irc.oftc.net #zumastor log beginning Sun Jun 1 00:00:01 PDT 2008 2008-06-01 04:35 -!- zumbi_(~zumbi@94.132.219.87.dynamic.jazztel.es) has joined #zumastor 2008-06-01 04:36 -!- zumbi_(~zumbi@94.132.219.87.dynamic.jazztel.es) has left #zumastor 2008-06-01 04:44 -!- zumbi_(~zumbi@94.132.219.87.dynamic.jazztel.es) has joined #zumastor 2008-06-01 06:51 -!- MaZe(~MaZe@c-24-6-86-168.hsd1.ca.comcast.net) has joined #zumastor 2008-06-01 08:08 -!- phoenix24(~phoenix24@122.167.69.80) has joined #zumastor 2008-06-01 13:05 -!- pgquiles(~pgquiles@130.Red-83-39-63.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-01 14:09 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-01 16:07 -!- pgquiles__(~pgquiles@130.Red-83-39-63.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-01 16:24 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-01 17:33 -!- pgquiles__(~pgquiles@130.Red-83-39-63.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-01 22:31 -!- phoenix24(~phoenix24@122.167.86.64) has joined #zumastor 2008-06-01 23:20 kernel panics are fun.. 2008-06-01 23:21 raid6 and sata_mv do not get along 2008-06-01 23:32 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-01 23:36 People are using zumastor in production since 1 hour ago 2008-06-01 23:36 ACTION crosses fingers :-) 2008-06-01 23:57 -!- pgquiles__(~pgquiles@130.Red-83-39-63.dynamicIP.rima-tde.net) has joined #zumastor irc.oftc.net #zumastor log beginning Mon Jun 2 00:00:01 PDT 2008 2008-06-02 00:23 -!- erwan_taf(~erwan@LAubervilliers-151-13-63-69.w217-128.abo.wanadoo.fr) has joined #zumastor 2008-06-02 05:52 -!- phoenix24(~phoenix24@122.167.70.243) has joined #zumastor 2008-06-02 08:08 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-02 08:53 -!- jiayingz(~jiayingz@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-02 08:55 zissues: Issue 150 in zumastor: Zumastor + VMWare Server+ Windows = hang 2008-06-02 10:18 -!- phoenix24(~phoenix24@122.167.86.64) has joined #zumastor 2008-06-02 10:20 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-02 11:50 -!- pgquiles(~pgquiles@130.Red-83-39-63.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-02 11:58 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-02 12:53 -!- phoenix24(~phoenix24@122.167.119.237) has joined #zumastor 2008-06-02 13:01 -!- pgquiles__(~pgquiles@130.Red-83-39-63.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-02 13:15 -!- dkegel(~chatzilla@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-02 13:29 -!- dld(~dld@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-02 14:30 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-02 15:25 03jiahotcake * r1674 10/trunk/ddsnap/kernel/dm-ddsnap.c: 2008-06-02 15:25 Add header file in dm-ddsnap.c to 2008-06-02 15:25 fix the compliling error introduced in commit 1672. 2008-06-02 15:32 03williamanowak * r1675 10/trunk/ (cbtb/uml/build.sh buildcurrent.sh cbtb/uml/smoke-test.sh): 2008-06-02 15:32 Update build/testing tools to detect/choose ARCH and DIST with a more 2008-06-02 15:32 appropriate outcome 2008-06-02 17:24 03jiahotcake * r1676 10/trunk/ddsnap/patches/2.6.24.2/ddsnap.base.patch: 2008-06-02 17:24 Export dm_copy_name_and_uuid in dm-ioctl.c so the kernel 2008-06-02 17:24 builds when dm is configured as a module. This is to fix 2008-06-02 17:24 the bug introduced by revision 1672. 2008-06-02 17:30 so, after resizing the snapstore, I did have to stop/start zumastor 2008-06-02 18:12 -!- jiayingz(~jiayingz@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-02 21:01 -!- phoenix24(~phoenix24@122.167.119.237) has joined #zumastor 2008-06-02 21:29 -!- phoenix24_(~phoenix24@122.167.107.201) has joined #zumastor 2008-06-02 21:54 my mistake, forgot zumastor resize 2008-06-02 22:43 -!- phoenix24(~phoenix24@122.167.70.243) has joined #zumastor irc.oftc.net #zumastor log beginning Tue Jun 3 00:00:01 PDT 2008 2008-06-03 00:38 -!- erwan_taf(~erwan@LAubervilliers-151-13-63-69.w217-128.abo.wanadoo.fr) has joined #zumastor 2008-06-03 02:00 -!- phoenix24(~phoenix24@122.167.86.64) has joined #zumastor 2008-06-03 02:30 -!- phoenix24(~phoenix24@122.167.86.64) has joined #zumastor 2008-06-03 04:53 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-03 06:24 -!- pgquiles__(~pgquiles@130.Red-83-39-63.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-03 08:50 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-03 08:52 -!- phoenix24(~phoenix24@122.167.86.64) has joined #zumastor 2008-06-03 09:16 -!- phoenix24(~phoenix24@122.167.86.64) has joined #zumastor 2008-06-03 09:30 -!- phoenix24(~phoenix24@122.167.86.64) has joined #zumastor 2008-06-03 10:17 03jiahotcake * r1677 10/trunk/test/large_volume_copy_test.sh: 2008-06-03 10:17 Update large_volume_copy test to refelct recent changes 2008-06-03 10:17 in zumastor package build. Also change ssh access checking 2008-06-03 10:17 to not ask for passwd. 2008-06-03 11:05 -!- pgquiles__(~pgquiles@130.Red-83-39-63.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-03 11:29 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-03 11:43 -!- MaZe(~MaZe@216-239-45-4.google.com) has left #zumastor 2008-06-03 11:51 -!- dkegel(~chatzilla@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-03 12:52 -!- natalie(~nataliep@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-03 13:33 -!- mitchg(~mitchg@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-03 13:47 03jiahotcake * r1678 10/trunk/test/uml/ (config_uml build_fs_root.sh build_uml.sh): Update uml test scripts to work with the current package build. 2008-06-03 13:47 03williamanowak * r1679 10/trunk/test/large_volume_copy_test.sh: Fix one spelling mistake, updated function definitions to be non-bashisms 2008-06-03 14:55 03williamanowak * r1680 10/trunk/ (openvz cbtb/uml/test-zuma-uml.sh): 2008-06-03 14:55 - Get rid of openvz cruft 2008-06-03 14:55 - uml test kernel logging patch from Steve VanDeBogart 2008-06-03 14:55 2008-06-03 16:02 zbuild: zumastor b0.9.0 r1676 build success 2008-06-03 16:10 hooray 2008-06-03 16:11 willn: there's a problem with zumastor-releases and gutsy 2008-06-03 16:11 they ignored my request for more quota there.. 2008-06-03 16:11 essentially, "copy packages" is broken 2008-06-03 16:12 ignored? I was given 5GB, it's my second increase :-? 2008-06-03 16:12 I opened a question and got no answer 2008-06-03 16:12 anyway, copying a source package is broken 2008-06-03 16:12 willn: they'll answer with approved or declined 2008-06-03 16:14 borken "copy packages" thread: http://thread.gmane.org/gmane.comp.cms.launchpad.user/3647 2008-06-03 16:14 *broken 2008-06-03 16:14 I got one approval, and one no answer 2008-06-03 16:14 https://answers.edge.launchpad.net/soyuz/+question/33228 2008-06-03 16:14 how long since you asked for more space? 2008-06-03 16:20 ~18 days? 2008-06-03 16:20 mm odd 2008-06-03 16:21 asked again. 2008-06-03 16:21 this time with a direct subject line 2008-06-03 16:21 ok 2008-06-03 16:22 how are the hardy and gutsy packages submitted: two different source uploads or one source upload + one copy source package ? 2008-06-03 16:22 zbuild: zumastor b0.9.0 r1676 install success 2008-06-03 16:23 zbuild: zumastor b0.9.0 r1680 build success 2008-06-03 16:26 one source, one copy 2008-06-03 16:26 it doesn't get happy with two uploads of the same version for different releases 2008-06-03 16:27 willn: one source, one copy is broken and ack'd by lauchpad developers 2008-06-03 16:27 the package is built but it's never made available by launchpad 2008-06-03 16:27 read this thread: http://thread.gmane.org/gmane.comp.cms.launchpad.user/3647 2008-06-03 16:28 zbuild: zumastor b0.9.0 r1680 install success 2008-06-03 16:29 the only possible fix atm is to have two source uploads with slightly different names in the changelog, for instance "zumastor 0.8.0~zumastor-gutsy" and "zumastor 0.8.0~zumastor-hardy" 2008-06-03 16:29 "changelog" meaning debian/changelog 2008-06-03 16:29 with copy packages, packages are built and transferred to the archive, but the Packages.gz does not include them 2008-06-03 16:30 I had a lot of trouble with that in my PPA until I discovered the fix myself :-/ 2008-06-03 16:45 ick. 2008-06-03 16:45 I think we may run with the 'only build for one release' idea for a while 2008-06-03 16:45 releases has a quota update 2008-06-03 16:46 granted? 2008-06-03 16:47 yep 2008-06-03 16:47 well, we are actually thinking in moving back to gutsy due to the bad performance of hardy and the vmware problems 2008-06-03 16:47 Since zumastor has moved to 2.6.24, things will get dicey 2008-06-03 16:47 discarded, then 2008-06-03 16:48 I think the problem with vmware server is zumastor writes are so slow with 2.6.24, after some time, when vmware tries to write, it hits some deadlock 2008-06-03 16:48 gutsy works flawlessly and hardy on a non-zumastored volume too 2008-06-03 16:49 vandebo: you may wish to export CONCURRENCY_LEVEL to `grep -c ^processor /proc/cpuinfo` to make make-kpkg speed up a bit 2008-06-03 16:49 that and or ccache, but ccache on nfs gets ugly fast 2008-06-03 16:49 you can point ccache to local disk 2008-06-03 16:50 None of our big source trees should be on nfs, IMHO 2008-06-03 16:50 willn: yea, I had put that change into at least one of my trees 2008-06-03 16:50 dkegel: agreed 2008-06-03 16:59 I keep everything on nfs, and just have some bits in my zsh config to put the right env variables in to make sure caches dont end up there 2008-06-03 17:00 worked for me until I ran out of quota :-) 2008-06-03 17:01 also symlinked zumastor/build to /var/tmp/zbuilds 2008-06-03 17:01 since that directory gets OMGHUGE in 3 or 4 builds 2008-06-03 17:24 turns out zumastor does not like running on top of lvs undergoing a pvmove 2008-06-03 17:25 it does not like it so much that it will eat your meachine 2008-06-03 17:27 Probably should be some functionality to prevent zumastor from invoking ddsnap status >1 time 2008-06-03 17:27 zumathumper1% ps -ef|grep -c 'ddsnap status' 2008-06-03 17:27 = 5 (all hung) 2008-06-03 17:32 ok, time to shoot it in the head 2008-06-03 17:33 zumastor faithfully started more ddsnap processes, even after telinit 6 2008-06-03 17:47 pgquiles, could you try 'echo 0 > /proc/sys/debug/exception-trace' in ur 2.6.24.2 kernel and see if that speeds up zumastor writes 2008-06-03 17:48 ping pgquiles__ 2008-06-03 17:48 jiayingz: pong 2008-06-03 17:49 jiayingz: I'll try that tomorrow, thanks 2008-06-03 17:49 so the vmware hanging problem only happens on 2.6.24.2 zumastor-ed volume? 2008-06-03 17:49 yes 2008-06-03 17:49 (at's 3AM here :-) ) 2008-06-03 17:49 *it's 2008-06-03 17:49 go to bed 2008-06-03 17:50 "yes, mom" 2008-06-03 17:50 :-D 2008-06-03 17:50 :) 2008-06-03 17:50 yes, I never saw the problem with 2.6.22 and I've checked again this afternoon - everything works fine 2008-06-03 17:51 but it's not as easily reproducible as I wrote in the issue tracker 2008-06-03 17:51 we found zumastor performance is about 30~50% slower with 2.6.24.2 kernel 2008-06-03 17:51 yesterday I was able to install 1 windows xp, 1 windows 2003 server and 1 ubuntu and it worked mostly fine, until a fourth vm brought everything down 2008-06-03 17:51 yes, I also noticed that 2008-06-03 17:52 that almost undos all of our optimization effort :( 2008-06-03 17:52 any luck in finding why? 2008-06-03 17:52 that exception-trace seems to count for half of the overhead 2008-06-03 17:52 what does it do? 2008-06-03 17:53 tracing for debugging 2008-06-03 17:53 not sure why it adds so much overhead 2008-06-03 17:56 well, it's time I go to bed. I'll test disabling exception tracing and report. 2008-06-03 17:56 thx! have a good night 2008-06-03 18:25 -!- mitchg(~mitchg@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-03 18:38 zbuild: zumastor b0.9.0 r1676 test failure 2 2008-06-03 19:53 zbuild: zumastor b0.9.0 r1680 test failure 2 2008-06-03 20:46 I wish I knew what was wrong with the dual node tests... 2008-06-03 23:18 -!- MaZe(~MaZe@c-24-6-86-168.hsd1.ca.comcast.net) has joined #zumastor irc.oftc.net #zumastor log beginning Wed Jun 4 00:00:01 PDT 2008 2008-06-04 00:13 -!- phoenix24(~phoenix24@122.167.86.64) has joined #zumastor 2008-06-04 00:57 -!- erwan_taf(~erwan@81.80.43.67) has joined #zumastor 2008-06-04 05:26 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-04 06:10 -!- pgquiles__(~pgquiles@130.Red-83-39-63.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-04 06:44 pgquiles__: ? 2008-06-04 06:45 zumbi_: yes? 2008-06-04 06:45 hola 2008-06-04 06:45 joer, qué pequeño es el mundo :-) 2008-06-04 06:45 :-) 2008-06-04 06:49 siempre somos los mismos en diferentes lugares... vi un mensaje tuyo de pasada en debian mentors :-) 2008-06-04 06:49 sí, 2008-06-04 06:49 bien bien.. seguimos en el buen camino 2008-06-04 06:49 de hecho tengo que subir una nueva versión de un paquete, a ver si consigo que alguien me lo suba al archivo 2008-06-04 06:49 :-D 2008-06-04 06:49 que paquete? 2008-06-04 06:50 yo no te lo puedo subir, pero conozco gente que a lo mejor se presta 2008-06-04 06:50 witty 2008-06-04 06:50 http://webtoolkit.eu 2008-06-04 06:52 cómo has ido a parar en zumastor? 2008-06-04 06:53 interesante paquete 2008-06-04 06:53 :-) 2008-06-04 06:53 dejé rails cuando descubrí wt 2008-06-04 06:53 jeje 2008-06-04 06:54 deberia estar en debian.. a ver si encontramos sponsor 2008-06-04 06:54 lo de zumastor, no se, creo que lei algo, ademas conoci Drake (dld) en una conferencia 2008-06-04 06:55 a ver si subo la versión nueva, que usa dpatch en vez de aplicar los parches a pelo, y soluciona un par de problemas de empaquetado de upstream, y ya pido que me lo suban 2008-06-04 06:55 ah, yo no conozco a nadie en persona 2008-06-04 06:55 pero llevo desde noviembre echando una mano 2008-06-04 06:55 este lunes íbamos a ponerlo en producción, pero surgió un problemilla tonto (nada que ver con zumastor) y se retrasa al lunes que viene 2008-06-04 06:56 bien.. esta gente es el alma del buscador este tan famosillo 2008-06-04 06:56 sí, y además son muy majos y siempre dispuestos a ayudar 2008-06-04 06:56 me entrevistaron para ir para alli, hable con kegel, pero parece que no fue muy bien 2008-06-04 06:56 y no me cogieron 2008-06-04 06:57 a mí kegel me envió un e-mail por si era estudiante y quería ir allí de becario, pero no era mi caso 2008-06-04 06:57 lo mismo, pero yo si era estudiante :-) 2008-06-04 06:58 bueno, yo sobre el papel todavía lo soy 2008-06-04 06:58 todavía no he leído el pfc 2008-06-04 06:58 :-/ 2008-06-04 06:58 llevo 5 años currando 2008-06-04 06:58 pues entonces si eres estudiante, podrias ir :-) 2008-06-04 06:58 tengo que leer YA porque temo que me pille (o me haya pillado ya) el cambio de plan de estudios :-( 2008-06-04 06:58 yo lei en febrero 2008-06-04 06:58 yo podía haber leído hace 2 años 2008-06-04 06:59 yo me fui a Barcelona 2008-06-04 06:59 pero no estaba 100% contento de mi pfc y esperé a algunas cosas de KDE4... y ahora mi pfc está totalmente desfasado 2008-06-04 06:59 yo no 2008-06-04 06:59 a mitad estudios y acabe alli en la UPC, he estado currando alli una larga temporada.. como tu, hasta que porfin di el paso y lei 2008-06-04 07:00 como tu, me refiero a currando sin leer 2008-06-04 07:00 yo estuve casi 4 años currando en una empresa muy pequeña de valencia, y luego, un buen día, me llegó un e-mail de otra empresa que estaba muy itneresada en mí 2008-06-04 07:00 y cambié 2008-06-04 07:00 y ahora estoy la mar de contento 2008-06-04 07:00 estuve año y medio currando en valencia para ellos, y ahora curro desde casa (novelda) 2008-06-04 07:00 y voy a valencia un día cada 2-3 semanas 2008-06-04 07:01 a qué te dedicas ahora? 2008-06-04 07:01 a mi me da yuyu el tema de las empresas.. he estado trabajando en la uni, en un proyecto (haciendo hardware), hasta que se acabaron los fondos (navidades) y me volvi para valencia... como el curro se paga tan mal, estoy intentando ir por libre, de freelance 2008-06-04 07:02 ahora me he venido a UK a trabajar una semanita en el proyecto que llevamos a cabo de software libre 2008-06-04 07:02 yo he tenido mucha suerte, estuve muy bien en la que estaba (incluso me ofrecieron el 10% de participación si me quedaba) y estoy muy bien ahora 2008-06-04 07:03 a que os dedicais? 2008-06-04 07:03 si no fuera por mi novia me hubiera ido a irlanda, alemania o escandinavia, obviamente 2008-06-04 07:03 software de control 2008-06-04 07:03 pero yo hago I+D 2008-06-04 07:03 interesante 2008-06-04 07:03 básicamente, me pagan para que pruebe todo lo que se me ocurra y que pueda ser útil en algún momento 2008-06-04 07:03 yo tambien me quiero quedar por Valencia :-) 2008-06-04 07:04 así que en realidad no sé cuándo estoy currando y cuando estoy de ocio, porque hago 99% lo mismo :-D 2008-06-04 07:04 en qué proyecto estás currando? 2008-06-04 07:04 si, te entiendo 2008-06-04 07:04 en debian para embebidos, emdebian.org 2008-06-04 07:05 ah, conozco a un chaval de cartagena que curra en algo parecido, pero en madrid 2008-06-04 07:05 por cierto, yo estoy por la zona de sagunto 2008-06-04 07:05 raúl siles, le conoces? 2008-06-04 07:06 aka kebianizao 2008-06-04 07:06 me suena de algo... pero lo mas seguro que sea... ah!! siii 2008-06-04 07:06 este es el de tuenti ? 2008-06-04 07:06 tuenti? 2008-06-04 07:06 tuenti es una red social, como el facebook 2008-06-04 07:07 ah, no lo sé 2008-06-04 07:07 pero no creo 2008-06-04 07:07 bueno.. pal caso, este chaval lo veo por ahi, por los canales de irc 2008-06-04 07:07 pues dí una charla de zumastor en la UPV hace un par de meses :-D 2008-06-04 07:08 ostras.. del ieee? 2008-06-04 07:08 no, polinux 2008-06-04 07:08 del ieee no sé nada desde hace mucho 2008-06-04 07:08 es que hace mucho que no aparezco por teleco 2008-06-04 07:08 mejor.. jeje 2008-06-04 07:09 y como también estaba haciendo informática, pues tengo más contacto con la gente de polinux 2008-06-04 07:09 pues de haberme enterado de la charla, habria ido 2008-06-04 07:09 mejor? terremoto en ieee? qué fue de la gente que había allí? 2008-06-04 07:10 ni idea.. no tenia mucha relacion con ellos, con tormo y rapul 2008-06-04 07:10 muchos andan por la gmlist 2008-06-04 07:10 gmlist? 2008-06-04 07:10 te acuerdas.. que en nuestra epoca empezamos a jugar con lo de las listas en ente? pues al final se creo una lista en google 2008-06-04 07:10 la gmlist 2008-06-04 07:11 se pasan el dia enviando chorradas 2008-06-04 07:11 ah, ni idea 2008-06-04 07:11 recibía taaaanto correo que al final me desuscribí de un porrón de listas 2008-06-04 07:11 si quieres apuntarte... puedo preguntar 2008-06-04 07:11 y aún así recibo muuuucho más de lo que puedo digerir :-D 2008-06-04 07:11 es privada 2008-06-04 07:12 vale la pena suscribirse? 2008-06-04 07:12 yo tengo un filtro para quitarlas de la bandeja de entrada.. y cuando quiero leer chorraditas.. pues le doy a la etiqueta y ale... 2008-06-04 07:12 :-D 2008-06-04 07:12 pues lo cierto es que no le hago mucho caso 2008-06-04 07:12 por ver a gente que conoces y eso 2008-06-04 07:13 por nada mas 2008-06-04 07:13 yo es que tuve un problema con ellos 2008-06-04 07:13 con gololo 2008-06-04 07:13 me cancelo mi cuenta en morgoth 2008-06-04 07:13 sin avisar ni nada, por todo el morro 2008-06-04 07:14 y me queje.. le compare con un troll.. 2008-06-04 07:14 yo ví que un buen día habían formateado la máquina, instaladado gentoo y ya 2008-06-04 07:14 pero bueno, tampoco usaba morgoth para nada, así que pasé 2008-06-04 07:14 si, gololo, jose maria 2008-06-04 07:14 sí, aznar 2008-06-04 07:15 el caso es que yo si lo hacia.. y el tio se llevo sus piezas para actualizar gololo y se lo apropio 2008-06-04 07:15 pff 2008-06-04 07:15 la gente del ieee (los nuevos) decian que morgoth era suyo 2008-06-04 07:15 y que estaba en su derecho de hacer lo que le diera la gana 2008-06-04 07:16 parece que la memoria histórica es importante :-) 2008-06-04 07:17 fue algo muy fuerte... ademas como el estaba alli todos los dias y tenia relacion con los nuevos del ieee, y se sintio afectado por mis comentarios, todos salieron en defensa suya 2008-06-04 07:17 yo al final recupere mi cuenta, pero no quise saber mas de ellos 2008-06-04 07:17 y rapul, tormo, etc? 2008-06-04 07:17 rapul estaba haciendo pruebas por ds2 2008-06-04 07:18 tormo ni idea.. y muchos otros siguen por teleco 2008-06-04 07:18 el german ese, mazcuñan, .. 2008-06-04 07:18 javito 2008-06-04 07:19 umm 2008-06-04 07:20 el rafa siles este, esta en #dudes en oftc 2008-06-04 07:20 pasate por alli, hay algunos DDs 2008-06-04 07:24 bueno, me alegro de xarlar contigo.. a ver si coincidimos en carne y hueso... y para lo que necesites por aqui me encontraras :-) 2008-06-04 07:25 :-D 2008-06-04 07:25 coincidir va a ser difícil :-) 2008-06-04 07:28 en alguna conf 2008-06-04 07:28 podria ser 2008-06-04 09:27 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-04 10:00 -!- pgquiles__(~pgquiles@130.Red-83-39-63.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-04 11:13 hi pgquiles 2008-06-04 11:14 jiayingz: hi 2008-06-04 11:14 i don't think u need to try turning off exception-trace 2008-06-04 11:14 that does not seem to be relavent 2008-06-04 11:15 :-( 2008-06-04 11:15 it seems that one of the vmware crashes damaged one of the zumastored volumes, now deltas cannot be applied 2008-06-04 11:15 can I fix that using ddsnap? 2008-06-04 11:15 dropping the delta, maybe? 2008-06-04 11:15 i got some worse performance yesterday. then it became a little better. i thought that was because i turned off exception-trace 2008-06-04 11:15 but it remained the same after i turned it back 2008-06-04 11:16 cold cache, maybe? 2008-06-04 11:16 not sure. does not seem like 2008-06-04 11:16 so that happened on downstream? 2008-06-04 11:17 checksum error 2008-06-04 11:17 you got replication errors after vmware crashed? 2008-06-04 11:18 after vmware crashed, the system was dog slow and vmware processes went defunct 2008-06-04 11:18 you may need to restart replication. i mean start from initial one 2008-06-04 11:18 do you mean removing the volume and re-creating it downstream? 2008-06-04 11:18 but vmware crashes should not damage the zumastor volumes 2008-06-04 11:19 that is one way to do that 2008-06-04 11:19 well, I needed to kill ddsnap ungracefully to be able to reboot the machine :-/ 2008-06-04 11:19 or you can echo -1 to /var/lib/zumastor/$vol/source/hold 2008-06-04 11:19 that's much better :-) 2008-06-04 11:19 even that should not leave ddsnap in an inconsistent state 2008-06-04 11:20 wait a sec and I'll post the error here 2008-06-04 11:20 thx! 2008-06-04 11:21 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-04 11:22 Wed Jun 4 15:05:56 2008: [6210] daemonize: starting at Wed Jun 4 15:05:56 2008 2008-06-04 11:22 got client connection 2008-06-04 11:22 processing 2008-06-04 11:22 Wed Jun 4 15:06:00 2008: [6211] apply_delta_extents: delta header checksum '1186031', actual checksum '791523' 2008-06-04 11:22 Wed Jun 4 15:06:00 2008: [6211] apply_delta_extents: checksum failed for 1 chunk extent with start address of 112742891520 snapshot0 is not the same on the upstream and the downstream 2008-06-04 11:22 Wed Jun 4 15:06:00 2008: [6211] ddsnap_delta_server: closing connection on error: unable to apply upstream delta to device "/dev/mapper/zumavm" 2008-06-04 11:22 Wed Jun 4 15:06:00 2008: [6210] ddsnap_delta_server: unable to accept connection: Interrupted system call 2008-06-04 11:22 Wed Jun 4 15:06:00 2008: [6210] ddsnap_delta_server: Caught signal 17 2008-06-04 11:22 uh, pastebin would've been better O:-) 2008-06-04 11:23 could u also post server log? 2008-06-04 11:25 downstream server.log: http://pastebin.com/m42bb179c 2008-06-04 11:25 zumastor status upstream: http://pastebin.com/m78510d35 (only relevant part) 2008-06-04 11:25 and downstream: http://pastebin.com/d35cf670f 2008-06-04 11:26 upstream is stuck sending chunk 191324 2008-06-04 11:26 and downstream is stuck in "apply: 38 0/238" 2008-06-04 11:29 does source.log say anything about snapshot 38 before the crash? 2008-06-04 11:30 no 2008-06-04 11:30 actually snapshot 38 was taken after the crash 2008-06-04 11:30 two days before 2008-06-04 11:31 after the crash? when did crash hanppen? 2008-06-04 11:31 replication started failing 2 days ago but only for that volume, which was the one VMWare blocked 2008-06-04 11:31 right 2008-06-04 11:31 :) 2008-06-04 11:32 I guess some blocks are inconsistent due to the crash 2008-06-04 11:32 i wonder if that is a corner case we did not cover in replication. 2008-06-04 11:32 that's why the deltas are not applying, they don't find the blocks they are looking for 2008-06-04 11:32 I don't know 2008-06-04 11:33 if it's useful, these servers are using hardware RAID with battery-backed cache 2008-06-04 11:33 may due to a write error to the disk 2008-06-04 11:33 need to go to a meeting. could u file an issue for this. i will take a closer look after i come back 2008-06-04 11:36 ok 2008-06-04 11:36 thanks 2008-06-04 11:40 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-04 12:00 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-04 12:15 -!- pgquiles__(~pgquiles@130.Red-83-39-63.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-04 13:25 -!- phoenix24(~phoenix24@122.167.71.233) has joined #zumastor 2008-06-04 14:29 -!- zumalog(~zumalog@yzf.shapor.com) has joined #zumastor 2008-06-04 15:04 03williamanowak * r1681 10/trunk/cbtb/tests/1/bashisms.sh: 2008-06-04 15:04 Update the bashisms test to do a little detection before wreaking havoc on the 2008-06-04 15:04 test environment. 2008-06-04 15:09 03williamanowak * r1682 10/trunk/cbtb/tests/1/bashisms.sh: On second thought, let's remove the bashisms test all together. 2008-06-04 15:10 ACTION remembers doing that a while ago 2008-06-04 15:12 ACTION hands shapor a cookie 2008-06-04 15:29 zbuild: zumastor b0.9.0 r1681 build failure 1 2008-06-04 15:30 bah 2008-06-04 15:30 apt phail 2008-06-04 15:30 local mirror failed? (haven't read it) 2008-06-04 15:34 Yes 2008-06-04 15:34 Our 2.6.24 seems to really suck. 2008-06-04 15:34 ddsnap inflight bio vecs 1000, pending requests: 533, query requests 467, release requests 0, locked requests 0 2008-06-04 15:36 ddsnap inflight bio vecs 769, pending requests: 533, query requests 236, release requests 0, locked requests 0 2008-06-04 15:36 all together not fun 2008-06-04 15:37 also 2008-06-04 15:37 15:37:25 up 22:06, 2 users, load average: 9.71, 10.72, 8.78 2008-06-04 15:49 zbuild: zumastor b0.9.0 r1682 build failure 1 2008-06-04 16:26 03williamanowak * r1683 10/trunk/test/large_volume_copy_test.sh: 2008-06-04 16:26 Update the large volume copy test 2008-06-04 16:26 - accept configuration with environment variables 2008-06-04 16:26 - a few small style changes 2008-06-04 16:27 -!- MaZe(~MaZe@c-24-6-86-168.hsd1.ca.comcast.net) has joined #zumastor 2008-06-04 16:34 -!- phillips_(~phillips@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-04 16:44 zbuild: zumastor b0.9.0 r1683 build success 2008-06-04 16:59 03jiahotcake * r1684 10/trunk/ddsnap/ (buffer.c ddsnapd.c buffer.h): (log message trimmed) 2008-06-04 16:59 Use deferred metadata writes in ddsnap commit_transaction. 2008-06-04 16:59 Basically, after a dirty buffer is written to the on-disk 2008-06-04 16:59 journal, it is moved to a new list, called journaled_buffers. 2008-06-04 16:59 Instead of writing the buffer data to metadata store directly, 2008-06-04 16:59 we flush the journaled_buffers list when the list is half full. 2008-06-04 16:59 According to my measurements, the patch provides about 5-10% 2008-06-04 16:59 zbuild: zumastor b0.9.0 r1683 install success 2008-06-04 17:19 zbuild: zumastor b0.9.0 r1684 build success 2008-06-04 17:25 zbuild: zumastor b0.9.0 r1684 install success 2008-06-04 17:53 03vandebo * r1685 10/trunk/ddsnap/kernel/dm-ddsnap.c: Fix an off by 2 memory allocation. 2008-06-04 18:35 -!- MaZe(~MaZe@c-67-180-254-133.hsd1.ca.comcast.net) has joined #zumastor 2008-06-04 18:46 -!- dank(~chatzilla@cpe-76-94-199-31.socal.res.rr.com) has joined #zumastor 2008-06-04 18:50 zbuild: zumastor b0.9.0 r1683 test failure 2 2008-06-04 19:50 zbuild: zumastor b0.9.0 r1684 test failure 2 2008-06-04 20:05 -!- MaZe(~MaZe@c-24-6-86-168.hsd1.ca.comcast.net) has joined #zumastor 2008-06-04 21:14 -!- phoenix24(~phoenix24@122.167.77.33) has joined #zumastor 2008-06-04 21:52 -!- dank(~chatzilla@cpe-76-94-199-31.socal.res.rr.com) has joined #zumastor 2008-06-04 22:25 -!- phoenix24(~phoenix24@122.167.99.105) has joined #zumastor 2008-06-04 23:32 -!- phoenix24(~phoenix24@122.167.75.152) has joined #zumastor irc.oftc.net #zumastor log beginning Thu Jun 5 00:00:01 PDT 2008 2008-06-05 02:53 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-05 07:43 -!- jiayingz(~jiayingz@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-05 08:15 zbuild: zumastor b0.9.0 r1685 build success 2008-06-05 08:51 zbuild: zumastor b0.9.0 r1685 install success 2008-06-05 08:55 ACTION blinks 2008-06-05 08:58 -!- mitchg_(~mitchg@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-05 10:51 zbuild: zumastor b0.9.0 r1685 test success 2008-06-05 11:03 willn: ping 2008-06-05 11:04 pong 2008-06-05 11:05 are the scripts for building the ubuntu source packages (ddsnap, zumastor, kernel) in svn? 2008-06-05 11:07 Yea 2008-06-05 11:07 buildcurrentsrc 2008-06-05 11:07 and 2008-06-05 11:09 hmm 2008-06-05 11:09 where did that other one go 2008-06-05 11:09 zumastor/kernel/ubuntu-package/flavormaker.py 2008-06-05 11:11 ok, thanks 2008-06-05 11:11 I may try to upgrade the kernel to from 2.6.24-16.30 to the latest -server (2.6.24-19.33) 2008-06-05 11:14 alright. Let me know if flavormaker gives you troubles. It should be helpful in the generic sense now 2008-06-05 11:14 ok 2008-06-05 11:36 zbuild: zumastor b0.9.0 r1688 install success 2008-06-05 11:36 zbuild: zumastor b0.9.0 r1688 build success 2008-06-05 13:22 03williamanowak * r1686 10/trunk/cbtb/tests/1/ddsnap-kernel-namespace.sh: 2008-06-05 13:22 Fix the ddsnap-kernel-namespace test. 2008-06-05 13:22 grep returns with exit status 1 if no matches are found, which in this case is 2008-06-05 13:22 a good thing.. Add a few words to fix that bug, and the test should pass as of 2008-06-05 13:22 the changes in r1672 2008-06-05 13:23 03williamanowak * r1687 10/trunk/cbtb/tests/1/ddsnap-kernel-namespace.sh: Remove rogue q 2008-06-05 13:23 03williamanowak * r1688 10/trunk/cbtb/tests/1/ddsnap-kernel-namespace.sh: Invalid newline 2008-06-05 13:23 03jiahotcake * r1689 10/trunk/ddsnap/patches/2.6.24.2/ (bio.throttle.patch ddsnap.base.patch dm.inflight.patch): 2008-06-05 13:23 Add a sysctl interface /proc/sys/dev/dm/bio_throttle to enable/disable 2008-06-05 13:23 bio throttling. bio throttling is turned off by default for now. 2008-06-05 13:23 Also changed the dm_inflight() function to get inflight bios from 2008-06-05 13:23 the gendisk structure so it does not depend on the bio_throttling patch 2008-06-05 13:23 and merged it with ddsnap.base.patch. 2008-06-05 13:50 03williamanowak * r1690 10/trunk/cbtb/tests/1/ddsnap-kernel-namespace.sh: 2008-06-05 13:50 Fix ddsnap-kernel-namespace test 2008-06-05 13:50 - vgremove does not take a -f option 2008-06-05 14:02 zbuild: zumastor b0.9.0 r1688 test failure 3 2008-06-05 15:39 -!- pgquiles__(~pgquiles@130.Red-83-39-63.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-05 15:57 03williamanowak * r1691 10/trunk/ (5 files in 4 dirs): 2008-06-05 15:57 Some small packaging complaints 2008-06-05 15:57 - Fix builds so they work on version control systems that ignore empty 2008-06-05 15:57 directories 2008-06-05 15:57 - add /var/log/zumastor and /var/run/zumastor/volumes to debian/dirs 2008-06-05 15:57 - add something to make directories in debian/dirs when run by hand [non-.deb] 2008-06-05 15:57 - Clean up buildcurrent a tiny bit 2008-06-05 16:28 zissues: Issue 151 in zumastor: ddsnap performance is slow with 2.6.24.2 kernel 2008-06-05 16:30 -!- natalie(~nataliep@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-05 22:30 -!- MaZe(~MaZe@c-24-6-86-168.hsd1.ca.comcast.net) has joined #zumastor 2008-06-05 22:56 03williamanowak * r1692 10/trunk/buildcurrent.sh: Missing export on env variables used by dch 2008-06-05 23:42 -!- phoenix24(~phoenix24@122.167.86.64) has joined #zumastor irc.oftc.net #zumastor log beginning Fri Jun 6 00:00:01 PDT 2008 2008-06-06 00:09 -!- phoenix24_(~gopi_daii@122.167.86.64) has joined #zumastor 2008-06-06 00:09 phoenix24, ping! 2008-06-06 02:07 zbuild: zumastor b0.9.0 r1689 build success 2008-06-06 02:12 zbuild: zumastor b0.9.0 r1692 build success 2008-06-06 02:18 zbuild: zumastor b0.9.0 r1689 install success 2008-06-06 02:38 zbuild: zumastor b0.9.0 r1692 install success 2008-06-06 03:23 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-06 04:11 -!- pgquiles__(~pgquiles@130.Red-83-39-63.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-06 04:17 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-06 04:43 zbuild: zumastor b0.9.0 r1689 test failure 3 2008-06-06 07:13 zbuild: zumastor b0.9.0 r1692 test success 2008-06-06 09:58 io wait :( 2008-06-06 10:28 branching for 0.9. Autobuild/test is stopped. 2008-06-06 10:28 hooray 2008-06-06 10:28 a great cheer goes up throughout the land 2008-06-06 10:41 03drake.diedrich * r1693 10/branches/0.9: Branching from trunk to branches/0.9 2008-06-06 10:57 03drake.diedrich * r1694 10/trunk/ (4 files in 4 dirs): 2008-06-06 10:57 Update trunk version to 0.10 2008-06-06 10:57 Doc to date how the 0.9 branch was created - further steps depend on 2008-06-06 10:57 this commit. 2008-06-06 11:01 -!- pgquiles__(~pgquiles@130.Red-83-39-63.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-06 11:31 ACTION idles while a dapper install under qemu happens 2008-06-06 11:32 03drake.diedrich * r1695 10/trunk/ (7 files in 4 dirs): 2008-06-06 11:32 Updates to docs, through building a new base image for the trunk. 2008-06-06 11:32 Add init scripts the used to reside in other repositories. 2008-06-06 11:32 Update the .start script that launches all of the current branch builds 2008-06-06 11:32 and lives on trunk with the trunk-specific build/test scripts. 2008-06-06 11:41 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-06 11:43 dld: You may want to revert that last commit... 2008-06-06 12:17 -!- phoenix24(~phoenix24@122.167.110.147) has joined #zumastor 2008-06-06 12:22 -!- phoenix24_(~phoenix24@122.167.92.233) has joined #zumastor 2008-06-06 12:44 -!- Gramphos(~viktor@gramphos.csbnet.se) has joined #zumastor 2008-06-06 12:52 willn: about the same as email headers and logfiles from that machine. Nothing I'm worried about. 2008-06-06 13:00 How would I debug deadlocks with diskaccess from a zumastor volume? 2008-06-06 13:03 Someone (probably not me) will need more details. deadlocks have been hard to debug. 2008-06-06 13:03 (the devs are taking a break while I'm getting the autobuilder back up after the branch) 2008-06-06 13:04 well, doing ls on the zumastor origon mount point hangs on getdents64 2008-06-06 13:04 and killing stuff locks the kernel 2008-06-06 13:04 I'd try a dd if=blah from the raw volume, to see that it's all readable still 2008-06-06 13:05 That'll at least separate block layer from filesystem layer. 2008-06-06 13:05 the underlying volume, or the zumastor volume? 2008-06-06 13:05 I'd try both. 2008-06-06 13:06 snapshotts are still accessable 2008-06-06 13:09 (zumastor was tested with killing each of the processes and seeing that it recovered, so this is bad news that it's dying on a kill) 2008-06-06 13:09 I was able to dd 10 bytes from the zumastor volume to a file, but was not able to read the file. :hmmm: 2008-06-06 13:10 Is this your own build, or one of the released ones? (there have been lots of corner cases with particular kernels) 2008-06-06 13:10 It's my own build. 2008-06-06 13:11 There were no releases for my distro 2008-06-06 13:11 Patched onto another kernel, or a build of 2.6.24.2 with all of the zumastor/ddsnap patches? 2008-06-06 13:11 patched vanilla 2.6.24.2 with zumastor patches 2008-06-06 13:12 Gramphos if the machine is locked, sysrq-t will be helpful 2008-06-06 13:12 Well, that ought not to lock up. :/ Same zumastor .config ? 2008-06-06 13:13 I only have remote access to the machine, so that doesn't help 2008-06-06 13:13 have to cycle power 2008-06-06 13:14 ouch. I hate that. Too bad you don't have two that can control each other. 2008-06-06 13:18 Gramphos: that will make debugging a deadlock close to impossible 2008-06-06 13:18 any chance of getting a serial console on it? 2008-06-06 13:18 are your interactive shells locked? 2008-06-06 13:19 if not you could run "echo t >/proc/sysrq-trigger" 2008-06-06 13:19 graphus, before you cycle power.... 2008-06-06 13:19 grampos I mean 2008-06-06 13:19 the system doesn't lock down directly. But after some time or possible additional actions. 2008-06-06 13:19 ah, good 2008-06-06 13:19 gramhpos, do you know about alt-sysrq-t? 2008-06-06 13:20 flipz: hes remote to the machine, no keybaord 2008-06-06 13:20 alt-sysrq-t should work remote 2008-06-06 13:20 right, via the echo command 2008-06-06 13:20 yes 2008-06-06 13:20 which i just mentioned ;) 2008-06-06 13:21 :) 2008-06-06 13:22 that's often all we need to track down a deadlock 2008-06-06 13:22 which version of zumastor? 2008-06-06 13:22 which kernel? 2008-06-06 13:22 flipz read scrollback 2008-06-06 13:22 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-06 13:23 zumastor 0.8 2008-06-06 13:24 shapor, we should consider falling back to 2.6.23.8 2008-06-06 13:25 Gramphos, what kind of device are you using for your origin/snapstore? 2008-06-06 13:26 and whats the hardware setup in general 2008-06-06 13:26 a loopback device to a 40 GB file, split with lvm into two 20 GB files 2008-06-06 13:26 loopback... 2008-06-06 13:26 ahh 2008-06-06 13:27 ok, I hope we can see a task trace 2008-06-06 13:27 I'll see what I can do. 2008-06-06 13:27 loopback volumes are deadlock prone all by themselves 2008-06-06 13:27 the system is up again. 2008-06-06 13:27 If I don't do anything it will probably lock in an hour or so 2008-06-06 13:27 yeah i'm highly suspicous of ddsnap on top of loopback 2008-06-06 13:27 i think i tried that before 2008-06-06 13:28 can't remember the results 2008-06-06 13:28 I use loopback only for basic testing, I would not rely on it 2008-06-06 13:28 shapor: works for UML tests, but that's still different from real hardware. 2008-06-06 13:28 dld: yeah but isnt it only loopback in the host kernel? 2008-06-06 13:29 well, gramphos, if we can see the ""echo t >/proc/sysrq-trigger" task trace we can determine whether the lockup is related to loopback 2008-06-06 13:29 Some of the tests loopback inside the client (17T - which now that you mention it is failing...) 2008-06-06 13:30 yeah the kernel trace will end up in your /var/log/kern.log or equivalent 2008-06-06 13:30 if the problem is indeed loopback I could take a look at what would be necessary to fix that problem. It's been a problem in linux since forever 2008-06-06 13:31 shapor, /var/log/messages even 2008-06-06 13:31 I think it goes both places by default 2008-06-06 13:32 depends on the distro 2008-06-06 13:33 went both places here\ 2008-06-06 13:34 I wonder why all that cfq garbage got added to the task trace 2008-06-06 13:36 is there anyway to repair a broken snap store, or should I just reinitialize it 2008-06-06 13:37 -!- pgquiles_(~pgquiles@130.Red-83-39-63.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-06 13:38 zbuild: zumastor b0.9.0 r1693 build failure 1 2008-06-06 13:41 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-06 13:49 gramphos, just reinitialize it 2008-06-06 13:50 gramphos, how do you know it is broken? 2008-06-06 13:50 it gives an error about superblock 2008-06-06 13:50 the error is? 2008-06-06 13:51 start_server: Invalid superblock: please run 'ddsnap-sb' first to upgrade the superblock. 2008-06-06 14:07 Gramphos, did u run a new version of ddsnap with an old version of snapshot store? 2008-06-06 14:07 no 2008-06-06 14:08 wonder how u got inconsistent snapshot store 2008-06-06 14:08 It did work before the crash, but got corrupted (at least that's what I think happened) 2008-06-06 14:08 looks like so, although ddsnap should be able to recover from corruption 2008-06-06 14:09 I guess cycling power while the kernel is locked up has something to do with it 2008-06-06 14:10 have u written anything to snapshot device directly? 2008-06-06 14:10 no 2008-06-06 14:11 hmm. i guess u have to re-initialize snapshot store in that case 2008-06-06 14:13 zbuild: zumastor b0.9.0 r1695 build success 2008-06-06 14:14 I already did that. 2008-06-06 14:16 could u try the same test and get a sysrq-t trace? 2008-06-06 14:17 also, what revisions are u using? 2008-06-06 14:17 It's up and running. It needs some load before it hangs. 2008-06-06 14:20 03drake.diedrich * r1696 10/trunk/cbtb/host-setup/dapper-late.sh: 2008-06-06 14:20 [trunk] Add all packages required by the zumastor build to the template image 2008-06-06 14:20 before trying to generate a new one on trunk. 2008-06-06 14:26 I think I use r1685 2008-06-06 14:27 could u also try 1689 and see if it fixes ur problem? 2008-06-06 14:29 zbuild: zumastor b0.9.0 r1695 install success 2008-06-06 14:39 03drake.diedrich * r1697 10/trunk/cbtb/host-setup/dapper-late.sh: Add several packages used by tests, reducing dependency on mirrors and runtime at test time. 2008-06-06 15:52 -!- tim_vimm(~Tim@cpe-76-90-98-247.socal.res.rr.com) has joined #zumastor 2008-06-06 15:53 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-06 16:04 03williamanowak * r1698 10/trunk/cbtb/tests/2/ (7 files): - Add cleanup section to dual node tests 2008-06-06 16:16 -!- pgquiles(~pgquiles@130.Red-83-39-63.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-06 16:49 zbuild: zumastor b0.9.0 r1695 test success 2008-06-06 18:10 03drake.diedrich * r1699 10/trunk/doc/release-branch.txt: More docs on rebuilding the trunk image. 2008-06-06 18:12 Full kernel builds are running for the 0.9 branch and trunk (both under qemu on the same machine. Should keep it busy until Saturday night or Sunday morning, and then hopefully both will succeed and we'll be back in business. 2008-06-06 20:39 Now I have a hanged system again 2008-06-06 20:44 this time I am however still able to read from the device. 2008-06-06 20:45 but can't ls the snapshots 2008-06-06 20:52 -!- MaZe(~MaZe@c-24-6-86-168.hsd1.ca.comcast.net) has joined #zumastor 2008-06-06 21:53 -!- MaZe(~MaZe@c-24-6-86-168.hsd1.ca.comcast.net) has joined #zumastor 2008-06-06 22:06 Gramphos are you able to echo t >/proc/sysrq-trigger and get the output of your log file? 2008-06-06 22:07 I got some output from the logfile yes, but have rebooted now 2008-06-06 22:07 should I post it to some pasetbin? 2008-06-06 22:10 go ahead and post it to our mailing list 2008-06-06 22:12 http://www.pastebin.org/41222 2008-06-06 22:13 I'll post it to the mailing list later 2008-06-06 22:13 that one is just before I reboot 2008-06-06 22:14 where I've been trying to kill off/restart several processes 2008-06-06 22:31 oh that works, thanks! 2008-06-06 22:34 i'd never heard of pastebin before, looks cool 2008-06-06 22:46 I think it did truncate it 2008-06-06 22:47 it appears to have the important part (ddsnap server trace), but it would be nice to have it all 2008-06-06 22:51 http://www.pastebin.org/41226 2008-06-06 22:54 is the mysql using the zumastor device? 2008-06-06 22:54 yes 2008-06-06 22:59 what distro are you running 2008-06-06 23:06 CentOS 4.6 irc.oftc.net #zumastor log beginning Sat Jun 7 00:00:01 PDT 2008 2008-06-07 04:57 -!- natalie(~nataliep@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-07 07:55 zbuild: zumastor b0.10.0 r1697 build success 2008-06-07 08:00 zbuild: zumastor b0.10.0 r1699 build success 2008-06-07 08:05 zbuild: zumastor b0.10.0 r1697 install success 2008-06-07 08:25 zbuild: zumastor b0.10.0 r1699 install success 2008-06-07 10:40 zbuild: zumastor b0.10.0 r1697 test success 2008-06-07 11:39 shapor, around? 2008-06-07 13:00 zbuild: zumastor b0.10.0 r1699 test failure 1 2008-06-07 16:46 -!- pgquiles(~pgquiles@150.Red-83-38-36.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-07 17:30 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-07 18:11 -!- pgquiles__(~pgquiles@150.Red-83-38-36.dynamicIP.rima-tde.net) has joined #zumastor irc.oftc.net #zumastor log beginning Sun Jun 8 00:00:01 PDT 2008 2008-06-08 00:41 -!- MaZe(~MaZe@c-24-6-86-168.hsd1.ca.comcast.net) has joined #zumastor 2008-06-08 12:26 -!- phoenix24(~phoenix24@122.167.66.171) has joined #zumastor 2008-06-08 12:56 -!- pgquiles(~pgquiles@192.Red-79-148-70.staticIP.rima-tde.net) has joined #zumastor 2008-06-08 13:38 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-08 16:30 -!- pgquiles__(~pgquiles@192.Red-79-148-70.staticIP.rima-tde.net) has joined #zumastor 2008-06-08 22:37 -!- Gramphos(~viktor@gramphos.csbnet.se) has joined #zumastor 2008-06-08 22:37 -!- erwan_taf(~erwan@81.80.43.67) has joined #zumastor irc.oftc.net #zumastor log beginning Mon Jun 9 00:00:01 PDT 2008 2008-06-09 00:10 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-09 06:43 -!- dank(~chatzilla@cpe-76-94-199-31.socal.res.rr.com) has joined #zumastor 2008-06-09 06:43 pgquiles_, what kind of load are you putting on zumastor when it locks up? 2008-06-09 06:45 dank: hi 2008-06-09 06:45 hi 2008-06-09 06:45 dank: the only thing I know is a coworker was trying to write hundreds of small files over Samba 2008-06-09 06:45 That must have been the shortest production deployment in history :-( 2008-06-09 06:45 but that might just be a coincidence 2008-06-09 06:45 dank: indeed :-/ 2008-06-09 06:46 it worked very well until it froze 2008-06-09 06:47 Um, did you happen to do alt-sysrq-t when the freeze happened, to get the kernel stacks? Or was it too hung? 2008-06-09 06:47 no, actually the kernel runs fine 2008-06-09 06:47 and I've got two virtual machines running on non-zumastored volumes 2008-06-09 06:47 Oh, ok. 2008-06-09 06:47 and several non-zumastored partitions shared with Samba and they work fine 2008-06-09 06:47 it's only the zumastor volume 2008-06-09 06:48 will killing "ddsnap create" corrupt data or snapshots? 2008-06-09 06:49 I don't know. Is that one of the things that's hung? 2008-06-09 06:49 BTW we have a stress test where we kill everything randomly and repeatedly, but I don't think it does any ddsnap creates. 2008-06-09 06:50 dank: it's not hung but in suspended on I/O 2008-06-09 06:51 root 30398 0.0 0.0 2032 520 ? S 14:54 0:00 ddsnap create /var/run/zumastor/servers/zumaf 68 2008-06-09 06:51 Jiaying is flying home for a couple weeks, so flips and shapor are the folks most likely to be able to help. I'm not hands-on enough. 2008-06-09 06:51 ok 2008-06-09 06:51 thanks anyway 2008-06-09 06:51 it must be really late over there, btw :-) 2008-06-09 06:51 7am! A fine hour. (Did you try the zumastor create after everything else had already hung?) 2008-06-09 06:52 "odd behavior" is a bit of an understatement. 2008-06-09 06:54 no, I did not fire ddsnap myself, it was automatically ran by zumastor after I ran "zumastor stop", which is why I said "odd": if I'm requesting zumastor to stop my volume, the last thing in the world I'm expecting is a snapshot being started :-/ 2008-06-09 06:55 7 am? oh, I don't know why I though it was about 4am :-o 2008-06-09 06:55 Oh. Yes, that's pretty odd. 2008-06-09 06:58 Maybe it was a scheduled snapshot? 2008-06-09 06:58 (I'm showing my ignorance here.) 2008-06-09 07:00 the logs show snapshots were taken daily around 14.55h, about the same time the snapshot started today 2008-06-09 07:00 that's OK 2008-06-09 07:00 oh, ok 2008-06-09 07:00 now I understand 2008-06-09 07:00 the snapshots are taken by a cron job, not by zumastor itself 2008-06-09 07:00 stopping zumastor does not prevent snapshots from being taken 2008-06-09 07:01 unless zumastor stop has already finished 2008-06-09 07:01 and thus the volume is no longer available 2008-06-09 07:02 in my case, I requested zumastor stop at 14.20 and it had not finished at 14.55, therefore the snapshot started 2008-06-09 07:02 at least now I know why the snapshot was started :-) 2008-06-09 10:30 -!- pgquiles__(~pgquiles@192.Red-79-148-70.staticIP.rima-tde.net) has joined #zumastor 2008-06-09 10:47 -!- phoenix24(~phoenix24@122.167.71.236) has joined #zumastor 2008-06-09 10:50 I'm building the latest ubuntu hardy kernel with jia's patch to remove the per-device throttle 2008-06-09 10:51 pgquiles: hello 2008-06-09 10:51 shapor: hi 2008-06-09 10:51 did you get a kernel traceback? 2008-06-09 10:52 impossible, gdb never shows me a prompt :-( 2008-06-09 10:52 oh, kernel 2008-06-09 10:52 dont need gdb 2008-06-09 10:52 yeah 2008-06-09 10:52 just t sysrq-trigger 2008-06-09 10:52 echo t >/proc/sysrq-trigger 2008-06-09 10:52 ok 2008-06-09 10:53 if not, useful ps output would be: ps axo state,comm,pid,wchan 2008-06-09 10:53 I'm 20% uploading the kernel to my PPA, when I'm done I'll VPN to the office and try sysrq 2008-06-09 10:53 cool! thanks 2008-06-09 10:53 its hard to tell why ddsnapd is blocking otherwise 2008-06-09 10:53 ddsnapd isn't supposed to block like that ;) 2008-06-09 10:54 :-) 2008-06-09 10:54 what kind of device are you using underneath zumastor? 2008-06-09 10:54 it was working perfectly fine until someone dumped hundreds of files over samba. After that, zumastor got really slow and after reboot it showed "status: failed" :-/ 2008-06-09 10:55 hm sounds like 2 bugs 2008-06-09 10:55 getting slow, and not recovering 2008-06-09 10:55 7.2Krpm SATA disks with HP P400 RAID controllers in RAID6, 512MB of BBWC 2008-06-09 10:55 just for completeness, is there any lvm or software raid underneath zumastor, or just raw block devices? 2008-06-09 10:56 I needed to reboot the server but I didn't kill -9 ddsnap or zumastor, I expected zumastor to recover 2008-06-09 10:56 yes, LVM2, no software RAID 2008-06-09 10:56 ok 2008-06-09 10:56 this worked great on gutsy with 0.8 :-( 2008-06-09 10:57 so you were running 0.9 on hardy, yes? 2008-06-09 10:57 no, 0.8, too 2008-06-09 10:57 gutsy + 0.8 was never into production but it behaved well in my stress tests 2008-06-09 10:58 and in fact hardy + 0.8 was working fine when dumpind data over rsync 2008-06-09 10:58 samba has triggered deadlocks for me in the past 2008-06-09 10:58 I'm not sure if Samba is also to blame here, yesterday I installed Samba 3.2.0rc1 2008-06-09 10:58 using it on top of aoe 2008-06-09 10:58 i think there are two modes, one uses mmap and one doesn't 2008-06-09 10:58 mmm I don't know 2008-06-09 10:58 trying to remember the details, but this was long ago, without zumastor, back in 2.6.11 or so 2008-06-09 10:59 if its a samba only bug perhaps its related to mmap 2008-06-09 10:59 anyway, it's frozen again and now it's not due to Samba 2008-06-09 10:59 but thats a wild guess without any tracebacks ;) 2008-06-09 10:59 because this time I shut down Samba and moved files around myself 2008-06-09 11:06 pgquiles, going to do the "echo t >/proc/sysrq-trigger" ? 2008-06-09 11:07 pgquiles, it's great you eliminated samba from the question 2008-06-09 11:07 is it possible to run on the raw volumes without lvm2? 2008-06-09 11:07 flipz: 10:53 < pgquiles> I'm 20% uploading the kernel to my PPA, when I'm done I'll VPN to the office and try sysrq 2008-06-09 11:08 I imagine that would require a hardy reinstall 2008-06-09 11:08 hes going to 2008-06-09 11:08 thanks 2008-06-09 11:08 and i dont think samba is eliminated 2008-06-09 11:08 flipz: yes, I'll try sysrq as soon as I finish uploading the new hardy kernel to my PPA (still about 30MB to go) 2008-06-09 11:08 oh i see 2008-06-09 11:08 nm 2008-06-09 11:08 apparently we both have reading comprehension issues :P 2008-06-09 11:08 flipz: reinstall? no, no need for that. Data and operating system are on different disks and different RAIDs. 2008-06-09 11:09 thank goodness 2008-06-09 11:09 so it would be possible to try it without lvm2? 2008-06-09 11:09 it's probably I cannot explain myself well O:-) 2008-06-09 11:09 flipz: yes, but not today 2008-06-09 11:09 ok, seeing the traceback would be a big step for today 2008-06-09 11:10 either the kernel with jiaying's patch to get rid of per-device throttle works fine, or I move the data to non-zumastored volumes 2008-06-09 11:10 but I've got two more servers to test everything until it works 2008-06-09 11:10 :-) 2008-06-09 11:11 pgquiles, I will be surprised if turning off throttling does anything good 2008-06-09 11:12 flipz: :-( 2008-06-09 11:13 reading http://code.google.com/p/zumastor/issues/detail?id=151 gave me hope 2008-06-09 11:13 pgquiles, oh I see, my mistake 2008-06-09 11:14 you are talking about removing peterz's per-device throttling, I thought you were talking about our bio.throttle patch 2008-06-09 11:14 yes, this is quite likely to help 2008-06-09 11:15 oh, no, bio.throttle says there 2008-06-09 11:15 fact is 2.6.24 (.23?) is harmful for zumastor :-) 2008-06-09 11:15 I would say so 2008-06-09 11:16 in fact, dirty limits in general are harmful to zumastor and everybody else in my opinion, I just need some solid evidence and I will make them go away 2008-06-09 11:17 this might be solid evidence, I'm anxiously awaiting the result 2008-06-09 11:19 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-09 11:25 how do I bring back a volume from "status: failed" to "status: OK" ? 2008-06-09 11:39 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-09 11:40 ok, I've VPNed to work 2008-06-09 11:40 shapor: here is the ps output as requested: http://pastie.org/211625 2008-06-09 11:42 hrm, wchan's are all "-" :( 2008-06-09 11:42 are you able to get the sysrq trace? 2008-06-09 11:44 pgquiles_: does cat /proc/4497/wchan show anything? 2008-06-09 11:45 and the sysrq trace: www.elpauer.org/tmp/zumacrash-20080609.tar.bz2 2008-06-09 11:45 shapor: cat /proc/4497/wchan shows nothing 2008-06-09 11:45 ok 2008-06-09 11:46 I've uploaded /var/log/{kern.log,messages,syslog}, is that enough? 2008-06-09 11:46 perfect 2008-06-09 11:47 hmm i dont see ddsnap_server (pid 4497) in the kernel trace 2008-06-09 11:47 :( 2008-06-09 11:48 can you run it again and see if it ends up in the log, sometimes the kernel or syslog drops them 2008-06-09 11:49 i think its less likely to happen if you use "W" rather than "T" 2008-06-09 11:49 because w just shows blocked tasks 2008-06-09 11:49 its a new option in 2.6.2? 2008-06-09 11:53 ok 2008-06-09 11:53 now lvdisplay is locked, too :-/ 2008-06-09 11:55 ok, with w there's info about 4497 2008-06-09 11:56 new logs: http://www.elpauer.org/tmp/zumacrash-20080609-2.tar.bz2 2008-06-09 12:16 shapor: flipz: have you found anything? 2008-06-09 12:25 looking, the trace is helpful 2008-06-09 12:25 shapor: ok 2008-06-09 12:26 I'm going to the swimming pool and I'll be back in about 1 hour 2008-06-09 12:26 after 13.5 hours in front of the computer, I need some rest :-) 2008-06-09 12:36 indeed, have a nice swim! 2008-06-09 12:36 thanks for the bug report 2008-06-09 12:46 pgquiles_ 2008-06-09 13:18 pgquiles_, is there software raid in this setup? 2008-06-09 13:19 ACTION thinks no 2008-06-09 13:22 he said no earlier 2008-06-09 13:22 just lvm on hardware raod 2008-06-09 13:22 raid 2008-06-09 13:31 pgquiles_: are you using a loopback block device by any chance? 2008-06-09 13:32 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-09 13:42 shapor: loopback? not that I can remember of 2008-06-09 13:42 let me check 2008-06-09 13:43 hmm your traceback is confusing us :) 2008-06-09 13:44 http://pastie.org/211732 2008-06-09 13:44 no mounted loopback devices 2008-06-09 13:44 shapor: confusing? why? 2008-06-09 13:44 ok :) 2008-06-09 13:44 I'm the master of disaster, the... 2008-06-09 13:44 because the call trace doesnt make sense 2008-06-09 13:44 mmm :-D 2008-06-09 13:44 can you post your kernel config? 2008-06-09 13:45 it's the -zumastor kernel from zumastor-releases PPA 2008-06-09 13:46 http://www.elpauer.org/tmp/config-2.6.24-16-zumastor 2008-06-09 13:48 # CONFIG_4KSTACKS is not set 2008-06-09 13:48 just checking 2008-06-09 13:48 because the tracebacks are strange 2008-06-09 13:48 functions in the chain that can't possibly be called 2008-06-09 13:48 for example: 2008-06-09 13:49 3445 Jun 9 20:54:34 maquina1 kernel: [16009.444662] [ext3:__blockdev_direct_IO+0xb0b/0x1d30] __blockdev_direct_IO+0xb0b/0xe10 2008-06-09 13:49 3446 Jun 9 20:54:34 maquina1 kernel: [16009.444674] [blkdev_direct_IO+0x5b/0x70] blkdev_direct_IO+0x5b/0x70 2008-06-09 13:49 should not be calling into ext3 if the block device is not loopback mounted 2008-06-09 13:50 the device is mounted by zumastor 2008-06-09 13:50 and the /media/justreplicated is another partition, ext3 on top of raw blocks 2008-06-09 13:50 this call chain is all happening underneath the mount 2008-06-09 13:50 UUID=4fdbf16a-abc9-4766-bd12-d10bbf75a7a7 /media/justreplicated ext3 relatime,errors=remount-ro 0 1 2008-06-09 13:51 do you need more information? any test? I can reboot the server if you want 2008-06-09 13:51 # CONFIG_FRAME_POINTER is not set <- ok, this is likely the explanation for the funny traces 2008-06-09 13:57 pgquiles_, we really need CONFIG_FRAME_POINTER set on in order to make sense of the backtraces 2008-06-09 13:58 flipz: ok, I'll see what I can do 2008-06-09 13:59 but given that a kernel build takes about 6 hours, probably no useful information until tomorrow 2008-06-09 13:59 tomorrow is fine 2008-06-09 13:59 flipz: does CONFIG_FRAME_POINTER harm performance? 2008-06-09 13:59 not measurably 2008-06-09 14:00 ok 2008-06-09 14:01 should CONFIG_4KSTACKS also be set? 2008-06-09 14:03 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-09 14:04 no, not 4KSTACKS 2008-06-09 14:04 risks overflow 2008-06-09 14:04 and set the printk buffer size to the max 2008-06-09 14:09 what's the option for the printk buffer size? I can'f find one :-? 2008-06-09 14:10 CONFIG_LOG_BUF_SHIFT = 17 2008-06-09 14:11 under "general setup" 2008-06-09 14:12 not clear why it isn't under debugging 2008-06-09 14:12 maybe because we call that "hacking", not debugging? 2008-06-09 14:12 ok 2008-06-09 14:14 what should I use tomorrow, 0.8 stable or 0.9 trunk? 2008-06-09 14:28 0.9 is next stable 2008-06-09 14:28 pending the finish of a test 2008-06-09 14:31 0.10 is current trunk 2008-06-09 14:35 willn: when will that test finish? 2008-06-09 14:36 another day or two 2008-06-09 14:50 the 2.6.25.4 branch seems to not have broken yet. Been running for a couple days now 2008-06-09 15:10 willn: what about performance? IIRC jiaying said 2.6.25 was better than .24 2008-06-09 15:19 its slightly better, no numbers from me though 2008-06-09 15:22 how do I fix a volume which is in "status: failed" ? 2008-06-09 15:24 This is running zumastor status as root, yes? 2008-06-09 15:28 I thought we had a global '/bin/zumastor must be run as root' function somewhere... Apparently not 2008-06-09 15:29 willn: yes, running status as root 2008-06-09 15:32 willn: this? http://code.google.com/p/zumastor/issues/detail?id=51 2008-06-09 15:40 Yea, status still lets you run it as non-root though 2008-06-09 15:42 willn: why shouldn't status be allowed as non-root? 2008-06-09 15:42 if you can run "mount" as non-root, "zumastor status" should be as well 2008-06-09 15:43 it provides more or less the same information, unless I'm overlooking something :-? 2008-06-09 15:57 It still makes calls to things that don't work as non-root 2008-06-09 15:57 so a volume will always show up as failed for non-root users 2008-06-09 16:08 -!- CIA-4(~CIA@208.69.182.149) has joined #zumastor 2008-06-09 16:08 zbuild: zumastor b0.10.0 r1700 build success 2008-06-09 16:13 zbuild: zumastor b0.10.0 r1700 install success 2008-06-09 16:18 willn: interesting, I didn't know 2008-06-09 16:18 willn: anyway, my volume is showing as failed even when running zumastor status as root :-/ 2008-06-09 16:19 status: failed is determined at 2008-06-09 16:19 line 1541 of /bin/zumastor 2008-06-09 16:20 as of rev 1701 2008-06-09 16:20 last changed 1602 2008-06-09 16:21 thanks, I'll take a look 2008-06-09 16:25 try `ddsnap status /var/run/zumastor/servers/$VOL --list` 2008-06-09 17:12 03williamanowak * r1702 10/trunk/kernel/config/ (2.6.25.4-um-i386-uml 2.6.25.4-i386-full 2.6.25.4-amd64-full): Proper kernel configs this time. 2008-06-09 18:04 03vandebo * r1703 10/trunk/ddsnap/kernel/dm-ddsnap.c: 2008-06-09 18:04 Resolve issue 141 and more: 2008-06-09 18:04 Convert to the single direct use of the socket code to use the API. 2008-06-09 18:04 Remove the use of underlying fd's and use the sockets directly. 2008-06-09 18:37 03williamanowak * r1704 10/trunk/cbtb/tests/2/nfs-test.sh: Try and not fail with the nfs-test 2008-06-09 18:38 zbuild: zumastor b0.10.0 r1700 test failure 1 2008-06-09 20:19 -!- pgquiles__(~pgquiles@192.Red-79-148-70.staticIP.rima-tde.net) has joined #zumastor 2008-06-09 23:03 rsync needs to stop dying 2008-06-09 23:45 -!- phoenix24(~phoenix24@122.167.99.34) has joined #zumastor irc.oftc.net #zumastor log beginning Tue Jun 10 00:00:01 PDT 2008 2008-06-10 01:57 -!- MaZe(~MaZe@c-24-6-86-168.hsd1.ca.comcast.net) has joined #zumastor 2008-06-10 06:19 zbuild: zumastor b0.10.0 r1701 build success 2008-06-10 06:34 zbuild: zumastor b0.10.0 r1701 install success 2008-06-10 08:40 -!- pgquiles(~pgquiles@192.Red-79-148-70.staticIP.rima-tde.net) has joined #zumastor 2008-06-10 08:43 bah 2008-06-10 08:59 couple more kernels to build before we're in the clear 2008-06-10 09:00 zbuild: zumastor b0.10.0 r1701 test failure 1 2008-06-10 10:10 flipz: ping 2008-06-10 10:59 willn, pong 2008-06-10 11:18 You've got mail 2008-06-10 14:28 willn: I tried building 2.6.24-19.33 with zumastor patches but it fails :-( 2008-06-10 14:29 zissues: Issue 153 in zumastor: Zumastor cron jobs should be refactored 2008-06-10 14:30 pgquiles, do you have .rej files? 2008-06-10 14:34 pgquiles: with flavormaker? 2008-06-10 14:35 willn: yes 2008-06-10 14:36 hmm, what arguments did you pass 2008-06-10 14:36 flipz: no, the problem is not about .rej files but about some missing options in the config file. Unfortunately, my system rebooted accidentally and I can't remember the output :-/ 2008-06-10 14:37 I'll give it a shot locally. 2008-06-10 14:38 I also added jiaying's patch for issue 151 2008-06-10 14:38 it might be that 2008-06-10 14:49 I wish it would build reverse alphabetically 2008-06-10 14:50 building with ccache should help. 2008-06-10 14:54 pgquiles: How early did you get a failure? 2008-06-10 14:55 willn: I tried but I can't get pbuilder to use distcc/cache :-/ 2008-06-10 14:55 willn: mmm I'd say quite soon after the -zumastor flavour starts building 2008-06-10 14:55 I moved "zumastor" to be the first flavour to be built, btw 2008-06-10 14:56 that way I get an early failure, no need to wait for lpia, rt, xen, etc 2008-06-10 14:59 zissues: Issue 152 in zumastor: Clean up kernel messages at boot 2008-06-10 16:10 ACTION spins 2008-06-10 17:50 pgquiles: looks like the build is going ok 2008-06-10 17:51 I did a ./buildcurrent.sh, which dropped AUTO.* into the ddsnap patches directories 2008-06-10 17:51 then 2008-06-10 17:51 ./flavormaker.py -i -a -n zumastor -m http://archive.ubuntu.com/ubuntu -d /pool/main/l/linux/linux_2.6.24-19.33.dsc -r hardy -e 'Zumastor Builder ' -v 'ppa6' -c 'CONFIG_DM_DDSNAP=m' -p ~/zumastor/ddsnap/patches/2.6.24.2 2008-06-10 20:20 zbuild: zumastor b0.10.0 r1704 build success 2008-06-10 20:28 woo 2008-06-10 20:28 finally 2008-06-10 20:30 zbuild: zumastor b0.10.0 r1704 install success 2008-06-10 23:06 zbuild: zumastor b0.10.0 r1704 test failure 1 irc.oftc.net #zumastor log beginning Wed Jun 11 00:00:01 PDT 2008 2008-06-11 10:57 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-11 11:24 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-11 12:53 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-11 13:30 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-11 14:15 willn: should the zumastor package have a dep on dmsetup ? 2008-06-11 14:21 it should 2008-06-11 14:21 i thought it did 2008-06-11 14:22 yes. 2008-06-11 14:22 hmm, my mistake 2008-06-11 14:22 Depends: ddsnap (>= 0.10.0), nblock-write, bash (>= 3.1), cron, dmsetup, util-linux (>= 2.12), ${shlibs:Depends}, ${misc:Depends} 2008-06-11 14:26 pgquiles: to reproduce your hang, I should try writing a bunch of small files with hardy? 2008-06-11 15:02 vandebo: hes on the other side of the world, probably sleeping, might want to email (and cc the list) 2008-06-11 15:07 shapor: yea, thought he might be around 2008-06-11 19:35 03Daniel.Raymond.Phillips * r1705 10/trunk/ddsnap/patches/2.6.23.8/ (ddsnap.base.patch bio.throttle.patch): 2008-06-11 19:35 Patch sets for 2.6.22.18 2.6.23.8 were broken by the introduction of dm_copy_name_and_uuid 2008-06-11 19:35 in dm-ddsnap.c, a new device mapper function added to recent kernels, in particular 2.6.24.2 2008-06-11 19:35 and 2.6.25.4, which we support. I backported dm_copy_name_and_uuid to 2.6.23.8 to avoid 2008-06-11 19:35 having to do something major to dm-ddsnap.c. Now 2.6.23.8 compiles, better than it was. 2008-06-11 19:35 Not tested yet, since our automated testing system does not handle older kernels. 2008-06-11 19:37 flipz: the uml smoke test will test anything, provided a uml kernel config exists and you update the KernelVersion file accordingly 2008-06-11 21:13 oy. 2008-06-11 21:14 If zumastor define volume fails, it will sill leave residue behind 2008-06-11 21:14 so you have to forget the volume before you can try again 2008-06-11 21:14 also, the ssh failure error message is really silly. For permission denied it tells you to set source, not check the ssh setup 2008-06-11 21:31 zissues: Issue 155 in zumastor: ddsnap/zumastor cachesize option not well documented 2008-06-11 21:31 zissues: Issue 154 in zumastor: Failed `zumastor define volume` leaves a mess if it fails 2008-06-11 21:42 zissues: Issue 156 in zumastor: Zumastor failed ssh log message is overloaded 2008-06-11 21:46 I'm stuck on an ssh issue. 2008-06-11 21:47 ssh t1 from t2 works. ssh t2 from that t1 session works. ssh t1 from that t2 session works. ssh t2 from that t1 session asks for a password 2008-06-11 21:48 what gives? 2008-06-11 21:53 well, its replicating now. 1T volume, replicating compressed data at ~10M/s, no snapshots on the upstream yet 2008-06-11 21:53 writing data there though 2008-06-11 21:53 we shall see what happens tommorow 2008-06-11 21:53 did chained replication for kicks too. 2008-06-11 22:16 Its probably time to go home. irc.oftc.net #zumastor log beginning Thu Jun 12 00:00:01 PDT 2008 2008-06-12 01:03 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-12 01:11 -!- MaZe(~MaZe@c-24-6-86-168.hsd1.ca.comcast.net) has joined #zumastor 2008-06-12 01:16 -!- pgquiles__(~pgquiles@192.Red-79-148-70.staticIP.rima-tde.net) has joined #zumastor 2008-06-12 03:19 -!- MaZe(~MaZe@c-24-6-86-168.hsd1.ca.comcast.net) has joined #zumastor 2008-06-12 05:48 -!- erwan_taf(~erwan@81.80.43.67) has joined #zumastor 2008-06-12 06:48 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-12 09:29 -!- pgquiles__(~pgquiles@192.Red-79-148-70.staticIP.rima-tde.net) has joined #zumastor 2008-06-12 09:45 ACTION blinkks 2008-06-12 09:45 ACTION waves willn 2008-06-12 09:45 howdy 2008-06-12 09:46 are you going to upload 2.6.24-19.33 to zumastor-team and/or zumastor-stable? 2008-06-12 09:46 (btw, build failed for me because I had not added -c "DDSNAP..." ) 2008-06-12 09:47 I've not yet, but I certainly can 2008-06-12 09:47 it'd be nice :-) 2008-06-12 09:47 zbuild: zumastor b0.10.0 r1705 build success 2008-06-12 09:48 This 1T volume is taking forever to populate/replicate 2008-06-12 09:48 btw, is is possible to use multiple -c and -p parameters? 2008-06-12 09:48 Yep 2008-06-12 09:48 willn: I know, my volumes are 1TB each :-) 2008-06-12 09:48 This is the hardware test cluster environment,so no hardware raid, just software raid 0 2008-06-12 09:49 ouch 2008-06-12 09:49 that hurts 2008-06-12 09:49 No overhead, when we were trying software raid 5 it was real bad. 2008-06-12 09:49 oh, ok 2008-06-12 09:49 somehow I had read "just software raid 5" 2008-06-12 09:50 :P 2008-06-12 09:50 I was reading on next OS X Server including ZFS 2008-06-12 09:51 I've been meaning to write the 3 lines of shell around flavormaker to generate the AUTO.patch files, run flavormaker, run dpkg-buildpackage -S, and upload 2008-06-12 09:51 :-D 2008-06-12 09:51 what are the AUTO.patch :-? 2008-06-12 09:52 ddsnap/scripts/genpatch.sh 2008-06-12 09:57 zbuild: zumastor b0.10.0 r1705 install success 2008-06-12 10:05 pgquiles: are you actually using r1659? the 0.8 release is 1639 2008-06-12 10:06 vandebo: mmm ok, it's probably 1639. I wrote that out of my memory :-) 2008-06-12 10:06 I only missed one digit :-D 2008-06-12 10:07 pgquiles: I uploaded a 2.6.25-19.33 package to the -team ppa 2008-06-12 10:08 willn: you mean 2.6.24-19.33, right? 2008-06-12 10:09 ~/zumastor/ddsnap/scripts/genallpatches.sh; ./flavormaker.py -i -a -n zumastor -m http://zuma14/mirrors/ubuntu -d /pool/main/l/linux/linux_2.6.24-19.33.dsc -r hardy -e 'Zumastor Builder ' -v 'ppa6' -c 'CONFIG_DM_DDSNAP=m' -p ~/zumastor/ddsnap/patches/2.6.24.2; cd /tmp/kernelbuild.694/linux-2.6.24; dpkg-buildpackage -S; cd ..; dput zumastor-team *.changes 2008-06-12 10:09 pgquiles: Yeah. 2008-06-12 10:10 willn: it might be a good idea to have README.txt in the flavormaker.py directory with an example invocation, i. e. what you just pasted above 2008-06-12 10:10 Indeed. 2008-06-12 10:13 zbuild: [PPA zumastor-team] Accepted: linux 2.6.24-19.33~ppa6 (source) 2008-06-12 10:17 03williamanowak * r1706 10/trunk/kernel/ubuntu-package/README: Added flavormaker kernel-build howto readme 2008-06-12 10:38 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-12 10:52 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-12 11:06 Guess that triggered a kernel build :( 2008-06-12 12:18 zbuild: zumastor b0.10.0 r1705 test failure 1 2008-06-12 12:18 03williamanowak * r1707 10/trunk/cbtb/tests/2/nfs-test.sh: Attempt to fix the nfs-test again. 2008-06-12 15:22 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-12 16:13 gah. ABI failure 2008-06-12 17:17 -!- pgquiles__(~pgquiles@192.Red-79-148-70.staticIP.rima-tde.net) has joined #zumastor 2008-06-12 18:38 zissues: Issue 157 in zumastor: On-disk data not 64 bit clean. 2008-06-12 19:34 -!- shapor(~shapor@yzf.shapor.com) has joined #zumastor 2008-06-12 21:00 03williamanowak * r1708 10/trunk/ (35 files in 18 dirs): 2008-06-12 21:00 Check in testcluster base software. This includes everything for basic cluster 2008-06-12 21:00 operations. Autotest configuration and tests will be another commit. 2008-06-12 21:00 README's scattered about indicate what the directory contains/is used for 2008-06-12 21:00 CLUSTEROPS has usage information for the running cluster. 2008-06-12 21:00 These configs/utilities should be useful to our project, and other projects 2008-06-12 21:00 wishing to automate testing on bare metal 2008-06-12 23:24 -!- hgichon(~hgichon@116.39.193.213) has joined #zumastor 2008-06-12 23:39 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-12 23:49 -!- pgquiles__(~pgquiles@192.Red-79-148-70.staticIP.rima-tde.net) has joined #zumastor 2008-06-12 23:53 -!- MaZe(~MaZe@c-24-6-86-168.hsd1.ca.comcast.net) has joined #zumastor irc.oftc.net #zumastor log beginning Fri Jun 13 00:00:01 PDT 2008 2008-06-13 00:33 zbuild: zumastor b0.10.0 r1708 build success 2008-06-13 00:33 zbuild: zumastor b0.10.0 r1706 build success 2008-06-13 00:44 zbuild: zumastor b0.10.0 r1706 install success 2008-06-13 00:54 zbuild: zumastor b0.10.0 r1708 install success 2008-06-13 02:23 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-13 03:19 zbuild: zumastor b0.10.0 r1706 test failure 1 2008-06-13 05:49 zbuild: zumastor b0.10.0 r1708 test failure 1 2008-06-13 05:53 -!- phoenix24(~phoenix24@122.167.85.32) has joined #zumastor 2008-06-13 07:15 -!- phoenix24(~phoenix24@122.167.85.32) has joined #zumastor 2008-06-13 08:42 03williamanowak * r1709 10/trunk/cbtb/tests/2/replication-zumastor-names.sh: Quick fix for failing replication-zumastor-names.sh 2008-06-13 09:04 zbuild: zumastor b0.10.0 r1709 build success 2008-06-13 09:29 zbuild: zumastor b0.10.0 r1709 install success 2008-06-13 10:17 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-13 10:49 -!- phoenix24_(~phoenix24@122.167.69.94) has joined #zumastor 2008-06-13 11:07 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-13 11:11 -!- phoenix24(~phoenix24@122.167.108.169) has joined #zumastor 2008-06-13 11:42 pgquiles_: ping 2008-06-13 11:45 zbuild: zumastor b0.10.0 r1709 test success 2008-06-13 12:01 willn: friday night here 2008-06-13 12:23 willn: pong 2008-06-13 12:39 pgquiles_: I'm sorting out the abi issues with the upload, should be ready to do another sometime before the end of the day (PST) 2008-06-13 12:42 willn: OK. I patched flavormaker yesterday to get rid of them but I'd say I invoked flavormaker with the wrong parameters because the kernel build failed. My patch is http://pastie.org/214664 and the failure log is http://launchpadlibrarian.net/15263601/buildlog_ubuntu-hardy-i386.linux_2.6.24-19.33~ppa7_FAILEDTOBUILD.txt.gz 2008-06-13 12:42 I did not commit the patch, though 2008-06-13 12:49 pgquiles_: I ran my test in 2GB (vmware) and 10GB volumes. 2008-06-13 12:49 vandebo: empty volumes? 2008-06-13 12:49 you suggested that the problem was reproducable moving less than a gigabyte 2008-06-13 12:49 yes 2008-06-13 12:49 I ran those scripts to generate files on the empty voluemes, yes 2008-06-13 12:50 but maybe having so many files, or the 1TB volume, makes zumastor slower 2008-06-13 12:50 I moved 2.3G without issue 2008-06-13 12:50 anyway, next week I'll setup two servers for testing 2008-06-13 12:51 vandebo: zumastor worked fine for me quite long, that's why the subject in my e-mail was "odd behavior" :-) 2008-06-13 12:51 I didn't expect it to fail 2008-06-13 12:51 yea... harder to debug when it's not clear what triggers the bug :-/ 2008-06-13 12:52 indeed :-( 2008-06-13 12:53 in the old times (zumastor 0.3, IIRC), large volumes were impossible to use, but I think the problem was in the zumastor shell-script, not in ddsnap 2008-06-13 12:53 this time I feel it's ddsnap what's causing trouble 2008-06-13 12:54 for instance, "ddsnap list" froze the volume immediately 2008-06-13 12:54 how long do you let it try? 2008-06-13 12:57 about 14 hours 2008-06-13 12:57 so even if it wasn't completely hung, it was as good as hung... 2008-06-13 12:57 indeed 2008-06-13 12:58 and I was in a no-turn-back point, so I had to find a solution quickly 2008-06-13 12:59 which in the end meant no zumastor and using rsync to synchronize data between the servers 2008-06-13 12:59 it's terrible when technology works on the test bench, but then fails when you try to actually use it 2008-06-13 12:59 until I find out why zumastor failed and replace the rsynced servers with zumastored servers 2008-06-13 13:00 yes 2008-06-13 13:00 it's the worst failure I have had in 5 years :-( 2008-06-13 13:04 funniest one was about two years ago when I was called overnight to install a SAN, as the client was really angry. It was as simple as pressing the hard disk to make sure they made contact with the enclosure :-D 2008-06-13 13:10 haha 2008-06-13 13:42 pgquiles_: You don't happen to have the command line that you passed? 2008-06-13 13:42 [to flavormaker] 2008-06-13 13:43 and it looks like your missing the patches from ddsnap/scripts/genpatch.sh 2008-06-13 13:44 willn: I don't have the command line. Well, I *might* have, but I have 14 shells open and have been working for about 14 hours since I passed it, so it's almost impossible for me to find it :-D 2008-06-13 13:44 :P 2008-06-13 13:44 willn: you are problaby right about the missing patches 2008-06-13 13:45 mmm two 14's, is that my lucky number? :-D 2008-06-13 13:45 I've got to move flavormaker out of svn/trunk/kernel, otherwise changes trigger kernel builds 2008-06-13 13:46 not only flavormaker but also the docs 2008-06-13 13:52 that whole ubuntu-package directory 2008-06-13 13:57 actually I meant zumastor/doc, zumastor/ddsnap/man, etc. Or are those already filtered? 2008-06-13 14:00 oh, the docs all still cause builds 2008-06-13 14:01 kernel builds are especially bad, since the kernel is building in qemu 2008-06-13 14:01 and takes about 12 hours 2008-06-13 14:03 :-D 2008-06-13 14:04 willn: what about skipping building the lpia, rt, xen, etc flavours? they are already available from the official ubuntu repository 2008-06-13 14:04 Could do that 2008-06-13 14:04 and probably skipping -server and others would also be a good idea, just build -zumastor 2008-06-13 14:12 building locally and on lp 2008-06-13 14:13 hopefully the lp build will give me some early indication, since i've got the build iin ccache 2008-06-13 14:14 er, the local build 2008-06-13 14:30 zbuild: [PPA zumastor-team] Accepted: linux 2.6.24-19.33~ppa7 (source) 2008-06-13 15:15 oy 2008-06-13 15:15 going to fail again 2008-06-13 15:15 Checking ABI for generic...previous or current ABI file missing! /tmp/kernelbuild.482/linux-2.6.24/debian/abi/2.6.24-19.33~ppa7/amd64/generic 2008-06-13 15:39 zissues: Issue 158 in zumastor: Using seperate metadata volumes are broken for the first exception 2008-06-13 16:58 Yea, that one worked. 2008-06-13 16:59 pgquiles_: debian/scripts/misc/getabis is our friend 2008-06-13 17:30 zbuild: [PPA zumastor-team] Accepted: linux 2.6.24-19.33~ppa8 (source) 2008-06-13 19:41 -!- MaZe(~MaZe@c-24-6-86-168.hsd1.ca.comcast.net) has joined #zumastor 2008-06-13 20:08 oy vey. 2008-06-13 20:09 SHould have updated instead of incremented the changelog 2008-06-13 23:00 -!- phoenix24(~phoenix24@122.167.108.169) has joined #zumastor irc.oftc.net #zumastor log beginning Sat Jun 14 00:00:01 PDT 2008 2008-06-14 01:00 -!- phoenix24_(~phoenix24@122.167.78.22) has joined #zumastor 2008-06-14 02:53 willn: getabis? I didn't know that existed! :-O 2008-06-14 04:04 -!- pgquiles__(~pgquiles@192.Red-79-148-70.staticIP.rima-tde.net) has joined #zumastor 2008-06-14 12:11 It takes the format `debian/scripts/misc/getabis 2.6.24 19.33` and will download abis for that kernel into your abi dir 2008-06-14 13:02 -!- phoenix24(~phoenix24@122.167.78.22) has joined #zumastor 2008-06-14 17:43 -!- pgquiles(~pgquiles@223.Red-81-32-36.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-14 17:49 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-14 20:19 -!- pgquiles__(~pgquiles@223.Red-81-32-36.dynamicIP.rima-tde.net) has joined #zumastor irc.oftc.net #zumastor log beginning Sun Jun 15 00:00:01 PDT 2008 2008-06-15 02:56 -!- phoenix24(~phoenix24@122.167.113.245) has joined #zumastor 2008-06-15 06:43 -!- phoenix24(~phoenix24@122.167.75.89) has joined #zumastor 2008-06-15 07:13 -!- phoenix24_(~phoenix24@122.167.83.37) has joined #zumastor 2008-06-15 14:46 -!- pgquiles__(~pgquiles@100.Red-83-41-45.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-15 17:24 -!- phoenix24_(~phoenix24@122.167.83.37) has joined #zumastor 2008-06-15 21:24 -!- phoenix24_(~phoenix24@122.167.83.37) has joined #zumastor 2008-06-15 21:42 can i use zumastor for realtime replication? 2008-06-15 21:43 i used source period =0 2008-06-15 21:43 but update is not immediatable - -; 2008-06-15 22:19 Your period shouldnt be shorter than a realisic time for churn irc.oftc.net #zumastor log beginning Mon Jun 16 00:00:01 PDT 2008 2008-06-16 00:48 hmm... ok thanks. 2008-06-16 00:49 Is possible mount targeted volume for rw? 2008-06-16 00:53 i means... 2008-06-16 00:54 if original volume is died by accident, then i want to service ... using replicated volume.. 2008-06-16 02:15 hgichon: yes, it's possible to mount it for rw but you need to setup something for HA 2008-06-16 02:28 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-16 03:02 -!- pgquiles__(~pgquiles@100.Red-83-41-45.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-16 05:03 pgquiles__:thanks ^^ 2008-06-16 05:15 -!- pgquiles_(~pgquiles@123.Red-81-33-103.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-16 05:21 -!- pgquiles__(~pgquiles@118.Red-81-36-120.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-16 05:45 -!- nataliep_(~nataliep@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-16 08:25 -!- erwan_taf(~erwan@81.80.43.67) has joined #zumastor 2008-06-16 10:16 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-16 10:25 -!- mitchg(~mitchg@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-16 11:37 hgichon: You just want to make sure you don't have two ends of the replication rw simultaneously 2008-06-16 11:37 It should always be master/slave configuration 2008-06-16 11:38 zbuild: [PPA zumastor-team] Accepted: linux 2.6.24-19.33~ppa9 (source) 2008-06-16 11:40 lets hope that one doesnt suffer abi failure 2008-06-16 11:40 lets also hope these test cluster machines don't continue to have grub issues 2008-06-16 12:05 -!- dkegel(~chatzilla@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-16 12:10 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-16 13:03 zbuild: [PPA zumastor-team] Accepted: zumastor 0.10.0-r1709 (source) 2008-06-16 13:03 zbuild: [PPA zumastor-team] Accepted: ddsnap 0.10.0-r1709 (source) 2008-06-16 13:11 -!- dkegel_(~chatzilla@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-16 13:12 flips, do you have slides or a recording of your linuxtag talk somewhere? Someone on lvm list is asking. We should link to them from zumastor.org... 2008-06-16 13:37 -!- pgquiles(~pgquiles@192.Red-79-148-70.staticIP.rima-tde.net) has joined #zumastor 2008-06-16 13:42 -!- dkegel__(~chatzilla@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-16 13:47 dkegel, ping :-) 2008-06-16 13:51 Oh wonderful. Now the ppa build failed for something else 2008-06-16 13:57 pgquiles: ... The amd64 build succeeded! 2008-06-16 13:58 willn: cool! 2008-06-16 13:58 The i386 and lpia builds failed because I was getting to crafty and trimming out non-zumastor builds 2008-06-16 13:58 so other scripts that depend on those builds failed, causing the build to fail 2008-06-16 13:58 oh 2008-06-16 13:58 there are some builds which could probably be taken out safely, i. e. xen and rt 2008-06-16 13:58 and maybe others, too 2008-06-16 14:00 I think I killed those 2008-06-16 14:00 mmm I didn't expect -server to depend on -rt or -xen :-? 2008-06-16 14:01 Nope. The i386 build failed because I killed -386 2008-06-16 14:01 which some script tried to extract something from 2008-06-16 14:01 and the .deb wasn't there after the build finished 2008-06-16 14:43 zbuild: [PPA zumastor-team] Accepted: linux 2.6.24-19.33~ppa10 (source) 2008-06-16 21:52 pgquiles: build finished. feel free to give it a whirl irc.oftc.net #zumastor log beginning Tue Jun 17 00:00:01 PDT 2008 2008-06-17 01:49 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-17 02:19 -!- MaZe(~MaZe@c-24-6-86-168.hsd1.ca.comcast.net) has joined #zumastor 2008-06-17 07:03 -!- pgquiles__(~pgquiles@192.Red-79-148-70.staticIP.rima-tde.net) has joined #zumastor 2008-06-17 07:57 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-17 10:46 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-17 12:30 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-17 12:35 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-17 12:57 -!- pgquiles_(~pgquiles@192.Red-79-148-70.staticIP.rima-tde.net) has joined #zumastor 2008-06-17 13:08 -!- pgquiles__(~pgquiles@192.Red-79-148-70.staticIP.rima-tde.net) has joined #zumastor 2008-06-17 14:01 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-17 14:17 -!- pgquiles__(~pgquiles@192.Red-79-148-70.staticIP.rima-tde.net) has joined #zumastor 2008-06-17 14:23 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-17 14:30 03williamanowak * r1710 10/trunk/ (8 files in 4 dirs): 2008-06-17 14:30 Move utils into utils dir 2008-06-17 14:30 - Move ubuntu kernel package tool out of kernel config dir 2008-06-17 14:30 - Move ddsnap/zumastor src pkg builder 2008-06-17 14:30 - Move build_packages.sh to utils dir 2008-06-17 15:11 -!- pgquiles__(~pgquiles@192.Red-79-148-70.staticIP.rima-tde.net) has joined #zumastor 2008-06-17 16:43 zissues: Issue 159 in zumastor: Enable frame pointers in our kernel configs 2008-06-17 16:52 03Daniel.Raymond.Phillips * r1711 10/trunk/kernel/config/ (5 files): All configs should have frame pointers on so we get sane kernel stack traces 2008-06-17 17:25 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-17 17:29 03williamanowak * r1712 10/trunk/testcluster/ (12 files in 5 dirs): Update testcluster software and configurations 2008-06-17 17:33 -!- pgquiles__(~pgquiles@192.Red-79-148-70.staticIP.rima-tde.net) has joined #zumastor 2008-06-17 18:03 ddddddd 2008-06-17 18:04 -!- nataliep_(~nataliep@66-102-14-1.google.com) has joined #zumastor 2008-06-17 18:08 well said. 2008-06-17 18:10 -!- mitchg(~mitchg@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-17 18:11 -!- jiayingz(~jiayingz@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-17 18:11 -!- dkegel(~chatzilla@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-17 18:11 -!- phillips_(~phillips@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-17 22:01 -!- MaZe(~MaZe@c-24-6-86-168.hsd1.ca.comcast.net) has joined #zumastor irc.oftc.net #zumastor log beginning Wed Jun 18 00:00:01 PDT 2008 2008-06-18 02:50 -!- pgquiles_(~pgquiles@192.Red-79-148-70.staticIP.rima-tde.net) has joined #zumastor 2008-06-18 04:50 zbuild: zumastor b0.10.0 r1710 build success 2008-06-18 04:55 zbuild: zumastor b0.10.0 r1710 install success 2008-06-18 07:26 zbuild: zumastor b0.10.0 r1710 test success 2008-06-18 08:52 -!- pgquiles__(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-18 09:36 -!- pgquiles_(~pgquiles@192.Red-79-148-70.staticIP.rima-tde.net) has joined #zumastor 2008-06-18 09:55 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-18 10:13 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-18 10:19 -!- pgquiles__(~pgquiles@192.Red-79-148-70.staticIP.rima-tde.net) has joined #zumastor 2008-06-18 10:40 -!- pgquiles___(~pgquiles@192.Red-79-148-70.staticIP.rima-tde.net) has joined #zumastor 2008-06-18 10:45 -!- pgquiles_(~pgquiles@192.Red-79-148-70.staticIP.rima-tde.net) has joined #zumastor 2008-06-18 11:42 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-18 12:24 zissues: Issue 160 in zumastor: "zumastor define volume" should warn if cache too small 2008-06-18 12:48 -!- pgquiles__(~pgquiles@192.Red-79-148-70.staticIP.rima-tde.net) has joined #zumastor 2008-06-18 14:17 -!- flips(~phillips@phunq.net) has joined #zumastor 2008-06-18 15:55 -!- CIA-4(~CIA@208.69.182.149) has joined #zumastor 2008-06-18 16:56 03williamanowak * r1713 10/trunk/testcluster/usr/local/ (33 files in 15 dirs): 2008-06-18 16:56 Landing autotest tests and machine utility updates 2008-06-18 16:56 - Client and serverside tests 2008-06-18 16:56 - Updated reinstallmach and installzumastor 2008-06-18 16:56 - Added autotestreinstall which wraps reinstallmach and installautotest 2008-06-18 17:31 pgquiles: Do you want to try and do a 2.6.24-19.33 upload with frame pointers enabled? 2008-06-18 19:02 zbuild: zumastor b0.10.0 r1712 build success 2008-06-18 19:07 zbuild: zumastor b0.10.0 r1712 install success 2008-06-18 19:07 zbuild: zumastor b0.10.0 r1713 build success 2008-06-18 19:27 zbuild: zumastor b0.10.0 r1713 install success 2008-06-18 21:32 zbuild: zumastor b0.10.0 r1712 test success 2008-06-18 23:34 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-18 23:42 -!- pgquiles__(~pgquiles@192.Red-79-148-70.staticIP.rima-tde.net) has joined #zumastor irc.oftc.net #zumastor log beginning Thu Jun 19 00:00:01 PDT 2008 2008-06-19 00:12 zbuild: zumastor b0.10.0 r1713 test success 2008-06-19 06:57 -!- pgquiles__(~pgquiles@189.Red-79-147-232.staticIP.rima-tde.net) has joined #zumastor 2008-06-19 08:10 -!- Gramphos(~viktor@gramphos.csbnet.se) has left #zumastor 2008-06-19 22:10 -!- pgquiles__(~pgquiles@189.Red-79-147-232.staticIP.rima-tde.net) has joined #zumastor 2008-06-19 22:10 -!- zumbi(~zumbi@94.132.219.87.dynamic.jazztel.es) has joined #zumastor 2008-06-19 22:10 -!- vandebo(~vandebo@abydos.nerdbox.net) has joined #zumastor 2008-06-19 22:21 -!- flips(~phillips@phunq.net) has joined #zumastor 2008-06-19 22:21 -!- nataliep_(~nataliep@66-102-14-1.google.com) has joined #zumastor 2008-06-19 22:21 -!- shapor(~shapor@yzf.shapor.com) has joined #zumastor irc.oftc.net #zumastor log beginning Fri Jun 20 00:00:01 PDT 2008 2008-06-20 01:20 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-20 01:33 -!- pgquiles__(~pgquiles@79.147.232.189) has joined #zumastor 2008-06-20 07:01 -!- erwan_taf(~erwan@81.80.43.67) has joined #zumastor 2008-06-20 07:31 -!- vandebo(~vandebo@abydos.nerdbox.net) has joined #zumastor irc.oftc.net #zumastor log beginning Sat Jun 21 00:00:01 PDT 2008 2008-06-21 07:02 -!- pgquiles(~pgquiles@232.Red-83-42-63.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-21 07:09 -!- pgquiles(~pgquiles@232.Red-83-42-63.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-21 10:40 -!- phoenix24(~phoenix24@122.167.100.117) has joined #zumastor 2008-06-21 10:57 -!- phoenix24(~phoenix24@122.167.100.117) has joined #zumastor 2008-06-21 13:17 -!- MaZe(~MaZe@c-24-6-86-168.hsd1.ca.comcast.net) has joined #zumastor irc.oftc.net #zumastor log beginning Sun Jun 22 00:00:01 PDT 2008 2008-06-22 14:27 -!- pgquiles(~pgquiles@54.Red-79-153-82.staticIP.rima-tde.net) has joined #zumastor 2008-06-22 21:18 -!- flips(~phillips@phunq.net) has joined #zumastor irc.oftc.net #zumastor log beginning Mon Jun 23 00:00:01 PDT 2008 2008-06-23 03:22 -!- flips(~phillips@phunq.net) has joined #zumastor 2008-06-23 04:08 -!- zumalog(~zumalog@yzf.shapor.com) has joined #zumastor 2008-06-23 04:08 -!- MaZe(~MaZe@c-24-6-86-168.hsd1.ca.comcast.net) has joined #zumastor 2008-06-23 04:08 -!- CIA-4(~CIA@208.69.182.149) has joined #zumastor 2008-06-23 04:08 -!- phillips_(~phillips@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-23 04:08 -!- jiayingz(~jiayingz@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-23 04:09 -!- CIA-6(~CIA@208.69.182.149) has joined #zumastor 2008-06-23 04:09 -!- jiayingz(~jiayingz@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-23 04:10 -!- phillips_(~phillips@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-23 04:10 -!- MaZe(~MaZe@c-24-6-86-168.hsd1.ca.comcast.net) has joined #zumastor 2008-06-23 04:10 -!- zbot(wan@209.67.252.126) has joined #zumastor 2008-06-23 06:08 -!- erwan_taf(~erwan@81.80.43.67) has joined #zumastor 2008-06-23 10:22 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-23 22:12 -!- MaZe(~MaZe@c-24-6-86-168.hsd1.ca.comcast.net) has joined #zumastor irc.oftc.net #zumastor log beginning Tue Jun 24 00:00:01 PDT 2008 2008-06-24 01:19 -!- erwan_taf(~erwan@LAubervilliers-151-13-63-69.w217-128.abo.wanadoo.fr) has joined #zumastor 2008-06-24 05:01 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-24 05:06 -!- pgquiles__(~pgquiles@54.Red-79-153-82.staticIP.rima-tde.net) has joined #zumastor 2008-06-24 08:31 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-24 08:36 -!- pgquiles__(~pgquiles@54.Red-79-153-82.staticIP.rima-tde.net) has joined #zumastor 2008-06-24 09:42 -!- MaZe(~MaZe@c-24-6-86-168.hsd1.ca.comcast.net) has joined #zumastor 2008-06-24 10:53 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-24 11:54 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-24 15:07 -!- SEJeff(~jeff__@66.151.59.138) has joined #zumastor 2008-06-24 15:10 Anyone around to answer a few quick questions about the cbtb tests? 2008-06-24 20:22 What do you want to know? 2008-06-24 23:54 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor irc.oftc.net #zumastor log beginning Wed Jun 25 00:00:01 PDT 2008 2008-06-25 06:55 -!- erwan_taf(~erwan@LAubervilliers-151-13-63-69.w217-128.abo.wanadoo.fr) has joined #zumastor 2008-06-25 17:30 Ubuntu pushed a fix for the hardy installer, will update the luster configs 2008-06-25 17:47 03williamanowak * r1714 10/trunk/testcluster/var/www/ (boot.cfg p.cfg): LP#234486 was copied to hardy-updates, updating preseed and boot configuration accordingly. 2008-06-25 17:49 -!- vandebo(~vandebo@abydos.nerdbox.net) has joined #zumastor 2008-06-25 18:10 zbuild: zumastor b0.10.0 r1714 build success 2008-06-25 18:25 zbuild: zumastor b0.10.0 r1714 install success 2008-06-25 20:45 zbuild: zumastor b0.10.0 r1714 test success irc.oftc.net #zumastor log beginning Thu Jun 26 00:00:01 PDT 2008 2008-06-26 00:33 -!- pgquiles(~pgquiles@54.Red-79-153-82.staticIP.rima-tde.net) has joined #zumastor 2008-06-26 01:09 -!- erwan_taf(~erwan@81.80.43.67) has joined #zumastor 2008-06-26 01:09 -!- pgquiles(~pgquiles@212.Red-88-0-136.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-26 04:32 -!- erwan_taf(~erwan@81.80.43.67) has joined #zumastor 2008-06-26 05:11 willn_, Still around? 2008-06-26 05:50 SEJeff: Yep 2008-06-26 05:50 Hey you said you could answer some questions about the tests 2008-06-26 05:51 Sure 2008-06-26 05:51 So previously they used passwordless root ssh keys. Now they use passwordless unpriv keys with sudo, correct? 2008-06-26 05:51 With a small wrapper script, we can make it use ssh triggers 2008-06-26 05:52 Which tests are we talking, the cbtb tests under uml? 2008-06-26 05:52 Yes 2008-06-26 05:52 But really a design of zumastor that requires root access and ssh 2008-06-26 05:53 And in specific, a way to improve it 2008-06-26 05:53 afaik, there haven't been any changes in the rpc [root ssh] methods 2008-06-26 05:53 Here is an overly aggressive example of a script I'm talking about 2008-06-26 05:54 http://www.digitalprognosis.com/opensource/scripts/ssh-trigger-verify 2008-06-26 05:54 Its a to-do last I heard 2008-06-26 05:54 Are you familar with ssh triggers? 2008-06-26 05:54 Great, maybe they would let me do it :) 2008-06-26 05:55 We always accept patches :) 2008-06-26 05:55 You make a passwordless root key 2008-06-26 05:55 Then you prepend something like this to it: 2008-06-26 05:55 I started on something along those lines, but havent had the chance to finish it 2008-06-26 05:55 no-port-forwarding,no-X11-forwarding,no-agent-forwarding,command="/usr/bin/ssh-trigger-verify" ssh-dss sshkeyhere comment 2008-06-26 05:56 If you put that in the authorized_keys, every time someone logs in with the matching key, it runs the command under command= 2008-06-26 05:56 That command has access to a variable named $SSH_ORIGINAL_COMMAND 2008-06-26 05:56 Right 2008-06-26 05:57 So you have a small script that checks to make sure $SSH_ORIGINAL_COMMAND starts with zumastor or ddsnap 2008-06-26 05:57 Would you wrap the ssh-trigger-verify into zumastor, or leave it standalone 2008-06-26 05:57 And disallow things like i/o redirection 2008-06-26 05:57 Well it needs to be standalone for a reason 2008-06-26 05:57 Actually, it doesn't 2008-06-26 05:57 You could just add to zumastor a function to verify arguments 2008-06-26 05:58 It would be easier with a seperate script I think and have the script call zumastor. What do you think? 2008-06-26 05:58 Essentially you could ssh in with the proper key and just do "zumastor some command" 2008-06-26 05:59 A small verification script would make sure the command being ran was either zumastor or ddsnap and then run $SSH_ORIGINAL_COMMAND. If you wrap that into zumastor itsself, zumastor will need to exec itsself. Is that ok? 2008-06-26 06:01 I'd almost like to see zumastor only call zumastor on remote hosts 2008-06-26 06:01 I don't like the direct calls to ddsnap remotely or the 'cat'ing of files remotly to verify their status 2008-06-26 06:02 Ok so in the authorized_keys you have something along the lines of: 2008-06-26 06:02 no-port-forwarding,no-X11-forwarding,no-agent-forwarding,command="/usr/bin/zumastor --verify-ssh" 2008-06-26 06:03 Or something along those lines where it checks the command isn't malicious and then execs zumastor command 2008-06-26 06:03 Should this go to the ML? 2008-06-26 06:21 You can either do that 2008-06-26 06:22 or comment on issue 40 2008-06-26 06:22 http://code.google.com/p/zumastor/issues/detail?id=40 2008-06-26 06:24 ok 2008-06-26 06:29 -!- pgquiles(~pgquiles@212.Red-88-0-136.dynamicIP.rima-tde.net) has left #zumastor 2008-06-26 06:46 -!- pgquiles(~pgquiles@212.Red-88-0-136.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-26 06:53 zissues: Issue 40 in zumastor: bidirectional root ssh authentication required 2008-06-26 10:18 03vandebo * r1715 10/trunk/cbtb/tests/1/snapshot-microtests.sh: 2008-06-26 10:18 A new ddsnap test. This test creates various configurations of snapshots and 2008-06-26 10:18 modifications to the origin and those snapshots. It tests those results 2008-06-26 10:18 against a golden value that is generated on the fly by writing to a file and 2008-06-26 10:18 copying that file for snapshots. 2008-06-26 10:47 zbuild: zumastor b0.10.0 r1715 build success 2008-06-26 10:51 -!- mitchg(~mitchg@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-26 10:52 zbuild: zumastor b0.10.0 r1715 install success 2008-06-26 12:58 http://code.google.com/p/shflags/wiki/Documentation10x 2008-06-26 13:00 perhaps that would make zumastor a whole lot shorter/more readable 2008-06-26 13:27 zbuild: zumastor b0.10.0 r1715 test failure 1 2008-06-26 13:35 03vandebo * r1716 10/trunk/ (4 files in 3 dirs): 2008-06-26 13:35 Change the package rule to enable a nostrip option 2008-06-26 13:35 Change the uml script to copy out the zumastor logs after each test. 2008-06-26 13:35 Update the howto to include a uml problem that might come up because of the loggin change. 2008-06-26 13:42 03vandebo * r1717 10/trunk/cbtb/tests/1/snapshot-microtests.sh: Add expect fail to new test and attempt to debug it (works in uml cbtb) 2008-06-26 13:52 zbuild: zumastor b0.10.0 r1716 build success 2008-06-26 13:55 Has any work been done on a ddsnap enabled kernel for zumastor in RHEL/CentOS? 2008-06-26 13:55 I don't know, willn might 2008-06-26 13:56 I'm one of the contributors to redhat's ovirt project and want to use it to setup a virtual zumastor cluster 2008-06-26 13:56 For testing and development 2008-06-26 13:57 zbuild: zumastor b0.10.0 r1717 build success 2008-06-26 14:04 SEJeff: it has been installed on rhel, using the "from source" directions 2008-06-26 14:04 i think that the 2.6.22 patches apply 2008-06-26 14:04 that was a while ago though 2008-06-26 14:06 shapor, To a 2.6.18 based kernel? 2008-06-26 14:06 Really? 2008-06-26 14:07 zbuild: zumastor b0.10.0 r1716 install success 2008-06-26 14:10 SEJeff: no, i must be remembering incorrectly 2008-06-26 14:11 Ok, well it wouldn't be a huge deal to run a custom kernel with an rpm 2008-06-26 14:11 Just kind of too bad 2008-06-26 14:22 zbuild: zumastor b0.10.0 r1717 install success 2008-06-26 14:23 i haven't build an rpm in ... years 2008-06-26 14:24 if you build one perhaps a script or a note on how you did it would be nice 2008-06-26 14:24 ACTION wonders if theres a service like the launchpad ppa for rpms? 2008-06-26 14:24 If I do it you'll get a patch 2008-06-26 14:24 Yeah the opensuse build service 2008-06-26 14:24 sweet :) 2008-06-26 14:24 it would be nice to have autogen packages 2008-06-26 14:24 http://en.opensuse.org/Build_Service 2008-06-26 14:25 at least for releases 2008-06-26 14:25 Thats exactly what you want 2008-06-26 14:25 So should it be a vanilla kernel with the ddsnap code? 2008-06-26 14:25 thats what we were doing before 2008-06-26 14:25 but it seems to make sense to keep the distro patches 2008-06-26 14:26 Yes, but that means you have to massage zumastor patches around them 2008-06-26 14:26 shouldn't be a huge deal 2008-06-26 14:26 also our patches are not too invasive to core stuff 2008-06-26 14:26 So it isn't as easy to automate things. Don't you guys still fiddle with the block layer? Or did jiaying prove bio throttling is not necessary with new kernels 2008-06-26 14:27 although hrm.. when using 2.6.18 i think we still have peter z's anti deadlock patches 2008-06-26 14:27 which are quite invasive, and probably wouldnt apply cleanly 2008-06-26 14:27 Hmmmm wonder if those are already in RHEL/CentOS 2008-06-26 14:27 doubtful 2008-06-26 14:27 which kernel are they shipping? 2008-06-26 14:28 A heavily modified 2.6.18 2008-06-26 14:28 WIth a ton of things backported 2008-06-26 14:31 hm it seems we've dropped support for 2.6.18 2008-06-26 14:31 src/zumaroot/trunk/ddsnap/patches$ echo * 2008-06-26 14:31 2.6.22.18 2.6.23.8 2.6.24.2 2.6.25.4 2008-06-26 14:34 ah this was recently discussed on the list 2008-06-26 14:34 "RHEL5 2.6.18 support? 2008-06-26 14:36 No big deal 2008-06-26 14:36 Most people won't use zumastor seriously until it is upstream anyways 2008-06-26 15:00 shapor still around? 2008-06-26 15:00 Do you mind answering a few questions about the zumastor codebase if you are? 2008-06-26 15:22 shapor will see questions eventually, but I may be able to answer some questions 2008-06-26 15:25 Ok well I'm looking at restricting the commands that can be ran remotely 2008-06-26 15:25 So I found the run_remote function and what calls it 2008-06-26 15:26 SEJeff yeah i'm here 2008-06-26 15:27 run_remote should really only call zumastor 2008-06-26 15:27 even though that is not true right now 2008-06-26 15:27 Mainly only zumastor replicate ($vol|$remotevol) $HOSTNAME $wait, zumastor receive (start|done) $remotevol, ddsnap zumastor status, etc 2008-06-26 15:27 Right 2008-06-26 15:27 anything that needs to be done remotely should be a zumastor command 2008-06-26 15:27 it is a pretty trivial change 2008-06-26 15:28 Ok so that will be the first set of patches in my series 2008-06-26 15:28 excellent :) 2008-06-26 15:28 My questions were mainly on how those commands are really used without wrapping my head around a 1722 line shell script 2008-06-26 15:30 If you say: zumastor replicate $remote_vol $MYHOSTNAME --nowait 2008-06-26 15:30 That ssh's from the origin to the destination and then pulls from the origin? 2008-06-26 15:30 Is that right? 2008-06-26 15:30 Pulling from the destination with: zumastor receive start|done ? 2008-06-26 15:31 it is a push model, not pull 2008-06-26 15:31 so when you run "zumastor replicate" upstream 2008-06-26 15:31 it does a few things 2008-06-26 15:31 1) creates a new snapshot to replicate 2008-06-26 15:31 2) ssh's downstream and starts a "ddsnap listen" 2008-06-26 15:32 3) runs "ddsnap transmit" on upstream, which connects to downstream "listen" process, sending the delta 2008-06-26 15:32 4) ssh's downstream and runs "zumastor replicate done", which kills the listener, set the new snapshot, and rotates them 2008-06-26 15:39 -!- zumalog(~zumalog@yzf.shapor.com) has joined #zumastor 2008-06-26 15:39 -!- SEJeff(~jeff__@66.151.59.138) has joined #zumastor 2008-06-26 15:39 What does ddsnap do, send binary junk over an encrypted ssh stream? 2008-06-26 15:39 -!- CIA-4(~CIA@208.69.182.149) has joined #zumastor 2008-06-26 15:40 -!- zbot(wan@209.67.252.126) has joined #zumastor 2008-06-26 15:41 -!- phillips_(~phillips@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-26 15:41 -!- jiayingz(~jiayingz@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-26 15:41 SEJeff, no ddsnap sends delta plain text over its own socket 2008-06-26 15:41 hence the listen/transmit pair 2008-06-26 15:41 Oh 2008-06-26 15:41 So you send data unencrypted over the wire with ddsnap? 2008-06-26 15:41 oh yeah! 2008-06-26 15:41 Thats hawt 2008-06-26 15:42 indeed 2008-06-26 15:42 i have run it through an ssh tunnel 2008-06-26 15:42 that should be an option 2008-06-26 15:42 So what is the point of ssh if the most important thing (data) is sent unencrypted :) 2008-06-26 15:42 wow 2008-06-26 15:43 Why did danK want to convert it all to a pull? Perhaps I misunderstood his email. Did you read that? 2008-06-26 15:43 i'm pretty sure that is what most commercial replication solutions do too 2008-06-26 15:43 therefore most people just use a vpn 2008-06-26 15:43 although, i agree, it is really bad ;) 2008-06-26 15:43 there no authentication either 2008-06-26 15:44 easy to accidentally blast the wrong port/ip with data 2008-06-26 15:44 and corrupt stuff 2008-06-26 15:44 Not even tls? 2008-06-26 15:44 nothing 2008-06-26 15:44 it jsut waits for a header packet, and then data packets 2008-06-26 15:44 It shouldn't be impossible to use ssl if nothing else 2008-06-26 15:45 yeah that wouldnt be a very hard hack 2008-06-26 15:45 Ok so the only things that aren't zumastor commands are ddsnap status and cat 2008-06-26 15:45 yeah 2008-06-26 15:45 *Aren't zumastor commands that are ran remotely 2008-06-26 15:45 So how should that be done? Would you add 2 new toplevel zumastor commands? 2008-06-26 15:46 maybe additional sub-commands of zumastor receive 2008-06-26 15:46 hmm 2008-06-26 15:46 well the ddsnap status it does just returns size 2008-06-26 15:46 so perhaps a zumastor status --size 2008-06-26 15:47 "zumastor status --size" 2008-06-26 15:48 That should be easy enough 2008-06-26 15:48 yeah 2008-06-26 15:48 and perhaps useful othersize 2008-06-26 15:48 s/size/wise/ 2008-06-26 15:48 "cat $VOLUMES/$remote_vol/source/hostname" 2008-06-26 15:49 What is the actual point of that? 2008-06-26 15:49 that is to check to make sure you are indeed the machine it expects data from 2008-06-26 15:49 Why does it care? 2008-06-26 15:49 "tell me who your upstream is?" 2008-06-26 15:49 just a safety check really 2008-06-26 15:50 Well if the strategy is push only, why does it care? As in how is that truly a safety check? 2008-06-26 15:50 What does it prevent? The fact that the ddsnap stuff is dumb at the moment? 2008-06-26 15:53 its essentially a check to make sure that the configuration matches upstream and downstream 2008-06-26 15:53 ok 2008-06-26 15:54 Are there cases where that could potentially be wrong? 2008-06-26 15:54 In specific cases when setup with the zumastor command? 2008-06-26 15:54 might be nice to ask "zumastor status --upstream" 2008-06-26 15:54 which either tells you who the upstream is 2008-06-26 15:54 or that it is master 2008-06-26 15:54 hmm.. maybe 2008-06-26 15:55 Ok thats not a bad idea 2008-06-26 15:55 Is that implemented? 2008-06-26 15:55 no, just thinking out loud 2008-06-26 15:56 zumastor status --upstream would just replace cat $VOLUMES/$remote_vol/source/hostname, right? 2008-06-26 15:56 this is one argument for adding get/set to configure 2008-06-26 15:56 yes 2008-06-26 15:56 zumastor get 2008-06-26 15:56 Ok and in the case you are upstream, what should it do? 2008-06-26 15:57 Would it just be cat $VOLUMES/$vol/source/hostname? 2008-06-26 15:59 if you are upstream it should probably communicate that somehow 2008-06-26 15:59 perhaps an error code 2008-06-26 15:59 or some string which could never be a hostname 2008-06-26 15:59 *UPSTREAM* 2008-06-26 16:01 Has anyone tried zumastor under kvm? UML is really slow 2008-06-26 16:03 yeah drake did a lot 2008-06-26 16:03 buthe found kvm unstable 2008-06-26 16:03 iirc 2008-06-26 16:03 s/th/t h/ 2008-06-26 16:03 When did he try it? 2008-06-26 16:04 And do you know with what kvm version? It has came a LONG way 2008-06-26 16:04 it wasn't too long ago 2008-06-26 16:04 Do you know with what kvm version and what was "unstable" about it? 2008-06-26 16:05 random hangs on boot i think 2008-06-26 16:05 Let me guess, you were doing it with the old kernel in dapper? 2008-06-26 16:05 "kvm isn't 2008-06-26 16:05 reliably running the tests or installs, so those happen in qemu. " 2008-06-26 16:05 from drake on the mailing list 2008-06-26 16:05 you'd really have to ask him 2008-06-26 16:05 ok 2008-06-26 16:05 heh, sorry 2008-06-26 16:06 No problem 2008-06-26 16:06 but it seemed to work fine for the builds 2008-06-26 16:06 since they were way too slow on qemu 2008-06-26 16:07 Well I'd like to get zumastor up and running under ovirt 2008-06-26 16:07 It is planning on being the FLOSS equivalent of ESX 2008-06-26 16:10 It is based on kvm 2008-06-26 16:15 cool 2008-06-26 16:20 http://ovirt.org/ 2008-06-26 16:31 keep us updated, sounds cool 2008-06-26 16:31 let me know if you need any other zumastor insight 2008-06-26 16:33 Will do, thanks for the ideas 2008-06-26 16:33 Saving an irc log and will start hacking sometime in the next week 2008-06-26 16:43 zbuild: zumastor b0.10.0 r1716 test success 2008-06-26 17:32 -!- vandebo(~vandebo@abydos.nerdbox.net) has joined #zumastor 2008-06-26 17:32 hi steve 2008-06-26 17:32 hey 2008-06-26 18:11 ACTION waves 2008-06-26 19:23 zbuild: zumastor b0.10.0 r1717 test success 2008-06-26 21:45 -!- MaZe(~MaZe@c-24-6-86-168.hsd1.ca.comcast.net) has joined #zumastor irc.oftc.net #zumastor log beginning Fri Jun 27 00:00:01 PDT 2008 2008-06-27 00:38 -!- CIA-4(~CIA@208.69.182.149) has joined #zumastor 2008-06-27 00:39 -!- MaZe(~MaZe@c-24-6-86-168.hsd1.ca.comcast.net) has joined #zumastor 2008-06-27 00:39 -!- flipz(~phillips@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-27 00:39 -!- nataliep(~nataliep@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-27 01:08 -!- MaZe(~MaZe@c-24-6-86-168.hsd1.ca.comcast.net) has joined #zumastor 2008-06-27 01:08 -!- flipz(~phillips@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-27 01:08 -!- nataliep(~nataliep@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-27 02:32 -!- pgquiles(~pgquiles@183.Red-79-152-102.staticIP.rima-tde.net) has joined #zumastor 2008-06-27 07:29 -!- MaZe(~MaZe@c-24-6-86-168.hsd1.ca.comcast.net) has joined #zumastor 2008-06-27 10:08 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-27 17:10 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor irc.oftc.net #zumastor log beginning Sat Jun 28 00:00:01 PDT 2008 2008-06-28 00:18 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-28 02:01 -!- MaZe(~MaZe@c-24-6-86-168.hsd1.ca.comcast.net) has joined #zumastor 2008-06-28 12:15 -!- nataliep_(~nataliep@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-28 12:28 -!- MaZe(~MaZe@c-24-6-86-168.hsd1.ca.comcast.net) has joined #zumastor 2008-06-28 12:28 -!- flipz(~phillips@207.47.98.129.static.nextweb.net) has joined #zumastor 2008-06-28 16:33 -!- pgquiles(~pgquiles@216.Red-81-44-176.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-28 19:19 -!- pgquiles_(~pgquiles@216.Red-81-44-176.dynamicIP.rima-tde.net) has joined #zumastor irc.oftc.net #zumastor log beginning Sun Jun 29 00:00:01 PDT 2008 2008-06-29 15:33 versioned volumes fuzz test now runs to 10,000 iterations 2008-06-29 15:34 each iteration a random write to a random snapshot, and every ten or so iterations randomly create a new snapshot of a snapshot or delete a snapshot 2008-06-29 15:35 10,000 is as high as I've tested 2008-06-29 15:35 one glitch to fix in incremental upate of version tree during version delete 2008-06-29 19:12 -!- pgquiles__(~pgquiles@130.Red-83-38-37.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-29 20:12 glitch fixed, now it runs to 100,000, haven't tested it higher 2008-06-29 20:12 found about 4 bugs in total with the fuzz tester 2008-06-29 20:12 and dozens with unit testing 2008-06-29 20:13 full system testing is a weak method in comparison 2008-06-29 20:13 (a word to dkegel) 2008-06-29 22:36 -!- erwan_taf(~erwan@81.80.43.67) has joined #zumastor irc.oftc.net #zumastor log beginning Mon Jun 30 00:00:01 PDT 2008 2008-06-30 00:13 -!- pgquiles_(~pgquiles@62.43.226.52.static.user.ono.com) has joined #zumastor 2008-06-30 00:19 -!- pgquiles__(~pgquiles@130.Red-83-38-37.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-30 00:20 -!- IceGuest_7(~IceChat7@202.106.180.68) has joined #zumastor 2008-06-30 00:27 -!- zumastor(~IceChat7@202.106.180.68) has joined #zumastor 2008-06-30 00:54 -!- zumastor_(~IceChat7@202.106.180.68) has joined #zumastor 2008-06-30 00:54 -!- erwan_taf(~erwan@81.80.43.67) has joined #zumastor 2008-06-30 02:15 -!- zumastor_(~IceChat7@202.106.180.68) has joined #zumastor 2008-06-30 06:03 -!- zumastor_(~IceChat7@221.219.14.187) has joined #zumastor 2008-06-30 11:05 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2008-06-30 12:55 -!- CIA-4(~CIA@208.69.182.149) has joined #zumastor 2008-06-30 13:07 -!- pgquiles__(~pgquiles@154.Red-81-38-97.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-30 13:13 -!- pgquiles__(~pgquiles@76.Red-81-39-154.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-30 13:56 -!- pgquiles__(~pgquiles@27.Red-81-44-176.dynamicIP.rima-tde.net) has joined #zumastor 2008-06-30 15:09 -!- CIA-5(~CIA@208.69.182.149) has joined #zumastor 2008-06-30 16:07 03Daniel.Raymond.Phillips * r1718 10/trunk/ddsnap/ (ddsnap-sb.c ddsnapd.c): 2008-06-30 16:07 Apply patch from first time contributor hhuu to clarify what 2008-06-30 16:07 the user is expected to do in response to an error about a superblock mismatch. 2008-06-30 16:31 zbuild: zumastor b0.10.0 r1718 build success 2008-06-30 16:41 zbuild: zumastor b0.10.0 r1718 install success 2008-06-30 17:14 -!- zumastor_(~IceChat7@202.106.180.68) has joined #zumastor 2008-06-30 17:42 -!- zumastor_(~IceChat7@202.106.180.68) has joined #zumastor 2008-06-30 18:24 -!- zumastor_(~IceChat7@202.106.180.68) has joined #zumastor 2008-06-30 19:12 zbuild: zumastor b0.10.0 r1718 test success 2008-06-30 21:05 versioned volumes fuzz test now includes origin writes 2008-06-30 21:06 vandebo, as a matter of fact there was a bug 2008-06-30 21:06 but since it was the same bug as the last one, it it didn't last long 2008-06-30 23:18 -!- zumastor(~IceChat7@202.106.180.68) has joined #zumastor 2008-06-30 23:27 hi 2008-06-30 23:36 -!- jiayingz_(~IceChat7@202.106.180.68) has joined #zumastor 2008-06-30 23:37 hi jiayingz_ 2008-06-30 23:37 hi 2008-06-30 23:37 how are things in beijing? 2008-06-30 23:38 it is ok. trying to get zumastor running here 2008-06-30 23:39 time to sleep here 2008-06-30 23:39 good luck 2008-06-30 23:39 indeed 2008-06-30 23:39 see all the progress with improved snapshot encoding 2008-06-30 23:40 steve is going to take a run at implementing it 2008-06-30 23:40 new bugs... 2008-06-30 23:40 i thought u were implementing that 2008-06-30 23:40 -!- jiayingz__(~jiayingz@202.106.180.68) has joined #zumastor 2008-06-30 23:40 just the unit testing version of it 2008-06-30 23:41 i c 2008-06-30 23:41 how is the hanging problem? 2008-06-30 23:41 still waiting to see a trace with frame pointers enabled 2008-06-30 23:41 the traces didn't make a lot of sense last time 2008-06-30 23:41 we haven't seen any hangs on our own machines 2008-06-30 23:42 i heard there is some almost hanging problem on the new server