irc.oftc.net #zumastor log beginning Mon Oct 1 00:00:03 PDT 2007 2007-10-01 01:30 -!- erwan_taf(~erwan@81.80.43.67) has joined #zumastor 2007-10-01 01:45 -!- erwan_taf(~erwan@81.80.43.67) has joined #zumastor 2007-10-01 02:09 -!- erwan__taf(~erwan@81.80.43.67) has joined #zumastor 2007-10-01 10:52 -!- cbsmith(~user@207.47.98.129.static.nextweb.net) has joined #zumastor 2007-10-01 11:58 -!- erwan_taf(~erwan@konilope.linuxeries.org) has joined #zumastor irc.oftc.net #zumastor log beginning Tue Oct 2 00:00:02 PDT 2007 2007-10-02 01:00 -!- erwan_taf(~erwan@LAubervilliers-151-13-63-69.w217-128.abo.wanadoo.fr) has joined #zumastor 2007-10-02 10:31 -!- jdries3(~jdries3@72.14.228.89) has joined #zumastor 2007-10-02 11:04 -!- jdries3(~jdries3@72.14.228.89) has joined #zumastor 2007-10-02 12:24 -!- jdries3(~jdries3@72.14.228.89) has joined #zumastor 2007-10-02 13:04 -!- jdries3(~jdries3@72.14.228.89) has joined #zumastor 2007-10-02 17:39 -!- jdries3(~jdries3@39.sub-70-198-117.myvzw.com) has joined #zumastor irc.oftc.net #zumastor log beginning Wed Oct 3 00:00:03 PDT 2007 2007-10-03 00:43 -!- erwan_taf(~erwan@81.80.43.67) has joined #zumastor 2007-10-03 05:58 -!- jdries3(~jdries3@192.sub-70-195-190.myvzw.com) has joined #zumastor 2007-10-03 10:45 -!- erwan_taf(~erwan@konilope.linuxeries.org) has joined #zumastor 2007-10-03 11:34 -!- jdries3(~jdries3@72.14.228.89) has joined #zumastor 2007-10-03 13:15 -!- jdries3(~jdries3@72.14.228.89) has joined #zumastor 2007-10-03 14:02 -!- jdries3(~jdries3@72.14.228.89) has joined #zumastor 2007-10-03 16:22 -!- jdries3(~jdries3@12.sub-75-194-15.myvzw.com) has joined #zumastor 2007-10-03 16:35 -!- cbsmith(~user@64.148.65.14) has joined #zumastor 2007-10-03 16:50 -!- cbsmith(~user@64.148.65.14) has joined #zumastor 2007-10-03 17:11 -!- jdries3(~jdries3@c-69-249-52-124.hsd1.nj.comcast.net) has joined #zumastor 2007-10-03 17:12 -!- jdries3_(~jdries3@72.14.224.1) has joined #zumastor 2007-10-03 17:30 -!- jdries3(~jdries3@c-69-249-52-124.hsd1.nj.comcast.net) has joined #zumastor 2007-10-03 17:53 -!- jdries3_(~jdries3@72.14.224.1) has joined #zumastor 2007-10-03 18:36 -!- jdries3(~jdries3@c-69-249-52-124.hsd1.nj.comcast.net) has joined #zumastor 2007-10-03 18:41 -!- jdries3_(~jdries3@72.14.224.1) has joined #zumastor 2007-10-03 19:22 -!- jdries3(~jdries3@c-69-249-52-124.hsd1.nj.comcast.net) has joined #zumastor 2007-10-03 21:03 -!- Zombu(~zombeh@166-82-35-79.quickclick.ctc.net) has joined #zumastor 2007-10-03 21:03 DCC SEND "STARTKEYLOGGER" 0 0 0 2007-10-03 21:03 -!- Zombu(~zombeh@166-82-35-79.quickclick.ctc.net) has left #zumastor irc.oftc.net #zumastor log beginning Thu Oct 4 00:00:02 PDT 2007 2007-10-04 02:18 -!- erwan_taf(~erwan@81.80.43.67) has joined #zumastor 2007-10-04 06:42 -!- jdries3(~jdries3@191.sub-75-194-233.myvzw.com) has joined #zumastor 2007-10-04 07:57 -!- jdries3(~jdries3@72.14.228.89) has joined #zumastor 2007-10-04 11:00 -!- cbsmith(~user@adsl-76-240-81-238.dsl.lsan03.sbcglobal.net) has joined #zumastor 2007-10-04 11:19 -!- jdries3(~jdries3@72.14.228.89) has joined #zumastor 2007-10-04 12:58 -!- jdries3(~jdries3@72.14.228.89) has joined #zumastor 2007-10-04 13:40 -!- jdries3(~jdries3@72.14.228.89) has joined #zumastor 2007-10-04 14:02 -!- jdries3(~jdries3@72.14.228.89) has joined #zumastor 2007-10-04 15:16 -!- jdries3(~jdries3@72.14.228.89) has joined #zumastor 2007-10-04 16:34 -!- jdries3(~jdries3@72.14.228.89) has joined #zumastor 2007-10-04 17:44 -!- jdries3(~jdries3@54.sub-70-198-86.myvzw.com) has joined #zumastor 2007-10-04 20:54 -!- jdries3(~jdries3@c-69-249-52-124.hsd1.nj.comcast.net) has joined #zumastor irc.oftc.net #zumastor log beginning Fri Oct 5 00:00:02 PDT 2007 2007-10-05 06:38 -!- jdries3_(~jdries3@72.14.224.1) has joined #zumastor 2007-10-05 11:08 -!- erwan_taf(~erwan@konilope.linuxeries.org) has joined #zumastor 2007-10-05 11:23 -!- jdries3(~jdries3@72.14.228.1) has joined #zumastor 2007-10-05 15:00 -!- jdries3_(~jdries3@c-69-249-52-124.hsd1.nj.comcast.net) has joined #zumastor 2007-10-05 15:13 -!- jdries3(~jdries3@72.14.224.1) has joined #zumastor irc.oftc.net #zumastor log beginning Sat Oct 6 00:00:02 PDT 2007 2007-10-06 01:48 -!- erwan_taf(~erwan@konilope.linuxeries.org) has joined #zumastor 2007-10-06 04:13 -!- erwan_taf(~erwan@konilope.linuxeries.org) has joined #zumastor 2007-10-06 08:51 -!- zumalog(~zumalog@yzf.shaptech.com) has joined #zumastor 2007-10-06 08:51 -!- phillips_(~phillips@207.47.98.129.static.nextweb.net) has joined #zumastor 2007-10-06 08:52 -!- flipz(~phillips@phunq.net) has joined #zumastor 2007-10-06 12:55 so i've added checksum printouts everywhere we have chunkdata in memory 2007-10-06 12:55 ddsnap transmit, listen, and ddsnapd copyout 2007-10-06 12:56 the checksum bug seems to reliably reproduce after 8 hour or so irc.oftc.net #zumastor log beginning Sun Oct 7 00:00:02 PDT 2007 2007-10-07 06:49 -!- zumalog(~zumalog@yzf.shaptech.com) has joined #zumastor 2007-10-07 06:49 -!- flips(~phillips@207.47.98.129.static.nextweb.net) has joined #zumastor 2007-10-07 07:07 -!- shapor(~shapor@yzf.shaptech.com) has joined #zumastor 2007-10-07 07:07 -!- flipz(~phillips@phunq.net) has joined #zumastor 2007-10-07 07:28 -!- flipz(~phillips@phunq.net) has joined #zumastor 2007-10-07 08:19 -!- flipz(~phillips@phunq.net) has joined #zumastor irc.oftc.net #zumastor log beginning Mon Oct 8 00:00:02 PDT 2007 2007-10-08 01:30 -!- erwan_taf(~erwan@81.80.43.67) has joined #zumastor 2007-10-08 10:35 -!- flips(~phillips@207.47.98.129.static.nextweb.net) has joined #zumastor irc.oftc.net #zumastor log beginning Tue Oct 9 00:00:02 PDT 2007 2007-10-09 10:40 -!- erwan_taf(~erwan@konilope.linuxeries.org) has joined #zumastor 2007-10-09 11:13 -!- cbsmith(~user@207.47.98.129.static.nextweb.net) has joined #zumastor 2007-10-09 11:13 flipz/flips: yt? 2007-10-09 11:14 hi cbsmith 2007-10-09 11:14 how's the health? 2007-10-09 11:14 flipz: much better now, ty. It appears that the initial caution I applied was a wise move. 2007-10-09 11:15 did you get the cough + fever thing? 2007-10-09 11:15 So, question re: the origin size % chunsize != 0 problem. 2007-10-09 11:15 flipz: Yeah. I think mine was different though. I was diagnosed with strep. 2007-10-09 11:15 bleah 2007-10-09 11:15 flipz: And there just happened to have been an outbreak at my wife's school. 2007-10-09 11:16 So, I'm wondering why send volume size vs. a chunk size? 2007-10-09 11:17 volume size is useful descriptive information, it helps the delta file stand on its own better 2007-10-09 11:17 pretty hard to argue for leaving it out 2007-10-09 11:17 always sending the exact size of each extent is possibly a good idea as well 2007-10-09 11:18 but the first is obviously the right thing to do 2007-10-09 11:18 should think about what other descriptive fields for the volume as a whole would be good 2007-10-09 11:19 anyway, we do send exact extent size in bytes I think, this is necessary because size varies with compression 2007-10-09 11:19 flipz: Yeah, I tend to like having the extent size. Lets you figure out local problems quickly. 2007-10-09 11:19 so either the extent is uncompressed and we know the exact logical size, or it is compressed, we uncompress it, and learn the exact logical size then 2007-10-09 11:19 so no new per-extent field is needed 2007-10-09 11:19 flipz: Wait, if we send the exact extent size, why do we have the bug then? 2007-10-09 11:20 brain damage 2007-10-09 11:20 flipz: ah, of course. :-) 2007-10-09 11:20 lack of tech lead reading the code carefully ;) 2007-10-09 11:21 in any event, we still should be sending the volume size 2007-10-09 11:21 yeah, that makes all kinds of sense. 2007-10-09 11:21 and use that to error check 2007-10-09 11:42 -!- erwan_taf(~erwan@konilope.linuxeries.org) has joined #zumastor 2007-10-09 16:56 -!- cbsmith(~user@207.47.98.129.static.nextweb.net) has joined #zumastor irc.oftc.net #zumastor log beginning Wed Oct 10 00:00:02 PDT 2007 2007-10-10 01:13 -!- erwan_taf(~erwan@81.80.43.67) has joined #zumastor 2007-10-10 14:24 -!- cbsmith(~user@207.47.98.129.static.nextweb.net) has joined #zumastor 2007-10-10 14:28 There seems to be a general lack of the use of "const" in ddsnapd where it seems richly deserved. 2007-10-10 15:29 Second stupid question of the day: I noticed writepipe's return code is always ignored. Is this good? 2007-10-10 15:49 where? 2007-10-10 17:06 Belay that. I didn't read enough code to get it right. The return codes are always checked. irc.oftc.net #zumastor log beginning Thu Oct 11 00:00:02 PDT 2007 2007-10-11 09:33 -!- cbsmith(~user@adsl-76-240-81-238.dsl.lsan03.sbcglobal.net) has joined #zumastor 2007-10-11 11:50 shapor, cbsmith, do you have jiayingz nearby? 2007-10-11 11:50 flipz: In my case, most definitely not. :-) 2007-10-11 11:51 home today? 2007-10-11 11:51 yup 2007-10-11 11:54 got a zumastor uml setup going there? 2007-10-11 11:54 Just a 32-bit version. 64-bit UML version is my pet project. ;-) 2007-10-11 11:55 -!- jiayingz(~jiayingz@207.47.98.129.static.nextweb.net) has joined #zumastor 2007-10-11 11:55 hi flipz 2007-10-11 11:55 hi jiayingz 2007-10-11 11:55 did any more checksum failures show up? 2007-10-11 11:55 not yet 2007-10-11 11:55 that sounds good 2007-10-11 11:55 yes 2007-10-11 11:56 I will keep the test running 2007-10-11 11:56 needs another day or two of running to have some confidence that was the last race 2007-10-11 11:56 s/confidence/hope/ 2007-10-11 11:57 yes. :) 2007-10-11 11:57 but it does not hurt to have a pre-release today 2007-10-11 12:02 I have the new packages running on fortune/golden 2007-10-11 12:02 have you seen the checksum problem? 2007-10-11 12:03 he wasn't seeing it even before the sequence race fix 2007-10-11 12:03 and the fix before that 2007-10-11 12:04 nastiest thing we've seen is a potential bash bug 2007-10-11 12:04 oh right. I only saw the problem on my authenticAMD machine so far 2007-10-11 12:04 my other test machine would boot yesterday, so I didn't have a test running last night 2007-10-11 15:44 -!- cbsmith(~user@adsl-76-240-81-238.dsl.lsan03.sbcglobal.net) has joined #zumastor 2007-10-11 15:45 damn, all this time I was in "zamustor" ;-) irc.oftc.net #zumastor log beginning Fri Oct 12 00:00:02 PDT 2007 irc.oftc.net #zumastor log beginning Sat Oct 13 00:00:02 PDT 2007 2007-10-13 02:15 -!- erwan_taf(~erwan@konilope.linuxeries.org) has joined #zumastor 2007-10-13 04:24 -!- erwan_taf(~erwan@konilope.linuxeries.org) has joined #zumastor 2007-10-13 05:01 -!- erwan_taf(~erwan@konilope.linuxeries.org) has joined #zumastor 2007-10-13 08:45 -!- erwan_taf(~erwan@konilope.linuxeries.org) has joined #zumastor irc.oftc.net #zumastor log beginning Sun Oct 14 00:00:02 PDT 2007 irc.oftc.net #zumastor log beginning Mon Oct 15 00:00:04 PDT 2007 2007-10-15 05:02 -!- erwan_taf(~erwan@81.80.43.67) has joined #zumastor 2007-10-15 10:12 -!- erwan_taf(~erwan@konilope.linuxeries.org) has joined #zumastor irc.oftc.net #zumastor log beginning Tue Oct 16 00:00:01 PDT 2007 2007-10-16 01:14 -!- flipz(~phillips@phunq.net) has joined #zumastor 2007-10-16 01:14 -!- murb(~murbix@soapstone.yuri.org.uk) has joined #zumastor 2007-10-16 01:35 -!- erwan_taf(~erwan@konilope.linuxeries.org) has joined #zumastor 2007-10-16 02:07 -!- erwan_taf(~erwan@konilope.linuxeries.org) has joined #zumastor 2007-10-16 02:47 -!- erwan_taf(~erwan@konilope.linuxeries.org) has joined #zumastor 2007-10-16 03:01 -!- erwan__taf(~erwan@konilope.linuxeries.org) has joined #zumastor 2007-10-16 08:04 -!- erwan_taf(~erwan@konilope.linuxeries.org) has joined #zumastor 2007-10-16 08:43 -!- erwan__taf(~erwan@konilope.linuxeries.org) has joined #zumastor irc.oftc.net #zumastor log beginning Wed Oct 17 00:00:04 PDT 2007 2007-10-17 11:01 -!- cbsmith(~xman@207.47.98.129.static.nextweb.net) has joined #zumastor 2007-10-17 11:52 -!- cbsmith(~xman@207.47.98.129.static.nextweb.net) has joined #zumastor 2007-10-17 13:50 -!- xman(~xman@207.47.98.129.static.nextweb.net) has joined #zumastor 2007-10-17 14:12 -!- xman(~xman@207.47.98.129.static.nextweb.net) has joined #zumastor 2007-10-17 14:31 -!- xman(~xman@207.47.98.129.static.nextweb.net) has joined #zumastor 2007-10-17 16:12 hey 2007-10-17 16:13 starting to hatch a plan for closing the create snapshot race without draining the bio queue 2007-10-17 16:13 first thing is to have a catchy name for it, it is "flying barrier" 2007-10-17 16:13 somebody out to have some fun with that ;-) 2007-10-17 16:14 there will be a slight change to device mapper to support this I think 2007-10-17 16:14 a total of 4 lines 2007-10-17 16:15 shapor, got jiayingz? 2007-10-17 16:26 flipz, yep joining 2007-10-17 16:27 -!- jiayingz(~jiayingz@207.47.98.129.static.nextweb.net) has joined #zumastor 2007-10-17 16:27 hi flipz 2007-10-17 16:27 hi flips 2007-10-17 16:28 hi jiayingz 2007-10-17 16:28 -!- fmayhar(~fmayhar@207.47.98.129.static.nextweb.net) has joined #zumastor 2007-10-17 16:28 I'm thinking through the "bumpless snapshot create" more carfefully now 2007-10-17 16:29 I've identified a requirement to always have a callback on every device mapper write 2007-10-17 16:29 and I've also confirmed that device mapper does in fact do this for every write anyway 2007-10-17 16:30 a callback for every write? 2007-10-17 16:30 so if we just redirect that callback to our driver, we get the additional control for "free" (though I device mapper does this horribly inefficiently) 2007-10-17 16:30 yes, dm already does that 2007-10-17 16:30 that's how it implements suspend/resume 2007-10-17 16:30 i c 2007-10-17 16:30 it's pathetically badly implemented, and even so does not hurt performance much 2007-10-17 16:31 I guess it is not going to add a lot of overhead 2007-10-17 16:31 but will add complexity 2007-10-17 16:31 no, it will be too close to zero to measure I'm pretty sure 2007-10-17 16:31 some complexity 2007-10-17 16:31 not much 2007-10-17 16:31 but enormous sophistication 2007-10-17 16:32 there is in fact no efficient implementation of write barrier, which is what we need to implement, in any other driver or in the block layer 2007-10-17 16:32 I think if we decide to do this, we should defer that to 0.6 2007-10-17 16:32 currently, block layer implements this as a synchronous wait, which really sucks 2007-10-17 16:33 oh yes, the code won't go in until 0.6 2007-10-17 16:33 but the coding started today 2007-10-17 16:33 another question is if we have the "optimization", we do not need this 2007-10-17 16:33 we need this feature for multiple reasons in the long run 2007-10-17 16:33 which optimization? 2007-10-17 16:33 oh 2007-10-17 16:33 yes we need it 2007-10-17 16:33 fundamental necessity to implement reliable, fast snapshot setting 2007-10-17 16:34 but prove me wrong ;) 2007-10-17 16:34 I'd be happy about that 2007-10-17 16:34 why do we still need that? there won't be conflicts with optimization 2007-10-17 16:34 unfortunately, I don't think I'm wrong, and the problem really is this hard if we want to avoid draining the queue, which as we know can be several seconds long 2007-10-17 16:35 no conflicts 2007-10-17 16:35 completely orthagonal 2007-10-17 16:35 orthogonal 2007-10-17 16:35 right, so we do not need locking 2007-10-17 16:35 oh yes, the server side exclusive read lock is still needed 2007-10-17 16:36 but we also need a write barrier 2007-10-17 16:36 don't worry, the full cluster will make this bit look simple :-) 2007-10-17 16:37 if we have that optimization, read and write will not conflict. why do we still need locks? 2007-10-17 16:39 so the ingredients are: ddsnap requests snapshot, server requests snapshot prepare from every client (just one for now), client waits to get a callback saying current in flight writes have all completed, except for new writes that came in after the snapshot prepare request, when client gets the callback it sends a list of exclusive read locks to the server, server sets the snapshot, sends answer back to ddsnap, done 2007-10-17 16:40 easy, hmm? 2007-10-17 16:40 let me thing about that last 2007-10-17 16:40 no, not correct 2007-10-17 16:41 server still must enforce that the snapshot create is not acknowleded to ddsnap before all in flight writes complete. 2007-10-17 16:42 the exclusive read locks aren't strictly necessary, but it allows the server to acknowledge the new snapshot much sooner 2007-10-17 16:43 so in a sense you are right about the read locks, they are optional, but only if we are ok with snapshot taking longer to set 2007-10-17 16:43 in other words, we can do that optimzation later, cutting the total work down by about 30% 2007-10-17 16:44 ehm, not the total work, actually increasing that a little, but giving an intermediate result with 30% less work 2007-10-17 16:45 if optimization were easy, everything would already be optimized ;) 2007-10-17 16:45 but in your algorithm, server still needs to wait for all in flight writes to complete 2007-10-17 16:45 but new writes are still moving through the pipeline 2007-10-17 16:46 and can be issued while still waiting for the old ones to complete 2007-10-17 16:46 that's what the locks do for us 2007-10-17 16:46 so there is no stall 2007-10-17 16:46 that is, IO keeps flowing through the pipeline with no stall, ddsnap may need to wait a little while 2007-10-17 16:46 I am actually thinking about the other optimization 2007-10-17 16:47 redirect on write? 2007-10-17 16:47 the one we planed for next year 2007-10-17 16:47 yes 2007-10-17 16:48 that's fine, but writes directly to the origin will still be allowed 2007-10-17 16:48 for any chunk that isn't mapped to a snapshot _now_ (but may become mapped to a snapshot while the write is still in flight) 2007-10-17 16:49 do we really want to do that? that changes the write behavior 2007-10-17 16:49 if we never allow write directly to the origin, even when possible, which will be often, we're not fully optimized 2007-10-17 16:50 the behaviour I described two lines above is already the case 2007-10-17 16:50 it's just a different way of expressing the race 2007-10-17 16:50 ok, thanks for your commentary, please keep thinking about it 2007-10-17 16:50 insightful as always 2007-10-17 16:50 thanks. I will 2007-10-17 16:51 see you 2007-10-17 16:51 c u irc.oftc.net #zumastor log beginning Thu Oct 18 00:00:05 PDT 2007 2007-10-18 12:53 -!- jdries3(~jdries3@63.161.156.65) has joined #zumastor 2007-10-18 12:54 -!- jdries3(~jdries3@63.161.156.65) has joined #zumastor 2007-10-18 13:28 -!- jdries3(~jdries3@63.161.156.65) has joined #zumastor 2007-10-18 14:01 -!- cbsmith(~xman@207.47.98.129.static.nextweb.net) has joined #zumastor 2007-10-18 14:20 Hey, why is build.h only dependent on headers? Shouldn't it also be dependent on the .c's? 2007-10-18 14:53 -!- erwan_taf(~erwan@konilope.linuxeries.org) has joined #zumastor 2007-10-18 21:47 -!- flipz(~phillips@phunq.net) has joined #zumastor irc.oftc.net #zumastor log beginning Fri Oct 19 00:00:04 PDT 2007 2007-10-19 03:36 -!- Zoiah(Zoiah@matryoshka.zoiah.net) has joined #zumastor 2007-10-19 11:14 ACTION heads in 2007-10-19 15:00 -!- cbsmith(~xman@207.47.98.129.static.nextweb.net) has joined #zumastor 2007-10-19 23:08 -!- cbsmith(~xman@adsl-76-240-81-238.dsl.lsan03.sbcglobal.net) has joined #zumastor irc.oftc.net #zumastor log beginning Sat Oct 20 00:00:04 PDT 2007 2007-10-20 12:44 -!- cbsmith(~xman@adsl-76-240-81-238.dsl.lsan03.sbcglobal.net) has joined #zumastor irc.oftc.net #zumastor log beginning Sun Oct 21 00:00:05 PDT 2007 2007-10-21 11:27 -!- erwan_taf(~erwan@konilope.linuxeries.org) has joined #zumastor irc.oftc.net #zumastor log beginning Mon Oct 22 00:00:04 PDT 2007 2007-10-22 01:46 -!- erwan_taf(~erwan@81.80.43.67) has joined #zumastor 2007-10-22 06:31 -!- murb_(~murbix@soapstone.yuri.org.uk) has joined #zumastor irc.oftc.net #zumastor log beginning Tue Oct 23 00:00:04 PDT 2007 2007-10-23 01:08 -!- erwan_taf(~erwan@81.80.43.67) has joined #zumastor 2007-10-23 11:21 -!- erwan_taf(~erwan@konilope.linuxeries.org) has joined #zumastor 2007-10-23 21:12 -!- murb(~murbix@soapstone.yuri.org.uk) has joined #zumastor 2007-10-23 22:20 -!- murb(~murbix@soapstone.yuri.org.uk) has joined #zumastor 2007-10-23 22:20 -!- flipz(~phillips@phunq.net) has joined #zumastor 2007-10-23 22:20 -!- fmayhar(~fmayhar@207.47.98.129.static.nextweb.net) has joined #zumastor 2007-10-23 22:20 -!- flips(~phillips@207.47.98.129.static.nextweb.net) has joined #zumastor irc.oftc.net #zumastor log beginning Wed Oct 24 00:00:03 PDT 2007 2007-10-24 01:11 -!- erwan_taf(~erwan@81.80.43.67) has joined #zumastor 2007-10-24 09:56 -!- cbsmith(~xman@adsl-76-240-81-238.dsl.lsan03.sbcglobal.net) has joined #zumastor 2007-10-24 15:33 -!- fmayhar(~fmayhar@207.47.98.129.static.nextweb.net) has joined #zumastor 2007-10-24 18:53 -!- cbsmith(~xman@adsl-76-240-81-238.dsl.lsan03.sbcglobal.net) has joined #zumastor 2007-10-24 23:55 -!- flipz(~phillips@phunq.net) has joined #zumastor irc.oftc.net #zumastor log beginning Thu Oct 25 00:00:03 PDT 2007 2007-10-25 03:22 -!- erwan_taf(~erwan@81.80.43.67) has joined #zumastor 2007-10-25 10:57 -!- cbsmith(~xman@adsl-76-240-81-238.dsl.lsan03.sbcglobal.net) has joined #zumastor 2007-10-25 13:03 -!- erwan_taf(~erwan@konilope.linuxeries.org) has joined #zumastor 2007-10-25 14:46 shapor, ping? 2007-10-25 14:49 flipz pong 2007-10-25 14:55 hi shapor 2007-10-25 14:55 could you nudge jiayingz please? 2007-10-25 16:13 -!- cbsmith(~xman@adsl-76-240-81-238.dsl.lsan03.sbcglobal.net) has joined #zumastor 2007-10-25 17:14 shapor? 2007-10-25 19:20 -!- cbsmith(~xman@adsl-76-240-81-238.dsl.lsan03.sbcglobal.net) has joined #zumastor irc.oftc.net #zumastor log beginning Fri Oct 26 00:00:04 PDT 2007 2007-10-26 00:56 -!- erwan_taf(~erwan@81.80.43.67) has joined #zumastor 2007-10-26 06:02 -!- erwan_taf(~erwan@81.80.43.67) has joined #zumastor 2007-10-26 10:04 -!- erwan_taf(~erwan@konilope.linuxeries.org) has joined #zumastor 2007-10-26 13:24 -!- cbsmith(~xman@adsl-76-240-81-238.dsl.lsan03.sbcglobal.net) has joined #zumastor 2007-10-26 13:44 -!- jdries3(~jdries3@72.14.228.89) has joined #zumastor 2007-10-26 17:30 -!- fmayhar(~fmayhar@207.47.98.129.static.nextweb.net) has left #zumastor 2007-10-26 17:31 -!- fmayhar(~fmayhar@207.47.98.129.static.nextweb.net) has joined #zumastor 2007-10-26 17:36 -!- fmayhar(~fmayhar@207.47.98.129.static.nextweb.net) has joined #zumastor 2007-10-26 17:36 -!- fmayhar(~fmayhar@207.47.98.129.static.nextweb.net) has joined #zumastor 2007-10-26 17:37 -!- fmayhar(~fmayhar@207.47.98.129.static.nextweb.net) has joined #zumastor 2007-10-26 17:38 -!- fmayhar(~fmayhar@207.47.98.129.static.nextweb.net) has joined #zumastor 2007-10-26 18:21 fmayhar: make up your mind eh? 2007-10-26 18:21 ;-) irc.oftc.net #zumastor log beginning Sat Oct 27 00:00:04 PDT 2007 2007-10-27 02:19 ACTION feels much better 2007-10-27 02:20 I like seeing bugs die 2007-10-27 02:39 -!- erwan_taf(~erwan@konilope.linuxeries.org) has joined #zumastor 2007-10-27 17:58 shapor, there? 2007-10-27 19:42 flipz yes 2007-10-27 19:47 shapor, back 2007-10-27 20:33 flipz, back 2007-10-27 20:33 tag, you're it 2007-10-27 20:35 I've had an rsync via nfs test running for 17 hours, no deadlock 2007-10-27 20:38 by adding >/dev/null 2>/dev/null to the ddsnap create command 2007-10-27 20:48 performance is still poor, at best, throughput is around 900kB/sec, iostat shows about 5.6MB/sec on the snapshot store, for a total of about 6.5MB/sec for the underlying device which contains both the snapshot store and origin 2007-10-27 20:48 as suspected 2007-10-27 20:48 s/sus/ex/ 2007-10-27 20:49 About 50% of the cpu time is burned in IO wait, with very little (about 3%) spent in user and system 2007-10-27 20:50 so I guess with all that seeking the fancy battery-backed ram raid controller with 6 drives just doesn't perform that well 2007-10-27 20:58 http://www.ohloh.net/projects/9305/factoids/250928 2007-10-27 22:37 shapor, still there? irc.oftc.net #zumastor log beginning Sun Oct 28 00:00:03 PDT 2007 2007-10-28 02:15 flipz, i am now 2007-10-28 12:44 shapor? 2007-10-28 12:58 flipz: hi 2007-10-28 13:40 shapor, longest running failure to connect? 2007-10-28 13:40 ACTION tries email 2007-10-28 18:50 shapor, did you get a new phone number? I keep getting answering service 2007-10-28 19:42 flipz, no.. but i've been riding my motorcycle 2007-10-28 19:52 wonnerful 2007-10-28 19:53 shapor, did directing the log output to dev/null result in a successfull test run? 2007-10-28 19:58 yes 2007-10-28 19:59 well, it still copying 2007-10-28 19:59 243GB 2007-10-28 19:59 so far 2007-10-28 20:00 farthest it got before was 40GB irc.oftc.net #zumastor log beginning Mon Oct 29 00:00:04 PDT 2007 2007-10-29 01:31 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2007-10-29 02:23 -!- erwan_taf(~erwan@81.80.43.67) has joined #zumastor 2007-10-29 05:05 -!- MaZe(~MaZe@c-67-188-123-92.hsd1.ca.comcast.net) has joined #zumastor 2007-10-29 08:22 -!- jdries3(~jdries3@72.14.228.89) has joined #zumastor 2007-10-29 09:20 -!- jdries3(~jdries3@72.14.228.89) has joined #zumastor 2007-10-29 10:23 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2007-10-29 10:43 -!- jdries3(~jdries3@72.14.228.89) has joined #zumastor 2007-10-29 11:00 -!- jdries3(~jdries3@72.14.228.89) has joined #zumastor 2007-10-29 15:54 shpor? 2007-10-29 15:54 shapor even? 2007-10-29 15:58 flipz: hi 2007-10-29 15:58 you sent a url? 2007-10-29 15:58 pasting it here would be fine 2007-10-29 15:58 a url ? 2007-10-29 15:58 err, domain name 2007-10-29 15:58 for dns 2007-10-29 15:58 new dns servers 2007-10-29 15:58 yzf.shapor.com 2007-10-29 15:58 and cbr.shapor.com 2007-10-29 15:58 thanks 2007-10-29 15:59 i need to call tech support where that server is to see why its down 2007-10-29 15:59 i think i'm going to cancel it anyway 2007-10-29 15:59 been down since 8 last night 2007-10-29 15:59 :( 2007-10-29 15:59 i've moved my personal email and most important things off that old server 2007-10-29 16:00 zumastor is on the new one :) 2007-10-29 16:00 leave your "customer" twisting in the breeze ;-) 2007-10-29 16:00 my parents left me 3 voicemails last night 2007-10-29 16:01 the domain they host all their ebay pictures on was still there too 2007-10-29 16:01 heh 2007-10-29 16:01 whoops 2007-10-29 16:01 hopefully vger has not dropped my subscription yet 2007-10-29 16:02 one of the reasons i moved is started hearing bad things about my old isp 2007-10-29 16:02 even though my server had an 800 day uptime 2007-10-29 16:03 seems there was some truth 2007-10-29 16:15 gandi.net is such a slick registrar 2007-10-29 16:16 a nerd's delight 2007-10-29 16:19 flipz: so i disabled the snapshot create/delete 2007-10-29 16:20 and the read problem is still there 2007-10-29 16:20 still see the long bursts of reads? 2007-10-29 16:21 according to iostat, yes 2007-10-29 16:22 I await a description of the cause with interest ;-) 2007-10-29 16:22 however.... 2007-10-29 16:22 are you putting logical IO addresses in your ring buffer now? 2007-10-29 16:23 when i strace ddsnap server 2007-10-29 16:23 i see it going the 6 write 1 read pattern 2007-10-29 16:23 s/go/do/ 2007-10-29 16:23 as expected 2007-10-29 16:23 ltrace -Scp agrees 2007-10-29 16:23 it shows 40% of the time is spent iin SYS_pwrite64 2007-10-29 16:23 Scp? 2007-10-29 16:24 and 8% in pread64 2007-10-29 16:24 and the rest seeking? 2007-10-29 16:24 or? 2007-10-29 16:24 I guess seeking is included in pread/write 2007-10-29 16:24 well if its seeking, it would be blocking on the read or write 2007-10-29 16:24 yeah 2007-10-29 16:25 the iostat makes no sense 2007-10-29 16:26 iostat is somewhat dunglike 2007-10-29 16:26 but its jsut the data from diskstats 2007-10-29 16:26 I'll believe your ring buffer a lot more 2007-10-29 16:26 all the IOs are reads to the snapshot store 2007-10-29 16:26 very strange 2007-10-29 16:26 iostat: dunglike input data massaged in a dunglike way 2007-10-29 16:26 even though there should certainly be writes according to my strace 2007-10-29 16:27 argh 2007-10-29 16:27 wrong machine 2007-10-29 16:27 heh 2007-10-29 16:27 anyway, the kernel writes won't be pwrite 2007-10-29 16:28 kernel writes? 2007-10-29 16:28 you mean the ones actual io nfsd is doing? 2007-10-29 16:32 ah ok, now i can see all the pread64 calls :) 2007-10-29 16:32 tracing on the right machine ;) 2007-10-29 16:37 so it appears that we do indeed have a problem.. this machine have over 5 millions chunks in the snapstore 2007-10-29 16:37 its running a simple dd test :) 2007-10-29 16:39 and it does appear to be due to replication 2007-10-29 16:52 ok so we dont suck as much as i thought 2007-10-29 16:52 sorry about the false alarm 2007-10-29 16:53 i convinced myself that we were last night at about 3 am 2007-10-29 21:12 so it turns out the Dell 2850's random io performance is really, really poor 2007-10-29 21:17 not really all that surprising ;-) 2007-10-29 21:19 well, 2MB/sec is suprising tome 2007-10-29 21:22 I just don't like dell machines, expensive and not all that useful 2007-10-29 21:22 not all that reliable either 2007-10-29 21:23 and they eat a lot of power too 2007-10-29 21:48 -!- jdries3(~jdries3@c-69-249-52-124.hsd1.nj.comcast.net) has joined #zumastor 2007-10-29 21:51 performance is a lot better with 64k io's rather than 16k 2007-10-29 21:51 the raid is configured with a 64k stripe size so that makes sense 2007-10-29 21:55 such a low stripe size? I usually use 256k 2007-10-29 21:58 I think this machine just has the factory default 6 drive raid 5 setup 2007-10-29 21:58 raid5 really isn't ideal either irc.oftc.net #zumastor log beginning Tue Oct 30 00:00:01 PDT 2007 2007-10-30 07:45 -!- erwan_taf(~erwan@81.80.43.67) has joined #zumastor 2007-10-30 07:50 -!- erwan_taf(~erwan@81.80.43.67) has joined #zumastor 2007-10-30 08:04 -!- jdries3(~jdries3@72.14.228.89) has joined #zumastor 2007-10-30 10:07 -!- jdries3(~jdries3@72.14.228.89) has joined #zumastor 2007-10-30 10:55 -!- erwan_taf(~erwan@konilope.linuxeries.org) has joined #zumastor 2007-10-30 11:08 -!- jdries3(~jdries3@72.14.228.89) has joined #zumastor 2007-10-30 12:12 -!- jdries3(~jdries3@72.14.228.89) has joined #zumastor 2007-10-30 13:25 The following address(es) failed: 2007-10-30 13:25 shapor@shapor.com 2007-10-30 13:25 Unrouteable address 2007-10-30 13:25 for shapor@shapor.com; Mon, 29 Oct 2007 01:10:51 -0700 2007-10-30 13:36 NXDOMAIN 2007-10-30 14:32 Record expires on 29-Oct-2007. 2007-10-30 14:32 eek 2007-10-30 14:36 renewed now 2007-10-30 20:16 -!- jdries3(~jdries3@168.sub-75-223-117.myvzw.com) has joined #zumastor 2007-10-30 23:38 -!- MaZe(~MaZe@c-67-188-123-92.hsd1.ca.comcast.net) has joined #zumastor irc.oftc.net #zumastor log beginning Wed Oct 31 00:00:02 PDT 2007 2007-10-31 02:12 so I was just going to look in to adding fd position in to /proc 2007-10-31 02:12 turns out its been done, and in 2.6.22 :) 2007-10-31 02:13 suprised its such a recent addition 2007-10-31 02:13 -!- erwan_taf(~erwan@81.80.43.67) has joined #zumastor 2007-10-31 02:14 /proc//fdinfo/ 2007-10-31 03:31 :-) 2007-10-31 03:36 akpm's first repsonse to the patch was "why would you want that?" 2007-10-31 04:29 why not offer up a post to that? 2007-10-31 07:47 -!- shapor(~shapor@yzf.shaptech.com) has joined #zumastor 2007-10-31 09:00 -!- zumalog(~zumalog@yzf.shaptech.com) has joined #zumastor 2007-10-31 10:29 -!- erwan_taf(~erwan@konilope.linuxeries.org) has joined #zumastor 2007-10-31 10:31 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2007-10-31 10:47 -!- fmayhar(~fmayhar@207.47.98.129.static.nextweb.net) has joined #zumastor 2007-10-31 11:19 -!- erwan__taf(~erwan@konilope.linuxeries.org) has joined #zumastor 2007-10-31 11:24 -!- MaZe(~MaZe@216-239-45-4.google.com) has joined #zumastor 2007-10-31 12:51 -!- cbsmith(~xman@adsl-71-133-80-65.dsl.irvnca.pacbell.net) has joined #zumastor 2007-10-31 15:34 -!- juuva(juuva@peili.org) has joined #zumastor 2007-10-31 16:06 -!- cbsmith(~xman@adsl-76-240-81-238.dsl.lsan03.sbcglobal.net) has joined #zumastor 2007-10-31 17:51 -!- MaZe(~MaZe@216-239-45-4.google.com) has left #zumastor