13:01 <@jrandom> 0) hi
13:01 <@jrandom> 1) 0.5.0.3
13:01 <@jrandom> 2) batching
13:01 <@jrandom> 3) updating
13:01 <@jrandom> 4) ???
13:01 <@jrandom> 0) hi
13:01  * jrandom waves
13:01 <@jrandom> the just-now-posted weekly status notes are up @ http://dev.i2p.net/pipermail/i2p/2005-March/000654.html
13:02 <+detonate> hi
13:02 <+cervantes> 'lo
13:02 <@jrandom> jumpin' right in to 1) 0.5.0.3
13:02 <@jrandom> the release came out a few days ago, and reports have been positive
13:02 <+cervantes> jrandom: let us know when the blue dancing dwarves climb onto your monitor and we'll stop the meeting early
13:03 <@jrandom> heh cervantes 
13:03 <@jrandom> (thank Bob for editable meeting logs ;)
13:04 <@jrandom> i dont really have much to add wrt 0.5.0.3 than whats in that message
13:04 <@jrandom> anyone have any comments/questions/concerns on it?
13:04 < bla> jrandom: Any new measurements on the fast-peers-selection code?
13:05 <@jrandom> ah, i knew there was something else in 0.5.0.3 that i had neglected ;)
13:06 <@jrandom> i dont have any hard metrics yet, but anecdotally the fast peer selection has found the peers that i know explicitly to be 'fast' (e.g. routers on the same box, etc)
13:07 < bla> jrandom: Sometimes, eepsites still require a number of retries to find a good tunnel to use
13:07 <@jrandom> reports have come in for fairly reasonable throughput rates on occation as well (in the 10-20KBps range), but thats still not frequent (we still have the parameters tuned down)
13:08 <+ant> <Connelly> oops there's a meeting
13:09 <@jrandom> hmm, yes, i've found that reliability still isn't where it need to be.  retrying more than once really isn't a solution though - if a site doesnt load after 1 retry, give it 5-10m before retrying
13:09 <@jrandom> what i've seen on the net though is some not-infrequent-enough transport layer delay spikes
13:10 <@jrandom> e.g. taking 5-20+ seconds just to flush a 1-2KB message through a socket
13:10 <@jrandom> tie that up with a 5 hop path (2 hop tunnels) and you can run into trouble
13:11 <@jrandom> thats actually part of whats driving the batching code - reducing the # of messages to be sent
13:13 <@jrandom> ok, anyone else have any questions/comments/concerns on 0.5.0.3?
13:13 < bla> jrandom: Looks good. I will ask more about it in the next "section"
13:14 <@jrandom> w3rd, ok, perhaps we can move on there then - 2) batching
13:15 <@jrandom> the email and my blog (jrandom.dev.i2p</spam>) should describe the basics of whats planned
13:15 <@jrandom> and, well, really its some pretty basic stuff
13:15 <@jrandom> the current preprocessor we have was the simplest possible one to implement (class name: TrivialPreprocessor) ;)
13:16 <@jrandom> this new one has some tunable parameters for batching frequency, as well as some outbound tunnel affinity within individual tunnel pools (where we try to select the same outbound tunnel for subsequent requests for up to e.g. 500ms, so that we can optimize the batching)
13:17 <@jrandom> that's about all i have to add on that though - any questions/comments/concerns?  
13:18 < bla> Does this require all participating nodes to run the new preprocessor, or can a mix of Trivial/NewOne coexist?
13:18 <+Ragnarok> this will add .5 s to latency, right?
13:19 <@jrandom> bla: nah, this preprocessor is only used on the tunnel gateway, and its up to that gateway to decide how or whether to batch
13:20 <@jrandom> Ragnarok: not usually - message 1 may be delayed for up to .5s, but messages 2-15 get transferred much faster than they would have otherwise.  its also a simple threshold - as soon as a full tunnel message worth of data is available, its flushed
13:20 <+Ragnarok> cool
13:20 <+Ragnarok> how much overhead does it save?
13:21 <@jrandom> substantial ;)
13:21 <+Ragnarok> substantial is good, if vague :P
13:21 <@jrandom> look on your http://localhost:7657/oldstats.jsp#tunnel.smallFragments
13:21 <@jrandom> compare that to #tunnel.fullFragments
13:22 < bla> jrandom: Does this concern endpoint->IB-gateway communication only? 
13:22 <@jrandom> with batching, the ratio of full to small will go up, and the # of pad bytes in the small will go down
13:23 <@jrandom> bla: hmm, it concerns all tunnel gateways, whether inbound or outbound
13:24 < mihi> full fragments: lifetime average value: 1,00 over 1.621,00 events
13:24 < bla> jrandom: ok
13:24 < mihi> can there be a frational number of fragments?
13:24 <@jrandom> full: 1.00 over 807,077.00 events small: 746.80 over 692,682.00 events
13:25 <@jrandom> heh mihi
13:25 <@jrandom> (that small: 746 means that on those 692k messages, 746 out of 996 bytes were wasted pad bytes!)
13:26 <@jrandom> well, not quite wasted - they served their purpose
13:26 <+detonate> needless overhead anyway
13:27 <@jrandom> yep, much of which we should be able to shed with the batching preprocessor
13:28 <@jrandom> unfortunately, that won't be bundled in the next release
13:28 <@jrandom> but it will be bundled in the 0.5.0.6 rev (or perhaps 0.5.1)
13:28 <@jrandom> erm, 0.5.0.5, or 0.5.1
13:28  * jrandom gets confused with #s
13:29 < bla> jrandom: How come not?
13:29 <+cervantes> hash and pills...damn
13:30 <@jrandom> !thwap cervantes 
13:30 <@jrandom> bla: there's a bug in 0.5.0.3 (and before) where the fragmented message handler will cause subsequent fragments within the same tunnel message to be discarded
13:31 <@jrandom> if we deployed the batching preprocessor directly, we'd have a substantial number of lost messages
13:31 <@jrandom> its not a worry, we've got other neat stuff up our sleeves though, so this coming 0.5.0.4 won't be totally boring ;)
13:31 < bla> jrandom: Ah, so that
13:32 < bla> jrandom: Ah, so that is why we have to do that after 0.5.0.4 is mainstream.. I see. Thnx.
13:33 <@jrandom> yeah, it'd be nice if the fragment handler was able to deal with it, and it does, generally, it just releases the byte buffer too soon, zeroing out subsequent fragments (oops)
13:33 <@jrandom> ok, anything else on 2), or shall we move on to 3) updating?
13:35 <@jrandom> ok, as mentioned in the status notes (and discussed for a while in various venues), we're going to add some functionality for simple and safe updating that doesn't require the end user to go to the website, read the mailing list, or read the topic in the channel :)
13:36 <+detonate> cool
13:36 <@jrandom> smeghead has put together some code to help automate and secure the process, working with cervantes to tie in with fire2pe as well as the normal routerconsole
13:37 <@jrandom> the email lists the general description of whats going to be available, and while most of it is functional, there are still a few pieces not yet fully hashed out
13:37 <@jrandom> unlike the batching, this /will/ be available in the next rev, though people won't be able to make much use of it until 0.5.0.5 (when it comes time to update)
13:39 <+Ragnarok> so... what's the cool stuff for 5.0.4, then?
13:42 <@jrandom> with the update code comes the ability to pull announcement data, displaying e.g. a snippet of news on the top of the router console.  in addition to that, as part of the update code we've got a new reliable download component which works either directly or through the eepproxy, retrying and continuing along the way.  perhaps there'll be some relatd features built off that, but no promises
13:42 <+Ragnarok> neat
13:43 <@jrandom> ok, anyone else have any questions/comments/concerns on 3) updating?
13:45 <@jrandom> if not, moving on to 4) ???
13:45 <@jrandom> anything else anyone wants to bring up?  i'm sure i've missed soem things
13:45 <+detonate> i2p's known to work with the sun jvm in OpenBSD 3.7 :)
13:45 <@jrandom> nice!
13:47 < bla> What is the status on the UDP transport?
13:48 <+detonate> udp is going to be messy, i think it would be better to steal the pipelining code from bt and adapt it ;)
13:48 <@jrandom> *cough*
13:49 <@jrandom> i dont expect there to be much trouble, but there's certainly work to be done
13:49 <@jrandom> how the queueing policy operates, as well as the bw throttling for admission to the queue will be interesting
13:50 < bla> Who was doing that prelim work?
13:50 <@jrandom> bla: detonate and mule
13:50 <+detonate> yeah.. i've been slacking off the last month or so though
13:50 < bla> detonate: I assume you jest, with your BT remark?
13:51 <+detonate> i'm half-serious
13:51 <+detonate> you could at least cut the thread count for the transport in half if you did that
13:51  * jrandom flings a bucket of mud at detonate 
13:51 < jdot> woohoo.  my router is now running on my dedicated server rather than my POS cable connection.
13:51 <@jrandom> nice1 jdot 
13:52 <@jrandom> we'll be able to get by with 3-5 threads in the transport layer for all comm with all peers
13:52 < bla> detonate: But half is not going to cut it, when the net becomes large (> couple hundred nodes)
13:52 < jdot> with 1000GB of b/w at its disposal
13:53 < jdot> unforunately j.i2p and the chat.i2p will be down for a few more hours while i migrate things
13:53 < duck> detonate: addressbook on halt too?
13:53 <+detonate> yeah, it's on halt too
13:54 <+detonate> the only thing that isn't on halt is the monolithic profile storage, i was meaning to get that working later today
13:54 <@jrandom> w3rd
13:54 <+detonate> then i2p won't fragment the drive massively
13:54 < jdot> jrandom: as far as BW limits go, are they averages?
13:54 <+frosk> modern filesystems don't fragment, silly
13:55 <+detonate> ntfs does
13:55 <@jrandom> jdot: the bandwidth limits are strict token buckets
13:55 <@jrandom> jdot: if you set the burst duration out, thats how long of a period it averages out through
13:56 <@jrandom> (well, 2x burst == period)
13:56 <@jrandom> ((ish))
13:56 < jdot> hmmm...  well i have 1000GB and want i2p to be able to take up to 800GB/mo....
13:56 <+ant> <mihi> detonate: ntfs stores really small files in mft which means nealy no fragmentation
13:57 < jdot> and i dont care what it bursts to
13:57 <+detonate> well, when you run the defragmenter, it spends 10 minutes moving all 6000 profiles around.. so they must fragment
13:58 <@jrandom> jdot: hmm, well, 800GB is probably more than it'll want to push anyway, so you can probably go without limits ;)  
13:58 <@jrandom> otoh, if you could describe a policy that you'd like implemented, we might be able to handle it
13:58 < jdot> jrandom: i'll do that for now and see how it works
13:58 < bla> detonate: NTFS, IIRC, is a journalling FS. So even a monolotic file will get fragmented if you write small portions to it 
13:58 <+detonate> everything gets written at once
13:59 <+detonate> and read at once
13:59 < bla> detonate: Ok. I see.
13:59 < jdot> jrandom: well, lets wait until we figure out if it'll even be a problem.
13:59 < bla> detonate: Good work in that case!
13:59 <+detonate> i'd be interested in knowing how much usage there really is if you leave it uncapped
14:00 <+detonate> on a good connection
14:00 < jdot> i'm interested too!
14:00 <@jrandom> my colo routers run uncapped, though cpu constrained
14:00 <+Ragnarok> can you use a bitbucket to average over a month?
14:00 < jdot> jrandom: cpu contrianed?  what kind of cpu?
14:01 <@jrandom> 4d transfer 3.04GB/2.73GB
14:01 <+detonate> hmm, was expecting less
14:01 <@jrandom> jdot: cpu constrained because i run 3 routers on it, plus a few other jvms, sometimes profiling
14:01 <+detonate> must be those bt people
14:01 <+detonate> once the batching stuff is online, it would be interesting to see how that changes
14:02 <@jrandom> detonate: some of that transfer is also me pushing 50MB files between itself ;)
14:02 <+detonate> heh
14:02 < jdot> ahh.  ok.  well, we'll see how this system does.  its an AMD XP 2400 w/ 512MB and a 10Mbit connection
14:02 <@jrandom> Ragnarok: token buckets dont really work that way
14:02 <@jrandom> jdot: word, yeah, this is a p4 1.6 iirc
14:03 <@jrandom> Ragnarok: in a token bucket, every (e.g.) second you add in some tokens, according to the rate.  if the bucket is full (size = burst period), the tokens are discarded
14:04 <@jrandom> whenever you want to transfer data, you need to get a sufficient amount of tokens
14:04 <@jrandom> (1 token = 1 byte)
14:04 <+Ragnarok> I know how they work... what happens if you just make the bucket really big?
14:05 <+detonate> then you never stop sending data
14:05 <+detonate> if it's infinite in size
14:05 <+detonate> err, and it's filled with tokens
14:05 <@jrandom> if its really big, it may go out and burst to unlimited rates after low usage
14:06 <@jrandom> though perhaps thats desired in some cases
14:07 <@jrandom> the thing is, you can't just set the token bucket to 800GB, as that wouldnt constrain the total amount transferred
14:08 <+detonate> you need a field there where you can set the number of tokens per second, then you can just divide the bandwidth usage per month by the number of seconds
14:08 <+detonate> :)
14:10 <@jrandom> thats the same as just setting the rate averaged over the month, which would be unbalanced.  but, anyway, lots of scenarios available - if anyone has any that address their needs that can't be met with whats available, please, get in touch
14:10 <+Ragnarok> but if you set the rate to the average you want... I think 308 kB/s here, and then set the bitbucket to something very larger... why doesn't that work?
14:11 <+Ragnarok> s/larger/large/
14:12 <+detonate> well, you could set it so that it never sends more than 800GB/44000 in a 60 second burst period
14:12 <+detonate> 44000 being roughly the number of minutes in a month
14:13 <@jrandom> the bucket size / burst duration describes how much we'll send without constraint, and most people /do/ want constraints, so the router doesnt gobble 10mbps for 5 minutes while draining the bucket (or whatever)
14:14 <@jrandom> an additional throttle of tokens coming out of the bucket is also possible (and should that throttle have its own token bucket, and that bucket have its own throttle, etc)
14:16 <+Ragnarok> I thought the bucket only got paid into when there was bandwidth not being used
14:16 <@jrandom> tokens are added to the bucket at a constant rate (e.g. 64k tokens per second)
14:17 <@jrandom> anything that wants bandwidth always asks the bucket
14:18 <+Ragnarok> ah.. ok
14:19 <@jrandom> ok cool, anyone else have anything they want to bring up for the meeting?
14:21 <@jrandom> ok if not
14:21  * jrandom winds up
14:21  * jrandom *baf*s the meeting closed