id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc
87	use banana+compression for flogfiles on wire	Brian Warner		"0.3.0 stores log events on disk using pickle, with optional compression. The compression is a big win, usually about 10x.

But when we send these events over the wire in an Incident Report, we decompress the file, pickle.load into memory, serialize with Banana over the wire, then the incident-gatherer at the other end must deserialize the Banana, reserialize with pickle, recompress, then write to a file.

The idea was to avoid the unsafe pickle deserialization at the receiving end. But we could use banana instead. We almost have enough Tub.serialize code in place to let us do streaming safe serialization of the log events, which we could then compress as normal. We can add a new incident-subscription protocol (rather a replacement for get_incident) which returns a big string of bzip2'ed banana'ed log events, instead of getting a stream of distinct log events.

I think this would reduce the CPU usage of catching up the incident-gatherer considerably. At the moment it takes a second or two per incident, and probably 10x as much network traffic as we really want.
"	enhancement	new	major	undecided	logging	0.3.0