use banana+compression for flogfiles on wire

[Brian Warner]

0.3.0 stores log events on disk using pickle, with optional compression. The compression is a big win, usually about 10x.

But when we send these events over the wire in an Incident Report, we decompress the file, pickle.load into memory, serialize with Banana over the wire, then the incident-gatherer at the other end must deserialize the Banana, reserialize with pickle, recompress, then write to a file.

The idea was to avoid the unsafe pickle deserialization at the receiving end. But we could use banana instead. We almost have enough Tub.serialize code in place to let us do streaming safe serialization of the log events, which we could then compress as normal. We can add a new incident-subscription protocol (rather a replacement for get_incident) which returns a big string of bzip2'ed banana'ed log events, instead of getting a stream of distinct log events.

I think this would reduce the CPU usage of catching up the incident-gatherer considerably. At the moment it takes a second or two per incident, and probably 10x as much network traffic as we really want.

[Brian Warner]

too big for this impending release

[Brian Warner]

oh, the banana-serialized files on disk should be named .flob instead of .flog