logging tools to analyze hard failures like segfaults

[Brian Warner]

We were analyzing a Tahoe failure that resulted from an overloaded logger (#72). One intermediate hypothesis was that python segfaulted in response to some inbound callRemote, and the log.msg that would have reported the inbound method call was lost while it was still inside the sending process's transmit buffer. So we were discussing ways to get log messages from a process to a gatherer that didn't depend upon python or foolscap working.

One approach that I looked at for a different project many years ago was to use shared memory. The sending process allocates a buffer (perhaps 1MB) that is shared with a receiver, and writes some structures into place that define a circular buffer (or a slab to which we write in chunks from one end to the other, and use memmove when we reach the end instead of wrapping around). The messages should have checksums, and the structure should have a bunch of pointers, so that the receiver can look at the pointers at any moment (in between python opcodes) and read any message that was fully written by that point.

Shared memory is not the easiest thing to manipulate from python, but it's possible. The log messages would be serialized (perhaps safely with Tub.serialize) before being added to the buffer.