notifyOnDisconnect doesn't always fire

[Brian Warner]

The analysis that zooko did in ​tahoe#653, plus work we did together, provides strong evidence that foolscap 0.4.2-ish notifyOnDisconnect is not always firing when it's supposed to. I suspect that replacement connections (where an asymettric connection close occurs, and one side must drop its dead-but-it-doesnt-know-it-yet connection in preference to a fresh new inbound connection) are a factor, and that the notifications are being dropped. Maybe intentional disconnect follows a different code path than non-intentional, and the notifications are being skipped on that one.

I also noticed that the notifyOnDisconnect code is buggy, it should use a newly-minted empty Marker class instance as the cancel-handle, instead of a (callback,args,kwargs) tuple. Using the tuple could cause problems if the same code subscribes the same callback multiple times. I do not believe that this issue could have caused the tahoe problems, but it's worth fixing.

It might just be a good idea to put a bright orange warning sticker on notifyOnDisconnect, discouraging people from relying upon it being called the right way under significant connect/disconnect/reconnect cycles, since I don't really use it enough myself to prioritize development of the challenging test framework that would be needed to nail this one down.