make DeadReferenceError qualify as RemoteException

[Brian Warner]

​Tahoe#889 indicates a place where some logging code was surprised to see DeadReferenceError qualify as a "local" exception (and thus got logged with more severity than intended). When I wrote that logging code, I had expected DeadReferenceError to qualify as remote, via f.check(RemoteException).

I'd like to fix this, so that code which does f.check(RemoteException) will catch DeadReferenceError. On the other hand, it is useful for clients to spot DeadReferenceError easily, because they typically respond to it by dropping the connection or trying to establish a new one (as opposed to other remote errors that don't affect the connection).

I'd like to make this use of f.check work for both the DeadReferenceErrors that occur when you lose the connection while the request is outstanding, and for the ones that occur when the connection is already gone by the time you call callRemote. These are delivered via two separate code paths. I believe (but haven't confirmed) that the former code path wraps the DeadReferenceError in a RemoteException, but the latter does not.

So I'm considering making DeadReferenceError simply inherit from RemoteException. The downside of this would be that code which does f.check(RemoteException) could not also just do f.value.failure.check, because these DeadReferenceErrors would not have a .failure attribute. This would be a significant compatibility burden.

The alternative would be to have the callRemote-side path also wrap its DeadReferenceError in a RemoteException. This would means that application code would never see callRemote errback with DeadReferenceError, so code which wanted to handle lost connections specially would have to be updated. This affects backwards compatibility too.

[Brian Warner]

Looking at the code paths, it appears that DeadReferenceError is never wrapped in RemoteException. DRE is created in three places:

• raised during callRemote/Broker.newRequestId (sending on an already-broken rref)
• sent to PendingRequest.fail during Broker.abandonAllRequests (connection loss, mapped from ConnectionLost or ConnectionDone)
• sent to Broker.abandonAllRequests from Negotiation when a connection is replaced by a newer one

And RemoteException is only created on receipt of an (error) sequence, which is only sent by ErrorSlicer.

So the current behavior is at least consistent.

I guess I'll leave this alone, because the compatibility problems are hard. Existing code that uses the current API will have trouble with moving DeadReferenceError under RemoteException (code which log.WEIRDs anything but RemoteException will create noisy incidents for mere lost connections, and code which reacts to DeadReferenceError would stop working).

Instead, application code should anticipate the following categories of exceptions (when setOption("expose-remote-exception-types", False) is in effect):

• RemoteException, on which it is always safe to examine f.value.failure. These can only result from exceptions that occur on the far end of the wire, from either an inbound schema Violation or an exception raised by the remote_* method.
• DeadReferenceError, caused when the connection is lost before the response is received (possibly before the request was even sent). A serious banana-level error will drop the connection, causing DeadReferenceError for all outstanding requests.
• Violation, caused when a local inbound schema violation is detected, because the remote end sent us something unexpected
• anything else is a local exception caused by something higher in the Deferred chain, or by an internal Foolscap error. In particular, there is no way for the remote side to produce normal Python exceptions like IndexError or some other application-specific exception type.

So application code that should probably have a handler that looks like this:

def handle(f):
    if f.check(RemoteException):
        if f.value.failure.check(IndexError): return "remote indexerror"
        if f.value.failure.check(AppSpecificError): return "known"
        return f
    if f.check(DeadReferenceError):
        self.stop_using_this_connection()
        return None
    return f # something local