make it easy to distinguish server-side failures/exceptions from client-side

[Zooko]

In Tahoe-LAFS I'm writing code which wraps a call to a remote server in several layers of processing the response from the server. I would like to reliably distinguish between exceptions/failures caused processing on the client side (e.g. bugs in the client) and exceptions/failures incurred during processing on the server side.

Ideally there would be a single type that I could trap to catch all and only exceptions/failures occurring in the server side application logic, and another type that I could trap to catch all and only exceptions/failures from the foolscap layer, i.e. disconnection, foolscap version incompatibility, bug in the foolscap code, etc..

I *think* that CopiedFailure serves for the former purpose and DeadReferenceError serves for the latter, but the foolscap docs don't state that foolscap offers this as part of its contract, and I'm unsure if those will catch all errors of the type I want to catch or only most such errors. Also foolscap issue #4 looks a bit scary.

So I think currently I'm going to wrap each call to .callRemote() by immediately attaching an errback which wraps all resulting failures in a wrapper class named ServerFailure.

[Zooko]

This might be related to #4 (improve CopiedFailure? processing).

[Brian Warner]

propsed API for a failure_is_remote(f) predicate

[Brian Warner]

I started to try and address this need with the attached failure_is_remote predicate, but in talking further with Zooko, I don't think this would be satisfying.

We spoke of two basic classes of users. One is in the "E" camp, in which remote exceptions should look exactly like local exceptions (because E mistrusts local subroutines just as much as remote ones). The other (of which he'd like Tahoe to be a member) want to treat remote exceptions very differently: in a sense they're more trusting of locally-generated exceptions, in that they're willing to have their control flow influenced by the type of locally-generated exceptions, but not by the type of remotely-generated exceptions.

There are cases in the current Tahoe codebase where we do influence control flow by the type of a remotely-generated exceptions, for good reasons (generally debugability). If we move Tahoe into this second class, we will need to update them.

I want to keep Foolscap usable by both, and I want to get rid of the wrapper that Tahoe uses (because it breaks EQ, and because I think it's confusing). So my current plan is to have the application declare which class it wants to belong in, by providing a Tub option control the exception-reporting behavior of all RemoteReferences:

• create foolscap.RemoteException, an exception class. This will wrap a Failure instance (most likely a CopiedFailure), which will be accessible as its .failure attribute.
• add Tub.setOption("expose-remote-exception-types", bool), with True being the default value (which retains the current behavior). If this is set to False, the RemoteReferences that the Tub creates will have a flag that causes the following behavior changes:
  • all remote exceptions are reported in Failures of type RemoteException
  • (possibly including DeadReferenceError, I'm not sure)
  • f.value.failure can be used to extract the original CopiedFailure, on which you could call f.trap and f.check (with the caveat that f.trap would re-raise the remote failure instead of the local RemoteException-based one)

Code that sets expose-remote-exception-types=False, but which still wants to pay attention to the remote exception type, can use an errback like the following:

d = rref.callRemote("renew-lease", storage_index)

def _ignore_key_error(f):
    f.trap(foolscap.RemoteException)
    if f.value.failure.check(KeyError):
        return None
    return f
d.addErrback(_ignore_key_error)

def _log_remote_errors(f):
    f.trap(foolscap.RemoteException)
    log.msg("not my fault", failure=f.value.failure)
    return f
d.addErrback(_log_remote_errors)

def _retry_only_my_errors(f):
    if f.check(foolscap.RemoteException):
        return None
    return retry_something()
d.addErrback(_retry_only_my_errors)

(note that using f.value.failure.trap(KeyError) instead of check would result in the remote exception being re-raised, causing it to look like a local exception, violating the "class 2" expectations described above)

Comments welcome!

[Brian Warner]

2nd version of proposed API

[Brian Warner]

I've attached a new proposal. Zooko's approach is growing on me, so I'm considering making it the default in a future version of Foolscap (I think it would break too much existing error-handling code to change the default without some warning period first).

[Brian Warner]

[362cd28f9e5dd7a5e4fbe129132d7265697db78d] implements the new option. I'm going to close this ticket, and open a new one (destined for a future release, maybe 0.5.0) to consider setting the default to always use RemoteExceptions .

[Brian Warner]

#123 talks about making this RemoteException mode the default.