better control over unexpected-exception logging

[Brian Warner]

Foolscap currently has an option to let you log (via twisted.python.log) all exceptions that occur during remote method calls. There is a separate option which logs all exceptions that occur during local method calls (ones that were run on behalf of some distant caller). This is handy when tracking down problems, since the process in which the exception occurred is likely to have more information (either in the logfile or in the head of the human watching it) than any other.

However, using this feature has the annoying property that it discourages developers from using exceptions as part of the normal remote API of their objects. For example, robk has a protocol that uses python's normal KeyError to signal that a given resource is unavailable, a situation which is actually fairly common. The huge numbers of KeyError exceptions then spam the logfiles with their tracebacks, masking other problems and consuming resources like crazy.

It would be nice if there were a way to tell Foolscap that, for this particular method, KeyError is a normal part of the API, and that it doesn't need to be logged.

oldpb did this by creating a special exception class, pb.Error, with the property that any exceptions inheriting from it would not be logged. Any exceptions that occur during message processing that do not inherit from pb.Error will be logged locally.

We're thinking of better ways to express this sort of distinction. The problem with pb.Error is that it discourages you from using standard python exception types as part of your API, since something like KeyError has no way of inheriting from pb.Error . This is slightly annoying but not majorly so.. you have to do things like:

  class MyKeyError(pb.Error):
    pass

  def remote_fetch(self, key):
    if key in self.data:
      return self.data[key]
    else:
        raise MyKeyError("key '%s' not in my data" % (key,))

instead of:

  def remote_fetch(self, key):
    return self.data[key]

Rob's proposal was to use the RemoteInterface? to indicate a list of exceptions that are a "normal" part of the API. Foolscap would know that it shouldn't log any instances of these exceptions. (note that foolscap won't log anything about exceptions unless you enable it, but it's so useful that usually you do enable it).

In trying to figure out how to implement this proposal, two ways come to mind. The first is to add a special magic argument to the prototype method definition inside the RemoteInterface?:

class RIFoo(RemoteInterface):
  def fetch(key=str, _raises=(KeyError, IndexError)):
    return int

(and of course _raises could be either a single exception class or a sequence of them).

The second is to use the established epydoc conventions and have the RemoteInterface? code parse the docstring for a '@raises' field:

class RIFoo(RemoteInterface):
  def fetch(key=str):
    """Fetch a key.
    @raises: KeyError if the key does not exist
    """
    return int

the docstring approach is trickier, for several reasons:

• docstrings are traditionally intended for humans, so they aren't as rigorously defined as code
• scoping: builtins like KeyError are pretty straightforward to turn into exception classes (or compare against existing exception classes), but what about more local classes? Either the docstring must use the fully-qualified exception classname (what a drag!) or the RemoteInterface? parser needs to be aware of the module names that are currently in scope and use them to locate a fully-qualified classname for everything in there.

If we go with the docstring approach, it also hint that we should use @param fields for the type information rather than our current default-values trick.

The way that remote method schemas are defined is a large area to explore.. zope.schema has some stuff to examine, as are some of the method-signature-annotation changes proposed for python itself. The goal has to be to make the RemoteInterface? definition easy for humans to read and understand, and feasible for foolscap to parse and enforce.

[Brian Warner]

ooh, with the new logging infrastructure, I can imagine adding a callRemote() argument that tells foolscap how you want it to log the failure (if it happens). This might be slightly easier than capturing the errback and attaching a log handler to it, although really that's what you ought to be doing.