tests fail against latest twisted-17.9.0: Failure is no longer pickleable

[Brian Warner]

Unit tests are failing when run against the latest Twisted release (17.9.0). In this release, Failure now inherits from BaseException, which means it is no longer pickleable.

There are two sets of test failures. The first is in test_copyable where we explicitly try to round-trip a CopiedFailure through pickle:

[ERROR]
Traceback (most recent call last):
  File "/Users/warner/stuff/python/foolscap/src/foolscap/test/test_copyable.py", line 138, in _testFailure1_1
    p = pickle.dumps(f)
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1380, in dumps
    Pickler(file, protocol).dump(obj)
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 425, in save_reduce
    save(state)
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 669, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 759, in save_global
    (obj, module, name))
pickle.PicklingError: Can't pickle <class exceptions.RuntimeError at 0x10e6a8c80>: it's not the same object as exceptions.RuntimeError

foolscap.test.test_copyable.Copyable.testFailure1

and the second is in the logging tests, where Failures are pickled into the serialized (internal) log buffers:

[ERROR]
Traceback (most recent call last):
  File "/Users/warner/stuff/python/foolscap/src/foolscap/eventual.py", line 26, in _turn
    cb(*args, **kwargs)
  File "/Users/warner/stuff/python/foolscap/src/foolscap/logging/log.py", line 484, in msg
    from_="local", rx_time=time.time())
  File "/Users/warner/stuff/python/foolscap/src/foolscap/logging/flogfile.py", line 20, in serialize_wrapper
    pickle.dump(wrapper, f)
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1376, in dump
    Pickler(file, protocol).dump(obj)
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 669, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 669, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 425, in save_reduce
    save(state)
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 669, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 759, in save_global
    (obj, module, name))
pickle.PicklingError: Can't pickle <class foolscap.test.test_logging.SampleError at 0x10e8a7188>: it's not the same object as foolscap.test.test_logging.SampleError

foolscap.test.test_logging.Dumper.test_dump

I haven't seen evidence of these affecting actual use (the Tahoe unit tests are still passing), but it causes a FTBFS (Fails To Build From Source) error on both Debian and Ubuntu, and we need to fix this ASAP to about having Foolscap removed entirely from debian (and Tahoe too, along with all other downstream dependencies).

• ​https://bugs.launchpad.net/ubuntu/+source/foolscap/+bug/1733075
• ​https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=877206

I've been meaning to replace pickle with JSON in the logging code for years now (#247, see also #4, #110, #130). I've got a few old branches, but I haven't looked at them for a long time, so I don't think they're ready for a quick fix. I'll dust them off and see what state they're in, but I think I might just remove exceptions from pickled logs entirely for now, and try to switch to JSON-based data in the next release.

[Brian Warner]

fixed by [0708923]