Passing third-party references causes assertion error

[djfroofy]

In my simple example:

class Lobby(foolscap.Referenceable)
    lobby = {}
    ...
    def remote_befriend(self, inviter, invitee, token):
        lobby = self.lobby
        assert inviter in lobby and invitee in lobby
        inviter_ref, tk = self.lobby[inviter]
        assert token == tk
        invitee_ref, _tk = self.lobby[invitee]
        invitee_ref.callRemote('befriend', inviter, inviter_ref)

I get the following exception when I try to pass the inviter RemoteReference? on the the invitee:

2007-10-24 11:01:44-0400 [-] exception in produce
2007-10-24 11:01:44-0400 [-] exception in produce
2007-10-24 11:01:44-0400 [-] SendBanana.sendFailed: [Failure instance: Traceback: <type 'exceptions.AssertionError'>: 
2007-10-24 11:01:44-0400 [-] /opt/pyinstances/twisted/lib/python2.5/site-packages/foolscap/slicers/root.py:107:send
2007-10-24 11:01:44-0400 [-] /opt/pyinstances/twisted/lib/python2.5/site-packages/twisted/internet/defer.py:239:callback
2007-10-24 11:01:44-0400 [-] /opt/pyinstances/twisted/lib/python2.5/site-packages/twisted/internet/defer.py:304:_startRunCallbacks
2007-10-24 11:01:44-0400 [-] /opt/pyinstances/twisted/lib/python2.5/site-packages/twisted/internet/defer.py:317:_runCallbacks
2007-10-24 11:01:44-0400 [-] --- <exception caught here> ---
2007-10-24 11:01:44-0400 [-] /opt/pyinstances/twisted/lib/python2.5/site-packages/foolscap/banana.py:197:produce
2007-10-24 11:01:44-0400 [-] /opt/pyinstances/twisted/lib/python2.5/site-packages/foolscap/referenceable.py:614:slice
2007-10-24 11:01:44-0400 [-] ]
2007-10-24 11:01:44-0400 [-] Sendfailed.sendfailed
2007-10-24 11:01:44-0400 [-] Unhandled Error
        Traceback (most recent call last):
          File "/opt/pyinstances/twisted/lib/python2.5/site-packages/foolscap/slicers/root.py", line 107, in send
            d.callback(None)
          File "/opt/pyinstances/twisted/lib/python2.5/site-packages/twisted/internet/defer.py", line 239, in callback
            self._startRunCallbacks(result)
          File "/opt/pyinstances/twisted/lib/python2.5/site-packages/twisted/internet/defer.py", line 304, in _startRunCallbacks
            self._runCallbacks()
          File "/opt/pyinstances/twisted/lib/python2.5/site-packages/twisted/internet/defer.py", line 317, in _runCallbacks
            self.result = callback(self.result, *args, **kw)
        --- <exception caught here> ---
          File "/opt/pyinstances/twisted/lib/python2.5/site-packages/foolscap/banana.py", line 197, in produce
            obj = next()
          File "/opt/pyinstances/twisted/lib/python2.5/site-packages/foolscap/referenceable.py", line 614, in slice
            assert isinstance(tracker.url, str)
        exceptions.AssertionError: 

I am only registering a reference to the Lobby in the tub, although lobby of course holds references to the Chatter Referenceable objects as each joins the lobby. According to the documentation under third-party references, I should not have to explicitly register the remote references I pass. Also, to further complicate matters, it appears the reference to invitee is broken after this call, since I get this message when the lobby tries to relay normal chat lines:

2007-10-24 11:01:47-0400 [-] Unhandled error in Deferred:
2007-10-24 11:01:47-0400 [-] Unhandled Error
        Traceback (most recent call last):
          File "/opt/pyinstances/twisted/lib/python2.5/site-packages/foolscap/broker.py", line 545, in _doCall
            return obj.doRemoteCall(delivery.methodname, args, kwargs)
          File "/opt/pyinstances/twisted/lib/python2.5/site-packages/foolscap/referenceable.py", line 59, in doRemoteCall
            res = meth(*args, **kwargs)
          File "chat.py", line 72, in remote_chat
            chatter.callRemote('chat', time.asctime(), name, message)
          File "/opt/pyinstances/twisted/lib/python2.5/site-packages/foolscap/referenceable.py", line 379, in callRemote
            return defer.maybeDeferred(self._callRemote, _name, *args, **kwargs)
        --- <exception caught here> ---
          File "/opt/pyinstances/twisted/lib/python2.5/site-packages/twisted/internet/defer.py", line 107, in maybeDeferred
            result = f(*args, **kw)
          File "/opt/pyinstances/twisted/lib/python2.5/site-packages/foolscap/referenceable.py", line 418, in _callRemote
            reqID = broker.newRequestID()
          File "/opt/pyinstances/twisted/lib/python2.5/site-packages/foolscap/broker.py", line 444, in newRequestID
            raise DeadReferenceError("Calling Stale Broker")
        foolscap.ipb.DeadReferenceError: Calling Stale Broker

Sorry, no version. This is off mercurial trunk as of ... yesterday. Latest changeset: 256:69975a3ca4da.

[djfroofy]

The full example:

$ svn co svn://enterthefoo.com/scraps/foolscap-chat foolscap-chat

To reproduce, run the server and then connect two or more clients. Example:

$ python client.py <furl> jill
$ python client.py <furl> jack

Now, in jack's terminal, type:

@befriend jill

[Brian Warner]

My initial hunch is that the host of the Referenceable that you're trying to pass around did not do a tub.setLocation(), so the Referenceable doesn't know where it lives, and consequently (given our current approach to gifts) it cannot be gifted.

(think of it this way: when jack receives a reference to jill, what IP address / port does he connect to?)

I don't see a Tub.setLocation in client.py .. does this seem like the right answer?

I want to change the way we do gifts to behave more like E's (well, CapTP/VatTP's, to be specific) approach, which would clean up a few things and provide better ordering guarantees on gifts, but it wouldn't remove the need to have the giftable Referenceable live in a Tub which has a location.

In our "Tahoe" project (​http://allmydata.org), we have code that scans all the network interfaces and builds up a list of IP addresses at startup, then does Tub.setLocation() with those. This doesn't provide hostnames, and of course doesn't work behind a NAT box (although there's code to let you add extra connection hints, in case you've manually set up a port forwarding), but it does allow gifts to work betwee machines that are either locally visible to each other or actually connected to the internet.

To make NAT/etc work is to get into the realm of UPnP, hole-punching, and relay, and frankly we're putting off figuring that stuff out for as long as we can.

hope that helps,

-Brian

[djfroofy]

My initial hunch is that the host of the Referenceable that you're trying to pass around did not do a tub.setLocation(), so the Referenceable doesn't know where it lives, and consequently (given our current approach to gifts) it cannot be gifted.

Hunch verified.

(think of it this way: when jack receives a reference to jill, what IP address / port does he connect to?)

I don't see a Tub.setLocation in client.py .. does this seem like the right answer?

Yes, my crude proof-of-concept application (​svn://enterthefoo.com/scraps/foolscap-chat) now works after some simple changes, but I still have the problem with NAT traversal which I certainly don't want to deal with right now. Since I want to use this in the context of a game where the clients are behind NAT 99% of the time, I'll probably just create server proxies as relay points.

Thanks for the feedback.

-Drew