remove UnauthenticatedTub

[Brian Warner]

As recommended in #66, there may be good reasons to have Foolscap simply require pyOpenSSL unconditionally, and remove the option of using UnauthenticatedTub. It needs some more thought, though.

[bgranger]

Replying to warner:

As recommended in #66, there may be good reasons to have Foolscap simply require pyOpenSSL unconditionally, and remove the option of using UnauthenticatedTub. It needs some more thought, though.

Please don't do this. The IPython project is now using foolscap in its parallel computing infrastructure. In some cases:

1) Connecting processes on loopback only

2) In heavily secured contexts where the system is secured through other means

We really need to be able to use foolscap in its insecure mode. Also, we are not confident that openssl is available on some of the more exotic systems (supercomputers) we will be running on. Finally, in these contexts, we need to be able to use foolscap without SSL for performance reasons.

Summary: we are loving foolscap, but definitely prefer to have pyOpenSSL optional. I should mention though that in IPython the default mode is secure!

[Zooko]

This is an opportunity to learn about the costs of securing foolscap.

Why is it important to bgranger to avoid the requirement on pyOpenSSL?

Possible reasons:

1. Key management makes things harder. The foolscap design should make this not be the case. If it is still the case I would love to know why. (This is a unique feature of foolscap over alternative secure RPC mechanisms -- that the object-capability paradigm makes the key management problem go away.)

2. CPU cost of SSL/AES/HMAC/SHA. I would be surprised if this were actually an issue for bgranger's deployments, but maybe it is. Is there evidence that this is a problem? (If it is, then this is useful information for the future, in the future foolscap could use a more CPU-efficient encryption and authentication scheme. I was just describing to Brian a few minutes ago how there are new improved stream ciphers out now which seem to be both faster and safer than AES, for example.)

3. Time and network utilization costs of SSL startup. Again, our measurements have shown that this is negligible. What was it -- something like 10,000 connections per second? However, bgranger's network performance characteristics might be different, and if so I would like to learn about it. (A future version of foolscap could use a crypto protocol with faster startup than SSL.)

4. Availability of OpenSSL on deployment target. This one I totally believe is an issue for bgranger. I myself have had problems configuring and installing OpenSSL on Windows, Cygwin, and Mac OS X. I wouldn't be surprised if it either doesn't port to the supercomputer at all, or if it does so only with extensive configuration tweaks. Also, even if the source code can be easily built on a deployment target, that isn't the same as it being sufficiently convenient to depend on it being there. I wouldn't be surprised if the worst problem with requiring pyOpenSSL was that there is no simple, cross-platform way to express the fact that pyOpenSSL requires OpenSSL, and have that requirement automatically resolved.

5. Other? Is there any problem with using pyOpenSSL in your environment, bgranger, other than the problems mentioned above?

Thank you very much!

[Zooko]

Correction: Brian showed me the spot on the whiteboard where we had scribbled our notes about OpenSSL benchmarking. It was 100 connections per second, which caused a 30% CPU load. Presumably therefore it could handle 300 connections per second if it didn't need to do anything else with its CPU. This was on a typical server from a year or two ago -- 1 CPU, 1 core, about 2 or 3 GHz, Athlon64 or x86 architecture.

Also note that this isn't measuring latency incurred by the SSL connection negotiation, it is only measuring the "connection establishment throughput" -- connections established over some long time period (tens of seconds) divided by that time period.

[bgranger]

Replying to zooko:

This is an opportunity to learn about the costs of securing foolscap.

Why is it important to bgranger to avoid the requirement on pyOpenSSL?

Possible reasons:

1. Key management makes things harder. The foolscap design should make this not be the case. If it is still the case I would love to know why. (This is a unique feature of foolscap over alternative secure RPC mechanisms -- that the object-capability paradigm makes the key management problem go away.)

This is a not an issue for us for the reason you mention.

2. CPU cost of SSL/AES/HMAC/SHA. I would be surprised if this were actually an issue for bgranger's deployments, but maybe it is. Is there evidence that this is a problem? (If it is, then this is useful information for the future, in the future foolscap could use a more CPU-efficient encryption and authentication scheme. I was just describing to Brian a few minutes ago how there are new improved stream ciphers out now which seem to be both faster and safer than AES, for example.)

This _could_ be a huge issue for us. Our bottleneck process maintains a large number of persistent (hundreds) connections to workers. We need extremely low latency in all of these connections and if this single process has to do a lot of CPU bound work for encryption related things, we could have a problem. I would be very interested in learning about how to use the newer ciphers.

The other performance related thing is that much of the time, we are running over loopback in a single user workstation - in this context, most users will not need security. It seems crazy to force it on them.

3. Time and network utilization costs of SSL startup. Again, our measurements have shown that this is negligible. What was it -- something like 10,000 connections per second? However, bgranger's network performance characteristics might be different, and if so I would like to learn about it. (A future version of foolscap could use a crypto protocol with faster startup than SSL.)

All of our connections are persistent, so this is not a big issue.

4. Availability of OpenSSL on deployment target. This one I totally believe is an issue for bgranger. I myself have had problems configuring and installing OpenSSL on Windows, Cygwin, and Mac OS X. I wouldn't be surprised if it either doesn't port to the supercomputer at all, or if it does so only with extensive configuration tweaks. Also, even if the source code can be easily built on a deployment target, that isn't the same as it being sufficiently convenient to depend on it being there. I wouldn't be surprised if the worst problem with requiring pyOpenSSL was that there is no simple, cross-platform way to express the fact that pyOpenSSL requires OpenSSL, and have that requirement automatically resolved.

This is a huge one for us. We _have_ to be able to run anywhere that Python+twisted runs. The more exotic platforms for us include AIX and hacked up linux versions on supercomputers. If we can't run foolscap without SSL, we will have to consider dropping foolscap and instead go with plain PB for these contexts. But I _really_ don't want to have to maintain a PB version of all of our stuff.

5. Other? Is there any problem with using pyOpenSSL in your environment, bgranger, other than the problems mentioned above?

Thank you very much!

[Zooko]

The IPython project has long since dropped using foolscap and switched to "ZeroMQ". Their primary reason for that switch appers to be Python 3 support:

​http://ipython.org/ipython-doc/stable/development/ipythonzmq.html

(Aside: the work-arounds for the lack of access control/confidentiality/integrity in ZeroMQ appear to amount to a bit of hassle and/or vulnerability: ​http://ipython.org/ipython-doc/dev/parallel/parallel_security.html )

This probably means there are no users of foolscap who require insecure mode.

By the way, I'm reminded of Ian Grigg's immortal rules for secure protocol design:

"H3: There Is Only One Mode And It Is Secure"

​http://iang.org/ssl/h3_there_is_only_one_mode_and_it_is_secure.html

[Zooko]

Here's the first patch for this ticket: ​https://github.com/warner/foolscap/pull/16

[Brian Warner]

moving this to 0.7.0 . We'll put the minor stuff in the next minor release (0.6.5), and this more significant (API-breaking) change in the more-major release (0.7.0).

[Zooko]

For the record, I've recently had two users show up on IRC saying "I don't know why Tahoe-LAFS doesn't work! It says 'Failed to load application: crypto for PB is not available, try importing foolscap.crypto and see what happens'.".

(These users do not appear to know how to "try importing foolscap.crypto". They presumably don't know how to run Python code themselves, and are just getting this output when they try to execute tahoe-lafs.)

[Zooko]

I wrote ​a branch for this ticket in which I searched through all of the Foolscap source code and attempted to excise all of the Unauthenticated stuff. Unfortunately the result doesn't pass unit tests and I think I need Brian's help to make it do so.

[Brian Warner]

Ok, first step is a deprecation warning. The next release will contain that, then the following release will apply zooko's patch to remove everything unauthenticated, and require pyopenssl unconditionally.

[Brian Warner]

I'm not totally sure how I want to add the deprecation warning. The twisted approach would be to add a twisted.python.deprecate.deprecatedModuleAttribute to pb.py:

deprecatedModuleAttribute(Version("Foolscap", 0,7,0),
  "UnauthenticatedTub was deprecated in Foolscap-0.7.0 and will be removed in 0.8.0 . All Tubs will be authenticated. There is only one mode, and it is secure.",
  UnauthenticatedTub.__module__, UnauthenticatedTub.__name__)

However this currently isn't noisy enough: the only way to see the message is to run your foolscap-using program with -Wall, and no users do that. Ideally all foolscap-using downstream programs would run their test suites with -Wall, and discover the problem that way, but I don't think any developers do that either.

Twisted has a discussion about this issue (​https://twistedmatrix.com/trac/ticket/6348), from which I gather two things. One is that they intend to run the Twisted test suite with -Wall, so they can discover problems with their own upstream dependencies, but that bug means that they currently don't. The second is that the general compatibility/deprecation policy/contract they maintain for their downstream users is this: if you get a clean test run with -Wall against Twisted version N, then you should be able to run without problems (although possibly with deprecation warnings) against Twisted version N+1. I'd like Foolscap to offer a similar contract.

I'm on the fence as to whether enough downstream Foolscap users follow such practice. That, plus the relatively large magnitude of this change (there's not a whole lot of Foolscap users anyways, and I imagine a smaller percentage of them are using UnauthenticatedTub, but if they are then this is a fairly catastrophic change, as it will probably require a significant redesign), makes me lean towards making user-visible deprecation warnings. Specifically, by changing UnauthenticatedTub.__init__ to make the following call:

        warnings.warn("UnauthenticatedTub was deprecated in Foolscap-0.7.0 and will be removed in Foolscap-0.8.0. All Tubs will be authenticated. There is only one mode, and it is secure.", stacklevel=2)

That will show up on stderr in any program that uses UnauthenticatedTub . Note that we can't use warn(.., category=DeprecationWarning) because these are not displayed without -Wall (I think this was part of the "py2.7 is the last py2.x ever, stop warning about things that can't be fixed" policy for Python itself).

I think I want to add this line just before we tag 0.7.0, and then just after we tag it, immediately rip out all of the tests for UnauthenticatedTub, followed slowly by the implementation. We could add the line and remove the tests now, but then it might bitrot and break before the release. We could also add the line now and remove the tests later, but then our tests might get noisy and mask actual problems between now and the release.

Given the other changes pending (the Tor/endpoint stuff), I'm uncertain how to time this. We could do an 0.7.0 this week (deprecating UnT but not adding endpoints), then an 0.8.0 in maybe a month (adding endpoints). I want to give at least a month between the releases for the UnT warning to be noticed (3 or 6 months would be more appropriate, but I'm impatient to get this done with). I'll check with the Endpoint folks and see if that will be ok for their plans.

[Brian Warner]

For reference: the Foolscap test suite emits 35 warnings when run with this patch in place. The stacklevel=2 ought to mean we get one warning per call site, however I'm seeing the same test-case filename+lineno being printed multiple times, so something's not suppressing duplicates the way I thought it would.

[Brian Warner]

The deprecation was pushed in [bd164cafb3cc092d3224ebf5651c960418ab7448], and released in 0.7.0 .

Now moving this ticket out to 0.8.0 for the second phase: actually removing the code. ​https://github.com/warner/foolscap/tree/no-unauth currently contains the branch for that: it combines zooko's excellent work with some cleanups and fixes.

[Brian Warner]

I released 0.7.0 an hour ago, so I've landed the removal branch (with extra patches to fix up the remaining unit tests and reformat some code), in [9221b51cef08fe7f805ad1e64c5214babfb044da].

Awesome work Zooko, you performed some extensive and super-thorough surgery on that code. Nicely done!

I think we still need an install_requires= change to setup.py to mandate the dependency on pyopenssl, I'll hold off on closing this until that lands and gets tested.

[Brian Warner]

I've pushed the install_requires=['pyOpenSSL'] change, in 1b12906. UnauthenticatedTubs are gone!

One problem (recorded in #231) is that installs break when I include service_identity as a dependency. Twisted complains if you don't have service_identity installed (even though we don't use it, as it provides CA/X.509 -style cert checking, and Foolscap uses hash-of-certificate instead). So it'd be nice to depend upon it directly.

[Brian Warner]

Oh, there's probably a bit more code that can be removed: the Negotiation process allows the remote tub ID to be None, for example.