make "flogtool", "flappclient", and "flappserver" executables that work on Windows

[Zooko]

I'm use to running command-lines using flogtool. On Windows, however, that command emits an error message, and I have to use cygwin bash and enter the following script in order to run flogtool:

python `cygpath -m \`type -p flogtool \` `

If you're interested, Brian, I'll happily submit a patch which uses setuptools, if available, to make a flogtool executable for the target platform..

[Zooko]

I'm trying to automate the upload of Python packages for pycryptopp using flappserver. Flappserver is great! It works nicely on Linux:

​http://allmydata.org/buildbot-pycryptopp/builders/linux-amd64-ubuntu-jaunty-yukyuk/builds/26/steps/upload%20egg/logs/stdio

and cygwin:

​http://allmydata.org/buildbot-pycryptopp/builders/windows-cygwin/builds/76/steps/upload%20egg/logs/stdio

but unfortunately not on Windows:

​http://allmydata.org/buildbot-pycryptopp/builders/windows-native/builds/100/steps/upload%20egg/logs/stdio

Foolscap was installed with "easy_install -U foolscap" on all three platforms.

Here is a patch which (probably -- not tested) makes "easy_install -U foolscap" create working flogtool, flappclient, and flappserver on Windows.

HACK yukyuk:~/playground/foolscap/foolscap-trunk$ hg diff
diff -r f2bc95a71091 setup.py
--- a/setup.py  Wed Jun 24 17:21:48 2009 -0700
+++ b/setup.py  Sun Jul 26 21:33:12 2009 -0600
@@ -62,6 +62,11 @@
     setup_args['extras_require'] = { 'secure_connections' : ["pyOpenSSL"] }
     # note that pyOpenSSL-0.7 and recent Twisted causes unit test failures,
     # see bug #62
+    setup_args['console_scripts'] = [
+        'flogtool = foolscap.logging.cli:run_flogtool',
+        'flappserver = foolscap.appserver.cli:run_flappserver',
+        'flappclient = foolscap.appserver.client:run_flappclient',
+        ]

 if __name__ == '__main__':
     setup(**setup_args)

[Zooko]

Nope, that patch didn't work. Here's one that was actually tested and does work, as shown in this buildbot result:

​http://allmydata.org/buildbot-pycryptopp/builders/windows-native/builds/101/steps/upload%20egg

HACK yukyuk:~/playground/foolscap/foolscap-trunk$ hg diff
diff -r f2bc95a71091 setup.py
--- a/setup.py  Wed Jun 24 17:21:48 2009 -0700
+++ b/setup.py  Sun Jul 26 21:42:49 2009 -0600
@@ -48,7 +48,6 @@

         'packages': ["foolscap", "foolscap/slicers", "foolscap/logging",
                      "foolscap/appserver", "foolscap/test"],
-        'scripts': ["bin/flogtool", "bin/flappserver", "bin/flappclient"],
 }

 try:
@@ -56,12 +55,17 @@
     # arguments to the setup args.
     import setuptools
 except ImportError:
-    pass
+    setup_args['scripts'] = ["bin/flogtool", "bin/flappserver", "bin/flappclient"]
 else:
     setup_args['install_requires'] = ['twisted >= 2.4.0']
     setup_args['extras_require'] = { 'secure_connections' : ["pyOpenSSL"] }
     # note that pyOpenSSL-0.7 and recent Twisted causes unit test failures,
     # see bug #62
+    setup_args['entry_points'] = { 'console_scripts': [
+        'flogtool = foolscap.logging.cli:run_flogtool',
+        'flappserver = foolscap.appserver.cli:run_flappserver',
+        'flappclient = foolscap.appserver.client:run_flappclient',
+        ] }

 if __name__ == '__main__':
     setup(**setup_args)

[Zooko]

The previous buildbot link shows flappclient being successfully invoked on Windows, which is what this patch is intended to accomplish. However, it quickly errored out due to finding no furl file. The following buildbot link will hopefully soon show a successful upload after finding the furl file:

​http://allmydata.org/buildbot-pycryptopp/builders/windows-native/builds/102/steps/upload%20egg

[Zooko]

Okay, that buildbot run also failed, because the upload-egg.furl file had an extra "FURL is " prepended to the furl, but this run succeeded:

​http://allmydata.org/buildbot-pycryptopp/builders/windows-native/builds/103/steps/upload%20egg/logs/stdio

[Brian Warner]

I'm in an airport now, but when I get back to a unix box tomorrow, I'll see what that setup.py change does in the unix/debian world. I remember resisting this sort of change in the past, because it caused my debian /usr/bin/flogtool binaries to start depending upon a bunch of setuptools things which didn't always work. I'd really prefer the flogtool/flappserver/etc binaries which go into debian packages to not have the extra setuptools dependency checks: they've caused problems in the past.

[Zooko]

So with the patch as is, it branches on whether import setuptools raises an ImportError or not. If it does, then the build, including the resulting scripts, proceeds as before. If that's not good enough control over this for you I'll be happy to help with other ideas.

For what it is worth, those issues with the setuptools dependency checks seem to be receding into the past. Most of them had to do with Debian packages that didn't have normal distutils-generated .egg-info files, as far as I recall.

[Brian Warner]

Life would be easiest if we only cared about sid :-). My concern is with older debian releases: I want Foolscap to remain usable on dapper, even if dapper doesn't have valid .egg-info files for e.g. pyopenssl. I don't know offhand what the state of those older platforms are.. I'll need to do some testing.

(one fallback position would be to switch on sys.platform=="win32", and only use the entrypoint scripts on windows. I'll investigate the older debian platforms before considering such a compromise, though).

[Zooko]

I would like to facilitate Peter Secor using flappclient to upload Windows packages for the tahoe-w32-client that he just announced the Open Sourcement of:

​http://allmydata.org/trac/tahoe-w32-client/wiki

I could do this by patching foolscap to fix this issue and then installing my patched foolscap on Peter's Windows buildslave, but in the long run it will probably enable Peter and Brian to work together without as much help from me if Peter can install upstream foolscap instead of a Zooko-patched foolscap.

So, could you think about this ticket today?

I checked, and the Twisted v2.2.0 (for Python 2.4) which comes with Dapper, doesn't seem to have .egg-info files, so I wouldn't expect the executable generated by setuptools to work on dapper. (Unless of course a dapper user ran sudo easy_install foolscap, in which case it probably would work, but would install a nice new Twisted 8.2.0 egg first.)

Hm, how about this. Here is a patch which attempts to avoid the "console scripts" feature on older Linux distributions. See the patch comments for details.

Oh, by the way, it looks like the current release of foolscap specifies that it requires Twisted >= 2.4.0. If you want to support Dapper, maybe you should relax that to Twisted >= 2.2.0.

I'll attach this patch as an attachment as well.

HACK yukyuk:~/playground/foolscap/foolscap-trunk$ hg diff
diff -r f2bc95a71091 setup.py
--- a/setup.py  Wed Jun 24 17:21:48 2009 -0700
+++ b/setup.py  Sun Aug 16 14:12:18 2009 -0600
@@ -48,21 +48,43 @@

         'packages': ["foolscap", "foolscap/slicers", "foolscap/logging",
                      "foolscap/appserver", "foolscap/test"],
-        'scripts': ["bin/flogtool", "bin/flappserver", "bin/flappclient"],
 }

+use_console_scripts = None
+
 try:
     # If setuptools is installed, then we'll add setuptools-specific
     # arguments to the setup args.
     import setuptools
 except ImportError:
-    pass
+    use_console_scripts = False
 else:
     setup_args['install_requires'] = ['twisted >= 2.4.0']
     setup_args['extras_require'] = { 'secure_connections' : ["pyOpenSSL"] }
     # note that pyOpenSSL-0.7 and recent Twisted causes unit test failures,
     # see bug #62

+    # Now if setuptools is < 0.6c8 then we might be on an old Linux
+    # distribution where packages have had their .egg-info files stripped out.
+    # (Hardy and Lenny don't have this problem, and they each have
+    # setuptools-0.6c8.)
+    import pkg_resources # comes with setuptools
+    pv = pkg_resources.parse_version
+    if pv(setuptools.__version__) < pv("0.6c8"):
+        # Don't use the console_scripts feature -- it interacts badly with old
+        # Linux distributions that have stripped out the .egg-info files.
+        use_console_scripts = False
+    else:
+        use_console_scripts = True
+
+if use_console_scripts:
+    setup_args['entry_points'] = { 'console_scripts': [
+        'flogtool = foolscap.logging.cli:run_flogtool',
+        'flappserver = foolscap.appserver.cli:run_flappserver',
+        'flappclient = foolscap.appserver.client:run_flappclient',
+        ] }
+else:
+    setup_args['scripts'] = ["bin/flogtool", "bin/flappserver", "bin/flappclient"]
+
 if __name__ == '__main__':
     setup(**setup_args)
-

[Zooko]

Oh by the way I haven't tested this patch yet. If it looks potentially acceptable to you I could test it, for example by applying it to foolscap trunk and then installing the result onto Peter's Windows buildslave. :-)

[Zooko]

Brian told me on the phone the other day that he thought the kludge of using setuptools version to infer age of Linux distro was too fragile, and anyway he doesn't see any benefit to the console scripts feature on Linux, and at least one drawback -- it makes it harder for people to figure out what Python code a given script will run. At the time we discussed using console scripts only on Windows, but when editing this patch just now I started thinking that it might be a good idea on Mac, too. It makes sure that the resulting script has a shebang line invoking a particular python interpreter, which can help in the situation (all too common on Mac) that different versions of Python are or have been installed on the system. (There is the shipped-by-Apple, the fink, the macports, and the downloaded-from-python.org, not to mention the compiled-yerself. For some of these pairs, pyOpenSSL compiled for one will just ImportError or segfault or something when imported by the other.)

Here is a patch which enables console scripts except on Linux.

diff -r 7a10069855c4 setup.py
--- a/setup.py	Thu Jul 30 17:17:53 2009 -0700
+++ b/setup.py	Fri Sep 04 20:49:04 2009 -0600
@@ -48,21 +48,39 @@ object reference system, and a capabilit
 
         'packages': ["foolscap", "foolscap/slicers", "foolscap/logging",
                      "foolscap/appserver", "foolscap/test"],
-        'scripts': ["bin/flogtool", "bin/flappserver", "bin/flappclient"],
 }
+
+use_console_scripts = None
 
 try:
     # If setuptools is installed, then we'll add setuptools-specific
     # arguments to the setup args.
     import setuptools
 except ImportError:
-    pass
+    use_console_scripts = False
 else:
     setup_args['install_requires'] = ['twisted >= 2.4.0']
     setup_args['extras_require'] = { 'secure_connections' : ["pyOpenSSL"] }
     # note that pyOpenSSL-0.7 and recent Twisted causes unit test failures,
     # see bug #62
 
+    # Some old Linux distribution stripped the .egg-info files from
+    # some python packages. (Hardy and Lenny don't have this problem, nor
+    # does Fedora going back at least to Fedora 9.)
+    import platform
+    if platform.system() == 'Linux':
+        use_console_scripts = False
+    else:
+        use_console_scripts = True
+
+if use_console_scripts:
+    setup_args['entry_points'] = { 'console_scripts': [
+        'flogtool = foolscap.logging.cli:run_flogtool',
+        'flappserver = foolscap.appserver.cli:run_flappserver',
+        'flappclient = foolscap.appserver.client:run_flappclient',
+        ] }
+else:
+    setup_args['scripts'] = ["bin/flogtool", "bin/flappserver", "bin/flappclient"]
+
 if __name__ == '__main__':
     setup(**setup_args)
-

Brian: please support Windows somehow -- however you prefer -- as soon as possible, to make it easier for me to help Peter Secor, Shawn Willden, et al. support Tahoe-LAFS and Tahoe-w32-client on Windows.

[Brian Warner]

Doesn't the normal "python setup.py install" take care of the shbang lines? I don't think we need entrypoint scripts to accomplish that.

As far as I can tell, entrypoint scripts provide two features over regular distutils scripts=:

• automatic windows .bat-file generation
• import "right" version of Foolscap in the presence of multiple installed versions, via pkg_resources.require

I consider the latter situation to be a bad idea, so I'm only really interested in supporting the former.

I'll apply a patch which uses entry-point scripts when sys.platform=="win32" and doesn't otherwise. I'll try to write this patch tonight.

[Brian Warner]

only use entrypoint scripts when on windows

[Brian Warner]

Argh, the patch I tried to use (attached as "setuptools-on-windows.txt"), even though it doesn't use entry_points=, still creates impenetrable /bin/flogtool scripts. In this case, it looks like:

import pkg_resources
pkg_resouces.run_script('foolscap==0.4.2-', 'flogtool')

which presumeably finds the matching .egg, extracts EGG-INFO/scripts/flogtool (to a temporary directory? each time it is run?), puts the .egg on sys.path, then runs the extracted script.

ick. I must investigate further. Maybe I want to take a step further and only do import setuptools on windows...

[Brian Warner]

Ok, so strace tells me that it's not actually writing the "flogtool" script to a tempdir. Instead, it opens the .egg file *55* times to execute a basic "flogtool --version". The runtime of this setuptools-generated script (as opposed to running ./bin/flogtool directly) is the same, so I can't complain/claim that those 55 times are slowing things down. (in fact, on OS-X+Filevault, the setuptools form runs marginally faster, probably due to file IO being so slow that a single .egg file can be loaded faster than dozens of separate .py files).

But I'm still annoyed by the generated script: it has most of the negative properties that I complained about in ​http://allmydata.org/pipermail/tahoe-dev/2009-August/002702.html . I notice that the /usr/bin/flogtool on my debian system doesn't look like this: I think that means that the debian packager must have commented out the setuptools import, or explictly built the package on a setuptools-free system. (incidentally, having the build results change depending upon whether setuptools was installed or not is a bad thing, and I'm sure the debian packagers would prefer it not behave this way).

I'll try to look into this further later in the week. I don't know what the best answer is. If I can't find a better solution, I'm willing to use these icky generated scripts, but I won't like it :).

[Zooko]

I guess you can accomplish what you want by making sure not to import setuptools when you don't want to produce these icky scripts. I would also be satisfied if you just lived with the icky scripts, especially if you also open a bug report with the Distribute project saying what is icky about them and how the Distribute project can make them less icky: ​http://bitbucket.org/tarek/distribute/issues/new/ . Most of all, I just want Peter to be able to install foolscap and then use flappclient to upload Windows binaries of Tahoe-LAFS, as soon as possible, any way that works.

[Brian Warner]

Interesting: two things affect the generated script: setup.py's scripts=-vs-entry_points= argument, and invoking setup.py install with or without --single-version-externally-managed. (when not using --svem, you must also set PYTHONPATH and create all the target directories first). Then you get the following:

• scripts=, no --svem:

  import pkg_resources
  pkg_resources.run_script("foolscap==0.4.2-", "flogtool")
  

• scripts=, yes --svem: the original bin/flogtool, from the source tree
• entry_points=, no --svem

  from pkg_resources import load_entry_point
  sys.exit(load_entry_point("foolscap==0.4.2-", "console_scripts", "flogtool")())
  

• entry_points=, yes --svem:

  from pkg_resources import load_entry_point
  sys.exit(load_entry_point("foolscap==0.4.2-", "console_scripts", "flogtool")())
  

So it's likely that the debian packagers have taken to using --single-version-externally-managed to cause setup.py install to behave more like it does under distutils. (of course, that also means that setuptools must be installed, i.e. listed as a build-time dependency, because otherwise --svem would cause an error). And that I must do the same (*and* use scripts= instead of entry_points=) to get my original bin/flogtool to be installed instead of the pkg_resources-based wrapper.

Ok, so that means my use-entrypoints-on-windows patch *should* work, I'm just forced to use --svem on any other system that has setuptools installed. This feels like an acceptable compromise. I just pushed that patch (as [69e32419fd13b45d00c55ed8d236416efc67f3ed]) which should close this ticket. Please let me know if this fixes the problem.

[Zooko]

To make it easy for Black Dew to test this out I did:

hg clone ​http://foolscap.lothar.com/repos/trunk foolscap-trunk cd foolscap-trunk python setup.py sdist # this produces "dist/foolscap-0.4.2-.tar.gz"

Then I scp'ed the .tar.gz to my web server so that Black Dew can install it with "easy_install -U ​http://zooko.com/foolscap-0.4.2-.tar.gz"

[Brian Warner]

No reports of failure yet, so I'm assuming this one can be closed.