Issue2570

OS X is now failing in certain cases, namely yolk, because of a regression on #1112 For now, this may require patching setuptools for Jython itself, as was done earlier; see #2298

To reproduce (and this should be part of any release testing, as a smoke test):

1. Build an installer (in dist/jython-installer.jar):

$ ant all-jars  # will emit warnings like crazy, mostly around doc tags

2. Run the installer. There are many ways to run this, and they should all be ideally tested. More info on possible installer options:

$ java -jar dist/jython-installer.jar --help

However, the CLI is the easiest way to do any testing - especially since the installer has been very stable, with only pip/setuptools integration being added to it in the last 10 or so years:

$ java -jar dist/jython-installer.jar -s -d ~/jython-2.7.1-test-RC1
Performing silent installation
 10 %
 20 %
 30 %
 40 %
 50 %
 60 %
 70 %
 80 %
Generating start scripts ...
Installing pip and setuptools
 90 %
Collecting setuptools
Collecting pip
Installing collected packages: setuptools, pip
Successfully installed pip-9.0.1 setuptools-28.8.0
 100 %
Congratulations! You successfully installed Jython 2.7.1rc1 to directory /Users/jbaker/jython-2.7.1-test-RC1.

3. OK, let's try it out. First, does pip work?

$ cd ~/jython-2.7.1-test-RC1/
jimbaker:jython-2.7.1-test-RC1 jbaker$ bin/pip install yolk
Collecting yolk
/Users/jbaker/jython-2.7.1-test-RC1/Lib/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:310: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  warnings.warn(
  Using cached yolk-0.4.3.tar.gz
Requirement already satisfied: setuptools in ./Lib/site-packages (from yolk)
Installing collected packages: yolk
  Running setup.py install for yolk ... done
Successfully installed yolk-0.4.3

This SNI warning is due to the fact I'm running on Java 7; if you use Java 8, it goes away, and SNI is used (as in fact is tested in our regrtest).

4. Does setuptools work, in conjunction with yolk?

The reason I chose yolk is that it creates a executable in the bin directory for Jython:

$ ls -ls bin/yolk
8 -rwxr-xr-x  1 jbaker  staff  397 Mar 13 16:31 bin/yolk

However, right now it's not working on OS X, because yolk is being interpreted as a **shell script**:
$ bin/yolk
bin/yolk: line 3: __requires__: command not found
Version: ImageMagick 6.9.3-6 Q16 x86_64 2016-02-28 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2016 ImageMagick Studio LLC
License: http://www.imagemagick.org/script/license.php
... # most output elided but more of the same as the confusing text below!
By default, 'file' is written in the MIFF image format.  To
specify a particular image format, precede the filename with an image
format name and a colon (i.e. ps:image) or specify the image type as
the filename suffix (i.e. image.ps).  Specify 'file' as '-' for
standard input or output.
import: delegate library support not built-in `' (X11) @ error/import.c/ImportImageCommand/1297.
from: can't read /var/mail/pkg_resources
./yolk: line 9: syntax error near unexpected token `('
./yolk: line 9: `    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])'

This bug is because OS X does not properly support recursive shebangs, as noted in #1112 (and also https://www.in-ulm.de/~mascheck/various/shebang/#interpreter-script). To fix, instead of a shebang like the following:

#!/Users/jbaker/jython-2.7.1-test-RC1/bin/jython

We need this:

#!/usr/bin/env /Users/jbaker/jython-2.7.1-test-RC1/bin/jython

With just this small change, yolk will now work:

$ bin/yolk --list
pip 9.0.1 has no metadata
setuptools 28.8.0 has no metadata
wsgiref         - 0.1.2        - active development (/Users/jbaker/jython-2.7.1-test-RC1/Lib)
yolk            - 0.4.3        - active

Maybe related to https://hg.python.org/jython/rev/63fcb6b8993a

So, in a Jython sprint we investigated this issue and found the regression point. It's actually a regression in setup tools:
https://github.com/pypa/setuptools/commit/2b3cf9b12b23ac6beca6808c4c7620aa77f168e3
(added darjus to nosy for this)

Take a look at the discarded method as_header:
It contained the shebang correction that is vital for OSX.

In Jython-side this commit pulled the regression into Jython:
https://github.com/jythontools/jython/commit/5c5b20ed24ef5a9bc7b97abab9f45e506b9f25da

So, how should we fix it? Restore the relevant code in setuptools?
Or maybe better monkeypatch setuptools from Jython, so we have better control over this aspect, wich might be convenient if launchers change over time etc...
E.g. I suppose this problem would go away if we could introduce LiJy-launch as default launcher: https://github.com/Stewori/LiJy-launch
(It is constrained to OpenJDK's license (GPL with classpath exception) because it inherits from OpenJDK C-code. Don't know what trouble it would make to have a stricter license just for the launcher. The launcher is usually not subject to library binding etc.; Also, I will take a look if I can maybe remove or rewrite the relevant code, or maybe we can ask OpenJDK for permission to use the code snippet in question under PSF2 license.)

Agreed that using something like LiJy would make this problem go away, because we would have a real executable as a launcher, much as we see with the Windows launcher. Better yet, it would be direct. But there are both build (need to do this for every platform, although creative use of CPython could help) and licensing problems as pointed out.

If we could "seamlessly" monkeypatch setuptools CommandSpec as necessary, that would give us control for this thorny issue, without having to maintain a Jython-specific patch of setuptools (thereby limiting upgrades of pypa upstream work like pip), while preserving the ability to revise the installer for a specific version of Jython. Maybe we could manage through some suitable cleverness in site.py? Again, this only works because CommandSpec is relatively stable... And it is monkeypatching!

I think I fixed it. Will push it soon...

Stefan, thanks for working on this, and I look forward to seeing and trying out the commit!

Fixed as of https://hg.python.org/jython/rev/aed63ef4113d.

That is an interesting place to do the monkeypatch! I wonder what further power could be exposed there, given its centrality.

Works for me! I think we can safely close, this is a good approach for the time being.

Guys, sorry for this and thanks for the fix.

@Jim
> That is an interesting place to do the monkeypatch! I wonder what further power could be exposed there, given its centrality.

Hmmm, it was the only place I could think of that can guarantee the monkeypatch to be applied before the patched module is used. And at the same time avoids to patch it twice. It puzzles me a bit that the name 'easy_install' never shows up there, which is why I check for 'setuptools.command' (_fix_setuptools_jython_osx.py asserts that actually easy_install is the target).

@Darjus
NP ;)