Issue2732

classification
Title: Regression in large module support for pip
Type: crash Severity: normal
Components: Versions: Jython 2.7
Milestone: Jython 2.7.2
process
Status: pending Resolution: fixed
Dependencies: Superseder:
Assigned To: jeff.allen Nosy List: jeff.allen, stefan.richthofer
Priority: low Keywords:

Created on 2019-01-08.08:52:19 by jeff.allen, last changed 2019-03-18.07:58:44 by jeff.allen.

Messages
msg12296 (view) Author: Jeff Allen (jeff.allen) Date: 2019-01-08.08:52:19
Compilation of large modules (and modules with large methods) operates by spawning a CPython complation command. This works correctly, but no longer in the context of pip install.

Large module and method compilation works only if a path to CPython 2.7 is given via a Java system property:

PS> del large_module.pyc
PS> jython -J-Dcpython_cmd=C:\Python\2.7.15\python.exe large_module.py
PS> dir large_module.*
...
-a----       07/01/2019     20:21        1028129 large_module.py
-a----       08/01/2019     06:55         791318 large_module.pyc

Without the property, this is the result, even with CPython on the PATH:

PS> del large_module.pyc
PS> jython large_module.py
java.lang.RuntimeException:
Encountered too large method code in
large_module.py

Please provide a CPython 2.7 bytecode file (.pyc) to proceed, e.g. run
python -m py_compile large_module.py
and try again.

Alternatively provide proper CPython 2.7 execute command via
cpython_cmd property, e.g. call
    jython -J-Dcpython_cmd=python
or if running pip on Jython:
    pip install --global-option="-J-Dcpython_cmd=python" <package>
...

(On my machine, this advice comes out twice.) That this does not find the CPython available on the PATH may be a defect. But the main point of the issue I'm raising is that advice given with respect to pip no longer works.

PS> pip install --global-option=-J-Dcpython_cmd=C:\Python\2.7.15\python.exe mpmath
...
    error: option -J not recognized

This is the mpmath setup.py replying that it doesn't understand the Java property definition passed to it through --global-option. (See https://pip.pypa.io/en/stable/reference/pip/#build-options.)

Previously -J options given anywhere on the jython command line would (I think) be collected in our command line processing, and fronted, but this was technically incorrect and disappeared with #2686. The options after the executable file argument (-c, -m or a *.py) belongs to that program. The other "obvious" possibility:

PS> jython -Dcpython_cmd=C:\Python\2.7.15\python.exe -m pip install mpmath

fails because it is not the top-level Jython that needs to find CPython, but a subprocess created by pip, and Java properties are not inherited.

Solution ideas include:
1. Restore the non-standard processing of -J (or -J-D) options.
2. Find CPython another way (like PYTHONHOME or PATH).
3. Use a compiler we provide in a reliable location (e.g. a script in ./bin).
4. Advise defining cpython_cmd via JAVA_OPTS or JYTHON_OPTS or the registry.
msg12298 (view) Author: Jeff Allen (jeff.allen) Date: 2019-01-08.20:17:56
Just a note to say I did, in fact, succeed with:

PS> $env:JYTHON_OPTS='-Dcpython_cmd=C:\Python\2.7.15\python.exe'
PS> pip install mpmath

and with sympy as a whole, and the equivalent on Linux:

$ export JYTHON_OPTS="-Dcpython_cmd=python"
$ pip install mpmath

If we like this answer, then it is a documentation (error message) change only.

All our other properties are namespaced to python.*: I was wondering about python.c.python, or python.cpython2, or similar for consistency?
msg12302 (view) Author: Stefan Richthofer (stefan.richthofer) Date: 2019-01-08.21:09:18
> That this does not find the CPython available on the PATH may be a defect.

When I implemented that I felt like if it called into native CPython totally silently that would be too much magic under the hood. The message is also meant to let the user consciously acknowledge that it is invoking CPython now.
We can decide to add support for automatic CPython calls. Hoever I also wanted to let the user explicitly name the CPython to be used to ensure we won't get Python3 vs 2 bytecode incompatibility issues.

> If we like this answer, then it is a documentation (error message) change only.

Let's just fix the error message for now (i.e. for this release).
msg12362 (view) Author: Jeff Allen (jeff.allen) Date: 2019-03-17.07:30:36
@Stefan Yes, let's.

I've done this now, with a bit of rationalisation of the strings too. Now the advice reads:

PS jython-jvm9> dist\bin\jython large_module.py
java.lang.RuntimeException: Module or method too large in `large_module.py`.

Please provide a CPython 2.7 bytecode file (.pyc), e.g. run
    python -m py_compile large_module.py

Alternatively, specify a CPython 2.7 command via the python.cpython2 property, e.g.:
    jython -Dpython.cpython2=python
or (e.g. for pip) through the environment variable JYTHON_OPTS:
    export JYTHON_OPTS="-Dpython.cpython2=python"

And this advice works (at least in the development environment):

PS jython-jvm9> dist\bin\jython "-Dpython.cpython2=C:\Python\2.7.15\python.exe" large_module.py
PS jython-jvm9> del large_*.pyc
PS jython-jvm9> $env:JYTHON_OPTS="-Dpython.cpython2=C:\Python\2.7.15\python.exe"
PS jython-jvm9> dist\bin\jython large_module.py

In an installed jython, in response to "pip install mpmath", the failure message comes out fairly prominantly (for tests\test_fp), but this works:
PS issue2726> $env:JYTHON_OPTS="-Dpython.cpython2=C:\Python\2.7.15\python.exe"
PS issue2726> pip install mpmath
...
Successfully installed mpmath-1.1.0
...
msg12370 (view) Author: Jeff Allen (jeff.allen) Date: 2019-03-18.07:22:48
Now claiming fixed at https://hg.python.org/jython/rev/96917c0883da

Regression tests check that Stefan's technique it works when it should, and I haven't broken it, but not that it produces the right error messages in all circumstances. I've verified some manually. Tyre kicking by others would be welcome.
msg12376 (view) Author: Jeff Allen (jeff.allen) Date: 2019-03-18.07:58:44
Now fixed at https://hg.python.org/jython/rev/96917c0883da

Or fixed "for now" as Stefan suggests above. But I will move it towards closed, and we can open another issue if it bites again.
History
Date User Action Args
2019-03-18 07:58:44jeff.allensetmessages: + msg12376
2019-03-18 07:22:48jeff.allensetstatus: open -> pending
resolution: accepted -> fixed
messages: + msg12370
2019-03-17 07:30:37jeff.allensetassignee: jeff.allen
messages: + msg12362
2019-01-08 21:09:18stefan.richthofersetmessages: + msg12302
2019-01-08 20:17:56jeff.allensetpriority: normal -> low
nosy: + stefan.richthofer
resolution: accepted
messages: + msg12298
milestone: Jython 2.7.2
2019-01-08 08:52:19jeff.allencreate