Issue2732

classification
Title: Regression in large module support for pip
Type: crash Severity: normal
Components: Versions: Jython 2.7
Milestone: Jython 2.7.2
process
Status: open Resolution: accepted
Dependencies: Superseder:
Assigned To: Nosy List: jeff.allen, stefan.richthofer
Priority: low Keywords:

Created on 2019-01-08.08:52:19 by jeff.allen, last changed 2019-01-08.21:09:18 by stefan.richthofer.

Messages
msg12296 (view) Author: Jeff Allen (jeff.allen) Date: 2019-01-08.08:52:19
Compilation of large modules (and modules with large methods) operates by spawning a CPython complation command. This works correctly, but no longer in the context of pip install.

Large module and method compilation works only if a path to CPython 2.7 is given via a Java system property:

PS> del large_module.pyc
PS> jython -J-Dcpython_cmd=C:\Python\2.7.15\python.exe large_module.py
PS> dir large_module.*
...
-a----       07/01/2019     20:21        1028129 large_module.py
-a----       08/01/2019     06:55         791318 large_module.pyc

Without the property, this is the result, even with CPython on the PATH:

PS> del large_module.pyc
PS> jython large_module.py
java.lang.RuntimeException:
Encountered too large method code in
large_module.py

Please provide a CPython 2.7 bytecode file (.pyc) to proceed, e.g. run
python -m py_compile large_module.py
and try again.

Alternatively provide proper CPython 2.7 execute command via
cpython_cmd property, e.g. call
    jython -J-Dcpython_cmd=python
or if running pip on Jython:
    pip install --global-option="-J-Dcpython_cmd=python" <package>
...

(On my machine, this advice comes out twice.) That this does not find the CPython available on the PATH may be a defect. But the main point of the issue I'm raising is that advice given with respect to pip no longer works.

PS> pip install --global-option=-J-Dcpython_cmd=C:\Python\2.7.15\python.exe mpmath
...
    error: option -J not recognized

This is the mpmath setup.py replying that it doesn't understand the Java property definition passed to it through --global-option. (See https://pip.pypa.io/en/stable/reference/pip/#build-options.)

Previously -J options given anywhere on the jython command line would (I think) be collected in our command line processing, and fronted, but this was technically incorrect and disappeared with #2686. The options after the executable file argument (-c, -m or a *.py) belongs to that program. The other "obvious" possibility:

PS> jython -Dcpython_cmd=C:\Python\2.7.15\python.exe -m pip install mpmath

fails because it is not the top-level Jython that needs to find CPython, but a subprocess created by pip, and Java properties are not inherited.

Solution ideas include:
1. Restore the non-standard processing of -J (or -J-D) options.
2. Find CPython another way (like PYTHONHOME or PATH).
3. Use a compiler we provide in a reliable location (e.g. a script in ./bin).
4. Advise defining cpython_cmd via JAVA_OPTS or JYTHON_OPTS or the registry.
msg12298 (view) Author: Jeff Allen (jeff.allen) Date: 2019-01-08.20:17:56
Just a note to say I did, in fact, succeed with:

PS> $env:JYTHON_OPTS='-Dcpython_cmd=C:\Python\2.7.15\python.exe'
PS> pip install mpmath

and with sympy as a whole, and the equivalent on Linux:

$ export JYTHON_OPTS="-Dcpython_cmd=python"
$ pip install mpmath

If we like this answer, then it is a documentation (error message) change only.

All our other properties are namespaced to python.*: I was wondering about python.c.python, or python.cpython2, or similar for consistency?
msg12302 (view) Author: Stefan Richthofer (stefan.richthofer) Date: 2019-01-08.21:09:18
> That this does not find the CPython available on the PATH may be a defect.

When I implemented that I felt like if it called into native CPython totally silently that would be too much magic under the hood. The message is also meant to let the user consciously acknowledge that it is invoking CPython now.
We can decide to add support for automatic CPython calls. Hoever I also wanted to let the user explicitly name the CPython to be used to ensure we won't get Python3 vs 2 bytecode incompatibility issues.

> If we like this answer, then it is a documentation (error message) change only.

Let's just fix the error message for now (i.e. for this release).
History
Date User Action Args
2019-01-08 21:09:18stefan.richthofersetmessages: + msg12302
2019-01-08 20:17:56jeff.allensetpriority: normal -> low
nosy: + stefan.richthofer
resolution: accepted
messages: + msg12298
milestone: Jython 2.7.2
2019-01-08 08:52:19jeff.allencreate