Issue2732

Compilation of large modules (and modules with large methods) operates by spawning a CPython complation command. This works correctly, but no longer in the context of pip install.

Large module and method compilation works only if a path to CPython 2.7 is given via a Java system property:

PS> del large_module.pyc
PS> jython -J-Dcpython_cmd=C:\Python\2.7.15\python.exe large_module.py
PS> dir large_module.*
...
-a----       07/01/2019     20:21        1028129 large_module.py
-a----       08/01/2019     06:55         791318 large_module.pyc

Without the property, this is the result, even with CPython on the PATH:

PS> del large_module.pyc
PS> jython large_module.py
java.lang.RuntimeException:
Encountered too large method code in
large_module.py

Please provide a CPython 2.7 bytecode file (.pyc) to proceed, e.g. run
python -m py_compile large_module.py
and try again.

Alternatively provide proper CPython 2.7 execute command via
cpython_cmd property, e.g. call
    jython -J-Dcpython_cmd=python
or if running pip on Jython:
    pip install --global-option="-J-Dcpython_cmd=python" <package>
...

(On my machine, this advice comes out twice.) That this does not find the CPython available on the PATH may be a defect. But the main point of the issue I'm raising is that advice given with respect to pip no longer works.

PS> pip install --global-option=-J-Dcpython_cmd=C:\Python\2.7.15\python.exe mpmath
...
    error: option -J not recognized

This is the mpmath setup.py replying that it doesn't understand the Java property definition passed to it through --global-option. (See https://pip.pypa.io/en/stable/reference/pip/#build-options.)

Previously -J options given anywhere on the jython command line would (I think) be collected in our command line processing, and fronted, but this was technically incorrect and disappeared with #2686. The options after the executable file argument (-c, -m or a *.py) belongs to that program. The other "obvious" possibility:

PS> jython -Dcpython_cmd=C:\Python\2.7.15\python.exe -m pip install mpmath

fails because it is not the top-level Jython that needs to find CPython, but a subprocess created by pip, and Java properties are not inherited.

Solution ideas include:
1. Restore the non-standard processing of -J (or -J-D) options.
2. Find CPython another way (like PYTHONHOME or PATH).
3. Use a compiler we provide in a reliable location (e.g. a script in ./bin).
4. Advise defining cpython_cmd via JAVA_OPTS or JYTHON_OPTS or the registry.

Just a note to say I did, in fact, succeed with:

PS> $env:JYTHON_OPTS='-Dcpython_cmd=C:\Python\2.7.15\python.exe'
PS> pip install mpmath

and with sympy as a whole, and the equivalent on Linux:

$ export JYTHON_OPTS="-Dcpython_cmd=python"
$ pip install mpmath

If we like this answer, then it is a documentation (error message) change only.

All our other properties are namespaced to python.*: I was wondering about python.c.python, or python.cpython2, or similar for consistency?

> That this does not find the CPython available on the PATH may be a defect.

When I implemented that I felt like if it called into native CPython totally silently that would be too much magic under the hood. The message is also meant to let the user consciously acknowledge that it is invoking CPython now.
We can decide to add support for automatic CPython calls. Hoever I also wanted to let the user explicitly name the CPython to be used to ensure we won't get Python3 vs 2 bytecode incompatibility issues.

> If we like this answer, then it is a documentation (error message) change only.

Let's just fix the error message for now (i.e. for this release).

@Stefan Yes, let's.

I've done this now, with a bit of rationalisation of the strings too. Now the advice reads:

PS jython-jvm9> dist\bin\jython large_module.py
java.lang.RuntimeException: Module or method too large in `large_module.py`.

Please provide a CPython 2.7 bytecode file (.pyc), e.g. run
    python -m py_compile large_module.py

Alternatively, specify a CPython 2.7 command via the python.cpython2 property, e.g.:
    jython -Dpython.cpython2=python
or (e.g. for pip) through the environment variable JYTHON_OPTS:
    export JYTHON_OPTS="-Dpython.cpython2=python"

And this advice works (at least in the development environment):

PS jython-jvm9> dist\bin\jython "-Dpython.cpython2=C:\Python\2.7.15\python.exe" large_module.py
PS jython-jvm9> del large_*.pyc
PS jython-jvm9> $env:JYTHON_OPTS="-Dpython.cpython2=C:\Python\2.7.15\python.exe"
PS jython-jvm9> dist\bin\jython large_module.py

In an installed jython, in response to "pip install mpmath", the failure message comes out fairly prominantly (for tests\test_fp), but this works:
PS issue2726> $env:JYTHON_OPTS="-Dpython.cpython2=C:\Python\2.7.15\python.exe"
PS issue2726> pip install mpmath
...
Successfully installed mpmath-1.1.0
...

Now claiming fixed at https://hg.python.org/jython/rev/96917c0883da

Regression tests check that Stefan's technique it works when it should, and I haven't broken it, but not that it produces the right error messages in all circumstances. I've verified some manually. Tyre kicking by others would be welcome.

Now fixed at https://hg.python.org/jython/rev/96917c0883da

Or fixed "for now" as Stefan suggests above. But I will move it towards closed, and we can open another issue if it bites again.