Created on 2019-01-08.08:52:19 by jeff.allen, last changed 2019-03-18.07:58:44 by jeff.allen.
|msg12296 (view)||Author: Jeff Allen (jeff.allen)||Date: 2019-01-08.08:52:19|
Compilation of large modules (and modules with large methods) operates by spawning a CPython complation command. This works correctly, but no longer in the context of pip install. Large module and method compilation works only if a path to CPython 2.7 is given via a Java system property: PS> del large_module.pyc PS> jython -J-Dcpython_cmd=C:\Python\2.7.15\python.exe large_module.py PS> dir large_module.* ... -a---- 07/01/2019 20:21 1028129 large_module.py -a---- 08/01/2019 06:55 791318 large_module.pyc Without the property, this is the result, even with CPython on the PATH: PS> del large_module.pyc PS> jython large_module.py java.lang.RuntimeException: Encountered too large method code in large_module.py Please provide a CPython 2.7 bytecode file (.pyc) to proceed, e.g. run python -m py_compile large_module.py and try again. Alternatively provide proper CPython 2.7 execute command via cpython_cmd property, e.g. call jython -J-Dcpython_cmd=python or if running pip on Jython: pip install --global-option="-J-Dcpython_cmd=python" <package> ... (On my machine, this advice comes out twice.) That this does not find the CPython available on the PATH may be a defect. But the main point of the issue I'm raising is that advice given with respect to pip no longer works. PS> pip install --global-option=-J-Dcpython_cmd=C:\Python\2.7.15\python.exe mpmath ... error: option -J not recognized This is the mpmath setup.py replying that it doesn't understand the Java property definition passed to it through --global-option. (See https://pip.pypa.io/en/stable/reference/pip/#build-options.) Previously -J options given anywhere on the jython command line would (I think) be collected in our command line processing, and fronted, but this was technically incorrect and disappeared with #2686. The options after the executable file argument (-c, -m or a *.py) belongs to that program. The other "obvious" possibility: PS> jython -Dcpython_cmd=C:\Python\2.7.15\python.exe -m pip install mpmath fails because it is not the top-level Jython that needs to find CPython, but a subprocess created by pip, and Java properties are not inherited. Solution ideas include: 1. Restore the non-standard processing of -J (or -J-D) options. 2. Find CPython another way (like PYTHONHOME or PATH). 3. Use a compiler we provide in a reliable location (e.g. a script in ./bin). 4. Advise defining cpython_cmd via JAVA_OPTS or JYTHON_OPTS or the registry.
|msg12298 (view)||Author: Jeff Allen (jeff.allen)||Date: 2019-01-08.20:17:56|
Just a note to say I did, in fact, succeed with: PS> $env:JYTHON_OPTS='-Dcpython_cmd=C:\Python\2.7.15\python.exe' PS> pip install mpmath and with sympy as a whole, and the equivalent on Linux: $ export JYTHON_OPTS="-Dcpython_cmd=python" $ pip install mpmath If we like this answer, then it is a documentation (error message) change only. All our other properties are namespaced to python.*: I was wondering about python.c.python, or python.cpython2, or similar for consistency?
|msg12302 (view)||Author: Stefan Richthofer (stefan.richthofer)||Date: 2019-01-08.21:09:18|
> That this does not find the CPython available on the PATH may be a defect. When I implemented that I felt like if it called into native CPython totally silently that would be too much magic under the hood. The message is also meant to let the user consciously acknowledge that it is invoking CPython now. We can decide to add support for automatic CPython calls. Hoever I also wanted to let the user explicitly name the CPython to be used to ensure we won't get Python3 vs 2 bytecode incompatibility issues. > If we like this answer, then it is a documentation (error message) change only. Let's just fix the error message for now (i.e. for this release).
|msg12362 (view)||Author: Jeff Allen (jeff.allen)||Date: 2019-03-17.07:30:36|
@Stefan Yes, let's. I've done this now, with a bit of rationalisation of the strings too. Now the advice reads: PS jython-jvm9> dist\bin\jython large_module.py java.lang.RuntimeException: Module or method too large in `large_module.py`. Please provide a CPython 2.7 bytecode file (.pyc), e.g. run python -m py_compile large_module.py Alternatively, specify a CPython 2.7 command via the python.cpython2 property, e.g.: jython -Dpython.cpython2=python or (e.g. for pip) through the environment variable JYTHON_OPTS: export JYTHON_OPTS="-Dpython.cpython2=python" And this advice works (at least in the development environment): PS jython-jvm9> dist\bin\jython "-Dpython.cpython2=C:\Python\2.7.15\python.exe" large_module.py PS jython-jvm9> del large_*.pyc PS jython-jvm9> $env:JYTHON_OPTS="-Dpython.cpython2=C:\Python\2.7.15\python.exe" PS jython-jvm9> dist\bin\jython large_module.py In an installed jython, in response to "pip install mpmath", the failure message comes out fairly prominantly (for tests\test_fp), but this works: PS issue2726> $env:JYTHON_OPTS="-Dpython.cpython2=C:\Python\2.7.15\python.exe" PS issue2726> pip install mpmath ... Successfully installed mpmath-1.1.0 ...
|msg12370 (view)||Author: Jeff Allen (jeff.allen)||Date: 2019-03-18.07:22:48|
Now claiming fixed at https://hg.python.org/jython/rev/96917c0883da Regression tests check that Stefan's technique it works when it should, and I haven't broken it, but not that it produces the right error messages in all circumstances. I've verified some manually. Tyre kicking by others would be welcome.
|msg12376 (view)||Author: Jeff Allen (jeff.allen)||Date: 2019-03-18.07:58:44|
Now fixed at https://hg.python.org/jython/rev/96917c0883da Or fixed "for now" as Stefan suggests above. But I will move it towards closed, and we can open another issue if it bites again.
|2019-03-18 07:58:44||jeff.allen||set||messages: + msg12376|
|2019-03-18 07:22:48||jeff.allen||set||status: open -> pending|
resolution: accepted -> fixed
messages: + msg12370
|2019-03-17 07:30:37||jeff.allen||set||assignee: jeff.allen|
messages: + msg12362
|2019-01-08 21:09:18||stefan.richthofer||set||messages: + msg12302|
|2019-01-08 20:17:56||jeff.allen||set||priority: normal -> low|
nosy: + stefan.richthofer
messages: + msg12298
milestone: Jython 2.7.2