Issue2579

java.lang.RuntimeException: java.lang.RuntimeException:
Encountered too large method code in
__pyclasspath__/ucsmsdk/ucsmeta.py

Please provide a CPython 2.7 bytecode file (.pyc) to proceed, e.g. run
python -m py_compile __pyclasspath__/ucsmsdk/ucsmeta.py
and try again.

Alternatively provide proper CPython 2.7 execute command via
cpython_cmd property, e.g. call
    jython -J-Dcpython_cmd=python
or if running pip on Jython:
    pip install --global-option="-J-Dcpython_cmd=python" <package>

        at org.python.core.Py.JavaError(Py.java:548)
        at org.python.core.ParserFacade.fixParseEr



Python Exception..
Traceback (most recent call last):
  File "<string>", line 24, in <module>
  File "__pyclasspath__/ucsmsdk/ucshandle.py", line 20, in <module>
  File "__pyclasspath__/ucsmsdk/ucscoreutils.py", line 27, in <module>
ImportError: No module named ucsmsdk.ucsmeta

        at org.python.core.Py.ImportError(Py.java:330)
        at org.python.core.imp.import_first(imp.java:871)
        at org.python.core.imp.import_module_level(imp.java:957)
        at org.python.core.imp.importName(imp.java:1043)
        at org.python.core.ImportFunction.__call__(__builtin__.java:1280)
        at org.python.core.PyObject.__call__(PyObject.java:450)
        at org.python.core.__builtin__.__import__(__builtin__.java:1232)
        at org.python.core.imp.importFromAs(imp.java:1135)
        at org.python.core.imp.importFrom(imp.java:1110)
        at ucsmsdk.ucscoreutils$py.f$0(__pyclasspath__/ucsmsdk/ucscoreutils.py:638)
        at ucsmsdk.ucscoreutils$py.call_function(__pyclasspath__/ucsmsdk/ucscoreutils.py)
        at org.python.core.PyTableCode.call(PyTableCode.java:167)

I am using jython standalone jar with my java module..
i was facing an issue with loading external modules with bigger size with 2.7. As jvm bytecode limit is only 65kB.
i learnt from IRC /chat forum that with latest code base this issue is fixed. and been advised to take latest code and build the jar. and procide pyc files build from CPython 2.7. I did so and i found that it is not able to locate the pyc files.

Parisa, when the path of the py-file contains __pyclasspath__, Jython doesn't track its exact position, but just knows it's somewhere on the classpath. The search mechanism for pyc files currently doesn't consider that and breaks in such a case.

I implemented a possible fix for this: https://github.com/Stewori/jython/commit/343740e5a34afd90f4953ba0eab18ccf863da860

But so far I find it tedious to test.

So it would be nice if you could try it, e.g. by cloning my Jython fork at https://github.com/Stewori/jython

If that solves the problem I will merge it into the main repo.

Hi Stefan,
Thanks a lot for quick update...
I will test and update the same...

Hi Stefan,
Unable to import with latest code from your git branch i tried to import a module. Please find error below... 

ImportError: No module named ucsmsdk.ucsmeta

        at org.python.core.Py.ImportError(Py.java:330)
        at org.python.core.imp.import_first(imp.java:871)
        at org.python.core.imp.import_module_level(imp.java:957)
        at org.python.core.imp.importName(imp.java:1043)
        at org.python.core.ImportFunction.__call__(__builtin__.java:1280)
        at org.python.core.PyObject.__call__(PyObject.java:450)
        at org.python.core.__builtin__.__import__(__builtin__.java:1232)
        at org.python.core.imp.importFromAs(imp.java:1135)
        at org.python.core.imp.importFrom(imp.java:1110)
        at ucsmsdk.ucscoreutils$py.f$0(__pyclasspath__/ucsmsdk/ucscoreutils.py:638)
        at ucsmsdk.ucscoreutils$py.call_function(__pyclasspath__/ucsmsdk/ucscoreutils.py)
        at org.python.core.PyTableCode.call(PyTableCode.java:167)
        at org.python.core.PyCode.call(PyCode.java:18)
        at org.python.core.imp.createFromCode(imp.java:432)
        at org.python.core.util.importer.importer_load_module(importer.java:116)
        at org.python.core.ClasspathPyImporter.ClasspathPyImporter_load_module(ClasspathPyImporter.java:170)
        at org.python.core.ClasspathPyImporter$ClasspathPyImporter_load_module_exposer.__call__(Unknown Source)
        at org.python.core.PyBuiltinMethodNarrow.__call__(PyBuiltinMethodNarrow.java:46)
        at org.python.core.imp.loadFromLoader(imp.java:591)
        at org.python.core.imp.find_module(imp.java:543)
        at org.python.core.PyModule.impAttr(PyModule.java:111)
        at org.python.core.PyModule.__findattr_ex__(PyModule.java:140)
        at org.python.core.PyObject.__findattr__(PyObject.java:965)
        at org.python.core.PyObject.__findattr__(PyObject.java:950)
        at org.python.core.imp.ensureFromList(imp.java:1004)
        at org.python.core.imp.ensureFromList(imp.java:978)
        at org.python.core.imp.import_module_level(imp.java:972)
        at org.python.core.imp.importName(imp.java:1043)
        at org.python.core.ImportFunction.__call__(__builtin__.java:1280)
        at org.python.core.PyObject.__call__(PyObject.java:450)
        at org.python.core.__builtin__.__import__(__builtin__.java:1232)
at org.python.core.PyTableCode.call(PyTableCode.java:167)
        at org.python.core.PyCode.call(PyCode.java:18)
        at org.python.core.imp.createFromCode(imp.java:432)
        at org.python.core.util.importer.importer_load_module(importer.java:116)
        at org.python.core.ClasspathPyImporter.ClasspathPyImporter_load_module(ClasspathPyImporter.java:170)
        at org.python.core.ClasspathPyImporter$ClasspathPyImporter_load_module_exposer.__call__(Unknown Source)
        at org.python.core.PyBuiltinMethodNarrow.__call__(PyBuiltinMethodNarrow.java:46)
        at org.python.core.imp.loadFromLoader(imp.java:591)
        at org.python.core.imp.find_module(imp.java:543)
        at org.python.core.PyModule.impAttr(PyModule.java:111)
        at org.python.core.imp.import_next(imp.java:837)
        at org.python.core.imp.import_logic(imp.java:897)
        at org.python.core.imp.import_module_level(imp.java:965)
        at org.python.core.imp.importName(imp.java:1043)
        at org.python.core.ImportFunction.__call__(__builtin__.java:1280)
        at org.python.core.PyObject.__call__(PyObject.java:450)
        at org.python.core.__builtin__.__import__(__builtin__.java:1232)
        at org.python.core.imp.importFromAs(imp.java:1135)
        at org.python.core.imp.importFrom(imp.java:1110)
        at org.python.pycode._pyx0.f$0(<string>:124)
        at org.python.pycode._pyx0.call_function(<string>)
        at org.python.core.PyTableCode.call(PyTableCode.java:167)
        at org.python.core.PyCode.call(PyCode.java:18)
        at org.python.core.Py.runCode(Py.java:1401)
        at org.python.core.Py.exec(Py.java:1445)
        at org.python.util.PythonInterpreter.exec(PythonInterpreter.java:267)
        at com.pari.jelly.engine.JythonEngine$2.call(JythonEngine.java:201)
        at com.pari.jelly.engine.JythonEngine$2.call(JythonEngine.java:151)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Parisa,

could you please specify the command you use for launching?
What is the classpath configuration?
What is the actual location of ucsmsdk/ucsmeta.py?
It's okay to provide paths in an abstract sense, I just need this to reproduce your setup in terms of working directory, classpath, location of your modules and import hierarchy.

As a potential workaround, please try to add the absolute path of ucsmsdk/ucsmeta.py to sys.path (before the relevant import statement); put it at the beginning, e.g. via sys.path.insert(0, 'whatever'). This should avoid __pyclasspath__ to occur in the path.

Hi Stefan,
As per your suggestion with your jar when i provide absolute path it is able to load the module from pyc file.
But in production my modules and pyc files will be added to class path.
So internally jython should be able to load pyc file from path relative to its launch path.

Also if bigger module does not have pyc file its throwing below exeption


java.lang.NullPointerException

	at org.python.compiler.Module.loadPyBytecode(Module.java:740)

	at org.python.compiler.Module.compile(Module.java:925)

	at org.python.core.imp.compileSource(imp.java:367)

	at org.python.core.imp.compileSource(imp.java:352)

	at org.python.core.util.importer.getModuleCode(importer.java:229)

	at org.python.core.util.importer.importer_load_module(importer.java:102)

	at org.python.core.ClasspathPyImporter.ClasspathPyImporter_load_module(ClasspathPyImporter.java:170)

	at org.python.core.ClasspathPyImporter$ClasspathPyImporter_load_module_exposer.__call__(Unknown Source)

	at org.python.core.PyBuiltinMethodNarrow.__call__(PyBuiltinMethodNarrow.java:46)

	at org.python.core.imp.loadFromLoader(imp.java:591)

	at org.python.core.imp.find_module(imp.java:543)

	at org.python.core.PyModule.impAttr(PyModule.java:111)

	at org.python.core.imp.import_next(imp.java:837)

	at org.python.core.imp.import_module_level(imp.java:952)

	at org.python.core.imp.importName(imp.java:1043)

	at org.python.core.ImportFunction.__call__(__builtin__.java:1280)

	at org.python.core.PyObject.__call__(PyObject.java:450)

	at org.python.core.__builtin__.__import__(__builtin__.java:1232)

	at org.python.core.imp.importFromAs(imp.java:1135)

	at org.python.core.imp.importFrom(imp.java:1110)

	at ucsmsdk.ucscoreutils$py.f$0(__pyclasspath__/ucsmsdk/ucscoreutils.py:638)

	at ucsmsdk.ucscoreutils$py.call_function(__pyclasspath__/ucsmsdk/ucscoreutils.py)

	at org.python.core.PyTableCode.call(PyTableCode.java:167)

	at org.python.core.PyCode.call(PyCode.java:18)

	at org.python.core.imp.createFromCode(imp.java:432)

	at org.python.core.util.importer.importer_load_module(importer.java:116)

	at org.python.core.ClasspathPyImporter.ClasspathPyImporter_load_module(ClasspathPyImporter.java:170)

	at org.python.core.ClasspathPyImporter$ClasspathPyImporter_load_module_exposer.__call__(Unknown Source)

	at org.python.core.PyBuiltinMethodNarrow.__call__(PyBuiltinMethodNarrow.java:46)

	at org.python.core.imp.loadFromLoader(imp.java:591)

	at org.python.core.imp.find_module(imp.java:543)

	at org.python.core.PyModule.impAttr(PyModule.java:111)

	at org.python.core.PyModule.__findattr_ex__(PyModule.java:140)

	at org.python.core.PyObject.__findattr__(PyObject.java:965)

	at org.python.core.PyObject.__findattr__(PyObject.java:950)

	at org.python.core.imp.ensureFromList(imp.java:1004)

	at org.python.core.imp.ensureFromList(imp.java:978)

	at org.python.core.imp.import_module_level(imp.java:972)

	at org.python.core.imp.importName(imp.java:1043)

	at org.python.core.ImportFunction.__call__(__builtin__.java:1280)

	at org.python.core.PyObject.__call__(PyObject.java:450)

	at org.python.core.__builtin__.__import__(__builtin__.java:1232)

	at org.python.core.imp.importFromAs(imp.java:1135)

	at org.python.core.imp.importFrom(imp.java:1110)

	at ucsmsdk.ucshandle$py.f$0(__pyclasspath__/ucsmsdk/ucshandle.py:29)

	at ucsmsdk.ucshandle$py.call_function(__pyclasspath__/ucsmsdk/ucshandle.py)

	at org.python.core.PyTableCode.call(PyTableCode.java:167)

	at org.python.core.PyCode.call(PyCode.java:18)

	at org.python.core.imp.createFromCode(imp.java:432)

	at org.python.core.util.importer.importer_load_module(importer.java:116)

	at org.python.core.ClasspathPyImporter.ClasspathPyImporter_load_module(ClasspathPyImporter.java:170)

	at org.python.core.ClasspathPyImporter$ClasspathPyImporter_load_module_exposer.__call__(Unknown Source)

	at org.python.core.PyBuiltinMethodNarrow.__call__(PyBuiltinMethodNarrow.java:46)

	at org.python.core.imp.loadFromLoader(imp.java:591)

	at org.python.core.imp.find_module(imp.java:543)

	at org.python.core.PyModule.impAttr(PyModule.java:111)

	at org.python.core.imp.import_next(imp.java:837)

	at org.python.core.imp.import_logic(imp.java:897)

	at org.python.core.imp.import_module_level(imp.java:965)

	at org.python.core.imp.importName(imp.java:1043)

	at org.python.core.ImportFunction.__call__(__builtin__.java:1280)

	at org.python.core.PyObject.__call__(PyObject.java:450)

	at org.python.core.__builtin__.__import__(__builtin__.java:1232)

	at org.python.core.imp.importFromAs(imp.java:1135)

	at org.python.core.imp.importFrom(imp.java:1110)

	at org.python.pycode._pyx0.f$0(<string>:870)

	at org.python.pycode._pyx0.call_function(<string>)

	at org.python.core.PyTableCode.call(PyTableCode.java:167)

	at org.python.core.PyCode.call(PyCode.java:18)

	at org.python.core.Py.runCode(Py.java:1401)

	at org.python.core.Py.exec(Py.java:1445)

	at org.python.util.PythonInterpreter.exec(PythonInterpreter.java:267)

> But in production my modules and pyc files will be added to class path.

Once a class file for an oversize-issued module has been successfully created (e.g. ucsmeta$py.class), the pyc files are not required any more; the CPython bytecode gets embedded into the classfile. So you only need to distribute classfiles. Under some circumstances Jython does not write the classfiles to disc. In that case you can enforce classfile creation as follows:

Write a script (e.g. compile_ucsmeta.py) like

import compileall, sys
sys.path.append('dir_containing_oversize_issued_module') # so it's workable with current pyc-file search mechanism
compileall.compile_dir('dir_containing_oversize_issued_module')

Run it e.g. like this
java -cp .:jython-standalone.jar:your_classpath org.python.util.jython


That said, I agree we need a proper fix for this issue.
I improved the previous attempt in
https://github.com/Stewori/jython/commit/9d45e3979ad82f86a65a46e33d80ca3fa7261f4d

and successfully tested it as far as I was able to reproduce your scenario. Please test my fork at
https://github.com/Stewori/jython
again regarding this issue.
Especially the NullPointerException if no pyc-file exists was fixed; now it should print the usual instructions.
It should adjust the filename to a version not containing __pyclasspath__, but the actual path.

Regarding your result with
ImportError: No module named ucsmsdk.ucsmeta

I get output like this if classpath is not properly set. If you continue to see this, please sketch your import hierarchy w.r.t classpath configuration, so I can reproduce.

Stefan,
With your changes, I am able to load large modules now.
Thanks a lot.
Please let me know when this will be available in main branch https://github.com/jythontools/jython

Thank you

Fixed as of https://hg.python.org/jython/rev/c382818607a0.

@Parisa
> Thanks a lot.
Happy to :)