Issue527524

Sourceforge won't let me create an account right now, 
sorry.

Attached is a 2500 line snippet from a 15,000 line 
unicode initialization table.  Commenting out the last 
executable line will let it execute, but the line 
itself does not seem to be the problem.

Using Jython 2.1 on java1.3.0_01 (JIT:null) on MSW 2000

--Th

Logged In: YES 
user_id=72446

Hi,

I'm new to SourceForge, I can't seem to find the file that
you are referring to, without it, it's hard to reproduce the
problem.

Pedro

Logged In: NO 

I have been able to recreate this problem using my own script 
file that is around 2500 lines in length.

When I cut the script down to a smaller size it works, as the 
size increases it fails.

Logged In: YES 
user_id=759503

I have a file that recreates this problem, but I can't seem to 
attach it now.

Logged In: YES 
user_id=117665

9 months later, no code.  Closing it.

Logged In: NO 

You should be able to recreate this bug with a py file of 
significant size.

This is a serious scaling limitation to python, I suggest you fix 
the issue and not close it.

Logged In: YES 
user_id=117665

I have now found a problem with large files, but I don't
know if it is the same as the original bug or not.

Steps to do what I've done:
- Increase heap space available to Jython, by modifying
'jython' script to include -Xmx256m -Xms256m 

- Save the 'createBigFile.py' (attached) to some directory
with lots of space.

- jython createBigFile.py (creates big.py)

- jython big.py
Traceback (innermost last):
  (no code object) at line 0
java.lang.ClassFormatError: org/python/pycode/_pyx0 (Code of
a method longer than 65535 bytes)
	at java.lang.ClassLoader.defineClass0(Native Method)
	at java.lang.ClassLoader.defineClass(Unknown Source)
	at
org.python.core.BytecodeLoader2.loadClassFromBytes(BytecodeLoader2.java)
	at
org.python.core.BytecodeLoader.makeClass(BytecodeLoader.java)
	at org.python.core.BytecodeLoader.makeCode(BytecodeLoader.java)
	at org.python.core.Py.compile_flags(Py.java)
	at org.python.core.Py.compile_flags(Py.java)
	at org.python.core.__builtin__.execfile_flags(__builtin__.java)
	at
org.python.util.PythonInterpreter.execfile(PythonInterpreter.java)
	at org.python.util.jython.main(jython.java)

java.lang.ClassFormatError: java.lang.ClassFormatError:
org/python/pycode/_pyx0 (Code of a method longer than 65535
bytes)

Note that this is partly a Java limitation, and partly a
"mapping Python code to Java" issue.

Is this the error that was intended with this bug?

Logged In: NO 

The exception trace on createBigFile.py isn't the same thing I 
saw.  This exception looks like there is a method that is very 
large.  What I saw was a parsing problem whith a py file 
around 2500 lines long, that at a certain point the python 
parser loses infomation (like it is cut off at a certain point).  
So the parse fails due to the cut off.

Logged In: YES 
user_id=117665

Can you attach a file, include any exceptions, etc.?

You can also email me a file to my username
@users.sourceforge.net

The problem appears to be that that the method executing exceeds the size constraints of the ClasFile format for a method.  In the VM spec, this is stated as a maximum size of 65536, which coincides with the error message.  Looking at this from a Java 6, this limit appears to have grown to: 99877

which Im going to assume is because of jsr 202:
http://www.jcp.org/en/jsr/detail?id=202

It would be nice to find info of the max this has been changed to.

leouser

This bug seems to be still there by 2009.
This is a blocker preventing to load the best oss ERP - OpenERP - on
Jython cause it uses the reportlab reporting engine and reportlab sets
up large table of char sequences when loading.
Included is the file causing the problem for me. I could verify that
this problem is caused by large lists too.
Indeed, I trimmed all the datastructures intinialized in the _fondata.py
and then it loaded (not attached ws just a test).

The command leading to a crash (either from command line or in OpenERP) is:

from reportlab.platypus import SimpleDocTemplate

of course to try this you should put the reportlab package in your path,
either the last 2.2 release, either trunk from here:
http://www.reportlab.org/downloads.html

The the shown command will result in that stack trace:

>>> from reportlab.platypus import SimpleDocTemplate
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File
"/home/rvalyi/DEV/jython/dist/Lib/reportlab/platypus/__init__.py", line
10, in <module>
    from reportlab.platypus.paragraph import Paragraph,
cleanBlockQuotedText, ParaLines
  File
"/home/rvalyi/DEV/jython/dist/Lib/reportlab/platypus/paragraph.py", line
9, in <module>
    from reportlab.pdfbase.pdfmetrics import stringWidth, getFont,
getAscentDescent
  File
"/home/rvalyi/DEV/jython/dist/Lib/reportlab/pdfbase/pdfmetrics.py", line
23, in <module>
    from reportlab.pdfbase import _fontdata
java.lang.ClassFormatError: Invalid method Code length 66082 in class
file reportlab/pdfbase/_fontdata$py
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:642)
	at
org.python.core.BytecodeLoader$Loader.loadClassFromBytes(BytecodeLoader.java:116)
	at org.python.core.BytecodeLoader.makeClass(BytecodeLoader.java:35)
	at org.python.core.BytecodeLoader.makeCode(BytecodeLoader.java:65)
	at org.python.core.imp.createFromSource(imp.java:231)
	at org.python.core.imp.loadFromSource(imp.java:448)
	at org.python.core.imp.find_module(imp.java:354)
	at org.python.core.PyModule.impAttr(PyModule.java:109)
	at org.python.core.imp.import_next(imp.java:554)
	at org.python.core.imp.import_name(imp.java:682)
	at org.python.core.imp.importName(imp.java:712)
	at org.python.core.ImportFunction.__call__(__builtin__.java:1265)
	at org.python.core.PyObject.__call__(PyObject.java:337)
	at org.python.core.__builtin__.__import__(__builtin__.java:1236)
	at org.python.core.imp.importFromAs(imp.java:788)
	at org.python.core.imp.importFrom(imp.java:765)
	at
reportlab.pdfbase.pdfmetrics$py.f$0(/home/rvalyi/DEV/jython/dist/Lib/reportlab/pdfbase/pdfmetrics.py:822)
	at
reportlab.pdfbase.pdfmetrics$py.call_function(/home/rvalyi/DEV/jython/dist/Lib/reportlab/pdfbase/pdfmetrics.py)
	at org.python.core.PyTableCode.call(PyTableCode.java:199)
	at org.python.core.PyCode.call(PyCode.java:14)
	at org.python.core.imp.createFromCode(imp.java:269)
	at org.python.core.imp.createFromPyClass(imp.java:117)
	at org.python.core.imp.loadFromSource(imp.java:441)
	at org.python.core.imp.find_module(imp.java:354)
	at org.python.core.PyModule.impAttr(PyModule.java:109)
	at org.python.core.imp.import_next(imp.java:554)
	at org.python.core.imp.import_logic(imp.java:611)
	at org.python.core.imp.import_name(imp.java:667)
	at org.python.core.imp.importName(imp.java:712)
	at org.python.core.ImportFunction.__call__(__builtin__.java:1265)
	at org.python.core.PyObject.__call__(PyObject.java:337)
	at org.python.core.__builtin__.__import__(__builtin__.java:1236)
	at org.python.core.imp.importFromAs(imp.java:788)
	at org.python.core.imp.importFrom(imp.java:765)
	at
reportlab.platypus.paragraph$py.f$0(/home/rvalyi/DEV/jython/dist/Lib/reportlab/platypus/paragraph.py:1626)
	at
reportlab.platypus.paragraph$py.call_function(/home/rvalyi/DEV/jython/dist/Lib/reportlab/platypus/paragraph.py)
	at org.python.core.PyTableCode.call(PyTableCode.java:199)
	at org.python.core.PyCode.call(PyCode.java:14)
	at org.python.core.imp.createFromCode(imp.java:269)
	at org.python.core.imp.createFromPyClass(imp.java:117)
	at org.python.core.imp.loadFromSource(imp.java:441)
	at org.python.core.imp.find_module(imp.java:354)
	at org.python.core.PyModule.impAttr(PyModule.java:109)
	at org.python.core.imp.import_next(imp.java:554)
	at org.python.core.imp.import_logic(imp.java:611)
	at org.python.core.imp.import_name(imp.java:667)
	at org.python.core.imp.importName(imp.java:712)
	at org.python.core.ImportFunction.__call__(__builtin__.java:1265)
	at org.python.core.PyObject.__call__(PyObject.java:337)
	at org.python.core.__builtin__.__import__(__builtin__.java:1236)
	at org.python.core.imp.importFromAs(imp.java:788)
	at org.python.core.imp.importFrom(imp.java:765)
	at
reportlab.platypus$py.f$0(/home/rvalyi/DEV/jython/dist/Lib/reportlab/platypus/__init__.py:16)
	at
reportlab.platypus$py.call_function(/home/rvalyi/DEV/jython/dist/Lib/reportlab/platypus/__init__.py)
	at org.python.core.PyTableCode.call(PyTableCode.java:199)
	at org.python.core.PyCode.call(PyCode.java:14)
	at org.python.core.imp.createFromCode(imp.java:269)
	at org.python.core.imp.createFromPyClass(imp.java:117)
	at org.python.core.imp.loadFromSource(imp.java:441)
	at org.python.core.imp.find_module(imp.java:354)
	at org.python.core.PyModule.impAttr(PyModule.java:109)
	at org.python.core.imp.import_next(imp.java:554)
	at org.python.core.imp.import_logic(imp.java:611)
	at org.python.core.imp.import_name(imp.java:667)
	at org.python.core.imp.importName(imp.java:712)
	at org.python.core.ImportFunction.__call__(__builtin__.java:1265)
	at org.python.core.PyObject.__call__(PyObject.java:337)
	at org.python.core.__builtin__.__import__(__builtin__.java:1236)
	at org.python.core.imp.importFromAs(imp.java:788)
	at org.python.core.imp.importFrom(imp.java:765)
	at org.python.pycode._pyx23.f$0(<stdin>:1)
	at org.python.pycode._pyx23.call_function(<stdin>)
	at org.python.core.PyTableCode.call(PyTableCode.java:199)
	at org.python.core.PyCode.call(PyCode.java:14)
	at org.python.core.Py.runCode(Py.java:1206)
	at org.python.core.Py.exec(Py.java:1237)
	at org.python.util.PythonInterpreter.exec(PythonInterpreter.java:133)
	at
org.python.util.InteractiveInterpreter.runcode(InteractiveInterpreter.java:90)
	at
org.python.util.InteractiveInterpreter.runsource(InteractiveInterpreter.java:71)
	at
org.python.util.InteractiveInterpreter.runsource(InteractiveInterpreter.java:46)
	at org.python.util.InteractiveConsole.push(InteractiveConsole.java:110)
	at org.python.util.InteractiveConsole.interact(InteractiveConsole.java:90)
	at org.python.util.jython.run(jython.java:293)
	at org.python.util.jython.main(jython.java:113)

java.lang.ClassFormatError: java.lang.ClassFormatError: Invalid method
Code length 66082 in class file reportlab/pdfbase/_fontdata$py
>>> 


Hope this helps.

Guys this is really an annoying bug for us, I see no workaround, so if
you could fix it, that would just help having a super great oss ERP on
the Java platform.

Raphaël Valyi.

This will be fixed by the addition of the Python bytecode VM, which has
no such limitations, at the cost of lower performance compared to Java
bytecode.

The PBCVM work can be tracked here:
https://jython.svn.sourceforge.net/svnroot/jython/branches/pbcvm/

This includes methods that represent the top-level of the module itself.

Deferred to 2.5.1

I am removing Kevin Butler from the nosy list because mails to his
account are bouncing. I sent him a private mail some time back about
this but there was no response  - Raghu.

Back in March 2009 on msg4240 Jim Baker said this issue would be
deferred to 2.5.1, which has since been released (without a fix).

Is there a new planned milestone?

This is of interest to me in getting Biopython to work nicely under
Jython (excluding those bits needing C code and/or NumPy). While we may
be able to work around this JVM size limit, it is frustrating.

re msg5318 by Peter:
2.5.1 proved to be a smaller increment for us. I have changed the 
version to being Deferred for the time being.

Here's what needs to be implemented:
a compiler that given an AST, produces Python bytecode

This would support many things for Jython, not just long code bodies.

Note that you can work around this problem today by compiling your code 
with CPython to .pyc files, perhaps with the compileall module, then 
using "import pycimport" to enable executing .pyc code by the Python 
bytecode VM. Incidentally, this capability has been available, quietly, 
since 2.5.0. While not an ideal solution, this toolchain is certainly 
doable. For instance, I used it to test the PBC-VM against regrtest.

I should mention, the workaround is briefly documented in the Jython book:
http://www.jython.org/jythonbook/en/1.0/ModulesPackages.html#sys-meta-path

And you can see its usage in Lib/test/test_pbcvm.py. I should make the change noted in the comments to use more of regrtest via subprocess.

Fix with a Python bytecode compiler. We probably want to emit such bytecode in $py.class files however, at least by default.

I implemented Jim's suggestion in github.com/jythontools/jython/commit/2c0dda9be0b7bb9f094960a251cbe93c1dd170aa. Now Jython scans for an appropriate CPython 2.7 pyc-file and embeds the required bytecode from there into the generated class-file. This should be sufficient to state that this issue will hardly bother any more.

What still needs to be done is to let Jython generate the pyc-file itself. However that's a major effort and I'd suggest to directly target Jython 3 for such a project.

Somehow this process is broken on Windows. I am getting ValueError: bad marshal data. It however works fine on Windows if I use classfiles I created on Linux (i.e. with embedded CPython bytecode from a pyc-file). So the write-process must have some platform-dependent flaw.

Will investigate...


Jython 2.7.1b3 (, Feb 21 2017, 21:22:24)
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java1.8.0_121
Type "help", "copyright", "credits" or "license" for more information.
>>> import sympy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\workspace\Jython\lib_test\dist\Lib\site-packages\sympy\__init__.py", line 49, in <module>
    from .polys import *
  File "D:\workspace\Jython\lib_test\dist\Lib\site-packages\sympy\polys\__init__.py", line 21, in <module>
    from . import numberfields
  File "D:\workspace\Jython\lib_test\dist\Lib\site-packages\sympy\polys\numberfields.py", line 25, in <module>
    from sympy.polys.rootoftools import CRootOf
  File "D:\workspace\Jython\lib_test\dist\Lib\site-packages\sympy\polys\rootoftools.py", line 19, in <module>
    from sympy.polys.polyroots import (
  File "D:\workspace\Jython\lib_test\dist\Lib\site-packages\sympy\polys\polyroots.py", line 26, in <module>
    from sympy.polys.polyquinticconst import PolyQuintic
ValueError: bad marshal data
>>>

Okay, fixed it as of https://hg.python.org/jython/rev/5861112d6d39.
pyc-file was opened with 'r' mode, but must be 'rb' to work properly also on Windows. A shame how long this one-letter-fix took me to find :/

As of https://hg.python.org/jython/rev/03f4808038f8 you can run things like

pip install --global-option="-J-Dcpython_cmd=python" sympy

on Jython (try it!). Jython will use the provided cpython 2.7 command to automatically create CPython bytecode for oversized functions as needed. It will build  workable class-files for sympy (or whatever package you install) without bothering you again about pyc-files.