Issue527524

classification
Title: Cannot compile to use methods exceeding JVM size restrictions
Type: Severity: normal
Components: Core Versions: Jython 2.7
process
Status: open Resolution: remind
Dependencies: Superseder:
Assigned To: zyasoft Nosy List: amak, draghuram, ericmoore, fwierzbicki, leouserz, nobody, p2sam, pjac, rvalyi, zyasoft
Priority: low Keywords:

Created on 2002-03-08.20:01:28 by anonymous, last changed 2013-03-01.00:21:06 by amak.

Files
File name Uploaded Description Edit Remove
createBigFile.py kevinbutler, 2003-10-23.23:25:17 Create a big file to demonstrate this failure
_fontdata.py rvalyi, 2009-01-29.23:23:57
Messages
msg611 (view) Author: Nobody/Anonymous (nobody) Date: 2002-03-08.20:01:28
Sourceforge won't let me create an account right now, 
sorry.

Attached is a 2500 line snippet from a 15,000 line 
unicode initialization table.  Commenting out the last 
executable line will let it execute, but the line 
itself does not seem to be the problem.

Using Jython 2.1 on java1.3.0_01 (JIT:null) on MSW 2000

--Th
msg612 (view) Author: Pedro Sam (p2sam) Date: 2003-04-25.12:18:23
Logged In: YES 
user_id=72446

Hi,

I'm new to SourceForge, I can't seem to find the file that
you are referring to, without it, it's hard to reproduce the
problem.

Pedro
msg613 (view) Author: Nobody/Anonymous (nobody) Date: 2003-04-25.17:13:34
Logged In: NO 

I have been able to recreate this problem using my own script 
file that is around 2500 lines in length.

When I cut the script down to a smaller size it works, as the 
size increases it fails.
msg614 (view) Author: Eric Moore (ericmoore) Date: 2003-04-25.23:45:52
Logged In: YES 
user_id=759503

I have a file that recreates this problem, but I can't seem to 
attach it now.
msg615 (view) Author: Kevin J. Butler (kevinbutler) Date: 2003-10-23.22:15:27
Logged In: YES 
user_id=117665

9 months later, no code.  Closing it.
msg616 (view) Author: Nobody/Anonymous (nobody) Date: 2003-10-23.22:37:26
Logged In: NO 

You should be able to recreate this bug with a py file of 
significant size.

This is a serious scaling limitation to python, I suggest you fix 
the issue and not close it.
msg617 (view) Author: Kevin J. Butler (kevinbutler) Date: 2003-10-23.23:25:17
Logged In: YES 
user_id=117665

I have now found a problem with large files, but I don't
know if it is the same as the original bug or not.

Steps to do what I've done:
- Increase heap space available to Jython, by modifying
'jython' script to include -Xmx256m -Xms256m 

- Save the 'createBigFile.py' (attached) to some directory
with lots of space.

- jython createBigFile.py (creates big.py)

- jython big.py
Traceback (innermost last):
  (no code object) at line 0
java.lang.ClassFormatError: org/python/pycode/_pyx0 (Code of
a method longer than 65535 bytes)
	at java.lang.ClassLoader.defineClass0(Native Method)
	at java.lang.ClassLoader.defineClass(Unknown Source)
	at
org.python.core.BytecodeLoader2.loadClassFromBytes(BytecodeLoader2.java)
	at
org.python.core.BytecodeLoader.makeClass(BytecodeLoader.java)
	at org.python.core.BytecodeLoader.makeCode(BytecodeLoader.java)
	at org.python.core.Py.compile_flags(Py.java)
	at org.python.core.Py.compile_flags(Py.java)
	at org.python.core.__builtin__.execfile_flags(__builtin__.java)
	at
org.python.util.PythonInterpreter.execfile(PythonInterpreter.java)
	at org.python.util.jython.main(jython.java)

java.lang.ClassFormatError: java.lang.ClassFormatError:
org/python/pycode/_pyx0 (Code of a method longer than 65535
bytes)

Note that this is partly a Java limitation, and partly a
"mapping Python code to Java" issue.

Is this the error that was intended with this bug?
msg618 (view) Author: Nobody/Anonymous (nobody) Date: 2003-10-23.23:36:47
Logged In: NO 

The exception trace on createBigFile.py isn't the same thing I 
saw.  This exception looks like there is a method that is very 
large.  What I saw was a parsing problem whith a py file 
around 2500 lines long, that at a certain point the python 
parser loses infomation (like it is cut off at a certain point).  
So the parse fails due to the cut off.
msg619 (view) Author: Kevin J. Butler (kevinbutler) Date: 2003-10-24.00:38:37
Logged In: YES 
user_id=117665

Can you attach a file, include any exceptions, etc.?

You can also email me a file to my username
@users.sourceforge.net
msg620 (view) Author: Deleted User leouserz (leouserz) Date: 2006-12-21.15:37:54
The problem appears to be that that the method executing exceeds the size constraints of the ClasFile format for a method.  In the VM spec, this is stated as a maximum size of 65536, which coincides with the error message.  Looking at this from a Java 6, this limit appears to have grown to: 99877

which Im going to assume is because of jsr 202:
http://www.jcp.org/en/jsr/detail?id=202

It would be nice to find info of the max this has been changed to.

leouser
msg4109 (view) Author: Raphaël Valyi (rvalyi) Date: 2009-01-29.23:23:56
This bug seems to be still there by 2009.
This is a blocker preventing to load the best oss ERP - OpenERP - on
Jython cause it uses the reportlab reporting engine and reportlab sets
up large table of char sequences when loading.
Included is the file causing the problem for me. I could verify that
this problem is caused by large lists too.
Indeed, I trimmed all the datastructures intinialized in the _fondata.py
and then it loaded (not attached ws just a test).

The command leading to a crash (either from command line or in OpenERP) is:

from reportlab.platypus import SimpleDocTemplate

of course to try this you should put the reportlab package in your path,
either the last 2.2 release, either trunk from here:
http://www.reportlab.org/downloads.html

The the shown command will result in that stack trace:

>>> from reportlab.platypus import SimpleDocTemplate
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File
"/home/rvalyi/DEV/jython/dist/Lib/reportlab/platypus/__init__.py", line
10, in <module>
    from reportlab.platypus.paragraph import Paragraph,
cleanBlockQuotedText, ParaLines
  File
"/home/rvalyi/DEV/jython/dist/Lib/reportlab/platypus/paragraph.py", line
9, in <module>
    from reportlab.pdfbase.pdfmetrics import stringWidth, getFont,
getAscentDescent
  File
"/home/rvalyi/DEV/jython/dist/Lib/reportlab/pdfbase/pdfmetrics.py", line
23, in <module>
    from reportlab.pdfbase import _fontdata
java.lang.ClassFormatError: Invalid method Code length 66082 in class
file reportlab/pdfbase/_fontdata$py
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:642)
	at
org.python.core.BytecodeLoader$Loader.loadClassFromBytes(BytecodeLoader.java:116)
	at org.python.core.BytecodeLoader.makeClass(BytecodeLoader.java:35)
	at org.python.core.BytecodeLoader.makeCode(BytecodeLoader.java:65)
	at org.python.core.imp.createFromSource(imp.java:231)
	at org.python.core.imp.loadFromSource(imp.java:448)
	at org.python.core.imp.find_module(imp.java:354)
	at org.python.core.PyModule.impAttr(PyModule.java:109)
	at org.python.core.imp.import_next(imp.java:554)
	at org.python.core.imp.import_name(imp.java:682)
	at org.python.core.imp.importName(imp.java:712)
	at org.python.core.ImportFunction.__call__(__builtin__.java:1265)
	at org.python.core.PyObject.__call__(PyObject.java:337)
	at org.python.core.__builtin__.__import__(__builtin__.java:1236)
	at org.python.core.imp.importFromAs(imp.java:788)
	at org.python.core.imp.importFrom(imp.java:765)
	at
reportlab.pdfbase.pdfmetrics$py.f$0(/home/rvalyi/DEV/jython/dist/Lib/reportlab/pdfbase/pdfmetrics.py:822)
	at
reportlab.pdfbase.pdfmetrics$py.call_function(/home/rvalyi/DEV/jython/dist/Lib/reportlab/pdfbase/pdfmetrics.py)
	at org.python.core.PyTableCode.call(PyTableCode.java:199)
	at org.python.core.PyCode.call(PyCode.java:14)
	at org.python.core.imp.createFromCode(imp.java:269)
	at org.python.core.imp.createFromPyClass(imp.java:117)
	at org.python.core.imp.loadFromSource(imp.java:441)
	at org.python.core.imp.find_module(imp.java:354)
	at org.python.core.PyModule.impAttr(PyModule.java:109)
	at org.python.core.imp.import_next(imp.java:554)
	at org.python.core.imp.import_logic(imp.java:611)
	at org.python.core.imp.import_name(imp.java:667)
	at org.python.core.imp.importName(imp.java:712)
	at org.python.core.ImportFunction.__call__(__builtin__.java:1265)
	at org.python.core.PyObject.__call__(PyObject.java:337)
	at org.python.core.__builtin__.__import__(__builtin__.java:1236)
	at org.python.core.imp.importFromAs(imp.java:788)
	at org.python.core.imp.importFrom(imp.java:765)
	at
reportlab.platypus.paragraph$py.f$0(/home/rvalyi/DEV/jython/dist/Lib/reportlab/platypus/paragraph.py:1626)
	at
reportlab.platypus.paragraph$py.call_function(/home/rvalyi/DEV/jython/dist/Lib/reportlab/platypus/paragraph.py)
	at org.python.core.PyTableCode.call(PyTableCode.java:199)
	at org.python.core.PyCode.call(PyCode.java:14)
	at org.python.core.imp.createFromCode(imp.java:269)
	at org.python.core.imp.createFromPyClass(imp.java:117)
	at org.python.core.imp.loadFromSource(imp.java:441)
	at org.python.core.imp.find_module(imp.java:354)
	at org.python.core.PyModule.impAttr(PyModule.java:109)
	at org.python.core.imp.import_next(imp.java:554)
	at org.python.core.imp.import_logic(imp.java:611)
	at org.python.core.imp.import_name(imp.java:667)
	at org.python.core.imp.importName(imp.java:712)
	at org.python.core.ImportFunction.__call__(__builtin__.java:1265)
	at org.python.core.PyObject.__call__(PyObject.java:337)
	at org.python.core.__builtin__.__import__(__builtin__.java:1236)
	at org.python.core.imp.importFromAs(imp.java:788)
	at org.python.core.imp.importFrom(imp.java:765)
	at
reportlab.platypus$py.f$0(/home/rvalyi/DEV/jython/dist/Lib/reportlab/platypus/__init__.py:16)
	at
reportlab.platypus$py.call_function(/home/rvalyi/DEV/jython/dist/Lib/reportlab/platypus/__init__.py)
	at org.python.core.PyTableCode.call(PyTableCode.java:199)
	at org.python.core.PyCode.call(PyCode.java:14)
	at org.python.core.imp.createFromCode(imp.java:269)
	at org.python.core.imp.createFromPyClass(imp.java:117)
	at org.python.core.imp.loadFromSource(imp.java:441)
	at org.python.core.imp.find_module(imp.java:354)
	at org.python.core.PyModule.impAttr(PyModule.java:109)
	at org.python.core.imp.import_next(imp.java:554)
	at org.python.core.imp.import_logic(imp.java:611)
	at org.python.core.imp.import_name(imp.java:667)
	at org.python.core.imp.importName(imp.java:712)
	at org.python.core.ImportFunction.__call__(__builtin__.java:1265)
	at org.python.core.PyObject.__call__(PyObject.java:337)
	at org.python.core.__builtin__.__import__(__builtin__.java:1236)
	at org.python.core.imp.importFromAs(imp.java:788)
	at org.python.core.imp.importFrom(imp.java:765)
	at org.python.pycode._pyx23.f$0(<stdin>:1)
	at org.python.pycode._pyx23.call_function(<stdin>)
	at org.python.core.PyTableCode.call(PyTableCode.java:199)
	at org.python.core.PyCode.call(PyCode.java:14)
	at org.python.core.Py.runCode(Py.java:1206)
	at org.python.core.Py.exec(Py.java:1237)
	at org.python.util.PythonInterpreter.exec(PythonInterpreter.java:133)
	at
org.python.util.InteractiveInterpreter.runcode(InteractiveInterpreter.java:90)
	at
org.python.util.InteractiveInterpreter.runsource(InteractiveInterpreter.java:71)
	at
org.python.util.InteractiveInterpreter.runsource(InteractiveInterpreter.java:46)
	at org.python.util.InteractiveConsole.push(InteractiveConsole.java:110)
	at org.python.util.InteractiveConsole.interact(InteractiveConsole.java:90)
	at org.python.util.jython.run(jython.java:293)
	at org.python.util.jython.main(jython.java:113)

java.lang.ClassFormatError: java.lang.ClassFormatError: Invalid method
Code length 66082 in class file reportlab/pdfbase/_fontdata$py
>>> 


Hope this helps.

Guys this is really an annoying bug for us, I see no workaround, so if
you could fix it, that would just help having a super great oss ERP on
the Java platform.

Raphaël Valyi.
msg4112 (view) Author: Jim Baker (zyasoft) Date: 2009-01-30.14:38:07
This will be fixed by the addition of the Python bytecode VM, which has
no such limitations, at the cost of lower performance compared to Java
bytecode.

The PBCVM work can be tracked here:
https://jython.svn.sourceforge.net/svnroot/jython/branches/pbcvm/
msg4113 (view) Author: Jim Baker (zyasoft) Date: 2009-01-30.14:52:10
This includes methods that represent the top-level of the module itself.
msg4240 (view) Author: Jim Baker (zyasoft) Date: 2009-03-12.08:07:18
Deferred to 2.5.1
msg4247 (view) Author: Raghuram Devarakonda (draghuram) Date: 2009-03-12.13:17:18
I am removing Kevin Butler from the nosy list because mails to his
account are bouncing. I sent him a private mail some time back about
this but there was no response  - Raghu.
msg5318 (view) Author: Peter (pjac) Date: 2009-11-23.17:28:11
Back in March 2009 on msg4240 Jim Baker said this issue would be
deferred to 2.5.1, which has since been released (without a fix).

Is there a new planned milestone?

This is of interest to me in getting Biopython to work nicely under
Jython (excluding those bits needing C code and/or NumPy). While we may
be able to work around this JVM size limit, it is frustrating.
msg5320 (view) Author: Jim Baker (zyasoft) Date: 2009-11-23.17:41:57
re msg5318 by Peter:
2.5.1 proved to be a smaller increment for us. I have changed the 
version to being Deferred for the time being.

Here's what needs to be implemented:
a compiler that given an AST, produces Python bytecode

This would support many things for Jython, not just long code bodies.

Note that you can work around this problem today by compiling your code 
with CPython to .pyc files, perhaps with the compileall module, then 
using "import pycimport" to enable executing .pyc code by the Python 
bytecode VM. Incidentally, this capability has been available, quietly, 
since 2.5.0. While not an ideal solution, this toolchain is certainly 
doable. For instance, I used it to test the PBC-VM against regrtest.
msg6061 (view) Author: Jim Baker (zyasoft) Date: 2010-09-11.04:22:43
I should mention, the workaround is briefly documented in the Jython book:
http://www.jython.org/jythonbook/en/1.0/ModulesPackages.html#sys-meta-path

And you can see its usage in Lib/test/test_pbcvm.py. I should make the change noted in the comments to use more of regrtest via subprocess.
History
Date User Action Args
2013-03-01 00:21:06amaksetnosy: + amak
2013-02-19 18:55:47fwierzbickisetresolution: remind
versions: + Jython 2.7, - Deferred
2012-08-13 18:11:44fwierzbickisetnosy: + fwierzbicki
2010-09-11 04:22:44zyasoftsetmessages: + msg6061
2009-11-23 17:41:57zyasoftsetmessages: + msg5320
versions: + Deferred, - 2.5.1
2009-11-23 17:28:12pjacsetnosy: + pjac
messages: + msg5318
2009-03-12 13:17:19draghuramsetnosy: + draghuram, - kevinbutler
messages: + msg4247
2009-03-12 08:09:06zyasoftsetversions: + 2.5.1
2009-03-12 08:07:19zyasoftsetmessages: + msg4240
2009-01-30 14:52:11zyasoftsetmessages: + msg4113
title: Long list of initializations fails -> Cannot compile to use methods exceeding JVM size restrictions
2009-01-30 14:38:07zyasoftsetassignee: zyasoft
messages: + msg4112
nosy: + zyasoft
2009-01-29 23:24:00rvalyisetfiles: + _fontdata.py
nosy: + rvalyi
messages: + msg4109
2008-12-14 17:50:17fwierzbickisetcomponents: + Core, - None
2002-03-08 20:01:28anonymouscreate