Issue1744

classification
Title: jython script total characters limited to 100000
Type: behaviour Severity: normal
Components: None Versions: Jython 2.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: amak Nosy List: TheHappyRock, amak, fwierzbicki, jtrim, philba, reljicb, zyasoft
Priority: low Keywords:

Created on 2011-04-27.11:04:21 by reljicb, last changed 2014-06-19.07:06:14 by zyasoft.

Files
File name Uploaded Description Edit Remove
Main.java reljicb, 2011-04-27.11:04:20 test
Messages
msg6518 (view) Author: Bojan (reljicb) Date: 2011-04-27.11:04:20
I use Jython as a java ScriptEngine, and I load scripts longer then 100000 characters, which throws IOException:

javax.script.ScriptException: java.io.IOException: java.io.IOException: Mark invalid
        at org.python.jsr223.PyScriptEngine.scriptException(PyScriptEngine.java:196)
        at org.python.jsr223.PyScriptEngine.compileScript(PyScriptEngine.java:75)
        at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:31)
        at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:247)
        at testjython.Main.run(Main.java:36)
        at testjython.Main.main(Main.java:56)
Caused by: java.io.IOException: Mark invalid

java.io.IOException: java.io.IOException: Mark invalid

        at org.python.core.PyException.fillInStackTrace(PyException.java:70)
        at java.lang.Throwable.<init>(Throwable.java:181)
        at java.lang.Exception.<init>(Exception.java:29)
        at java.lang.RuntimeException.<init>(RuntimeException.java:32)
        at org.python.core.PyException.<init>(PyException.java:46)
        at org.python.core.PyException.<init>(PyException.java:43)
        at org.python.core.Py.JavaError(Py.java:481)
        at org.python.core.ParserFacade.fixParseError(ParserFacade.java:104)
        at org.python.core.ParserFacade.parseExpressionOrModule(ParserFacade.java:130)


Test ended.
        at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:259)
        at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:256)
        at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:250)
        at org.python.jsr223.PyScriptEngine.compileScript(PyScriptEngine.java:70)
        ... 4 more
Caused by: java.io.IOException: Mark invalid
        at java.io.BufferedReader.reset(BufferedReader.java:485)
        at org.python.core.ParserFacade.parseExpressionOrModule(ParserFacade.java:127)
        ... 8 more


I am not saying this is a bug, but it is certainly a behavior which limits me from executing not that big python scripts.

Is there a way to increase this limit?

Cheers,
Bojan

PS. See attached java file to reproduce the problem.
msg6654 (view) Author: Philipp Bach (philba) Date: 2011-09-28.11:35:17
I would also like to increase the MARK_LIMIT.
msg6857 (view) Author: Frank Wierzbicki (fwierzbicki) Date: 2012-03-19.20:16:15
We can't really increase the mark limit - it causes out of memory errors pretty quickly in our regression tests if it is large, which leads me to believe that it can be leaky. Routinely rejecting >100K files isn't what we want either - but I'm not sure of a way to fix this yet. The two uses we have are checking the very beginning of the file for unicode markers - these we could replace with a pushback inputstream (http://docs.oracle.com/javase/1.4.2/docs/api/java/io/PushbackInputStream.html)

 The harder part is printing lines for parse errors - we might still be able to use a pushback input stream for this too, but it would take some investigation. If anyone would like to try to submit a patch for this I'd be more than willing to review it.
msg7532 (view) Author: Frank (TheHappyRock) Date: 2012-11-21.04:23:30
Just wanted to keep this alive as it is a big deal for programs that create large Jython scripts.

I don't know the Jython code base at all, but hopefully this catches someone's eye that does.
msg7724 (view) Author: Alan Kennedy (amak) Date: 2013-02-20.09:48:18
Frank, perhaps we could expose the MARK_LIMIT as a registry property, and give the users control over it?

http://hg.python.org/jython/file/a30708945630/src/org/python/core/ParserFacade.java#l40

We could still default it to 100,000 if no explicit value is specified.

That way, the user could set it higher if required, and they have the memory to support it.
msg7751 (view) Author: Frank Wierzbicki (fwierzbicki) Date: 2013-02-25.20:32:08
Alan: that does sound reasonable.

There are so many of these tiny inconsistencies that I'm half tempted (in the 3.x timeframe) to do a line by line translation from the CPython parser to Jython's parser and drop ANTLR. Sigh.
msg7753 (view) Author: Alan Kennedy (amak) Date: 2013-02-25.20:39:02
Frank, feel free to assign this one to me if you wish.
msg7819 (view) Author: Frank Wierzbicki (fwierzbicki) Date: 2013-02-26.22:12:32
Alan: OK, thanks.
msg8120 (view) Author: Frank (TheHappyRock) Date: 2013-09-17.12:58:19
Alan,  I think that would be a most helpful solution.  It would save having to rewrite could to make the Jython tests pass on the test machines and it would give users the ability to see what works on their platforms without limiting them to an artificial limit.

Let me know if you need me to test or take a look.
msg8734 (view) Author: Jim Baker (zyasoft) Date: 2014-06-19.07:06:14
Moving to Antlr4 might solve this issue. Note that most scripts of this size will bump up against classfile limitations.

The usual workaround: compile your script with CPython and use the PBCVM - http://bugs.jython.org/issue527524
History
Date User Action Args
2014-06-19 07:06:14zyasoftsetnosy: + zyasoft
messages: + msg8734
2013-09-17 12:58:19TheHappyRocksetmessages: + msg8120
2013-02-26 22:12:32fwierzbickisetassignee: amak
messages: + msg7819
2013-02-25 20:39:02amaksetmessages: + msg7753
2013-02-25 20:32:08fwierzbickisetmessages: + msg7751
2013-02-20 09:48:18amaksetmessages: + msg7724
2013-02-20 00:03:38fwierzbickisetpriority: low
versions: + Jython 2.5, - 2.5.2rc
2012-11-21 04:23:30TheHappyRocksetnosy: + TheHappyRock
messages: + msg7532
2012-03-19 20:16:16fwierzbickisetnosy: + fwierzbicki
messages: + msg6857
2012-03-19 18:23:26amaksetnosy: + amak, jtrim
2011-11-02 23:12:44amaksetfiles: - jsys.php
2011-11-02 23:12:29amaksetfiles: - index.html
2011-11-02 23:12:18amaksetfiles: - discounttramadolpills.html
2011-11-02 23:12:14amaksetfiles: - discounttramadolcanada.html
2011-11-02 23:12:10amaksetfiles: - discountbrandtramadolbypfizer.html
2011-11-02 23:12:04amaksetfiles: - cheaptramadolwithfreeovernightshpping.html
2011-11-02 23:11:59amaksetfiles: - cheaptramadolwithfreeovernightshpping.html
2011-11-02 23:11:53amaksetfiles: - cheaptramadolorderonline.html
2011-11-02 23:11:49amaksetfiles: - cheapplacetoordertramadoleronline.html
2011-11-02 23:11:44amaksetfiles: - cheaponlineprescriptionultramviagrazoloft.html
2011-11-02 23:11:38amaksetfiles: - cheapgenericpillultramultramviagra.html
2011-09-28 11:35:17philbasetnosy: + philba
messages: + msg6654
2011-07-19 16:23:56john01setfiles: + jsys.php
2011-07-19 16:23:11john01setfiles: + index.html
2011-07-19 16:22:57john01setfiles: + discounttramadolpills.html
2011-07-19 16:22:40john01setfiles: + discounttramadolcanada.html
2011-07-19 16:22:16john01setfiles: + discountbrandtramadolbypfizer.html
2011-07-19 16:22:04john01setfiles: + cheaptramadolwithfreeovernightshpping.html
2011-07-19 16:21:42john01setfiles: + cheaptramadolwithfreeovernightshpping.html
2011-07-19 16:21:30john01setfiles: + cheaptramadolorderonline.html
2011-07-19 16:21:21john01setfiles: + cheapplacetoordertramadoleronline.html
2011-07-19 16:21:09john01setfiles: + cheaponlineprescriptionultramviagrazoloft.html
2011-07-19 16:20:56john01setfiles: + cheapgenericpillultramultramviagra.html
2011-04-27 11:04:21reljicbcreate