Issue1744

I use Jython as a java ScriptEngine, and I load scripts longer then 100000 characters, which throws IOException:

javax.script.ScriptException: java.io.IOException: java.io.IOException: Mark invalid
        at org.python.jsr223.PyScriptEngine.scriptException(PyScriptEngine.java:196)
        at org.python.jsr223.PyScriptEngine.compileScript(PyScriptEngine.java:75)
        at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:31)
        at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:247)
        at testjython.Main.run(Main.java:36)
        at testjython.Main.main(Main.java:56)
Caused by: java.io.IOException: Mark invalid

java.io.IOException: java.io.IOException: Mark invalid

        at org.python.core.PyException.fillInStackTrace(PyException.java:70)
        at java.lang.Throwable.<init>(Throwable.java:181)
        at java.lang.Exception.<init>(Exception.java:29)
        at java.lang.RuntimeException.<init>(RuntimeException.java:32)
        at org.python.core.PyException.<init>(PyException.java:46)
        at org.python.core.PyException.<init>(PyException.java:43)
        at org.python.core.Py.JavaError(Py.java:481)
        at org.python.core.ParserFacade.fixParseError(ParserFacade.java:104)
        at org.python.core.ParserFacade.parseExpressionOrModule(ParserFacade.java:130)


Test ended.
        at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:259)
        at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:256)
        at org.python.util.PythonInterpreter.compile(PythonInterpreter.java:250)
        at org.python.jsr223.PyScriptEngine.compileScript(PyScriptEngine.java:70)
        ... 4 more
Caused by: java.io.IOException: Mark invalid
        at java.io.BufferedReader.reset(BufferedReader.java:485)
        at org.python.core.ParserFacade.parseExpressionOrModule(ParserFacade.java:127)
        ... 8 more


I am not saying this is a bug, but it is certainly a behavior which limits me from executing not that big python scripts.

Is there a way to increase this limit?

Cheers,
Bojan

PS. See attached java file to reproduce the problem.

I would also like to increase the MARK_LIMIT.

We can't really increase the mark limit - it causes out of memory errors pretty quickly in our regression tests if it is large, which leads me to believe that it can be leaky. Routinely rejecting >100K files isn't what we want either - but I'm not sure of a way to fix this yet. The two uses we have are checking the very beginning of the file for unicode markers - these we could replace with a pushback inputstream (http://docs.oracle.com/javase/1.4.2/docs/api/java/io/PushbackInputStream.html)

 The harder part is printing lines for parse errors - we might still be able to use a pushback input stream for this too, but it would take some investigation. If anyone would like to try to submit a patch for this I'd be more than willing to review it.

Just wanted to keep this alive as it is a big deal for programs that create large Jython scripts.

I don't know the Jython code base at all, but hopefully this catches someone's eye that does.

Frank, perhaps we could expose the MARK_LIMIT as a registry property, and give the users control over it?

http://hg.python.org/jython/file/a30708945630/src/org/python/core/ParserFacade.java#l40

We could still default it to 100,000 if no explicit value is specified.

That way, the user could set it higher if required, and they have the memory to support it.

Alan: that does sound reasonable.

There are so many of these tiny inconsistencies that I'm half tempted (in the 3.x timeframe) to do a line by line translation from the CPython parser to Jython's parser and drop ANTLR. Sigh.

Frank, feel free to assign this one to me if you wish.

Alan: OK, thanks.

Alan,  I think that would be a most helpful solution.  It would save having to rewrite could to make the Jython tests pass on the test machines and it would give users the ability to see what works on their platforms without limiting them to an artificial limit.

Let me know if you need me to test or take a look.

Moving to Antlr4 might solve this issue. Note that most scripts of this size will bump up against classfile limitations.

The usual workaround: compile your script with CPython and use the PBCVM - http://bugs.jython.org/issue527524