Issue1735864

Running PythonInterpreters in multiple java threads leads to non-deterministic ImportErrors and NameErrors in scripts which 'otherwise' are just fine.

An initial discussion can be found here:
http://www.nabble.com/Using-Jython-in-separate-threads-tf3263616.html

I was able to reproduce the effects with the attached test program.
Platforms: Ubuntu 6.10, Windows XP
Processors: single core, single core HT (hyper threading), dual core
JDK's: JDK1.4.2_14, JDK1.5.0_11

The test program takes two options:
 - number of threads to start
 - number of seconds to pause before starting the threads (useful for remote debugging)

ImportErrors or NameErrors occur between 5 and 50 threads on almost every platform / JDK combination.
(I was not able to reproduce them with JDK1.6.0_01 on my single core laptop).

Luckily problems do not go away if a debugger is attached and paused at ImportErrors.
File Added: Test.java

File Added: test.py

From debugging, I found that completely messed up import statements are issued from PyFrame (see stacktraces.txt):
  "from urllib.swing import JButton"
  "from List.util import List"
It looks like on part originates from one line, and the other part from another line of test.py

File Added: stacktraces.txt

The ImportErrors are compile time errors: There are wrong import statements generated in CodeCompiler.java, as documented in codecompiler.txt

I also had ThreadLocal under suspicion (http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6550283), but switching back to the pre 1.2 ThreadStateMapping made no difference.

The field:
  private static IParserHost literalMkrForParser = new LiteralMakerForParser()
in org.python.parser.java is highly under suspicion:
If I make it ThreadLocal, the situation dramatically improves.
I am investigating further.

I can't see how LiteralMakerForParser would cause any threading errors.  It doesn't store state between method invocations, so it should be safe to call from multiple threads.  Of course, I haven't found anything else that's shared between compilation instances.

I think the patch I'm attaching now may fix this.  TreeBuilder.java kept a single instance of each node type in a static nodes array, and just passed out the correct instance for each id as it was requested.  For JJTNAME, JJTSTRING, and JJTNUM the actual value of the node is stored in the image field of the returned node, so sharing those between threads would cause values to get swapped around.  The patch just returns a new node when any of those types come in rather than caching it.

The test failed reliably for me with 50 threads, but doesn't fail at all anymore.
File Added: no_string_node_cache.patch

The no_string_node_cache.patch solves the problem on my laptop.
I ran the same test as before, without any failures any more.

Alright, committed to trunk in r3255.

On all machines the test program runs fine with the no_string_node_cache.patch

I'd like to check the following:
 - what happens to this fix if we regenerate the parser
 - can we add a bugtest for this
and I'd like to fix it on the 2.3 branch, too.

A customer of mine has done early testing with this fix: problems went away.

The fix is not affected by parser generation.

SyntaxErrors can lead to incomplete nodes with null images.
test293.py got a NullPointerException in closeNode()
The attached patch fixes this.
File Added: 1735864-syntaxerror-patch

SyntaxError patch submitted:
http://jython.svn.sourceforge.net/viewvc/jython?view=rev&revision=3278