Issue2505

Unpack the attached jar file.

It contains a Jython file, with a test class extending unittest.TestCase, and a simple Java class that I reproduce here for convenience:

import org.python.core.PySystemState;
import org.python.util.PythonInterpreter;

public class InterpreterTest {
    public static void main(String[] args) {
        findTestCases(new PythonInterpreter(null, new PySystemState()));
        findTestCases(new PythonInterpreter());
    }

    private static void findTestCases(PythonInterpreter interpreter) {
        interpreter.exec("import unittest");
        interpreter.exec("import module.sometest");
        System.out.println(interpreter.eval("unittest.findTestCases(module.sometest)"));
    }
}

The purpose of this Java class is to print the test cases found in the Jython file.

As the Jython class contains two test methods, the result of executing InterpreterTest should be that they are printed twice (one by each interpreter). However, the output of this program is:

linux> java InterpreterTest
<unittest.suite.TestSuite tests=[]>
<unittest.suite.TestSuite tests=[<unittest.suite.TestSuite tests=[<module.sometest.SomeTest testMethod=testOne>, <module.sometest.SomeTest testMethod=testTwo>]>]>

I've been debugging and the cause is not easy to explain, but in summary it is due to the PySystemState passed to the first interpreter being lost, because Py.getSystemState() accesses in the end ThreadStateMapping.cachedThreadState, which has weak references to the ThreadState objects. As the PySystemState is accessed through the ThreadState, if the ThreadState is garbage collected, then the default initial PySystemState is used.

This, in combination of the fact that PyType.isSubType uses the == operator for checking equality of PyType objects, rather than the equals method, produces that two different instances of PyType representing unittest.TestCase are found to be different, so the example SomeTest is not detected to extend unittest.TestCase in the first case.

A solution would be to somehow make hard references to the ThreadState objects.

We've solve this by holding a WeakHashMap<PythonInterpreter, ThreadState> that only frees the ThreadState when its interpreter is garbage collected, but other better solution could be found.

Platform:

* Red Hat Enterprise Linux, release 6.8 (Santiago)
* Kernel Linux 2.6.32-642.1.1.el6.x86_64
* Oracle Java 1.8.0_91
* 8 Intel Core i7 processors

This has been a perennial challenge. Could you provide your patch using WeakHashMap?

This has been a perennial challenge. Could you provide your patch using WeakHashMap?

We create PythonInterpreter instances by means of a custom InterpreterFactory.

This class contains the following map:

// Map with hard references to thread states, depending on their interpreter
private static final Map<PythonInterpreter, ThreadState> threadStates = new WeakHashMap<>();

Then the method InterpreterFactory.newInterpreter, before returning the created PythonInterpreter, stores the interpreter in the auxiliary map. Something like:

PySystemState sysState = new PySystemState();
PythonInterpreter pythonInterpreter = new PythonInterpreter(null, sysState);
ThreadState threadState = Py.getThreadState(sysState);
threadStates.put(pythonInterpreter, threadState);

This ensures that the ThreadState of the created PythonInterpreter is not garbage collected during the interpreter's lifetime, with the net effect that the appropriate PySystemState will be used for every interpreter.

This idea might somehow be implemented in the Jython core.

Should get this fixed; per the docs in ThreadStateMapping:

"Because ThreadState itself has a weak reference to PySystemState, we also need to ensure a hard reference to it is separately maintained in the call stack."

Just revisited this, and it looks like this problem has been fixed in 2.7.1. In particular, with the test case provided, we now get the desired output:

<unittest.suite.TestSuite tests=[<unittest.suite.TestSuite tests=[<module.sometest.SomeTest testMethod=testOne>, <module.sometest.SomeTest testMethod=testTwo>]>]>
<unittest.suite.TestSuite tests=[<unittest.suite.TestSuite tests=[<module.sometest.SomeTest testMethod=testOne>, <module.sometest.SomeTest testMethod=testTwo>]>]>

I believe this problem was fixed with https://hg.python.org/jython/rev/dcff83c1c4ac, which directly addresses overly eager collection of PySystemState, because of the use of weakrefs.

Correct the milestone