Title: PySystemState is lost
Type: Severity: normal
Components: Versions: Jython 2.7
Milestone: Jython 2.7.2
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: jsaiz, zyasoft
Priority: Keywords:

Created on 2016-06-17.08:30:29 by jsaiz, last changed 2016-09-30.16:19:59 by zyasoft.

File name Uploaded Description Edit Remove
systemStateBug.jar jsaiz, 2016-06-17.08:30:27 Example files showing the problem
msg10865 (view) Author: Jaime (jsaiz) Date: 2016-06-17.08:30:27
Unpack the attached jar file.

It contains a Jython file, with a test class extending unittest.TestCase, and a simple Java class that I reproduce here for convenience:

import org.python.core.PySystemState;
import org.python.util.PythonInterpreter;

public class InterpreterTest {
    public static void main(String[] args) {
        findTestCases(new PythonInterpreter(null, new PySystemState()));
        findTestCases(new PythonInterpreter());

    private static void findTestCases(PythonInterpreter interpreter) {
        interpreter.exec("import unittest");
        interpreter.exec("import module.sometest");

The purpose of this Java class is to print the test cases found in the Jython file.

As the Jython class contains two test methods, the result of executing InterpreterTest should be that they are printed twice (one by each interpreter). However, the output of this program is:

linux> java InterpreterTest
<unittest.suite.TestSuite tests=[]>
<unittest.suite.TestSuite tests=[<unittest.suite.TestSuite tests=[<module.sometest.SomeTest testMethod=testOne>, <module.sometest.SomeTest testMethod=testTwo>]>]>

I've been debugging and the cause is not easy to explain, but in summary it is due to the PySystemState passed to the first interpreter being lost, because Py.getSystemState() accesses in the end ThreadStateMapping.cachedThreadState, which has weak references to the ThreadState objects. As the PySystemState is accessed through the ThreadState, if the ThreadState is garbage collected, then the default initial PySystemState is used.

This, in combination of the fact that PyType.isSubType uses the == operator for checking equality of PyType objects, rather than the equals method, produces that two different instances of PyType representing unittest.TestCase are found to be different, so the example SomeTest is not detected to extend unittest.TestCase in the first case.

A solution would be to somehow make hard references to the ThreadState objects.

We've solve this by holding a WeakHashMap<PythonInterpreter, ThreadState> that only frees the ThreadState when its interpreter is garbage collected, but other better solution could be found.
msg10866 (view) Author: Jaime (jsaiz) Date: 2016-06-17.08:38:21

* Red Hat Enterprise Linux, release 6.8 (Santiago)
* Kernel Linux 2.6.32-642.1.1.el6.x86_64
* Oracle Java 1.8.0_91
* 8 Intel Core i7 processors
msg10907 (view) Author: Jim Baker (zyasoft) Date: 2016-08-24.20:27:40
This has been a perennial challenge. Could you provide your patch using WeakHashMap?
msg10911 (view) Author: Jim Baker (zyasoft) Date: 2016-08-24.21:40:55
This has been a perennial challenge. Could you provide your patch using WeakHashMap?
msg10915 (view) Author: Jaime (jsaiz) Date: 2016-08-25.13:26:25
We create PythonInterpreter instances by means of a custom InterpreterFactory.

This class contains the following map:

// Map with hard references to thread states, depending on their interpreter
private static final Map<PythonInterpreter, ThreadState> threadStates = new WeakHashMap<>();

Then the method InterpreterFactory.newInterpreter, before returning the created PythonInterpreter, stores the interpreter in the auxiliary map. Something like:

PySystemState sysState = new PySystemState();
PythonInterpreter pythonInterpreter = new PythonInterpreter(null, sysState);
ThreadState threadState = Py.getThreadState(sysState);
threadStates.put(pythonInterpreter, threadState);

This ensures that the ThreadState of the created PythonInterpreter is not garbage collected during the interpreter's lifetime, with the net effect that the appropriate PySystemState will be used for every interpreter.

This idea might somehow be implemented in the Jython core.
Date User Action Args
2016-09-30 16:19:59zyasoftsetassignee: zyasoft ->
2016-09-14 16:23:38zyasoftsetassignee: zyasoft
2016-08-25 13:26:26jsaizsetmessages: + msg10915
2016-08-24 21:40:55zyasoftsetmessages: + msg10911
2016-08-24 20:27:40zyasoftsetnosy: + zyasoft
messages: + msg10907
milestone: Jython 2.7.2
2016-06-17 08:38:22jsaizsetmessages: + msg10866
2016-06-17 08:30:29jsaizcreate