Title: PySystemState is lost
Type: Severity: normal
Components: Versions: Jython 2.7
Milestone: Jython 2.7.1
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: jsaiz, zyasoft
Priority: Keywords:

Created on 2016-06-17.08:30:29 by jsaiz, last changed 2017-09-24.16:13:22 by zyasoft.

File name Uploaded Description Edit Remove
systemStateBug.jar jsaiz, 2016-06-17.08:30:27 Example files showing the problem
msg10865 (view) Author: Jaime (jsaiz) Date: 2016-06-17.08:30:27
Unpack the attached jar file.

It contains a Jython file, with a test class extending unittest.TestCase, and a simple Java class that I reproduce here for convenience:

import org.python.core.PySystemState;
import org.python.util.PythonInterpreter;

public class InterpreterTest {
    public static void main(String[] args) {
        findTestCases(new PythonInterpreter(null, new PySystemState()));
        findTestCases(new PythonInterpreter());

    private static void findTestCases(PythonInterpreter interpreter) {
        interpreter.exec("import unittest");
        interpreter.exec("import module.sometest");

The purpose of this Java class is to print the test cases found in the Jython file.

As the Jython class contains two test methods, the result of executing InterpreterTest should be that they are printed twice (one by each interpreter). However, the output of this program is:

linux> java InterpreterTest
<unittest.suite.TestSuite tests=[]>
<unittest.suite.TestSuite tests=[<unittest.suite.TestSuite tests=[<module.sometest.SomeTest testMethod=testOne>, <module.sometest.SomeTest testMethod=testTwo>]>]>

I've been debugging and the cause is not easy to explain, but in summary it is due to the PySystemState passed to the first interpreter being lost, because Py.getSystemState() accesses in the end ThreadStateMapping.cachedThreadState, which has weak references to the ThreadState objects. As the PySystemState is accessed through the ThreadState, if the ThreadState is garbage collected, then the default initial PySystemState is used.

This, in combination of the fact that PyType.isSubType uses the == operator for checking equality of PyType objects, rather than the equals method, produces that two different instances of PyType representing unittest.TestCase are found to be different, so the example SomeTest is not detected to extend unittest.TestCase in the first case.

A solution would be to somehow make hard references to the ThreadState objects.

We've solve this by holding a WeakHashMap<PythonInterpreter, ThreadState> that only frees the ThreadState when its interpreter is garbage collected, but other better solution could be found.
msg10866 (view) Author: Jaime (jsaiz) Date: 2016-06-17.08:38:21

* Red Hat Enterprise Linux, release 6.8 (Santiago)
* Kernel Linux 2.6.32-642.1.1.el6.x86_64
* Oracle Java 1.8.0_91
* 8 Intel Core i7 processors
msg10907 (view) Author: Jim Baker (zyasoft) Date: 2016-08-24.20:27:40
This has been a perennial challenge. Could you provide your patch using WeakHashMap?
msg10911 (view) Author: Jim Baker (zyasoft) Date: 2016-08-24.21:40:55
This has been a perennial challenge. Could you provide your patch using WeakHashMap?
msg10915 (view) Author: Jaime (jsaiz) Date: 2016-08-25.13:26:25
We create PythonInterpreter instances by means of a custom InterpreterFactory.

This class contains the following map:

// Map with hard references to thread states, depending on their interpreter
private static final Map<PythonInterpreter, ThreadState> threadStates = new WeakHashMap<>();

Then the method InterpreterFactory.newInterpreter, before returning the created PythonInterpreter, stores the interpreter in the auxiliary map. Something like:

PySystemState sysState = new PySystemState();
PythonInterpreter pythonInterpreter = new PythonInterpreter(null, sysState);
ThreadState threadState = Py.getThreadState(sysState);
threadStates.put(pythonInterpreter, threadState);

This ensures that the ThreadState of the created PythonInterpreter is not garbage collected during the interpreter's lifetime, with the net effect that the appropriate PySystemState will be used for every interpreter.

This idea might somehow be implemented in the Jython core.
msg11583 (view) Author: Jim Baker (zyasoft) Date: 2017-09-10.00:29:51
Should get this fixed; per the docs in ThreadStateMapping:

"Because ThreadState itself has a weak reference to PySystemState, we also need to ensure a hard reference to it is separately maintained in the call stack."
msg11589 (view) Author: Jim Baker (zyasoft) Date: 2017-09-11.03:37:38
Just revisited this, and it looks like this problem has been fixed in 2.7.1. In particular, with the test case provided, we now get the desired output:

<unittest.suite.TestSuite tests=[<unittest.suite.TestSuite tests=[<module.sometest.SomeTest testMethod=testOne>, <module.sometest.SomeTest testMethod=testTwo>]>]>
<unittest.suite.TestSuite tests=[<unittest.suite.TestSuite tests=[<module.sometest.SomeTest testMethod=testOne>, <module.sometest.SomeTest testMethod=testTwo>]>]>

I believe this problem was fixed with, which directly addresses overly eager collection of PySystemState, because of the use of weakrefs.
msg11590 (view) Author: Jim Baker (zyasoft) Date: 2017-09-11.03:38:07
Correct the milestone
Date User Action Args
2017-09-24 16:13:22zyasoftsetstatus: pending -> closed
2017-09-11 03:38:07zyasoftsetmessages: + msg11590
milestone: Jython 2.7.2 -> Jython 2.7.1
2017-09-11 03:37:39zyasoftsetstatus: open -> pending
resolution: fixed
messages: + msg11589
2017-09-10 00:29:52zyasoftsetmessages: + msg11583
2016-09-30 16:19:59zyasoftsetassignee: zyasoft ->
2016-09-14 16:23:38zyasoftsetassignee: zyasoft
2016-08-25 13:26:26jsaizsetmessages: + msg10915
2016-08-24 21:40:55zyasoftsetmessages: + msg10911
2016-08-24 20:27:40zyasoftsetnosy: + zyasoft
messages: + msg10907
milestone: Jython 2.7.2
2016-06-17 08:38:22jsaizsetmessages: + msg10866
2016-06-17 08:30:29jsaizcreate