Title: ImportError when importing in multiple PyScriptEngine concurrently
Type: crash Severity: normal
Components: Core Versions: Jython 2.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: amoebam, jeff.allen, tcdelaney
Priority: normal Keywords: thread-interpreter context

Created on 2017-11-08.01:34:17 by tcdelaney, last changed 2019-08-08.20:20:06 by amoebam.

msg11652 (view) Author: Tim Delaney (tcdelaney) Date: 2017-11-08.01:34:15
I have an application which creates multiple PyScriptEngine instances on different threads. If multiple threads concurrently try to import I often get errors like:

Traceback (most recent call last):
  File ".\", line 34, in <module>
    import threading
  File ".\lib\jython-standalone-2.7.1.jar\Lib\", line 88, in <module>
AttributeError: type object 'java.lang.Thread' has no attribute 'State'


Traceback (most recent call last):
  File ".\", line 44, in <module>
    import gzip
  File ".\lib\jython-standalone-2.7.1.jar\Lib\", line 10, in <module>
  File ".\lib\jython-standalone-2.7.1.jar\Lib\", line 69, in <module>
  File ".\lib\jython-standalone-2.7.1.jar\Lib\", line 50, in <module>
ImportError: cannot import name DEFAULT_BUFFER_SIZE

Whilst I haven't been able to verify it, it appears to only be when importing Java classes.

By experimentation, if only a single thread is importing at a time the errors do not occur. As a workaround for my application I'm using a global import lock rather than an import lock per PySystemState, which seems to have resolved the issue. It's possible this is overkill and only part of the import process needs to be protected with a global lock.
msg11718 (view) Author: Jeff Allen (jeff.allen) Date: 2018-02-25.22:45:22
Things not having attributes when you know they do, and concurrency is in play, is reminiscent of #2609. A good thing to check in 2.7.2 beta, when it appears (or build your own alpha).
msg11851 (view) Author: Jeff Allen (jeff.allen) Date: 2018-03-23.20:45:00
Bot promising ofr 2.7.2 as I think concurrent interpreters is a topic needing careful thought (and testing). However, it is still possible #2609 has fixed it in this case.
msg12626 (view) Author: Andy Merton (amoebam) Date: 2019-08-08.20:20:06
Just dropping in to report that contrary to Jeff's hope, #2609 did not fix this.

I've checked with a 2.7.2a1+ built from hg master at the time of posting, and it is still necessary to prevent Python imports from coinciding on different JVM threads -- otherwise we observe "impossible" errors such as the Python code "from collections import deque" failing with an ImportError "cannot import name deque".

The underlying cause is, as you might guess, that the first thread to request an import starts initializing the new object, but the second thread receives a reference to the same object while the first thread is still only halfway through the classDictInit method, requests an attribute that has not been added yet, and dies.

So, a global lock it is, for now.  But we're not quite confident we're locking the right things - we've stopped seeing crashes, but maybe we've just made them much rarer? - and it would be much better if whatever locking is actually required were done by Jython itself.
Date User Action Args
2019-08-08 20:20:06amoebamsetnosy: + amoebam
messages: + msg12626
2018-03-23 20:45:01jeff.allensetkeywords: + thread-interpreter context
priority: normal
messages: + msg11851
milestone: Jython 2.7.2 ->
2018-02-25 22:45:22jeff.allensetnosy: + jeff.allen
messages: + msg11718
milestone: Jython 2.7.1 -> Jython 2.7.2
2017-11-08 01:34:17tcdelaneycreate