Issue2487

classification
Title: PyType.fromClass publication bug on slow systems (travis, circleci)
Type: Severity: critical
Components: Core Versions: Jython 2.7
Milestone: Jython 2.7.1
process
Status: open Resolution: accepted
Dependencies: Superseder:
Assigned To: Nosy List: darjus, jamesmudd, stefan.richthofer, zyasoft
Priority: urgent Keywords:

Created on 2016-03-18.00:08:39 by darjus, last changed 2017-04-04.23:25:59 by zyasoft.

Files
File name Uploaded Description Edit Remove
thread-dump-2462.txt darjus, 2016-03-18.00:08:36
Messages
msg10823 (view) Author: Jim Baker (zyasoft) Date: 2016-03-18.02:43:55
Fixed as of https://hg.python.org/jython/rev/cffe5b824a21
msg10828 (view) Author: Darjus Loktevic (darjus) Date: 2016-03-19.07:58:49
Mar 19, 2016 6:56:12 PM io.netty.channel.ChannelInitializer exceptionCaught
WARNING: Failed to initialize a channel. Closing: [id: 0xabe1bf32, /127.0.0.1:63059 => /127.0.0.1:63054]
Traceback (most recent call last):
  File "/Users/darjus/Documents/jython/dist/Lib/_socket.py", line 639, in initChannel
    child._ensure_post_connect()
  File "/Users/darjus/Documents/jython/dist/Lib/_socket.py", line 1421, in _ensure_post_connect
    self._post_connect()
  File "/Users/darjus/Documents/jython/dist/Lib/_socket.py", line 877, in _post_connect
    self.channel.pipeline().addLast(self.python_inbound_handler)
SystemError: __getattribute__ not found on type DefaultChannelPipeline


Looks like there's a bug in the new synchronization code. Found it while running some unrelated stuff.
msg10854 (view) Author: Stefan Richthofer (stefan.richthofer) Date: 2016-05-08.18:35:52
Hey. Is there a way to reproduce this issue (on a fast system)?
Does someone have jstack output available for the deadlock?
msg10906 (view) Author: Jim Baker (zyasoft) Date: 2016-08-24.20:10:11
Fixed by https://hg.python.org/jython/rev/f3458ef83e08
msg11296 (view) Author: James Mudd (jamesmudd) Date: 2017-04-04.22:09:28
Not sure if its actually related to this problem or not as its closed but a CI build just experienced something related to this. See: https://travis-ci.org/jythontools/jython/jobs/218647190
msg11297 (view) Author: Jim Baker (zyasoft) Date: 2017-04-04.23:24:50
It is in fact a bug, a classic object publication/data race to be precise. The problem is that it is possible for a Java type to be published by one thread in classToType, as held by PyType#LazyClassToTypeHolder; but for another thread to simultaneously see/lookup this type for the first time.

We used to synchronize on this map, but that caused deadlocks (the original bug here). Then we introduced polling with the bug fix (https://hg.python.org/jython/rev/f3458ef83e08), but we realized it was a hack, and linked it to this bug.

Probably the right way to fix this bug is to use a scheme actually seen in Lib/_sockets.py - use associated condition variables to mark when the publication is finished. This should be close to the current code: continue polling on the existence of __getattribute__, but wait for the specific publication. So we can start with a ConcurrentMap<Class<?>, Condition>, publish it from one thread the specific condition variable using #putIfAbsent, and cv wait for it in https://github.com/jythontools/jython/blob/master/src/org/python/core/Deriveds.java#L51
History
Date User Action Args
2017-04-04 23:25:59zyasoftsettitle: PyType.fromClass deadlocks on slow systems (circleci for example) -> PyType.fromClass publication bug on slow systems (travis, circleci)
2017-04-04 23:25:12zyasoftsetpriority: immediate -> urgent
assignee: darjus ->
status: closed -> open
resolution: fixed -> accepted
2017-04-04 23:24:51zyasoftsetmessages: + msg11297
2017-04-04 22:09:28jamesmuddsetnosy: + jamesmudd
messages: + msg11296
2016-08-24 20:10:11zyasoftsetstatus: open -> closed
messages: + msg10906
2016-05-08 18:35:53stefan.richthofersetnosy: + stefan.richthofer
messages: + msg10854
2016-03-19 07:58:50darjussetstatus: pending -> open
messages: + msg10828
2016-03-18 02:43:55zyasoftsetstatus: open -> pending
resolution: fixed
messages: + msg10823
nosy: + zyasoft
2016-03-18 00:08:39darjuscreate