Issue2625

classification
Title: Repeated calls to the socket module crashes during execution
Type: crash Severity: normal
Components: None Versions: Jython 2.7
Milestone:
process
Status: open Resolution: accepted
Dependencies: Superseder:
Assigned To: Nosy List: amak, fviale, paraita, zyasoft
Priority: Keywords:

Created on 2017-09-15.07:54:16 by paraita, last changed 2017-09-17.04:59:33 by zyasoft.

Files
File name Uploaded Description Edit Remove
Main.java paraita, 2017-09-15.07:54:15 Code to reproduce the issue
Messages
msg11598 (view) Author: Paraita Wohler (paraita) Date: 2017-09-15.07:54:15
Successive calls to socket.gethostname() or any other functions of this module makes the script execution crash with the following trace:

javax.script.ScriptException: java.lang.IllegalStateException: java.lang.IllegalStateException: failed to create a child event loop in <script> at line number 1
	at org.python.jsr223.PyScriptEngine.scriptException(PyScriptEngine.java:222)
	at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:59)
	at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:31)
	at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
	at testjython.Main.main(Main.java:49)
Caused by: Traceback (most recent call last):
  File "<script>", line 1, in <module>
  File "/home/paraita/src/local_java_stuff/lib/jython-standalone-2.7.1.jar/Lib/socket.py", line 3, in <module>
  File "/home/paraita/src/local_java_stuff/lib/jython-standalone-2.7.1.jar/Lib/_socket.py", line 217, in <module>
	at org.python.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:88)
	at org.python.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58)
	at org.python.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47)
	at org.python.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59)
	at org.python.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:77)
	at org.python.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:72)
	at org.python.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:59)
	at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.python.core.PyReflectedConstructor.constructProxy(PyReflectedConstructor.java:211)
java.lang.IllegalStateException: java.lang.IllegalStateException: failed to create a child event loop

	at org.python.core.Py.JavaError(Py.java:552)
	at org.python.core.Py.JavaError(Py.java:543)
	at org.python.core.PyReflectedConstructor.constructProxy(PyReflectedConstructor.java:221)
	at org.python.core.PyReflectedConstructor.__call__(PyReflectedConstructor.java:180)
	at org.python.core.PyObject.__call__(PyObject.java:438)
	at org.python.core.PyMethod.instancemethod___call__(PyMethod.java:237)
	at org.python.core.PyMethod.__call__(PyMethod.java:228)
	at org.python.core.PyMethod.__call__(PyMethod.java:223)
	at org.python.core.Deriveds.dispatch__init__(Deriveds.java:19)
	at org.python.core.PyObjectDerived.dispatch__init__(PyObjectDerived.java:1112)
	at org.python.core.PyType.type___call__(PyType.java:1822)
	at org.python.core.PyType.__call__(PyType.java:1805)
	at org.python.core.PyObject.__call__(PyObject.java:497)
	at org.python.core.PyObject.__call__(PyObject.java:501)
	at _socket$py.f$0(/home/paraita/src/local_java_stuff/lib/jython-standalone-2.7.1.jar/Lib/_socket.py:1965)
	at _socket$py.call_function(/home/paraita/src/local_java_stuff/lib/jython-standalone-2.7.1.jar/Lib/_socket.py)
	at org.python.core.PyTableCode.call(PyTableCode.java:171)
	at org.python.core.PyCode.call(PyCode.java:18)
	at org.python.core.imp.createFromCode(imp.java:436)
	at org.python.core.util.importer.importer_load_module(importer.java:116)
	at org.python.modules.zipimport.zipimporter.zipimporter_load_module(zipimporter.java:170)
	at org.python.modules.zipimport.zipimporter$zipimporter_load_module_exposer.__call__(Unknown Source)
	at org.python.core.PyBuiltinMethodNarrow.__call__(PyBuiltinMethodNarrow.java:46)
	at org.python.core.imp.loadFromLoader(imp.java:593)
	at org.python.core.imp.find_module(imp.java:547)
	at org.python.core.imp.import_next(imp.java:838)
	at org.python.core.imp.import_module_level(imp.java:957)
	at org.python.core.imp.importName(imp.java:1057)
	at org.python.core.ImportFunction.__call__(__builtin__.java:1280)
	at org.python.core.PyObject.__call__(PyObject.java:450)
	at org.python.core.__builtin__.__import__(__builtin__.java:1232)
	at org.python.core.imp.importFromAs(imp.java:1149)
	at org.python.core.imp.importFrom(imp.java:1124)
	at socket$py.f$0(/home/paraita/src/local_java_stuff/lib/jython-standalone-2.7.1.jar/Lib/socket.py:132)
	at socket$py.call_function(/home/paraita/src/local_java_stuff/lib/jython-standalone-2.7.1.jar/Lib/socket.py)
	at org.python.core.PyTableCode.call(PyTableCode.java:171)
	at org.python.core.PyCode.call(PyCode.java:18)
	at org.python.core.imp.createFromCode(imp.java:436)
	at org.python.core.util.importer.importer_load_module(importer.java:116)
	at org.python.modules.zipimport.zipimporter.zipimporter_load_module(zipimporter.java:170)
	at org.python.modules.zipimport.zipimporter$zipimporter_load_module_exposer.__call__(Unknown Source)
	at org.python.core.PyBuiltinMethodNarrow.__call__(PyBuiltinMethodNarrow.java:46)
	at org.python.core.imp.loadFromLoader(imp.java:593)
	at org.python.core.imp.find_module(imp.java:547)
	at org.python.core.imp.import_next(imp.java:838)
	at org.python.core.imp.import_module_level(imp.java:957)
	at org.python.core.imp.importName(imp.java:1057)
	at org.python.core.ImportFunction.__call__(__builtin__.java:1280)
	at org.python.core.PyObject.__call__(PyObject.java:450)
	at org.python.core.__builtin__.__import__(__builtin__.java:1232)
	at org.python.core.imp.importOne(imp.java:1076)
	at org.python.pycode._pyx270.f$0(<script>:3)
	at org.python.pycode._pyx270.call_function(<script>)
	at org.python.core.PyTableCode.call(PyTableCode.java:171)
	at org.python.core.PyCode.call(PyCode.java:18)
	at org.python.core.Py.runCode(Py.java:1614)
	at org.python.core.__builtin__.eval(__builtin__.java:497)
	at org.python.core.__builtin__.eval(__builtin__.java:501)
	at org.python.util.PythonInterpreter.eval(PythonInterpreter.java:259)
	at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:57)
	... 3 more
Caused by: java.lang.IllegalStateException: failed to create a child event loop
	at org.python.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:88)
	at org.python.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58)
	at org.python.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47)
	at org.python.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59)
	at org.python.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:77)
	at org.python.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:72)
	at org.python.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:59)
	at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.python.core.PyReflectedConstructor.constructProxy(PyReflectedConstructor.java:211)
	... 60 more
Caused by: org.python.netty.channel.ChannelException: failed to open a new selector
	at org.python.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:175)
	at org.python.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:149)
	at org.python.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:127)
	at org.python.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:36)
	at org.python.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
	... 70 more
Caused by: java.io.IOException: Trop de fichiers ouverts
	at sun.nio.ch.IOUtil.makePipe(Native Method)
	at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:65)
	at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
	at org.python.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:173)
	... 74 more

This only happens with Jython 2.7.1, I could not reproduce with 2.7.0.
I also noticed that the execution is slightly slower on 2.7.1.
See the attached code to reproduce.
msg11599 (view) Author: Jim Baker (zyasoft) Date: 2017-09-17.03:59:53
So the key exception here is

Caused by: java.io.IOException: Too many open files in system 

(or equivalently in French, as seen in the reported traceback, Caused by: java.io.IOException: Trop de fichiers ouverts)

which then causes this exception:

Caused by: java.lang.IllegalStateException: failed to create a child event loop

So presumably this problem is manifesting in 2.7.1 because we chose to isolate PySystemState (= sys) in 2.7.1 per #2154. This isolation in turn means that we are constantly re-initializing PySystemState, including imports of socket. But it's not itself a situation where we are creating new threads; at least on OS X, the thread count remains stable. However, lsof reports a constant growth in files, specifically KQUEUE and PIPE files, before the number of files get too high (on my laptop, that's around 8K open files).

Note this is only seen in the import of the socket module itself.

Also our JSR223 support is perceptibly quite slow given this sys isolation.
msg11600 (view) Author: Jim Baker (zyasoft) Date: 2017-09-17.04:59:32
So to narrow this down, I removed this statement from Lib/_socket.py:

NIO_GROUP = NioEventLoopGroup(_NUM_THREADS, DaemonThreadFactory("Jython-Netty-Client-%s"))

This deletion is obviously going to cause failures for any actual usage of the socket module, but for testing the import itself of the socket module, this changedoes stabilize the count of open files from lsof (at 40 on my OS X laptop).

One thought I have here is that we should upgrade to Netty 4.1.x from 4.0.x; in particular, this allows us to use Executors for Netty event loops, and in particular ForkJoinPool and on Java 8 the common pool (https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ForkJoinPool.html#commonPool--). We also get other benefits, at the cost of having to do the upgrade (but this needs to be done regardless). Netty 4.1 continues to work against Java 7, so this doesn't impact Jython 2.7 dependencies. This change may also reduce selector usage.

We may need to look at other cleanup as well.
History
Date User Action Args
2017-09-17 04:59:33zyasoftsetmessages: + msg11600
2017-09-17 03:59:54zyasoftsetresolution: accepted
messages: + msg11599
2017-09-17 00:41:05zyasoftsetnosy: + zyasoft
2017-09-15 16:31:25amaksetnosy: + amak
2017-09-15 07:54:16paraitacreate