Issue2625

Successive calls to socket.gethostname() or any other functions of this module makes the script execution crash with the following trace:

javax.script.ScriptException: java.lang.IllegalStateException: java.lang.IllegalStateException: failed to create a child event loop in <script> at line number 1
	at org.python.jsr223.PyScriptEngine.scriptException(PyScriptEngine.java:222)
	at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:59)
	at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:31)
	at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
	at testjython.Main.main(Main.java:49)
Caused by: Traceback (most recent call last):
  File "<script>", line 1, in <module>
  File "/home/paraita/src/local_java_stuff/lib/jython-standalone-2.7.1.jar/Lib/socket.py", line 3, in <module>
  File "/home/paraita/src/local_java_stuff/lib/jython-standalone-2.7.1.jar/Lib/_socket.py", line 217, in <module>
	at org.python.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:88)
	at org.python.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58)
	at org.python.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47)
	at org.python.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59)
	at org.python.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:77)
	at org.python.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:72)
	at org.python.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:59)
	at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.python.core.PyReflectedConstructor.constructProxy(PyReflectedConstructor.java:211)
java.lang.IllegalStateException: java.lang.IllegalStateException: failed to create a child event loop

	at org.python.core.Py.JavaError(Py.java:552)
	at org.python.core.Py.JavaError(Py.java:543)
	at org.python.core.PyReflectedConstructor.constructProxy(PyReflectedConstructor.java:221)
	at org.python.core.PyReflectedConstructor.__call__(PyReflectedConstructor.java:180)
	at org.python.core.PyObject.__call__(PyObject.java:438)
	at org.python.core.PyMethod.instancemethod___call__(PyMethod.java:237)
	at org.python.core.PyMethod.__call__(PyMethod.java:228)
	at org.python.core.PyMethod.__call__(PyMethod.java:223)
	at org.python.core.Deriveds.dispatch__init__(Deriveds.java:19)
	at org.python.core.PyObjectDerived.dispatch__init__(PyObjectDerived.java:1112)
	at org.python.core.PyType.type___call__(PyType.java:1822)
	at org.python.core.PyType.__call__(PyType.java:1805)
	at org.python.core.PyObject.__call__(PyObject.java:497)
	at org.python.core.PyObject.__call__(PyObject.java:501)
	at _socket$py.f$0(/home/paraita/src/local_java_stuff/lib/jython-standalone-2.7.1.jar/Lib/_socket.py:1965)
	at _socket$py.call_function(/home/paraita/src/local_java_stuff/lib/jython-standalone-2.7.1.jar/Lib/_socket.py)
	at org.python.core.PyTableCode.call(PyTableCode.java:171)
	at org.python.core.PyCode.call(PyCode.java:18)
	at org.python.core.imp.createFromCode(imp.java:436)
	at org.python.core.util.importer.importer_load_module(importer.java:116)
	at org.python.modules.zipimport.zipimporter.zipimporter_load_module(zipimporter.java:170)
	at org.python.modules.zipimport.zipimporter$zipimporter_load_module_exposer.__call__(Unknown Source)
	at org.python.core.PyBuiltinMethodNarrow.__call__(PyBuiltinMethodNarrow.java:46)
	at org.python.core.imp.loadFromLoader(imp.java:593)
	at org.python.core.imp.find_module(imp.java:547)
	at org.python.core.imp.import_next(imp.java:838)
	at org.python.core.imp.import_module_level(imp.java:957)
	at org.python.core.imp.importName(imp.java:1057)
	at org.python.core.ImportFunction.__call__(__builtin__.java:1280)
	at org.python.core.PyObject.__call__(PyObject.java:450)
	at org.python.core.__builtin__.__import__(__builtin__.java:1232)
	at org.python.core.imp.importFromAs(imp.java:1149)
	at org.python.core.imp.importFrom(imp.java:1124)
	at socket$py.f$0(/home/paraita/src/local_java_stuff/lib/jython-standalone-2.7.1.jar/Lib/socket.py:132)
	at socket$py.call_function(/home/paraita/src/local_java_stuff/lib/jython-standalone-2.7.1.jar/Lib/socket.py)
	at org.python.core.PyTableCode.call(PyTableCode.java:171)
	at org.python.core.PyCode.call(PyCode.java:18)
	at org.python.core.imp.createFromCode(imp.java:436)
	at org.python.core.util.importer.importer_load_module(importer.java:116)
	at org.python.modules.zipimport.zipimporter.zipimporter_load_module(zipimporter.java:170)
	at org.python.modules.zipimport.zipimporter$zipimporter_load_module_exposer.__call__(Unknown Source)
	at org.python.core.PyBuiltinMethodNarrow.__call__(PyBuiltinMethodNarrow.java:46)
	at org.python.core.imp.loadFromLoader(imp.java:593)
	at org.python.core.imp.find_module(imp.java:547)
	at org.python.core.imp.import_next(imp.java:838)
	at org.python.core.imp.import_module_level(imp.java:957)
	at org.python.core.imp.importName(imp.java:1057)
	at org.python.core.ImportFunction.__call__(__builtin__.java:1280)
	at org.python.core.PyObject.__call__(PyObject.java:450)
	at org.python.core.__builtin__.__import__(__builtin__.java:1232)
	at org.python.core.imp.importOne(imp.java:1076)
	at org.python.pycode._pyx270.f$0(<script>:3)
	at org.python.pycode._pyx270.call_function(<script>)
	at org.python.core.PyTableCode.call(PyTableCode.java:171)
	at org.python.core.PyCode.call(PyCode.java:18)
	at org.python.core.Py.runCode(Py.java:1614)
	at org.python.core.__builtin__.eval(__builtin__.java:497)
	at org.python.core.__builtin__.eval(__builtin__.java:501)
	at org.python.util.PythonInterpreter.eval(PythonInterpreter.java:259)
	at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:57)
	... 3 more
Caused by: java.lang.IllegalStateException: failed to create a child event loop
	at org.python.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:88)
	at org.python.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58)
	at org.python.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47)
	at org.python.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59)
	at org.python.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:77)
	at org.python.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:72)
	at org.python.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:59)
	at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.python.core.PyReflectedConstructor.constructProxy(PyReflectedConstructor.java:211)
	... 60 more
Caused by: org.python.netty.channel.ChannelException: failed to open a new selector
	at org.python.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:175)
	at org.python.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:149)
	at org.python.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:127)
	at org.python.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:36)
	at org.python.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
	... 70 more
Caused by: java.io.IOException: Trop de fichiers ouverts
	at sun.nio.ch.IOUtil.makePipe(Native Method)
	at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:65)
	at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
	at org.python.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:173)
	... 74 more

This only happens with Jython 2.7.1, I could not reproduce with 2.7.0.
I also noticed that the execution is slightly slower on 2.7.1.
See the attached code to reproduce.

So the key exception here is

Caused by: java.io.IOException: Too many open files in system 

(or equivalently in French, as seen in the reported traceback, Caused by: java.io.IOException: Trop de fichiers ouverts)

which then causes this exception:

Caused by: java.lang.IllegalStateException: failed to create a child event loop

So presumably this problem is manifesting in 2.7.1 because we chose to isolate PySystemState (= sys) in 2.7.1 per #2154. This isolation in turn means that we are constantly re-initializing PySystemState, including imports of socket. But it's not itself a situation where we are creating new threads; at least on OS X, the thread count remains stable. However, lsof reports a constant growth in files, specifically KQUEUE and PIPE files, before the number of files get too high (on my laptop, that's around 8K open files).

Note this is only seen in the import of the socket module itself.

Also our JSR223 support is perceptibly quite slow given this sys isolation.

So to narrow this down, I removed this statement from Lib/_socket.py:

NIO_GROUP = NioEventLoopGroup(_NUM_THREADS, DaemonThreadFactory("Jython-Netty-Client-%s"))

This deletion is obviously going to cause failures for any actual usage of the socket module, but for testing the import itself of the socket module, this changedoes stabilize the count of open files from lsof (at 40 on my OS X laptop).

One thought I have here is that we should upgrade to Netty 4.1.x from 4.0.x; in particular, this allows us to use Executors for Netty event loops, and in particular ForkJoinPool and on Java 8 the common pool (https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ForkJoinPool.html#commonPool--). We also get other benefits, at the cost of having to do the upgrade (but this needs to be done regardless). Netty 4.1 continues to work against Java 7, so this doesn't impact Jython 2.7 dependencies. This change may also reduce selector usage.

We may need to look at other cleanup as well.