Issue1765

classification
Title: subprocess.Popen leaks fds
Type: crash Severity: normal
Components: Library Versions: Jython 2.5
process
Status: closed Resolution: invalid
Dependencies: Superseder:
Assigned To: Nosy List: fwierzbicki, halfie, mkz, zyasoft
Priority: normal Keywords:

Created on 2011-07-03.06:00:35 by mkz, last changed 2014-06-19.05:56:40 by zyasoft.

Files
File name Uploaded Description Edit Remove
proctest.py mkz, 2011-07-03.06:00:34 python code
Messages
msg6561 (view) Author: Marko Kreen (mkz) Date: 2011-07-03.06:00:34
Code that repeatedly calls subprocess.Popen() will crash with
'Too many files open' error.

Attached files reproduce it.  Crash is reproducable also with default
ulimit (1024).

Python code contains several patterns of Popen usage, all leaky.

Excpected: .communicate() and code that closes fds excplicitly
should be safe.

test script:
--------------------------
#! /bin/sh

# crash faster
ulimit -n 32

jython ./proctest.py

# jython leaves tty messed up
stty sane

--------------------------
Output:

$ ./test.sh
0
1
2
3
4
5
Traceback (most recent call last):
  File "./proctest.py", line 9, in <module>
    p = Popen(['true'])
  File "/opt/apps/jython/Lib/subprocess.py", line 751, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/opt/apps/jython/Lib/subprocess.py", line 1265, in _execute_child
    raise OSError(e.getMessage() or e)
OSError: Cannot run program "true" (in directory "/home/marko/src/niisama/rarfile/jxx"): java.io.IOException: error=24, Too many open files
msg6562 (view) Author: Marko Kreen (mkz) Date: 2011-07-03.06:11:08
My setup: Jython 2.5.2, Ubuntu 10.04

$ java -version
java version "1.6.0_20"
OpenJDK Runtime Environment (IcedTea6 1.9.8) (6b20-1.9.8-0ubuntu1~10.04.1)
OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
msg6814 (view) Author: halfie (halfie) Date: 2012-03-19.08:45:43
I can re-produce the bug (using proctest.py) on Oracle JDK and Open JDK 6.x series. However, the bug magically disappears when using Oracle JDK 7. Seems to be a JVM bug (and not a Jython bug).
msg6815 (view) Author: Marko Kreen (mkz) Date: 2012-03-19.09:24:37
The question is, does subprocess module leaves fd.close() to GC?

If yes it's subprocess problem, either in Jython-specific code, or maybe even generic core subprocess problem, except it only is seen on non-refcounting python implementations.

Not sure why newer JVM makes issue disapper.  More aggressive GC?  But does it guarantee the issue is fixed now?

OTOH, if subprocess closes it's objects properly, but still JVM leaves thing hanging, then it's indeed JVM problem.
msg8717 (view) Author: Jim Baker (zyasoft) Date: 2014-06-19.05:56:39
Looks like an earlier JDK problem, since it also works for me on 2.7 trunk against JDK 7. Re subprocess and closing FD resources, I would think the underlying Process.waitFor, as used by our implementation of communicate for example, will take care of closing.
History
Date User Action Args
2014-06-19 05:56:40zyasoftsetstatus: open -> closed
resolution: invalid
messages: + msg8717
nosy: + zyasoft
2013-02-25 20:16:57fwierzbickisetpriority: normal
nosy: + fwierzbicki
versions: + Jython 2.5, - 2.5.2
2012-03-19 09:24:37mkzsetmessages: + msg6815
2012-03-19 08:45:43halfiesetnosy: + halfie
messages: + msg6814
2011-07-03 06:11:08mkzsetmessages: + msg6562
2011-07-03 06:00:35mkzcreate