Title: os.popen variants hang when executed command produces a lot to stderr
Type: crash Severity: normal
Components: Library Versions: 2.2.2, 2.5b1
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: pjenvey Nosy List: fwierzbicki, pekka.klarck, pjenvey, zyasoft
Priority: high Keywords:

Created on 2008-09-08.22:31:00 by pekka.klarck, last changed 2009-04-03.06:21:17 by pjenvey.

File name Uploaded Description Edit Remove pekka.klarck, 2008-09-08.22:30:59 example script demonstrating the problem
msg3493 (view) Author: Pekka Klärck (pekka.klarck) Date: 2008-09-08.22:30:59
Jython hangs when os.popen, or some other os.popen variants, is used to
execute external commands that produce a lot of text to the standard
error stream.

An example script demonstrating this problem is attached. It takes the
number of characters to write into stdout and stderr as argument, and 
when I run it on my system (OS X 10.5.4 Core 2 Duo w/ Jython 2.2 on Java
1.5.0_13) it executes successfully with argument 16383 but hangs with

If '2>&1' is added after the executed command, or the script is changed
to write only to the stdout, execution passes without problems. Even
though the script uses only os.popen, I was able to reproduce the same
issue with os.popen3, and based on a brief look at the code also other
os.popen variants are affected.

I've tested this only with 2.2, but I can test also against 2.5a2 if
there's a need.
msg4224 (view) Author: Jim Baker (zyasoft) Date: 2009-03-12.06:41:34
Pekka, can you please retry your tests with 2.5b3? Thanks.
msg4252 (view) Author: Pekka Klärck (pekka.klarck) Date: 2009-03-12.23:19:44
Exactly same results with 2.5b3 when using the same machine as in the
original report.
msg4254 (view) Author: Frank Wierzbicki (fwierzbicki) Date: 2009-03-13.14:16:15
Reproduced, so set priority to high.
msg4340 (view) Author: Jim Baker (zyasoft) Date: 2009-03-23.13:01:52
An interesting magic number here: the hanging occurs for any count >=
16384 for this test script.
msg4343 (view) Author: Pekka Klärck (pekka.klarck) Date: 2009-03-23.14:14:11
The magic number is definitely system dependent. I just now tried this
on Jython 2.2/Java 1.6.0_03/WinXP, and only 511 was enough to hang the
script. Are you Jim possibly running on OS X like I've done before?

Notice that because the script automatically writes \n, the real number
of chars written to stderr is actually one more than the given number.
It's probably not just a coincident that pow(2, 9) == 512 and pow(2, 14)
== 16384.
msg4390 (view) Author: Philip Jenvey (pjenvey) Date: 2009-03-30.16:44:57
Need to check if subprocess exhibits the same problem. It's a bit of a 
cleanup of the popen code so switching all the popen to use subprocess is 
a possibility
msg4415 (view) Author: Philip Jenvey (pjenvey) Date: 2009-04-03.06:21:16
indeed, subprocess didn't have this problem, so I've fixed this by 
rewriting popen2 to use it. r6154

I've bumped the size of one of the test_subprocess tests which should 
test this condition (I think)

The script won't deadlock now, but note that you may see the size of the 
final output being smaller than what CPython produces. This is because 
not all the stderr from the child process will always be printed (stdout 
is fine because you're explicitly reading it via PIPE).

In Jython the parent and child processes can't actually share file 
descriptors (Java doesn't allow it), so we're forced to keep the two in 
sync via a separate thread. This thread is a daemon thread (otherwise it 
could prevent Jython from exiting, IIRC) so Jython can exit before it 
captures all the child's stderr

However the output will be like CPython's if the program sleeps for a 
short time before exiting
Date User Action Args
2009-04-03 06:21:18pjenveysetstatus: open -> closed
resolution: accepted -> fixed
messages: + msg4415
2009-04-02 19:19:39pjenveysetassignee: pjenvey
2009-03-30 16:44:57pjenveysetnosy: + pjenvey
messages: + msg4390
2009-03-23 14:14:11pekka.klarcksetmessages: + msg4343
2009-03-23 13:01:53zyasoftsetmessages: + msg4340
2009-03-14 15:09:09fwierzbickisetversions: + 2.5b1
2009-03-13 14:16:16fwierzbickisetpriority: high
nosy: + fwierzbicki
resolution: accepted
messages: + msg4254
2009-03-12 23:19:45pekka.klarcksetmessages: + msg4252
2009-03-12 06:41:35zyasoftsetnosy: + zyasoft
messages: + msg4224
2008-09-08 22:31:00pekka.klarckcreate