Issue1124
Created on 2008-09-08.22:31:00 by pekka.klarck, last changed 2009-04-03.06:21:17 by pjenvey.
File name |
Uploaded |
Description |
Edit |
Remove |
jython_popen_bug.py
|
pekka.klarck,
2008-09-08.22:30:59
|
example script demonstrating the problem |
|
|
msg3493 (view) |
Author: Pekka Klärck (pekka.klarck) |
Date: 2008-09-08.22:30:59 |
|
Jython hangs when os.popen, or some other os.popen variants, is used to
execute external commands that produce a lot of text to the standard
error stream.
An example script demonstrating this problem is attached. It takes the
number of characters to write into stdout and stderr as argument, and
when I run it on my system (OS X 10.5.4 Core 2 Duo w/ Jython 2.2 on Java
1.5.0_13) it executes successfully with argument 16383 but hangs with
16384.
If '2>&1' is added after the executed command, or the script is changed
to write only to the stdout, execution passes without problems. Even
though the script uses only os.popen, I was able to reproduce the same
issue with os.popen3, and based on a brief look at the code also other
os.popen variants are affected.
I've tested this only with 2.2, but I can test also against 2.5a2 if
there's a need.
|
msg4224 (view) |
Author: Jim Baker (zyasoft) |
Date: 2009-03-12.06:41:34 |
|
Pekka, can you please retry your tests with 2.5b3? Thanks.
|
msg4252 (view) |
Author: Pekka Klärck (pekka.klarck) |
Date: 2009-03-12.23:19:44 |
|
Exactly same results with 2.5b3 when using the same machine as in the
original report.
|
msg4254 (view) |
Author: Frank Wierzbicki (fwierzbicki) |
Date: 2009-03-13.14:16:15 |
|
Reproduced, so set priority to high.
|
msg4340 (view) |
Author: Jim Baker (zyasoft) |
Date: 2009-03-23.13:01:52 |
|
An interesting magic number here: the hanging occurs for any count >=
16384 for this test script.
|
msg4343 (view) |
Author: Pekka Klärck (pekka.klarck) |
Date: 2009-03-23.14:14:11 |
|
The magic number is definitely system dependent. I just now tried this
on Jython 2.2/Java 1.6.0_03/WinXP, and only 511 was enough to hang the
script. Are you Jim possibly running on OS X like I've done before?
Notice that because the script automatically writes \n, the real number
of chars written to stderr is actually one more than the given number.
It's probably not just a coincident that pow(2, 9) == 512 and pow(2, 14)
== 16384.
|
msg4390 (view) |
Author: Philip Jenvey (pjenvey) |
Date: 2009-03-30.16:44:57 |
|
Need to check if subprocess exhibits the same problem. It's a bit of a
cleanup of the popen code so switching all the popen to use subprocess is
a possibility
|
msg4415 (view) |
Author: Philip Jenvey (pjenvey) |
Date: 2009-04-03.06:21:16 |
|
indeed, subprocess didn't have this problem, so I've fixed this by
rewriting popen2 to use it. r6154
I've bumped the size of one of the test_subprocess tests which should
test this condition (I think)
The script won't deadlock now, but note that you may see the size of the
final output being smaller than what CPython produces. This is because
not all the stderr from the child process will always be printed (stdout
is fine because you're explicitly reading it via PIPE).
In Jython the parent and child processes can't actually share file
descriptors (Java doesn't allow it), so we're forced to keep the two in
sync via a separate thread. This thread is a daemon thread (otherwise it
could prevent Jython from exiting, IIRC) so Jython can exit before it
captures all the child's stderr
However the output will be like CPython's if the program sleeps for a
short time before exiting
|
|
Date |
User |
Action |
Args |
2009-04-03 06:21:18 | pjenvey | set | status: open -> closed resolution: accepted -> fixed messages:
+ msg4415 |
2009-04-02 19:19:39 | pjenvey | set | assignee: pjenvey |
2009-03-30 16:44:57 | pjenvey | set | nosy:
+ pjenvey messages:
+ msg4390 |
2009-03-23 14:14:11 | pekka.klarck | set | messages:
+ msg4343 |
2009-03-23 13:01:53 | zyasoft | set | messages:
+ msg4340 |
2009-03-14 15:09:09 | fwierzbicki | set | versions:
+ 2.5b1 |
2009-03-13 14:16:16 | fwierzbicki | set | priority: high nosy:
+ fwierzbicki resolution: accepted messages:
+ msg4254 |
2009-03-12 23:19:45 | pekka.klarck | set | messages:
+ msg4252 |
2009-03-12 06:41:35 | zyasoft | set | nosy:
+ zyasoft messages:
+ msg4224 |
2008-09-08 22:31:00 | pekka.klarck | create | |
|