Issue1972

classification
Title: jython 2.5.3 sys.stdin.readline() hangs when jython launched as subprocess on Mac OS X
Type: crash Severity: critical
Components: Core, Library Versions: Jython 2.5
process
Status: closed Resolution: fixed
Dependencies: Can only read 8192 bytes
View: 1967
Superseder:
Assigned To: Nosy List: amak, emcdowell, fwierzbicki, jeff.allen, pjenvey, vitaly, zyasoft
Priority: normal Keywords:

Created on 2012-09-28.01:17:23 by vitaly, last changed 2014-05-22.01:53:32 by zyasoft.

Files
File name Uploaded Description Edit Remove
test_jython_readline_isue.py vitaly, 2012-09-28.01:17:22 run as: python test_jython_readline_issue.py all
issue1972.py jeff.allen, 2013-05-09.21:11:58 Variant of test allowing for Windows
Messages
msg7465 (view) Author: Vitaly Kruglikov (vitaly) Date: 2012-09-28.01:17:22
1. Make sure jython 2.5.3 is the default jython on the system (shows up first in PATH);
2. Run the attached file as:

python test_jython_readline_issue.py all

Expected: all tests complete successfully and the test app exits normally.

Actual: the test app hangs when jython 2.5.3 is the default jython (the python test app starts jython as a supbrocess; see details below); however, this test works fine when jython 2.5.2 is the default jython on the system.

Details:

The test_jython_readline_issue.py python script (executed via python)  uses subprocess.Popen() to execute this jython command-line: 

["jython", "-c", "import sys; sys.stdout.write(sys.stdin.readline()); sys.stdout.flush();"].

This jython process reads a line from stdin, writes it to stdout, then exits.

Each test in the parent process (test_jython_readline_issue.py) then writes a newline-terminated line to the jython subprocess's stdin, flushes the subprocess's stdin, attempts to read a line from the subprocess's stdout, closes the subprocess's stdin, waits for the suprocess to finish, and then returns.  The test hangs with jython 2.5.3, but works fine with jython 2.5.2

If I replace the jython subprocess command line in the test script with (python instead of jython): ["python", "-c", "import sys; sys.stdout.write(sys.stdin.readline()); sys.stdout.flush();"]

then it works fine, too.  It also works fine if the subprocess command is: ["cat"] (unix catenate command)


My System info:
==
System Version:	Mac OS X 10.7.5 (11G56)
Kernel Version:	Darwin 11.4.2
Jython 2.5.3
java version "1.6.0_35"
Java(TM) SE Runtime Environment (build 1.6.0_35-b10-428-11M3811)
Java HotSpot(TM) 64-Bit Server VM (build 20.10-b01-428, mixed mode)
Python 2.6.7
==
msg7466 (view) Author: Vitaly Kruglikov (vitaly) Date: 2012-09-28.21:10:10
Actually, the issue even shows up if I simply run the following command from a terminal (I'm using bash):

jython -c "import sys; sys.stdout.write(sys.stdin.readline()); sys.stdout.flush();"

with Jython 2.5.2, if I type in a few letters followed by newline (Return key) after starting the above command, the command echoes back the line and exits.  However, with Jython 2.5.3, the same command simply hangs.
msg7589 (view) Author: Vitaly Kruglikov (vitaly) Date: 2013-01-29.04:57:26
<PING> Any progress on this issue? It's a real show-stopper. I wish I could help, but I am not a Java/Jython developer :(

Thank you,
Vitaly
msg7618 (view) Author: Jeff Allen (jeff.allen) Date: 2013-02-04.22:48:41
I played around with this and I think resolution of #1967 is necessary (maybe sufficient) to resolve this. See discussion there.

Ditching the customised Channel implementation made readline() work correctly in the IDE (Eclipse). Run at a DOS prompt, there seems to be a problem recognising \r as a line-ending in TextIOWrapper. But you're not using DOS.
msg7641 (view) Author: Frank Wierzbicki (fwierzbicki) Date: 2013-02-07.20:59:19
Fixed in http://hg.python.org/jython/rev/ba4ec099655d Now I just need to roll a 2.5.4 rc and make sure I didn't break the app at my job :O

If I did break it then that will delay a final for sure.
msg7672 (view) Author: Edward McDowell (emcdowell) Date: 2013-02-13.20:00:34
sys.stdin.readline() hangs on Windows as well.  This problem persists in both jython 254RC1 and 27b1.  raw_input must be used to successfully read from the keyboard.  I have encountered this problem using the standalone jar distributions on both Windows 7 and Windows XP. To reproduce the problem:

Start the jython shell: java -jar jython.jar
>>> import sys
>>> sys.stdin.readline()
Type a line of input.  Will hang.
msg7673 (view) Author: Frank Wierzbicki (fwierzbicki) Date: 2013-02-13.20:49:59
Vitaly: is this working for you now?
Jeff: do you see the same problem as Edward? I don't have a windows machine handy...
msg7674 (view) Author: Jeff Allen (jeff.allen) Date: 2013-02-13.21:54:07
Yes, this happens for me in 2.7b1+ on Windows 7 x64. (I tried it when I saw Edward's post.)

As remarked in my previous post (2013-02-04) I thought fixing #1967 was necessary, but there's an extra problem at the DOS prompt. If I remember rightly, in that context InputStream provides characters one at a time and the line ending is just \r.

I find I get satisfactory results from the alternative console org.python.util.InteractiveConsole although I lose the interactive line recall, of course.

dev>jython -Dpython.console=org.python.util.InteractiveConsole
Jython 2.7b1+ (default:1af38173c96d, Feb 12 2013, 15:40:07)
[Java HotSpot(TM) 64-Bit Server VM (Sun Microsystems Inc.)] on java1.6.0_35
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.stdin.readline()
abcdef
'abcdef\n'
>>>

Am I in the best position to fix it?
msg7676 (view) Author: Frank Wierzbicki (fwierzbicki) Date: 2013-02-13.22:53:43
Jeff: you probably are in the best place to fix it (at least at the moment). I have a license from MS for Python testing purposes, but I haven't found the time to get a windows VM going. I will eventually, but other things have priority for now... If you have an interest in looking at this I'd definitely appreciate it! But don't feel obligated, I will get to it eventually.
msg7679 (view) Author: Edward McDowell (emcdowell) Date: 2013-02-13.23:42:02
Using org.python.util.InteractiveConsole resolves the problem for me too. Thanks for the workaround.
msg7681 (view) Author: Alan Kennedy (amak) Date: 2013-02-14.07:05:39
I can confirm that this happens on my windows box as well.

This is another instance of jline showing clear problems.

Here are some more

System.console().readLine only reads a single character
http://bugs.jython.org/issue1987

Windows only CTRL-C no longer works since Jython 2.5.2
http://bugs.jython.org/issue1957

KeyboardInterrupt does not catch ctrl-c
http://bugs.jython.org/issue1313

I'm sure there are more.

Is it time we considered shipping the old InteractiveConsole with the standard installation, instead of jline? Which clearly wasn't tested with all platforms, and is quite a gnarly external dependency to get to grips with, what with all of its platform specific behaviours.

If people want the fancy command line editing facilities of jline, we can make it available to them as an option.
msg7682 (view) Author: Jeff Allen (jeff.allen) Date: 2013-02-14.19:54:35
I'll see what I can do.
msg7683 (view) Author: Frank Wierzbicki (fwierzbicki) Date: 2013-02-14.21:27:36
That's too bad - I'm ok with reverting to the old interactive console given these problems. Really it would only take flipping the default in the registry file.

I tried to upgrade further than JLine 1.0 - but unless I'm just confused I think they may have re-implemented in Scala. I should go bug the JRuby guys and see if they have problems on Windows - they use a newer version (that may be Scala based).
msg7684 (view) Author: Frank Wierzbicki (fwierzbicki) Date: 2013-02-14.21:35:54
Nope ignore that - it looks like it is still implemented in Java from what I can tell. Maybe JLine 2.x can be a target of Jython3 - with extra testing on windows to make sure all is well.
msg7883 (view) Author: Philip Jenvey (pjenvey) Date: 2013-03-01.20:17:17
It sounds like disabling JLine works to solve the sys.stdin.readline example in the REPL on Windows. However I doubt this will affect the original issue that interacts with a subprocess.

The subprocess should already have JLine disabled because it's not interactive. Jython only enables JLine if stdin is a tty or you explicitly specified interactive mode
msg7886 (view) Author: Alan Kennedy (amak) Date: 2013-03-01.20:37:12
Looking at Vitaly's test code in test_jython_readline_isue.py, the following lines are key.

# Doesn't work with jython 2.5.3 (but works fine with jython 2.5.2)
_REMOTE_CMD = ["jython", "-c", "import sys; sys.stdout.write(sys.stdin.readline()); sys.stdout.flush();"]

So what change was made between 2.5.2 and 2.5.3 that stopped it from working?

Was it this change?

http://hg.python.org/jython/rev/c810cd7c8b4a

But that change happened back in 2.5b4, which was years before 2.5.2 and 2.5.3.
msg7888 (view) Author: Philip Jenvey (pjenvey) Date: 2013-03-01.20:47:22
Here's what I mean:

$ dist/bin/jython -c "import sys;print(sys.stdin.isatty(), sys._jy_interpreter)"
(True, org.python.util.JLineConsole@66e8c7db)

check.py:
import subprocess, sys
subprocess.call([sys.executable, '-c',
                 'import sys; print(sys.stdin.isatty(), sys._jy_interpreter)'])

$ dist/bin/jython check.py
(False, org.python.util.InteractiveConsole@455118a7)
msg7889 (view) Author: Philip Jenvey (pjenvey) Date: 2013-03-01.20:48:37
Also I would recommend that you use Popen.communicate here instead of manually using the p.stdin/stdout. Though I'm not sure that will affect the issue
msg7905 (view) Author: Vitaly Kruglikov (vitaly) Date: 2013-03-06.04:31:47
I can't use Popen.communicate because jython script is acting as a proxy service to a Java-based API. The client sends commands via the Jython script's stdin and reads responses from the Jython script's stdout.
msg7906 (view) Author: Vitaly Kruglikov (vitaly) Date: 2013-03-06.04:38:49
My results from running pjenvey tty checks:

$ jython --version
Jython 2.5.2
$ jython -c "import sys;print(sys.stdin.isatty(), sys._jy_interpreter)"
(True, org.python.util.JLineConsole@3a2cd728)
$ jython check.py 
(False, org.python.util.InteractiveConsole@6a1e5cf0)
msg7907 (view) Author: Vitaly Kruglikov (vitaly) Date: 2013-03-06.04:46:46
What I meant to say earlier was that I can't use Popen.communicate because *my production/real* jython script is acting as a proxy service to a Java-based API. The client sends commands via the Jython script's stdin and reads responses from the Jython script's stdout. And this back-and-forth continues for a very long period of time.
msg7992 (view) Author: Jeff Allen (jeff.allen) Date: 2013-04-13.17:19:54
I see what is happening, running interactively under the debugger, but I'm not sure what to do about it. Two things are playing badly together: our expectations of jline and the design of (mainly) TextIOWrapper.

jline.WindowsTerminal.initializeTerminal() calls the low-level C-API setconsolemode (http://msdn.microsoft.com/en-us/library/windows/desktop/ms686033(v=vs.85).aspx) to:
1. Suppress echo
2. Suppress system trapping of the ctrl-C key (Issue #1313 ?)
3. Put the input into single character mode
These changes affect all input, whether read explicitly through jline or directly through System.in.

It's the last of these which is giving us the difference from InteractiveConsole. When sys.stdin.readline calls System.in.read(), it receives just one byte. That's ok, because this is happening within TextIOWrapper.readline(), which simply accumulates characters one at a time, looking for a line ending. When the user hits return, the character received is '\r', and TextIOWrapper goes back expecting a '\n'. If you sneak in a real line feed (Alt+0010) at this point, readline() returns, but who's going to think of that?

dev>jython -J-agentlib:jdwp=transport=dt_socket,server=y,address=8000,suspend=y
Listening for transport dt_socket at address: 8000
Jython 2.7b1+ (default:96e2e620b894+, Mar 29 2013, 00:21:14)
[Java HotSpot(TM) 64-Bit Server VM (Sun Microsystems Inc.)] on java1.6.0_35
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> repr(sys.stdin.readline())
"'a\\n'"
>>>
Opening stdin in universal mode does not fix it because the check for the '\n' causes an extra byte to be read: readline returns, when you type an extra key after '\r', but it is then waiting in the input.

I should also investigate how this behaves when jython is a subprocess, in case it is differently broken.

Solution possibilities in my mind (assuming we want to keep jline):
1. Configuration of jline (but does that affect System.in).
2. Open sys.stdin so it treats just '\r' as the newline (when using jline).
3. sys.stdin needs to wrap calls to jline instead of calls to System.in.read().
4. Turn jline on and off (if possible) when reading the console, so other use of System.in is well-behaved.

I don't presently know how to do any of these, but could probably work it out, if it can be done.
msg8007 (view) Author: Jeff Allen (jeff.allen) Date: 2013-05-05.13:16:08
Philip is correct: we've mixed up two bugs here. The original problem (from Vitaly) is not connected with jline because jline is not in use when Jython is launched as a subprocess.

The superficially similar behaviour in which interactive readline() hangs (reported by Edward), is related to jline and is fixed by switching console. The critical difference is almost certainly that with jline active, System.in is effectively told to terminate lines with \r, instead of the system-specific line.separator.

There may be a common element in the strategy used by TextIOWrapper to recognise an line endings. I'm investigating the interaction between this strategy, System.in, and line endings when piping text between a process and a subprocess.
msg8008 (view) Author: Jeff Allen (jeff.allen) Date: 2013-05-09.21:11:58
The readline problem that affects interactive sessions (including scripts specified with the -c option from the shell prompt) is not the same one that Vitaly originally reports affecting subprocesses on Mac OS/X.

I have tried Vitaly's script on Windows 7. I have had to modify it because Windows does not support fcntl (or os.O_NONBLOCK). I can therefore only run the cat-block and cat-fdblock tests. Also, the close_fds option to Popen is not supported, and the jython command (being a batch file) can only be found if I use the Popen Shell=True option.

These do not hang for me on 2.5.2, 2.5.4rc1 or 2.7b1+; they hang on 2.5.3.

Although they do not hang, they fail because the returned line is terminated by \r while the sent line is terminated by \n. With that finessed they work. I've posted my variant of the test.

So it looks like this issue is fixed for Windows in 2.5.4rc1 (as long as we are talking about the right bug). I'm not closing as I don't know if the original problem (on Mac) persists. I'm de-assigning myself as I don't have a Mac.

Vitaly: have you tried this with 2.5.4rc1 on Mac?
msg8549 (view) Author: Jim Baker (zyasoft) Date: 2014-05-22.01:53:32
Appears to be resolved in 2.5.4 per Jeff Allen's note (msg8008). Also works fine on 2.7 trunk on OS X.

Closing this out since it's been over a year since msg8008
History
Date User Action Args
2014-05-22 01:53:32zyasoftsetstatus: open -> closed
resolution: fixed
messages: + msg8549
nosy: + zyasoft
2013-05-09 21:12:00jeff.allensetfiles: + issue1972.py
assignee: jeff.allen ->
messages: + msg8008
2013-05-05 13:16:08jeff.allensetmessages: + msg8007
2013-04-13 17:19:55jeff.allensetmessages: + msg7992
2013-03-06 04:46:47vitalysetmessages: + msg7907
2013-03-06 04:38:49vitalysetmessages: + msg7906
2013-03-06 04:31:47vitalysetmessages: + msg7905
2013-03-05 22:37:20amaksetkeywords: - console
2013-03-01 20:48:37pjenveysetmessages: + msg7889
2013-03-01 20:47:22pjenveysetmessages: + msg7888
2013-03-01 20:37:12amaksetmessages: + msg7886
2013-03-01 20:17:17pjenveysetnosy: + pjenvey
messages: + msg7883
2013-02-25 21:56:20amaksetkeywords: + console
2013-02-19 23:14:17fwierzbickisetversions: + Jython 2.5, - 2.5.3b2
2013-02-14 21:35:54fwierzbickisetmessages: + msg7684
2013-02-14 21:27:37fwierzbickisetmessages: + msg7683
2013-02-14 19:54:35jeff.allensetassignee: fwierzbicki -> jeff.allen
resolution: fixed -> (no value)
messages: + msg7682
2013-02-14 07:05:39amaksetmessages: + msg7681
2013-02-13 23:42:02emcdowellsetmessages: + msg7679
2013-02-13 22:53:43fwierzbickisetmessages: + msg7676
2013-02-13 22:53:00fwierzbickisetmessages: - msg7675
2013-02-13 22:52:31fwierzbickisetmessages: + msg7675
2013-02-13 21:54:07jeff.allensetmessages: + msg7674
2013-02-13 20:49:59fwierzbickisetmessages: + msg7673
2013-02-13 20:00:34emcdowellsetnosy: + emcdowell
messages: + msg7672
2013-02-07 20:59:19fwierzbickisetpriority: normal
assignee: fwierzbicki
resolution: fixed
messages: + msg7641
nosy: + fwierzbicki
2013-02-04 22:48:41jeff.allensetnosy: + jeff.allen
dependencies: + Can only read 8192 bytes
messages: + msg7618
2013-01-29 04:57:26vitalysetmessages: + msg7589
2012-11-04 14:51:39amaksetnosy: + amak
2012-09-28 21:13:10vitalysetseverity: urgent -> critical
2012-09-28 21:10:11vitalysetmessages: + msg7466
components: + Library
2012-09-28 01:17:23vitalycreate