Issue2664

When I execute the code in file with jython, the program crash with this error:
Traceback (most recent call last):
  File "/tmp/jython_program.py/Lib/runpy.py", line 161, in _run_module_as_main
  File "/tmp/jython_program.py/Lib/runpy.py", line 72, in _run_code
  File "/tmp/jython_program.py/Lib/tnsping.py", line 22, in <module>
  File "/tmp/jython_program.py/Lib/tnsping.py", line 12, in system_call
  File "/tmp/jython_program.py/Lib/tnsping.py", line 12, in system_call
  File "/tmp/jython_program.py/Lib/subprocess.py", line 988, in communicate
  File "/tmp/jython_program.py/Lib/subprocess.py", line 1023, in _communicate
IOError: Stream closed

I took a quick look at the. I have replicated the error. When I debug, stdin returns a -1 and it crashes at the call to subprocess.Popen. Doing some more research.

Mary, I read up on the documentation for the subprocess module, and learned a couple of things:

PIPE indicates that a
    new pipe to the child should be created.

stdin
        If the stdin argument is PIPE, this attribute is a file object
        that provides input to the child process.  Otherwise, it is None.
======
That being said, when I debugged your code, both PIPE and stdin were returning -1. So, I manually passed a string and a file via PIPE, and stdin is still closing the stream., apparently, with a obscure IOError.

On a side note, Jeff or Jim, i notieced that subprocess does a test for jython, it sets the stdin, stdout, and stderr to None. Is that the expected behavior?

Mary, you apparently have tnsping on your system. Is this a Bash script ;file that you created?

I was able to get your code to work, when I passed an executable Bash script I have on my system. 

I have a Bash script that starts Jython 2.72, I changed the cmd variable to read:

cmd = "/home/patrick/devjy --version

and the code runs with the timestamp with the text returned by the --version parameter. I know that the I/O can be really picky..

Basically, stdin is expecting something from the PIPE. When it doesn't receive anything, it closes.

It does not appear to be a bug.

tnsping is apparently an Oracle utility http://www.orafaq.com/wiki/Tnsping

It looks like it produces 9-ish lines of output. We can test with some other command (maybe regular ping?), but if that does not reproduce it for us then we need to know whether Mary has problems with only that command, or can reproduce it with a commonly-available command.

It is also essential I think to be sure of the version of Jython in use. The subprocess module attracts a lot of change. It seems to break easily in the face of other changes (for example changes to the console or our attempts to get encoding right). I would not be surprised to find a bug here, or that something failed on 2.7.0 that is fixed in 2.7.2a1.

There are tests in test.test_subprocess that appear to exercise PIPE streams.

Yes, we test whether the platform is Jython, and in that case the sub-process is constructed with java.lang.ProcessBuilder. This is done by making alternative definitions of several of the functions found in subprocess.py. So there's a fair amount of difference from CPython.

When I tackle things like this I use pdb, inserting set_trace() and print statements into the copy of (in this case) subprocess.py that is in dist/Lib. That way I'm not corrupting the codebase. (Then ant clean developer-build will wash it all away.) Low-tech, I know.

Thanks. I will look at it some more.\\I have a pretty good debugger in 
IntelliJ, and it pointed to the fact that stdin and PIPE were returning 
a -1, so I followed that road.

I agree. More testing with different parameters .  It is noted that 
2.7.2 actually has a clearer error code: it says IOError: Stream closed.

On 04/04/2018 03:14 PM, Jeff Allen wrote:
> Jeff Allen <ja.py@farowl.co.uk> added the comment:
>
> tnsping is apparently an Oracle utility http://www.orafaq.com/wiki/Tnsping
>

I was developing a script to demonstrate the subprocess call with a 
shell and without shell. I got the two processes working as they should. 
If if the parameters are not passed correctly, it will cause the stdin 
stream to close.

What I also discovered is that, the communicate call simply communicates 
with your child process, printing the returned information. While the 
documentation states you can pass in a string, doing so will cause the 
stream to close.

return p.communicate() <----- fine

return p.communicate(input)  <-------NOT fine. Maybe because it's trying 
to put something back in the stream.

Anyway, I'm still looking into this and study the communicate() call 
more closely.

Attaching test_subprocess.py

On 04/04/2018 06:28 PM, Patrick Palczewski wrote:
> Patrick Palczewski <psykiatris@gmail.com> added the comment:
>
> Thanks. I will look at it some more.\\I have a pretty good debugger in
> IntelliJ, and it pointed to the fact that stdin and PIPE were returning
> a -1, so I followed that road.
>
> I agree. More testing with different parameters .  It is noted that
> 2.7.2 actually has a clearer error code: it says IOError: Stream closed.
>
> On 04/04/2018 03:14 PM, Jeff Allen wrote:
>> Jeff Allen <ja.py@farowl.co.uk> added the comment:
>>
>> tnsping is apparently an Oracle utility http://www.orafaq.com/wiki/Tnsping
>>
> _______________________________________
> Jython tracker <report@bugs.jython.org>
> <http://bugs.jython.org/issue2664>
> _______________________________________

return p.communicate(input)  <-------NOT fine. Maybe because it's trying to put something back in the stream.

Not really "back in the stream". It writes to the stdin of the subprocess.

Suppose on Jython you try essentially Mary's code, but with a command we've got (like the normal ping). Do you get the output you expect from p.communicate() ?

Now from p.communicate("\n"*10) ?

And on CPython 2?

Yes, I saw in the documentation after I posted. It was pretty late. :/

I managed to get it working with a regular ping. I changed argv into a 
list, changed shell to False and added executable. I left the input 
string (as it was assigned to stdoutdata, but I didn't observe any 
newlines, so I will test again with a different string, now that I know 
it works.

Thank you for your patience.

Attaching patch files.

On 04/07/2018 01:10 AM, Jeff Allen wrote:
> Jeff Allen <ja.py@farowl.co.uk> added the comment:
>
> return p.communicate(input)  <-------NOT fine. Maybe because it's trying to put something back in the stream.
>
> Not really "back in the stream". It writes to the stdin of the subprocess.
>

Final report and recommendation:

1. If shell-True, subprocess will execute /bin/bash. Code works if an 
executable shell script and arguments are provided.

2. it is better to pass arguments in as a list. Example: cmd=['ping', 
'8.8.8.8;, '-c 10']. If you do this, you do not need to specify an 
executable= setting.

3. p.communicate() closes the stdin, stdout, stderr streams, collects 
all input for output. Inserting 10 newlines (\n) has no effect. It is 
appropriate to leave it empty, or use None.

4. The only way text can be inserted into the stdin is if the external 
program has a function to do the following: 
sys.stdout.write(sys.stdin.read(). This can be done internally with 
Jython and Python code.

I've attached the updated tnsping.py scrpt. My previous patches were 
poorly thought out and will not work. My apologies.

Recommendation: Mark this issue as invalid as the code works with no 
errors with the minor changes that were made.

On 04/07/2018 09:23 AM, Patrick Palczewski wrote:
> Patrick Palczewski <psykiatris@gmail.com> added the comment:
>
> Yes, I saw in the documentation after I posted. It was pretty late. :/
>
>

And for system programs, set shell=False.

On 04/10/2018 09:19 PM, Patrick Palczewski wrote:
> Patrick Palczewski <psykiatris@gmail.com> added the comment:
>
> Final report and recommendation:
>
> 1. If shell-True, subprocess will execute /bin/bash. Code works if an
> executable shell script and arguments are provided.
>
> 2. it is better to pass arguments in as a list. Example: cmd=['ping',
> '8.8.8.8;, '-c 10']. If you do this, you do not need to specify an
> executable= setting.
>
> 3. p.communicate() closes the stdin, stdout, stderr streams, collects
> all input for output. Inserting 10 newlines (\n) has no effect. It is
> appropriate to leave it empty, or use None.
>
> 4. The only way text can be inserted into the stdin is if the external
> program has a function to do the following:
> sys.stdout.write(sys.stdin.read(). This can be done internally with
> Jython and Python code.
>
> I've attached the updated tnsping.py scrpt. My previous patches were
> poorly thought out and will not work. My apologies.
>
> Recommendation: Mark this issue as invalid as the code works with no
> errors with the minor changes that were made.
>
> On 04/07/2018 09:23 AM, Patrick Palczewski wrote:
>> Patrick Palczewski <psykiatris@gmail.com> added the comment:
>>
>> Yes, I saw in the documentation after I posted. It was pretty late. :/
>>
>>
> Added file: http://bugs.jython.org/file1623/unnamed
> Added file: http://bugs.jython.org/file1624/tnsping.py
>
> _______________________________________
> Jython tracker <report@bugs.jython.org>
> <http://bugs.jython.org/issue2664>
> _______________________________________

Patrick: thanks for resolving this.

Bottom line: writing to the input of a program that isn't expecting it raises an IOError (in CPython too). In the original program, the default argument communicatestr=None would be a good choice.

Setting resolution here to invalid because we don't think this is a bug in Jython, and status=pending in case Mary wants to respond.