Issue2238

execution on HPUX
Jython 2.7b3 (default:e81256215fb0, Aug 4 2014, 02:39:51)
[Java HotSpot(TM) Server VM (xyz)] on java1.7.0.09
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.system("ls")
255
>>>


execution on RHEL 5.8

Jython 2.7b3 (default:e81256215fb0, Aug 4 2014, 02:39:51)
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java1.7.0_60
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.system("ls")
jython.sh  west.py
0
>>>


same code is working on linux but not working on hpux. It returns with code 255 for any of the command tried to run using OS.system().

os.system, unlike many other aspects of the os module should be portable, since it uses standard Java resources, namely java.lang.ProcessBuilder, and not the Java Native Runtime, which doesn't have support for HPUX.

Can you tell me what is reported by the following:

$ jython27 -c "import os; print os._name"

(where jython27 is whatever is used to executed your Jython command)

It should output posix; if not, that's an issue in terms of it figuring out its underlying OS.

Next step would to get this information:

import sys; sys.builtin_module_names

which on OS X reports the following:

('sys', 'gc', '_json', 'jffi', '_sre', '_threading', 'posix', 'exceptions', '_csv', 'time', '_imp', '_py_compile', 'zipimport', 'struct', 'thread', '_functools', '_jythonlib', 'ucnhash', 'bz2', 'synchronize', 'binascii', '_codecs', '_random', '_systemrestart', 'cmath', '_marshal', 'cStringIO', '_io', '_ast', 'operator', 'jarray', 'math', 'errno', '_collections', '_hashlib', '__builtin__', 'itertools', '_weakref', 'array', 'cPickle')

Hi,

Thanks for replying, below is the output you had asked.

# sh jython.sh
Jython 2.7b3 (default:e81256215fb0, Aug 4 2014, 02:39:51)
[Java HotSpot(TM) Server VM (Hewlett-Packard Company)] on java1.7.0.09
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> print os._name
posix
>>> import sys
>>> sys.builtin_module_names
('sys', 'imp', 'gc', 'jffi', '_sre', '_threading', 'posix', '_csv', 'exceptions', 'time', '_py_compile', 'zipimport', 'struct', '_functools', 'thread', 'bz2', 'ucnhash', 'synchronize', 'binascii', '_codecs', '_random', '_systemrestart', 'cmath', '_marshal', 'cStringIO', '_io', '_ast', 'operator', 'jarray', 'math', '_collections', 'errno', '_hashlib', '__builtin__', '_weakref', 'itertools', 'cPickle', 'array')
>>>

I can see the difference between the output you had shared and what i have got on HPUX. In HPUX output, "_json" and "_jythonlib" is not there and also in place of "_imp" HPUX shows "imp".

So posix is not missing, that's the important one. You can alternatively try to use the subprocess module, and see if that works at all. There's also a small chance that this bug, recently fixed, might be causing the underlying problem:
http://bugs.jython.org/issue2205, but you are not reporting these exceptions being thrown.

As for the remaining differences, _imp, _json, _jythonlib are newly added since beta 3:
* _imp used to be called imp, but in the process of adding some functionality to the imp module (NullImporter), I chose to write it in Python and place it in Lib/imp.py rather than in Java, since that's way easier. My preference is always to reserve writing Java for code that is either performance intensive or can cause other issues, such as seen in our object proxying support.
* _json is for the port of json.c
* _jythonlib is a new module for managing some newly exposed Jython runtime internals - in part to minimize having said object proxying code in Java, using _jythonlib's new support

Below is the output you had asked,

>>> import subprocess as sub
>>> p = sub.Popen('ls',stdout=sub.PIPE,stderr=sub.PIPE)
>>> output, errors = p.communicate()
>>> print output

>>> import subprocess as sub
>>> p = sub.Popen('ls',stdout=sub.PIPE,stderr=sub.PIPE)
>>> output, errors = p.communicate()
>>> print errors

>>> import subprocess
>>> status = subprocess.call("ls"+ "", shell=True)
>>> print status
255

in the first two cases, output and errors both are coming empty where as subprocess.call returns 255 again.

Hi,

Is there any other way of achieving this? or is this a bug in jython 2.7 b3?

One thing to try is to see if it works at all with Jython 2.5.

Supporting HPUX will be challenging since we do not have access to such a system for debugging.

it does work on jython 2.5 as currently we are using it on hpux and works like magic. Due to recent fixes expected in jython 2.7, we are trying to check the compatibility on different os and it is failing on hpux with 2.7 b3 version.

Hi Jim,

As you have stated "Supporting HPUX will be challenging since we do not have access to such a system for debugging."

Can I take it as officially jython 2.7 will not be supported on HPUX?

Thanks,
Shishir

Also, please suggest when is the 2.7 b4 is planned for release? Earlier it was in Dec-2014. Also, is there any tentative timeline for official jython 2.7 version?

We are happy to commit a bugfix specific to HPUX, so long as you can assure us it works on that platform. But for platforms that we do not have access to - HPUX, AIX, AS/400, OS/390 - that's really the best we can do.

One more suggestion: you can probably write a workable version of os.system directly using ProcessBuilder, and monkeypatch that in. See our book on that approach, http://www.jython.org/jythonbook/en/1.0/Concurrency.html#interruption

Re beta 4 (last beta!) - I committed the last bug fix earlier today. We plan to release a soft beta later today. Frank may be able to get one more fix in before that time.

Hi Jim,

Thanks for the help. Could you please suggest what can be the scope of testing of bugfix? If the effort is not much, i can plan for it and verify the bug on HPUX. Please let me know your thoughts on that.
I'll also have a look into the book and see how can i use process builder to achieve os.system

Thanks,
Shishir

Shishir, we did tag the beta yesterday, so this should be released shortly.

I reviewed our implementation of os.system. Right now, it's Java code that's directly wrapping a call to Python. This is unnecessary:

    public static PyString __doc__system = new PyString(
        "system(command) -> exit_status\n\n" +
        "Execute the command (a string) in a subshell.");
    public static PyObject system(PyObject command) {
        // import subprocess; return subprocess.call(command, shell=True)
        return imp.load("subprocess").invoke("call", command, new PyObject[] {Py.True},
                                             new String[] {"shell"});
    }

(PosixModule.java)

The other thing is that it uses subprocess in its implementation. I believe the right thing to do is to rewrite as follows:

from java.lang import Runtime

def system(command)
    """system(command) -> exit_status

    Execute the command (a string) in a subshell."""
    return Runtime.exec(command).exitValue()

Easy!

This avoids using subprocess and possibly any problems in its implementation on a specific system, as we have seen in HPUX. Presumably it would just work then, given the simplicity of the above code. No monkeypatching needed as well by Shishir, always a bonus! :)

The other thing is that it better corresponds to the docs seen here: https://docs.python.org/2/library/os.html#os.system

Please try the latest in trunk given the recent fix here: https://hg.python.org/jython/rev/7ea0c8aa8b50

At the very least, the much simpler implementation of os.system (which is in Lib/subprocess.py named _os_system, since it still depends on functions in subprocess) means you might have some chance of debugging it if it's still broken.

Also I should point out that it wasn't quite as easy as I had hoped, given that Runtime.exec is also broken. Fortunately we had done all the hard work earlier in working around its problems.

Marking as fixed, if original poster does not respond