Issue2441

'NoneType' is not iterable in _nt_quote_args in spawn.py when setup() is called. This was tested using Jython 2.7.1b2. 

Here is a stacktrace:

08:42:12   File "setup.py", line 25, in <module>
08:42:12     setup(
08:42:12   File "/jenkins_data/jenkins/workspace/Util_OpsCenterBuild_Faster/ripcord/opscenterd/lib/jvm/com/datastax/opscenter/jython/2.7.1-rc2/jython-2.7.1-rc2.jar/Lib/distutils/core.py", line 152, in setup
08:42:12   File "/jenkins_data/jenkins/workspace/Util_OpsCenterBuild_Faster/ripcord/opscenterd/lib/jvm/com/datastax/opscenter/jython/2.7.1-rc2/jython-2.7.1-rc2.jar/Lib/distutils/core.py", line 152, in setup
08:42:12   File "/jenkins_data/jenkins/workspace/Util_OpsCenterBuild_Faster/ripcord/opscenterd/lib/jvm/com/datastax/opscenter/jython/2.7.1-rc2/jython-2.7.1-rc2.jar/Lib/distutils/dist.py", line 953, in run_commands
08:42:12   File "/jenkins_data/jenkins/workspace/Util_OpsCenterBuild_Faster/ripcord/opscenterd/lib/jvm/com/datastax/opscenter/jython/2.7.1-rc2/jython-2.7.1-rc2.jar/Lib/distutils/dist.py", line 972, in run_command
08:42:12   File "/jenkins_data/jenkins/workspace/Util_OpsCenterBuild_Faster/ripcord/opscenterd/lib/jvm/com/datastax/opscenter/jython/2.7.1-rc2/jython-2.7.1-rc2.jar/Lib/distutils/command/install.py", line 589, in run
08:42:12   File "/jenkins_data/jenkins/workspace/Util_OpsCenterBuild_Faster/ripcord/opscenterd/lib/jvm/com/datastax/opscenter/jython/2.7.1-rc2/jython-2.7.1-rc2.jar/Lib/distutils/cmd.py", line 326, in run_command
08:42:12   File "/jenkins_data/jenkins/workspace/Util_OpsCenterBuild_Faster/ripcord/opscenterd/lib/jvm/com/datastax/opscenter/jython/2.7.1-rc2/jython-2.7.1-rc2.jar/Lib/distutils/dist.py", line 972, in run_command
08:42:12   File "/jenkins_data/jenkins/workspace/Util_OpsCenterBuild_Faster/ripcord/opscenterd/lib/jvm/com/datastax/opscenter/jython/2.7.1-rc2/jython-2.7.1-rc2.jar/Lib/distutils/command/install_lib.py", line 101, in run
08:42:12   File "/jenkins_data/jenkins/workspace/Util_OpsCenterBuild_Faster/ripcord/opscenterd/lib/jvm/com/datastax/opscenter/jython/2.7.1-rc2/jython-2.7.1-rc2.jar/Lib/distutils/command/install_lib.py", line 140, in byte_compile
08:42:12   File "/jenkins_data/jenkins/workspace/Util_OpsCenterBuild_Faster/ripcord/opscenterd/lib/jvm/com/datastax/opscenter/jython/2.7.1-rc2/jython-2.7.1-rc2.jar/Lib/distutils/util.py", line 528, in byte_compile
08:42:12   File "/jenkins_data/jenkins/workspace/Util_OpsCenterBuild_Faster/ripcord/opscenterd/lib/jvm/com/datastax/opscenter/jython/2.7.1-rc2/jython-2.7.1-rc2.jar/Lib/distutils/spawn.py", line 40, in spawn
08:42:12   File "/jenkins_data/jenkins/workspace/Util_OpsCenterBuild_Faster/ripcord/opscenterd/lib/jvm/com/datastax/opscenter/jython/2.7.1-rc2/jython-2.7.1-rc2.jar/Lib/distutils/spawn.py", line 158, in _spawn_java
08:42:12   File "/jenkins_data/jenkins/workspace/Util_OpsCenterBuild_Faster/ripcord/opscenterd/lib/jvm/com/datastax/opscenter/jython/2.7.1-rc2/jython-2.7.1-rc2.jar/Lib/distutils/spawn.py", line 57, in _nt_quote_args
08:42:12 TypeError: 'NoneType' object is not iterable

And here is a patch: https://github.com/jythontools/jython/pull/26

Explanation: This arguments list eventually gets joined via a ' '.join([arguments]) and there is a None in that list. Obviously you can't join None to a string. I'm not sure *how* the arguments list has a None value, but I ran into this.

So it appears that the root cause of this issue is that sys.executable is None for standalone installations.

This makes calls to setup() with install/build commands broken and is the cause for the symptom in which I thought was the original problem above.

Here is where the error originates, and why a None is passed to the command list in the arguments to spawn()
https://github.com/teeohhem/jython/blob/master/Lib/distutils/util.py#L523

Here is my proposed fix: https://github.com/jythontools/jython/pull/27

Down the rabbit hole: https://mail.python.org/pipermail/python-dev/2006-March/062453.html https://mail.python.org/pipermail/python-list/2015-March/687466.html http://forum.freecadweb.org/viewtopic.php?t=8399

To summarize: in CPython if it is embedded, they provide the executable path of the executing program and not None, even though sys.executable may be None according to Python documentation. I literally could not find a case on the web for 'if sys.executable is None' except in Jython. And apparently this is a recurring issue ( http://bugs.jython.org/issue1452 ) for us.

The usage of sys.executable from the perspective of the user in an embedded system is mostly useless (you can't execute python again via uwsgi or py2exe), but it saves it from the Billion Dollar Mistake (https://en.wikipedia.org/wiki/Null_pointer).

Where I'm going with this is instead of sprinkling sys.executable is None all over the world, we stick java.home property + /bin/java(.exe).

What do you guys think?

OK, after a chat with Jim (@zyasoft, feel free to comment if i missed anythin) offline about this issue. Here's the somewhat new proposal:

As we do today we look for python.executable property. If that does not exist, instead of None, we will return sys.perfix + bin/jython(.exe) depending on the platform.

The jython file may not exist, but it gives users the ability to install/write a wrapper that would work, especially in cases where someone does want to spawn/fork from existing standalone jar. It should also work nicely with custom filesystems if proper wrappers are generated (will not implement as part of this issue).

And just to reiterate, it is still customizable via -Dpython.executable.

This diverges from Jython 2.7.0, but as i said before i could not actually find any use of it on the internet (does not mean it does not exist of course), but i think the benefit outweighs the risk (being closer to CPython implementation for one).

1. If python.executable is specified, we always use that for sys.executable. That this option is not currently honored is a bug.

2. If sys.prefix is None, sys.executable will remain None. I believe the only circumstance we cannot compute sys.prefix is in applet-like contexts, but we will not be launching subprocesses in any event. (It is possible that webstart bundled applications could see the same behavior however, but that's a further refinement we can look at later.)

3. Otherwise sys.executable is the equivalent of os.path.join(sys.prefix, "bin/jython"), or on Windows, "bin/jython.exe", which ties into nicely to a variety of ways users can actually setup.

What this convention avoids is unnecessary customization for Jython in the wider Python ecosystem. We already have enough of that for os.name vs os._name!

I think the proposed solution is fine and should fix this issue. We have to be fairly certain that sys.executable will not be None in the majority of cases to avoid running into situations as described in this ticket.

Hey Tom,

Indeed, this should cover most situations (still not all as sys.prefix can be null, though a lot of things must go wrong for that to happen (see https://github.com/jythontools/jython/blob/master/src/org/python/core/PySystemState.java#L719 ).

Committed in: https://github.com/jythontools/jython/commit/a20f5f0e0a53bac6c4e43490f0a366dbc3720900