Issue2396

classification
Title: test_threading and others fail under cygwin / sys.executable setting doesn't allow subprocess.Popen() to find jython
Type: behaviour Severity: normal
Components: Core Versions: Jython 2.7
Milestone:
process
Status: closed Resolution: accepted
Dependencies: Superseder:
Assigned To: jeff.allen Nosy List: adamburke, jeff.allen, zyasoft
Priority: Keywords: patch

Created on 2015-09-13.01:57:03 by adamburke, last changed 2015-10-22.20:52:10 by zyasoft.

Files
File name Uploaded Description Edit Remove
jython_path_2396.diff adamburke, 2015-09-13.03:00:16
Messages
msg10251 (view) Author: Adam Burke (adamburke) Date: 2015-09-13.01:57:01
Invoking the regression test from cygwin using the jython script results in some failures, eg


======================================================================
ERROR: test_finalize_with_trace (test.test_threading.ThreadTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\cygwin64\home\Adam\jython\dist\Lib\test\test_threading.py", line 314, in test_finalize_with_trace
    p = subprocess.Popen([sys.executable, "-c", """if 1:
  File "C:\cygwin64\home\Adam\jython\dist\Lib\subprocess.py", line 830, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\cygwin64\home\Adam\jython\dist\Lib\subprocess.py", line 1352, in _execute_child
    raise OSError(e.getMessage() or e)
OSError: Cannot run program "C:\cygwin64\home\Adam\jython\dist\bin\jython" (in directory "C:\cygwin64\home\Adam\jython"): CreateProcess error=193, %1 is not a valid Win32 application

======================================================================
ERROR: test_join_nondaemon_on_shutdown (test.test_threading.ThreadTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\cygwin64\home\Adam\jython\dist\Lib\test\test_threading.py", line 348, in test_join_nondaemon_on_shutdown
    p = subprocess.Popen([sys.executable, "-c", """if 1:
  File "C:\cygwin64\home\Adam\jython\dist\Lib\subprocess.py", line 830, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\cygwin64\home\Adam\jython\dist\Lib\subprocess.py", line 1352, in _execute_child
    raise OSError(e.getMessage() or e)
OSError: Cannot run program "C:\cygwin64\home\Adam\jython\dist\bin\jython" (in directory "C:\cygwin64\home\Adam\jython"): CreateProcess error=193, %1 is not a valid Win32 application

Invoking using jython.exe directly, even from cygwin, runs fine. 

Digging around a bit, it seems to be because shell invocation uses the property python.executable in PySystemState.initExecutable() rather than Py.getDefaultExecutable(). Py.getDefaultExecutable() has some platform checking that resolves this by looking explicitly for jython.exe under windows.
msg10252 (view) Author: Adam Burke (adamburke) Date: 2015-09-13.02:48:24
Jeff Allen also reported by mail

 jython -m test.regrtest -e
...
17 fails unexpected:
     test___all__ test_classpathimporter test_file2k test_httpservers
     test_jython_initializer test_netrc test_os_jy test_runpy
     test_select test_shutil test_sys test_sys_jy test_tarfile
     test_threading test_urllib2 test_zipfile test_zipimport_jy

Three of these are expected failures excluded by the suggested fix for regression tests in http://bugs.jython.org/issue2393

so I get 


14 tests failed:
    test_cmd_line test_codecs_jy test_compile_jy test_import_jy
    test_java_visibility test_json test_jython_launcher test_pdb
    test_quopri test_site test_site_jy test_threading test_warnings
    test_zipimport_support
14 fails unexpected:
    test_cmd_line test_codecs_jy test_compile_jy test_import_jy
    test_java_visibility test_json test_jython_launcher test_pdb
    test_quopri test_site test_site_jy test_threading test_warnings
    test_zipimport_support
msg10253 (view) Author: Adam Burke (adamburke) Date: 2015-09-13.03:00:16
As it seems to be very tied to cygwin invocation, one possible fix is the following four line tweak, also attached

diff -r 7ebdc6c80d55 src/shell/jython
--- a/src/shell/jython  Fri Sep 11 01:43:27 2015 +0000
+++ b/src/shell/jython  Sun Sep 13 10:00:00 2015 +0800
@@ -98,7 +98,7 @@
   # switch delimiter only after building a Unix style $CP
   CP_DELIMITER=";"
   CP=`cygpath -p -w "$CP"`
-  PRG=`cygpath -w "$PRG"`
+  PRG=`cygpath -w "$PRG"`.exe
 fi

 # ----- Execute the requested command -----------------------------------------
msg10254 (view) Author: Adam Burke (adamburke) Date: 2015-09-13.03:04:48
Meant to say - am not clear why this suddenly started failing under 2.7.1, though.
msg10255 (view) Author: Jeff Allen (jeff.allen) Date: 2015-09-13.08:19:52
I test at the prompt, not on cygwin, so it is nearly certain that no-one has tested it on cygwin and the but is as old as the code. Another difference is that when testing we invoke as dist\bin\jython, whereas after an install, as just jython, implying a path search by the cygwin shell, finding (I suppose) the script jython first.

Your patch, as I read it, ensures that the OS (which is what subprocess will talk to, I expect) finds jython.exe.

The failure of test_threading I saw isn't the one you have (and I can't repeat mine standalone).
msg10262 (view) Author: Adam Burke (adamburke) Date: 2015-09-16.13:23:10
Right, that's all the patch does. It mirrors the behaviour of Py.getDefaultExecutable() that way.
msg10344 (view) Author: Jeff Allen (jeff.allen) Date: 2015-10-10.22:02:39
I got the chance today to reproduce the bug. (I don't normally use cygwin but have it now ... sort of.)

Adam's change does indeed make the regrtest run, for me (Win 7 64-bit) with substantially the same results as at the DOS prompt. An extra failure stems from locale-related tests that are skipped on DOS but not skipped on Cygwin. That is a separate issue, I'm sure, so this patch looks good to me.

Shall I just do it?
msg10345 (view) Author: Jeff Allen (jeff.allen) Date: 2015-10-10.23:25:07
I decided this was low-risk & it would be nice to have in beta 2.

At https://hg.python.org/jython/rev/b03890ba8504
History
Date User Action Args
2015-10-22 20:52:10zyasoftsetstatus: pending -> closed
2015-10-10 23:25:08jeff.allensetstatus: open -> pending
assignee: jeff.allen
resolution: accepted
messages: + msg10345
2015-10-10 22:02:40jeff.allensetmessages: + msg10344
2015-09-22 17:42:48zyasoftsetnosy: + zyasoft
2015-09-16 13:23:11adamburkesetmessages: + msg10262
2015-09-13 08:19:52jeff.allensetnosy: + jeff.allen
messages: + msg10255
2015-09-13 03:04:48adamburkesetmessages: + msg10254
2015-09-13 03:00:17adamburkesetfiles: + jython_path_2396.diff
keywords: + patch
messages: + msg10253
2015-09-13 02:48:25adamburkesetmessages: + msg10252
2015-09-13 01:57:03adamburkecreate