Issue2521

classification
Title: Windows installation (all) fails on Windows 10
Type: behaviour Severity: urgent
Components: Installer Versions: Jython 2.7
Milestone: Jython 2.7.1
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: stefan.richthofer Nosy List: jeff.allen, stefan.richthofer, wymannmi, zyasoft
Priority: urgent Keywords:

Created on 2016-09-24.14:40:51 by wymannmi, last changed 2017-02-22.03:27:22 by stefan.richthofer.

Files
File name Uploaded Description Edit Remove
jython-install.log wymannmi, 2016-09-24.14:48:33 Installation Log
Messages
msg10954 (view) Author: Michael Wymann (wymannmi) Date: 2016-09-24.14:48:33
Environment: 
Windows 10 (Version 1607 Build 14393.187)

Java:
C:\Users\wym>java -version
java version "1.8.0_102"
Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode) 

1. Start installation with 
   > java -jar jython-installer-2.7.0.jar --console
2. See details in jython-install.log
3. Installation reports error 
   AttributeError: 'module' object has no attribute 'geteuid'
   (Stack-trace in jython-install.log)  

Note: JYTHON_HOME is not set.

The installation continues after this point.

However, the installation does not contain pip or any other tools.

Here is the content of the bin directory:

C:\Progs\Jython27\bin>dir
 Datenträger in Laufwerk C: ist Windows
 Volumeseriennummer: D889-2638

 Verzeichnis von C:\Progs\Jython27\bin

24.09.2016  16:34    <DIR>          .
24.09.2016  16:34    <DIR>          ..
29.04.2015  02:25         1'236'153 jython.exe
29.04.2015  02:25         2'459'136 python27.dll
               2 Datei(en),      3'695'289 Bytes
               2 Verzeichnis(se), 864'769'847'296 Bytes frei


Attached: 
- jython-install.log: The console output of a console installation
msg11031 (view) Author: Stefan Richthofer (stefan.richthofer) Date: 2017-01-20.05:44:13
I think this boils down to faulty Windows-detection in pip/compat:

# windows detection, covers cpython and ironpython
WINDOWS = (sys.platform.startswith("win") or
           (sys.platform == 'cli' and os.name == 'nt'))

I actually wonder how this could have worked on Windows with Jython at all. I see basically three possible solutions:

1) Fix pip to properly detect Windows in Jython-case
2) Let sys.platform be backed by something like PyShadowString in JyNI
3) Adjust PosixModule.geteuid() such taht it also behaves reasonable on Windows

I'd suggest to apply a combination of 1) and 2) or 1) and 3), given that 1) yields the risk of rejection by pip-devs and would take some time in best case, also restricting Jython to use newest pip version.

Regarding 2) see github.com/Stewori/JyNI/blob/master/JyNI-Java/src/JyNI/PyShadowString.java
It would have to be extended such that it also works with startswith.

Regarding 3) we would perform Windows-detection like in PosixModule.uname() and then let PosixModule.geteuid return 
System.getProperty("user.name") or System.getenv("USERNAME") (does it make a difference? Maybe System.getenv("USERNAME") would be more resilient against some faulty launcher.)

Jim: Which would you prefer? Or do you see another option?

Looking forward to forget about issues like this in Jython 3...


P.S. Am I right to think that this issue is a release blocker for Jython 2.7.1?
msg11047 (view) Author: Jeff Allen (jeff.allen) Date: 2017-01-31.22:56:48
I haven't used the console installer before. But I get the same effect ... nearly always. The time I tried to follow what it was doing by looking in the temp folder, it completed without the error, and I have pip in the bin folder. It finished:
 70 %
 80 %
Generating start scripts ...
Installing pip and setuptools
 90 %
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking setuptools
Downloading/unpacking pip
Installing collected packages: setuptools, pip
Successfully installed setuptools pip
Cleaning up...
 100 %
Do you want to show the contents of README ? [y/N] >>> N
Congratulations! You successfully installed Jython 2.7.0 to directory C:\...\issue2521\Jython270.

The visual installer has worked for me recently on Windows 10, or appeared to. And again I got an installed copy of pip. However, there's not much comfort in it as I can confirm pip doesn't work for me on Windows. It dies on geteuid() again.

I favour idea 3, but would a user name be viable in whatever use a program may subsequently do with the result? I don't understand 2.

I notice that in bash for Windows the id command (id (GNU coreutils) 8.25) comes back with a numerical uid but I don't know how.

I also fervently wish sys.platform were not as it is in Jython.
msg11048 (view) Author: Jeff Allen (jeff.allen) Date: 2017-01-31.23:41:26
Looking again at your explanation of the cause in compat.py ...
# windows detection, covers cpython and ironpython
WINDOWS = (sys.platform.startswith("win") or
           (sys.platform == 'cli' and os.name == 'nt'))

I notice that the version of pip installed by Jython 2.7.0 is older than this, and does its Windows detection in-line in locations.py, where the error arises. In this version there is some Jython-specific detection, but not consistently. Maybe idea #1 is feasible (fix pip).
msg11050 (view) Author: Stefan Richthofer (stefan.richthofer) Date: 2017-02-01.01:12:50
Jython 2.7.0 bundeled this custom pip: https://github.com/jythontools/pip.
However nowadays the Jython-friendly implementation of __get_username disappeared there too. At some point after Jython 2.7.0 this specialized pip was removed, because upstream pip was believed to be suitable for Jython now (forgot to test on Windows?).
I tried to search in https://github.com/pypa/pip to see whether __get_username ever behaved Jython-friendly, but couldn't find it in the vast history. (Also remarkable: The pip bundled with Jython 2.7.0 describes itself as 1.6 while there never was a pip version with that number.) Anyway, based on this I suppose we shouldn't go for option 1) exclusively.

>I also fervently wish sys.platform were not as it is in Jython.

This was a constant pain in JyNI as well. E.g. native ctypes uses os.name for platform-check, and presenting it with 'java' wasn't quite helpful^^ There I solved this by monkeypatching sys.platform (and os.name) once JyNI is loaded. I insert a custom PyString subclass (PyShadowString) that displays like "java", but having __eq__ accept the current platform string as well (the one stored in os._name). So on linux you would have
print sys.platform result in 'java'
while sys.platform == 'posix' still yields True.
This manipulation could somewhat break Jython's startup, which I fixed in
https://github.com/jythontools/jython/commit/92c51dcd0aa9151f0c5a374474baccc8fe662c99
Before that fix I applied the monkeypatching after startup, but that was a bit ugly.

Hope this explains option 2) a bit better.

As far as I know in Jython 3 repo this stuff (os.name, sys.platform, what else?) was aligned with CPython behavior again. So hopefully such problems will find their end there.
msg11054 (view) Author: Stefan Richthofer (stefan.richthofer) Date: 2017-02-01.20:27:03
The advantage of option 2) would be that it yields a solution to the ongoing pain with sys.platform-based platform detection.
On the other hand it would potentially break stuff that already checks for Jython, e.g. like

if sys.platform.startswith('win'):
   ... whatever
elif sys.platform == 'java':
   ... something java-specific

(However so far I never saw this; usually packages just ignore that Jython exists)

Anyway, I'd suggest to go for 3) since it yields minimal risk. We can experiment with 2) in 2.7.2 or so. I'll setup Jython on Windows and check out viability of 3)...
msg11061 (view) Author: Jeff Allen (jeff.allen) Date: 2017-02-02.07:55:43
On the face of it, 2 sounds masssively confusing for anyone who attempts to debug code that involves it. I think at least the repr should tell you it's not just a str. And going further with this thought, could it look like a tuple of the values it considers equal to itself?
msg11062 (view) Author: Stefan Richthofer (stefan.richthofer) Date: 2017-02-02.15:03:04
> On the face of it, 2 sounds masssively confusing for anyone who attempts to debug code that involves it.

Agreed. To be clear: I also don't like this solution very much and I'm concerned about possible consequences. However it still is the best approach I found so far to deal with the platform-detection problem on the whole. I once discussed it also with Jim and he agreed on that solution (at least for JyNI).

> at least the repr should tell you it's not just a str

Sure. And there should be a big fat warning in __doc__ and javadoc about it. (The PyShadowString in JyNI linked above is just a draft.)

One more thing: This only works for
sys.platform == "posix" or so,
not for
"posix" == sys.platform
(too bad, in Python '==' is asymmetric :-/ )
We would have to make PyString aware of PyShadowString (or whatever we would cal it) and have it delegate the check to the PyShadowString if compared to one.
Actually "posix" == sys.platform style is rare, but I stumbled on it e.g. in PyOpenGL. So far I just changed the source; one of the reasons to have JyOpenGL. (If this would go into Jython, it might be possible (together with the advances in #527524) for JyNI to support upstream PyOpenGL)
msg11069 (view) Author: Jeff Allen (jeff.allen) Date: 2017-02-02.19:44:09
Actually I'm warming to this.

The question of symmetry in == popped into my head later in the day too. In fact it works because str does not know what to do with your object and __eq__ is its own reverse. With obvious definitions, one can demonstrate it in pure Python (on Jython 2.7.0, of course).

>>> sys.platform = Multi('win32', 'java')
>>> sys.platform == 'java'
True
>>> 'java' == sys.platform
True
>>> sys.platform.startswith('win')
True
>>> sys.platform
Multi(set(['win32', 'java']))
>>>

Python: can't help but smile some days.

I guess this magical string should be unhashable, in case it ended up as a key in a dictionary?
msg11101 (view) Author: Stefan Richthofer (stefan.richthofer) Date: 2017-02-19.20:34:29
Linking to corresponding issue in pip bugtracker:
https://github.com/pypa/pip/issues/2979
msg11107 (view) Author: Stefan Richthofer (stefan.richthofer) Date: 2017-02-22.03:27:21
I just tried this actually on Windows and must say that I cannot reproduce it:

D:\workspace\linux\Jython\installer_test>java -jar jython-installer.jar --console
Willkommen bei Jython !
Sie sind im Begriff, Jython Version 2.7.1b3 zu installieren.
(Sie koennen die Installation jederzeit durch Eingabe von c abbrechen)
Die folgenden Sprachen sind fuer den Installationsvorgang verfuegbar: Englisch, Deutsch
Bitte waehlen Sie Ihre Sprache [E/d] >>> E
Do you want to read the license agreement now ? [y/N] >>> N
Do you accept the license agreement ? [Y/n] >>> Y
The following installation types are available:
  1. All (everything, including sources)
  2. Standard (core, library modules, demos and examples, documentation)
  3. Minimum (core)
  9. Standalone (a single, executable .jar)
Please select the installation type [ 1 /2/3/9] >>> 1
Do you want to exclude parts from the installation ? [y/N] >>> N
Please enter the target directory >>> D:\workspace\linux\Jython\installer_test\2.7.1
Unable to find directory D:\workspace\linux\Jython\installer_test\2.7.1, create it ? [Y/n] >>> Y
Your java version to start Jython is: Oracle Corporation / 1.8.0_121
Your operating system version is: Windows 10 / 10.0
Summary:
  - mod: true
  - demo: true
  - doc: true
  - src: true
  - ensurepip: true
  - JRE: C:\Program Files\Java\jre1.8.0_121
Please confirm copying of files to directory D:\workspace\linux\Jython\installer_test\2.7.1 [Y/n] >>> Y
 10 %
 20 %
 30 %
 40 %
 50 %
 60 %
 70 %
 80 %
Generating start scripts ...
Installing pip and setuptools
 90 %
Collecting setuptools
Collecting pip
Installing collected packages: setuptools, pip
Successfully installed pip-9.0.1 setuptools-28.8.0
 100 %
Do you want to show the contents of README ? [y/N] >>> N
Congratulations! You successfully installed Jython 2.7.1b3 to directory D:\workspace\linux\Jython\installer_test\2.7.1.

Then I tried to install some random package, e.g. pathlib:

D:\workspace\linux\Jython\installer_test\2.7.1>bin\pip install pathlib
Collecting pathlib
  Using cached pathlib-1.0.1.tar.gz
Installing collected packages: pathlib
  Running setup.py install for pathlib ... [?25ldone
[?25hSuccessfully installed pathlib-1.0.1

Despite from the font/codepage/whatever display issues it works well for me (Windows 10, Java 8 current Jython trunk version).

So could you please try it with current versions? Note the versions reported for pip and setuptools, i.e. pip-9.0.1 setuptools-28.8.0.

Then, if pip installation of some package fails, please describe the steps in detail; maybe it is a separate issue...

E.g. scandir setup fails via pip, because it tries to build a C-extension. Actually scandir itself is workable, becuase it has fallbacks for missing C-extension. I'going to file that in a separate issue however.

So it would be good to get an update about the state and validness of this issue, also given that it is marked urgent and everything.
History
Date User Action Args
2017-02-22 03:27:22stefan.richthofersetmessages: + msg11107
2017-02-19 20:34:29stefan.richthofersetmessages: + msg11101
2017-02-02 19:44:10jeff.allensetmessages: + msg11069
2017-02-02 15:03:05stefan.richthofersetmessages: + msg11062
2017-02-02 07:55:43jeff.allensetmessages: + msg11061
2017-02-01 20:27:03stefan.richthofersetmessages: + msg11054
2017-02-01 01:12:51stefan.richthofersetmessages: + msg11050
2017-01-31 23:41:27jeff.allensetmessages: + msg11048
2017-01-31 22:56:49jeff.allensetnosy: + jeff.allen
messages: + msg11047
2017-01-20 05:44:13stefan.richthofersetseverity: normal -> urgent
nosy: + stefan.richthofer, zyasoft
messages: + msg11031
priority: urgent
assignee: stefan.richthofer
milestone: Jython 2.7.1
type: behaviour
2016-09-24 14:48:34wymannmisetfiles: + jython-install.log
messages: + msg10954
2016-09-24 14:40:51wymannmicreate