Issue2521

Environment: 
Windows 10 (Version 1607 Build 14393.187)

Java:
C:\Users\wym>java -version
java version "1.8.0_102"
Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode) 

1. Start installation with 
   > java -jar jython-installer-2.7.0.jar --console
2. See details in jython-install.log
3. Installation reports error 
   AttributeError: 'module' object has no attribute 'geteuid'
   (Stack-trace in jython-install.log)  

Note: JYTHON_HOME is not set.

The installation continues after this point.

However, the installation does not contain pip or any other tools.

Here is the content of the bin directory:

C:\Progs\Jython27\bin>dir
 Datenträger in Laufwerk C: ist Windows
 Volumeseriennummer: D889-2638

 Verzeichnis von C:\Progs\Jython27\bin

24.09.2016  16:34    <DIR>          .
24.09.2016  16:34    <DIR>          ..
29.04.2015  02:25         1'236'153 jython.exe
29.04.2015  02:25         2'459'136 python27.dll
               2 Datei(en),      3'695'289 Bytes
               2 Verzeichnis(se), 864'769'847'296 Bytes frei


Attached: 
- jython-install.log: The console output of a console installation

I think this boils down to faulty Windows-detection in pip/compat:

# windows detection, covers cpython and ironpython
WINDOWS = (sys.platform.startswith("win") or
           (sys.platform == 'cli' and os.name == 'nt'))

I actually wonder how this could have worked on Windows with Jython at all. I see basically three possible solutions:

1) Fix pip to properly detect Windows in Jython-case
2) Let sys.platform be backed by something like PyShadowString in JyNI
3) Adjust PosixModule.geteuid() such taht it also behaves reasonable on Windows

I'd suggest to apply a combination of 1) and 2) or 1) and 3), given that 1) yields the risk of rejection by pip-devs and would take some time in best case, also restricting Jython to use newest pip version.

Regarding 2) see github.com/Stewori/JyNI/blob/master/JyNI-Java/src/JyNI/PyShadowString.java
It would have to be extended such that it also works with startswith.

Regarding 3) we would perform Windows-detection like in PosixModule.uname() and then let PosixModule.geteuid return 
System.getProperty("user.name") or System.getenv("USERNAME") (does it make a difference? Maybe System.getenv("USERNAME") would be more resilient against some faulty launcher.)

Jim: Which would you prefer? Or do you see another option?

Looking forward to forget about issues like this in Jython 3...


P.S. Am I right to think that this issue is a release blocker for Jython 2.7.1?

I haven't used the console installer before. But I get the same effect ... nearly always. The time I tried to follow what it was doing by looking in the temp folder, it completed without the error, and I have pip in the bin folder. It finished:
 70 %
 80 %
Generating start scripts ...
Installing pip and setuptools
 90 %
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking setuptools
Downloading/unpacking pip
Installing collected packages: setuptools, pip
Successfully installed setuptools pip
Cleaning up...
 100 %
Do you want to show the contents of README ? [y/N] >>> N
Congratulations! You successfully installed Jython 2.7.0 to directory C:\...\issue2521\Jython270.

The visual installer has worked for me recently on Windows 10, or appeared to. And again I got an installed copy of pip. However, there's not much comfort in it as I can confirm pip doesn't work for me on Windows. It dies on geteuid() again.

I favour idea 3, but would a user name be viable in whatever use a program may subsequently do with the result? I don't understand 2.

I notice that in bash for Windows the id command (id (GNU coreutils) 8.25) comes back with a numerical uid but I don't know how.

I also fervently wish sys.platform were not as it is in Jython.

Looking again at your explanation of the cause in compat.py ...
# windows detection, covers cpython and ironpython
WINDOWS = (sys.platform.startswith("win") or
           (sys.platform == 'cli' and os.name == 'nt'))

I notice that the version of pip installed by Jython 2.7.0 is older than this, and does its Windows detection in-line in locations.py, where the error arises. In this version there is some Jython-specific detection, but not consistently. Maybe idea #1 is feasible (fix pip).

Jython 2.7.0 bundeled this custom pip: https://github.com/jythontools/pip.
However nowadays the Jython-friendly implementation of __get_username disappeared there too. At some point after Jython 2.7.0 this specialized pip was removed, because upstream pip was believed to be suitable for Jython now (forgot to test on Windows?).
I tried to search in https://github.com/pypa/pip to see whether __get_username ever behaved Jython-friendly, but couldn't find it in the vast history. (Also remarkable: The pip bundled with Jython 2.7.0 describes itself as 1.6 while there never was a pip version with that number.) Anyway, based on this I suppose we shouldn't go for option 1) exclusively.

>I also fervently wish sys.platform were not as it is in Jython.

This was a constant pain in JyNI as well. E.g. native ctypes uses os.name for platform-check, and presenting it with 'java' wasn't quite helpful^^ There I solved this by monkeypatching sys.platform (and os.name) once JyNI is loaded. I insert a custom PyString subclass (PyShadowString) that displays like "java", but having __eq__ accept the current platform string as well (the one stored in os._name). So on linux you would have
print sys.platform result in 'java'
while sys.platform == 'posix' still yields True.
This manipulation could somewhat break Jython's startup, which I fixed in
https://github.com/jythontools/jython/commit/92c51dcd0aa9151f0c5a374474baccc8fe662c99
Before that fix I applied the monkeypatching after startup, but that was a bit ugly.

Hope this explains option 2) a bit better.

As far as I know in Jython 3 repo this stuff (os.name, sys.platform, what else?) was aligned with CPython behavior again. So hopefully such problems will find their end there.

The advantage of option 2) would be that it yields a solution to the ongoing pain with sys.platform-based platform detection.
On the other hand it would potentially break stuff that already checks for Jython, e.g. like

if sys.platform.startswith('win'):
   ... whatever
elif sys.platform == 'java':
   ... something java-specific

(However so far I never saw this; usually packages just ignore that Jython exists)

Anyway, I'd suggest to go for 3) since it yields minimal risk. We can experiment with 2) in 2.7.2 or so. I'll setup Jython on Windows and check out viability of 3)...

On the face of it, 2 sounds masssively confusing for anyone who attempts to debug code that involves it. I think at least the repr should tell you it's not just a str. And going further with this thought, could it look like a tuple of the values it considers equal to itself?

> On the face of it, 2 sounds masssively confusing for anyone who attempts to debug code that involves it.

Agreed. To be clear: I also don't like this solution very much and I'm concerned about possible consequences. However it still is the best approach I found so far to deal with the platform-detection problem on the whole. I once discussed it also with Jim and he agreed on that solution (at least for JyNI).

> at least the repr should tell you it's not just a str

Sure. And there should be a big fat warning in __doc__ and javadoc about it. (The PyShadowString in JyNI linked above is just a draft.)

One more thing: This only works for
sys.platform == "posix" or so,
not for
"posix" == sys.platform
(too bad, in Python '==' is asymmetric :-/ )
We would have to make PyString aware of PyShadowString (or whatever we would cal it) and have it delegate the check to the PyShadowString if compared to one.
Actually "posix" == sys.platform style is rare, but I stumbled on it e.g. in PyOpenGL. So far I just changed the source; one of the reasons to have JyOpenGL. (If this would go into Jython, it might be possible (together with the advances in #527524) for JyNI to support upstream PyOpenGL)

Actually I'm warming to this.

The question of symmetry in == popped into my head later in the day too. In fact it works because str does not know what to do with your object and __eq__ is its own reverse. With obvious definitions, one can demonstrate it in pure Python (on Jython 2.7.0, of course).

>>> sys.platform = Multi('win32', 'java')
>>> sys.platform == 'java'
True
>>> 'java' == sys.platform
True
>>> sys.platform.startswith('win')
True
>>> sys.platform
Multi(set(['win32', 'java']))
>>>

Python: can't help but smile some days.

I guess this magical string should be unhashable, in case it ended up as a key in a dictionary?

Linking to corresponding issue in pip bugtracker:
https://github.com/pypa/pip/issues/2979

I just tried this actually on Windows and must say that I cannot reproduce it:

D:\workspace\linux\Jython\installer_test>java -jar jython-installer.jar --console
Willkommen bei Jython !
Sie sind im Begriff, Jython Version 2.7.1b3 zu installieren.
(Sie koennen die Installation jederzeit durch Eingabe von c abbrechen)
Die folgenden Sprachen sind fuer den Installationsvorgang verfuegbar: Englisch, Deutsch
Bitte waehlen Sie Ihre Sprache [E/d] >>> E
Do you want to read the license agreement now ? [y/N] >>> N
Do you accept the license agreement ? [Y/n] >>> Y
The following installation types are available:
  1. All (everything, including sources)
  2. Standard (core, library modules, demos and examples, documentation)
  3. Minimum (core)
  9. Standalone (a single, executable .jar)
Please select the installation type [ 1 /2/3/9] >>> 1
Do you want to exclude parts from the installation ? [y/N] >>> N
Please enter the target directory >>> D:\workspace\linux\Jython\installer_test\2.7.1
Unable to find directory D:\workspace\linux\Jython\installer_test\2.7.1, create it ? [Y/n] >>> Y
Your java version to start Jython is: Oracle Corporation / 1.8.0_121
Your operating system version is: Windows 10 / 10.0
Summary:
  - mod: true
  - demo: true
  - doc: true
  - src: true
  - ensurepip: true
  - JRE: C:\Program Files\Java\jre1.8.0_121
Please confirm copying of files to directory D:\workspace\linux\Jython\installer_test\2.7.1 [Y/n] >>> Y
 10 %
 20 %
 30 %
 40 %
 50 %
 60 %
 70 %
 80 %
Generating start scripts ...
Installing pip and setuptools
 90 %
Collecting setuptools
Collecting pip
Installing collected packages: setuptools, pip
Successfully installed pip-9.0.1 setuptools-28.8.0
 100 %
Do you want to show the contents of README ? [y/N] >>> N
Congratulations! You successfully installed Jython 2.7.1b3 to directory D:\workspace\linux\Jython\installer_test\2.7.1.

Then I tried to install some random package, e.g. pathlib:

D:\workspace\linux\Jython\installer_test\2.7.1>bin\pip install pathlib
Collecting pathlib
  Using cached pathlib-1.0.1.tar.gz
Installing collected packages: pathlib
  Running setup.py install for pathlib ... [?25ldone
[?25hSuccessfully installed pathlib-1.0.1

Despite from the font/codepage/whatever display issues it works well for me (Windows 10, Java 8 current Jython trunk version).

So could you please try it with current versions? Note the versions reported for pip and setuptools, i.e. pip-9.0.1 setuptools-28.8.0.

Then, if pip installation of some package fails, please describe the steps in detail; maybe it is a separate issue...

E.g. scandir setup fails via pip, because it tries to build a C-extension. Actually scandir itself is workable, becuase it has fallbacks for missing C-extension. I'going to file that in a separate issue however.

So it would be good to get an update about the state and validness of this issue, also given that it is marked urgent and everything.

pip and setuptools now works, as of recent changes!

C:\Users\jimba>c:\jython2.7.1-rc1-A\bin\pip install yolk
Collecting yolk
  Downloading yolk-0.4.3.tar.gz (86kB)
    100% |################################| 92kB 100kB/s ta 0:00:01
[?25hRequirement already satisfied: setuptools in c:\jython2.7.1-rc1-a\lib\site-packages (from yolk)
Installing collected packages: yolk
  Running setup.py install for yolk ... [?25ldone
[?25hSuccessfully installed yolk-0.4.3

C:\Users\jimba>c:\jython2.7.1-rc1-A\bin\yolk.exe --list
pip 9.0.1 has no metadata
pytz 2016.10 has no metadata
setuptools 28.8.0 has no metadata
wsgiref         - 0.1.2        - active development (c:\jython2.7.1-rc1-a\lib)
yolk            - 0.4.3        - active

Yes, that's my way of suggesting this we may be ready for a release candidate! There are some other issues that might still be worthwhile, but windows support of pip was the last one that was really outstanding.

NOT FIXED YET.

Looks like an additional step is needed to rename or remove a TEMP folder.

Also got an error after installing on Windows 10.

  File "E:\jython2.7.0\Lib\site-packages\pip\locations.py", line 90, in _get_build_prefix
    if file_uid != os.geteuid():
AttributeError: 'module' object has no attribute 'geteuid'

My fix was to rename the TEMP\pip_build_Thad folder to XXX_pip_build which had only 1 file under it called 'pip-delete-this-directory.txt' coincidentally.

After the renamed folder, I then tried 'pip install yolk' and it worked.

I found the fix after reading about the issue here: https://github.com/robotframework/Selenium2Library/issues/409

Thad, could you please try the following:

in Lib/os.py

right after line 'import sys, errno' add the line
sys.platform.addtarget("pip\.compat\.*")
(Remember to delete os.py's corresponding class file, so this change finds its way into bytecode; won't work if Lib is packaged into jar!)

It should make pip detect Windows properly and thus not call getusereid on non-posix systems.
Please report back if that solves the problem, then I will include it into Jython. (I would try myself if I could reproduce the issue)

Meanwhile I will try to reproduce this error. Why do I not see it? Are you logged in as a different user than the temp-folder belongs to or something otherwise special?

Should rather be sys.platform.addtarget("pip\.compat.*")

The reason MIGHT just be that I am an advanced Windows user who has MOVED
their normal Temp folder to a faster Raid array since I work with Big Data
sometimes ? :)   I set local user variables in Windows 10...
TEMP=E:\TEMP_THAD and TMP=E:\TEMP_THAD

On Wed, Mar 1, 2017 at 6:53 AM Stefan Richthofer <report@bugs.jython.org>
wrote:

>
> Stefan Richthofer added the comment:
>
> Should rather be sys.platform.addtarget("pip\.compat.*")
>
> _______________________________________
> Jython tracker <report@bugs.jython.org>
> <http://bugs.jython.org/issue2521>
> _______________________________________
>

tried fix...

Microsoft Windows [Version 10.0.14393]
(c) 2016 Microsoft Corporation. All rights reserved.

C:\Users\Thad>pip install yolk
Exception in thread "main" Traceback (most recent call last):
  File "E:\jython2.7.0\Lib\site.py", line 68, in <module>
    import os
  File "E:\jython2.7.0\Lib\os.py", line 27, in <module>
    sys.platform.addtarget("pip\.compat.*")
AttributeError: 'str' object has no attribute 'addtarget'

The functionality I referred to has been added to Jython just some days ago. There is no way to get it working in Jython 2.7.0 (actually there maybe _is_ a way by monkeypatching sys.platform; if you really really need it in Jython 2.7.0 I could give advice on that path). Anyway, fixes we are talking about here are 2.7.1 only (2.7.0 is done) and I assumed you were using the newest build from trunk (which I recommend).

That said, if prefer to avoid building Jython yourself, luckily Frank has  created a soft release candidate recently that is also capable of the potential fix I suggested. Find the links at https://sourceforge.net/p/jython/mailman/message/35692550/

I'll see if I can reproduce the issue with the user variables you describe.

Fixed !

2.7.1rc1 works out of the box for me now with 'pip install yolk'

(I also observed the E:\THAD_TEMP\pip_build_xxx folders getting created then deleted as yolk was downloaded and installed)

Good Job.  Looking forward to 2.7.1 release!

Microsoft Windows [Version 10.0.14393]
(c) 2016 Microsoft Corporation. All rights reserved.

C:\Users\Thad>pip install yolk
Collecting yolk
  Downloading yolk-0.4.3.tar.gz (86kB)
    100% |################################| 92kB 365kB/s ta 0:00:01
[?25hRequirement already satisfied: setuptools in e:\jython2.7.1rc1\lib\site-packages (from yolk)
Installing collected packages: yolk
  Running setup.py install for yolk ... [?25ldone
[?25hSuccessfully installed yolk-0.4.3

Thad:
Good to hear it's really fixed!