Issue2862

classification
Title: Jython fails on Linux for normal user when installed by root
Type: behaviour Severity: normal
Components: Core, Installer Versions: Jython 2.7.3
Milestone: Jython 2.7.2
process
Status: open Resolution: accepted
Dependencies: Superseder:
Assigned To: jeff.allen Nosy List: jeff.allen, r_walter
Priority: normal Keywords:

Created on 2020-02-12.13:57:04 by r_walter, last changed 2020-02-20.22:41:12 by jeff.allen.

Messages
msg12986 (view) Author: Roland Walter (r_walter) Date: 2020-02-12.13:57:03
OS: RHEL 7, openSUSE 15.1
Java 1.8.0 

I created a snapshot installer build on 2020-02-05. I used that to install jython in /usr/local/jython2.7.2b3-rwalter20200205.

Calling jython as root works. Calling jython as user roland fails with

Exception in thread "main" 
Exception: org.python.core.PyException thrown from the UncaughtExceptionHandler in thread "main"

A workaround is to remove all *$py.class files from the Lib directory of the jython installation.

This error was not there in snapshot builds before 2020-01-20.
msg12988 (view) Author: Roland Walter (r_walter) Date: 2020-02-12.14:00:54
Calling with jython -vvv gives in a German environment:

org.python FEIN initializer: 'META-INF/services/org.python.core.JythonInitializer' not found on sun.misc.Launcher$AppClassLoader@33909752
org.python FEIN initializer: 'META-INF/services/org.python.core.JythonInitializer' not found on sun.misc.Launcher$AppClassLoader@33909752
org.python.package FEIN reading cache of '/usr/local/jython2.7.2b3-rwalter20200205/jython.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/resources.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/rt.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/jsse.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/jce.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/charsets.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/ext/jfxrt.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/ext/sunec.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/ext/nashorn.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/ext/zipfs.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/ext/jaccess.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/ext/sunjce_provider.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/ext/sunpkcs11.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/ext/dnsns.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/ext/icedtea-sound.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/ext/cldrdata.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/ext/localedata.jar'
org.python FEIN threadstate: no current system state
org.python.core KONFIGURATION sys module instance created
org.python.import FEIN # trying org.python.core.exceptions as builtin module in SysPathJavaLoader
org.python.import KONFIGURATION import exceptions # builtin org.python.core.exceptions
org.python.import FEIN # trying precompiled /usr/local/jython2.7.2b3-rwalter20200205/Lib/site$py.class
Exception in thread "main" org.python.import FEIN # trying precompiled /usr/local/jython2.7.2b3-rwalter20200205/Lib/encodings/__init__$py.class

Exception: org.python.core.PyException thrown from the UncaughtExceptionHandler in thread "main"
msg12990 (view) Author: Roland Walter (r_walter) Date: 2020-02-12.14:09:49
The real workaround is to set read rights for group and other:

find Lib -type f -name "*py.class" | xargs chmod 644

This was the case for builds on Windows 10 and Linux.
msg12992 (view) Author: Jeff Allen (jeff.allen) Date: 2020-02-15.07:34:21
Thanks for trying this. This is surely a side-effect of #2044.

It was always a dubious design to cache compiled files by default into the installation directory. Compilation to there during installation is reasonable, and that the files should then be world-read, but caches specific to the application ought to be private to the user (and local or in TEMP, I think).
msg12994 (view) Author: Jeff Allen (jeff.allen) Date: 2020-02-20.18:29:18
When we compile, since #2044, we make class files private to the current user.

When we install, the files come out of a JAR. Since the process does not preserve permissions, the files have the default permissions for the installing user, 
and end up world-readable (fortuitously, almost). 

However, we also run an installation script for pip, and at that point it seems we recompile a proportion (not all) of the library. These class files become private to the installing user. I believe this happens because either :

1. the source appears newer because we touch it somehow after the class is created.
2. the source time in the class file does not match the one rounded by the JAR format. 

This is a Bad Thing generally, and not just because it has this effect. It will makes Jython try to ignore (always try to rewrite) compiled version we supply. Good discovery.

At the moment, I'm looking at the rounding theory.
msg12996 (view) Author: Jeff Allen (jeff.allen) Date: 2020-02-20.22:41:12
It seems I have a choice forming the JAR to round up or down. Let's choose down, so that the source via the JAR not older than the mtime in the $py.class.
If I log this difference, it does indeed seem to be spread from 0 to 2000 ms:

runpy$py.class time is 1,161 ms later than source
imp$py.class time is 1,599 ms later than source
os$py.class time is 1,755 ms later than source
ntpath$py.class time is 958 ms later than source
stat$py.class time is 1,271 ms later than source
genericpath$py.class time is 83 ms later than source
warnings$py.class time is 677 ms later than source
linecache$py.class time is 849 ms later than source
types$py.class time is 661 ms later than source
UserDict$py.class time is 1,864 ms later than source
_abcoll$py.class time is 1,255 ms later than source
abc$py.class time is 1,911 ms later than source
_weakrefset$py.class time is 1,317 ms later than source
jythonlib$py.class time is 1,661 ms later than source
pkgutil$py.class time is 1,786 ms later than source
ensurepip\__init__$py.class time is 1,552 ms later than source
__future__$py.class time is 1,880 ms later than source
shutil$py.class time is 1,208 ms later than source
fnmatch$py.class time is 36 ms later than source
...

Tolerating this, we should be able to avoid the recompilation that affects the permissions, without reverting the fix of the CVE.

A dry run on Linux confirms this, and I installed as root (with sudo) and can run it (sort of) as an ordinary user.

pip and setuptools installed by ensurepip, which surely involves some compilation, do not result in private files, so probably the installation of packages takes care of permissions.

The fault now is that Jython is trying to place the JAR cache in the installation directory.
History
Date User Action Args
2020-02-20 22:41:12jeff.allensetassignee: jeff.allen
type: behaviour
messages: + msg12996
2020-02-20 18:29:18jeff.allensetmessages: + msg12994
2020-02-15 07:34:21jeff.allensetpriority: normal
resolution: accepted
messages: + msg12992
nosy: + jeff.allen
milestone: Jython 2.7.2
2020-02-12 14:20:30r_waltersetcomponents: + Installer
2020-02-12 14:09:49r_waltersetmessages: + msg12990
2020-02-12 14:00:54r_waltersetmessages: + msg12988
2020-02-12 13:57:04r_waltercreate