Issue2862

OS: RHEL 7, openSUSE 15.1
Java 1.8.0 

I created a snapshot installer build on 2020-02-05. I used that to install jython in /usr/local/jython2.7.2b3-rwalter20200205.

Calling jython as root works. Calling jython as user roland fails with

Exception in thread "main" 
Exception: org.python.core.PyException thrown from the UncaughtExceptionHandler in thread "main"

A workaround is to remove all *$py.class files from the Lib directory of the jython installation.

This error was not there in snapshot builds before 2020-01-20.

Calling with jython -vvv gives in a German environment:

org.python FEIN initializer: 'META-INF/services/org.python.core.JythonInitializer' not found on sun.misc.Launcher$AppClassLoader@33909752
org.python FEIN initializer: 'META-INF/services/org.python.core.JythonInitializer' not found on sun.misc.Launcher$AppClassLoader@33909752
org.python.package FEIN reading cache of '/usr/local/jython2.7.2b3-rwalter20200205/jython.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/resources.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/rt.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/jsse.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/jce.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/charsets.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/ext/jfxrt.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/ext/sunec.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/ext/nashorn.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/ext/zipfs.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/ext/jaccess.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/ext/sunjce_provider.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/ext/sunpkcs11.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/ext/dnsns.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/ext/icedtea-sound.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/ext/cldrdata.jar'
org.python.package FEIN reading cache of '/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/ext/localedata.jar'
org.python FEIN threadstate: no current system state
org.python.core KONFIGURATION sys module instance created
org.python.import FEIN # trying org.python.core.exceptions as builtin module in SysPathJavaLoader
org.python.import KONFIGURATION import exceptions # builtin org.python.core.exceptions
org.python.import FEIN # trying precompiled /usr/local/jython2.7.2b3-rwalter20200205/Lib/site$py.class
Exception in thread "main" org.python.import FEIN # trying precompiled /usr/local/jython2.7.2b3-rwalter20200205/Lib/encodings/__init__$py.class

Exception: org.python.core.PyException thrown from the UncaughtExceptionHandler in thread "main"

The real workaround is to set read rights for group and other:

find Lib -type f -name "*py.class" | xargs chmod 644

This was the case for builds on Windows 10 and Linux.

Thanks for trying this. This is surely a side-effect of #2044.

It was always a dubious design to cache compiled files by default into the installation directory. Compilation to there during installation is reasonable, and that the files should then be world-read, but caches specific to the application ought to be private to the user (and local or in TEMP, I think).

When we compile, since #2044, we make class files private to the current user.

When we install, the files come out of a JAR. Since the process does not preserve permissions, the files have the default permissions for the installing user, 
and end up world-readable (fortuitously, almost). 

However, we also run an installation script for pip, and at that point it seems we recompile a proportion (not all) of the library. These class files become private to the installing user. I believe this happens because either :

1. the source appears newer because we touch it somehow after the class is created.
2. the source time in the class file does not match the one rounded by the JAR format. 

This is a Bad Thing generally, and not just because it has this effect. It will makes Jython try to ignore (always try to rewrite) compiled version we supply. Good discovery.

At the moment, I'm looking at the rounding theory.

It seems I have a choice forming the JAR to round up or down. Let's choose down, so that the source via the JAR not older than the mtime in the $py.class.
If I log this difference, it does indeed seem to be spread from 0 to 2000 ms:

runpy$py.class time is 1,161 ms later than source
imp$py.class time is 1,599 ms later than source
os$py.class time is 1,755 ms later than source
ntpath$py.class time is 958 ms later than source
stat$py.class time is 1,271 ms later than source
genericpath$py.class time is 83 ms later than source
warnings$py.class time is 677 ms later than source
linecache$py.class time is 849 ms later than source
types$py.class time is 661 ms later than source
UserDict$py.class time is 1,864 ms later than source
_abcoll$py.class time is 1,255 ms later than source
abc$py.class time is 1,911 ms later than source
_weakrefset$py.class time is 1,317 ms later than source
jythonlib$py.class time is 1,661 ms later than source
pkgutil$py.class time is 1,786 ms later than source
ensurepip\__init__$py.class time is 1,552 ms later than source
__future__$py.class time is 1,880 ms later than source
shutil$py.class time is 1,208 ms later than source
fnmatch$py.class time is 36 ms later than source
...

Tolerating this, we should be able to avoid the recompilation that affects the permissions, without reverting the fix of the CVE.

A dry run on Linux confirms this, and I installed as root (with sudo) and can run it (sort of) as an ordinary user.

pip and setuptools installed by ensurepip, which surely involves some compilation, do not result in private files, so probably the installation of packages takes care of permissions.

The fault now is that Jython is trying to place the JAR cache in the installation directory.

Here we deal with the spurious recompilation that occurs when root runs Jython to install pip:
https://hg.python.org/jython/rev/2a77fdc4417c
and here relocate the cache directory to user space, so it works again in shared installations, after #2044:
https://hg.python.org/jython/rev/33ea06302061

I thought a longer name was necessary, now the cache it is out of the Jython installation context.

I considered placing it in the user's home directory. This would have the advantage that Jython run from different directories by the same user would not experience the start-up delay of rebuilding the cache. However, the down-side is a potential for interaction when different versions of Java or different versions of the JARs are in play. This seemed the safer choice to make, this close to a release.