Issue2362

classification
Title: Import * doesn't work on JDK9 for java.*, jdk.* namespaces
Type: behaviour Severity: normal
Components: Core Versions: Jython 2.7
Milestone: Jython 2.7.2
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: jeff.allen Nosy List: alanb, amak, darjus, fwierzbicki, jeff.allen, mr, rpan, sergei175, sfelts, zyasoft
Priority: urgent Keywords:

Created on 2015-05-28.17:56:24 by sfelts, last changed 2019-03-08.07:41:24 by jeff.allen.

Files
File name Uploaded Description Edit Remove
a.txt sergei175, 2019-01-13.15:19:59
Messages
msg10089 (view) Author: Stephen Felts (sfelts) Date: 2015-05-28.17:56:23
I had "from java.io import *"
It no longer works.  I needed to replace it with the explicit imports needed.  For example,
import java.io.ByteArrayOutputStream
import java.io.FileInputStream
import java.io.PrintStream
msg10090 (view) Author: Jim Baker (zyasoft) Date: 2015-05-28.19:18:04
Java 9 has changed jar formats; I believe this is summarized by http://openjdk.java.net/jeps/220 This means that our package scanner also needs to change.

We should start testing against Java 9, in prep for its release date in just over 1 year; Sept 22, 2016 was recently announced.

Ideally we can start supporting by2.7.1, but certainly by 2.7.2.
msg10276 (view) Author: Jim Baker (zyasoft) Date: 2015-09-22.18:52:13
This will most likely slip to 2.7.2, to be released in May 2016. I will note that I'm posting this note on Sept 22, 2015, exactly one year in advance of the planned released date for Java 9 :)

For Java 9, one thing I like is that it seems feasible to remove package scanning altogether for Java SE components, given the possibility of module reflection (http://cr.openjdk.java.net/~mr/jigsaw/spec/api/java/lang/reflect/Module.html). This seems at first glance preferable to our current approach of scanning rt.jar and also seems in the spirit of Project Jigsaw as well. We may be able to extend this to arbitrary jars in general, TBD.

It will be easy enough to test out these assumptions in Python code before  attempting changes in import logic.
msg10634 (view) Author: Jim Baker (zyasoft) Date: 2016-01-12.00:42:47
Moving to 2.7.2 - we just don't have time to get such a major change in 2.7.1. In contrast, 2.7.2 lets us get this right.

But it will be a blocker for 2.7.2, so marking urgent as a consequence.
msg10649 (view) Author: Darjus Loktevic (darjus) Date: 2016-01-15.12:16:07
Looks like Java 9 release is slipping by 6mo which means the GA will be 2017/03/23 . In a way, lucky for us this gives us a bit more time work on Java 9 support as we ourselves are slipping with 2.7.1 release of Jython. But this does not change the priority of making sure Java 9 works with Jython 2.7.2

http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-December/003237.html
msg10651 (view) Author: Jim Baker (zyasoft) Date: 2016-01-15.15:22:22
Darjus, good to see the alignment in our schedules. As I see it, we should triage this bug as the second most important bug after #2182 in 2.7.2. (Regardless, we would expect work on these two bugs to be done in parallel.)

Also we need to use the jrt filesystem (http://openjdk.java.net/jeps/220), which corresponds in any event with the existing package scanner. As far as I can tell, java.lang.reflect.Module and related introspection does not provide sufficient support to get at package hierarchies.
msg11331 (view) Author: Alan Bateman (alanb) Date: 2017-04-28.08:40:39
See also this issue in the OpenJDK JIRA where it initially looked like a startup performance issue in JDK 9:
https://bugs.openjdk.java.net/browse/JDK-8166236
msg11475 (view) Author: (rpan) Date: 2017-07-14.06:56:58
The OpenJDK bug https://bugs.openjdk.java.net/browse/JDK-8166236 was closed. Its corresponding Jython issue is: http://bugs.jython.org/issue2605
msg11930 (view) Author: Jim Baker (zyasoft) Date: 2018-05-01.02:07:28
We should look at the use of such tooling as Reflections, ClassIIndex, and Fast Class Scanner, possibly directly incorporating, depending on their licensing. Something like ClassIndex might work, we specifically need the JRT support introduced by Jigsaw: https://github.com/atteo/classindex/issues/42, I would assume the ahead-of-time aspects are not so relevant for Jython's model. ClassIndex's license is directly compatible with Jython (see similar inclusions like https://hg.python.org/jython/file/tip/ACKNOWLEDGMENTS)

Note this would be a big change for how we index!
msg11931 (view) Author: Jim Baker (zyasoft) Date: 2018-05-01.02:09:12
To clarify: a big change for our *index file format* if we go with ClassIndex. Otherwise I would expect package scanning to remain the same as something we cache on demand.
msg11932 (view) Author: Stephen Felts (sfelts) Date: 2018-05-01.02:18:29
The performance hit is visibly noticible to users.  That's why we even started doing the analysis.  It's important that this is fixed.
msg11933 (view) Author: Jim Baker (zyasoft) Date: 2018-05-02.00:29:21
We may not need a third party library. This works on Java 9 and Java 10 to list all classes in the java.base module:

from java.lang.module import ModuleFinder
from contextlib import closing

finder = ModuleFinder.ofSystem()
ref = finder.find('java.base')
with closing(ref.get().open()) as reader:
    resources = reader.list().toArray()
    print resources

Ignores such niceties as the Stream interface etc. Improve accordingly for real usage, such as writing in Java and putting into the Jython runtime.

Also it's important to know that import * works on any library that is not in a Jigsaw module. This presumably only applies for what used to be in the rt.jar, plus additions.

A third party library might result in a faster scanner however.
msg12173 (view) Author: Alan Bateman (alanb) Date: 2018-11-04.20:01:40
Accessing classes in the run-time image is very easy with JDK 9 and newer. Here's a simple example that lists all resources in all modules in the run-time image. More details on this in JEP 220.

FileSystem jrtfs = FileSystems.getFileSystem(URI.create("jrt:/"));
Path top = jrtfs.getPath("/");
Files.walk(top).forEach(System.out::println);
msg12174 (view) Author: Jeff Allen (jeff.allen) Date: 2018-11-09.02:09:21
Thanks Alan, that's a good pointer. I think the answer is probably to have a list of selected modules, then enumerate packages in each, somewhat as if each were a JAR.

Having this as a notional file system means I can write this using classes Java 7 understands. (So it compiles, but fails to find a provider for jrt: at run-time.) I wonder what happens if I compile on 7 and run on 9?
msg12175 (view) Author: Alan Bateman (alanb) Date: 2018-11-09.08:12:32
The contents of jrt file system is organized by both module and package to allow for different usages. As I understand it, jython creates its index on first usage that it used in subsequent runs. It should be able to do this without needing to have a list of modules in advance.
msg12181 (view) Author: Jeff Allen (jeff.allen) Date: 2018-11-22.07:43:17
I've made a bit of headway using the jrt: file system (thanks Alan), and following the pattern set for classes in JARs. However, asm-5.2 cannot parse classes from the JDK 9 image (see e.g. https://issues.apache.org/jira/browse/WW-4866), which it must in order to read the accessibility information.

For some reason I haven't figured out yet, we are pinned to asm-5.2, as build.xml says:
            <!-- pin to ASM 5.2 until we upgrade compilation -->
However, running with ASM 7.0 seems fine so far (on Java 9).
msg12182 (view) Author: Alan Bateman (alanb) Date: 2018-11-22.07:58:28
The class file version was rev'ed in Java SE 9, 10 and 11 so you will need to upgrade ASM.

For "accessibility information" then you will, at some point, need to also understand module membership too. An easy start would be to only index the packages that are exported to all modules. That information is in the module-info.class that newer releases of ASM can parse.
msg12184 (view) Author: Jeff Allen (jeff.allen) Date: 2018-11-25.08:14:34
I have a change set that seems to work now, with the limitations that it reads only a configured list of modules, and does no caching. I'll push this because it is quite big already, and I want to see if it works on other OSes.
https://hg.python.org/jython/rev/f662c11b87a0

I'm finding the API here paints me into a bit of a corner. It's very JAR-specific, while at the same time containing fragments useful for modules too. I'd quite like to re-work it to use URI and NIO, but it is a long-standing technically public API. It invites extension (of a sort) but I can find no reference to anyone doing so.
msg12185 (view) Author: Jeff Allen (jeff.allen) Date: 2018-11-25.08:15:19
@Alan: if an application defined modules of its own, would they show up in jrt:/modules/ too? Or in another file system I have to open from the module path? I'd like to solve this in a way that could be extended to this future possibility (by not assuming things to be unique when they're not).
msg12186 (view) Author: Alan Bateman (alanb) Date: 2018-11-25.08:57:58
You can use the jlink tool to create a run-time image containing your own modules and you'll see a directory for each of your modules in /modules. The only module that is guaranteed to be present is java.base.

The jrt file system provider was developed for tools, such as IDEs, that run on one JDK but compile/build an application using a different JDK. The IDE might running on JDK 8 but targeting JDK 11 for example. The cross targeting scenario might not be interesting for Jython so maybe you have the flexibility to use java.lang.module APIs, in which case you could use the java.lang.module APIs to easily get the set of modules that export packages and the packages that they export.

As regards a --module-path equivalent then Jython could use the ModuleFinder.of(Path ...) API to create a ModuleFinder to find modules on the module path. That will do most of the work for you in the event that you do add this option in the future.
msg12187 (view) Author: Jeff Allen (jeff.allen) Date: 2018-11-25.13:56:37
@Alan: thanks for the reply; didn't mean to disturb your Sunday.

Good, so my understanding of jlink was right. The case in point would not be a --module-path argument to Jython, so much as an application written in Python that expects to import packages from a module path given the JVM. We'd have to scan that, I assume, but not as we scan jrt.

I looked at the API Java 9 offers. However, I'm trying only to write code that still compiles on Java 7. In that sense, we are cross-targeting. I think this can wait until we no longer support pre-9 JVMs and the API can be used.
msg12306 (view) Author: Jeff Allen (jeff.allen) Date: 2019-01-08.22:42:49
At the moment, I am using the jrt filesystem provider as Alan suggests, but scanning only the modules from a short list. THis is not what I intend to end up with. However, I think it might be useful for an installation to specify a subset of modules, for performance reasons, to discourage use (I wonb't say prevent), or avoid a problem case. MAybe a list of regexes?

I'm noting here as a defect for further study that on Java 11 (and not before) the import of java.util.regex is incomplete:

PS jython-trunk> dist\bin\jython -i .\dist\Lib\test\import_star_from_java.py
Traceback (most recent call last):
  File ".\dist\Lib\test\import_star_from_java.py", line 4, in <module>
    p = Pattern.compile("foo")
NameError: name 'Pattern' is not defined
>>> import java.util.regex as r
>>> dir(r)
['MatchResult', 'PatternSyntaxException', '__name__']

Where are Matcher and Pattern? On Java 10 we see no exception and:

>>> dir(r)
['MatchResult', 'Matcher', 'Pattern', 'PatternSyntaxException', '__name__']
msg12314 (view) Author: Jeff Allen (jeff.allen) Date: 2019-01-12.14:02:30
That last defect concerning visibility was fixed in: https://hg.python.org/jython/rev/37365325d253

I now have this scanning all java and jdk paths in https://hg.python.org/jython/rev/7fd811df6d33 .

If you ask for it, it will log as:

PS jython-jvm9> dist\bin\jython -v -S -c "raise SystemExit"
*sys-package-mgr*: reading packages from /modules/java.base
*sys-package-mgr*: reading packages from /modules/java.compiler
*sys-package-mgr*: reading packages from /modules/java.datatransfer
...
*sys-package-mgr*: reading packages from /modules/jdk.accessibility
*sys-package-mgr*: reading packages from /modules/jdk.aot
...

This fixes the issue. I think more should be done in this area, but not necessarily for 2.7.2, so I'll make a separate issue of it (#2734).
msg12318 (view) Author: Sergei (sergei175) Date: 2019-01-13.15:19:59
Hi,  Jeff 
 
 Thanks. I've tested this fix by creating jython.jar and from GIT and moving this jar file to my test application https://jwork.org/dmelt/.
I've replaced lib/jython/jython.jar with the new jar file. After I've started this application, I've got java.lang.IllegalArgumentExceptio at org.objectweb.asm.ClassVisitor.

I've attached the full log file. I've got the same issue on JDK8 and JDK12.
msg12320 (view) Author: Jeff Allen (jeff.allen) Date: 2019-01-13.16:38:07
Sergei: thanks, I appreciate you trying this out.

It's difficult to be sure, but given the particular error, quite likely you have not noticed that we upgraded to ASM 7. It would be worth checking the vintage of other JARs too (ANTLR?). Or just build the installer target and use that.
msg12322 (view) Author: Sergei (sergei175) Date: 2019-01-13.22:18:27
Hi, Jeff 

 Ok, thanks for the comment. Indeed, I had to rearrange jar files inside DataMelt to make it to work with your Jython fix. 

I think everything works now. I can import Java classes using "*" inside Jython. I've tested it on JDK12 and JDK8.  I've added your fix to DataMelt (and will release a stable version of DataMelt when this Jython version will be stable).

best
History
Date User Action Args
2019-03-08 07:41:24jeff.allensetstatus: pending -> closed
2019-01-13 22:18:27sergei175setmessages: + msg12322
2019-01-13 16:38:07jeff.allensetmessages: + msg12320
2019-01-13 15:19:59sergei175setfiles: + a.txt
nosy: + sergei175
messages: + msg12318
2019-01-12 14:02:30jeff.allensetstatus: open -> pending
resolution: accepted -> fixed
messages: + msg12314
2019-01-08 22:42:49jeff.allensetmessages: + msg12306
2018-11-25 13:56:37jeff.allensetmessages: + msg12187
2018-11-25 08:57:58alanbsetmessages: + msg12186
2018-11-25 08:15:20jeff.allensetmessages: + msg12185
2018-11-25 08:14:36jeff.allensetmessages: + msg12184
2018-11-22 07:58:28alanbsetmessages: + msg12182
2018-11-22 07:43:17jeff.allensetmessages: + msg12181
2018-11-09 08:12:33alanbsetmessages: + msg12175
2018-11-09 02:09:22jeff.allensetmessages: + msg12174
2018-11-04 20:01:41alanbsetmessages: + msg12173
2018-11-04 17:32:17jeff.allensetassignee: jeff.allen
nosy: + jeff.allen
2018-05-02 00:29:23zyasoftsetmessages: + msg11933
title: Import * doesn't work on JDK9 -> Import * doesn't work on JDK9 for java.*, jdk.* namespaces
2018-05-01 02:18:29sfeltssetmessages: + msg11932
2018-05-01 02:09:12zyasoftsetmessages: + msg11931
2018-05-01 02:07:29zyasoftsetmessages: + msg11930
2017-07-14 06:56:58rpansetnosy: + rpan
messages: + msg11475
2017-04-28 08:40:40alanbsetnosy: + alanb
messages: + msg11331
2016-01-15 15:22:22zyasoftsetmessages: + msg10651
2016-01-15 12:16:08darjussetnosy: + darjus
messages: + msg10649
2016-01-12 00:42:47zyasoftsetpriority: high -> urgent
nosy: + fwierzbicki
messages: + msg10634
milestone: Jython 2.7.1 -> Jython 2.7.2
2015-12-30 00:03:53zyasoftsetpriority: normal -> high
2015-12-30 00:00:50zyasoftsetmilestone: Jython 2.7.1
2015-09-22 18:52:14zyasoftsetmessages: + msg10276
2015-09-17 15:38:34mrsetnosy: + mr
2015-06-05 10:11:21amaksetnosy: + amak
2015-05-28 19:18:05zyasoftsetpriority: normal
resolution: accepted
messages: + msg10090
nosy: + zyasoft
2015-05-28 17:56:24sfeltscreate