Issue2656

classification
Title: Illegal reflective access warnings from Java 9
Type: behaviour Severity: minor
Components: Core Versions: Jython 2.7
Milestone: Jython 2.7.2
process
Status: open Resolution:
Dependencies: Determine console encoding without access violation (Java 9), IllegalAccessException accessing public abstract method via PyReflectedFunction
View: 2659

View: 2662
Superseder:
Assigned To: jeff.allen Nosy List: jeff.allen
Priority: high Keywords: Java Roadmap

Created on 2018-03-11.09:29:59 by jeff.allen, last changed 2018-05-17.19:54:10 by jeff.allen.

Messages
msg11778 (view) Author: Jeff Allen (jeff.allen) Date: 2018-03-11.09:29:58
On Java 9:

PS bugs> jython -J--illegal-access=warn
WARNING: Illegal reflective access by org.python.core.PySystemState (file:/C:/Jython/2.7.2a1/jython.jar) to method java.io.Console.encoding()
WARNING: Illegal reflective access by jnr.posix.JavaLibCHelper (file:/C:/Jython/2.7.2a1/jython.jar) to method sun.nio.ch.SelChImpl.getFD()
WARNING: Illegal reflective access by jnr.posix.JavaLibCHelper (file:/C:/Jython/2.7.2a1/jython.jar) to field sun.nio.ch.FileChannelImpl.fd
WARNING: Illegal reflective access by jnr.posix.JavaLibCHelper (file:/C:/Jython/2.7.2a1/jython.jar) to field java.io.FileDescriptor.fd
WARNING: Illegal reflective access by jnr.posix.JavaLibCHelper (file:/C:/Jython/2.7.2a1/jython.jar) to field java.io.FileDescriptor.handle
Jython 2.7.2a1+ (default:d74f8c2cd56f, Feb 24 2018, 17:18:53)
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java9.0.1

By 2.7.2 it would be nice not to be the poster-child for this new JDK feature:
https://docs.oracle.com/javase/9/migrate/#GUID-7BB28E4D-99B3-4078-BDC4-FC24180CE82B

The infamous one is fixed in jnr.posix (https://github.com/jnr/jnr-posix/issues/108), but others follow if one gives -J--illegal-access=warn. I feel we should at least fix the ones we're directly responsible for.
msg11780 (view) Author: Jeff Allen (jeff.allen) Date: 2018-03-11.13:03:25
A reasonably complete set of cases (made by running the regrtest and a bit of clean-up):

By jnr.posix.JavaLibCHelper (jython.jar) to field java.io.FileDescriptor.fd
By jnr.posix.JavaLibCHelper (jython.jar) to field java.io.FileDescriptor.handle
By jnr.posix.JavaLibCHelper (jython.jar) to field sun.nio.ch.FileChannelImpl.fd
By jnr.posix.JavaLibCHelper (jython.jar) to method sun.nio.ch.SelChImpl.getFD()
By jnr.posix.util.FieldAccess (jython.jar) to field java.io.FileDescriptor.fd
By org.python.core.PyBeanProperty (jython.jar) to method com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.setEntityResolver(org.xml.sax.EntityResolver)
By org.python.core.PyBeanProperty (jython.jar) to method com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.setErrorHandler(org.xml.sax.ErrorHandler)
By org.python.core.PyReflectedFunction (jython.jar) to field java.lang.ProcessImpl.handle
By org.python.core.PyReflectedFunction (jython.jar) to method
By org.python.core.PyReflectedFunction (jython.jar) to method com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl.newSAXParser()
By org.python.core.PyReflectedFunction (jython.jar) to method com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl.setFeature(java.lang.String,boolean)
By org.python.core.PyReflectedFunction (jython.jar) to method com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.getFeature(java.lang.String)
By org.python.core.PyReflectedFunction (jython.jar) to method com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(org.xml.sax.InputSource)
By org.python.core.PyReflectedFunction (jython.jar) to method com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.setFeature(java.lang.String,boolean)
By org.python.core.PyReflectedFunction (jython.jar) to method com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.getXMLReader()
By org.python.core.PyReflectedFunction (jython.jar) to method com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.setContentHandler(org.xml.sax.ContentHandler)
By org.python.core.PyReflectedFunction (jython.jar) to method com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.setDTDHandler(org.xml.sax.DTDHandler)
By org.python.core.PyReflectedFunction (jython.jar) to method com.sun.tools.javac.api.JavacTaskImpl.call()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.net.www.protocol.jar.JarURLConnection.getDefaultUseCaches()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.net.www.protocol.jar.JarURLConnection.setDefaultUseCaches(boolean)
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.ext.Big5.newDecoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.ext.Big5.newEncoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.ext.Big5_HKSCS.newDecoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.ext.Big5_HKSCS.newEncoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.ext.EUC_CN.newDecoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.ext.EUC_CN.newEncoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.ext.EUC_JP.newDecoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.ext.EUC_JP.newEncoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.ext.EUC_KR.newDecoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.ext.EUC_KR.newEncoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.ext.GB18030.newDecoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.ext.GB18030.newEncoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.ext.IBM942.newDecoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.ext.IBM942.newEncoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.ext.IBM949.newDecoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.ext.IBM949.newEncoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.ext.IBM950.newDecoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.ext.IBM950.newEncoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.ext.ISO2022_JP.newDecoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.ext.ISO2022_JP.newEncoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.ext.ISO2022_JP_2.newDecoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.ext.ISO2022_JP_2.newEncoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.ext.ISO2022_KR.newDecoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.ext.ISO2022_KR.newEncoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.ext.SJIS_0213.newDecoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.ext.SJIS_0213.newEncoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.GBK.newDecoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.GBK.newEncoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.Johab.newDecoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.Johab.newEncoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.MS936.newEncoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.SJIS.newDecoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.nio.cs.SJIS.newEncoder()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.security.x509.X509CertImpl.toString()
By org.python.core.PyReflectedFunction (jython.jar) to method sun.util.calendar.ZoneInfo.getRawOffset()
By org.python.core.PySystemState (jython.jar) to method java.io.Console.encoding()
By org.python.modules.gc (jython.jar) to field java.util.ArrayList.elementData
By org.python.netty.util.internal.ReflectionUtil (jython.jar) to constructor java.nio.DirectByteBuffer(long,int)
By org.python.netty.util.internal.ReflectionUtil (jython.jar) to field sun.nio.ch.SelectorImpl.publicSelectedKeys
By org.python.netty.util.internal.ReflectionUtil (jython.jar) to field sun.nio.ch.SelectorImpl.selectedKeys
By org.python.netty.util.internal.ReflectionUtil (jython.jar) to method java.nio.Bits.unaligned()
msg11951 (view) Author: Jeff Allen (jeff.allen) Date: 2018-05-07.07:36:43
Upgrading jnr-posix to 3.0.44 has reduced the number of occurrences. test_java_integration passes, as Mark Reinhold's shouting no longer drowns out Rover's 42 barks.

In other places, I'm beginning to wonder if we can really quiet the code. If we insist on getting a real C API file descriptor, and it only exists in a private field, it's bound to involve private members like java.io.FileDescriptor.fd and sun.nio.ch.SelChImpl.getFD(). See also #2679, peeled off from this issue. Maybe jnr.posix, netty and the like will have to provide a DLL in the way JLine does?

Ideally, Jython would run perfectly well on a standard Java API anything further is an opt-in.  We've always stayed very close to that ideal.
msg11952 (view) Author: Jeff Allen (jeff.allen) Date: 2018-05-07.12:18:05
Updating netty helps too. https://hg.python.org/jython/rev/ead1cae7f8de

The list currently produced from:
    dist\bin\jython -J--illegal-access=warn -m test.regrtest -e >refacc.log 2>&1
and in bash:
    cat refacc.log | grep reflective | sort | uniq
with a bit of clean up is:

WARNING: An illegal reflective access operation has occurred
By jnr.posix.JavaLibCHelper$ReflectiveAccess (jnr-posix-3.0.44.jar) to field java.io.FileDescriptor.fd
By jnr.posix.JavaLibCHelper$ReflectiveAccess (jnr-posix-3.0.44.jar) to field java.io.FileDescriptor.handle
By jnr.posix.JavaLibCHelper$ReflectiveAccess (jnr-posix-3.0.44.jar) to field sun.nio.ch.FileChannelImpl.fd
By jnr.posix.JavaLibCHelper$ReflectiveAccess (jnr-posix-3.0.44.jar) to method sun.nio.ch.SelChImpl.getFD()
By jnr.posix.util.FieldAccess (jnr-posix-3.0.44.jar) to field java.io.FileDescriptor.fd
By org.python.core.PyJavaType (jython-dev.jar) to method java.lang.Object.finalize()
By org.python.core.PyReflectedFunction (jython-dev.jar) to field java.lang.ProcessImpl.handle
By org.python.modules.gc (jython-dev.jar) to field java.util.ArrayList.elementData
msg11965 (view) Author: Jeff Allen (jeff.allen) Date: 2018-05-08.22:17:45
Yay! In Java 9, Process has a pid() method so we no longer have to steal java.lang.ProcessImpl.handle

https://hg.python.org/jython/rev/61eda8fd9356
msg11967 (view) Author: Jeff Allen (jeff.allen) Date: 2018-05-10.07:56:28
The remaining violations that surface from jnr-posix are from org.python.core.io.FileIO.__int__, or from jnr.posix.JavaLibCHelper$ReflectiveAccess. In both cases as a result of calls to os.fstat (that is org.python.modules.posix.PosixModule$FstatFunction.__call__).

For the most part, these in turn are the result of calls to _pyio.open where it attempts to get the filesystem block size, although there are other root causes (fileinput$py.readline, SimpleHTTPServer$py.send_head). Note that it is only _pyio.open that does this. the pure Java io.open has a fixed idea of the ideal block size.

Seems like issues #1736 and #2320 just go on giving. C file descriptors are a terrible idea. They don't *describe* anything unless you also have access to a table we can't see. As we have worked this a few times already I incline to think that a good solution is not imminent, so not for 2.7.2.
msg11992 (view) Author: Jeff Allen (jeff.allen) Date: 2018-05-17.19:54:10
The last four really intrusive reflective access errors all come from this line where we check whether an apparent filename is actually a tty:
https://hg.python.org/jython/file/61eda8fd9356/src/org/python/util/jython.java#l355

However, there seems no easy fix here without first clarifying the logic of the run() method as a whole. See #2686.
History
Date User Action Args
2018-05-17 19:54:10jeff.allensetmessages: + msg11992
2018-05-10 07:56:29jeff.allensetmessages: + msg11967
2018-05-08 22:17:45jeff.allensetmessages: + msg11965
2018-05-07 12:18:05jeff.allensetmessages: + msg11952
2018-05-07 07:36:43jeff.allensetmessages: + msg11951
2018-04-01 08:07:44jeff.allensetdependencies: + Determine console encoding without access violation (Java 9), - Detail message is not set on PyException from PythonInterpreter
2018-04-01 08:07:19jeff.allensetdependencies: + Detail message is not set on PyException from PythonInterpreter, IllegalAccessException accessing public abstract method via PyReflectedFunction
2018-03-23 20:29:35jeff.allensettitle: Illegal reflective access warnings form Java 9 -> Illegal reflective access warnings from Java 9
2018-03-11 13:03:25jeff.allensetmessages: + msg11780
2018-03-11 13:02:54jeff.allensetmessages: - msg11779
2018-03-11 13:00:19jeff.allensetmessages: + msg11779
2018-03-11 09:30:11jeff.allensetkeywords: + Java Roadmap
2018-03-11 09:30:00jeff.allencreate