Issue2810

C:\Users\Adam\jython\jython2>dist\bin\jython.exe -m test.regrtest -e -v test_jython_initializer
== 2.7.2a1+ (uncontrolled:+, Oct. 2 2019, 11:47:29)
== [Java HotSpot(TM) 64-Bit Server VM ("Oracle Corporation")]
== platform: java10.0.2
== encodings: stdin=cp850, stdout=cp850, FS=utf-8
== locale: default=('en_AU', 'windows-1252'), actual=(None, None)
test_jython_initializer
test_syspath_initializer (test.test_jython_initializer.TestUsingInitializer) ... Exception in thread "main" java.lang.NoSuchMethodError: java.nio.ByteBuffer.clear()Ljava/nio/ByteBuffer;
        at org.python.core.io.BufferedReader.clear(BufferedReader.java:147)
        at org.python.core.io.BufferedReader.<init>(BufferedReader.java:27)
        at org.python.core.PyFile.createBuffer(PyFile.java:227)
        at org.python.core.PyFile.file___init__(PyFile.java:185)
        at org.python.core.PyFile.file___init__(PyFile.java:178)
        at org.python.core.PyFile.<init>(PyFile.java:101)
        at org.python.core.PySystemState.<init>(PySystemState.java:237)
        at org.python.core.PySystemState.doInitialize(PySystemState.java:1222)
        at SyspathAppendingInitializer.initialize(SyspathAppendingInitializer.java:15)
        at org.python.core.PySystemState.initialize(PySystemState.java:1171)
        at org.python.core.PySystemState.initialize(PySystemState.java:1102)
        at org.python.core.PySystemState.initialize(PySystemState.java:1085)
        at org.python.core.PySystemState.initialize(PySystemState.java:1080)
        at org.python.core.PySystemState.initialize(PySystemState.java:1075)
        at org.python.util.jython.main(jython.java:531)
FAIL

Current trunk.

Seems to be due to a removed, previously deprecated, method in Java 10.

Currently this is the only regrtest failure on Windows under Java 10+.

JRuby dealt with this by switching to the buffer classes in the backport9 project 
https://github.com/jruby/jruby/issues/5450
https://github.com/headius/backport9

The JRuby fix works locally in a pretty straightforward way. There are some loose ends though, and will be too busy here to submit the patch until at least next week.

I've seen this when compiling on 9 and running on something earlier, but I don't see why this would happen compiling on 8 and running on something later. It's not so much deprecation as an unfortunate specialisation. (I've done similar things myself.)

A suspicious element is that we're looking for clear() returning ByteBuffer while the signature (both in Buffer and ByteBuffer) returns Buffer. When I look at org.python.core.io.BufferedReader.clear() with Oracle 1.8.0_211-b12, I see:

public void clear();
  Code:
     0: aload_0
     1: getfield      #5                  // Field buffer:Ljava/nio/ByteBuffer;
     4: invokevirtual #13                 // Method java/nio/ByteBuffer.clear:()Ljava/nio/Buffer;
     7: iconst_0
     8: invokevirtual #29                 // Method java/nio/Buffer.limit:(I)Ljava/nio/Buffer;
    11: pop
    12: return

I wonder if this is an openjdk thing? Possibly: http://hg.openjdk.java.net/jdk/jdk11/file/1ddf9a99e4ad/src/java.base/share/classes/java/nio/X-Buffer.java.template#l1137

Maybe the answer here is to cast ((Buffer) buffer).clear() so as to seek the right one?

Could indeed be an openjdk thing. More details:

Under OpenJDK 11

C:\Users\Adam\jython\jython5\jython>java -version
openjdk version "11.0.1" 2018-10-16
OpenJDK Runtime Environment 18.9 (build 11.0.1+13)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.1+13, mixed mode)

Before fix:
  public void clear();
    Code:
       0: aload_0
       1: getfield      #5                  // Field buffer:Ljava/nio/ByteBuffer;
       4: invokevirtual #13                 // Method java/nio/ByteBuffer.clear:()Ljava/nio/ByteBuffer;
       7: iconst_0
       8: invokevirtual #11                 // Method java/nio/ByteBuffer.limit:(I)Ljava/nio/ByteBuffer;
      11: pop
      12: return


With fix
  public void clear();
    Code:
       0: aload_0
       1: getfield      #5                  // Field buffer:Ljava/nio/ByteBuffer;
       4: invokestatic  #13                 // Method com/headius/backport9/buffer/Buffers.clearBuffer:(Ljava/nio/Buffer;)Ljava/nio/Buffer;
       7: pop
       8: aload_0
       9: getfield      #5                  // Field buffer:Ljava/nio/ByteBuffer;
      12: iconst_0
      13: invokestatic  #11                 // Method com/headius/backport9/buffer/Buffers.limitBuffer:(Ljava/nio/Buffer;I)Ljava/nio/Buffer;
      16: pop
      17: return

As you can sort of see in the bytecode, the backport9 fix is to invoke statically, using a generic to always match the type signatures.

public class Buffers {
    /**
     * Invoke Buffer.clear always using Buffer as the target, to avoid binary incompatibility on Java 8.
     *
     */
    public static <T extends Buffer> T clearBuffer(T buf) {
        return (T) buf.clear();
    }


Jython 2.7.2a1+ (uncontrolled:+, Oct. 6 2019, 20:17:35)
[OpenJDK 64-Bit Server VM (Oracle Corporation)] on java11.0.1
Type "help", "copyright", "credits" or "license" for more information.

My Java 8 is indeed Oracle

:\Users\Adam\jython>java -version
java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)

I get it on Oracle jdk 11.0.4 too though

C:\Users\Adam\jython\jython5\jython>dist\bin\jython.exe -m test.regrtest -e -v test_jython_initializer
== 2.7.2a1+ (uncontrolled:+, Oct. 7 2019, 18:17:02)
== [Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)]
== platform: java11.0.4
== encodings: stdin=cp850, stdout=cp850, FS=utf-8
== locale: default=('en_AU', 'windows-1252'), actual=(None, None)
test_jython_initializer
test_syspath_initializer (test.test_jython_initializer.TestUsingInitializer) ... Exception in thread "main" java.lang.NoSuchMethodError: java.nio.ByteBuffer.clear()Ljava/nio/ByteBuffer;
        at org.python.core.io.BufferedReader.clear(BufferedReader.java:147)
        at org.python.core.io.BufferedReader.<init>(BufferedReader.java:27)
        at org.python.core.PyFile.createBuffer(PyFile.java:227)
        at org.python.core.PyFile.file___init__(PyFile.java:185)
        at org.python.core.PyFile.file___init__(PyFile.java:178)
        at org.python.core.PyFile.<init>(PyFile.java:101)
        at org.python.core.PySystemState.<init>(PySystemState.java:237)
        at org.python.core.PySystemState.doInitialize(PySystemState.java:1222)
        at SyspathAppendingInitializer.initialize(SyspathAppendingInitializer.java:15)
        at org.python.core.PySystemState.initialize(PySystemState.java:1171)
        at org.python.core.PySystemState.initialize(PySystemState.java:1102)
        at org.python.core.PySystemState.initialize(PySystemState.java:1085)
        at org.python.core.PySystemState.initialize(PySystemState.java:1080)
        at org.python.core.PySystemState.initialize(PySystemState.java:1075)
        at org.python.util.jython.main(jython.java:531)
FAIL

======================================================================
FAIL: test_syspath_initializer (test.test_jython_initializer.TestUsingInitializer)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\Adam\jython\jython5\jython\dist\Lib\test\test_jython_initializer.py", line 23, in test_syspath_initializer
    self.assertEquals(0, subprocess.call([sys.executable, fn], env=env))
AssertionError: 0 != 1

----------------------------------------------------------------------
Ran 1 test in 0.911s


Further weirdness - other classes have this same idiom, but don't trigger the bug. A good example is org/python/util/ConsoleOutputStream.java

  protected java.lang.CharSequence getPrompt(java.nio.charset.Charset);
    Code:
       0: aload_0
       1: getfield      #4                  // Field buf:Ljava/nio/ByteBuffer;
       4: invokevirtual #13                 // Method java/nio/ByteBuffer.flip:()Ljava/nio/ByteBuffer;
       7: pop
       8: aload_1
       9: aload_0
      10: getfield      #4                  // Field buf:Ljava/nio/ByteBuffer;
      13: invokevirtual #14                 // Method java/nio/charset/Charset.decode:(Ljava/nio/ByteBuffer;)Ljava/nio/CharBuffer;
      16: astore_2
      17: aload_0
      18: getfield      #4                  // Field buf:Ljava/nio/ByteBuffer;
      21: invokevirtual #15                 // Method java/nio/ByteBuffer.compact:()Ljava/nio/ByteBuffer;
      24: pop
      25: aload_2
      26: areturn
}

getPrompt() is called on every evaluation of the Jython console. I even put println() statements in to prove it was getting called, without this crash. JIT is only thing I can think of that would cause that? Since it seems to have the same invokevirtual instruction on the same method.

PR https://github.com/jythontools/jython/pull/155

Bit uneasy about not fully explaining when it happens, but also stuck on what else to do about it.

It happens when you compile against an API in which ByteBuffer provides an implementation of clear() that declares it returns ByteBuffer (this seems to be the case from JDK 9), but you run on a JVM where ByteBuffer declares that clear() returns Buffer.

https://github.com/eclipse/jetty.project/issues/3244

The reason JRuby need the backport patch is, if I've understood correctly, that they compile on Java 9+ (to get various compiler features) but want to be able run on 8.

Maybe your IDE is compiling for you using a different profile from the ant build. Does this happen if you run "ant clean developer-build" at the prompt where javac means Java 8?

However, the first exhibit in the issue, where a Java 10 JVM is unable to find java.nio.ByteBuffer.clear()Ljava/nio/ByteBuffer; makes no sense to me. Has something messed up your JVM boot classpath, JYTHON_OPTS or similar "invisible" options?

If I set Java 11 and run a clean build, I get exactly that dump only once I reduce the environment to 8.

As noted in the PR, this was caused by jython calling out to jython and defaulting to a Java 8 in the environment. More detail there.

I can reproduce this, setting the system java to 8, although I have to work quite hard against my usual configuration scripts. Thanks for this: I agree it is good hygiene to avoid the pitfall for others.

Now in at https://hg.python.org/jython/rev/6d3659465010 .