Issue2700

Hi,

I am using openhab with the jsr223 scripting engine & jython to do some home automation.
With jython 2.7.1 (latest version from maven) it seems that modules do not get imported properly.

I have two files (A&B) which import a self written library:
   import testModule

In the library I set a property:

   import testModule
   print( 'TESTPROP: {}'.format(hasattr(testModule, 'TESTPROP')))
   testModule.TESTPROP = 'TEST'
   print( 'TESTPROP: {}'.format(hasattr(testModule, 'TESTPROP')))

This results in the following output:

   TESTPROP: 0
   TESTPROP: 1
   
   TESTPROP: 0
   TESTPROP: 1

It seems that the library does get loaded multiple times.
With jython 2.7.0 the output is correct as the library only gets loaded once:

   TESTPROP: 0
   TESTPROP: 1


Steps to reproduce:
- Download Openhab2.3 from https://www.openhab.org/
- Copy Jython Standalone to Openhab2\runtime\lib\boot\
  Use 2.7.1 to reproduce problem, use 2.7.0 to see it work
- unzip attached file into Folder Openhab2\conf\
- Edit Openhab2\runtime\bin\setenv.bat Line 118:
  Insert:
    -Dpython.path=%OPENHAB_CONF%/automation/lib ^
  to add library path to JAVA_OPTS
- Start openhab (start.bat)
- See https://www.openhab.org/docs/tutorial/1sttimesetup.html and select Expert
- Activate JSR223 scripting (Next-Gen Rule Engine)
  https://www.openhab.org/docs/configuration/rules-ng.html#installation
- Stop Openhab and restart and watch output in console window
  Scripts are located in automation/jsr223
  Library is located in automation/lib

So, first step is to find a minimal configuration to reproduce this, ideally independently from openhab. I quickly tried to reproduce with a plain Jython (dev tip):

ModuleA.py:
print "A imported"

ModuleB.py:

import ModuleA

print "B import started"
print( 'TESTPROP: {}'.format(hasattr(ModuleA, 'TESTPROP')))
ModuleA.TESTPROP = 'TEST'
print( 'TESTPROP: {}'.format(hasattr(ModuleA, 'TESTPROP')))
print "B imported"


Output is as expected:

A imported
B import started
TESTPROP: 0
TESTPROP: 1
B imported



So this is triggered by something more advanced.

Hi,
this was not exactly what I was trying to do.
Could you try this:

Folder TestModule
   __init__.py:
      import test_file
   test_file.py:
      import testModule
      print( 'TESTPROP: {}'.format(hasattr(testModule, 'TESTPROP')))
      testModule.TESTPROP = 'TEST'
      print( 'TESTPROP: {}'.format(hasattr(testModule, 'TESTPROP')))

ModuleA.py:
   import TestModule

ModuleB.py:
   import TestModule


Thank you very much for your quick reply!

I notice that testModule (TestModule) is differently capitalised here and there and that we are on Windows. I think Jython will make some attempt to ignore case when finding files on Windows, but it won't ignore case internally.

With exactly the capitalisation shown I get:

PS 271-sa> python ModuleA.py
Traceback (most recent call last):
  File "ModuleA.py", line 1, in <module>
    import TestModule
  File "C:\Users\Jeff\Documents\Jython\271-sa\TestModule\__init__.py", line 1, in <module>
    import test_file
  File "C:\Users\Jeff\Documents\Jython\271-sa\TestModule\test_file.py", line 1, in <module>
    import testModule
ImportError: No module named testModule
PS 271-sa> java -Xmx512m -Xss2560k -cp "C:\Jython\2.7.1-sa\jython-standalone-2.7.1.jar" org.python.util.jython ModuleA.py
Traceback (most recent call last):
  File "ModuleA.py", line 1, in <module>
    import TestModule
  File "C:\Users\Jeff\Documents\Jython\271-sa\TestModule\__init__.py", line 1, in <module>
    import test_file
  File "C:\Users\Jeff\Documents\Jython\271-sa\TestModule\test_file.py", line 1, in <module>
    import testModule
ImportError: No module named testModule

Now with the latest Jython I get:

PS 271-sa> java -Xmx512m -Xss2560k -cp "C:\Users\Jeff\.m2\repository\org\python\jython\2.7.2a1+\jython-2.7.2a1+.jar;.\extlibs\*;"  org.python.util.jython ModuleA.py
TESTPROP: 0
TESTPROP: 1
TESTPROP: 1
TESTPROP: 1

But I can't help thinking that's not right, maybe because it is "differently tolerant" of capitalisation. With consistent capitalisation I get:

PS 271-sa> python ModuleA.py
TESTPROP: False
TESTPROP: True
PS 271-sa> java -Xmx512m -Xss2560k -cp "C:\Jython\2.7.1-sa\jython-standalone-2.7.1.jar" org.python.util.jython ModuleA.py
TESTPROP: 0
TESTPROP: 1
PS 271-sa> java -Xmx512m -Xss2560k -cp "C:\Users\Jeff\.m2\repository\org\python\jython\2.7.2a1+\jython-2.7.2a1+.jar;.\extlibs\*;"  org.python.util.jython ModuleA.py
TESTPROP: 0
TESTPROP: 1

This looks right to me, except that it illustrates #2691.

Hi, you are right - I did mess up the capitalization in my example.
However - if you look in the attached automation.zip you'll see that I have it correct there. Should have copy-pasted. Sorry! :S

Correct should be the following:

Folder TestModule
   __init__.py:
      import test_file
   test_file.py:
      import TestModule
      print( 'TESTPROP: {}'.format(hasattr(testModule, 'TESTPROP')))
      testModule.TESTPROP = 'TEST'
      print( 'TESTPROP: {}'.format(hasattr(testModule, 'TESTPROP')))

ModuleA.py:
   import TestModule

ModuleB.py:
   import ModuleA
   import TestModule



Upon execution of ModuleB I then would expect the output to be (behavior of)
   TESTPROP: 0
   TESTPROP: 1

In my openhab instance it gives me with 2.7.1 this:
   TESTPROP: 0
   TESTPROP: 1

   TESTPROP: 0
   TESTPROP: 1

Ah, I wondered what ModuleB was for. If we are standardising on uppercase "TestModule", then your text for test_file is still wrong.

Resorting to bash for a moment, I can show you without mistakes that I have exactly this:

$ tail -n +1 -- Module*.py TestModule/*.py
==> ModuleA.py <==
import TestModule

==> ModuleB.py <==
import ModuleA
import TestModule

==> TestModule/__init__.py <==
import test_file

==> TestModule/test_file.py <==
import TestModule
print( 'TESTPROP: {}'.format(hasattr(TestModule, 'TESTPROP')))
TestModule.TESTPROP = 'TEST'
print( 'TESTPROP: {}'.format(hasattr(TestModule, 'TESTPROP')))

Now in a PowerShell I get:
PS 271-sa> java -Xmx512m -Xss2560k -cp "C:\Jython\2.7.1-sa\jython-standalone-2.7.1.jar" org.python.util.jython ModuleB.py
TESTPROP: 0
TESTPROP: 1
PS 271-sa>

My setup is:
PS 271-sa> java -Xmx512m -Xss2560k -cp "C:\Jython\2.7.1-sa\jython-standalone-2.7.1.jar" org.python.util.jython
Jython 2.7.1 (default:0df7adb1b397, Jun 30 2017, 19:02:43)
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java1.7.0_60

But on a quick check I get the same behaviour with Java 8 and 9.

I'm puzzled by "In my openhab instance ...". If you try your own example at the prompt, do you not get the same as I? And if it's different in some other environment ... well, it's an integration problem so it could be either party. I would be nice to have them work together. But I think we're barking up the wrong tree doing this at the Windows prompt.

The JSR-223 wrapper can make a material difference (usually in the area of console i/o). Maybe openhab creates and disposes of interpreters, which is somewhat like reloading sys and sys.modules. At the Jython prompt I can do this:

>>> from javax.script import ScriptEngineManager as SEM
>>> sem = SEM()
>>> se = sem.getEngineByName('jython')
>>> se.eval("import sys; sys.path[:0]=['']; print sys.path")
['', 'C:\\Jython\\2.7.1-sa\\Lib', 'C:\\Jython\\2.7.1-sa\\jython-standalone-2.7.1.jar\\Lib', '__classpath__', '__pyclasspath__/', 'C:\\Users\\Jeff\\.local\\lib\\jython2.7\\site-packages']
>>> se.eval("import ModuleB")
TESTPROP: 0
TESTPROP: 1
>>> se.eval("import ModuleB")
>>>

The second import has no effect as you'd expect. But create yourself another interpreter (se2 = ...), and you have to do all that afresh, and you get the output again.

I think I can explain this now. I repeated the last sequence with 2.7.0 and the results are different.

PS 271-sa> java -Xmx512m -Xss2560k -cp "C:\Jython\2.7.0-sa\jython-standalone-2.7.0.jar" org.python.util.jython
Jython 2.7.0 (default:9987c746f838, Apr 29 2015, 02:25:11)
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java1.7.0_60
Type "help", "copyright", "credits" or "license" for more information.
>>> from javax.script import ScriptEngineManager as SEM
>>> sem = SEM()
>>> se = sem.getEngineByName('jython')
>>> se.eval('import sys; print sys.path')
['', 'C:\\Jython\\2.7.0-sa\\Lib', 'C:\\Jython\\2.7.0-sa\\jython-standalone-2.7.0.jar\\Lib', '__classpath__', '__pyclasspath__/']
>>>

The first difference to notice here is that I already have '' on the sys.path. Now try:
>>> se.eval("import ModuleB")
TESTPROP: 0
TESTPROP: 1
>>> se.eval("import ModuleB")
>>> se2 = sem.getEngineByName('jython')
>>> se2.eval("import ModuleB")

What's more, even in the main interpreter:
>>> import ModuleB
produces no output. This is because in 2.7.0 all the interpreter instances, while providing you distinct namespaces, share a lot of state including sys.modules. We decided this was incorrect: you are owed a clean interpreter (a clean sys module) with no baggage from other interpreters.

It's still only a theory for you to check, but I'd say openhab creates two interpreters and both of them run your code. Maybe this is something you can prevent. Or maybe it is harmless.

A way to ask "which interpreter am I" is to insert a use of System.identityHashCode in your test_file.py:

==> TestModule/test_file.py <==
import TestModule

from java.lang import System
print 'TestModule = ', System.identityHashCode(TestModule)

print( 'TESTPROP: {}'.format(hasattr(TestModule, 'TESTPROP')))
TestModule.TESTPROP = 'TEST'
print( 'TESTPROP: {}'.format(hasattr(TestModule, 'TESTPROP')))

Then I get (in a 2.7.1 Jython console, after preparing two interpreters):

>>> se.eval("import sys; sys.path[:0]=['']")
>>> se.eval("import ModuleB")
TestModule =  324099565
TESTPROP: 0
TESTPROP: 1
>>> se2.eval("import sys; sys.path[:0]=['']")
>>> se2.eval("import ModuleB")
TestModule =  192838628
TESTPROP: 0
TESTPROP: 1

Maybe you could try the same test_file.py under openhab.

Hi,
your theory seems plausible.
I modified the file as you suggested and get the following output:

jython 2.7.0:
   Loading testModule
   TestModule =  1514880709
   TESTPROP: 0
   TESTPROP: 1
   Loaded testModule
   A Loaded
   B Loaded

jython 2.7.1:
   Loading testModule
   TestModule =  2059507207
   TESTPROP: 0
   TESTPROP: 1
   Loaded testModule
   A Loaded
   Loading testModule
   TestModule =  188259898
   TESTPROP: 0
   TESTPROP: 1
   Loaded testModule
   B Loaded

Is it possible to configure this behaviour?
This makes the use of librarys in openhab kind of pointless ... .

Glad I was able to guess right.

The use of library modules has the same point as ever: you get to write the code once and use it from lots of places. This statement in the openHAB Jython scripting guide, however, is incorrect (from 2.7.1) in the third paragraph: https://github.com/OH-Jython-Scripters/openhab2-jython#jython-scripts-and-modules .

That was true up to 2.7.0, but that was a defect. If you compare it with what PEP-554 has to say: https://www.python.org/dev/peps/pep-0554/#interpreter-isolation you'll see what I mean. PEP-554 is comparatively recent, and is really about exposing the facility for multiple interpreters in a new stdlib module. For that, it documents how interpreters have worked for a long time at the C-level.

It looks very much like openHAB intends each script to execute in a fresh context, and now that context is unpolluted by modules (and their state) loaded by other scripts. It's slower, of course. OpenHAB could make this configurable by allowing you to choose to re-use the script engine (or script context?), which would mean the same modules, loaded once.

Thank you very much for your analysis and help.
I'll try my luck with an issue for openhab.