Issue1879

classification
Title: -m command line option broken for scripts inside a jar file
Type: Severity: normal
Components: Core Versions: Jython 2.7
Milestone:
process
Status: pending Resolution: fixed
Dependencies: Superseder:
Assigned To: jeff.allen Nosy List: conflatedauto, darjus, fwierzbicki, jeff.allen, zyasoft
Priority: high Keywords:

Created on 2012-04-30.08:37:52 by conflatedauto, last changed 2015-05-17.08:43:20 by jeff.allen.

Messages
msg7074 (view) Author: Adam (conflatedauto) Date: 2012-04-30.08:37:52
jython -m broken for scripts inside jar files

The workaround jython -c "import module; module.blah()" will work, but not if you don't have a blah function.

This is jy.bat


Version 2.5.2

C:\Working\working>java -classpath C:\Working\tools\maven\repository2\org\robotf
ramework\robotframework\2.7.1\robotframework-2.7.1.jar;C:\Working\tools\maven\re
pository2\tools\tools-core\1.3.1-SNAPSHOT\tools-core-1.3.1-SNAPSHOT.jar org.python
.util.jython
Jython 2.5.2 (Release_2_5_2:7206, Mar 2 2011, 23:12:06)
[Java HotSpot(TM) Client VM (Sun Microsystems Inc.)] on java1.6.0_26
Type "help", "copyright", "credits" or "license" for more information.

C:\Working\working>jy -m editor.vimgen

C:\Working\working>java -classpath C:\Working\tools\maven\repository2\org\robotf
ramework\robotframework\2.7.1\robotframework-2.7.1.jar;C:\Working\tools\maven\re
pository2\tools\tools-core\1.3.1-SNAPSHOT\tools-core-1.3.1-SNAPSHOT.jar org.python
.util.jython -m editor.vimgen
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Working\tools\maven\repository2\org\robotframework\robotframework\2.7
.1\robotframework-2.7.1.jar\Lib\runpy.py", line 88, in run_module
AttributeError: 'ClasspathPyImporter' object has no attribute 'get_code'
C:\Working\working>jy -m editor/vimgen

C:\Working\working>java -classpath C:\Working\tools\maven\repository2\org\robotf
ramework\robotframework\2.7.1\robotframework-2.7.1.jar;C:\Working\tools\maven\re
pository2\tools\tools-core\1.3.1-SNAPSHOT\tools-core-1.3.1-SNAPSHOT.jar org.python
.util.jython -m editor/vimgen
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Working\tools\maven\repository2\org\robotframework\robotframework\2.7
.1\robotframework-2.7.1.jar\Lib\runpy.py", line 88, in run_module
AttributeError: 'ClasspathPyImporter' object has no attribute 'get_code'
C:\Working\working>jy -m editor/blah

Looking at the source it seems that ClasspathPyImporter is indeed lacking this method.


No such module is a different, correct error:

C:\Working\working>java -classpath C:\Working\tools\maven\repository2\org\robotf
ramework\robotframework\2.7.1\robotframework-2.7.1.jar;C:\Working\tools\maven\re
pository2\tools\tools-core\1.3.1-SNAPSHOT\tools-core-1.3.1-SNAPSHOT.jar org.python
.util.jython -m editor/blah
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Working\tools\maven\repository2\org\robotframework\robotframework\2.7
.1\robotframework-2.7.1.jar\Lib\runpy.py", line 87, in run_module
ImportError: No module named editor/blah



Still present under the latest beta release

C:\Working\working>jy -m editor/vimgen

C:\Working\working>rem set JYTHON=C:\Working\tools\maven\repository2\org\robotfr
amework\robotframework\2.7.1\robotframework-2.7.1.jar

C:\Working\working>set JYTHON=C:\Working\tools\jython\jython2.5.3b1\jython.jar

C:\Working\working>java -classpath C:\Working\tools\jython\jython2.5.3b1\jython.
jar;C:\Working\tools\maven\repository2\tools\tools-core\1.3.1-SNAPSHOT\tools-core-
1.3.1-SNAPSHOT.jar org.python.util.jython -m editor/vimgen
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Working\tools\jython\jython2.5.3b1\Lib\runpy.py", line 88, in run_mod
ule
    code = loader.get_code(mod_name)
AttributeError: 'ClasspathPyImporter' object has no attribute 'get_code'



Works for local scripts

C:\Working\working>dir *.py
 Volume in drive C has no label.
 Volume Serial Number is 7402-3955

 Directory of C:\Working\working

04/30/2012  11:11 AM                15 test.py
               1 File(s)             15 bytes
               0 Dir(s)  208,984,723,456 bytes free

C:\Working\working>jy -m test

C:\Working\working>rem set JYTHON=C:\Working\tools\maven\repository2\org\robotfr
amework\robotframework\2.7.1\robotframework-2.7.1.jar

C:\Working\working>set JYTHON=C:\Working\tools\jython\jython2.5.3b1\jython.jar

C:\Working\working>java -classpath C:\Working\tools\jython\jython2.5.3b1\jython.
jar;C:\Working\tools\maven\repository2\tools\tools-core\1.3.1-SNAPSHOT\tools-core-
1.3.1-SNAPSHOT.jar org.python.util.jython -m test
test.py


This is jy.bat just to show there is nothing weird there. Jython 2.5.2 was the version disted with the robot framework.

rem set JYTHON=%M2_REPO%\org\robotframework\robotframework\2.7.1\robotframework-
2.7.1.jar
set JYTHON=C:\Working\tools\jython\jython2.5.3b1\jython.jar


java -classpath %JYTHON%;%M2_REPO%\ets\tools-core\1.3.1-SNAPSHOT\tools-core-1.3.
1-SNAPSHOT.jar org.python.util.jython %1 %2 %3
msg7124 (view) Author: Darjus Loktevic (darjus) Date: 2012-05-21.19:57:41
Hey Adam,

Where is this editor.vimgen coming from? I am trying to reproduce your issue, but can't. I agree that get_code is missing, but first of all i want to make sure we can reproduce your problem.

Could you build a test that reproduces it (or at least clarify a bit more on reproducing it)?

java -cp ./dist/robotframework-trunk20120521.jar org.python.util.jython -m robot.run                                                    
[ ERROR ] Expected at least 1 argument, got 0.

Thanks!
Darjus
msg7144 (view) Author: Adam (conflatedauto) Date: 2012-05-25.00:02:47
Hi Darjus ā€“ thanks for looking into this. The vimgen.py script is inside the tools-core jar ā€“ I can see how it would be ambiguous without a jar listing now. Tools-core is part of the internal project Iā€™m working on.
 
$ jar tfv tools-core-1.3.2.jar
     0 Sun May 06 23:18:56 CST 2012 META-INF/
  127 Sun May 06 23:18:54 CST 2012 META-INF/MANIFEST.MF
     0 Sun May 06 23:18:34 CST 2012 editor/
   630 Sun May 06 23:18:34 CST 2012 editor/nameutil.py
     0 Sun May 06 23:18:34 CST 2012 editor/__init__.py
  7381 Sun May 06 23:18:34 CST 2012 editor/vimgen.py
 
Does that help?
Adam
msg9369 (view) Author: Jim Baker (zyasoft) Date: 2015-01-09.03:50:49
Related issue is #2058 - no get_data support

We should implement all the optional features in PEP 302:
https://www.python.org/dev/peps/pep-0302/#optional-extensions-to-the-importer-protocol
msg9371 (view) Author: Jim Baker (zyasoft) Date: 2015-01-09.03:53:27
Let's see if we can get into RC1
msg9531 (view) Author: Jim Baker (zyasoft) Date: 2015-02-19.15:02:32
Necessary for PEP 338, which is used by wrapper exe scripts as needed for #2251
msg9553 (view) Author: Jim Baker (zyasoft) Date: 2015-02-23.04:50:56
Demoting to high, but we will still try to get in for RC1

It's not actually the problem we saw for ensurepip, which was about making zip files executable by Jython.
msg10037 (view) Author: Jeff Allen (jeff.allen) Date: 2015-05-08.15:37:42
I don't understand much of the above, and some of it turned out to be a red herring. Does the following scenario represent the issue? I have a jarred-up the files I used in #2158 (and hidden the original):

> jar tf ulib.jar
META-INF/
META-INF/MANIFEST.MF
u/
u/v/
u/v/w/
u/v/w/x.py
u/v/w/y.py
u/v/w/__init__.py
u/v/y.py
u/v/__init__.py
u/y.py
u/__init__.py

When you invoke x.py, each of these will tell you its path. Now, from 2.7.0+ I get:

> java -cp ulib.jar;%JT%\dist\jython-dev.jar;%JT%\dist\javalib\* org.python.util.jython -c"import u.v.w.x"
Init: u\__init__.py
Init: u\v\__init__.py
Init: u\v\w\__init__.py
File: u\v\w\x.py
File: u\y.py

> java -cp ulib.jar;%JT%\dist\jython-dev.jar;%JT%\dist\javalib\* org.python.util.jython -m u.v.w.x
Init: u\__init__.py
Init: u\v\__init__.py
Init: u\v\w\__init__.py
... stack dump ...
AttributeError: 'ClasspathPyImporter' object has no attribute 'is_package'

The failure of the second one constitutes the issue, and is because our loaders do not implement the "optional" methods that are indispensible to runpy. Right?
msg10038 (view) Author: Jim Baker (zyasoft) Date: 2015-05-08.15:46:23
Jeff, I believe we just have to implement the "optional" features of PEP 302, per msg9369. Should be straightforward, since #2058 does have an attached PR that just needs to be reviewed, and Raphael has contributed other good commits in the past.
msg10039 (view) Author: Jeff Allen (jeff.allen) Date: 2015-05-08.16:31:52
I think that says I'm reproducing the right thing. Thanks, I'd missed Raphael's PR for get_data. The message here is about is_package, earlier it was get_code, and I suppose a failing test would be nice. So it's not the whole story, but #2058 is a good start.
msg10063 (view) Author: Jeff Allen (jeff.allen) Date: 2015-05-16.10:01:51
I've made some decent progress with this, committed #2058 locally, and implemented the optional methods. I'm not entirely sure what to test for, as I have no reference semantics when using a ClasspathPyImporter directly. (There is no direct CPython counterpart.) The requirement as I see it is to work correctly with import, -m (runpy) and pkgutil (which runpy uses) so that's what my tests look like.

At the moment I can do this:
>jar tf rlib.jar
META-INF/
META-INF/MANIFEST.MF
data/
data/r.dat
data/__init__.py

>java -cp rlib.jar;dist\jython-dev.jar;dist\javalib\* org.python.util.jython
Jython 2.7.0+ (default:67f36ecd9103+, May 16 2015, 00:30:35)
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java1.7.0_60
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, pkgutil
>>> from java.lang import System
>>> 'rlib.jar' in System.getProperty('java.class.path').split(';')
True
>>> i = pkgutil.get_importer('__pyclasspath__/')
>>> i.is_package('data')
True
>>> i.get_code('data')
<code object <module> at 0x2, file "__pyclasspath__/data/__init__.py", line 0>
>>> import data
Init data package
>>> i.get_data('data\\r.dat')
'some data'
>>> i.get_data('data\\r.dat')
''

That last bit is clearly wrong, and stems from re-using the cached InputStream from "entries". This puzzles me: why do we have that cache if it contains things we can only use once?

zipimporter avoids this by re-opening the entire archive and finding the resource again. I'll emulate that, and same for get_source. They could usefully share code, but it all seems a bit fragile to do in one step.

The entries cache is still there in zipimporter, and up it looks up modules in it, but then it doesn't use the result. I wonder if these open streams are behind the 100% unlink() failure I get from test_zipimporter?

It seems like a part of the code that could benefit from a cache, but not quite the one we have. My impression generally of Jython's import mechanism is that it contains some good stuff, but we lost our way. I can see that CPython wandered a long time before settling, so it's hardly surprising. It has to be revisited for Jy3 anyway: chance to re-think?
msg10064 (view) Author: Jeff Allen (jeff.allen) Date: 2015-05-16.14:52:02
Correction, zipimporter is doing its own thing with a table of contents. It's still a helpful study. In ClasspathPyImporter, a sneaky entries.remove happens inside makeBundle (if you use it), where one-time use is assured, and fixes my bug. This is a very odd proceeding.
msg10065 (view) Author: Jim Baker (zyasoft) Date: 2015-05-16.15:21:30
Jeff, sounds like the existing implementation was half baked, and it introduces a resource leak in the entries map. I think the best thing to do now is remove the entries map completely, and avoid caching in this layer. Instead let the underlying OS filesystem determine what caching needs to be done, if any. Alternatively users of the importer can do their own caching.
msg10066 (view) Author: Jeff Allen (jeff.allen) Date: 2015-05-16.15:41:14
No I was confused. It's not caching. It's holding something (in this case the InputStream around the resource) between finding it and using it. The trick (missed by Raphael in #2058) is to remove it when you use it. If you don't do that it *looks* like a cache because there's an entry for each thing you've accessed. This bit's good now. I think refactoring this is a separate little project. End in sight.
msg10068 (view) Author: Jeff Allen (jeff.allen) Date: 2015-05-17.08:43:19
I claim this is fixed, including the addition of desired PEP 302 optional methods, by change sets culminating in:
https://hg.python.org/jython/rev/dccf5d04d58e

I'm not *absolutely* sure that all the possible ways we might want to use PEP 302 methods are supported with least surprise semantics. (See also additions to test_classpathimporter.) Where there was no established expectation from CPython, I've used my judgement.

Bottom line:
> java -cp ulib.jar;%JT%\dist\jython-dev.jar;%JT%\dist\javalib\* org.python.util.jython -m u.v.w.x
Init: u\__init__.py
Init: u\v\__init__.py
Init: u\v\w\__init__.py
File: u\v\w\x.py
File: u\y.py
History
Date User Action Args
2015-05-17 08:43:20jeff.allensetstatus: open -> pending
resolution: accepted -> fixed
messages: + msg10068
2015-05-16 15:41:14jeff.allensetmessages: + msg10066
2015-05-16 15:21:30zyasoftsetmessages: + msg10065
2015-05-16 14:52:03jeff.allensetmessages: + msg10064
2015-05-16 10:01:53jeff.allensetmessages: + msg10063
versions: + Jython 2.7, - Jython 2.5
2015-05-08 16:31:52jeff.allensetmessages: + msg10039
2015-05-08 15:46:23zyasoftsetmessages: + msg10038
2015-05-08 15:37:43jeff.allensetmessages: + msg10037
2015-04-19 22:40:47jeff.allensetassignee: jeff.allen
nosy: + jeff.allen
2015-02-23 04:50:56zyasoftsetpriority: urgent -> high
messages: + msg9553
2015-02-19 15:02:32zyasoftsetpriority: high -> urgent
messages: + msg9531
2015-01-09 03:53:27zyasoftsetpriority: high
resolution: accepted
messages: + msg9371
2015-01-09 03:50:49zyasoftsetnosy: + zyasoft
messages: + msg9369
2013-02-19 21:50:47fwierzbickisetnosy: + fwierzbicki
versions: + Jython 2.5, - 2.5.2, 2.5.3b2
2012-05-25 00:02:47conflatedautosetmessages: + msg7144
2012-05-21 19:57:41darjussetnosy: + darjus
messages: + msg7124
2012-04-30 08:37:52conflatedautocreate