Issue2143

classification
Title: site-packages support in standalone jar
Type: rfe Severity: major
Components: Core Versions: Jython 2.7
Milestone:
process
Status: open Resolution: accepted
Dependencies: Superseder:
Assigned To: Nosy List: alex.gronholm, darjus, nonameentername, yanivnahoum, zyasoft
Priority: normal Keywords: patch

Created on 2014-05-13.22:09:47 by nonameentername, last changed 2018-03-07.17:28:55 by jeff.allen.

Files
File name Uploaded Description Edit Remove
patch.diff nonameentername, 2014-05-20.19:55:25
unnamed darjus, 2014-10-27.06:19:12
Messages
msg8399 (view) Author: werner (nonameentername) Date: 2014-05-13.22:09:47
site-packages inside a jar does not get imported by jython.

workaround:

import site
from org.python.util import jython
jar_location = jython().getClass().getProtectionDomain().getCodeSource().getLocation().getPath()
import site
import os.path
site.addsitedir(os.path.join(jar_location, 'Lib/site-packages'))
msg8400 (view) Author: Jim Baker (zyasoft) Date: 2014-05-13.22:11:35
Thanks for the workaround, this is very helpful for figuring out how to ensure this logic is always available in standalone mode.
msg8454 (view) Author: werner (nonameentername) Date: 2014-05-20.19:55:25
Add site-packages to the sys.path to allow finding site-packages inside a standalone jar
msg8532 (view) Author: Jim Baker (zyasoft) Date: 2014-05-22.01:07:12
More discussion in the duplicate bug #2149
msg8576 (view) Author: Jim Baker (zyasoft) Date: 2014-05-27.16:38:30
One idea I have here is that the built-in open function should be able to take a jar URL to the file in question and this can be resolved automagically against a jar that is on sys.path (including via __classpath__)

This change should then work transparently with normal search path resolution.

It should be fairly easy to test this idea out, given that we can directly take the result of getResourceAsStream and pass it to open.
msg9060 (view) Author: Alex Grönholm (alex.gronholm) Date: 2014-10-01.17:57:53
The workaround won't work in limited-permission use scenarios. "all-permissions" is required.
msg9061 (view) Author: Jim Baker (zyasoft) Date: 2014-10-01.17:59:51
Alex, even for using jar URLs? We will not be directly opening, but provide an alternative reader, via http://docs.oracle.com/javase/7/docs/api/java/net/JarURLConnection.html
msg9062 (view) Author: Alex Grönholm (alex.gronholm) Date: 2014-10-01.18:09:35
I hope this will shed some light to the issue: http://stackoverflow.com/questions/11208681/extract-some-content-from-jar-file-downloaded-with-jws
msg9063 (view) Author: Jim Baker (zyasoft) Date: 2014-10-01.19:09:15
Alex, that's actually not a bad idea to simply extract the jar to a temporary file directory for these types of operations, vs using the resource API to access. This is very similar to what is done in egg support iirc.
msg9171 (view) Author: Jim Baker (zyasoft) Date: 2014-10-27.01:55:06
I just worked on standalone support for #2196, and this gave me some fresh insight.

The best approach likely is to provide a new Clamp-specific runner similar to org.python.util.JarRunner. Let's assume a name like org.python.tools.ClampRunner. Clamp currently packs Lib files under /Lib in the jar so they can be used by existing but limited standalone support (see PySystemState#isStandalone for some specifics). Instead ClampRunner will unpack these files to a temporary directory (using the Java 7 support, Files#createTempDirectory), ensuring these are all deleted upon exit (File#deleteOnExit). Then it will call PySystemState#initialize with python.path property set to this unpacked directory.

Because it's a true directory, we will get standard file introspection that caused this bug to be submitted in the frist place.

Note that will also make it feasible to support the package cache using a similar scheme (python.cachedir property setting).

Lastly, this can be done completely outside of the Jython 2.7 dev cycle, but instead in Clamp instead. (Still need to verify that package cache can be constructed without such changes, but it can certainly be used without.)
msg9172 (view) Author: Darjus Loktevic (darjus) Date: 2014-10-27.06:19:13
Hey Jim,

Yeah, was thinking about something similar a while ago. The temp directory
approach may not work in all places, but should work for >80%

Darjus

On Mon, Oct 27, 2014 at 3:55 AM, Jim Baker <report@bugs.jython.org> wrote:

>
> Jim Baker added the comment:
>
> I just worked on standalone support for #2196, and this gave me some fresh
> insight.
>
> The best approach likely is to provide a new Clamp-specific runner similar
> to org.python.util.JarRunner. Let's assume a name like
> org.python.tools.ClampRunner. Clamp currently packs Lib files under /Lib in
> the jar so they can be used by existing but limited standalone support (see
> PySystemState#isStandalone for some specifics). Instead ClampRunner will
> unpack these files to a temporary directory (using the Java 7 support,
> Files#createTempDirectory), ensuring these are all deleted upon exit
> (File#deleteOnExit). Then it will call PySystemState#initialize with
> python.path property set to this unpacked directory.
>
> Because it's a true directory, we will get standard file introspection
> that caused this bug to be submitted in the frist place.
>
> Note that will also make it feasible to support the package cache using a
> similar scheme (python.cachedir property setting).
>
> Lastly, this can be done completely outside of the Jython 2.7 dev cycle,
> but instead in Clamp instead. (Still need to verify that package cache can
> be constructed without such changes, but it can certainly be used without.)
>
> ----------
> nosy: +darjus
>
> _______________________________________
> Jython tracker <report@bugs.jython.org>
> <http://bugs.jython.org/issue2143>
> _______________________________________
>
msg9181 (view) Author: Jim Baker (zyasoft) Date: 2014-10-29.00:13:08
Moving to low priority now that we have identified a good workaround:

* We will continue to support standalone for to minimize startup time
* But use the ClampRunner approach with unpacking for more general usage

We may want to revisit with Jython 3.x
msg9480 (view) Author: Jim Baker (zyasoft) Date: 2015-02-03.00:59:52
http://docs.oracle.com/javase/7/docs/technotes/guides/io/fsp/zipfilesystemprovider.html provides an approach that should work
msg11252 (view) Author: Yaniv Nahoum (yanivnahoum) Date: 2017-03-20.09:29:53
The workaround mentioned here [site.addsitedir(os.path.join(jar_location, 'Lib/site-packages'))] does not work for me - I assume it related to the fact that jython open() method for opening a file inside a jar does not work.
Currently the only way I can think of to provide a self contained jar is to copy the site-packages directory from the jar into a temp directory and add it to the sys.path, or set it as python.home.
Is there any progress on this issue?
History
Date User Action Args
2018-03-07 17:28:55jeff.allensetmilestone: Jython 2.7.2 ->
2017-03-21 04:47:36zyasoftsetassignee: zyasoft ->
2017-03-20 09:29:53yanivnahoumsetnosy: + yanivnahoum
messages: + msg11252
2015-12-29 23:55:08zyasoftsetmilestone: Jython 2.7.1 -> Jython 2.7.2
2015-04-23 17:42:58zyasoftsetpriority: low -> normal
milestone: Jython 2.7.1
2015-02-03 00:59:52zyasoftsetmessages: + msg9480
2014-10-29 00:13:09zyasoftsetpriority: high -> low
messages: + msg9181
2014-10-27 06:19:13darjussetfiles: + unnamed
messages: + msg9172
2014-10-27 01:55:07zyasoftsetnosy: + darjus
messages: + msg9171
2014-10-01 19:09:15zyasoftsetmessages: + msg9063
2014-10-01 18:09:36alex.gronholmsetmessages: + msg9062
2014-10-01 17:59:52zyasoftsetmessages: + msg9061
2014-10-01 17:57:53alex.gronholmsetnosy: + alex.gronholm
messages: + msg9060
2014-05-27 16:38:31zyasoftsetmessages: + msg8576
2014-05-22 01:07:12zyasoftsetmessages: + msg8532
2014-05-20 19:55:25nonameenternamesetfiles: + patch.diff
keywords: + patch
messages: + msg8454
2014-05-13 22:11:35zyasoftsetpriority: high
assignee: zyasoft
resolution: accepted
messages: + msg8400
nosy: + zyasoft
2014-05-13 22:09:47nonameenternamecreate