Issue1380

I have a test system, where I need to execute hundreds of commands using
Jython. After switching from 2.2 to 2.5, I noticed that the test run
took almost three times as long as previously (~1h vs. 2h50min). Since
even previously most of that time has been spent on interpreter startup
(With Python2.5, the tests run in ~6 min), I investigated this a bit
further. I ran this on my console:

$ time for i in `seq 20`; do jython -c 'import sys; print sys.path'; done

With 2.2, the result was:
real	0m33.868s
user	0m39.526s
sys	0m2.216s

With 2.5, however:
real	1m40.181s
user	2m14.572s
sys	0m4.924s

Almost three times slower here as well.

Here's the interpreter version:

Jython 2.5.0 (Release_2_5_0:6476, Jun 16 2009, 13:33:26) 
[Java HotSpot(TM) Server VM (Sun Microsystems Inc.)] on java1.6.0_13

For the JVM Language Summit, we plan to have a session on shared JVM
lang infrastructure. One thing we would like to explore is creating an
enhanced version of Nailgun that supports a secure model of
operation. Nailgun thus becomes an app container for running command
line apps, with effective startup times potentially below that of
CPython. See Charlie Grove's description:
http://www.cryptodira.com/2007/05/nailgun-your-way-to-quick-jython.html

It would be helpful to engage others in this design. Here are my initial
thoughts:

Share a secret key (perhaps in an environment variable. The
multiprocessing module in CPython does something similar;
http://docs.python.org/library/multiprocessing.html#multiprocessing-auth-keys.
Where
do we actually share the key? If in that process group, via an
environment variable might work.

Spin up a JVM.  My initial thought it to spin up the JVM container so
that it better matches the effective uid (or other OS equivalent). So
don't run it as a daemon or service, but instead as a regular
background process. Perhaps the client will spin it up on demand, with
it quietly terminating if not in use. Plus standard stuff to ensure we
only have one JVM of a given tuple like (process group, euid) or
whatever running. This might use an IPC semaphore; again we could
borrow from multiprocessing.

Any such wrappers necessarily would be written in C -- or against the
shell as necessary -- to minimize the overhead.

We investigated Nailgun a bit after Charlie wrote about it few years
ago. It looked really promising but couldn't make it to work with our
framework. A more ready-made solution based on it would be really cool.

As a Jython user I hope that something could be nevertheless done also
to the "normal" interpreter start-up time. Getting it at least to the
level it was in 2.2 would be great.

I've improved our startup time slightly in trunk (for 2.5.2), but we're 
still a bit slower than 2.2. Hopefully we can improve more before 2.5.2 
release

One large different between 2.2 and 2.5 is we now import the site module 
by default. If you're relying on 2.2, you obviously don't need this to 
happen, you can be disable with the -S option.

2.5 also has an experimental --boot option that improves startup time 
(however breaks some things, see #1127, #1129). We may eventually turn 
this on by default

On my machine (YMMV, comparing my trunk checkout to a 2.2 release and a 
2.5.1 install with a jython-full jar probably isn't even a fair 
comparison, but you can get an idea), we see around:

(jython)pjenvey@golgo13:~/src/java/jython$ time jython-Release_2_2maint 
-c "pass"

real	0m1.423s
user	0m1.753s
sys	0m0.257s

2.5.1:
(jython)pjenvey@golgo13:~/src/java/jython$ time 
~/Downloads/jython2.5.1/jython -c "pass"

real	0m3.217s
user	0m4.428s
sys	0m0.641s

As of r6886:

(jython)pjenvey@golgo13:~/src/java/jython$ time dist/bin/jython -c 
"pass"

real	0m2.399s
user	0m2.933s
sys	0m0.415s

r6886 sans site import:

(jython)pjenvey@golgo13:~/src/java/jython$ time dist/bin/jython -S -c 
"pass"

real	0m1.870s
user	0m2.038s
sys	0m0.283s

r6886 sans site, boot classpath

(jython)pjenvey@golgo13:~/src/java/jython$ time dist/bin/jython -S --
boot -c "pass"

real	0m1.217s
user	0m1.309s
sys	0m0.235s

My mistake, 2.2 does in fact import site by default

To be looked at prior to 2.5.2 rc1

I just tested the start-up time of 2.5.2b2 on Linux and Windows XP (different machines). I got both good and bad news, let's start with the former.

On Linux the start-up time has improved, but is still quite far away from 2.2. Running the same command as yanne in the original report gives these results:

2.5.2 beta 2:
real	0m53.332s
user	1m6.652s
sys	0m5.136s

2.5.1:
real	1m5.767s
user	1m35.698s
sys	0m6.288s

2.2:
real	0m28.580s
user	0m35.582s
sys	0m2.440s

The bad news is that on Windows the start-up time has gotten worse. I created a simple Python script to run similar test as on Linux. The average start-up time was 1.94s with 2.5.1 and 2.58s with 2.5.2b2. I don't have 2.2 installed on this machine.

Could the new start-up time regression on Windows be related to the recent jython.bat fixes? It was great to see that the problems we had reported are fixed (#1567 and #1599) but start-up time getting worse isn't so nice. Could the native Windows launcher (#1491) improve the situation?

We will be looking at this for the 2.5.2 release candidate.

Was this addressed in 2.5.2?

It's been looked at as much as I suspect it ever will be for 2.5 - I think this is outside the level of bugfix we'd do for 2.5. We could have another look in 2.7 though. I'll change the tag to 2.7a1 and see if we can give it a try.

The biggest easy gain we can make here is to use the boot classpath option by default. We had this on by default for a time before the 2.5 release, but we ended up reverting that change (IIRC due to some classpath issues and backwards compatibility).

We should lookup that original issue and consider re-enabling boot class path by default. For what it's worth, JRuby has enabled the boot class path feature for a long time now

I thought there was a specific reason we had to disable the bootclasspath thing...?

#1127 outlines the main issues that I remember

It seems to me that #1127 will cause enough of a surprising backwards compatibility problem often enough to keep this as a 2.7 discussion and leave this out of 2.5.

I don't know if this is something we really want to track in the bug system, but I'll leave it here for now.

There are a number of things we could do. I don't expect that we will really consider of any of them until 2.7.1 at the earliest.