Message6597

When our application (Java + Jython 2.5.1) started, we saw the following in the console log:

    [junit] *sys-package-mgr*: processing new jar, '/scratch/jenkins_workspace/GDA_8.14.tests.RH5_64_bad-0/thirdparty/eclipse/plugins/org.eclipse.swt.gtk.linux.x86_64_3.6.2.v3659b.jar'
    [junit] *sys-package-mgr*: processing new jar, '/scratch/jenkins_workspace/GDA_8.14.tests.RH5_64_bad-0/thirdparty/eclipse/plugins/org.eclipse.swt.gtk.solaris.sparc_3.6.2.v3659b.jar'
    [junit] *sys-package-mgr*: processing new jar, '/scratch/jenkins_workspace/GDA_8.14.tests.RH5_64_bad-0/thirdparty/eclipse/plugins/org.eclipse.swt.gtk.solaris.x86_3.6.2.v3659b.jar'
    [junit] *sys-package-mgr*: processing new jar, '/scratch/jenkins_workspace/GDA_8.14.tests.RH5_64_bad-0/thirdparty/eclipse/plugins/org.eclipse.swt.motif.aix.ppc_3.6.2.v3659b.jar'
    [junit] *sys-package-mgr*: can't write cache file for '/scratch/jenkins_workspace/GDA_8.14.tests.RH5_64_bad-0/thirdparty/eclipse/plugins/org.eclipse.swt.motif.aix.ppc_3.6.2.v3659b.jar'
    [junit] *sys-package-mgr*: processing new jar, '/scratch/jenkins_workspace/GDA_8.14.tests.RH5_64_bad-0/thirdparty/eclipse/plugins/org.eclipse.swt.motif.hpux.ia64_32_3.6.2.v3659b.jar'
    [junit] *sys-package-mgr*: skipping bad jar, '/scratch/jenkins_workspace/GDA_8.14.tests.RH5_64_bad-0/thirdparty/eclipse/plugins/org.eclipse.swt.motif.hpux.ia64_32_3.6.2.v3659b.jar'
    [junit] *sys-package-mgr*: processing new jar, '/scratch/jenkins_workspace/GDA_8.14.tests.RH5_64_bad-0/thirdparty/eclipse/plugins/org.eclipse.swt.motif.linux.x86_3.6.2.v3659b.jar'
    [junit] *sys-package-mgr*: skipping bad jar, '/scratch/jenkins_workspace/GDA_8.14.tests.RH5_64_bad-0/thirdparty/eclipse/plugins/org.eclipse.swt.motif.linux.x86_3.6.2.v3659b.jar'
    [junit] *sys-package-mgr*: processing new jar, '/scratch/jenkins_workspace/GDA_8.14.tests.RH5_64_bad-0/thirdparty/eclipse/plugins/org.eclipse.swt.motif.solaris.sparc_3.6.2.v3659b.jar'
    [junit] *sys-package-mgr*: skipping bad jar, '/scratch/jenkins_workspace/GDA_8.14.tests.RH5_64_bad-0/thirdparty/eclipse/plugins/org.eclipse.swt.motif.solaris.sparc_3.6.2.v3659b.jar'
 
The key messages here are "can't write cache file" followed by lots of "skipping bad jar". The application crashed shortly thereafter with
java.io.FileNotFoundException: <path>/InterfaceProvider.class (Too many open files)

If we disabled the Jython cache (we removed write permissions from the cache directory), then everything ran without problems.

We managed to reproduce the problem on our test system, and the problem is clearly _not_ with the specific jars that report "can't write cache file" or "skipping bad jar". We were running with the default "ulimit -n" of 1024, and if this was increased to 2048, the problem no longer occurred.

So, a couple of observations:
(1) The text "skipping bad jar" is not the correct description of the problem. I guess that this gets reported on any exception being thrown - this makes problem diagnosis difficult. It would be better if the error message gave more details about why it failed.

(2) I wonder if the *sys-package-mgr* code is cleaning up it's file handles properly. See the last comment in http://stackoverflow.com/questions/2272908/java-io-filenotfoundexception-too-many-open-files, and also http://coldfused.blogspot.com/2007/02/mystery-of-too-many-open-files.html.