Issue2495

classification
Title: Tomcat webapp classloader leak when using jython-standalone
Type: crash Severity: major
Components: None Versions: Jython 2.7
Milestone: Jython 2.7.0
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: amak, cbrouillet@pirel.com, zyasoft
Priority: Keywords:

Created on 2016-04-20.14:46:55 by cbrouillet@pirel.com, last changed 2016-04-29.12:31:56 by zyasoft.

Messages
msg10838 (view) Author: Christian Brouillet (cbrouillet@pirel.com) Date: 2016-04-20.14:46:53
Environment:
Windows 10 Pro 
Oracle Java 1.8.0_74
Tomcat 8.0.24
jython-standalone 2.7.0
Camunda Community edition 7.4.0

When using a Jython script inside a Camunda BPMN process, I can no longer do a live redeploy of the Camunda webapp. Upon live redeployment, I got the following SEVERE message from Tomcat:

20-Apr-2016 09:42:08.151 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [loan-approval-0.0.1-SNAPSHOT] created a ThreadLocal with key of type [org.python.core.ThreadStateMapping$1] (value [org.python.core.ThreadStateMapping$1@3900df7d]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;@719437fb]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

And after this message, Tomcat also log the following:
20-Apr-2016 09:42:08.151 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [loan-approval-0.0.1-SNAPSHOT] created a ThreadLocal with key of type [org.python.core.ThreadStateMapping$1] (value [org.python.core.ThreadStateMapping$1@3900df7d]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;@719437fb]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

This leaves the webapp in an unusable state and we have to shutdown Tomcat in order to be able to use the BPMN process application again.
msg10839 (view) Author: Christian Brouillet (cbrouillet@pirel.com) Date: 2016-04-25.18:38:16
I mistakenly pasted the same Tomcat message twice when I opened the issue. Here's the second Tomcat message that should had been pasted after the first one that reports a "probable memory leak":

20-Apr-2016 09:42:08.180 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.startup.ExpandWar.deleteDir [D:\Applications\Camunda\camunda-7.4.0\server\apache-tomcat-8.0.24\webapps\loan-approval-0.0.1-SNAPSHOT\WEB-INF\lib] could not be completely deleted. The presence of the remaining files may cause problems
msg10840 (view) Author: Jim Baker (zyasoft) Date: 2016-04-27.21:30:23
Looks like Tomcat is trying to do the right thing here to to resolve possible ClassLoader leaks - simply renewing the thread is a perfectly valid approach. The problem is seen in other containers that don't have that strategy, and thus we had to revise our context-scoped support in ThreadStateMapping.

I should point out that we have further revised this support in trunk (what will be 2.7.1); see https://hg.python.org/jython/rev/dcff83c1c4ac
msg10845 (view) Author: Christian Brouillet (cbrouillet@pirel.com) Date: 2016-04-28.11:09:16
Yes Tomcat is trying to do the right thing, alas, it's the second message reporting a problem leaving the webapp in an "unredeployable" state (a message I posted april 25th). 

The message states that Tomcat cannot delete the directory WEB-INF/lib. This directory only contains the jython jar. I tried to delete the directory manually. It did not work, I had to shutdown Tomcat to delete the jython jar and complete the redeploy cleanup by hand. 

If it were'nt of this, Tomcat could have circumvent the leak. It's as if the ThreadLocal object mentioned in the first message is holding an open file handle on the jython jar.

Good news about trunk 2.7.1. I'm willing to test it using my setup. Is 2.7.1 beta 3 a good candidate for me to try?

Meanwhile we have to use Groovy for our BPM scripting stuff and, for us, it's not a viable long term solution as Python is our dynamic programming language of choice.
msg10846 (view) Author: Christian Brouillet (cbrouillet@pirel.com) Date: 2016-04-28.14:31:42
Well, I tested 2.7.1b3 and the result is the same. I got the same messages, the one mentionaing the leak and the other one regarding the failed attempt at deleting the jython jar lib directory of the webpapp.
msg10847 (view) Author: Christian Brouillet (cbrouillet@pirel.com) Date: 2016-04-28.19:00:37
It crossed my mind that maybe, the failure to delete the jython standalone jar was because of Windows. So, I tested this setup on CENTOS 7.x. 

The class loader leak is still occuring but Tomcat is able to remove the webapp directory tree and redeploy a new version of the webapp without having to shutdown Tomcat.
msg10849 (view) Author: Jim Baker (zyasoft) Date: 2016-04-29.12:31:56
Very interesting details. In general, closing, then deleting files is problematic on Windows and is perhaps the most common bug that affects cross-platform Python code.
History
Date User Action Args
2016-04-29 12:31:56zyasoftsetmessages: + msg10849
2016-04-28 19:00:37cbrouillet@pirel.comsetmessages: + msg10847
2016-04-28 14:31:42cbrouillet@pirel.comsetmessages: + msg10846
2016-04-28 11:09:18cbrouillet@pirel.comsetmessages: + msg10845
2016-04-27 21:30:25zyasoftsetnosy: + zyasoft
messages: + msg10840
2016-04-25 18:38:17cbrouillet@pirel.comsetmessages: + msg10839
2016-04-22 17:03:19amaksetnosy: + amak
2016-04-20 14:46:55cbrouillet@pirel.comcreate