Issue2495

Environment:
Windows 10 Pro 
Oracle Java 1.8.0_74
Tomcat 8.0.24
jython-standalone 2.7.0
Camunda Community edition 7.4.0

When using a Jython script inside a Camunda BPMN process, I can no longer do a live redeploy of the Camunda webapp. Upon live redeployment, I got the following SEVERE message from Tomcat:

20-Apr-2016 09:42:08.151 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [loan-approval-0.0.1-SNAPSHOT] created a ThreadLocal with key of type [org.python.core.ThreadStateMapping$1] (value [org.python.core.ThreadStateMapping$1@3900df7d]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;@719437fb]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

And after this message, Tomcat also log the following:
20-Apr-2016 09:42:08.151 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [loan-approval-0.0.1-SNAPSHOT] created a ThreadLocal with key of type [org.python.core.ThreadStateMapping$1] (value [org.python.core.ThreadStateMapping$1@3900df7d]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;@719437fb]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

This leaves the webapp in an unusable state and we have to shutdown Tomcat in order to be able to use the BPMN process application again.

I mistakenly pasted the same Tomcat message twice when I opened the issue. Here's the second Tomcat message that should had been pasted after the first one that reports a "probable memory leak":

20-Apr-2016 09:42:08.180 SEVERE [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.startup.ExpandWar.deleteDir [D:\Applications\Camunda\camunda-7.4.0\server\apache-tomcat-8.0.24\webapps\loan-approval-0.0.1-SNAPSHOT\WEB-INF\lib] could not be completely deleted. The presence of the remaining files may cause problems

Looks like Tomcat is trying to do the right thing here to to resolve possible ClassLoader leaks - simply renewing the thread is a perfectly valid approach. The problem is seen in other containers that don't have that strategy, and thus we had to revise our context-scoped support in ThreadStateMapping.

I should point out that we have further revised this support in trunk (what will be 2.7.1); see https://hg.python.org/jython/rev/dcff83c1c4ac

Yes Tomcat is trying to do the right thing, alas, it's the second message reporting a problem leaving the webapp in an "unredeployable" state (a message I posted april 25th). 

The message states that Tomcat cannot delete the directory WEB-INF/lib. This directory only contains the jython jar. I tried to delete the directory manually. It did not work, I had to shutdown Tomcat to delete the jython jar and complete the redeploy cleanup by hand. 

If it were'nt of this, Tomcat could have circumvent the leak. It's as if the ThreadLocal object mentioned in the first message is holding an open file handle on the jython jar.

Good news about trunk 2.7.1. I'm willing to test it using my setup. Is 2.7.1 beta 3 a good candidate for me to try?

Meanwhile we have to use Groovy for our BPM scripting stuff and, for us, it's not a viable long term solution as Python is our dynamic programming language of choice.

Well, I tested 2.7.1b3 and the result is the same. I got the same messages, the one mentionaing the leak and the other one regarding the failed attempt at deleting the jython jar lib directory of the webpapp.

It crossed my mind that maybe, the failure to delete the jython standalone jar was because of Windows. So, I tested this setup on CENTOS 7.x. 

The class loader leak is still occuring but Tomcat is able to remove the webapp directory tree and redeploy a new version of the webapp without having to shutdown Tomcat.

Very interesting details. In general, closing, then deleting files is problematic on Windows and is perhaps the most common bug that affects cross-platform Python code.