Title: threading memory leak
Type: crash Severity: major
Components: Core Versions: Jython 2.7
Status: closed Resolution: invalid
Dependencies: Superseder:
Assigned To: Nosy List: amak, crankycoder, fwierzbicki, jeff.allen, treaves, zyasoft
Priority: normal Keywords:

Created on 2011-06-15.19:58:08 by treaves, last changed 2018-11-04.16:21:03 by jeff.allen.

File name Uploaded Description Edit Remove jeff.allen, 2018-03-17.10:55:46 Timothy's code with a repetitive client program
msg6553 (view) Author: Timothy Reaves (treaves) Date: 2011-06-15.19:58:07
Actually the version I am using is 2.5.2, but the Versions list does not offer that as a selection.  I upgraded to 2.5.2 from 2.5.2rc4 to see if this issue is resolved.

This issue may be 1660 - that issue is marked closed, but, if it is not that issue, it is another, very similar one.

I use threading for work in my code.  It's code copied from one of the Python doc pages.

class Worker(Thread):
	def __init__(self, tasks):
		self.tasks = tasks
		self.daemon = True

	def run(self):
		while True:
			func, args, kargs = self.tasks.get()
				func(*args, **kargs)
			except Exception, exception:
				logging.error(exception, extra={"sbs_app":"Worker"})

class ThreadPool:
	def __init__(self, num_threads):
		self.tasks = Queue(num_threads)
		for _ in range(num_threads):

	def add_task(self, func, *args, **kargs):
		self.tasks.put((func, args, kargs), True)

	def wait_completion(self):

In this case, Jython is being executed by ODI, an Oracle product (I don't really think that's relevant, but...).  I run this code with 32 threads, and then use the wait_for_completion().  All tasks run and finish fine.  However, none of the 32 threads ever goes away.

I run jconsole, and when I kick off my process, I see the thread count increase, typically by 35 threads (some are ODI's).  When the process has finished, and ODI finishes it's stuff, the delta is always exactly 32; it just keeps growing.  Eventually, the VM runs out of memory, or the ulimit is hit.

In jconsole, all of the threads are still visible, and show:
Name: Thread
State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@57cac7
Total blocked: 38  Total waited: 140

Stack trace: 
sun.misc.Unsafe.park(Native Method)
org.python.modules._threading.Condition$Condition_wait_exposer.__call__(Unknown Source)

Line 223 in is the except clause of the attempt at deleting self, when an exception has occurred. So for some reason, the threads do not seem able to self-destruct.

Any help getting this taken care will be greatly appreciated.
msg7632 (view) Author: Alan Kennedy (amak) Date: 2013-02-06.21:38:16
Now that this issue

Has been fixed, is this one still an issue?
msg7635 (view) Author: Timothy Reaves (treaves) Date: 2013-02-07.13:15:53
What makes you think this issue has been fixed?
msg7636 (view) Author: Alan Kennedy (amak) Date: 2013-02-07.13:30:49
Sorry, bad paste of URL.

Now that issue #1660 has been fixed

Is this issue (#1761) still happening?
msg7637 (view) Author: Timothy Reaves (treaves) Date: 2013-02-07.14:42:40
I see. :)

Yes, this issue still happens with 2.5.2.  1660 was closed in 2010, yet this bug still clearly existed in 2011 when I commented on 1660.  I wasn't sure if this defect was a new one (a regression introduced after 1660 was fixed), or if 1660 was never actually fixed (the fix was defective itself).

But this thread consumption still occurs, and with JConsole is easy to see.  I've had to write my own thread code to get around this.
msg7638 (view) Author: Frank Wierzbicki (fwierzbicki) Date: 2013-02-07.18:01:06
#1660 was *just* fixed, hopefully it resolves this issue as well (but no binary release has this fix yet). I am in the process of finding any more low hanging fixes - as soon as I am done with that I will be putting out a 2.5.4 rc and a 2.7 beta.
msg7639 (view) Author: Frank Wierzbicki (fwierzbicki) Date: 2013-02-07.18:05:13
oops, I forgot, #1660 is only for 2.7 as it is an API change and is not backwards compatible, so testing for this should be on the 2.7 beta.
msg11821 (view) Author: Jeff Allen (jeff.allen) Date: 2018-03-17.10:55:46
I checked this on 2.7.2a1, observed with jvisualvm, and I can reproduce what I understand is the OP's issue. "When the process has finished, and ODI finishes it's stuff, the delta is always exactly 32; it just keeps growing." So the OP's application is run repeatedly in the same JVM, and each time the threads remain alive (parked).

My test program is attached. It creates successive pools of 32 threads. A given pool behaves in the textbook way, but when the pool is dereferenced the JVM threads of the old one persist, so the JVM thread count shows a rising staircase. Other garbage collection is clearly going on happily. 

My difficulty is to know whether that is actually incorrect behaviour. does not return, so the thread should not finish. This stackoverflow post, with almost identical code (run on CPython I assume), is helpful:

I'm taking off 2.5 as a target as we won't fix that as far as I know.
msg12172 (view) Author: Jeff Allen (jeff.allen) Date: 2018-11-04.16:21:03
This is really old, and I think based in a misunderstanding identified above.
Date User Action Args
2018-11-04 16:21:03jeff.allensetstatus: pending -> closed
messages: + msg12172
2018-03-17 10:58:46jeff.allensetfiles: - unnamed
2018-03-17 10:58:36jeff.allensetresolution: fixed -> invalid
2018-03-17 10:55:47jeff.allensetstatus: open -> pending
files: +
versions: + Jython 2.7, - Jython 2.5
nosy: + jeff.allen
messages: + msg11821
resolution: remind -> fixed
2014-12-27 17:24:20zyasoftsetnosy: + zyasoft
2014-09-26 05:35:21zyasoftsetassignee: fwierzbicki ->
resolution: remind
2013-02-20 00:07:39fwierzbickisetversions: + Jython 2.5, - 2.5.2rc
2013-02-07 18:05:13fwierzbickisetmessages: + msg7639
2013-02-07 18:01:44fwierzbickisetpriority: normal
assignee: fwierzbicki
2013-02-07 18:01:06fwierzbickisetnosy: + fwierzbicki
messages: + msg7638
2013-02-07 14:42:40treavessetmessages: + msg7637
2013-02-07 13:30:49amaksetfiles: + unnamed
messages: + msg7636
2013-02-07 13:15:53treavessetmessages: + msg7635
2013-02-06 21:38:16amaksetnosy: + amak
messages: + msg7632
2011-08-11 13:56:23crankycodersetnosy: + crankycoder
2011-06-16 20:54:18treavessettype: crash
versions: + 2.5.2rc, - Deferred
2011-06-15 19:58:08treavescreate