Issue1761

classification
Title: threading memory leak
Type: crash Severity: major
Components: Core Versions: Jython 2.5
process
Status: open Resolution: remind
Dependencies: Superseder:
Assigned To: Nosy List: amak, crankycoder, fwierzbicki, treaves
Priority: normal Keywords:

Created on 2011-06-15.19:58:08 by treaves, last changed 2014-09-26.05:35:21 by zyasoft.

Files
File name Uploaded Description Edit Remove
unnamed amak, 2013-02-07.13:30:49
Messages
msg6553 (view) Author: Timothy Reaves (treaves) Date: 2011-06-15.19:58:07
Actually the version I am using is 2.5.2, but the Versions list does not offer that as a selection.  I upgraded to 2.5.2 from 2.5.2rc4 to see if this issue is resolved.

This issue may be 1660 - that issue is marked closed, but, if it is not that issue, it is another, very similar one.

I use threading for work in my code.  It's code copied from one of the Python doc pages.

class Worker(Thread):
	def __init__(self, tasks):
		Thread.__init__(self)
		self.tasks = tasks
		self.daemon = True
		self.start()

	def run(self):
		while True:
			func, args, kargs = self.tasks.get()
			try:
				func(*args, **kargs)
			except Exception, exception:
				logging.error(exception, extra={"sbs_app":"Worker"})
			self.tasks.task_done()

class ThreadPool:
	def __init__(self, num_threads):
		self.tasks = Queue(num_threads)
		for _ in range(num_threads):
			Worker(self.tasks)

	def add_task(self, func, *args, **kargs):
		self.tasks.put((func, args, kargs), True)

	def wait_completion(self):
		self.tasks.join()


In this case, Jython is being executed by ODI, an Oracle product (I don't really think that's relevant, but...).  I run this code with 32 threads, and then use the wait_for_completion().  All tasks run and finish fine.  However, none of the 32 threads ever goes away.

I run jconsole, and when I kick off my process, I see the thread count increase, typically by 35 threads (some are ODI's).  When the process has finished, and ODI finishes it's stuff, the delta is always exactly 32; it just keeps growing.  Eventually, the VM runs out of memory, or the ulimit is hit.

In jconsole, all of the threads are still visible, and show:
Name: Thread
State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@57cac7
Total blocked: 38  Total waited: 140

Stack trace: 
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
org.python.modules._threading.Condition.Condition_wait(Condition.java:87)
org.python.modules._threading.Condition$Condition_wait_exposer.__call__(Unknown Source)
org.python.core.PyObject.__call__(PyObject.java:375)
Queue$py.get$12(/u01/app/oracle/product/ODI/oracledi_J251/lib/scripting/Lib/Queue.py:179)
Queue$py.call_function(/u01/app/oracle/product/ODI/oracledi_J251/lib/scripting/Lib/Queue.py)
org.python.core.PyTableCode.call(PyTableCode.java:165)
org.python.core.PyBaseCode.call(PyBaseCode.java:301)
org.python.core.PyBaseCode.call(PyBaseCode.java:127)
org.python.core.PyFunction.__call__(PyFunction.java:317)
org.python.core.PyMethod.__call__(PyMethod.java:109)
sbs.threadsupport$py.run$3(/opt/csw/lib/python/site-packages/ImageProcessingPipeline-2.3.1-py2.6.egg/sbs/threadsupport.py:44)
sbs.threadsupport$py.call_function(/opt/csw/lib/python/site-packages/ImageProcessingPipeline-2.3.1-py2.6.egg/sbs/threadsupport.py)
org.python.core.PyTableCode.call(PyTableCode.java:165)
org.python.core.PyBaseCode.call(PyBaseCode.java:134)
org.python.core.PyFunction.__call__(PyFunction.java:317)
org.python.core.PyMethod.__call__(PyMethod.java:109)
threading$py._Thread__bootstrap$33(/u01/app/oracle/product/ODI/oracledi_J251/lib/scripting/Lib/threading.py:223)
threading$py.call_function(/u01/app/oracle/product/ODI/oracledi_J251/lib/scripting/Lib/threading.py)
org.python.core.PyTableCode.call(PyTableCode.java:165)
org.python.core.PyBaseCode.call(PyBaseCode.java:301)
org.python.core.PyBaseCode.call(PyBaseCode.java:194)
org.python.core.PyFunction.__call__(PyFunction.java:387)
org.python.core.PyMethod.instancemethod___call__(PyMethod.java:220)
org.python.core.PyMethod.__call__(PyMethod.java:211)
org.python.core.PyMethod.__call__(PyMethod.java:201)
org.python.core.PyMethod.__call__(PyMethod.java:196)
org.python.core.FunctionThread.run(FunctionThread.java:21)


Line 223 in threading.py is the except clause of the attempt at deleting self, when an exception has occurred. So for some reason, the threads do not seem able to self-destruct.

Any help getting this taken care will be greatly appreciated.
msg7632 (view) Author: Alan Kennedy (amak) Date: 2013-02-06.21:38:16
Now that this issue

http://bugs.jython.org/issue1761

Has been fixed, is this one still an issue?
msg7635 (view) Author: Timothy Reaves (treaves) Date: 2013-02-07.13:15:53
What makes you think this issue has been fixed?
msg7636 (view) Author: Alan Kennedy (amak) Date: 2013-02-07.13:30:49
Sorry, bad paste of URL.

Now that issue #1660 has been fixed

http://bugs.jython.org/issue1660

Is this issue (#1761) still happening?
msg7637 (view) Author: Timothy Reaves (treaves) Date: 2013-02-07.14:42:40
I see. :)

Yes, this issue still happens with 2.5.2.  1660 was closed in 2010, yet this bug still clearly existed in 2011 when I commented on 1660.  I wasn't sure if this defect was a new one (a regression introduced after 1660 was fixed), or if 1660 was never actually fixed (the fix was defective itself).

But this thread consumption still occurs, and with JConsole is easy to see.  I've had to write my own thread code to get around this.
msg7638 (view) Author: Frank Wierzbicki (fwierzbicki) Date: 2013-02-07.18:01:06
#1660 was *just* fixed, hopefully it resolves this issue as well (but no binary release has this fix yet). I am in the process of finding any more low hanging fixes - as soon as I am done with that I will be putting out a 2.5.4 rc and a 2.7 beta.
msg7639 (view) Author: Frank Wierzbicki (fwierzbicki) Date: 2013-02-07.18:05:13
oops, I forgot, #1660 is only for 2.7 as it is an API change and is not backwards compatible, so testing for this should be on the 2.7 beta.
History
Date User Action Args
2014-09-26 05:35:21zyasoftsetassignee: fwierzbicki ->
resolution: remind
2013-02-20 00:07:39fwierzbickisetversions: + Jython 2.5, - 2.5.2rc
2013-02-07 18:05:13fwierzbickisetmessages: + msg7639
2013-02-07 18:01:44fwierzbickisetpriority: normal
assignee: fwierzbicki
2013-02-07 18:01:06fwierzbickisetnosy: + fwierzbicki
messages: + msg7638
2013-02-07 14:42:40treavessetmessages: + msg7637
2013-02-07 13:30:49amaksetfiles: + unnamed
messages: + msg7636
2013-02-07 13:15:53treavessetmessages: + msg7635
2013-02-06 21:38:16amaksetnosy: + amak
messages: + msg7632
2011-08-11 13:56:23crankycodersetnosy: + crankycoder
2011-06-16 20:54:18treavessettype: crash
versions: + 2.5.2rc, - Deferred
2011-06-15 19:58:08treavescreate