Issue1761

Actually the version I am using is 2.5.2, but the Versions list does not offer that as a selection.  I upgraded to 2.5.2 from 2.5.2rc4 to see if this issue is resolved.

This issue may be 1660 - that issue is marked closed, but, if it is not that issue, it is another, very similar one.

I use threading for work in my code.  It's code copied from one of the Python doc pages.

class Worker(Thread):
	def __init__(self, tasks):
		Thread.__init__(self)
		self.tasks = tasks
		self.daemon = True
		self.start()

	def run(self):
		while True:
			func, args, kargs = self.tasks.get()
			try:
				func(*args, **kargs)
			except Exception, exception:
				logging.error(exception, extra={"sbs_app":"Worker"})
			self.tasks.task_done()

class ThreadPool:
	def __init__(self, num_threads):
		self.tasks = Queue(num_threads)
		for _ in range(num_threads):
			Worker(self.tasks)

	def add_task(self, func, *args, **kargs):
		self.tasks.put((func, args, kargs), True)

	def wait_completion(self):
		self.tasks.join()


In this case, Jython is being executed by ODI, an Oracle product (I don't really think that's relevant, but...).  I run this code with 32 threads, and then use the wait_for_completion().  All tasks run and finish fine.  However, none of the 32 threads ever goes away.

I run jconsole, and when I kick off my process, I see the thread count increase, typically by 35 threads (some are ODI's).  When the process has finished, and ODI finishes it's stuff, the delta is always exactly 32; it just keeps growing.  Eventually, the VM runs out of memory, or the ulimit is hit.

In jconsole, all of the threads are still visible, and show:
Name: Thread
State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@57cac7
Total blocked: 38  Total waited: 140

Stack trace: 
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
org.python.modules._threading.Condition.Condition_wait(Condition.java:87)
org.python.modules._threading.Condition$Condition_wait_exposer.__call__(Unknown Source)
org.python.core.PyObject.__call__(PyObject.java:375)
Queue$py.get$12(/u01/app/oracle/product/ODI/oracledi_J251/lib/scripting/Lib/Queue.py:179)
Queue$py.call_function(/u01/app/oracle/product/ODI/oracledi_J251/lib/scripting/Lib/Queue.py)
org.python.core.PyTableCode.call(PyTableCode.java:165)
org.python.core.PyBaseCode.call(PyBaseCode.java:301)
org.python.core.PyBaseCode.call(PyBaseCode.java:127)
org.python.core.PyFunction.__call__(PyFunction.java:317)
org.python.core.PyMethod.__call__(PyMethod.java:109)
sbs.threadsupport$py.run$3(/opt/csw/lib/python/site-packages/ImageProcessingPipeline-2.3.1-py2.6.egg/sbs/threadsupport.py:44)
sbs.threadsupport$py.call_function(/opt/csw/lib/python/site-packages/ImageProcessingPipeline-2.3.1-py2.6.egg/sbs/threadsupport.py)
org.python.core.PyTableCode.call(PyTableCode.java:165)
org.python.core.PyBaseCode.call(PyBaseCode.java:134)
org.python.core.PyFunction.__call__(PyFunction.java:317)
org.python.core.PyMethod.__call__(PyMethod.java:109)
threading$py._Thread__bootstrap$33(/u01/app/oracle/product/ODI/oracledi_J251/lib/scripting/Lib/threading.py:223)
threading$py.call_function(/u01/app/oracle/product/ODI/oracledi_J251/lib/scripting/Lib/threading.py)
org.python.core.PyTableCode.call(PyTableCode.java:165)
org.python.core.PyBaseCode.call(PyBaseCode.java:301)
org.python.core.PyBaseCode.call(PyBaseCode.java:194)
org.python.core.PyFunction.__call__(PyFunction.java:387)
org.python.core.PyMethod.instancemethod___call__(PyMethod.java:220)
org.python.core.PyMethod.__call__(PyMethod.java:211)
org.python.core.PyMethod.__call__(PyMethod.java:201)
org.python.core.PyMethod.__call__(PyMethod.java:196)
org.python.core.FunctionThread.run(FunctionThread.java:21)


Line 223 in threading.py is the except clause of the attempt at deleting self, when an exception has occurred. So for some reason, the threads do not seem able to self-destruct.

Any help getting this taken care will be greatly appreciated.

Now that this issue

http://bugs.jython.org/issue1761

Has been fixed, is this one still an issue?

What makes you think this issue has been fixed?

Sorry, bad paste of URL.

Now that issue #1660 has been fixed

http://bugs.jython.org/issue1660

Is this issue (#1761) still happening?

I see. :)

Yes, this issue still happens with 2.5.2.  1660 was closed in 2010, yet this bug still clearly existed in 2011 when I commented on 1660.  I wasn't sure if this defect was a new one (a regression introduced after 1660 was fixed), or if 1660 was never actually fixed (the fix was defective itself).

But this thread consumption still occurs, and with JConsole is easy to see.  I've had to write my own thread code to get around this.

#1660 was *just* fixed, hopefully it resolves this issue as well (but no binary release has this fix yet). I am in the process of finding any more low hanging fixes - as soon as I am done with that I will be putting out a 2.5.4 rc and a 2.7 beta.

oops, I forgot, #1660 is only for 2.7 as it is an API change and is not backwards compatible, so testing for this should be on the 2.7 beta.

I checked this on 2.7.2a1, observed with jvisualvm, and I can reproduce what I understand is the OP's issue. "When the process has finished, and ODI finishes it's stuff, the delta is always exactly 32; it just keeps growing." So the OP's application is run repeatedly in the same JVM, and each time the threads remain alive (parked).

My test program is attached. It creates successive pools of 32 threads. A given pool behaves in the textbook way, but when the pool is dereferenced the JVM threads of the old one persist, so the JVM thread count shows a rising staircase. Other garbage collection is clearly going on happily. 

My difficulty is to know whether that is actually incorrect behaviour. Worker.run does not return, so the thread should not finish. This stackoverflow post, with almost identical code (run on CPython I assume), is helpful:
https://stackoverflow.com/questions/18610144/python-threads-are-not-being-garbage-collected

I'm taking off 2.5 as a target as we won't fix that as far as I know.

This is really old, and I think based in a misunderstanding identified above.