Issue2199
Created on 2014-09-07.12:44:33 by Adel, last changed 2016-01-13.22:55:54 by zyasoft.
msg8959 (view) |
Author: Adel (Adel) |
Date: 2014-09-07.12:44:32 |
|
Memory Leak on the ShutdownCloser Objects that exist in a static map inside PySystemState that used in the Python Interpreter when create many Interpreter. These objects of type "ShutdownCloser" preserve in the memory and don't remove although if you call GC explicitly.
|
msg8960 (view) |
Author: Adel (Adel) |
Date: 2014-09-07.12:49:45 |
|
You can duplicate the case by run the following code and monitor the memory:-
public static void main(String[] args) {
try {
Thread.sleep(10 * 1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
PySystemState pySystemState = null;
PythonInterpreter interp = null;
for (int i = 1; i <= 100000; i++) {
pySystemState = new PySystemState();
interp = new PythonInterpreter(null, pySystemState);
interp.cleanup();
pySystemState.cleanup();
System.out.println("PythonInterpreter number [" + i + "] has been created and cleanup.");
}
System.gc();
// Please take Heap dump and check the number of objects for type "org.python.core.PySystemState$PySystemStateCloser$ShutdownCloser"
try {
Thread.sleep(5 * 60 * 1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
|
msg8961 (view) |
Author: Adel (Adel) |
Date: 2014-09-07.12:59:42 |
|
Please check the screenshot in the attachment section. It show the memory before and after run the upper code, and the number of objects that still in the memory although after we run the full GC
|
msg8965 (view) |
Author: Jim Baker (zyasoft) |
Date: 2014-09-07.23:15:18 |
|
I didn't have the patience to run this for the full 100,000 iterations, however I did verify to approx 1500 iterations of your example that the cleanup is occurring with the fix as of http://hg.python.org/jython/rev/d8233316cf09 (part of Jython 2.7 beta 3). In particular heap usage on my system stays at approx 18MB.
Please note we will not be backporting these fixes to 2.5 since they rely on Java 7 and some backwards breaking Jython API changes.
|
msg8971 (view) |
Author: Adel (Adel) |
Date: 2014-09-08.07:59:47 |
|
I fixed the issue by clear the reference for the "shutdownHook" instance variable in the "cleanup" method that exist in the static inner class PySystemStateCloser. It is only the last line in the cleanup method, As the following:-
private synchronized void cleanup() {
if (isCleanup) {
return;
}
this.isCleanup = true;
if (shutdownHook != null) {
try {
Runtime.getRuntime().removeShutdownHook(shutdownHook);
} catch (IllegalStateException e) {
}
}
for (Callable callable : this.resourceClosers) {
try {
callable.call();
} catch (Exception e) {
}
}
resourceClosers.clear();
// The below line is the only change to fix the issue.
+ shutdownHook = null;
}
Clear the strong reference which is "shutdownHook" instance variable inside "PySystemStateCloser" class, will remove all "ShutdownCloser" objects from the memory when call GC, since all objects of type "ShutdownCloser" is added to a static concurrent map called "syscloser" in the class "PySystemState" as a weak reference and this weak reference will be removed when call GC if the object doesn't has any strong reference to it or doesn't contains any strong reference, and the "shutdownHook" instance variable is a strong reference inside "ShutdownCloser" objects. So after remove its reference on the cleanup method, the GC will remove the objects from the static concurrent map "syscloser" and then from the memory.
After that, Is it possible to fix it inside 2.5 version?
|
msg8975 (view) |
Author: Jim Baker (zyasoft) |
Date: 2014-09-09.00:11:42 |
|
That looks like a very reasonable fix.
Target 2.5.4 - we need another release candidate, this should go in it
|
msg9025 (view) |
Author: Adel (Adel) |
Date: 2014-09-24.06:52:32 |
|
Is there a specific date for 2.5.4 to be released?
|
msg9028 (view) |
Author: Jim Baker (zyasoft) |
Date: 2014-09-24.18:31:15 |
|
We haven't set a specific date. It will not likely be before 2.7 RC1 however, since that should have some fixes to backport in general for 2.5.4
|
msg9179 (view) |
Author: Jim Baker (zyasoft) |
Date: 2014-10-28.23:18:03 |
|
This is not a bug for 2.7.0, so changing priority so it's out of the way
|
msg10624 (view) |
Author: Jim Baker (zyasoft) |
Date: 2016-01-11.04:28:57 |
|
Frank, assigning this to you, since I think you're the only committer who has expressed interest in working on 2.5.4. We may just want to close that out completely - limited resources and all.
|
|
Date |
User |
Action |
Args |
2016-01-13 22:55:54 | zyasoft | set | title: Memory Leak when create many Interpreter -> [2.5] Memory Leak when create many Interpreters |
2016-01-11 04:28:57 | zyasoft | set | assignee: fwierzbicki messages:
+ msg10624 |
2014-10-28 23:18:03 | zyasoft | set | priority: high -> normal messages:
+ msg9179 |
2014-09-24 18:31:15 | zyasoft | set | priority: high messages:
+ msg9028 |
2014-09-24 06:52:32 | Adel | set | messages:
+ msg9025 |
2014-09-09 00:11:54 | zyasoft | set | nosy:
+ fwierzbicki |
2014-09-09 00:11:42 | zyasoft | set | status: pending -> open resolution: works for me -> accepted messages:
+ msg8975 |
2014-09-08 07:59:47 | Adel | set | messages:
+ msg8971 |
2014-09-07 23:15:19 | zyasoft | set | status: open -> pending resolution: works for me messages:
+ msg8965 nosy:
+ zyasoft |
2014-09-07 12:59:42 | Adel | set | messages:
+ msg8961 |
2014-09-07 12:52:39 | Adel | set | files:
+ The memory After.jpg |
2014-09-07 12:52:26 | Adel | set | files:
+ The memory Before.jpg |
2014-09-07 12:49:46 | Adel | set | files:
+ Number Of Objects for ShutdownCloser.jpg messages:
+ msg8960 |
2014-09-07 12:47:27 | Adel | set | files:
- Number Of Class ShutdownCloser.jpg |
2014-09-07 12:44:33 | Adel | create | |
|