Issue1775
Created on 2011-07-22.07:16:30 by pekka.klarck, last changed 2013-02-26.18:08:14 by fwierzbicki.
msg6575 (view) |
Author: Pekka Klärck (pekka.klarck) |
Date: 2011-07-22.07:16:29 |
|
We noticed in our project that Jython leaks memory when calling zlib.compress. With something like 10000 calls compression started to be very slow and sometimes the end result was OutOfMemoryError.
I looked at the source code of Jython's zlib module and investigated java.util.zip.Deflater [1] it uses internally. I noticed that Deflator.end() as never called and after adding that to the end of _get_deflate_data helper things improved considerably.
[1] http://download.oracle.com/javase/1.5.0/docs/api/java/util/zip/Deflater.html
In our project we ended up implementing a custom version of compress based on the Jython's zlib.compress implementation. This version, for example, only instantiates Deflater once and calls its reset method when one string has been compressed. In the end our version is ~25% faster than zlib.compress with the fix I explained above.
I can create a patch to fix this issue and study could the enhancements we did to our compress be added too.
|
msg6576 (view) |
Author: Pekka Klärck (pekka.klarck) |
Date: 2011-07-22.07:30:21 |
|
For anyone interested, below is our compress implementation. It's not fully generic (i.e. doesn't allow setting compression level) but works great for us.
from java.util.zip import Deflater
import jarray
DEFLATOR = Deflater(9, False)
def compress(text):
DEFLATOR.setInput(text)
DEFLATOR.finish()
buf = jarray.zeros(1024, 'b')
compressed = []
while not DEFLATOR.finished():
length = DEFLATOR.deflate(buf, 0, 1024)
compressed.append(buf[:length].tostring())
DEFLATOR.reset()
return ''.join(compressed)
|
msg7796 (view) |
Author: Frank Wierzbicki (fwierzbicki) |
Date: 2013-02-26.18:08:14 |
|
This has been fixed as of 2.5.4rc1 and 2.7b1.
|
|
Date |
User |
Action |
Args |
2013-02-26 18:08:14 | fwierzbicki | set | status: open -> closed resolution: fixed messages:
+ msg7796 nosy:
+ fwierzbicki |
2011-07-22 07:30:22 | pekka.klarck | set | messages:
+ msg6576 |
2011-07-22 07:16:30 | pekka.klarck | create | |
|