Issue1775

classification
Title: zlib module leaks memory due to missing Deflater.end()
Type: Severity: normal
Components: Versions:
Milestone:
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: fwierzbicki, pekka.klarck
Priority: Keywords:

Created on 2011-07-22.07:16:30 by pekka.klarck, last changed 2013-02-26.18:08:14 by fwierzbicki.

Messages
msg6575 (view) Author: Pekka Klärck (pekka.klarck) Date: 2011-07-22.07:16:29
We noticed in our project that Jython leaks memory when calling zlib.compress. With something like 10000 calls compression started to be very slow and sometimes the end result was OutOfMemoryError.

I looked at the source code of Jython's zlib module and investigated java.util.zip.Deflater [1] it uses internally. I noticed that Deflator.end() as never called and after adding that to the end of  _get_deflate_data helper things improved considerably.

[1] http://download.oracle.com/javase/1.5.0/docs/api/java/util/zip/Deflater.html

In our project we ended up implementing a custom version of compress based on the Jython's zlib.compress implementation. This version, for example, only instantiates Deflater once and calls its reset method when one string has been compressed. In the end our version is ~25% faster than zlib.compress with the fix I explained above. 

I can create a patch to fix this issue and study could the enhancements we did to our compress be added too.
msg6576 (view) Author: Pekka Klärck (pekka.klarck) Date: 2011-07-22.07:30:21
For anyone interested, below is our compress implementation. It's not fully generic (i.e. doesn't allow setting compression level) but works great for us.


from java.util.zip import Deflater
import jarray

DEFLATOR = Deflater(9, False)

def compress(text):
    DEFLATOR.setInput(text)
    DEFLATOR.finish()
    buf = jarray.zeros(1024, 'b')
    compressed = []
    while not DEFLATOR.finished():
        length = DEFLATOR.deflate(buf, 0, 1024)
        compressed.append(buf[:length].tostring())
    DEFLATOR.reset()
    return ''.join(compressed)
msg7796 (view) Author: Frank Wierzbicki (fwierzbicki) Date: 2013-02-26.18:08:14
This has been fixed as of 2.5.4rc1 and 2.7b1.
History
Date User Action Args
2013-02-26 18:08:14fwierzbickisetstatus: open -> closed
resolution: fixed
messages: + msg7796
nosy: + fwierzbicki
2011-07-22 07:30:22pekka.klarcksetmessages: + msg6576
2011-07-22 07:16:30pekka.klarckcreate