Issue2449

classification
Title: requests library causes memory leak and 100% GC time when handling keep-alive
Type: crash Severity: normal
Components: Library Versions: Jython 2.7
Milestone: Jython 2.7.0
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: darjus, latifrons, zyasoft
Priority: Keywords:

Created on 2015-12-30.06:57:04 by latifrons, last changed 2016-01-03.22:01:08 by darjus.

Files
File name Uploaded Description Edit Remove
2015-12-30_145402.png latifrons, 2015-12-30.06:57:02 VisualVM Screenshot
Messages
msg10575 (view) Author: latifrons (latifrons) Date: 2015-12-30.06:57:02
requests library will cause memory leak on Jython when the target server is set to keep-alive and the maxKeepAliveRequests=-1 (unlimited)

Sample Tomcat server.xml configuration to reproduce the problem:

<Connector connectionTimeout="20000" maxThreads="100" port="8181"
    maxKeepAliveRequests="-1" 
    keepAliveTimeout="300000"
    protocol="org.apache.coyote.http11.Http11NioProtocol" redirectPort="8443"/>

======================
Sample python code (no problem when using Python 2.7):

from threading import Thread
import requests
import time

d = 'POST DATA'

def a():
    s = requests.Session()
    s.trust_env = False
    while True:
        resp = s.post('http://sample.com',data=d,headers = {'Content-Type':'application/json'})
        resp.encoding = 'utf-8'
        print len(resp.text),time.time()

for i in range(20):
    t = Thread(target=a)
    t.start()

while True:
    time.sleep(1000)


======================
[]$ /apps/lpsadmin/jdk1.8.0_25/bin/jstat -gcutil 3072 1s
  S0     S1     E      O      M     CCS    YGC     YGCT    FGC    FGCT     GCT   
 20.98   0.00  95.67  50.47  91.15  95.76     72    4.045     2    0.143    4.188
  0.00  95.35  52.35  51.89  91.15  95.76     73    4.080     2    0.143    4.223
 20.70   0.00  20.76  53.29  91.15  95.76     74    4.115     2    0.143    4.258
 20.70   0.00  97.44  53.29  91.15  95.76     74    4.115     2    0.143    4.258
  0.00  97.72  73.99  53.29  91.15  95.76     75    4.150     2    0.143    4.293
 48.61   0.00  33.77  53.29  91.15  95.76     76    4.190     2    0.143    4.333
 48.61   0.00  81.00  53.29  91.15  95.76     76    4.190     2    0.143    4.333
  0.00  59.74  59.67  53.29  91.15  95.76     77    4.235     2    0.143    4.378
 69.17   0.00  30.18  53.29  91.15  95.76     78    4.286     2    0.143    4.428
 69.17  43.48 100.00  54.02  91.15  95.76     79    4.286     2    0.143    4.428
  0.00  64.22  79.09  54.55  91.15  95.76     79    4.345     2    0.143    4.488
 63.10   0.00  52.62  55.40  91.15  95.76     80    4.401     2    0.143    4.544
  0.00  98.57  16.72  57.04  91.15  95.76     81    4.470     2    0.143    4.613
  0.00  98.57  95.24  57.04  91.15  95.76     81    4.470     2    0.143    4.613
 46.98   0.00  65.85  58.77  91.15  95.76     82    4.523     2    0.143    4.666
  0.00  37.98  40.66  60.54  91.15  95.76     83    4.571     2    0.143    4.714
 24.98   0.00  11.54  62.59  91.15  95.76     84    4.627     2    0.143    4.770
 24.98   0.00  81.58  62.59  91.15  95.76     84    4.627     2    0.143    4.770
  0.00  25.76  55.75  64.05  91.15  95.76     85    4.665     2    0.143    4.808
 26.22   0.00  25.68  65.55  91.15  95.76     86    4.701     2    0.143    4.844
 26.22   0.00  89.92  65.55  91.15  95.76     86    4.701     2    0.143    4.844
  0.00  26.57  63.18  67.04  91.15  95.76     87    4.739     2    0.143    4.882
 27.87   0.00  36.69  68.57  91.15  95.76     88    4.777     2    0.143    4.920
  0.00  27.95   8.99  70.09  91.15  95.76     89    4.816     2    0.143    4.958
  0.00  27.95  81.64  70.09  91.15  95.76     89    4.816     2    0.143    4.958
 29.14   0.00  51.78  71.62  91.15  95.76     90    4.853     2    0.143    4.996
  0.00  29.60  25.41  73.17  91.15  95.76     91    4.891     2    0.143    5.034
  0.00  29.60  93.34  73.17  91.15  95.76     91    4.891     2    0.143    5.034
 31.29   0.00  64.18  74.69  91.15  95.76     92    4.929     2    0.143    5.072
  0.00  32.32  34.08  76.31  91.15  95.76     93    4.969     2    0.143    5.112
 34.14   0.00   2.05  77.88  91.15  95.76     94    5.009     2    0.143    5.152
 34.14   0.00  71.19  77.88  91.15  95.76     94    5.009     2    0.143    5.152
  0.00  34.49  43.58  79.49  91.15  95.76     95    5.050     2    0.143    5.193
 26.66  34.49 100.00  81.04  91.15  95.76     96    5.050     2    0.143    5.193
 37.21   0.00  56.60  81.15  91.15  95.76     96    5.134     2    0.143    5.277
 37.21   0.00  87.03  81.15  91.15  95.76     96    5.134     2    0.143    5.277
  0.00  39.46  23.16  82.79  91.15  95.76     97    5.175     2    0.143    5.318
  0.00  39.46  63.83  82.79  91.15  95.76     97    5.175     2    0.143    5.318
 43.43   0.00  28.68  84.43  91.15  95.76     98    5.216     2    0.143    5.359
 43.43   0.00  76.11  84.43  91.15  95.76     98    5.216     2    0.143    5.359
  0.00  74.46  18.74  84.43  91.15  95.76     99    5.259     2    0.143    5.401
  0.00  74.46  45.63  84.43  91.15  95.76     99    5.259     2    0.143    5.401
  0.00  74.46  88.79  84.43  91.15  95.76     99    5.259     2    0.143    5.401
 91.38   0.00  30.25  84.43  91.15  95.76    100    5.309     2    0.143    5.452
 91.38   0.00  73.04  84.43  91.15  95.76    100    5.309     2    0.143    5.452
  0.00  99.91  16.74  84.43  91.15  95.76    101    5.365     2    0.143    5.508
  0.00  99.91  52.57  84.43  91.15  95.76    101    5.365     2    0.143    5.508
 99.91   0.00  25.27  85.48  91.15  95.76    102    5.439     2    0.143    5.581
  0.00  99.93   0.00  87.90  91.15  95.76    103    5.505     2    0.143    5.647
  0.00  99.93  74.45  87.90  91.15  95.76    103    5.505     2    0.143    5.647
 60.21   0.00  60.70  90.10  91.15  95.76    104    5.567     2    0.143    5.709
  0.00  96.42  30.82  93.10  91.15  95.76    105    5.623     2    0.143    5.765
 24.25   0.00  14.75  95.38  91.15  95.76    106    5.680     2    0.143    5.823
 24.25   0.00  94.18  95.38  91.15  95.76    106    5.680     2    0.143    5.823
  0.00  24.67  76.00  96.94  91.15  95.76    107    5.721     2    0.143    5.863
 25.16   0.00   0.00  98.46  91.15  95.76    108    5.759     3    0.143    5.902
 25.16   0.00   0.00  98.46  91.15  95.76    108    5.759     3    0.143    5.902
  0.00   0.00  80.40  86.01  91.00  95.67    108    5.759     3    2.405    8.164
  0.00  37.01  47.35  86.01  91.00  95.67    109    5.788     3    2.405    8.193
 27.36   0.00  19.44  88.09  91.00  95.67    110    5.833     3    2.405    8.238
 27.36   0.00  99.80  88.09  91.00  95.67    111    5.833     3    2.405    8.238
  0.00  28.23  78.82  89.61  91.00  95.67    111    5.871     3    2.405    8.276
 29.76   0.00  51.06  91.19  91.00  95.67    112    5.910     3    2.405    8.315
  0.00  30.74  22.98  92.81  91.00  95.67    113    5.950     3    2.405    8.355
  0.00  30.74  98.99  92.81  91.00  95.67    113    5.950     3    2.405    8.355
 31.73   0.00  79.99  94.42  91.00  95.67    114    5.989     3    2.405    8.394
  0.00  33.13  54.78  96.03  91.00  95.67    115    6.029     3    2.405    8.435
 36.05   0.00   0.00  97.67  91.00  95.67    116    6.069     4    2.405    8.474
  0.00   0.00  59.93  92.22  90.96  95.63    116    6.069     4    3.621    9.690
  0.00  52.06  20.92  92.22  90.96  95.63    117    6.099     4    3.621    9.720
  0.00  52.06  89.32  92.22  90.96  95.63    117    6.099     4    3.621    9.720
 78.76   0.00  27.60  92.22  90.96  95.63    118    6.138     4    3.621    9.758
 78.76  81.69 100.00  92.22  90.96  95.63    119    6.138     4    3.621    9.758
  0.00  98.08  83.66  92.22  90.96  95.63    119    6.183     4    3.621    9.804
 99.88   0.00  51.65  92.36  90.96  95.63    120    6.235     4    3.621    9.856
  0.00  60.15  39.67  95.40  90.96  95.63    121    6.295     4    3.621    9.916
 67.74  60.15 100.00  95.40  90.96  95.63    122    6.295     4    3.621    9.916
 74.77   0.00  82.65  95.40  90.96  95.63    122    6.348     4    3.621    9.969
  0.00  99.11  72.54  95.40  90.96  95.63    123    6.409     4    3.621   10.030
 89.16   0.00  59.43  95.40  90.96  95.63    124    6.476     4    3.621   10.096
  0.00  98.64  46.62  96.55  90.96  95.63    125    6.546     4    3.621   10.167
 57.89   0.00   0.00  99.01  90.96  95.63    126    6.616     5    3.621   10.236
 57.89   0.00   0.00  99.01  90.96  95.63    126    6.616     5    3.621   10.236
 17.33   0.00  92.42  99.87  90.96  95.63    126    6.616     5    5.792   12.408
 17.33   0.00 100.00  99.87  90.96  95.63    126    6.616     6    5.792   12.408
  0.00   0.00  56.21 100.00  90.96  95.63    126    6.616     6    7.323   13.939
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616     7    7.323   13.939
  0.00   0.00   8.51  99.95  90.96  95.63    126    6.616     7    8.895   15.511
  0.00   0.00 100.00  99.95  90.96  95.63    126    6.616     8    8.895   15.511
  0.00   0.00 100.00  99.95  90.96  95.63    126    6.616     8    8.895   15.511
  0.00   0.00 100.00  99.86  90.96  95.63    126    6.616     9   10.525   17.141
  0.00   0.00 100.00  99.86  90.96  95.63    126    6.616     9   10.525   17.141
  0.00   0.00 100.00  99.85  90.96  95.63    126    6.616    10   12.048   18.663
  0.00   0.00 100.00  99.85  90.96  95.63    126    6.616    10   12.048   18.663
  0.00   0.00 100.00  99.85  90.96  95.63    126    6.616    11   13.588   20.204
  0.00   0.00 100.00  99.85  90.96  95.63    126    6.616    11   13.588   20.204
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    12   14.734   21.350
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    12   14.734   21.350
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    13   16.251   22.866
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    14   17.781   24.397
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    14   17.781   24.397
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    15   19.330   25.945
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    15   19.330   25.945
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    16   20.904   27.520
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    16   20.904   27.520
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    17   22.483   29.099
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    17   22.483   29.099
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    18   24.046   30.662
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    19   25.617   32.233
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    19   25.617   32.233
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    20   27.208   33.824
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    20   27.208   33.824
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    21   28.760   35.375
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    22   30.334   36.950
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    22   30.334   36.950
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    23   31.908   38.523
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    23   31.908   38.523
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    24   33.538   40.154
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    25   35.154   41.770
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    25   35.154   41.770
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    26   36.709   43.324
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    26   36.709   43.324
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    27   38.290   44.905
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    28   39.639   46.255
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    29   40.829   47.445
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    29   40.829   47.445
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    30   42.412   49.027
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    30   42.412   49.027
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    31   43.976   50.592
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    32   45.545   52.161
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    32   45.545   52.161
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    33   47.103   53.718
  0.00   0.00  50.77 100.00  90.96  95.63    126    6.616    33   48.678   55.293
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    34   48.678   55.293
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    35   50.287   56.902
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    35   50.287   56.902
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    36   51.894   58.510
  0.00   0.00  53.94 100.00  90.96  95.63    126    6.616    36   53.443   60.058
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    37   53.443   60.058
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    38   54.853   61.469
  0.00   0.00 100.00 100.00  90.96  95.63    126    6.616    38   54.853   61.469


VisualVM shows most of the memory are spent on LinkedBlockingQueue.
msg10578 (view) Author: Darjus Loktevic (darjus) Date: 2016-01-03.10:31:06
Confirmed
msg10579 (view) Author: Darjus Loktevic (darjus) Date: 2016-01-03.12:07:40
The no threaded version does not leak, though even if i do threading with 1 thread it seems to leak. Digging deeper.
msg10581 (view) Author: Jim Baker (zyasoft) Date: 2016-01-03.16:53:25
LinkedBlockingQueue is used to handle incoming chunks for recv, as part of the inbound handler.
msg10582 (view) Author: Darjus Loktevic (darjus) Date: 2016-01-03.22:01:07
Hey Jim,

Saw that too. That's as far as i got yesterday :)
History
Date User Action Args
2016-01-03 22:01:08darjussetmessages: + msg10582
2016-01-03 16:53:25zyasoftsetmessages: + msg10581
2016-01-03 16:23:59zyasoftsetnosy: + zyasoft
2016-01-03 12:07:41darjussetmessages: + msg10579
2016-01-03 10:31:07darjussetnosy: + darjus
messages: + msg10578
2015-12-30 06:57:04latifronscreate