Issue2622

classification
Title: json dumps error
Type: behaviour Severity: normal
Components: Versions: Jython 2.7
Milestone: Jython 2.7.2
process
Status: pending Resolution: fixed
Dependencies: Superseder:
Assigned To: zyasoft Nosy List: birkoff, stefan.richthofer, zyasoft
Priority: Keywords:

Created on 2017-08-31.13:37:10 by birkoff, last changed 2017-09-11.03:33:57 by zyasoft.

Messages
msg11558 (view) Author: birkoff (birkoff) Date: 2017-08-31.13:37:09
When I use PyZabbix ( https://github.com/adubkov/py-zabbix ) with simple code:

from pyzabbix import ZabbixMetric, ZabbixSender

# Send metrics to zabbix trapper
packet = [
    ZabbixMetric('hostname', 'test', 1),
]

result = ZabbixSender('127.0.0.1').send(packet)

Get error

Traceback (most recent call last):
  File "zabbix.py", line 8, in <module>
    result = ZabbixSender('127.0.0.1').send(packet)
  File "D:\Codes\zpy\pyzabbix\sender.py", line 383, in send
    result.parse(self._chunk_send(metrics[m:m + self.chunk_size]))
  File "D:\Codes\zpy\pyzabbix\sender.py", line 341, in _chunk_send
    messages = self._create_messages(metrics)
  File "D:\Codes\zpy\pyzabbix\sender.py", line 254, in _create_messages
    messages.append(str(m))
  File "D:\Codes\zpy\pyzabbix\sender.py", line 128, in __repr__
    result = json.dumps(self.__dict__)
  File "D:\Codes\zpy\jython-standalone-2.7.1.jar\Lib\json\__init__.py", line 243, in dumps
  File "D:\Codes\zpy\jython-standalone-2.7.1.jar\Lib\json\encoder.py", line 206, in encode
  File "D:\Codes\zpy\jython-standalone-2.7.1.jar\Lib\json\encoder.py", line 269, in iterencode
  File "D:\Codes\zpy\jython-standalone-2.7.1.jar\Lib\json\encoder.py", line 183, in default
TypeError: {'host': 'hostname', 'value': '1', 'key': 'test'} is not JSON serializable

---

On cpython 2.7.10 all work perfect

---

import json

data = {'host': 'hostname', 'value': '1', 'key': 'test'}

jsonf = json.dumps (data)

In Jython 2.7.1 works without any errors.
msg11559 (view) Author: birkoff (birkoff) Date: 2017-08-31.14:57:24
Did a dirty hack in pyzabbix\sender.py", line 128, in __repr__

result = json.dumps(self.__dict__)

change to

d = self.__dict__
jsond = {'hostname' : d['host'], 'value' : d['value'], 'key' : d['key']}
result = json.dumps(jsond)

---

But why self.__dict__ is not true dict object?
msg11570 (view) Author: Jim Baker (zyasoft) Date: 2017-09-05.18:49:11
So this can be readily reproduced:

$ jython27
Jython 2.7.1 (default:0df7adb1b397, Jun 30 2017, 19:02:43)
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java1.8.0_144
Type "help", "copyright", "credits" or "license" for more information.
>>> import json
>>> class X(object): pass
...
>>> x = X()
>>> x.foo = 42
>>> x.baz = "abc"
>>> x.__dict__
{'baz': 'abc', 'foo': 42}
>>> json.dumps(x.__dict__)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jbaker/jython2.7.1/Lib/json/__init__.py", line 243, in dumps
    return _default_encoder.encode(obj)
  File "/Users/jbaker/jython2.7.1/Lib/json/encoder.py", line 206, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Users/jbaker/jython2.7.1/Lib/json/encoder.py", line 269, in iterencode
    return _iterencode(o, 0)
  File "/Users/jbaker/jython2.7.1/Lib/json/encoder.py", line 183, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: {'baz': 'abc', 'foo': 42} is not JSON serializable

The workaround is also trivial, but obivously doesn't work well in existing code like Zabbix:

>>> json.dumps(dict(x.__dict__))
'{"foo": 42, "baz": "abc"}'

So we should fix this bug in the json module. Unfortunately, for a variety of historical reasons, the underlying object in Jython for __dict__ is actually something called PyStringMap, which actually isn't so restricted as its name implies. Maybe we can fix this in Jython 3; previous attempts at this refactoring didn't work however. For now, we have to do this workaround for code that directly sees the underlying Java representation, as is the case for the json module implementation.
msg11572 (view) Author: Jim Baker (zyasoft) Date: 2017-09-06.02:03:12
Fixed as of https://hg.python.org/jython/rev/efb72c73a78f

Always nice to have something easy to fix, as is the case here!
msg11573 (view) Author: birkoff (birkoff) Date: 2017-09-06.13:05:45
Thanks!
msg11575 (view) Author: Stefan Richthofer (stefan.richthofer) Date: 2017-09-07.08:12:49
Jim, I'd like to veto this fix. While it might work well, remember we unified PyDictionary and PyStringMap under AbstractDict.
So I'd suggest to modify Encoder.java to accept any AbstractDict in
https://github.com/jythontools/jython/blob/master/src/org/python/modules/_json/Encoder.java#L141

and also switch to AbstractDict in

https://github.com/jythontools/jython/blob/master/src/org/python/modules/_json/Encoder.java#L119

With that no further modification should be required (and it avoids to copy a dict).
AbstractDict was intended to be the migration path from PyDictionary/PyStringmap to a proper solution. I'd suggest we move to AbstractDict wherever issues regarding this pop up.
msg11579 (view) Author: Jim Baker (zyasoft) Date: 2017-09-08.04:58:17
Stefan, good point, and thanks for the review. (Commit, then review, is perfectly reasonable IMHO.) I will make the change to use AbstractDict.
msg11580 (view) Author: Jim Baker (zyasoft) Date: 2017-09-09.20:02:47
Now uses AbstractDict in https://hg.python.org/jython/rev/8d52ad2559f5
msg11587 (view) Author: Stefan Richthofer (stefan.richthofer) Date: 2017-09-10.21:12:39
Alright. One slight note:
The PyDictionary case can be removed then:
https://hg.python.org/jython/rev/8d52ad2559f5#l1.20
A PyDictionary import shoudn't be required any more.

(This is really minor, but I think it's better to keep the code as clean as possible.)
msg11588 (view) Author: Jim Baker (zyasoft) Date: 2017-09-11.03:33:56
Sounds good about removing the unnecessary PyDictionary import. I will also go ahead and remove similar unnecessary PyStringMap imports, which are at least in two files, as part of this cleanup.
History
Date User Action Args
2017-09-11 03:33:57zyasoftsetmessages: + msg11588
2017-09-10 21:12:39stefan.richthofersetmessages: + msg11587
2017-09-09 20:02:48zyasoftsetmessages: + msg11580
2017-09-08 04:58:18zyasoftsetmessages: + msg11579
2017-09-07 08:12:50stefan.richthofersetnosy: + stefan.richthofer
messages: + msg11575
2017-09-06 13:05:45birkoffsetmessages: + msg11573
2017-09-06 02:03:25zyasoftsetstatus: open -> pending
assignee: zyasoft
2017-09-06 02:03:12zyasoftsetresolution: accepted -> fixed
messages: + msg11572
2017-09-05 18:49:12zyasoftsetresolution: accepted
messages: + msg11570
nosy: + zyasoft
milestone: Jython 2.7.1 -> Jython 2.7.2
2017-08-31 14:57:24birkoffsetmessages: + msg11559
2017-08-31 13:37:10birkoffcreate