Issue2593

classification
Title: file.write(obj) if obj is not a str or buffer raises j.l.NullPointerException
Type: behaviour Severity: major
Components: Core Versions: Jython 2.7
Milestone: Jython 2.7.1
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: jeff.allen Nosy List: fwierzbicki, jeff.allen, tkanerva, zyasoft
Priority: urgent Keywords:

Created on 2017-05-21.21:19:44 by tkanerva, last changed 2017-06-10.13:25:49 by zyasoft.

Files
File name Uploaded Description Edit Remove
write_file_w_list.py tkanerva, 2017-05-21.21:19:44 write_file_w_list
Messages
msg11394 (view) Author: topi kanerva (tkanerva) Date: 2017-05-21.21:21:51
The title is misleading or a bit wrong.
Writing a list instead of a string is how the erroneous behaviour is triggered.
msg11421 (view) Author: Jim Baker (zyasoft) Date: 2017-06-09.02:32:26
In Jython 2.7.1rc2, as well as 2.7.0 (so no regression):

$ ~/jython2.7.1rc2/bin/jython t-write-list.py
Traceback (most recent call last):
  File "t-write-list.py", line 3, in <module>
    f.write(lst)
java.lang.NullPointerException
	at org.python.google.common.base.CharMatcher.matchesAllOf(CharMatcher.java:635)
	at org.python.core.Py.newStringOrUnicode(Py.java:674)
	at org.python.core.Py.newStringOrUnicode(Py.java:659)
	at org.python.core.PyException.<init>(PyException.java:61)
	at org.python.core.Py.TypeError(Py.java:265)
	at org.python.core.PyFile.asWritable(PyFile.java:512)
	at org.python.core.PyFile.file_write(PyFile.java:450)
	at org.python.core.PyFile$file_write_exposer.__call__(Unknown Source)
	at org.python.core.PyObject.__call__(PyObject.java:484)
	at org.python.pycode._pyx0.f$0(t-write-list.py:3)
	at org.python.pycode._pyx0.call_function(t-write-list.py)
	at org.python.core.PyTableCode.call(PyTableCode.java:171)
	at org.python.core.PyCode.call(PyCode.java:18)
	at org.python.core.Py.runCode(Py.java:1615)
	at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:296)
	at org.python.util.jython.run(jython.java:362)
	at org.python.util.jython.main(jython.java:142)
java.lang.NullPointerException: java.lang.NullPointerException

In Python 2.7:

$ python t-write-list.py
Traceback (most recent call last):
  File "t-write-list.py", line 3, in <module>
    f.write(lst)
TypeError: must be string or buffer, not list

Note that if the object being written is not a built-in type, one will get the following error:

$ python t-write-obj.py
Traceback (most recent call last):
  File "t-write-obj.py", line 9, in <module>
    f.write(foo)
TypeError: must be convertible to a buffer, not Foo

Ideally we can get this fixed for 2.7.1, since it is an easily observed NPE, which we try to never see in Jython.
msg11422 (view) Author: Jim Baker (zyasoft) Date: 2017-06-09.02:33:58
This looks like a blocker for us, and should be part of RC3, but should also be a very easy fix.
msg11425 (view) Author: Jeff Allen (jeff.allen) Date: 2017-06-09.21:42:22
Very. And my mistake.
msg11426 (view) Author: Jim Baker (zyasoft) Date: 2017-06-09.22:17:02
So this diff fixes by providig the appropriate TypeError - we are not using the formatted string as the new message. Note that for non-native types, CPython has a slightly different message "must be convertible to a buffer, not Foo", but I don't see this as being necessary for this fix (unless easy).

$ hg diff
diff -r c558ce4072ee src/org/python/core/PyFile.java
--- a/src/org/python/core/PyFile.java	Wed Jun 07 08:29:12 2017 +0100
+++ b/src/org/python/core/PyFile.java	Fri Jun 09 16:10:23 2017 -0600
@@ -506,7 +506,7 @@

         if (message == null) {
             // Messages differ for text or binary streams (CPython) but we always add the type
-            String.format("%s buffer, not %.200s", (binary ? "must be string or"
+            message = String.format("%s buffer, not %.200s", (binary ? "must be string or"
                     : "expected a character"), obj.getType().fastGetName());
         }
         throw Py.TypeError(message);

There is a somewhat similar problem in pickle/cPickle, but at least not a NPE. As we saw with Py.newStringOrUnicode, we might want to put this conversion logic in a common place:

Jython 2.7.1rc2 (default:c558ce4072ee+, Jun 9 2017, 16:10:36)
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java1.8.0_112
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> pickle.loads(47)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jbaker/jythondev/jython27/dist/Lib/pickle.py", line 1381, in loads
    file = StringIO(str)
TypeError: StringIO(): 1st arg can't be coerced to java.lang.CharSequence, org.python.core.PyArray
>>> import cPickle
>>> cPickle.loads(47)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
cPickle.UnpicklingError: invalid load key, '4'.
>>>
jimbaker:jython27 jbaker$ python
Python 2.7.10 |Anaconda 2.3.0 (x86_64)| (default, May 28 2015, 17:04:42)
[GCC 4.2.1 (Apple Inc. build 5577)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://binstar.org
>>> import pickle, cPickle
>>> pickle.loads(47)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jbaker/anaconda/lib/python2.7/pickle.py", line 1381, in loads
    file = StringIO(str)
TypeError: must be string or buffer, not int
>>> cPickle.loads(47)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: must be string, not int
>>>
msg11429 (view) Author: Jeff Allen (jeff.allen) Date: 2017-06-10.06:35:51
I went for fixing just the issue at hand, given we're on the eve of release. Now at https://hg.python.org/jython/rev/5d023369ff15.
msg11432 (view) Author: Jim Baker (zyasoft) Date: 2017-06-10.13:25:49
Works now as expected
History
Date User Action Args
2017-06-10 13:25:49zyasoftsetstatus: pending -> closed
resolution: accepted -> fixed
messages: + msg11432
2017-06-10 06:35:51jeff.allensetstatus: open -> pending
messages: + msg11429
2017-06-09 22:17:04zyasoftsetmessages: + msg11426
2017-06-09 21:42:22jeff.allensetassignee: jeff.allen
messages: + msg11425
2017-06-09 02:33:58zyasoftsetnosy: + fwierzbicki, jeff.allen
messages: + msg11422
2017-06-09 02:32:28zyasoftsetseverity: minor -> major
title: writing a buffer (instead of str) fails with NullPointerException instead of TypeError -> file.write(obj) if obj is not a str or buffer raises j.l.NullPointerException
nosy: + zyasoft
messages: + msg11421
priority: urgent
milestone: Jython 2.7.1
resolution: accepted
2017-05-21 21:21:51tkanervasetmessages: + msg11394
2017-05-21 21:19:44tkanervacreate