Issue2288

Used packages:
https://pypi.python.org/pypi/alabaster/0.7.2
https://pypi.python.org/pypi/Babel/1.3
https://pypi.python.org/pypi/docutils/0.12
https://pypi.python.org/pypi/Jinja2/2.7.3
https://pypi.python.org/pypi/MarkupSafe/0.23
https://pypi.python.org/pypi/mock/1.0.1
https://pypi.python.org/pypi/Pygments/2.0.2
https://pypi.python.org/pypi/roman/2.0.0
https://pypi.python.org/pypi/six/1.9.0
https://pypi.python.org/pypi/snowballstemmer/1.2.0
https://pypi.python.org/pypi/sphinx_rtd_theme/0.1.7
https://pypi.python.org/pypi/Sphinx/1.3

$ jython2.7 tests/run.py
Checking dependencies...
Temporary files will be placed in /tmp/Sphinx-1.3/tests/build.
Running Sphinx test suite (with Python 2.7b4+)...
.......Exception java.lang.NullPointerException: java.lang.NullPointerException in <bound method ZipFile.__del__ of <zipfile.ZipFile instance at 0x7f7>> ignored
.Exception java.lang.NullPointerException: java.lang.NullPointerException in <bound method ZipFile.__del__ of <zipfile.ZipFile instance at 0x7f8>> ignored
...........Exception java.lang.NullPointerException: java.lang.NullPointerException in <bound method ZipFile.__del__ of <zipfile.ZipFile instance at 0x800>> ignored
Exception java.lang.NullPointerException: java.lang.NullPointerException in <bound method ZipFile.__del__ of <zipfile.ZipFile instance at 0x801>> ignored
Exception java.lang.NullPointerException: java.lang.NullPointerException in <bound method ZipFile.__del__ of <zipfile.ZipFile instance at 0x802>> ignored
Exception java.lang.NullPointerException: java.lang.NullPointerException in <bound method ZipFile.__del__ of <zipfile.ZipFile instance at 0x803>> ignored
Exception java.lang.NullPointerException: java.lang.NullPointerException in <bound method ZipFile.__del__ of <zipfile.ZipFile instance at 0x804>> ignored
.........FException java.lang.NullPointerException: java.lang.NullPointerException in <bound method ZipFile.__del__ of <zipfile.ZipFile instance at 0x830>> ignored
Exception java.lang.NullPointerException: java.lang.NullPointerException in <bound method ZipFile.__del__ of <zipfile.ZipFile instance at 0x831>> ignored
EException java.lang.NullPointerException: java.lang.NullPointerException in <bound method ZipFile.__del__ of <zipfile.ZipFile instance at 0x838>> ignored
EException java.lang.NullPointerException: java.lang.NullPointerException in <bound method ZipFile.__del__ of <zipfile.ZipFile instance at 0x83d>> ignored
EEEEEEException java.lang.NullPointerException: java.lang.NullPointerException in <bound method ZipFile.__del__ of <zipfile.ZipFile instance at 0x856>> ignored
EEEEEEEE..Exception java.lang.NullPointerException: java.lang.NullPointerException in <bound method ZipFile.__del__ of <zipfile.ZipFile instance at 0x876>> ignored
Exception java.lang.NullPointerException: java.lang.NullPointerException in <bound method ZipFile.__del__ of <zipfile.ZipFile instance at 0x877>> ignored
Exception java.lang.NullPointerException: java.lang.NullPointerException in <bound method ZipFile.__del__ of <zipfile.ZipFile instance at 0x878>> ignored
EEEEEEEEEEEEEEEEEEEEEE........................EException java.lang.NullPointerException: java.lang.NullPointerException in <bound method ZipFile.__del__ of <zipfile.ZipFile instance at 0x1299>> ignored
Exception java.lang.NullPointerException: java.lang.NullPointerException in <bound method ZipFile.__del__ of <zipfile.ZipFile instance at 0x129a>> ignored
Exception java.lang.NullPointerException: java.lang.NullPointerException in <bound method ZipFile.__del__ of <zipfile.ZipFile instance at 0x129b>> ignored
Exception java.lang.NullPointerException: java.lang.NullPointerException in <bound method ZipFile.__del__ of <zipfile.ZipFile instance at 0x129c>> ignored
Exception java.lang.NullPointerException: java.lang.NullPointerException in <bound method ZipFile.__del__ of <zipfile.ZipFile instance at 0x129d>> ignored
Exception java.lang.NullPointerException: java.lang.NullPointerException in <bound method ZipFile.__del__ of <zipfile.ZipFile instance at 0x129e>> ignored
Exception java.lang.NullPointerException: java.lang.NullPointerException in <bound method ZipFile.__del__ of <zipfile.ZipFile instance at 0x129f>> ignored
Exception java.lang.NullPointerException: java.lang.NullPointerException in <bound method ZipFile.__del__ of <zipfile.ZipFile instance at 0x12a0>> ignored
................EFEEF..E..............................................Exception java.lang.NullPointerException: java.lang.NullPointerException in <bound method ZipFile.__del__ of <zipfile.ZipFile instance at 0x39b1>> ignored
Exception java.lang.NullPointerException: java.lang.NullPointerException in <bound method ZipFile.__del__ of <zipfile.ZipFile instance at 0x39b2>> ignored
.E.....Exception java.lang.NullPointerException: java.lang.NullPointerException in <bound method ZipFile.__del__ of <zipfile.ZipFile instance at 0x3bed>> ignored
EE......................E...................E..........EEEEE.EEEE.
======================================================================
ERROR: test_build.test_build_all('dirhtml', path(u'/tmp/Sphinx-1.3/tests/build/\u65e5\u672c\u8a9e'))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/share/jython-2.7/Lib/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/tmp/Sphinx-1.3/tests/test_build.py", line 40, in verify_build
    app.builder.build_all()
  File "/tmp/Sphinx-1.3/sphinx/builders/__init__.py", line 200, in build_all
    self.build(None, summary='all source files', method='all')
  File "/tmp/Sphinx-1.3/sphinx/builders/__init__.py", line 253, in build
    updated_docnames = set(self.env.update(self.config, self.srcdir,
  File "/tmp/Sphinx-1.3/sphinx/environment.py", line 617, in update
    self._read_serial(docnames, app)
  File "/tmp/Sphinx-1.3/sphinx/environment.py", line 637, in _read_serial
    self.read_doc(docname, app)
  File "/tmp/Sphinx-1.3/sphinx/environment.py", line 790, in read_doc
    pub.publish()
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/core.py", line 216, in publish
    self.document = self.reader.read(self.source, self.parser,
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/core.py", line 216, in publish
    self.document = self.reader.read(self.source, self.parser,
  File "/tmp/Sphinx-1.3/sphinx/environment.py", line 125, in read
    self.parse()
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/readers/__init__.py", line 78, in parse
    self.parser.parse(self.input, document)
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/parsers/rst/__init__.py", line 172, in parse
    self.statemachine.run(inputlines, document, inliner=self.inliner)
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/parsers/rst/states.py", line 169, in run
    results = StateMachineWS.run(self, input_lines, input_offset,
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/statemachine.py", line 238, in run
    context, next_state, result = self.check_line(
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/statemachine.py", line 238, in run
    context, next_state, result = self.check_line(
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/statemachine.py", line 238, in run
    context, next_state, result = self.check_line(
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/statemachine.py", line 460, in check_line
    return method(match, context, next_state)
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/parsers/rst/states.py", line 2726, in underline
    self.section(title, source, style, lineno - 1, messages)
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/parsers/rst/states.py", line 327, in section
    self.new_subsection(title, lineno, messages)
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/parsers/rst/states.py", line 393, in new_subsection
    newabsoffset = self.nested_parse(
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/parsers/rst/states.py", line 281, in nested_parse
    state_machine.run(block, input_offset, memo=self.memo,
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/parsers/rst/states.py", line 195, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/statemachine.py", line 238, in run
    context, next_state, result = self.check_line(
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/statemachine.py", line 238, in run
    context, next_state, result = self.check_line(
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/statemachine.py", line 238, in run
    context, next_state, result = self.check_line(
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/statemachine.py", line 460, in check_line
    return method(match, context, next_state)
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/parsers/rst/states.py", line 2301, in explicit_markup
    self.explicit_list(blank_finish)
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/parsers/rst/states.py", line 2326, in explicit_list
    newline_offset, blank_finish = self.nested_list_parse(
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/parsers/rst/states.py", line 318, in nested_list_parse
    state_machine.run(block, input_offset, memo=self.memo,
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/parsers/rst/states.py", line 195, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/statemachine.py", line 238, in run
    context, next_state, result = self.check_line(
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/statemachine.py", line 238, in run
    context, next_state, result = self.check_line(
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/statemachine.py", line 238, in run
    context, next_state, result = self.check_line(
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/statemachine.py", line 460, in check_line
    return method(match, context, next_state)
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/parsers/rst/states.py", line 2604, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/parsers/rst/states.py", line 2311, in explicit_construct
    return method(self, expmatch)
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/parsers/rst/states.py", line 2311, in explicit_construct
    return method(self, expmatch)
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/parsers/rst/states.py", line 2053, in directive
    return self.run_directive(
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/parsers/rst/states.py", line 2103, in run_directive
    result = directive_instance.run()
  File "/usr/share/jython-2.7/Lib/site-packages/docutils/parsers/rst/states.py", line 2103, in run_directive
    result = directive_instance.run()
  File "/tmp/Sphinx-1.3/sphinx/ext/autodoc.py", line 1467, in run
    documenter.generate(more_content=self.content)
  File "/tmp/Sphinx-1.3/sphinx/ext/autodoc.py", line 776, in generate
    if not self.import_object():
  File "/tmp/Sphinx-1.3/sphinx/ext/autodoc.py", line 385, in import_object
    __import__(self.modname)
  File "/tmp/Sphinx-1.3/sphinx/ext/autodoc.py", line 385, in import_object
    __import__(self.modname)
IllegalArgumentException: java.lang.IllegalArgumentException: Cannot create PyString with non-byte value

…

======================================================================
ERROR: test_ext_viewcode.test_viewcode
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/share/jython-2.7/Lib/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/tmp/Sphinx-1.3/tests/util.py", line 228, in deco
    func(app, status, warning, *args2, **kwargs2)
  File "/tmp/Sphinx-1.3/tests/test_ext_viewcode.py", line 19, in test_viewcode
    app.builder.build_all()
  File "/tmp/Sphinx-1.3/sphinx/builders/__init__.py", line 200, in build_all
    self.build(None, summary='all source files', method='all')
  File "/tmp/Sphinx-1.3/sphinx/builders/__init__.py", line 253, in build
    updated_docnames = set(self.env.update(self.config, self.srcdir,
  File "/tmp/Sphinx-1.3/sphinx/environment.py", line 617, in update
    self._read_serial(docnames, app)
  File "/tmp/Sphinx-1.3/sphinx/environment.py", line 637, in _read_serial
    self.read_doc(docname, app)
  File "/tmp/Sphinx-1.3/sphinx/environment.py", line 809, in read_doc
    app.emit('doctree-read', doctree)
  File "/tmp/Sphinx-1.3/sphinx/application.py", line 497, in emit
    results.append(callback(self, *args))
  File "/tmp/Sphinx-1.3/sphinx/ext/viewcode.py", line 83, in doctree_read
    modname = _get_full_modname(app, modname, fullname)
  File "/tmp/Sphinx-1.3/sphinx/ext/viewcode.py", line 28, in _get_full_modname
    return get_full_modname(modname, attribute)
  File "/tmp/Sphinx-1.3/sphinx/ext/viewcode.py", line 28, in _get_full_modname
    return get_full_modname(modname, attribute)
  File "/tmp/Sphinx-1.3/sphinx/util/__init__.py", line 269, in get_full_modname
    __import__(modname)
NullPointerException: java.lang.NullPointerException

…

Looks like 3 separate issues. Adding Stefan to nosy list re __del__ and NPE; my reading of https://docs.python.org/2/reference/datamodel.html#object.__del__ is that org.python.core.finalization.FinalizeTrigger#finalize should suppress the exception - printing it to stderr - instead of throwing Throwable.

>FinalizeTrigger#finalize should suppress the exception - printing it to stderr - instead of throwing Throwable.

Exceptions are currently not printed to stderr, but *should* already be suppressed. Doc of java.lang.Object#finalize() states:

If an uncaught exception is thrown by the finalize method, the exception is ignored and finalization of that object terminates.

and

Any exception thrown by the finalize method causes the finalization of this object to be halted, but is otherwise ignored.


Of course we can try whether it makes a difference, but according to these lines from the doc it should not. (Printing the exception to stderr should be added to match CPython behavior though.)

I guess the exception is somehow stored in the current ThreadState's exception-field (via Py.setException(...) ?), bypassing the ignore-exceptions-from-finalizers strategy of Java's gc-thread.

So removing "throws throwable" from FinalizeTrigger#finalize would probably not fix it. I think the finalize trigger would have to look at
ThreadState#exception, print it and clear it before it returns. Otherwise the exception is not entirely suppressed from Jython's point of view.

I'd suggest to write a minimalized test for this (as opposed to running the entire Sphinx test suite) to pin the issue and verify the fix without additional blur. However I won't find time for this before sunday.

On the other hand it should be quested why ZipFile.__del__ causes a NullPointerException anyway. Looks like a bug for me. Iff this is normal behavior (in an exceptional case though), shouldn't the exception be something more Python-like?
Something like TypeError or ValueError complaining something is None that shouldn't be? This should be investigated by someone familiar with the zipfile module.

Taking a quick look at zipfile.ZipFile code, there is this suspicious line in the class-body:
fp = None                   # Set here since __del__ checks it
But __del__ (i.e. close()) only looks at self.fp. So maybe the line should be in __init__ and read self.fp = None. However __init__ actually seems to init self.fp to *something* in any case. So I don't think this causes the issue, but it is still weird.

It seems, an exception in a finalizer usually does not cause a test to fail:

class GCTests_Finalization_Exception(unittest.TestCase):

    def test1(self):
        class A:
            def __del__(self):
                print "A __del__"
                A.txt = "text1"
                GCTestHelper.causeNullPointerException()
                A.txt = "text2"

        a = A()
        del a
        gc.collect()
        time.sleep(1)
        print "collect done"
        print A.txt
        self.assertEqual(A.txt, "text1")
        ts = Py.getThreadState()
        print ts.exception


Output is:

A __del__
Exception java.lang.NullPointerException: java.lang.NullPointerException in <bound method A.__del__ of <__main__.A instance at 0x2>> ignored
collect done
text1
None
.
----------------------------------------------------------------------
Ran 1 test in 1.209s

OK

So...
either NPE in __del__ is not the cause for the failure or things work different in the Sphinx test suite (maybe the issue only manifests with nose-testing). I will investigate this further.

I tried to compare current Jython exception-in-__del__ behavior to that of CPython. Of course, in CPython one cannot produce Java NullPointerExceptions, so I rewrote the test to use an ordinary python exception by calling "print None.test":

class GCTests_Finalization_Exception_CPythonCompliant(unittest.TestCase):

    def test1(self):
        class A:
            def __del__(self):
                print "A __del__"
                A.txt = "text1"
                print None.test
                A.txt = "text2"

        a = A()
        del a
        gc.collect()
        time.sleep(1)
        print "collect done"
        print A.txt
        self.assertEqual(A.txt, "text1")

if __name__ == "__main__":
    unittest.main()


The output of Jython is in principal equal that of CPython.
CPython:

A __del__
Exception AttributeError: "'NoneType' object has no attribute 'test'" in <bound method A.__del__ of <__main__.A instance at 0x166dd88>> ignored
collect done
text1
.
----------------------------------------------------------------------
Ran 1 test in 1.005s

OK


Jython:

A __del__
Exception AttributeError: "'NoneType' object has no attribute 'test'" in <bound method A.__del__ of <__main__.A instance at 0x2>> ignored
collect done
text1
.
----------------------------------------------------------------------
Ran 1 test in 1.309s

OK


So I seriously doubt that the error is finalization-related at all.
tbc

I finally managed to run the sphinx test suite on my own. However I cannot reproduce the output above. Especially I don't get any java.lang.NullPointerException in <bound method ZipFile.__del__ of <zipfile.ZipFile instance at 0x800>>

Instead I get tons of errors reading:
.../Babel-1.3/build/lib.linux-x86_64-2.7/babel/localtime/_unix.py", line 52, in _get_localzone
    pos = link_dst.find('/zoneinfo/')
AttributeError: 'NoneType' object has no attribute 'find'




Btw, CPython 2.7.5+ does not pass the Sphinx test suite error-less as well. I think this is due to some python2/python3 incompatibility:

.../sphinx-master/sphinx/ext/intersphinx.py", line 44, in <module>
    handlers = [request.ProxyHandler(), request.HTTPRedirectHandler(),
AttributeError: 'Module_six_moves_urllib_request' object has no attribute 'HTTPRedirectHandler'

> Instead I get tons of errors reading:
> .../Babel-1.3/build/lib.linux-x86_64-2.7/babel/localtime/_unix.py", line 52, in _get_localzone
>     pos = link_dst.find('/zoneinfo/')
> AttributeError: 'NoneType' object has no attribute 'find'

It is bug #2292.
(On my system /etc/localtime is a symlink to /usr/share/zoneinfo/posix/Poland.)

> Btw, CPython 2.7.5+ does not pass the Sphinx test suite error-less as well. I think this is due to some python2/python3 incompatibility:

Test suite of Sphinx passes for me with CPython 2.7.10_pre20150315.

> .../sphinx-master/sphinx/ext/intersphinx.py", line 44, in <module>
>     handlers = [request.ProxyHandler(), request.HTTPRedirectHandler(),
> AttributeError: 'Module_six_moves_urllib_request' object has no attribute 'HTTPRedirectHandler'

Do you use old version of six?

>It is bug #2292.
Thanks for pinning the actual bug behind my observation!

>(On my system /etc/localtime is a symlink to /usr/share/zoneinfo/posix/Poland.)
On my system it appears to be some non-human-readable binary file that is also not recognizable as a symlink or something. Maybe Babel has fallbacks for situations like this, but just can't leverage them due to bug #2292...

>Do you use old version of six?
Good point. I am using 1.9.0 too, but now that you mention it, I think I only configured Jython to use it, not CPython (I installed all dependencies locally by hand so I can modify them for debugging without messing up the system); I will retry with this fixed.

re IllegalArgumentException, this occurs during import of modules that are not ascii, eg u'/Users/jbaker/opensource/Sphinx-1.3.1/tests/build/\u65e5\u672c\u8a9e'

I believe/speculate this triggers a subsequent NullPointerException when the entry is not found in sys.modules.

Related issues include http://bugs.python.org/issue1342 for CPython

Defer this aspect to post 2.7.0, no way we have time to do this support now

Re importing unicode paths, maybe Sphinx is running Python 3 tests?

Running on Ubuntu 14.10:

$ python2
Python 2.7.8 (default, Oct 20 2014, 15:05:19)
[GCC 4.9.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> __import__(u"español")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf1' in position 4: ordinal not in range(128)

$ python3
Python 3.4.2 (default, Oct  8 2014, 13:08:17)
[GCC 4.9.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> __import__(u"español")
<module 'español' from '/home/jbaker/español.py'>

I filed some problems as separate bugs: #2293, #2294, #2295, #2296
I think that #2293 is more likely related to java.lang.IllegalArgumentException triggered by test suite of Sphinx.

In running the Sphinx test suite, I have traced it down: it's due to the use PyInstance (old style class), vs new style class in zipfile.ZipFile:

java.lang.Exception: Stack trace
	at java.lang.Thread.dumpStack(Thread.java:1342)
	at org.python.core.PyMethod.toString(PyMethod.java:327)
	at java.util.Formatter$FormatSpecifier.printString(Formatter.java:2838)
	at java.util.Formatter$FormatSpecifier.print(Formatter.java:2718)
	at java.util.Formatter.format(Formatter.java:2488)
	at java.util.Formatter.format(Formatter.java:2423)
	at java.lang.String.format(String.java:2797)
	at org.python.core.Py.writeUnraisable(Py.java:1285)
	at org.python.core.PyInstance.__del__(PyInstance.java:1958)
	at org.python.core.finalization.FinalizeTrigger.runFinalizer(FinalizeTrigger.java:88)
	at org.python.core.finalization.FinalizeTrigger.performFinalization(FinalizeTrigger.java:170)
	at org.python.core.finalization.FinalizeTrigger.finalize(FinalizeTrigger.java:209)
	at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
	at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:101)
	at java.lang.ref.Finalizer.access$100(Finalizer.java:32)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:190)
Exception java.lang.NullPointerException: java.lang.NullPointerException in <bound method ZipFile.__del__ of <zipfile.ZipFile instance at 0x122>> ignored

(A little instrumentation can help - I did this in PyMethod#toString)

The immediate fix is to convert ZipFile to a newstyle class - this is in fact what it is in CPython 2.7. We can review oldstyle classes at some future point to see if this can be fixed as well.

FWIW, ZipFile as fixed as of https://hg.python.org/jython/rev/57f8839eaac6 and the preceding commit, but we can look at the other issues as well, assuming they are not all broken out at this point.