Issue2635

classification
Title: AST.lineno ignored by compile
Type: behaviour Severity: normal
Components: Core Versions: Jython 2.7
Milestone: Jython 2.7.1
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: elkniwt
Priority: Keywords: patch

Created on 2017-10-27.14:46:35 by elkniwt, last changed 2017-10-27.14:46:35 by elkniwt.

Files
File name Uploaded Description Edit Remove
lineno.patch elkniwt, 2017-10-27.14:46:33 patch to pay attention to AST.lineno changes
Messages
msg11632 (view) Author: Jim Peterson (elkniwt) Date: 2017-10-27.14:46:32
I work a lot with transforming ast trees.  I've noticed that the stack traces that come from errors in exec-ing a transformed tree will ignore any updates to lineno made to the tree.  For example, the following differences in behavior between python and jython:

0[sparkle:~/personal/src/hg/jython]2002: python
Python 2.7.13 (default, Jan 19 2017, 14:48:08) 
[GCC 6.3.0 20170118] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from ast import *
>>> from traceback import *
>>> tree = Module(body=[Expr(value=Name(id='x', ctx=Load()),lineno=42)])
>>> tree = fix_missing_locations(tree)
>>> exec compile(tree,'tree','exec')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "tree", line 42, in <module>
NameError: name 'x' is not defined

0[sparkle:~/personal/src/hg/jython]2003: dist/bin/jython                                                      
Jython 2.7.1 (default:d638b2c5ef28+, Oct 27 2017, 10:17:48)                                                   
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java1.8.0_131                                     
Type "help", "copyright", "credits" or "license" for more information.                                        
>>> from ast import *                                                                                         
>>> from traceback import *                                                                                   
>>> tree = Module(body=[Expr(value=Name(id='x', ctx=Load()),lineno=42)])                                      
>>> tree = fix_missing_locations(tree)                                                                        
>>> exec compile(tree,'tree','exec')                                                                          
Traceback (most recent call last):                                                                            
  File "<stdin>", line 1, in <module>                                                                         
  File "tree", line 1, in <module>     <========= difference           
NameError: name 'x' is not defined                                                                            

Notice that the line number specified for "tree" is 42 in python, but 1 in jython.

I believe that this is due to the fact that org.python.antlr.PythonTree has public getLine() and getCharPositionInLine() methods that wholly ignore the descendents' lineno and col_offset attributes.  These public methods are then used by the compiler to mark up line information.

I recommend making getLine() and getCharPositionInLine() protected, adding getLine() and getCol_offset() to PythonTree which calls the protected methods, and changing all references outside the PythonTree descendents to call getLineno() and getCol_offset(), instead.  Then, the descendents will override getLineno() and getCol_offset() with their implementations, which already check the lineno and col_offset attributes, but default to calling getLine() and getCharPositionInLine() when they have not been set.

I have attached a patch which implements these changes.  The patch includes a change to test_ast.py, which fails before the other changes are made, but passes afterward.  I was uncertain about the general testing naming, assertion philosophies, etc., so someone may wish to review it.

Thanks,
--Jim
History
Date User Action Args
2017-10-27 14:46:35elkniwtcreate