Message11632

I work a lot with transforming ast trees.  I've noticed that the stack traces that come from errors in exec-ing a transformed tree will ignore any updates to lineno made to the tree.  For example, the following differences in behavior between python and jython:

0[sparkle:~/personal/src/hg/jython]2002: python
Python 2.7.13 (default, Jan 19 2017, 14:48:08) 
[GCC 6.3.0 20170118] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from ast import *
>>> from traceback import *
>>> tree = Module(body=[Expr(value=Name(id='x', ctx=Load()),lineno=42)])
>>> tree = fix_missing_locations(tree)
>>> exec compile(tree,'tree','exec')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "tree", line 42, in <module>
NameError: name 'x' is not defined

0[sparkle:~/personal/src/hg/jython]2003: dist/bin/jython                                                      
Jython 2.7.1 (default:d638b2c5ef28+, Oct 27 2017, 10:17:48)                                                   
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java1.8.0_131                                     
Type "help", "copyright", "credits" or "license" for more information.                                        
>>> from ast import *                                                                                         
>>> from traceback import *                                                                                   
>>> tree = Module(body=[Expr(value=Name(id='x', ctx=Load()),lineno=42)])                                      
>>> tree = fix_missing_locations(tree)                                                                        
>>> exec compile(tree,'tree','exec')                                                                          
Traceback (most recent call last):                                                                            
  File "<stdin>", line 1, in <module>                                                                         
  File "tree", line 1, in <module>     <========= difference           
NameError: name 'x' is not defined                                                                            

Notice that the line number specified for "tree" is 42 in python, but 1 in jython.

I believe that this is due to the fact that org.python.antlr.PythonTree has public getLine() and getCharPositionInLine() methods that wholly ignore the descendents' lineno and col_offset attributes.  These public methods are then used by the compiler to mark up line information.

I recommend making getLine() and getCharPositionInLine() protected, adding getLine() and getCol_offset() to PythonTree which calls the protected methods, and changing all references outside the PythonTree descendents to call getLineno() and getCol_offset(), instead.  Then, the descendents will override getLineno() and getCol_offset() with their implementations, which already check the lineno and col_offset attributes, but default to calling getLine() and getCharPositionInLine() when they have not been set.

I have attached a patch which implements these changes.  The patch includes a change to test_ast.py, which fails before the other changes are made, but passes afterward.  I was uncertain about the general testing naming, assertion philosophies, etc., so someone may wish to review it.

Thanks,
--Jim