Issue2635

classification
Title: AST.lineno ignored by compile
Type: behaviour Severity: normal
Components: Core Versions: Jython 2.7
Milestone: Jython 2.7.2
process
Status: open Resolution: accepted
Dependencies: Superseder:
Assigned To: Nosy List: elkniwt, fwierzbicki, jeff.allen
Priority: normal Keywords: patch

Created on 2017-10-27.14:46:35 by elkniwt, last changed 2018-02-26.21:13:16 by jeff.allen.

Files
File name Uploaded Description Edit Remove
lineno.patch elkniwt, 2017-10-27.14:46:33 patch to pay attention to AST.lineno changes
Messages
msg11632 (view) Author: Jim Peterson (elkniwt) Date: 2017-10-27.14:46:32
I work a lot with transforming ast trees.  I've noticed that the stack traces that come from errors in exec-ing a transformed tree will ignore any updates to lineno made to the tree.  For example, the following differences in behavior between python and jython:

0[sparkle:~/personal/src/hg/jython]2002: python
Python 2.7.13 (default, Jan 19 2017, 14:48:08) 
[GCC 6.3.0 20170118] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from ast import *
>>> from traceback import *
>>> tree = Module(body=[Expr(value=Name(id='x', ctx=Load()),lineno=42)])
>>> tree = fix_missing_locations(tree)
>>> exec compile(tree,'tree','exec')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "tree", line 42, in <module>
NameError: name 'x' is not defined

0[sparkle:~/personal/src/hg/jython]2003: dist/bin/jython                                                      
Jython 2.7.1 (default:d638b2c5ef28+, Oct 27 2017, 10:17:48)                                                   
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java1.8.0_131                                     
Type "help", "copyright", "credits" or "license" for more information.                                        
>>> from ast import *                                                                                         
>>> from traceback import *                                                                                   
>>> tree = Module(body=[Expr(value=Name(id='x', ctx=Load()),lineno=42)])                                      
>>> tree = fix_missing_locations(tree)                                                                        
>>> exec compile(tree,'tree','exec')                                                                          
Traceback (most recent call last):                                                                            
  File "<stdin>", line 1, in <module>                                                                         
  File "tree", line 1, in <module>     <========= difference           
NameError: name 'x' is not defined                                                                            

Notice that the line number specified for "tree" is 42 in python, but 1 in jython.

I believe that this is due to the fact that org.python.antlr.PythonTree has public getLine() and getCharPositionInLine() methods that wholly ignore the descendents' lineno and col_offset attributes.  These public methods are then used by the compiler to mark up line information.

I recommend making getLine() and getCharPositionInLine() protected, adding getLine() and getCol_offset() to PythonTree which calls the protected methods, and changing all references outside the PythonTree descendents to call getLineno() and getCol_offset(), instead.  Then, the descendents will override getLineno() and getCol_offset() with their implementations, which already check the lineno and col_offset attributes, but default to calling getLine() and getCharPositionInLine() when they have not been set.

I have attached a patch which implements these changes.  The patch includes a change to test_ast.py, which fails before the other changes are made, but passes afterward.  I was uncertain about the general testing naming, assertion philosophies, etc., so someone may wish to review it.

Thanks,
--Jim
msg11720 (view) Author: Jeff Allen (jeff.allen) Date: 2018-02-26.21:13:16
Jim:

Thanks for working on this. On a quick scan of the patch, this looks like a thorough job. We'll take a look. (Frank: care to? One of us eventually.)

It seems an odd thinking error to have passed us by for (probably) so long, that "org.python.antlr.PythonTree has public getLine() and getCharPositionInLine() methods that wholly ignore the descendents' lineno and col_offset attributes". This area doesn't get the sort of rapid change that breaks things.
History
Date User Action Args
2018-02-26 21:13:16jeff.allensetpriority: normal
nosy: + jeff.allen, fwierzbicki
resolution: accepted
messages: + msg11720
milestone: Jython 2.7.1 -> Jython 2.7.2
2017-10-27 14:46:35elkniwtcreate