Issue2158

classification
Title: Incorrect handling of relative, star imports ("from .module import *")
Type: behaviour Severity: normal
Components: Core Versions: Jython 2.7
Milestone:
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: jeff.allen Nosy List: Arfrever, fwierzbicki, jeff.allen, tkanerva, zyasoft
Priority: high Keywords:

Created on 2014-06-02.01:31:17 by Arfrever, last changed 2015-05-04.21:38:17 by jeff.allen.

Messages
msg8590 (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) Date: 2014-06-02.01:31:16
$ rm -fr /tmp/tests
$ mkdir /tmp/tests
$ cd /tmp/tests
$ echo "from .b import *" > a.py

  1. Incorrect exception for relative import in non-package:

$ python2.7 -c 'import a'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "a.py", line 1, in <module>
    from .b import *
ValueError: Attempted relative import in non-package
$ jython2.7 -c 'import a'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "a.py", line 1, in <module>
    from .b import *
ImportError: No module named b
$ 

  2. Incorrect success of searching for module:

$ touch b.py
$ python2.7 -c 'import a'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "a.py", line 1, in <module>
    from .b import *
ValueError: Attempted relative import in non-package
$ jython2.7 -c 'import a'
$
msg8592 (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) Date: 2014-06-02.01:42:13
A variant of above problem #1 with funny exception message:

$ echo "from . import *" > c.py
$ python2.7 -c 'import c'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "c.py", line 1, in <module>
    from . import *
ValueError: Attempted relative import in non-package
$ jython2.7 -c 'import c'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "c.py", line 1, in <module>
    from . import *
ImportError: No module named 
$
msg8593 (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) Date: 2014-06-02.02:14:59
3. Multiple dot-levels incorrectly handled:

$ mkdir -p x/x/x
$ touch x/__init__.py x/x/__init__.py x/x/x/__init__.py
$ echo "from ...y import *" > x/x/x/x.py
$ echo "print(1)" > x/y.py
$ echo "print(2)" > x/x/y.py
$ echo "print(3)" > x/x/x/y.py
$ python2.7 -c 'import x.x.x.x'
1
$ jython2.7 -c 'import x.x.x.x'
3
msg8594 (view) Author: Jim Baker (zyasoft) Date: 2014-06-02.13:42:44
test_import also currently fails, so this may be capturing some of these reported cases.
msg9113 (view) Author: Jim Baker (zyasoft) Date: 2014-10-06.16:36:41
Presuming this part of test_import
msg9899 (view) Author: Jeff Allen (jeff.allen) Date: 2015-04-19.22:43:26
I'll take this and a couple of other import-related issues to see if I can get to the bottom of them.
msg9995 (view) Author: Jeff Allen (jeff.allen) Date: 2015-05-03.07:39:18
Got it. The mistake is in our code generation walk, which sets the level argument to default, whenever the from list is ("*",). A new test case, following one of Jim's excellent PEP 328 examples, fails that way. Now to work out what the logic ought to be at this point ... .
msg9996 (view) Author: Jeff Allen (jeff.allen) Date: 2015-05-03.20:04:06
Fixed in my repo (which appears here as %jt%), to appear centrally soon:

1. Correct error identified:

> type a.py
from .b import *
> type b.py
The system cannot find the file specified.
> %jt%\dist\bin\jython -c"import a"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "a.py", line 1, in <module>
    from .b import *
ValueError: Attempted relative import in non-package

2. And of course b.py is not found:

> echo print "Hello from b" > b.py
> jython -c"import a"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "a.py", line 1, in <module>
    from .b import *
ValueError: Attempted relative import in non-package

3. in my variant:

> tree/f u
C:\ ... \U
│   y.py
│   __init__.py
│
└───v
    │   y.py
    │   __init__.py
    │
    └───w
            x.py
            y.py
            __init__.py

> type u\v\w\x.py
print("File: u\\v\\w\\x.py")
from ...y import *
> type u\y.py
print("File: u\\y.py")
> type u\v\w\y.py
print("File: u\\v\\w\\y.py")

... and so on: all the files print their path.

> %jt%\dist\bin\jython -c"import u.v.w.x"
Init: u\__init__.py
Init: u\v\__init__.py
Init: u\v\w\__init__.py
File: u\v\w\x.py
File: u\y.py

This shows we get the right y.py.

These are all fixed by getting the level parameter right in code generation. If you demonstrate the bug (which still exists in the released 2.7.0), you must clean up the class files before demonstrating the fixed version.
msg9998 (view) Author: Jim Baker (zyasoft) Date: 2015-05-04.01:29:30
Jeff, re out-of-date class files, we handle this by a bytecode magic. The easiest way to see how updating the magic works is to look at a relevant commit: https://hg.python.org/jython/rev/ae190b66bf4c#l4.8, specifically incrementing the APIVersion by one
msg10004 (view) Author: Jeff Allen (jeff.allen) Date: 2015-05-04.09:15:09
I didn't really see the bug-fix as an "API change". But I'll add that as it could conceivably bite a user. We're not close to running out of integers yet.

Playing with this, if having ensured my *$py.class files are old, then I hide the source x.py, the old Jython (API version matches x$py.class) runs, with the expected wrong result, while the new Jython (API version one up from x$py.class) says:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named x

rather than anything more nuanced about what really happened.

In comparable circumstances, CPython says:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: Bad magic number in u\v\w\x.pyc

although I suppose that could be put better too.
msg10006 (view) Author: Jim Baker (zyasoft) Date: 2015-05-04.12:58:22
Jeff, so our convention has been to update the APIVersion for any bytecode change that results in different behavior - including especially bug fixes. I think we still need to do this with subsequent releases in 2.7.x, especially since we want to open up opportunities for future optimization around generated bytecode.

Te specific ImportError message could be more nuanced, however. In cases like this I especially favor helpful diagnostic messages vs something terse.
msg10007 (view) Author: Jim Baker (zyasoft) Date: 2015-05-04.13:18:34
Some useful background on pyc files and handling of bytecode magic - http://nedbatchelder.com/blog/200804/the_structure_of_pyc_files.html
msg10019 (view) Author: Jeff Allen (jeff.allen) Date: 2015-05-04.17:33:48
I saw that. All the apparatus for better diagnosis is there in the code, but it was turned off (accidentally I assume) a long time ago.

But to be yet more helpful, I have changed the words to:
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: incompatible api version 36 (expected 37) in: u\v\w\x$py.class
msg10020 (view) Author: Jim Baker (zyasoft) Date: 2015-05-04.17:47:44
Maybe change it to something like

ImportError: incompatible compiled version 36 (expected version 37, fix by recompiling) in: u\v\w\x$py.class 

I'm sure the above message text can be improved!
msg10021 (view) Author: Jeff Allen (jeff.allen) Date: 2015-05-04.19:07:30
I agree "api version" is not that helpful to the uninitiated. I'll find a suitable colour.
msg10024 (view) Author: Jeff Allen (jeff.allen) Date: 2015-05-04.21:38:17
Fixes the problem: https://hg.python.org/jython/rev/9483b3252bd5

Version change and error messages: https://hg.python.org/jython/rev/cb5bc04f17d1
History
Date User Action Args
2015-05-04 21:38:17jeff.allensetstatus: open -> closed
resolution: fixed
messages: + msg10024
2015-05-04 19:07:30jeff.allensetmessages: + msg10021
2015-05-04 17:47:45zyasoftsetmessages: + msg10020
2015-05-04 17:33:48jeff.allensetmessages: + msg10019
2015-05-04 13:18:34zyasoftsetmessages: + msg10007
2015-05-04 12:58:23zyasoftsetmessages: + msg10006
2015-05-04 09:15:09jeff.allensetmessages: + msg10004
2015-05-04 01:29:31zyasoftsetmessages: + msg9998
2015-05-03 20:04:07jeff.allensetmessages: + msg9996
2015-05-03 07:39:19jeff.allensetmessages: + msg9995
2015-04-19 22:43:26jeff.allensetassignee: jeff.allen
type: behaviour
messages: + msg9899
nosy: + jeff.allen
2014-10-06 16:36:41zyasoftsetpriority: high
messages: + msg9113
2014-06-02 13:42:44zyasoftsetmessages: + msg8594
2014-06-02 02:14:59Arfreversetmessages: + msg8593
2014-06-02 01:42:13Arfreversetmessages: + msg8592
2014-06-02 01:31:17Arfrevercreate