Message12330

The desired feature is that we can have Python and Java packages by the same name, in the same place. The history of this is quite amusing to dig up, although the serious worry is that while it seems reasonable to expect, it might not be feasible without unacceptable side-effects.

The feature first appears in 2000 in https://hg.python.org/jython/file/7d2145df96de/org/python/core/PyModule.java#l17 where PyModule gets a __findattr__ that looks up missing attributes as sub-modules. The way this works morphs soon after: impAttr is invented roughly as it is now and __findattr__ calls it indirectly via impHook() and builtin.__import__(), see: https://hg.python.org/jython/file/e65ab08ab34e/org/python/core/PyModule.java#l72. (How short and simple everything seems.) This is relased in Jython 2.0.

A lot later (2008) we "remove the magic that automatically imports a module": https://hg.python.org/jython/rev/6d306db1e625. __findattr__ is simplified. This is how it looks in Jython 2.5.

__findattr__ is superseded by __findattr_ex__ for reasons we needn't go into, but PyModule doesn't need to override it, so what's left of PyModule.__findattr__ is simply removed.

In 2010, Jim adds a custom __findattr_ex__  to PyModule that calls impAttr to "re-enable mixing Java and Python code in the same directory" in https://hg.python.org/jython/rev/51ec6962b521. This was issue #1653. It is released in Jython 2.5.2.

Then Philip removes __findattr_ex__ while fixing #1811 (https://hg.python.org/jython/rev/4c672dbbcdb2#l4.8), although it is not clear that it is necessary to the fix. We release it this way in 2.5.3 and 2.7.0.

Darjus puts it back in https://hg.python.org/jython/rev/ac01360b4252 to fix #2455, and this is released in Jython 2.7.1. That re-introduces the automagical import of Python modules that this issue is about.

So, now we find ourselves considering removal again, but clearly we've been there and it's more subtle than that. It almost seems we don't quite know what we want. I imagine what we want is:

1. Python 2 semantics apply to Python packages and modules defined by __init__.py files and modules in packages as .py files. (All applies equally to $py.class and .pyc equivalents, and inside archives.)

2. A compiled Java packages is treated like a Python module (which may be a Python package), except a __init__ is not necessary to mark a package, and classes in the Java package define attributes of the Python module and are not valid targets for import. (This applies equally to file systems inside archives.)

3. These rules apply even when the directory in question is both a Python package and a Java package co-located (on the open file system or in an archive).

I believe this is possible in principle, but delicate. There is probably a rule missing too, that excepts Java classes in a package from rule 2, if they are intended to implement Python modules. A Java class implementing a module should not appear to be an attribute until explicitly imported.

I think this means that the return from __import__() ought always to be a PyModule, whereas at the moment it can return a module, javapackage or even a type object. Those modules containing or defined by Java classes may have values of __path__ and other attributes that CPython would not produce, but would still be modules.

This may be too complicated to get in 2.7.2, but I don't see an acceptable compromise position at the moment other than the divergent automagical behaviour.