Issue2305

Since https://hg.python.org/jython/rev/23c3effa5d4f sys.ps1 and sys.ps2 do not exist in non-interactive mode but they are still mentioned in dir(sys):

$ jython2.7 -c 'import sys; print(dir(sys))'
['JYTHON_DEV_JAR', 'JYTHON_JAR', 'PYTHON_CACHEDIR', 'PYTHON_CACHEDIR_SKIP', 'PYTHON_CONSOLE_ENCODING', 'PYTHON_IO_ENCODING', 'PYTHON_IO_ERRORS', '__delattr__', '__dict__', '__displayhook__', '__excepthook__', '__findattr_ex__', '__name__', '__new__', '__rawdir__', '__setattr__', '__stderr__', '__stdin__', '__stdout__', '_getframe', '_jy_console', '_mercurial', '_systemRestart', 'add_classdir', 'add_extdir', 'add_package', 'argv', 'builtin_module_names', 'builtins', 'byteorder', 'classDictInit', 'classLoader', 'cleanup', 'close', 'codecState', 'copyright', 'currentWorkingDir', 'defaultencoding', 'displayhook', 'doInitialize', 'dont_write_bytecode', 'exc_clear', 'exc_info', 'excepthook', 'exec_prefix', 'executable', 'exit', 'filesystemencoding', 'flags', 'float_info', 'float_repr_style', 'getBaseProperties', 'getBuiltin', 'getBuiltins', 'getClassLoader', 'getCodecState', 'getCurrentWorkingDir', 'getDefaultBuiltins', 'getFile', 'getImportLock', 'getPath', 'getPathLazy', 'getPlatform', 'getSyspathJavaLoader', 'getWarnoptions', 'getdefaultencoding', 'getfilesystemencoding', 'getprofile', 'getrecursionlimit', 'gettrace', 'hexversion', 'importLock', 'initialize', 'isPackageCacheEnabled', 'last_traceback', 'last_type', 'last_value', 'long_info', 'maxint', 'maxsize', 'maxunicode', 'meta_path', 'minint', 'modules', 'modules_reloading', 'packageManager', 'path', 'path_hooks', 'path_importer_cache', 'platform', 'prefix', 'ps1', 'ps2', 'py3kwarning', 'recursionlimit', 'refersDirectlyTo', 'registerCloser', 'registry', 'setBuiltins', 'setClassLoader', 'setCurrentWorkingDir', 'setPlatform', 'setWarnoptions', 'setprofile', 'setrecursionlimit', 'settrace', 'shadow', 'stderr', 'stdin', 'stdout', 'subversion', 'syspathJavaLoader', 'toString', 'traverse', 'unregisterCloser', 'version', 'version_info', 'warnoptions']
$ jython2.7 -c 'import sys; print(sys.ps1)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: '<reflected field public org.python.core.PyObject o' object has no attribute 'ps1'
$ jython2.7 -c 'import sys; print(sys.ps2)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: '<reflected field public org.python.core.PyObject o' object has no attribute 'ps2'


In CPython:

$ python2.7 -c 'import sys; print(dir(sys))'
['__displayhook__', '__doc__', '__excepthook__', '__name__', '__package__', '__stderr__', '__stdin__', '__stdout__', '_clear_type_cache', '_current_frames', '_getframe', '_mercurial', 'api_version', 'argv', 'builtin_module_names', 'byteorder', 'call_tracing', 'callstats', 'copyright', 'displayhook', 'dont_write_bytecode', 'exc_clear', 'exc_info', 'exc_type', 'excepthook', 'exec_prefix', 'executable', 'exit', 'flags', 'float_info', 'float_repr_style', 'getcheckinterval', 'getdefaultencoding', 'getdlopenflags', 'getfilesystemencoding', 'getprofile', 'getrecursionlimit', 'getrefcount', 'getsizeof', 'gettrace', 'hexversion', 'long_info', 'maxint', 'maxsize', 'maxunicode', 'meta_path', 'modules', 'path', 'path_hooks', 'path_importer_cache', 'platform', 'prefix', 'py3kwarning', 'setcheckinterval', 'setdlopenflags', 'setprofile', 'setrecursionlimit', 'settrace', 'stderr', 'stdin', 'stdout', 'subversion', 'version', 'version_info', 'warnoptions']
$ python2.7 -c 'import sys; print(sys.ps1)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: 'module' object has no attribute 'ps1'
$ python2.7 -c 'import sys; print(sys.ps2)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: 'module' object has no attribute 'ps2'

dir(sys) should respect PyAttributeDeleted. Most likely this should be done by overriding __dir__

Problem may also be in jython.java where we explicitly set them to empty strings instead of PyAttributeDeleted and that would fix it if fastGetDict or __rawdir__ already respect PyAttributeDeleted.

Easy?

I took some time on this one. Through my debug sessions, in jython.java, the vaiables ps1 and ps2 are assigned the JavaType 'attributedeleted', So far, so good. 

As the following  shows, both of these variables end up as type('str)...
===========
Jython 2.7.1 (, Sep 30 2017, 13:31:09) 
[OpenJDK 64-Bit Server VM (Oracle Corporation)] on java1.8.0_162
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> a = sys.ps1
>>> b = sys.ps2
>>> print a, b
>>>  ... 
>>> print type(a), type(b)
<type 'str'> <type 'str'>
>>> # Can you change the prompt?
>>> sys.ps1 = '===>'
===>
===># Yep
, But in this session only....
===>
======

When debugging, this code runs:

// Only defined if interactive, see https://docs.python.org/2/library/sys.html#sys.ps1
    public PyObject ps1 = PyAttributeDeleted.INSTANCE;
    public PyObject ps2 = PyAttributeDeleted.INSTANCE;

So, these two are set as type  PyAttribute Deleted at first, but they change to PyString, and could that be the reason they show up when calling the dir(sys) function?

I do see that  sys and PySystemState both return the same lis.

Still thinking....

Sme suggestions:

1. Change where in the process to set the PyAttributeDeleted type. as it may appear that it's set too early.
2. Disconnect the interactive prompt from PySystemState and make it its own type.

I say this because in InteractiveConsole.java, the systemstate.ps1 is saved as old_ps1, which retains the 'attributedeleted' type. Then .ps1 is assigned as a new String. Is it necessary to have a graphical prompt be part of PySystemState? 

Just asking because I'm still learning. :)

I just answered my own question:

Is it possible that ps1 (and ps2) are changed to a string type, losing its attributedeleted type, import sys is run (collecting all the attributes), then changed back to attributedeleted, therefore makingit unavailable when attempts are made to access the attribute that isn't there...

After I posted, I read the rest of InteractiveConsole.java and saw systemstate.ps1 = old_ps1. This should have made it an attributedeleted type again.

Time to debug....

Last note for now:

Whether i call jython with the -c or -m swithc, both ps1 and ps2 show up in the dir(sys) list.

In Python, it does not. it only appears, appropiately, in the interactive session. I'm debugging Python as well to see what goes on.

If setting them to PyAttributeDeleted makes them disappear from the list, then the place where we set them to "" is the problem. It looks right, but it's still an object in the dictionary.

The dir() function is defined in __builtins__.java . If you set a breakpoint there you could follow the action. At some point there is a map and dir() is enumerating its keys. Does that enumeration contain a skip when the value is PyAttributeDeleted? It probably should. Either that or we have to delete the prompts from the dictionary instead of setting them to "" or the PyAttributeDeleted singleton.

From what I can tell, when systemstate.ps1 is set to Py.EmptyString in jython.java, the type is still 'attributedeleted', so it should be okay.

i will follow the dir() in __builtins__.java to see if there's a skip for PyAttributeDeleted. If .ps1/ps2 is changed to a String prior to dir(),then whether the enumeration checks for it is moot. 

ps1/ps2 are the only objects set to PyAttributeDeleted at this point?

@Patrick: If you search the code base for PyAttributeDeleted you'll find it used in a few other places, but always in connection with PySystemState.

(Eclipse does not do this search well with our code base so I usually have another editor open. JEdit has a very effective search-in-directory function (globbing, regex, ...) that I use all the time for browsing both our code and CPython's.)

I'm not totally sure what the idea is of PyAttributeDeleted: I think it is connected with the fact that PySystemState is not a proper module, and so the actual Java variable has to be set to "I'm not here". There are a lot of // xxx: comments nearby suggesting whoever added this was not totally happy with it, but they haven't left us much else to go on. Sometimes code archeology (Mercurial file history) will give you a clue to who added it and what they were trying to do. And then sometimes one thinks of a better answer.

But this is difficult. My "easy" tag was based on the expectation we could go with the flow: use PyAttributeDeleted instead of the "" and ensure dir() respects it, maybe universally, maybe just for PySystemState (in __rawdir__ I think). If something else breaks as a result, we can reconsider.

When debugging, I see that the sys object, when passed to dir(), shows ps1 and ps2 as PyAttrubuteDeleted, so the way it is in jython.java is fine and is an empty string.

When dir() passes it to the Dictionary, it creates a stringmap, so by the time it gets to PylList, those attributes are now strings. I'm getting there and should have a solution soon.

I found the problem in dir(). There is no enumeration to check the types and remove the key . I'm not a Java guru (I'm having to do some refresher reading, so I'm attempting a solution that will call _setattr_ to remove the key, if it matches 'attribute deleted' && !interactive. Exactly where to put it is the hard part for me. I'm still working on it. Is it acceptable to create a new function, like public void respectAttributeDeleted()?

You don't need to condition this on "interactive" here, I would say. The problem (I think) is twofold, or one-and-a-half-fold, anyway.

In jython.java we are assigning an empty string when we should probably be assigning PyAttributeDeleted.INSTANCE to ps1 and ps2. Perhaps at that same place we should be removing them from __dict__. Take a look at PySystemState.__delattr__.

If that doesn't do it, then it's not so easy, and I think the focus is PySystemState.__rawdir__ . If you set a breakpoint there and watch it work you'll learn a lot about how hard Jython works under the covers to implement duck-typing. However, rather than use the generic dictionary update() method, in __rawdir__, maybe we need to use a loop where we step through __dict__ entries and don't add keys with a PyAttributeDeleted.INSTANCE value to accum. You have to remember that __dict__ is potentially *any* PyObject, so you can't assume it is the type you see in the debugger. You'll need something like PyObject.asIterable() and isMappingType().

Thanks for your thought provoking message.

Let me see if I can be clearer. Here is the code from jython.java:
=========
if (!opts.interactive) {
            // Not (really) interactive, so do not use console prompts
            systemState.ps1 = systemState.ps2 = Py.EmptyString;
        }
===========
I cannot change the above to PyAttributeDeleted.INSTANCE, because it's already set in PySystemState.java: as below:
============
public PyObject ps1 = PyAttributeDeleted.INSTANCE;
public PyObject ps2 = PyAttributeDeleted.INSTANCE;
===========

To be honest, (and I've tested this) the snippet in jython.java is redundant. Why? Because if it's not interactive, it's not going to show the prompts anyway, so why have code worrying about it when it's already set in SystemState?

I have attached screenshots that show ps1 and ps2 as JavaTypes 'attrbute deleted' and then becoming strings. 

It is a matter of doing a loop to remove these two keys from the __dict__, that challenge is finding the right spot.

In an earlier post, I questioned even having the interactive prompts connected to systemstate.ps1/ps2. I see those two variables used in other code in PySystemState.java and it's confusing....

I believe I have found the source of this issue. The __setattr__ and __delattr__ functions. From my debug sessions, both of thse functions appear to run only once, but I'll focuus on __delattr__ for now. 

In this bit of code, it is passed a name, then it sets that name to PyAttributeDeleted.INSTANCE, and then removes that key from the __dict__. 

Just an update..

hi Jeff,

I had hit a brick wall. I found the code for a overloaded __delattr__ method that specifically dealt with the PyAttributeDeleted said it needed to be fixed, so I had attempted to look at the invoker to see what was happening and try something, but basically, the loop only ran once and didn't delete all the deleted attributes. 

Also, through many debug sessions, I realized that the system seemed to not recognize it was in non-interactive mode (basing on the tty), so that was making me realize Ineeded to bus up on my Java, and now with Java 10 and 11 appearing, I sort of put Jython on the back burner. 

If you believe you've fixed it, great! I'll look for another issue to play with.

But, I am still interested and as i discover how IntelliJ deals with it, I amend my "how-to" blog post accordingly. Which Java version are you refactoring to?

I apologize for how long its been. A few crises have come along to make life more challenging, but still I plow on.

Thank you!

Don't worry: all contributions gratefully received, at whatever pace. The recent changes under #2686 don't fix this, they just removed one confusion factor. (My refactoring was based on main.c in Python 2.7.15, BTW.)

PS jython-jvm9> dist\bin\jython -c "import sys; print 'ps1' in dir(sys)"
True
PS jython-jvm9> python -c "import sys; print 'ps1' in dir(sys)"
False

In #2686 I removed a complicating factor where we were setting ps1 and ps2 in the Jython main to empty strings. (This had been an incorrect fix for a different bug.)

Here's how I see this one. When a module is defined in Java, the attributes Python sees will be the instance or static members that appear in the Java class, both data and methods. There is magic in PyType that fills the module __dict__ with entries obtained by inspection, where the key is the name of the Java attribute, and the value is a wrapper (PyReflectedField) that can get and sometimes set the Java member. Here "set" means write the Java member, not (as might be the case in Python) change the entry in the dictionary, so a reference from Java, that knows nothing about the __dict__, sees the changed value. It is hardly ever necessary to permit deletion, but here it is.

So how to represent that? Setting the value to null is a natural choice, but the normal wrapper maps null to None. So we invented the special value AttributeDeleted.INSTANCE (https://hg.python.org/jython/rev/7c709bc88e7f) and use it here for ps* (https://hg.python.org/jython/rev/23c3effa5d4f), but it means we have to give it special treatment, which we've neglected to do in dir().

A change to __rawdir__  to pretend that we didn't see any attribute with value AttributeDeleted.INSTANCE would fix this issue, I think. This test should have a companion that verifies that for dir(): https://hg.python.org/jython/file/23c3effa5d4f/Lib/test/test_sys_jy.py#l262

A side question: In SystemState.java, the following code, does it remove the deleted attribute?
====
public void __delattr__(String name) {
        checkMustExist(name);
        PyObject ret = getType().lookup(name); // xxx fix fix fix
        if (ret != null) {
            ret._doset(this, PyAttributeDeleted.INSTANCE);
        }
=============
This metod is executed once, and i thought it was part of a loop that should go through the rest of the __dict__ to remove other attrubytes tgat were set as deleted.

Also, I've noticed that using the -c option appears to not tell the program it is NOT interactive (meaning the interactive is set to true). So, if it believes it's interactive, then the ps* would display, right? 

I will examine dir() and __rawdir__ more closely. Thanks for being patient.

As explained above, you don't want to delete it properly from the dictionary, just have it ignored.

This code looks up the name, and if there's an entry in the dictionary, rather than remove the entry, it sets the member that to the special value AttributeDeleted.INSTANCE. Note that the __dict__ entry is a descriptor that manipulates the field Java sees, so we have to call its __set__ method, to set the new value, not just write in the distionary.

I don't understand the "fix fix fix" comment. (It has been there a long time.) AFAIK this code is ok. The problem is that dir() does not ignore the entry when AttributeDeleted.INSTANCE is stored there.

There was confusion in the old main() about two things that "interactive" means: 1. stdin/out is thought to be a terminal; and 2. Jython should start an interactive session on stdin/out to receive commands to execute them. Only the second sense is relevant to whether sys.ps1 and ps2 exist.

Code given with -c does not execute in an interactive session, so ps1, ps2 don't exist. If you give the -i option, they still don't exist while -c's code executes, but they will during the interactive session that follows.

On second thoughts ... my understanding of the interaction with __dict__  was incorrect.

I should have added that __delattr__ is the implementation of "del" at the REPL. But I was wrong in that we *do* delete ps* from __dict__, as well as set the field member to PyAttributeDeleted.INSTANCE during a call. However, we start in an inconsistent state with ps1, ps2 in the __dict__ but set to PyAttributeDeleted.

In a regular interactive REPL, observe this:

>>> import sys
>>> 'ps2' in dir(sys)
True
>>> del sys.ps2
>>> 'ps2' in dir(sys)
False
>>> sys.ps2 = '::: '
>>> 'ps2' in dir(sys)
False    # This is wrong.

I've set breakpoints in PySystemState.__setattr__ and __delattr__, so I can observe how it goes internally. You see that after mid-sequence above, ps2==PyAttributeDeleted and is absent from the dictionary (as reported by dir()). However, putting a value back in ps2 doesn't put it back in __dict__, which is wrong.

We should either 1. always remove/add to __dict__ on del/set, and always use del/set at the Java level or 2. del/set only the Java member (not __dict__) and then respect PyAttributeDeleted everywhere they crop up. It seems like the idea of PyAttributeDeleted is to do the latter, but __setattr__ is incorrect, and __rawdir__.