Title: sys.ps1 and sys.ps2 mentioned in dir(sys) in non-interactive mode
Type: Severity: normal
Components: Core Versions: Jython 2.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, jeff.allen, psykiatris, zyasoft
Priority: normal Keywords: easy

Created on 2015-03-30.19:54:46 by Arfrever, last changed 2018-05-23.14:40:41 by psykiatris.

File name Uploaded Description Edit Remove psykiatris, 2018-05-15.15:29:13 Screenshots showing ps1/ps2 in debug sessions
msg9738 (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) Date: 2015-03-30.19:54:45
Since sys.ps1 and sys.ps2 do not exist in non-interactive mode but they are still mentioned in dir(sys):

$ jython2.7 -c 'import sys; print(dir(sys))'
['JYTHON_DEV_JAR', 'JYTHON_JAR', 'PYTHON_CACHEDIR', 'PYTHON_CACHEDIR_SKIP', 'PYTHON_CONSOLE_ENCODING', 'PYTHON_IO_ENCODING', 'PYTHON_IO_ERRORS', '__delattr__', '__dict__', '__displayhook__', '__excepthook__', '__findattr_ex__', '__name__', '__new__', '__rawdir__', '__setattr__', '__stderr__', '__stdin__', '__stdout__', '_getframe', '_jy_console', '_mercurial', '_systemRestart', 'add_classdir', 'add_extdir', 'add_package', 'argv', 'builtin_module_names', 'builtins', 'byteorder', 'classDictInit', 'classLoader', 'cleanup', 'close', 'codecState', 'copyright', 'currentWorkingDir', 'defaultencoding', 'displayhook', 'doInitialize', 'dont_write_bytecode', 'exc_clear', 'exc_info', 'excepthook', 'exec_prefix', 'executable', 'exit', 'filesystemencoding', 'flags', 'float_info', 'float_repr_style', 'getBaseProperties', 'getBuiltin', 'getBuiltins', 'getClassLoader', 'getCodecState', 'getCurrentWorkingDir', 'getDefaultBuiltins', 'getFile', 'getImportLock', 'getPath', 'getPathLazy', 'getPlatform', 'getSyspathJavaLoader', 'getWarnoptions', 'getdefaultencoding', 'getfilesystemencoding', 'getprofile', 'getrecursionlimit', 'gettrace', 'hexversion', 'importLock', 'initialize', 'isPackageCacheEnabled', 'last_traceback', 'last_type', 'last_value', 'long_info', 'maxint', 'maxsize', 'maxunicode', 'meta_path', 'minint', 'modules', 'modules_reloading', 'packageManager', 'path', 'path_hooks', 'path_importer_cache', 'platform', 'prefix', 'ps1', 'ps2', 'py3kwarning', 'recursionlimit', 'refersDirectlyTo', 'registerCloser', 'registry', 'setBuiltins', 'setClassLoader', 'setCurrentWorkingDir', 'setPlatform', 'setWarnoptions', 'setprofile', 'setrecursionlimit', 'settrace', 'shadow', 'stderr', 'stdin', 'stdout', 'subversion', 'syspathJavaLoader', 'toString', 'traverse', 'unregisterCloser', 'version', 'version_info', 'warnoptions']
$ jython2.7 -c 'import sys; print(sys.ps1)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: '<reflected field public org.python.core.PyObject o' object has no attribute 'ps1'
$ jython2.7 -c 'import sys; print(sys.ps2)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: '<reflected field public org.python.core.PyObject o' object has no attribute 'ps2'

In CPython:

$ python2.7 -c 'import sys; print(dir(sys))'
['__displayhook__', '__doc__', '__excepthook__', '__name__', '__package__', '__stderr__', '__stdin__', '__stdout__', '_clear_type_cache', '_current_frames', '_getframe', '_mercurial', 'api_version', 'argv', 'builtin_module_names', 'byteorder', 'call_tracing', 'callstats', 'copyright', 'displayhook', 'dont_write_bytecode', 'exc_clear', 'exc_info', 'exc_type', 'excepthook', 'exec_prefix', 'executable', 'exit', 'flags', 'float_info', 'float_repr_style', 'getcheckinterval', 'getdefaultencoding', 'getdlopenflags', 'getfilesystemencoding', 'getprofile', 'getrecursionlimit', 'getrefcount', 'getsizeof', 'gettrace', 'hexversion', 'long_info', 'maxint', 'maxsize', 'maxunicode', 'meta_path', 'modules', 'path', 'path_hooks', 'path_importer_cache', 'platform', 'prefix', 'py3kwarning', 'setcheckinterval', 'setdlopenflags', 'setprofile', 'setrecursionlimit', 'settrace', 'stderr', 'stdin', 'stdout', 'subversion', 'version', 'version_info', 'warnoptions']
$ python2.7 -c 'import sys; print(sys.ps1)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: 'module' object has no attribute 'ps1'
$ python2.7 -c 'import sys; print(sys.ps2)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: 'module' object has no attribute 'ps2'
msg9739 (view) Author: Jim Baker (zyasoft) Date: 2015-03-30.20:54:55
dir(sys) should respect PyAttributeDeleted. Most likely this should be done by overriding __dir__
msg11747 (view) Author: Jeff Allen (jeff.allen) Date: 2018-03-06.08:25:49
Problem may also be in where we explicitly set them to empty strings instead of PyAttributeDeleted and that would fix it if fastGetDict or __rawdir__ already respect PyAttributeDeleted.

msg11955 (view) Author: Patrick Palczewski (psykiatris) Date: 2018-05-07.17:05:53
I took some time on this one. Through my debug sessions, in, the vaiables ps1 and ps2 are assigned the JavaType 'attributedeleted', So far, so good. 

As the following  shows, both of these variables end up as type('str)...
Jython 2.7.1 (, Sep 30 2017, 13:31:09) 
[OpenJDK 64-Bit Server VM (Oracle Corporation)] on java1.8.0_162
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> a = sys.ps1
>>> b = sys.ps2
>>> print a, b
>>>  ... 
>>> print type(a), type(b)
<type 'str'> <type 'str'>
>>> # Can you change the prompt?
>>> sys.ps1 = '===>'
===># Yep
, But in this session only....

When debugging, this code runs:

// Only defined if interactive, see
    public PyObject ps1 = PyAttributeDeleted.INSTANCE;
    public PyObject ps2 = PyAttributeDeleted.INSTANCE;

So, these two are set as type  PyAttribute Deleted at first, but they change to PyString, and could that be the reason they show up when calling the dir(sys) function?

I do see that  sys and PySystemState both return the same lis.

Still thinking....
msg11956 (view) Author: Patrick Palczewski (psykiatris) Date: 2018-05-07.17:33:31
Sme suggestions:

1. Change where in the process to set the PyAttributeDeleted type. as it may appear that it's set too early.
2. Disconnect the interactive prompt from PySystemState and make it its own type.

I say this because in, the systemstate.ps1 is saved as old_ps1, which retains the 'attributedeleted' type. Then .ps1 is assigned as a new String. Is it necessary to have a graphical prompt be part of PySystemState? 

Just asking because I'm still learning. :)
msg11957 (view) Author: Patrick Palczewski (psykiatris) Date: 2018-05-07.17:48:37
I just answered my own question:

Is it possible that ps1 (and ps2) are changed to a string type, losing its attributedeleted type, import sys is run (collecting all the attributes), then changed back to attributedeleted, therefore makingit unavailable when attempts are made to access the attribute that isn't there...

After I posted, I read the rest of and saw systemstate.ps1 = old_ps1. This should have made it an attributedeleted type again.

Time to debug....
msg11959 (view) Author: Patrick Palczewski (psykiatris) Date: 2018-05-07.19:52:55
Last note for now:

Whether i call jython with the -c or -m swithc, both ps1 and ps2 show up in the dir(sys) list.

In Python, it does not. it only appears, appropiately, in the interactive session. I'm debugging Python as well to see what goes on.
msg11960 (view) Author: Jeff Allen (jeff.allen) Date: 2018-05-07.22:01:13
If setting them to PyAttributeDeleted makes them disappear from the list, then the place where we set them to "" is the problem. It looks right, but it's still an object in the dictionary.

The dir() function is defined in . If you set a breakpoint there you could follow the action. At some point there is a map and dir() is enumerating its keys. Does that enumeration contain a skip when the value is PyAttributeDeleted? It probably should. Either that or we have to delete the prompts from the dictionary instead of setting them to "" or the PyAttributeDeleted singleton.
msg11961 (view) Author: Patrick Palczewski (psykiatris) Date: 2018-05-07.22:15:38
From what I can tell, when systemstate.ps1 is set to Py.EmptyString in, the type is still 'attributedeleted', so it should be okay.

i will follow the dir() in to see if there's a skip for PyAttributeDeleted. If .ps1/ps2 is changed to a String prior to dir(),then whether the enumeration checks for it is moot. 

ps1/ps2 are the only objects set to PyAttributeDeleted at this point?
msg11966 (view) Author: Jeff Allen (jeff.allen) Date: 2018-05-09.07:41:48
@Patrick: If you search the code base for PyAttributeDeleted you'll find it used in a few other places, but always in connection with PySystemState.

(Eclipse does not do this search well with our code base so I usually have another editor open. JEdit has a very effective search-in-directory function (globbing, regex, ...) that I use all the time for browsing both our code and CPython's.)

I'm not totally sure what the idea is of PyAttributeDeleted: I think it is connected with the fact that PySystemState is not a proper module, and so the actual Java variable has to be set to "I'm not here". There are a lot of // xxx: comments nearby suggesting whoever added this was not totally happy with it, but they haven't left us much else to go on. Sometimes code archeology (Mercurial file history) will give you a clue to who added it and what they were trying to do. And then sometimes one thinks of a better answer.

But this is difficult. My "easy" tag was based on the expectation we could go with the flow: use PyAttributeDeleted instead of the "" and ensure dir() respects it, maybe universally, maybe just for PySystemState (in __rawdir__ I think). If something else breaks as a result, we can reconsider.
msg11969 (view) Author: Patrick Palczewski (psykiatris) Date: 2018-05-11.02:43:45
When debugging, I see that the sys object, when passed to dir(), shows ps1 and ps2 as PyAttrubuteDeleted, so the way it is in is fine and is an empty string.

When dir() passes it to the Dictionary, it creates a stringmap, so by the time it gets to PylList, those attributes are now strings. I'm getting there and should have a solution soon.
msg11978 (view) Author: Patrick Palczewski (psykiatris) Date: 2018-05-14.13:24:45
I found the problem in dir(). There is no enumeration to check the types and remove the key . I'm not a Java guru (I'm having to do some refresher reading, so I'm attempting a solution that will call _setattr_ to remove the key, if it matches 'attribute deleted' && !interactive. Exactly where to put it is the hard part for me. I'm still working on it. Is it acceptable to create a new function, like public void respectAttributeDeleted()?
msg11980 (view) Author: Jeff Allen (jeff.allen) Date: 2018-05-15.08:07:22
You don't need to condition this on "interactive" here, I would say. The problem (I think) is twofold, or one-and-a-half-fold, anyway.

In we are assigning an empty string when we should probably be assigning PyAttributeDeleted.INSTANCE to ps1 and ps2. Perhaps at that same place we should be removing them from __dict__. Take a look at PySystemState.__delattr__.

If that doesn't do it, then it's not so easy, and I think the focus is PySystemState.__rawdir__ . If you set a breakpoint there and watch it work you'll learn a lot about how hard Jython works under the covers to implement duck-typing. However, rather than use the generic dictionary update() method, in __rawdir__, maybe we need to use a loop where we step through __dict__ entries and don't add keys with a PyAttributeDeleted.INSTANCE value to accum. You have to remember that __dict__ is potentially *any* PyObject, so you can't assume it is the type you see in the debugger. You'll need something like PyObject.asIterable() and isMappingType().
msg11981 (view) Author: Patrick Palczewski (psykiatris) Date: 2018-05-15.15:29:13
Thanks for your thought provoking message.

Let me see if I can be clearer. Here is the code from
if (!opts.interactive) {
            // Not (really) interactive, so do not use console prompts
            systemState.ps1 = systemState.ps2 = Py.EmptyString;
I cannot change the above to PyAttributeDeleted.INSTANCE, because it's already set in as below:
public PyObject ps1 = PyAttributeDeleted.INSTANCE;
public PyObject ps2 = PyAttributeDeleted.INSTANCE;

To be honest, (and I've tested this) the snippet in is redundant. Why? Because if it's not interactive, it's not going to show the prompts anyway, so why have code worrying about it when it's already set in SystemState?

I have attached screenshots that show ps1 and ps2 as JavaTypes 'attrbute deleted' and then becoming strings. 

It is a matter of doing a loop to remove these two keys from the __dict__, that challenge is finding the right spot.

In an earlier post, I questioned even having the interactive prompts connected to systemstate.ps1/ps2. I see those two variables used in other code in and it's confusing....
msg12001 (view) Author: Patrick Palczewski (psykiatris) Date: 2018-05-23.14:40:41
I believe I have found the source of this issue. The __setattr__ and __delattr__ functions. From my debug sessions, both of thse functions appear to run only once, but I'll focuus on __delattr__ for now. 

In this bit of code, it is passed a name, then it sets that name to PyAttributeDeleted.INSTANCE, and then removes that key from the __dict__. 

Just an update..
Date User Action Args
2018-05-23 14:40:41psykiatrissetmessages: + msg12001
2018-05-15 15:29:14psykiatrissetfiles: +
messages: + msg11981
2018-05-15 08:07:23jeff.allensetmessages: + msg11980
2018-05-14 13:24:46psykiatrissetmessages: + msg11978
2018-05-11 02:43:46psykiatrissetmessages: + msg11969
2018-05-09 07:41:49jeff.allensetmessages: + msg11966
2018-05-07 22:15:39psykiatrissetmessages: + msg11961
2018-05-07 22:01:13jeff.allensetmessages: + msg11960
2018-05-07 19:52:55psykiatrissetmessages: + msg11959
2018-05-07 17:48:37psykiatrissetmessages: + msg11957
2018-05-07 17:33:31psykiatrissetmessages: + msg11956
2018-05-07 17:05:53psykiatrissetnosy: + psykiatris
messages: + msg11955
2018-04-10 19:53:46jeff.allensetkeywords: + easy
2018-03-06 08:25:49jeff.allensetnosy: + jeff.allen
messages: + msg11747
milestone: Jython 2.7.2 ->
2015-12-29 23:54:28zyasoftsetmilestone: Jython 2.7.1 -> Jython 2.7.2
2015-04-20 21:05:51zyasoftsetmilestone: Jython 2.7.1
2015-04-02 16:05:35zyasoftsetpriority: normal
2015-03-30 20:54:55zyasoftsetmessages: + msg9739
2015-03-30 19:54:46Arfrevercreate