Issue2307

On Windows at least, there is come mis-handling of the normalisation of file paths. This shows as a test failure in test_chdir, but seems not to be related to actually changing directory. We are perhaps not handling paths in the same way as CPython.

Typical test_chdir output:

======================================================================
FAIL: test_chdir (__main__.ChdirTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\Jeff\Documents\Eclipse\jython-trunk\dist\Lib\test\test_chdir.py", line 155, in test_chdir
    self.assertEqual(os.path.realpath(self.dir1), cwd)
AssertionError: 'c:\\users\\jeff\\appdata\\local\\temp\\tmpxsjrue' != 'C:\\Users\\Jeff\\AppData\\Local\\Temp\\tmpxsjrue'

======================================================================
FAIL: test_relative_chdir (__main__.ChdirTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\Jeff\Documents\Eclipse\jython-trunk\dist\Lib\test\test_chdir.py", line 162, in test_relative_chdir
    self.assertEqual(os.getcwd(), os.path.realpath(self.dir1))
AssertionError: 'C:\\Users\\Jeff\\AppData\\Local\\Temp\\tmp1laq83' != 'c:\\users\\jeff\\appdata\\local\\temp\\tmp1laq83'

======================================================================
FAIL: test_realpath (__main__.FilesTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\Jeff\Documents\Eclipse\jython-trunk\dist\Lib\test\test_chdir.py", line 585, in test_realpath
    self.assertEqual(os.path.realpath(self.filename1),
AssertionError: 'c:\\users\\jeff\\appdata\\local\\temp\\tmpgxmxi1\\tmputfwjw' != 'C:\\Users\\Jeff\\AppData\\Local\\Temp\\tmpgxmxi1\\tmputfwjw'

======================================================================
FAIL: test_windows_chdir_dos_path (__main__.WindowsChdirTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\Jeff\Documents\Eclipse\jython-trunk\dist\Lib\test\test_chdir.py", line 194, in test_windows_chdir_dos_path
    self.assertEqual(os.getcwd(), os.path.realpath(dos_name))
AssertionError: 'C:\\Users\\Jeff\\AppData\\Local\\Temp\\tmpyok90j\\Program Files' != 'c:\\users\\jeff\\appdata\\local\\temp\\tmpyok90j\\progra~1'

======================================================================
FAIL: test_windows_chdir_slash_isabs (__main__.WindowsChdirTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\Jeff\Documents\Eclipse\jython-trunk\dist\Lib\test\test_chdir.py", line 233, in test_windows_chdir_slash_isabs
    self.assertEqual(os.path.normcase(os.getcwd()),
AssertionError: 'c:\\users\\jeff\\documents\\eclipse\\jython-trunk' != 'c:\\'

======================================================================
FAIL: test_windows_getcwd_ensures_drive_letter (__main__.WindowsChdirTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\Jeff\Documents\Eclipse\jython-trunk\dist\Lib\test\test_chdir.py", line 204, in test_windows_getcwd_ensures_drive_letter
    self.assertEqual(os.path.normcase(os.getcwd()),
AssertionError: 'c:\\users\\jeff\\documents\\eclipse\\jython-trunk' != 'c:\\'

test_chdir is a Jython-specific test. It reveals a bug in the tempfile module that CPython noticed (http://bugs.python.org/issue14255 refers) but only fixed in Python 3. I'll copy this for Jython. It suppresses three of the chdir failures.

I'll look into the other failures as distinct change sets. Jython's tempfile has fallen a little behind CPython's. I'll take in that and the problematic use of os.name too, but they're not the source of these failures AFAIK.

Fixed the failures related to tempfile, and updated, here:
https://hg.python.org/jython/rev/e6ca8b2cf467
Others are related to use of java.nio.Path. I will pull the most recent work before continuing. (My local repo is a bit old, thanks to others high productivity!)

This (https://hg.python.org/jython/rev/60972a7a26c5) passes on my C: drive, but on D: fails my especially cruel test:
======================================================================
FAIL: test_windows_getcwd_ensures_drive_letter (__main__.WindowsChdirTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\hg\jython-int\dist\Lib\test\test_chdir.py", line 227, in test_windows_getcwd_ensures_drive_letter
    self.assertEqual(os.path.normcase(os.getcwd()),
AssertionError: 'd:' != 'd:\\hg\\jython-int'

----------------------------------------------------------------------

Windows has this awkward feature that every drive has its own current working directory, and if you just call os.chdir('d:') you ought to land in the one previously defined (with os.chdir or in the shell) for the d-drive. Under #2117 we got this as close to CPython as possible using java.io.File (see https://hg.python.org/jython/rev/f553f6a07fb5#l3.74). On the plus side, I think it would be easier to get right with the help of Path.getRoot() and FileSystem.getRootDirectories().

Now seeing failures on OS X 10.10.2 for test_chdir:

======================================================================
FAIL: test_relative_chdir (test.test_chdir.ChdirTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jbaker/jythondev/jython27/dist/Lib/test/test_chdir.py", line 162, in test_relative_chdir
    self.assertEqual(os.getcwd(), os.path.realpath(self.dir1))
AssertionError: '/var/folders/kk/yrwpx1_x7ll3mn4mvqym2lv40000gn/T/tmpyMAziM' != '/private/var/folders/kk/yrwpx1_x7ll3mn4mvqym2lv40000gn/T/tmpyMAziM'

======================================================================
FAIL: test_realpath (test.test_chdir.FilesTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jbaker/jythondev/jython27/dist/Lib/test/test_chdir.py", line 585, in test_realpath
    self.assertEqual(os.path.realpath(self.filename1),
AssertionError: '/private/var/folders/kk/yrwpx1_x7ll3mn4mvqym2lv40000gn/T/tmphpwxZN/tmpXshvQq' != '/var/folders/kk/yrwpx1_x7ll3mn4mvqym2lv40000gn/T/tmphpwxZN/tmpXshvQq'


Also impacting test_posixpath

======================================================================
FAIL: test_realpath_resolve_parents (test.test_posixpath.PosixPathTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jbaker/jythondev/jython27/dist/Lib/test/test_posixpath.py", line 332, in test_realpath_resolve_parents
    self.assertEqual(realpath("a"), ABSTFN + "/y/a")
AssertionError: '/Users/jbaker/jythondev/jython27/$test_1_tmp/k/a' != '/Users/jbaker/jythondev/jython27/$test_1_tmp/y/a'

Sorry 'bout that, I've taken a deeper look.

I conclude that we cannot pass both the tests now failing on OS X, and the tests that were failing on Windows, as long as we use Path.toRealPath() uniformly. The issue is with the treatment of alternate (DOS 8.3) names on Windows. On Windows, CPython (2.8 and 3.4) does this:
>>> os.path.realpath('junk\PROGRA~1')
'C:\\Users\\Jeff\\Documents\\Jython\\2.7rc2\\junk\\PROGRA~1'
>>> os.chdir('junk\PROGRA~1')
>>> os.getcwd()
'C:\\Users\\Jeff\\Documents\\Jython\\2.7rc2\\junk\\PROGRA~1'

On Windows, java.nio.files does this:
>>> Paths.get('junk\PROGRA~1').toRealPath()
C:\Users\Jeff\Documents\Jython\2.7rc2\junk\Program Files
>>> Paths.get('junk\PROGRA~1').toAbsolutePath()
C:\Users\Jeff\Documents\Jython\2.7rc2\junk\PROGRA~1

So Java resolves PROGRA~1 to its full NT name, whereas CPython maintains the 8.3 name it was given.

On Linux I find that os.path.realpath resolves symbolic links (as required) but not hard links, which is also the behaviour of Java Path.toRealPath(). Is PROGRA~1 a sort of hard link, then?

It seems as if CPython os.chdir calls os.path.realpath (although I can't track that down). On Windows, os.path = ntpath and realpath is just abspath, which is not required resolve symbolic links. CPython ntpath.abspath calls nt._getfullpathname, but that doesn't resolve the 8.3 name to its NT name:
>>> nt._getfullpathname('junk\PROGRA~1')
'C:\\Users\\Jeff\\Documents\\Jython\\2.7rc2\\junk\\PROGRA~1'

Internally, CPython nt._getfullpathname involves the MS function GetFullPathName, which Java also uses in WindowsLinkSupport.getRealPath. While I'm still puzzled by the difference in result between ntpath and java.nio.file, it seems clear we have to make them agree on Windows as they do on Linux.

Jeff, makes perfect sense. We already have other logic that depends in the posix module on running on  Windows or not, so this is just more of the same.

I propose both an absolutePath and a realPath method, as close as possible to their os.path near namesakes' CPython behaviour, and with the same sensitivity to OS. chdir will call realPath.

absolutePath is called many places in PosixModule. It used to behave like os.path.realpath (except as noted), but now it behaves like os.path.abspath, and the only breakage seems to involve chdir. I suspect abspath behaviour is all we need in these cases and realpath behaviour may have been incorrect in some (unlink?). I'll leave them calling absolutePath(), unless tests prove me wrong.

Probably tomorrow's work.

Jeff, I would like to get a RC3 out on Monday with these changes, along with #2311 and #2319, or to temporarily roll them back if that's too ambitious. Anything else will go into 2.7.1.

(It's still possible that fixing #2311 sufficiently so we don't see a repeat of #2319 is also too ambitious, but that's not my sense of the scope.)

https://hg.python.org/jython/rev/3733bc1b2f38 fixes regression on Linux (msg9796) and https://hg.python.org/jython/rev/b14c97e4486c fixes the test_windows_getcwd_ensures_drive_letter failure (msg9793), without causing another regression on Linux.

Probably.

Hopefully! ;)

It's running fine on OS X 10.10 and Ubuntu 14.10, so I think we are now good. Thanks for your hard work on making it be compatible across platforms!