Message9274

So os.listdir has interesting semantics with respect to working with Unicode paths that I was not aware of until now (being US/ASCII centric I suppose ;)

I created the following directory layout:
unicode
└── 首页

(Note that at least on OSX, the tree and ls commands don't work from the command line. cd does. So I manually created the above tree diagram!)

Python 2.7
>>> os.listdir(u"unicode")
[u'\u9996\u9875']
>>> os.listdir("unicode")
['\xe9\xa6\x96\xe9\xa1\xb5']

Trying again with Jython 2.7 trunk:
>>> os.listdir(u"unicode")
[u'\u9996\u9875']
>>> os.listdir("unicode")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
	at org.python.core.PyString.<init>(PyString.java:60)
	at org.python.core.PyString.<init>(PyString.java:66)
	at org.python.core.PyString.createInstance(PyString.java:776)
	at org.python.modules.posix.PosixModule.listdir(PosixModule.java:499)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)

java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Cannot create PyString with non-byte value

Similar behavior is seen in say glob.glob("unicode/*") vs glob.glob(u"unicode/*"). os.stat can work with paths that are Unicode strings, but not as bytestrings:

os.stat('unicode/\xe9\xa6\x96\xe9\xa1\xb5')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 2] No such file or directory: '/Users/jbaker/test/unicode/\xe9\xa6\x96\xe9\xa1\xb5'

There is the outstanding and possibly related bug #2110 re updating our JNR Posix jar