Issue1568

Jython 2.5.1 (Release_2_5_1:6813, Sep 26 2009, 13:47:54)
[Java HotSpot(TM) Client VM (Sun Microsystems Inc.)] on java1.6.0_18
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.stdout.encoding
'Cp1252'

Python 2.6.4 (r264:75708, Oct 26 2009, 08:23:19) [MSC v.1500 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.stdout.encoding
'cp437'

Decoding console output using sys.stdout.encoding with Jython2.5 produces illegible characters.

Could you tell us which version of Windows you are using along with anything that might be unusual in your environment to help us reproduce this?

This is a long standing problem on the JVM. We use the JVM's file.encoding property for the console encoding. Unfortunately Windows uses a separate encoding for its console (usually the older cp437 codepage)

I believe Java 6's Console class now determines the correct encoding for stdout/err, but it keeps that information private. I suppose we could steal that information via reflection

In the meantime you can force the console encoding via the python.console.encoding property. You can edit your registry to force this or pass that argument on the command line

I have tested this on multiple machines with Windows XP SP3.

We have code that needs to work both on Jython and Python and on all common OSes. Currently we just work around this with an ugly if, and it seems to be working. 

Thanks for the replies.

Yes, Philip, you're right: we could steal that information via reflection. Here using Jython:
---
from java.lang import System

def get_encoding(console):
    """returns encoding of this console
    <!> untested if console is null (i.e. when not attached to a tty)
    <!> might fail if a security manager is set without "suppressAccessChecks"
    """
    encoding_method = console.getClass().getDeclaredMethod("encoding", None)
    encoding_method.setAccessible(True)
    return encoding_method.invoke(console, None)

def get_charset(console):
    """returns charset of this console
    <!> untested if console is null (i.e. when not attached to a tty)
    <!> might fail if a security manager is set without "suppressAccessChecks"
    """
    charset_field = console.getClass().getDeclaredField("cs")
    charset_field.setAccessible(True)
    return charset_field.get(console)

if __name__ == "__main__":
    print "java.io.Console's encoding:", get_encoding(System.console())
     # cp437 (string)

    print "java.io.Console's charset:", get_charset(System.console())
     # IBM437 (java.nio.charset.Charset object)
---
Regards,
Regis

Minimal implementation, working even if JVM is not connected to a TTY:
---
    from java.io import Console

    def get_console_encoding():
        encoding_method = Console.getDeclaredMethod("encoding", None)
        encoding_method.setAccessible(True)
        return encoding_method.invoke(None, None)
---

Console apps, where this is the problem, generally don't have security restrictions in place, so maybe we should try the reflection path to get the actual encoding and fail back to what Java is directly telling us.

Marking high, ideally we will have in 2.5.2

JavaOne BOF

A protoype on Windows shows that this only works using java 6:

C:\stuff>"c:\Program Files (x86)\Java\jdk1.5.0_22\bin\java" ConsoleTest
encoding: null

C:\stuff>java -version
java version "1.6.0_20"

C:\stuff>java ConsoleTest
encoding: cp437


Here is the source of ConsoleTest.java:

  private static String getConsoleEncoding() {
    String encoding = null;
    try {
      Class<?> consoleClass = Class.forName("java.io.Console");
      Method encodingMethod = consoleClass.getDeclaredMethod("encoding");
      encodingMethod.setAccessible(true);
      encoding = (String) encodingMethod.invoke(consoleClass);
    } catch (Exception e) {
      // ignore any exception
    }
    return encoding;
  }

  public static void main(String[] args) {
    System.out.println("encoding: " + getConsoleEncoding());
  }

a patch for PySystemState.java

fixed (on Java 6) with revision 7128