Issue1568

classification
Title: sys.stdout.encoding returns wrong value in Windows with Jython 2.5.1
Type: Severity: normal
Components: Versions:
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: otmarhumbel Nosy List: fwierzbicki, otmarhumbel, pekka.klarck, pjenvey, rdesgroppes, yanne, zyasoft
Priority: high Keywords:

Created on 2010-03-04.10:44:20 by yanne, last changed 2010-09-29.06:30:45 by otmarhumbel.

Files
File name Uploaded Description Edit Remove
1568patch.txt otmarhumbel, 2010-09-29.06:24:20 This patch fixes the issue, but only when running Jython on Java 6
Messages
msg5559 (view) Author: Janne Härkönen (yanne) Date: 2010-03-04.10:44:19
Jython 2.5.1 (Release_2_5_1:6813, Sep 26 2009, 13:47:54)
[Java HotSpot(TM) Client VM (Sun Microsystems Inc.)] on java1.6.0_18
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.stdout.encoding
'Cp1252'

Python 2.6.4 (r264:75708, Oct 26 2009, 08:23:19) [MSC v.1500 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.stdout.encoding
'cp437'

Decoding console output using sys.stdout.encoding with Jython2.5 produces illegible characters.
msg5560 (view) Author: Frank Wierzbicki (fwierzbicki) Date: 2010-03-04.20:14:59
Could you tell us which version of Windows you are using along with anything that might be unusual in your environment to help us reproduce this?
msg5631 (view) Author: Philip Jenvey (pjenvey) Date: 2010-04-04.19:01:41
This is a long standing problem on the JVM. We use the JVM's file.encoding property for the console encoding. Unfortunately Windows uses a separate encoding for its console (usually the older cp437 codepage)

I believe Java 6's Console class now determines the correct encoding for stdout/err, but it keeps that information private. I suppose we could steal that information via reflection

In the meantime you can force the console encoding via the python.console.encoding property. You can edit your registry to force this or pass that argument on the command line
msg5640 (view) Author: Janne Härkönen (yanne) Date: 2010-04-06.05:05:11
I have tested this on multiple machines with Windows XP SP3.

We have code that needs to work both on Jython and Python and on all common OSes. Currently we just work around this with an ugly if, and it seems to be working. 

Thanks for the replies.
msg5864 (view) Author: Régis Desgroppes (rdesgroppes) Date: 2010-06-30.16:00:14
Yes, Philip, you're right: we could steal that information via reflection. Here using Jython:
---
from java.lang import System

def get_encoding(console):
    """returns encoding of this console
    <!> untested if console is null (i.e. when not attached to a tty)
    <!> might fail if a security manager is set without "suppressAccessChecks"
    """
    encoding_method = console.getClass().getDeclaredMethod("encoding", None)
    encoding_method.setAccessible(True)
    return encoding_method.invoke(console, None)

def get_charset(console):
    """returns charset of this console
    <!> untested if console is null (i.e. when not attached to a tty)
    <!> might fail if a security manager is set without "suppressAccessChecks"
    """
    charset_field = console.getClass().getDeclaredField("cs")
    charset_field.setAccessible(True)
    return charset_field.get(console)

if __name__ == "__main__":
    print "java.io.Console's encoding:", get_encoding(System.console())
     # cp437 (string)

    print "java.io.Console's charset:", get_charset(System.console())
     # IBM437 (java.nio.charset.Charset object)
---
Regards,
Regis
msg5866 (view) Author: Régis Desgroppes (rdesgroppes) Date: 2010-07-01.09:40:06
Minimal implementation, working even if JVM is not connected to a TTY:
---
    from java.io import Console

    def get_console_encoding():
        encoding_method = Console.getDeclaredMethod("encoding", None)
        encoding_method.setAccessible(True)
        return encoding_method.invoke(None, None)
---
msg6050 (view) Author: Jim Baker (zyasoft) Date: 2010-09-08.19:37:24
Console apps, where this is the problem, generally don't have security restrictions in place, so maybe we should try the reflection path to get the actual encoding and fail back to what Java is directly telling us.
msg6051 (view) Author: Jim Baker (zyasoft) Date: 2010-09-08.19:37:55
Marking high, ideally we will have in 2.5.2
msg6091 (view) Author: Oti Humbel (otmarhumbel) Date: 2010-09-22.23:54:52
JavaOne BOF
msg6105 (view) Author: Oti Humbel (otmarhumbel) Date: 2010-09-29.05:46:33
A protoype on Windows shows that this only works using java 6:

C:\stuff>"c:\Program Files (x86)\Java\jdk1.5.0_22\bin\java" ConsoleTest
encoding: null

C:\stuff>java -version
java version "1.6.0_20"

C:\stuff>java ConsoleTest
encoding: cp437


Here is the source of ConsoleTest.java:

  private static String getConsoleEncoding() {
    String encoding = null;
    try {
      Class<?> consoleClass = Class.forName("java.io.Console");
      Method encodingMethod = consoleClass.getDeclaredMethod("encoding");
      encodingMethod.setAccessible(true);
      encoding = (String) encodingMethod.invoke(consoleClass);
    } catch (Exception e) {
      // ignore any exception
    }
    return encoding;
  }

  public static void main(String[] args) {
    System.out.println("encoding: " + getConsoleEncoding());
  }
msg6106 (view) Author: Oti Humbel (otmarhumbel) Date: 2010-09-29.06:24:20
a patch for PySystemState.java
msg6107 (view) Author: Oti Humbel (otmarhumbel) Date: 2010-09-29.06:30:44
fixed (on Java 6) with revision 7128
History
Date User Action Args
2010-09-29 06:30:45otmarhumbelsetstatus: open -> closed
resolution: fixed
messages: + msg6107
2010-09-29 06:24:21otmarhumbelsetfiles: + 1568patch.txt
messages: + msg6106
2010-09-29 05:46:36otmarhumbelsetmessages: + msg6105
2010-09-22 23:54:53otmarhumbelsetassignee: otmarhumbel
messages: + msg6091
nosy: + otmarhumbel
2010-09-08 19:37:55zyasoftsetpriority: high
messages: + msg6051
2010-09-08 19:37:25zyasoftsetnosy: + zyasoft
messages: + msg6050
2010-07-01 09:40:08rdesgroppessetmessages: + msg5866
2010-06-30 16:00:16rdesgroppessetnosy: + rdesgroppes
messages: + msg5864
2010-04-10 23:38:51pekka.klarcksetnosy: + pekka.klarck
2010-04-06 05:05:13yannesetmessages: + msg5640
2010-04-04 19:01:42pjenveysetnosy: + pjenvey
messages: + msg5631
2010-03-04 20:15:00fwierzbickisetnosy: + fwierzbicki
messages: + msg5560
2010-03-04 10:44:21yannecreate