Issue2073

strval = "schön"
hexval = ':'.join(x.encode('hex') for x in strval)
print hexval

correctly outputs: 73:63:68:c3:b6:6e
so the german o-umlaut (unicode \u00fc) is correctly converted to the utf-8 byte sequence "c3 b6"

It should be possible to configure jython in such a way that if I pass strval to a Java-Method, then the str is correctly converted back to "schön".

However, jython invokes the Java-method with the 6-character string value "schön", i.e., the o-umlaut has been replaced by the two characters \u00c3 \u00b6.

Sorry for my mast note beeing incomplete.
I have the described encoding problem.  I have reproduced the problem with jython 2.5.2 and 2.5.3 on Windows 7 with Java 7.
I know that I could change the jython code and use unicode-strings throughout.  However, this is no desirable an option for us because the jython code is not under my control. 

In any case I cannot imagine that the behaviour for multibyte-characters is intended by anybody.
I would expect that the conversion uses either Java's default charset, which can be specified by system property file.encoding, or some otherwise specifiable charset.

Another remark: in order for all this to work, my jython-code starts with the header below.  (I had experimented with other settings, too.)

# -*- coding: utf-8 -*-
from org.python.core import codecs
codecs.setDefaultEncoding('utf-8')

The attached program reproduces the problem.  I paste its output. let us see what happens to the umlauts in the display.
Mind the difference between the two lines prefixed by java-hex:
the first reveals the error: the java String has two characters for the german o-umlaut, obtained by padding each of the two bytes from its utf-8-sequence with a zero byte.


Executing:
# -*- coding: utf-8 -*-
from org.python.core import codecs
import Issue2073Main
codecs.setDefaultEncoding('utf-8')
print 'schön'
print 'python-hex:', ':'.join(x.encode('hex') for x in 'schön')
Issue2073Main.javaPrint('schön')
--------------------------------------------
schön
python-hex: 73:63:68:c3:b6:6e
javaPrint: schön
java-hex: 73:63:68:c3:b6:6e
--------------------------------------------
expected output:
javaPrint: schön
java-hex: 73:63:68:f6:6e

This is an unfortunate aspect of Python's str/unicode distinction. Jython used to ignore this distinction, but it caused significant incompatibility with running standard Python code.

The fact that you can construct bytestrings out of UTF-8 sequences in this way is very much orthogonal; I should also point out that the console is a bit problematic here (https://wiki.python.org/jython/ConsoleChoices). I will stick with scripts:

# -*- coding: utf-8 -*-
val = u"schön"
print val

Does print out what we expect:

$ jython27 test_utf8.py
schön

So please use unicode for unicode strings, str (or bytes) for byte strings.

Jython 3.x will make this work much better by making the usual string type be unicode, which will align better with Java usage, but that's very much vaporware for now.

Entwurf:

@Jim Baker, msg8473: Thank you for taking the time to investigate my problem.
I absolutely agree that it was a good decision to change the string handling between 2.1 and 2.5, in that unicode strings are now the default string type.
You are right, if the jython code uses type unicode as opposed to str, everything works nicely.  Unfortunately, we have thousands of lines of jython code that (1) use str, (2) used to work correctly for jython 2.1 and (3) are not shipped with our software but are stored in databases at our customers' sites.
Changing that code manually to use unicode rather than str is tedious and error-prone.

I insist that the current behaviour is undesirable. If you use utf8-encoding and pass the German letter o-umlaut (unicode: \u00f6) from java to jython and then back from jython to java, that single letter is currently replaced to two characters \u00c3\u00b6.  That cannot be intended!

The bug is in the method org.python.core.util.StringUtil.fromBytes(byte[], int, int).
There you find the following snippet, mind the comment:
------------------------------------------------------
// Yes, I know the method is deprecated, but it is the fastest
// way of converting between between byte[] and String
return new String(buf, 0, off, len);
------------------------------------------------------

That method StringUtil.fromBytes(byte[], int, int) is used in a couple of places where the correct encoding is not obvious from the soure code.  Fortunately, for the specific problem under consideration, that encoding *is* evident, so I propose the following fix:

Add two new methods in org.python.core.util.StringUtil, both overloading existing methods with an additional parameter of type Charset:
------------------------------------------------------
public static String fromBytes(ByteBuffer buf, Charset cs) {
  return fromBytes(buf.array(), buf.arrayOffset() + buf.position(),
      buf.arrayOffset() + buf.limit(), cs);
}

public static String fromBytes(byte[] buf, int off, int len, Charset cs) {
  return new String(buf, off, len, cs);
}
------------------------------------------------------

Then, in method org.python.antlr.GrammarActions.extractString(Token, String, boolean) (line 520) replace

------------------------------------------------------
string = StringUtil.fromBytes(decoded);
------------------------------------------------------
by
------------------------------------------------------
string = StringUtil.fromBytes(decoded, cs); // Issue2073: cs!
------------------------------------------------------