Issue1066

CPython 2.4 included the CJKCodecs package: http://cjkpython.i18n.org/

which provides codecs for chinese/japanese/korean etc charsets, 
implemented in C.

The lack of these codecs causes these tests to fail:

+        test_codecencodings_cn
+        test_codecencodings_hk
+        test_codecencodings_jp
+        test_codecencodings_kr
+        test_codecencodings_tw
+        test_codecmaps_cn
+        test_codecmaps_hk
+        test_codecmaps_jp
+        test_codecmaps_kr
+        test_codecmaps_tw

cjk also includes the _multibytecodec module, which affects these tests:

test_multibytecodec
test_multibytecodec_support

We should utilize the nio charsets for these. One gotcha is they encode 
to/decode from actual bytes, not chars (as they should) -- and of course 
our byte bucket (str) is based on chars.

In that case we could probably make the streaming from/to our 'byte 
bucket' more efficient by faking a ByteBuffer that gave back bytes 
from/put back bytes to an underlying char array. That'd avoid an extra 
conversion pass.

The Encoder/Decoder implementations seem to go through the actual 
ByteBuffer methods -- i.e. not through the underlying Buffer arrays 
directly. That'd allow this hack

A CharsetDecoder can take a ByteBuffer instance to fill into -- we'd 
have to use that for this hack, since Charset.encode returns an entirely 
new ByteBuffer

This hack would be kind of a lame, but would go away in Jython 3. Or we 
could just do the extra pass

Another gotcha would be -- can we still retain our error handling 
behavior with Java's Charsets? Briefly looking at them, they seem to 
have fairly similar error handling facilities

Java supports most of the cjkcodecs but not these:

cp932 (mskanji)
euc_jis_2004 (Japanese)
euc_jisx0213 (Japanese)
hz (Simplified Chinese)
iso2022_jp_1 (iso2022 variants)
iso2022_jp_2
iso2022_jp_2004
iso2022_jp_3
iso2022_jp_ext
shift_jis_2004 (Shiftjis variants)
shift_jisx0213

Determined from: http://java.sun.com/j2se/1.5.0/docs/guide/intl/encoding.doc.html

and:
$ grep getcodec *
big5.py:codec = _codecs_tw.getcodec('big5')
big5hkscs.py:codec = _codecs_hk.getcodec('big5hkscs')
cp932.py:codec = _codecs_jp.getcodec('cp932')
cp949.py:codec = _codecs_kr.getcodec('cp949')
cp950.py:codec = _codecs_tw.getcodec('cp950')
euc_jis_2004.py:codec = _codecs_jp.getcodec('euc_jis_2004')
euc_jisx0213.py:codec = _codecs_jp.getcodec('euc_jisx0213')
euc_jp.py:codec = _codecs_jp.getcodec('euc_jp')
euc_kr.py:codec = _codecs_kr.getcodec('euc_kr')
gb18030.py:codec = _codecs_cn.getcodec('gb18030')
gb2312.py:codec = _codecs_cn.getcodec('gb2312')
gbk.py:codec = _codecs_cn.getcodec('gbk')
hz.py:codec = _codecs_cn.getcodec('hz')
iso2022_jp.py:codec = _codecs_iso2022.getcodec('iso2022_jp')
iso2022_jp_1.py:codec = _codecs_iso2022.getcodec('iso2022_jp_1')
iso2022_jp_2.py:codec = _codecs_iso2022.getcodec('iso2022_jp_2')
iso2022_jp_2004.py:codec = _codecs_iso2022.getcodec('iso2022_jp_2004')
iso2022_jp_3.py:codec = _codecs_iso2022.getcodec('iso2022_jp_3')
iso2022_jp_ext.py:codec = _codecs_iso2022.getcodec('iso2022_jp_ext')
iso2022_kr.py:codec = _codecs_iso2022.getcodec('iso2022_kr')
johab.py:codec = _codecs_kr.getcodec('johab')
shift_jis.py:codec = _codecs_jp.getcodec('shift_jis')
shift_jis_2004.py:codec = _codecs_jp.getcodec('shift_jis_2004')
shift_jisx0213.py:codec = _codecs_jp.getcodec('shift_jisx0213')

Deferred to 2.5.1

When I looked at this, the nio charsets have similar default error 
handlers, but there's no way to make custom ones.  I think that rules 
using these charsets out with python, since codecs picked up the ability 
to use a user-defined error handling function in 2.3.  It has been a 
couple years since I looked at this though, so I may be misremembering 
things.

Actually it seems like we could do callable error handlers via nio's 
report error action. That would make the encoder/decoder return a 
CoderResult upon failure but without resetting its state

So we should be able to create a UnicodeError with its start/end/reason 
info from that CoderResult and the input Buffer (to pass to our error 
handler). Then we act upon the handler's result, restarting the 
encoder/decoder from where it left off if necessary

Ahh, that does sound workable.  Nice!

Let's see if we can write wrappers of NIO in time for 2.5.2.

I'm going to try to get this into 2.5.2rc2, so marking high. I think I know the APIs respectively well enough now to write a pure Jython version that leverages java.nio, following Phil's suggestion.

This will not make 2.5.2 unless there's a RC3. I recommend we should release as a separate package on PyPI.

Because of how one needs to do the buffering, it's necessary to use Java to manage the loop for reasonable performance.

FYI Yuji Yamano made some good progress on this task during the PyCon '11 sprint. He actually got it to the point that you could begin encoding asian characters via the codecs module.

I have a preliminary patch from him in a pastebin but I'm sure he'll eventually send us a later version of this patch, and then maybe we can get this in for 2.6

Here is the work in progess patch for the svn trunk.

* Some tests don't pass yet.
* There are still some problems, but I don't remember exectly:-<
* Too many debug log.

These codecs have become standard in Python 2.7 so the updated test_codecs regression test now fails (or acquires skips). Note related issue #2000. I observe that Python 2.7 has given us *codecs* for the missing asian script encodings but they depend on built-in modules I assume Yuji's patch aims to provide.

Is anyone competent and willing to review the patch?

I'm working on syncing the patch with the latest jython. 
See https://bitbucket.org/yyamano/jython/src/89bbdf124e6b/?at=issue1066

Yuji, what's the status of your branch to provide this functionality? Would it be possible to have this synced against Jython trunk?

For such syncing, please note the bitbucket mirror is currently down and has been in that state for the last couple of months; see https://bitbucket.org/site/master/issue/9315/https-bitbucketorg-jython-jython-no-longer, so you will need to sync with hg.python.org/jython

Targeting beta 4 of 2.7; required for work on https://github.com/html5lib/html5lib-python/pull/150

Currently working on this with the assumption we will use CoderResult for error management

I've started to make good progress, using shift_jis as a representative encoding. About 1/4 of the shift_jis tests now pass in test_codecencodings_jp, which seems to be pretty good considering this is mostly covering various error cases.

Given chunking, I suspect we can keep this in Python for now, although we can revisit at a later time.

Completed patch for shift_jis - all shift_jis tests pass in test_codecencodings_jp assuming that the following is changed from using a surrogate (not supportable in Jython unicode) in test_multibytecodec_support.py:

unmappedunicode = u'\ufffe'

The next step will be to register all encodings available in Java, ideally without a lot of boilerplate.

Congratulations on the progress.

0xfffe is a codepoint that is not a character (but it's not technically a surrogate).
http://www.unicode.org/charts/PDF/UFFF0.pdf
Is a unicode object a sequence of code points? Controversial area.

Fixed in http://hg.python.org/jython/rev/6c718e5e9ae9 to the extent possible by using java.nio.charset.Charset. Here are the codecs not available, more or less what Philip identified in msg3880:

euc_jis_2004
euc_jisx0213  
hz
iso2022_jp_1
iso2022_jp_2004
iso2022_jp_3
iso2022_jp_ext 
shift_jis_2004

hz could potentially be supported by preprocessing - it's a way of encoding GB2312 as 2 7-bit bytes, with escaping provided by ~{...~}. It's possible that ICU4J could potentially help as well.

We also potentially gain other encodings as well, such as cp1047, as needed by http://bugs.jython.org/issue550200, supporting EBCDIC.

The one remaining issue I see here is that there are a couple of minor corner cases around errors for trailing bytes where it is not final. It's not clear to me what can really be done here in this case, since it seems to be a property of the decoder; at the very least it's something that's picked up by our unit tests, so it's visible.

I get test failures from test_email and test_email_renamed about the decoding of euc-jp. In a sense this is an improvement, since that bit of the test is skipped if there is no such codec. But now there is ...

======================================================================
FAIL: test_body_encode (email.test.test_email.TestCharset)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\hg\jython-int\dist\Lib\email\test\test_email.py", line 2981, in test_body_encode
    eq('\x1b$B5FCO;~IW\x1b(B',
  File "D:\hg\jython-int\dist\Lib\email\test\test_email.py", line 2981, in test_body_encode
    eq('\x1b$B5FCO;~IW\x1b(B',
AssertionError: '\x1b$B5FCO;~IW\x1b(B' != '\x1b$B5FCO;~IW'

Is this the same for you?

Jeff, I'm seeing the same issues in test_email, but we will fix separately. Since it's in the regrtest, we see it every time it fails.