Title: Python built in function buffer not in Jython
Type: Severity: normal
Components: Core Versions: Jython 2.7
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: jeff.allen Nosy List: amak, fwierzbicki, jeff.allen, oberstet, pjac, santa4nt, zyasoft
Priority: high Keywords:

Created on 2009-12-08.14:27:32 by pjac, last changed 2014-05-22.02:20:39 by zyasoft.

msg5362 (view) Author: Peter (pjac) Date: 2009-12-08.14:27:32
I discovered this issue while trying some old Python code on Jython
2.5.1 on Mac OS X 10.5.

The following page lists built in Python functions,

At the end of the page there are four "Non-essential Built-in Functions"
* apply
* buffer
* coerce
* intern

There are all present in Jython 2.5.0 and 2.5.1 except buffer:

$ ~/jython2.5.1/jython
Jython 2.5.1 (Release_2_5_1:6813, Sep 26 2009, 13:47:54) 
[Java HotSpot(TM) Client VM (Apple Inc.)] on java1.5.0_20
Type "help", "copyright", "credits" or "license" for more information.
>>> apply
<built-in function apply>
>>> coerce
<built-in function coerce>
>>> intern
<built-in function intern>
>>> buffer
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'buffer' is not defined

Note that buffer is still in (C) Python 2.6 and the current
documentation for Python 2.7, and has not been deprecated.
msg5954 (view) Author: Jim Baker (zyasoft) Date: 2010-08-14.05:27:53
Implementing the buffer protocol for strings, arrays, and buffers would seem to be reasonable for Jython, and necessary for implementing the memoryview type for 2.6. However, it's rarely used and the workaround of 

buffer = str

although inefficient and not quite right, is often just what's needed.

Deferring to 2.6.
msg6875 (view) Author: Alan Kennedy (amak) Date: 2012-03-19.20:55:08
Deferring to 2.7
msg7923 (view) Author: Tobias Oberstein (oberstet) Date: 2013-03-08.11:08:06
This currently prevents Twisted to run (tested with Jython 2.7 beta).

Monkey patching (in twisted/python/

      # workaround for Jython, see:
      import __builtin__
      if not hasattr(__builtin__, 'buffer'):
          def _buffer(object, offset = None, size = None):
             if offset is None:
                offset = 0
             if size is None:
                size = len(object)
             return object[offset:offset+size]
          __builtin__.buffer = _buffer

works, but is less than ideal.

Would you agree to increase the prio of this ticket, so that it gets fixed before Jython 2.7 release?
msg7924 (view) Author: Frank Wierzbicki (fwierzbicki) Date: 2013-03-08.16:35:26
I've upped the priority - getting Twisted to work with Jython is very important to us. I don't know if this will block a 2.7.0 though - it might get pushed to a 2.7.x just because getting buffer fully implemented may be a very large task (not as bad as it used to be, we have a large part of memoryview implemented now)

I really want to see 2.7.0 out in the next few months at the latest. I don't think it will be a typical .0 release in that I think I will loudly warn that backwards compatibility won't be frozen as it is in normal releases since it is the last of the 2.x line. I expect we'll be learning a bunch by working on 3.x and I want to backport lots of these lessons. I am highly interested in getting Twisted to work on both 2.7 and the future 3.x of Jython.
msg7925 (view) Author: Tobias Oberstein (oberstet) Date: 2013-03-08.17:01:11
Thanks for upping prio!

There is at least one more problem with Twisted: with Jython, neither sys.platform nor allows to identify the underlying real OS .. which is necessary for various reasons (event loop etc etc).

Do you consider == 'java' to be a Jython bug or a feature?

If not, is 

import java.lang.System

the proper/supported way of getting the real OS?
msg7926 (view) Author: Tobias Oberstein (oberstet) Date: 2013-03-08.19:45:03
Here is the corresponding Twisted ticket:
msg7927 (view) Author: Frank Wierzbicki (fwierzbicki) Date: 2013-03-08.20:17:12
Back when it was implemented, being Java made sense as the original Jython was attempting to be OS unaware as Java had been doing at the time. Now that things have changed (we have a growing posix layer for example) it really doesn't make sense anymore, so I think it is now a bug. Unfortunately there is to much history here for changing in 2.x. I plan to make properly point to the real underlying OS in 3.x.

For now we have os._name that points to the real OS. I think we will probably keep that as an alias in 3.x for backwards compatibility.
msg7938 (view) Author: Frank Wierzbicki (fwierzbicki) Date: 2013-03-17.04:46:00
BTW for twisted, maybe memoryview() could be used instead of buffer? We have pretty good support for memoryview in 2.7b1
msg7945 (view) Author: Frank Wierzbicki (fwierzbicki) Date: 2013-03-20.16:18:45
Talked to Glyph briefly at PyCon and apparently memoryview isn't the right thing for this. He mentioned a strategy used for Python3 that could probably be used for us too while we get around to possibly doing a buffer. Knowing that something about Python3's buffer makes it unusable for Twisted makes me less sure we want to bother with it though...
msg7946 (view) Author: Frank Wierzbicki (fwierzbicki) Date: 2013-03-20.16:19:45
Of course I mean I'm not sure we want to bother with buffer() - I really want Twisted working one way or another as soon as we can.
msg8173 (view) Author: Jeff Allen (jeff.allen) Date: 2013-11-03.12:26:12
I've been working on this for a few weeks, as jython-dev followers know, but I just noticed we have an actual ticket for it.

There's a reasonably conformant buffer object in the trunk now.

But there are several parts to this work: the buffer API at Java level, which we've had for nearly a year; the memoryview object; the buffer object; and to make other objects in Jython both offer and accept the buffer API where they should. That last part is a big piece of work as there are several object types that should offer the API, and lots of methods where only str (bytes) is accepted now, but Python should allow any object with the buffer API. See for example #2011. On the plus side, there is only one buffer API in Jython, so whatever works with  buffer should work with memoryview.

Users should find that buffer and memoryview work nicely with str and bytearray, and with unicode as a method argument (e.g. u"Hello".endswith(buffer('llo')) returns True).

I'm working on array.array at present, first to make it offer the API, then to accept buffer arguments. array.array raises interesting questions about the buffer API, and about memoryview, when the element implementation type is not byte. For now, I'm just making it work for array.array('b').

BTW: It would be good if the issue and other Twisted blockers could be taken as distinct issues: this is big enough as the "buffer missing" ticket. I'm not up for a whole "get Twisted working" ticket.
msg8174 (view) Author: Frank Wierzbicki (fwierzbicki) Date: 2013-11-05.17:35:39
Jeff: Luckily other people are thinking specifically about "get Twisted working in Jython" :) -- but this is a nice big part of it!
msg8175 (view) Author: Frank Wierzbicki (fwierzbicki) Date: 2013-11-05.20:38:19
For the record, I completely agree about working on these things in small chunks :)
msg8554 (view) Author: Jim Baker (zyasoft) Date: 2014-05-22.02:20:39
Closing this out; test_buffer works, along with the underlying buffer protocol, even though there are related issues with Java integration and more tangential ones about Twisted. But those are separate chunks of work.
Date User Action Args
2014-05-22 02:20:39zyasoftsetstatus: open -> closed
resolution: fixed
messages: + msg8554
2013-11-05 20:38:19fwierzbickisetmessages: + msg8175
2013-11-05 17:35:39fwierzbickisetmessages: + msg8174
2013-11-03 12:26:13jeff.allensetassignee: jeff.allen
messages: + msg8173
nosy: + jeff.allen
2013-03-20 16:19:45fwierzbickisetmessages: + msg7946
2013-03-20 16:18:45fwierzbickisetmessages: + msg7945
2013-03-20 15:57:13santa4ntsetnosy: + santa4nt
2013-03-17 04:46:00fwierzbickisetmessages: + msg7938
2013-03-08 20:17:12fwierzbickisetmessages: + msg7927
2013-03-08 19:45:03oberstetsetmessages: + msg7926
2013-03-08 17:01:11oberstetsetmessages: + msg7925
2013-03-08 16:35:26fwierzbickisetpriority: low -> high
messages: + msg7924
2013-03-08 11:08:06oberstetsetnosy: + oberstet
messages: + msg7923
2013-02-26 17:28:01fwierzbickisetnosy: + fwierzbicki
2013-02-20 00:50:03fwierzbickisetversions: + Jython 2.7, - 2.7a1
2012-03-19 20:55:08amaksetnosy: + amak
messages: + msg6875
versions: + 2.7a1, - 2.5.1
2010-08-22 22:34:55zyasoftsetpriority: low
2010-08-14 05:27:54zyasoftsetnosy: + zyasoft
messages: + msg5954
2009-12-08 14:27:32pjaccreate