Issue1551

classification
Title: Java objects cannot be copied by the copy module
Type: behaviour Severity: critical
Components: Core Versions: 2.5.1
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: zyasoft Nosy List: juneau001, slucas, zyasoft
Priority: Keywords:

Created on 2010-02-08.16:43:21 by slucas, last changed 2010-08-11.14:16:13 by zyasoft.

Files
File name Uploaded Description Edit Remove
test pjenvey, 2010-02-15.21:29:07 test
Messages
msg5508 (view) Author: Sharon Lucas (slucas) Date: 2010-02-08.16:43:19
I'm trying to port our Java application that uses Jython 2.1 to use Jython 2.5.1.  However, I'm getting the following error using Jython 2.5.1 which prevents us from using it: 

  SystemError: Automatic proxy initialization should only occur on proxy classes 

I see this error after calling copy.copy or copy.deepcopy() to copy a Java type variable (such as one of type java.util.ArrayList for example).  The copy completes without any errors but as soon as the copied variable is accessed in any fashion, this SystemError occurs. 

Josh Juneau responded to my posting to the jython-user mailing list about this problem and said:

"Looks like your copied object is being treated as a proxy class and I don't think that should be happening.  The PyObject code has been completely overhauled for the 2.5.x release, and therefore it is difficult for me to look through the old code from 2.2.1 and compare...so I cannot tell you the exact cause.  This looks like a bug"

Here's a simple example of this problem after copy.copy() is used to copy a java.util.ArrayList instance, and then the copied variable is accessed.  This example works fine using Jython 2.1.   

C:\jython2.5.1>jython 
Jython 2.5.1 (Release_2_5_1:6813, Sep 26 2009, 13:47:54) 
[Java HotSpot(TM) Client VM (Sun Microsystems Inc.)] on java1.6.0_13 
Type "help", "copyright", "credits" or "license" for more information. 
>>> from java.util import ArrayList 
>>> gList = ArrayList() 
>>> gList 
[] 
>>> import copy 
>>> copyList = copy.copy(gList) 
>>> copyList 
Traceback (most recent call last): 
 File "<stdin>", line 1, in <module> 
SystemError: Automatic proxy initialization should only occur on proxy classes 
>>> globals() 
Traceback (most recent call last): 
 File "<stdin>", line 1, in <module> 
SystemError: Automatic proxy initialization should only occur on proxy classes 
>>> 

Note this problem occurs with some other Java classes as well, including some custom  classes we use, not just a Java list. 

It seems to me to be a very "bad" thing as globals() also then fails with this error.  This problem prevents us from migrating our Java application to use Jython 2.5.1.

Note:  Our application is STAX which is used by thousands of poeple and is part of the STAF open source project (http://staf.sourceforge.net).  Many of our customers have requested upgrading STAX to use Jython 2.5.1 (instead of Jython 2.1) so that they can take advantage of the new Python features.  However, this bug prevents use from using Jython 2.5.1.

Please let me know if there is a workaround or fix for this bug.  Or, if I can provide any more information to help resolve this problem.  Thank you.

If you n
msg5518 (view) Author: Juneau001 (juneau001) Date: 2010-02-09.20:06:36
I am looking into this issue starting with PyObject.  It seems that java libraries shouldn't be treated as proxy classes and for some reason a copy  of such an object is being treated as such.
msg5526 (view) Author: Juneau001 (juneau001) Date: 2010-02-14.15:58:13
A workaround to make shallow copies of native Java objects is to use the Java clone() technique.  In order to repair this issue and ensure that the Jython interpreter does not treat copies of native Java objects using copy.copy as proxy classes, we'll have to somehow alter the proxy objects (or perhaps the copy lib) to perform a check to see if indeed the object being copied is a native Java object.  If it is a native Java object, then treat differently then a Python object that inherits from a Java class.  

Perhaps the easiest fix would be to place a check in the copy lib that would simply use clone() as opposed to copy.copy if an object is found to be a native Java object.  The most difficult part of the problem is determining if an object is indeed native Java or a Python class that inherits from a Java object.
msg5551 (view) Author: Jim Baker (zyasoft) Date: 2010-02-23.17:17:57
As I mentioned in my email to jython-dev, you can determine whether something is a java object via isinstance(obj, java.lang.Object) and whether a type is a java class via issubclass(cls, java.lang.Object)
msg5553 (view) Author: Juneau001 (juneau001) Date: 2010-02-25.04:08:00
I've re-implemented copy.py so that it makes use of the clone() technique to create shallow copies of Java objects.  It now also has a java deep copy implementation.  Lastly, a function is_java(obj) has been included to help differentiate java objects from python.  This implementation of copy.py is fully functional.  However, it appears that copy.py in general needs to be optimized.  There is currently some development work towards implementing copy.py in java...this should help optimization.
msg5744 (view) Author: Jim Baker (zyasoft) Date: 2010-04-26.00:35:15
I have implemented a possible fix for this in r7045, with support for __copy__ and __deepcopy__ in proxy objects. This sidesteps the question of proxy initialization.

Consequently I'm marking this issue as pending. However, we need significantly more test coverage for such a feature, especially with any number of corner cases.
msg5915 (view) Author: Sharon Lucas (slucas) Date: 2010-07-20.19:53:34
I downloaded Jython 2.5.2 Beta 1 to verify that this bug I opened has been fixed.  Some of the issues have been fixed (e.g. problems copying Python objects).  But the problem copying a Java object still exists and this prevents my Java application, STAX, from working with Jython 2.5.2.  We need this fixed in Jython 2.5.2.  Should I reopen issue 1551 or open a new bug?  I would greatly appreciate any help you can provide in resolving this issue.

Here's an example of the problem that remains when copying a Java object (and that was fixed in a private fix to the copy module that Josh Juneau had provided earlier this year).
1) Create  an instance (e.g. me) of the Java Person class object.  Note that this class does not implement a clone method and isn't Serializable.  It's source code is at the bottom on this note.
2) Use copy.copy() to make a copy of me and call it copyMe. 
3) Accessing copyMe causes an Automatic proxy initialization error.  Even worse is that globals() fails with this error as well (and this prevents our Java application from working with Jython 2.5.2 as it essentially clones globals() and this is now failing).  In Jython 2.1 and 2.2.1, globals() would not fail.

C:\jython2.5.2b1>jython
Jython 2.5.2b1 (Release_2_5_2beta1:7075, Jun 28 2010, 07:44:20)
[Java HotSpot(TM) Client VM (Sun Microsystems Inc.)] on java1.6.0_20
Type "help", "copyright", "credits" or "license" for more information.
>>> import copy, sys
>>> sys.path.append("C:/dev/src/stax/PersonDir")
>>> import Person
>>> me = Person("Sharon", "Lucas")
>>> copyMe = copy.copy(me)
>>> copyMe
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
SystemError: Automatic proxy initialization should only occur on proxy classes
>>> globals()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
SystemError: Automatic proxy initialization should only occur on proxy classes
>>>

In Jython 2.1, doing a shallow copy of this Java class object resulted in error "un(shallow)copyable object of type org.python.core.PyJavaInstance".  Doing a shallow copy of globals(), would work and essentially appear to do a deep copy of the Java class object. 

C:\Jython-2.1>jython 
Jython 2.1 on java1.5.0_07 (JIT: null) 
Type "copyright", "credits" or "license" for more information. 
>>> import copy, sys 
>>> sys.path.append("C:/dev/src/stax/PersonDir") 
>>> import Person 
>>> me = Person("John", "Java") 
>>> copyMe = copy.copy(me) 
Traceback (innermost last): 
  File "<console>", line 1, in ? 
  File "C:\Jython-2.1\Lib\copy.py", line 78, in copy 
Error: un(shallow)copyable object of type org.python.core.PyJavaInstance 
>>> globals() 
{'sys': sys module, 'PyPerson': <module PyPerson at 3298006>, 'Person': <jclass 
Person at 31228301>, '__name__': '__main__', 'copy': <module  copy at 29199470>,
'me': Person@14ebadd, '__doc__': None} 
>>> copyGlobals = copy.copy(globals()) 
>>> copyGlobals        <== No error accessing the copy of globals() - looks like it actually did a deep copy of me 
{'sys': sys module, 'Person': <jclass Person at 31228301>, __name__': '__main__
', 'copy': <module copy at 29199470>, 'me': Person@14ebadd, '__doc__': None}                      | 
>>> me.setFirst("Jane") 
>>> me.printPerson() 
Jane Java 
>>> copyGlobals['me'].printPerson() 
Jane Java 


Here's the content of Person.java (its compiled Person.class resides in C:\dev\src\stax in the above examples).

public class Person {

    private String first;
    private String last;
    
    public Person(String first, String last){
        this.first = first;
        this.last = last;
    }
    
    public void printPerson(){
        System.out.println(first + " " + last);
    }
    
    public void setFirst(String first){
        this.first = first;
    }
    
    public void setLast(String last){
        this.last = last;
    }
}
msg5948 (view) Author: Jim Baker (zyasoft) Date: 2010-08-09.21:04:30
Added null versions of __copy__ and __deepcopy__ that raise a TypeError for Java classes not implementing Cloneable and Serializable respectively. Also I added a version of the Person test submitted by Sharon to test_java_integration.py

The globals problem observed is actually the result of the proxies not being configured properly, it doesn't have anything to do with globals itself, just accessing objects referenced by it.

Ideally immutable Java objects like java.lang.String should be copied too (Python semantics). Note this works for deepcopy.

Lastly, copy_reg which is somewhat related is no longer used by copy in CPython 2.5, so need to update our copy module support.

For these last two reasons, keeping this pending.
msg5950 (view) Author: Jim Baker (zyasoft) Date: 2010-08-11.14:16:11
Added immutable object support in r7089. Sharon reported in an email that copy is now working for her application. Lastly, we use the standard CPython copy module, so any copy_reg in there is as it should be apparently ;)

Closing this issue out at last. If there are further subtle issues in copy, possibly not 2.5.2 blockers, let's address in a separate issue.
History
Date User Action Args
2010-08-11 14:16:13zyasoftsetstatus: pending -> closed
messages: + msg5950
2010-08-09 21:04:32zyasoftsetmessages: + msg5948
2010-07-20 19:53:36slucassetmessages: + msg5915
2010-04-26 18:38:03zyasoftsettitle: SystemError: Automatic proxy initialization should only occur on proxy classes -> Java objects cannot be copied by the copy module
2010-04-26 01:19:29zyasoftsetresolution: fixed
2010-04-26 00:35:26zyasoftsetassignee: zyasoft
2010-04-26 00:35:15zyasoftsetstatus: open -> pending
messages: + msg5744
2010-02-25 04:08:01juneau001setmessages: + msg5553
2010-02-23 17:17:58zyasoftsetnosy: + zyasoft
messages: + msg5551
2010-02-15 21:29:07pjenveysetfiles: + test
2010-02-14 15:58:15juneau001setmessages: + msg5526
2010-02-09 20:06:36juneau001setnosy: + juneau001
messages: + msg5518
2010-02-08 16:43:21slucascreate