Issue1166

When trying to install jython-elementtree or setuptools trunk, or running our unit tests it reports the 
following error:

Traceback (most recent call last):
  File "./setup.py", line 53, in <module>
    setup(name = "PyAMF",
  File "/usr/local/src/jython-trunk/dist/Lib/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/local/src/jython-trunk/dist/Lib/distutils/dist.py", line 974, in run_commands
    self.run_command(cmd)
  File "/usr/local/src/jython-trunk/dist/Lib/distutils/dist.py", line 994, in run_command
    cmd_obj.run()
  File "/home/buildbot/Buildslaves/pyamf/amd64-ubuntu-jython/build/setuptools-0.6c8-
py2.5.egg/setuptools/command/install.py", line 56, in run
  File "/usr/local/src/jython-trunk/dist/Lib/distutils/command/install.py", line 517, in run
    self.run_command(cmd_name)
  File "/usr/local/src/jython-trunk/dist/Lib/distutils/cmd.py", line 333, in run_command
    self.distribution.run_command(command)
  File "/usr/local/src/jython-trunk/dist/Lib/distutils/dist.py", line 994, in run_command
    cmd_obj.run()
  File "/home/buildbot/Buildslaves/pyamf/amd64-ubuntu-jython/build/setuptools-0.6c8-
py2.5.egg/setuptools/command/install_lib.py", line 21, in run
  File "/usr/local/src/jython-trunk/dist/Lib/distutils/command/install_lib.py", line 116, in install
    outfiles = self.copy_tree(self.build_dir, self.install_dir)
  File "/home/buildbot/Buildslaves/pyamf/amd64-ubuntu-jython/build/setuptools-0.6c8-
py2.5.egg/setuptools/command/install_lib.py", line 50, in copy_tree
  File "/usr/local/src/jython-trunk/dist/Lib/distutils/cmd.py", line 385, in copy_tree
    return dir_util.copy_tree(
  File "/usr/local/src/jython-trunk/dist/Lib/distutils/dir_util.py", line 169, in copy_tree
    outputs.extend(
  File "/usr/local/src/jython-trunk/dist/Lib/distutils/dir_util.py", line 174, in copy_tree
    copy_file(src_name, dst_name, preserve_mode,
  File "/usr/local/src/jython-trunk/dist/Lib/distutils/file_util.py", line 172, in copy_file
    os.utime(dst, (st[ST_ATIME], st[ST_MTIME]))
  File "/usr/local/src/jython-trunk/dist/Lib/os.py", line 534, in utime
    _posix.utimes(path, long(atime * 1000), long(mtime * 1000))
TypeError: utimes(): 3rd arg can't be coerced to long

For a full issue report see our buildbot log at http://buildbot.pyamf.org/builders/amd64-ubuntu-
jython/builds/0/steps/jython-install/logs/stdio/text

I'm logging this ticket against 2.5alpha3 because the 2.5beta0 version nr isn't available in the list, 
might want to add that as well. Exact revision nr I was using is 5557.

Is this 64 bit java? I know there's an existing issue with utimes and 64 
bit java (on OS X, maybe others) -- though oddly enough this doesn't 
look like that issue at all.

I wasn't able to reproduce this on OS X 64 bit java (using jython trunk 
r5557, pyamf trunk).

Does the following simple case work?

import os
f = '/tmp/jython-foo'
open(f, 'w').close()
s = os.stat(f)
os.utime(f, (s.st_atime, s.st_mtime))

If you have a local Jython build and can reproduce the issue, it'd help 
to know what the actual arguments being passed to os.utime are (you 
could temporarily hack your os.py to print them out)

Also I'd like to double check what the value of 'os._native_posix' is on 
this platform

Yes it's a 64-bit machine. I put your test into a .py file and ran it 
with Jython and it didn't report any errors..

The results of the second request:

Jython 2.5b0+ (trunk:5557, Nov 8 2008, 16:33:47) 
[Java HotSpot(TM) 64-Bit Server VM (Sun Microsystems Inc.)] on 
java1.5.0_06
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os._native_posix
True
>>> 

The errors with jython-elementtree went away because they recently 
changed the installation procedure (now using an install.py instead of 
setuptools-based setup.py). I'll see if I can create a better reproducible test for this issue.

Alright so I'm able to reproduce this with jython-elementtree 
(http://code.google.com/p/jython-elementtree) r26 and jython r5587. 
running 'jython setup.py install' throws this error. Using setuptools-
0.6c8. Not sure if it's then a jython, jython-elementtree or setuptools 
issue, so let me know if I should open a ticket against a different 
project, but it all points to jython it seems.

hippyhacker noticed stat results on this platform (64 bit linux) are 
occasionally returning huge numbers (for things like st_mtime): http://pylonshq.com/pasties/1007

This must be the cause of this problem, I can reproduce the error 
message on any platform by passing a huge long such as 7308549900279964261 to os.utime

I have a Java 64/amd64 linux VM setup to work on this, I just need a 
file that can reproduce the problem. I'm positive it's the problem 
hippyhacker described, so I'm looking for a file that will return a huge 
long timestamp from stat

you should be able to tar up the file and send it to me (it'll preserve 
its timestamps), though hippyhacker tried and that didn't work (might 
have been his fault). It'd be nice if you could double check that the 
problem still happens after untarring the file

(and I mean huge, as shown here: http://pylonshq.com/pasties/1007 )

I tarred up the elementtree-jython trunk on my server using:

tar --atime-preserve -czvf et-jython.tar.gz jython-elementtree-trunk

I then downloaded it and tried to unpack it, which threw the following 
warnings:

tar: jython-elementtree-trunk/build/lib/elementtree/ElementTree.py: time 
stamp 2030-11-07 16:42:11 is 692845574 s in the future
tar: jython-elementtree-trunk/build/lib/elementtree/TidyTools.py: 
implausibly old time stamp 1970-01-01 01:00:00
tar: jython-elementtree-
trunk/build/lib/elementtree/TidyHTMLTreeBuilder.py: implausibly old time 
stamp 1970-01-01 01:00:00
tar: jython-elementtree-trunk/build/lib/xml/etree/ElementInclude.py: 
time stamp 2026-01-20 09:52:31 is 541448194 s in the future
tar: jython-elementtree-trunk/build/lib/xml/parsers/__init__.py: 
implausibly old time stamp 1970-01-01 01:00:00

So there's definitely something weird going on there. I attached that 
same tarball, I hope it helps solving this bug.. I have no idea how 
those timestamps became so screwed up, the time is and always has been 
correct on my server afaik. So I'm guessing Jython did something weird 
with the timestamps when trying to build the package? Just guessing..

I'm getting the same problem trying to install django.

jython 5.5b0+ (trunk, Dec 9 2008, 16:54:43)
[Java HotSpot(TM) 64-bit Server VM (Sun Microsystems Inc.)] on 
java1.6.0_04

django/trunk r9619

when I typed
# jython setup.py install
there were lots of

"copying django/this/that -> build/lib/django/this/that"

then, build/lib/django/contrib/csrf/tests.py throws up a traceback, 
with line 542 of os.py
_posix.utimes(path, long(atime * 1000), long(mtime * 1000))

throwing 

TypeError: utimes(): 3rd arg can't be coerced to long

I stuck some print statements into os.py, and 

DEBUG: mtime=7310012245875128690

what? Checking with stat
# stat build/lib/django/contrib/csrf/tests.py
  File: `build/lib/django/contrib/csrf/moved_tests.py'
  Size: 5037            Blocks: 24         IO Block: 4096   regular file
Device: 823h/2083d      Inode: 11255892    Links: 1
Access: (0644/-rw-r--r--)  Uid: ( XXXX/  XXXX)   Gid: (  XXXX/   XXXXX)
Access: 2008-12-10 09:00:48.000000000 +1100
Modify: 7165064483007524707
Change: 2008-12-10 09:00:45.000000000 +1100

Hmm. Other files don't stat like that (XXX overtyped by me by the 
way)...
# stat build/lib/django/contrib/gis/measure.py
  File: `build/lib/django/contrib/gis/measure.py'
  Size: 12528           Blocks: 40         IO Block: 4096   regular file
Device: 823h/2083d      Inode: 11256021    Links: 1
Access: (0644/-rw-r--r--)  Uid: ( XXXX/  XXXXXX)   Gid: (  XXX/   XXXXX)
Access: 2008-12-10 09:00:45.000000000 +1100
Modify: 1970-01-01 13:28:48.000000000 +1000
Change: 2008-12-10 09:00:45.000000000 +1100

That  that Modify time is not nonsence, but it's not right either. see 
also:

# ls -l build/lib/django/contrib/csrf/tests.py
-rw-r--r--  1 XXXXXX XXXXX 5037 7165064483007524707 
build/lib/django/contrib/csrf/tests.py

# ls -l  build/lib/django/contrib/gis/measure.py
-rw-r--r--  1 XXXXXX XXXXX 12528 Jan  1  1970 
build/lib/django/contrib/gis/measure.py

Yeah, right.

# touch build/lib/django/contrib/gis/measure.py
# ls -l build/lib/django/contrib/gis/measure.py
-rw-r--r--  1 XXXXXX XXXXX 12528 Dec 10 09:25 
build/lib/django/contrib/gis/measure.py

OK then,
# touch build/lib/django/contrib/csrf/tests.py
# ls -l build/lib/django/contrib/csrf/tests.py
-rw-r--r--  1 A01417 eawmd 5037 Dec 10 09:26 
build/lib/django/contrib/csrf/tests.py

Another "jython setup.py install" failed the same way further down, in 
contrib/webdesign. So I went on a touch rampage all over build/lib, 
then "jython setup.py install" completed OK.

But why was "jython setup.py build" making dodgy MTIMES in the build 
directory when stat in the source files looks fine?

Why is coreutils giving me nonsence with some of these dodgy MTIMES 
(but not all of them)?

I was finally able to reproduce this by manually setting a file to a 
huge timestamp. I later realized 64 bit CPython also allowed me to set 
these huge timestamps. This seemed to only work on Linux, other OSes I 
tried overflow back to smaller values

So the question is, who set the huge mtime value in the first place? 
Likely Jython -- but how? We already knew utimes had some issues on 64 
bit platforms, but how would it manage to set a timestamp to such a huge 
value if it runs into "3rd arg can't be coerced to long" on those huge 
values?

I'm thinking maybe that jna-posix utimes bug we already knew about 
somehow fudged another value into a much larger timestamp value

So I've a) fixed the 64 bit utimes bug we knew about, and also b) fixed 
utimes to allow setting these huge timestamps (as 64 bit CPython also 
allows it). So hopefully a) solves fudging timestamps, if we were in 
fact doing it before, and b) allows fudged timestamps to just pass on 
through. r5768

pjenvey@lamp:~/src/jython$ ll /tmp/foo.test 
-rw-r--r-- 1 pjenvey pjenvey 0 7165064483007524864 /tmp/foo.test
pjenvey@lamp:~/src/jython$ stat !$
stat /tmp/foo.test
  File: `/tmp/foo.test'
  Size: 0         	Blocks: 0          IO Block: 4096   regular 
empty file
Device: 801h/2049d	Inode: 78731       Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/ pjenvey)   Gid: ( 1000/ pjenvey)
Access: 7165064483007524864
Modify: 7165064483007524864
Change: 2008-12-15 00:40:34.000000000 +0000
pjenvey@lamp:~/src/jython$ touch /tmp/foo.test2
pjenvey@lamp:~/src/jython$ ll !$
ll /tmp/foo.test2
-rw-r--r-- 1 pjenvey pjenvey 0 2008-12-15 19:23 /tmp/foo.test2
pjenvey@lamp:~/src/jython$ dist/bin/jython
Jython 2.5b0+ (trunk:5768, Dec 15 2008, 19:21:04) 
[Java HotSpot(TM) 64-Bit Server VM (Sun Microsystems Inc.)] on 
java1.6.0_07
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.stat('/tmp/foo.test').st_mtime
7165064483007524864L
>>> os.utime('/tmp/foo.test2', (_, _))
>>> 
pjenvey@lamp:~/src/jython$ !-2
ll /tmp/foo.test2
-rw-r--r-- 1 pjenvey pjenvey 0 7165064483007524864 /tmp/foo.test2

Confirmed it works, cheers!