Issue1312

I'm trying to make pyftpdlib work on Jython but I'm encountering a bunch
of apparently weird problems.
When I run the test suite I get the following two errors.
The first one:

Exception in thread Thread:Traceback (most recent call last):
  File "/home/user/jython/Lib/threading.py", line 247, in _Thread__bootstrap
    self.run()
  File "test_ftpd.py", line 1998, in run
    self.server.serve_forever(timeout=self.__timeout, count=1,
  File "/home/user/jython/bin/ftpserver.py", line 3164, in serve_forever
    poll_fun(timeout, map)
  File "/home/user/jython/Lib/asyncore.py", line 121, in poll
    r, w, e = select.select(r, w, e, timeout)
  File "/home/user/jython/Lib/asyncore.py", line 121, in poll
    r, w, e = select.select(r, w, e, timeout)
  File "/home/user/jython/Lib/select.py", line 185, in native_select
    results = pobj.poll(timeout)
  File "/home/user/jython/Lib/select.py", line 134, in poll
    for k in selectedkeys.iterator():
  File "/home/user/jython/Lib/select.py", line 134, in poll
    for k in selectedkeys.iterator():
OutOfMemoryError: java.lang.OutOfMemoryError: Java heap space


The second one:

# An unexpected error has been detected by Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0xb7fb73a2, pid=7275, tid=3034299280
#
# Java VM: Java HotSpot(TM) Client VM (11.0-b15 mixed mode, sharing
linux-x86)
# Problematic frame:
# C  [ld-linux.so.2+0xe3a2]
#
# An error report file with more information is saved as:
# /home/user/jython/bin/hs_err_pid7275.log
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
./jython: line 216:  7275 Aborted                 "${JAVA_CMD[@]}"
$JAVA_OPTS "${java_args[@]}" -Dpython.home="$JYTHON_HOME"
-Dpython.executable="$PRG" org.python.util.jython $JYTHON_OPTS "$@"


Run on Ubuntu 8.04, JSE 1.6.0_10-b33, Jython 2.5b3

You should be able to get around it with more memory. Try running jython 
with -J-Xmx512m

Prioritzing this as high because I was meaning to bump our default Xmx to 
512 anyway

The latter crash is hopefully symptomatic of the memory problem -- 
otherwise, the JVM just isn't supposed to crash =]

It's exactly the same with -J-Xmx512m option.

Does any more memory help i.e. -J-Xmx1024m? I somewhat doubt your test 
suite really needs this much though -- maybe we have a leak somewhere

On a side note: did you tweak anything to deal with the partial support
for ipv6. I get an assertion failure:
Traceback (most recent call last):
  File "test/test_ftpd.py", line 94, in <module>
    SUPPORTS_IPV6 = socket.has_ipv6 and try_address('::1',
family=socket.AF_INET6)
  File "test/test_ftpd.py", line 85, in try_address
    s = socket.socket(family, socket.SOCK_STREAM)
  File "test/test_ftpd.py", line 85, in try_address
    s = socket.socket(family, socket.SOCK_STREAM)
  File "/home/kfitch/jython/dist/Lib/socket.py", line 1070, in __init__
    _sock = _realsocket(family, type, proto)
  File "/home/kfitch/jython/dist/Lib/socket.py", line 549, in _realsocket
    assert family == AF_INET, "Only AF_INET sockets are currently
supported on jython"
AssertionError: Only AF_INET sockets are currently supported on jython


See Issue1210.

So I set SUPPORTS_IPV6=False so I could try to reproduce this. Then when
I run the test suite:
cd pyftpdlib-0.5.1;jython test/test_ftpd.py 
I get numerous errors such as:
test_rein_during_transfer (__main__.TestFtpAuthentication) ... Traceback
(most recent call last):
  File "__pyclasspath__/pyftpdlib/ftpserver.py", line 1957, in push_dtp_data
  File "__pyclasspath__/pyftpdlib/ftpserver.py", line 894, in
push_with_producer
  File "__pyclasspath__/pyftpdlib/ftpserver.py", line 923, in initiate_send
  File "__pyclasspath__/pyftpdlib/ftpserver.py", line 923, in initiate_send
NameError: global name 'buffer' is not defined

ERROR

You can solve that by getting the latest pyftpdlib revision from the SVN
repository:
http://code.google.com/p/pyftpdlib/source/checkout

So I checked out the latest code, and the 'buffer' issue is gone, but I
still had to tweak SUPPORTS_IPV6 to get it to run. Also, the test suite
does not pass under CPython. The first thing that jumped out at me is
that it is checking for equality of floating point numbers, generally a
BIG no-no. In particular since, at a glance, those numbers are related
to something called 'CallLater'. Usually when scheduling something in
the future you can only so it will happen no-sooner-than, not exactly at.

That being said, I see nothing like what was described in the original
bug report. To be honest, my best guess is that this is actually a
hardware issue (bad ram?) or perhaps a currupted disk ... since the JVM
"shouldn't" crash.

> the 'buffer' issue is gone, but I still had to 
> tweak SUPPORTS_IPV6 to get it to run

Yes, I left it that way since I've noticed there's a bug already opened
for that on Jython issue tracker. 

> Also, the test suite does not pass under CPython

This is strange. Have you updated *also* test_ftpd.py to the latest
revision?

CPython 2.5.2 on Ubuntu 8.04 x64

I am at pyftpdlib r586 (the only mod is setting the IPv6 thing to false)

Latest jython from svn r6228

I'm not getting any failure. Maybe it's because I'm on a 32 bit sistem.
Could you please open a new issue on the pyftpdlib tracker?
http://code.google.com/p/pyftpdlib/issues/list

I realize I haven't answered some questions which have been asked to me.

> Does any more memory help i.e. -J-Xmx1024m? I somewhat 
> doubt your test  suite really needs this much though 
> -- maybe we have a leak somewhere

Tried with 1024 and also with 2048. Nothing changed.

> To be honest, my best guess is that this is actually a
> hardware issue (bad ram?) or perhaps a currupted disk 
> ... since the JVM "shouldn't" crash.

I'm running the tests on a virtualized Linux Ubuntu Server 8.04.
My actual workstation has 1.5 Gb of ram installed, the virtualized one
has 256 Mb reserved but it's a very minimal Ubuntu installation so I'm
sure it's not running out of memory.
I don't think I have any corrupted ram slot or physical drive issue.

The pyftpdlib related issues raised by Kevin should now be fixed in the
latest pyftpdlib SVN revision.

I'm attaching the log file generated by Jython when it crashes. Hope it
helps.

With the new changes to pyftpdlib things are better. I get two errors:

======================================================================
ERROR: test_permit_privileged_ports (__main__.TestConfigurableOptions)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_ftpd.py", line 1675, in test_permit_privileged_ports
    socket.getservbyport(port)
  File "test/test_ftpd.py", line 1675, in test_permit_privileged_ports
    socket.getservbyport(port)
  File "/home/kfitch/projects/jython/dist/Lib/socket.py", line 541, in
getservbyport
    raise NotImplementedError("getservbyport not yet supported on jython.")
NotImplementedError: getservbyport not yet supported on jython.

======================================================================
FAIL: test_dele (__main__.TestFtpFsOperations)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_ftpd.py", line 871, in test_dele
    self.assertRaises(ftplib.error_perm, self.client.delete, self.tempdir)
AssertionError: error_perm not raised

----------------------------------------------------------------------
Ran 94 tests in 24.557s

FAILED (failures=1, errors=1)

256MB is not very much. Java LOVES RAM, Python LOVES RAM ... put the two
together and ... well 256 probably isn't going to cut the mustard.

can you run

free -s 0.25 

In another terminal while you run the tests. On my box the amount of
free memory drops from ~274MB down to as low as ~24MB during the run ...
so it looks like it really wants about 250MB to run! So a box with only
256MB of RAM is going to have a tough time running this test.

It's surprising Jython uses so much memory!
With cPython the test suite uses 20 Mb at most.
Anyway, I bumped up the VMWare memory to 1028 Mb but the problem persists.
I ran free -s 0.25 in another terminal and noticed that I never run out
of free memory.

It seems that the problem might be somewhat related with the memory
since it appears only after a certain amount of time but I would say
that 1Gb of physical memory should be more than enough for this.

[billiejoex]
> It's surprising Jython uses so much memory!
> With cPython the test suite uses 20 Mb at most.
> Anyway, I bumped up the VMWare memory to 1028 Mb but the problem persists.

Are you sure that your "buffer" implementation is memory eficient?

Cpython's buffer implementation is efficient. But jython does not
support buffer. The "mock" version of buffer that you are using may
perhaps be causing the memory problem?

I got rid of my own "mock" buffer implementation in r572:
http://code.google.com/p/pyftpdlib/source/detail?r=572#

I was able to run through the entire test_ftpd (pyftpdlib r599/jython 
r6258) without any memory issues. Java used about 150MB of memory max. 
This was on OS X 64 bit Java 6

Output here:

http://pylonshq.com/pasties/ed53e91bc7ded8c93d3c8cf7be3281f7

If you're still seeing OOM errors I'm inclined to point the finger at 
your version of Java. Can you try the tests elsewhere?

It's been a while since any activity on this and the tests do run now, 
so I'm closing this out.

It looks like there are some other issues encountered in the test run 
but those should be narrowed down and have separate, specific bugs 
logged for them