Issue2508

We have noticed a problem that occurs sometimes when using non-blocking sockets with Jython.  Our application will call send() on a socket and then check the amount of bytes that are returned to see if the data has been sent.  If the number of bytes returned matched the number of bytes sent, then close() is called on the socket.  This logic works without problems when using cPython.  However, in Jython, if Netty has not finished delivering the data, then the close() will cause Netty to throw away whatever bytes have not been delivered yet.  Our application is unaware that this situation has occurred, and the client of the application receives a truncated response.

We can only reproduce this problem when the application and our client are communication over the slower network conditions of a VPN.  Normally Netty delivers the data before our application calls close() on the socket.

There is a FIXME in the send() method in Lib/_socket.py that reads:

  # FIXME are we sure we are going to be able to send this much data, especially async?
  return len(data)


I wanted to make sure that I understood the behavior of send() when used with non-blocking sockets, so I reviewed the Python documentation and the Posix specification for send() and write().  Two details in the Posix specification seem related to this situation:

1) If no flags are passed to send(), it is equivalent to write().  Jython is currently ignoring the flags that are passed to send().

    If the socket argument refers to a socket and the flags argument is 0, the send() function is equivalent to write().

See: http://pubs.opengroup.org/onlinepubs/9699919799/functions/send.html

2) write() will return the number of bytes written, even in non-blocking mode:

    If the O_NONBLOCK flag is set, write() shall not block the thread. If some data can be written without blocking the thread, write() shall write what it can and return the number of bytes written. Otherwise, it shall return -1 and set errno to EAGAIN.

See: http://pubs.opengroup.org/onlinepubs/9699919799/functions/write.html

This is the Posix non-blocking socket behavior.  For Python non-blocking sockets, the behavior is the same.  send() will return the number of bytes actually sent.

See: https://docs.python.org/2/library/socket.html - The description of send() says "Returns the number of bytes sent. " 
also see: http://stackoverflow.com/a/6718824

I did not see a way to easily implement the Posix behavior using the Netty API.  It might be possible to maintain some state concerning sockets that have not delivered all of their data so that a socket close() could internally delay calling close on the Netty channel until the data was fully delivered.

nonblocking-example.tar contains a simple example that should demonstrate this problem if 250ms or more of network latency is present.  nonblock.py  is a simplistic server and dl.py is a client.  These scripts should be run on separate physical or virtual machines. There is a README file that is included with instructions for setup and for simulating network delay in linux using the "tc" command.  If the problem is triggered, then the downloaded file will be truncated.

I thought about this behavior, but since this was not addressed by tests, or the docs I looked at, I simply let it go with the FIXME. Thanks for looking into this!

I had seen similar VPN issues with sockets in the past (on old versions of Windows at least), but had not considered this could be due to socket closing. It may have been a similar issue to what we are seeing here - Java sometimes reproduces lowest-common Windows behavior in other corner cases.

Most likely the solution here is to prevent early closes by validating data has been flushed, via flush notifications.

My reading of C socket semantics is that send simply means the data was copied over to the kernel for actual network send; this seems no different than it being copied over to Netty. But perhaps we can upgrade to 4.1 (perhaps easy); and use this API http://netty.io/4.1/api/io/netty/channel/Channel.html#bytesBeforeUnwritable() to get a more reliable estimate than the blind try to send all bytes into Netty that is done now.

So to make sure I'm understanding the approach correctly. The idea is that we call bytesBeforeUnwritable to see how many bytes netty will be able to immediately send, call writeAndFlush with that many bytes and return that length to the caller immediately expecting them to call send() again if not all bytes were sent. Right?

I've got a pr up against master implementing the logic described above:

https://github.com/jythontools/jython/pull/45

Also, the netty upgrade and the fix to send() are split into separate commits for (hopefully) easier reviewing.

Fixed as of https://hg.python.org/jython/rev/f7edd144b85a, using the patch provided by Nick in the PR.

Since socket.sendall is just aliased to socket.send, this change breaks socket.sendall, which is supposed to block until all data has been sent or an error occurs.

I would imagine this will break a lot of libraries and applications that expect to not have to call socket.sendall repeatedly until all data has been sent.

@Bernie, this is a good point, but let's open as a new bug, and then figure out a good test case. When I wrote the implementation of send, and aliased sendall to it, it was not clear to me that Netty is not doing this chunking for us, but it's something that can be readily investigated.

@Jim, I'll try to come up with a standalone reproduction script (shouldn't be difficult) and open another ticket. I discovered this when PyMongo's test suite locked up when run under Jython 2.7.1. sendall definitely no longer sends all under certain circumstances.

Filed http://bugs.jython.org/issue2618