Issue1005

1. The UDP socket implicitly created by the send/sendto methods is not
correctly configured (in relation to timeouts, etc).

2. Calling recv/recvfrom methods on an unbound UDP socket raises an
AssertionError (because a socket is not implicitly created).

See the jython-dev archives for further information.

http://www.nabble.com/UDP-Timeout-td15831206.html
http://sourceforge.net/mailarchive/forum.php?thread_name=8CA4C2CA69796CB-1590-4B7B%40FWM-M17.sysops.aol.com&forum_name=jython-dev

I've been reading up this bug some more, and realised that I've been
missing out on Bob's subtle point of implicitly created sockets being
bound to *some* address, even if not to a specific address, by virtue of
their creation; I'll refer to this address as the "assigned address".


So, what is the actual assigned address, in Java? Consider the following
code

//--------------------------------------------------------------
import java.io.IOException;

import java.nio.channels.DatagramChannel;

import java.nio.ByteBuffer;

import java.net.DatagramSocket;

import java.net.InetSocketAddress;



class udp_unbound_address

{

    public static void main ( String args[] )

        throws IOException

    {

        DatagramChannel chan = DatagramChannel.open();

        if (args.length > 0 && "sendto".compareTo(args[0]) == 0)

            chan.send(ByteBuffer.wrap("testdata".getBytes()), 

                new InetSocketAddress("www.jython.org", 8888));

        DatagramSocket sock = chan.socket();

        System.out.println("Is bound?: " + sock.isBound());

        System.out.println("Local address is: "

            + sock.getLocalAddress());

        System.out.println("Local port is: "

            + sock.getLocalPort());

        System.out.println("Local socket address is: "

            + sock.getLocalSocketAddress());

    }

}
//--------------------------------------------------------------

If you run this, you see that there is no assigned address when sockets
are created with java.nio. Here is the output from running on Unbuntu
7.10. (I get similar output on Windows Server 2003)

$ java -cp . udp_unbound_address 
Is bound?: false
Local address is: 0.0.0.0/0.0.0.0
Local port is: 0
Local socket address is: null

Making the socket do a sendto() first changes the picture slightly, in
that the socket now becomes bound, but there is still no assigned
hostname/IP or port.

$ java -cp . udp_unbound_address sendto
Is bound?: true
Local address is: 0.0.0.0/0.0.0.0
Local port is: 0
Local socket address is: 0.0.0.0/0.0.0.0:0

Note that this is different behaviour from the way old
java.net.DatagramSocket's behaved, which was to implicitly assign a port
to the socket, but not an IP address, regardless of the use of sendto.
Consider this code

//--------------------------------------------------------------
import java.io.IOException;

import java.net.DatagramPacket;

import java.net.DatagramSocket;

import java.net.InetSocketAddress;



class udp_unbound_address_net

{

    public static void main ( String args[] )

        throws IOException

    {

        DatagramSocket sock = new DatagramSocket();

        if (args.length > 0 && "sendto".compareTo(args[0]) == 0)

            sock.send(new DatagramPacket("testdata".getBytes(), 
                "testdata".getBytes().length,
                new InetSocketAddress("www.jython.org", 8888)
            ));

        System.out.println("Is bound?: " + sock.isBound());

        System.out.println("Local address is: "

            + sock.getLocalAddress());

        System.out.println("Local port is: "

            + sock.getLocalPort());

        System.out.println("Local socket address is: "

            + sock.getLocalSocketAddress());

    }

}
//--------------------------------------------------------------

Which gives the following results on Ubuntu, and similar on windows.

$ java -cp . udp_unbound_address_net
Is bound?: true
Local address is: 0.0.0.0/0.0.0.0
Local port is: 33066
Local socket address is: 0.0.0.0/0.0.0.0:33066

$ java -cp . udp_unbound_address_net sendto
Is bound?: true
Local address is: 0.0.0.0/0.0.0.0
Local port is: 33067
Local socket address is: 0.0.0.0/0.0.0.0:33067

(It is important to note that we cannot create UDP sockets using the old
java.net classes; to do so would prevent us from getting non-blocking
IO: it is only possible to do non-blocking IO on sockets created through
java.nio APIs).

Now lets look at cpython. If you try to get the address of a newly
created UDP socket, the result is an error on Windows

Python 2.4.4 (#71, Oct 18 2006, 08:34:43) [MSC v.1310 32 bit (Intel)] on
win32

>>> import socket

>>> s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

>>> s.getsockname()

Traceback (most recent call last):

  File "<stdin>", line 1, in ?

  File "<string>", line 1, in getsockname

socket.error: (10022, 'Invalid argument')

>>>

And on Unbuntu, we get the same result as Java, i.e. a null address

Python 2.5.1 (r251:54863, Oct  5 2007, 13:36:32) 
[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2
>>> import socket
>>> s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
>>> s.getsockname()
('0.0.0.0', 0)
>>> 

If we do a sendto before getting the sockname, then the situation
changes slightly on cpython

On both windows and Unbuntu, we now get a assigned port number, but
still a null machine address

Python 2.4.4 (#71, Oct 18 2006, 08:34:43) [MSC v.1310 32 bit (Intel)] on
win32

>>> import socket

>>> s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

>>> s.sendto("testdata", ("www.jython.org", 8888) )

8

>>> s.getsockname()

('0.0.0.0', 1246)

>>>

Python 2.5.1 (r251:54863, Oct  5 2007, 13:36:32) 
[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2
>>> import socket
>>> s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
>>> s.sendto("testdata", ("www.jython.org", 8888) )
8
>>> s.getsockname()
('0.0.0.0', 33063)
>>> 

So, let's try to relate that to our problem about whether or not
implicitly create a socket on unbound recvfrom, or raise an exception.
Implicitly creating a socket is only useful if there is a usable (i.e.
connectable/sendable from a different socket) assigned address, that can
somehow be publicized (other than sent through the socket itself). But
there is no usable assigned address, in java or cpython, so I see no
point in implicitly creating the socket; it can be used for nothing.

It is interesting to note that there is an outstanding bug in the Sun
database which complains that "The DatagramChannel.receive method does
not specify the behavior when the channel is not bound."


http://bugs.sun.com/view_bug.do?bug_id=6621689

It says

"""
Since 1.4, our implementation has returned null, even for the blocking
case. Throw NotNotBoundException or explicitly binding the channel's
socket might have been better choices but it cannot now be changed.
"""

So the original java.nio interpretation was different again; it
recognised the unbound state of the socket and returned null for all
receive() operations . Of the two alternatives proposed, i.e.

1. throw a NotBoundException
2. explicitly binding the channel's socket

I still favour the first, because it clearly illustrates to the user
that whatever they think they are expecting is not going to happen.
Cpython favors the second option, but you get different behaviour on
different platforms (see below).

If it were possible to get an usable assigned address for the implicitly
created socket, then it *might* be useful to implicitly create the
socket. But, as demonstrated above, there is no usable address.

It is worth noting that bug #6621689 has been accepted as a bug, and
that the fix is in progress, but no indication is given of what the
chosen fix is.

Lastly, I wanted to point out that there is different behaviour from
cpython on different platforms when a recvfrom is attempted on an
unbound socket.

Windows raises an error

c:\Python24>python

Python 2.4.4 (#71, Oct 18 2006, 08:34:43) [MSC v.1310 32 bit (Intel)] on
win32

>>> import socket

>>> s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

>>> s.recvfrom(1024)

Traceback (most recent call last):

  File "<stdin>", line 1, in ?

socket.error: (10022, 'Invalid argument')


Whereas Ubuntu hangs on the same code

Python 2.5.1 (r251:54863, Oct  5 2007, 13:36:32) 
[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
>>> result = s.recvfrom(1024)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyboardInterrupt

In Bob's original example, setting a timeout seems to have modified this
behaviour, so that both platforms run the code and timeout, i.e.

Python 2.4.4 (#71, Oct 18 2006, 08:34:43) [MSC v.1310 32 bit (Intel)] on
win32

Type "help", "copyright", "credits" or "license" for more information.

>>> import socket

>>> s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

>>> s.settimeout(1.0)

>>> s.recvfrom(1024)

Traceback (most recent call last):

  File "<stdin>", line 1, in ?

socket.timeout: timed out

>>>

Python 2.5.1 (r251:54863, Oct  5 2007, 13:36:32) 
[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
>>> s.settimeout(1.0)
>>> s.recvfrom(1024)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
socket.timeout: timed out

So, in summary, this whole area seems to be poorly understand, and
varyingly implemented.

Cpython gives different behaviour across different platforms, but gives
consistent behaviour for Bob's original code.

Java gives consistent behaviour on all platforms. 

But if we adopt the stance of "implicit create sockets for unbound
recvfrom()s", then there is nothing can be done with the resulting
socket; on the java platform anyway. (although perhaps the fix to bug
#6621689 might change this).

I need to think about this over the weekend.

All opinions welcome.

Alan.

[Alan]
So, in summary, this whole area seems to be poorly understand, and
varyingly implemented.

Truer words were never spoken.

Excellent analysis Alan.  From my view, the question is whether you can
you do a receive on an 'unbound' socket that has a timeout set.  I also
understand that with the history of c and unix and the way Java has been
  implemented, the design of a Jython solution is not easy.

I think the Python implementation is reasonable since if you don't set a
timeout and then do a recv on an 'unbound' port, the timeout defaults to
None which means the program will hang for ever and so it throws an
error.  But, if you set a timeout, it allows you to do the recv and then
throws an exception when the timer kicks off, and then the program
continues.    


Which means the following is really a 1 second delay;

import socket
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.settimeout(1.0)
result = s.recvfrom(1024)

I understand your point that the recvfrom will never actually receive a
packet and so it should be an error.  But consider the following:

i = 5
j = 10
print i

j is never used so is that an error?  I don't know but in a language
like Python or Jython, I would expect it to run.

So:

----------------------------
import socket

i = 5

s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

result = s.recvfrom(1024)

print i
-------------------

The print i statement will never be run, so I think it is reasonable
here to throw an exception on the recvfrom.

But, if you add the settimeout, then:

----------------------------
import socket

i = 5

s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s = settimeout(1.0)
result = s.recvfrom(1024)

print i
----------------

This will/should delay for a second then print '5' and then terminate
normally. 

I would expect that this is the way a program would work in Python or
Jython. 

Now, when you do a sendto, then:

----------------------------
import socket

s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.sendto("testdata", ('192.168.1.200', 10000))

result = s.recvfrom(1024)
--------------------------------------

In this case,whether or not a timeout is set, if there is a program
receiving at 192.168.1.200 port 10000, it will receive the "testdata"
packet and the hostname and port of our program that is attached to s.
The script doesn't really care what the number of the port is that the
OS attached to s just so we get our data when packets arrive back at
that port.

Finally, consider a multi threaded case. (the actual threaded scripts
have not been tested) 

----------------------------
import socket

s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.settimeout(2.0)

Have one thread that has access to s do the following:

while(True): 
    try:
        result = s.recvfrom(1024)
    except timeout, to:
        pass

And another thread that shares s do this:

------------------------------------------

while (True):

    time.sleep(10.0)

    s.sendto("testdata", ("192.168.1.200", 10000))

------------------------------------------------

It seems reasonable to me to expect that to work.

I honestly don't know which way is 'right',  but I think you are in for
a lot of push back if you make this change.

Let me know if you want more of my input.  And have a good weekend.

Hi Alan,

Well I've done some homework today too, and I have found some
interesting things.  First, the IETF standard for UDP, RFC 768 is still
in effect.  It is dated August 1980.  Yes, Jimmy Carter was the US
President and I was programming on a Control Data main frame with FORTRAN!

Anyway, it is literally less than 3 pages long and you can read it in
less than 5 minutes.  The lines that mainly concern us are these:


User Interface
--------------

A user interface should allow

  the creation of new receive ports,

  receive  operations  on the receive  ports that return the data octets
  and an indication of source port and source address,

  and an operation  that allows  a datagram  to be sent,  specifying the
  data, source and destination ports and addresses to be sent.
-----

That's it.  User interface in this case means the application above UDP.

I also have finally bought and begun to read:

UNIX Network Programming Volume 1, Third Edition: The Sockets Networking API
By 	W. Richard Stevens; Bill Fenner; Andrew M. Rudoff

It is a bit slow going as you may expect, but I can already recommend it
to you.  It talks about the history and the common problems that have
developed.

It is interesting that he calls what you just started calling assigned
ports, ephemeral ports.  I like your name better. He also talks about
the history of why UDP does not notify the application when it receives
the ICMP message saying that there is no server on the port.  It also
talks about host addresses.  That there are situations where the local
address can change with with every send to the same server.  That looks
like it may explain some of the issues you were finding.     

For our case, you already fixed this situation:

import socket

s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.sendto('testdata', ('192.168.1.103', 375))

buf, addr = s.recvfrom(2048)

If the server is up and it responds with a packet back, then this script
should read the packet and then the script should terminate.  Otherwise,
it should hang forever.  But, one thing that I wasn't thinking about was
this case:

import socket

s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.sendto('testdata', ('192.168.1.103', 375))
s.sendto('testdata', ('192.168.1.104', 375))  # <- different host
s.sendto('testdata', ('192.168.1.104', 3750))  # <- different port

buf, addr = s.recvfrom(2048)

Multiple sends to multiple server through the UDP port. 
This should respond just like the previous one.  If it gets one packet
from any of these servers, it will read it and terminate, otherwise it
will hang forever.  Interestingly, your code handles this, but CPython
does not!  It is clearly a bug in CPython.  I haven't tried Java yet and
I am also setting up to try the equivalent C programs.

For the other case, when there is a recv with no send, I am coming
closer and closer to your opinion, that an error should be generated
rather than allowing the read.  I guess my main concern was that
existing CPython code that was running may break.  But, maybe that is a
good thing.  What I am going to do is run the equivalent in c of the
following two examples and see how c handles it.

import socket

s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

buf, addr = s.recvfrom(2048)


and

import socket

s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.settimout(2.0)
buf, addr = s.recvfrom(2048)

I will let you know the results.

Thanks for all your research Bob, it has been most enlightening.

I have checked in some changes, namely

1. The simple fix for the obvious bug relating to failure to configure
the socket implicitly created by _udpsocket.sendto()

2. Cpython compatible behaviour for the recvfrom() method, in the case
where the socket is unbound.

In relation to the latter bug, I have checked in both options in the
code, but commented out my favoured approach, which is to raise an
exception.

Depending on further research results, the outcome of the Sun bug
relating to DatagramChannel.receive, and community input, we may yet
choose the exception route.

I'm leaving this bug open until we make a final decision.

I think that for the recv/recvfrom issue that this is a very reasonable
solution.  Jython 2.2.2 will test this even more.

I still think there is problem with the send/sendto timeout.  (I will
open a separate problem if you like, just let me know).   


----------------------
import socket

s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

print s.gettimeout()

s.settimeout(1.0)    # settimeout here done before the sendto

print s.gettimeout()

s.sendto('testdata', ('192.168.1.103', 9876))

print s.gettimeout()

print 'The program hangs here, even though s.gettimeout = 1.0'

buf, addr = s.recvfrom(2048)

---------------------------------------------

When the settimeout is performed after the sendto, it works properly.

I suspect that when config is run, that it sets the timeout to default
which is None, but the value of timeout in for the script is still 1.0.   

(By the way, because of this testing, I will be opening a problem with
Python 2.5.2 on Windows).

[Bob]
> I still think there is problem with the send/sendto timeout.
> (I will open a separate problem if you like, just let me know).

Are you sure about that?

Your code works just fine for me, on windows at least. (And I see no
reason it should operate differently on other platforms).

Here is my version, with some added print statements

#-=-=-=-=-=-=-=
import socket

timeout = 2.0

s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

print "Before timeout set, timeout is %s" % s.gettimeout()
s.settimeout(timeout)    # settimeout here done before the sendto
print "After timeout set, timeout is %s" % s.gettimeout()

s.sendto('testdata', ('192.168.1.103', 9876))
print "After socket created, timeout is %s" % s.gettimeout()

print "Under the covers, jsocket timeout is %d milliseconds" %
s._sock.sock_impl.jsocket.getSoTimeout()

print 'The program hangs here, even though s.gettimeout = %1.1lf' % timeout

buf, addr = s.recvfrom(2048)
#-=-=-=-=-=-=-=

Which gives the following output
#-------------------------------------
Before timeout set, timeout is None
After timeout set, timeout is 2.0
After socket created, timeout is 2.0
Under the covers, jsocket timeout is 2000 milliseconds
The program hangs here, even though s.gettimeout = 2.0
Traceback (innermost last):
  File "bob_luse_20080709.py", line 18, in ?
  File "C:\jython\release_22_maint\Lib\socket.py", line 785, in recvfrom
timeout: timed out
#-------------------------------------

Are you sure that you have the latest version of the code?

I would expect to see the behaviour you describe *only* if the new
one-line addition from revision 4194 to the sendto method has not been
made, i.e. line 753 should read "self._config()"; if it does not, you
have an old version.

My apologies.  I thought I had your last change in my test, but I did
not.  It now works perfectly forme a s well.  Thanks.

On re-reading the Sun java bug report listed below, I see that the
proposed resolution is to behave as cpython does, i.e. to implicitly
open the socket.

Since implicit opening is what both cpython and java programmers will
expect, we should adopt that resolution as well.

The code to implement it is already checked in, so I'm closing this bug.