Issue1782548

classification
Title: UDP send blocks with reader thread present
Type: Severity: normal
Components: Library Versions: 2.2.2
Milestone:
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: amak Nosy List: amak, rluse, wfouche
Priority: urgent Keywords:

Created on 2007-08-27.14:51:30 by wfouche, last changed 2008-02-28.19:00:03 by amak.

Files
File name Uploaded Description Edit Remove
udp_test.py wfouche, 2007-08-27.14:51:30 UDP test application
JavaTestUDPNIO.java amak, 2008-01-04.07:23:05
Messages
msg1864 (view) Author: wfouche (wfouche) Date: 2007-08-27.14:51:30
I've tried both Jython v2.2 and v2.2rc3 and have discovered that one of my UDP based applications deadlocks. The attached test program works in Python v2.4 and Jython v2.1 (using SUN Java v1.6.0_02), but deadlocks with Jython v2.2 (using SUN Java v1.6.0_02). All testing being done on Windows XP SP2.

Bug report: The second last line in the program (UDP send) never unblocks when Jython v2.2 is used.

This is what the program does. It creates a UDP socket, and binds port 5555 to it. A reader thread is then spawned to receive UDP packets sent to this port. The main application then continues to send a UDP packet originating from this port every second once the reader thread is running. If the reader thread is not instantiated, then Jython v2.2 is able to send packets every minute; otherwise the UDP send never unblocks.
msg1865 (view) Author: Bob Luse (rluse) Date: 2007-12-24.00:47:29
This does uncover an error handling problem in Jython, but I wouldn't say this is 'working' in Python.  If you add the except socket.error clause to the read loop(as I've done below), you'll see in Python that the read loop is blocking waiting for a receive in the thread, and then when the same socket tries to send from another place, Python gets an error and prints the following:

!!
(10054, 'Connection reset by peer')
   done.
!!
(10054, 'Connection reset by peer')
   done.
!!
(10054, 'Connection reset by peer')
   done.
''
''
''


Jython is blocking at the receive in the thread in the class, and then when the same socket tries to send from another location, the program hangs and doesn't raise the socket.error.  Jython does need to address the socket.error.  I remember there were similar problems with error handling in 2.2 for sockets.    I did try to write the same code in Java, but I wasn't even able to write it with just one socket reading in a thread while the same socket is writing from another part of the same program at the same time.  All you need to do, though, to make this work is to create another socket for your send.

Charlie, et all, I will attempt to have Jython throw a socket.error in this case, but I may need some guidance.

The updated script is as follows,  I don't know how to add an attachment to a comment, though.  If it doesn't come out well, I will send it to the dev forum.


import socket
import time
import threading

class Reader(threading.Thread):

    def __init__(this, udp_sock):
        threading.Thread.__init__(this)
        this.udp_sock = udp_sock
        this.running = 0
        
    def run(this):
        this.running = 1
        while this.running:
            try:
                text, addr = this.udp_sock.recvfrom(8000)
                print text
            except socket.error, e:
                print e
           
   
udp_sock=socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
udp_sock.bind(("", 5555))

reader = Reader(udp_sock)
reader.start()
while not reader.running:
    time.sleep(1.0)

while 1:
    time.sleep(1.0)
    print "!!"
    udp_sock.sendto("hi", ('localhost', 5556))
    print "   done."

msg1866 (view) Author: Bob Luse (rluse) Date: 2007-12-24.10:29:44
Okay, I did write what I think you are expecting in Java, and it does give the results that you want, I believe.  I am putting that program on the bottom of this comment.  

It is interesting that if you put a try except around the write on the Jython script and set blocking to False, both Python and Jython produce similar, (but wrong) results.  I think now that the problem is beyond simple error handling and I do not know the Socket.py code well enough.  I am willing to work on it, but I will need input from Alan or Mr. Envey ot someone else with more expertise than myself.

To me, its an unusual solution to require a UDP write from a bound UDP port.  However, there is no reason that I can think of why it should not be okay and obviously it works cleanly in Java, if not in Python.  It looks like the Python guys missed it too.  The documentation is vague on this.  Anyway, here is the Java code:

import java.net.*;
import java.io.*;

public class JavaTestUDP extends Thread {

	DatagramSocket socket;
	byte[] buffer = new byte[8000];
	DatagramPacket packet = new DatagramPacket(buffer, buffer.length);

	public JavaTestUDP(DatagramSocket socket) {
		this.socket = socket;
	}

	public void run() {

		while (true) {
			try {
				Thread.sleep(2000);
				System.out.println("Before socket receive");
				socket.receive(packet);
			} catch (Exception e) {
				System.out.println(e);
			}
			String s = new String(packet.getData(), 0, packet.getLength());
			System.out.println(packet.getAddress() + " at port "
					+ packet.getPort() + " says " + s);
			packet.setLength(buffer.length);
		}
	}

	public static void main(String[] args) throws SocketException {

		DatagramSocket socket = new DatagramSocket(5555);

		Thread t = new JavaTestUDP(socket);
		t.start();

		while (true) {
			try {
				Thread.sleep(1000);
			} catch (InterruptedException e) {
			}
			String sendString = "Hi";
			byte[] data = sendString.getBytes();
			try {
				InetAddress server = InetAddress.getByName("localhost");
				DatagramPacket sendPacket = new DatagramPacket(
                                                     data, data.length, server, 5556);
				System.out.println("!!");
				socket.send(sendPacket);
			} catch (UnknownHostException e) {
				System.out.println(e);
			}
			catch (IOException e) {
				System.out.println(e);
			}
			System.out.println("   done");
		}
	}
}
msg1867 (view) Author: Alan Kennedy (amak) Date: 2008-01-04.07:20:37
Hi,

I'll be posting notes about this bug in another message, but first just wanted to point out that the example code is a little confusing.

The UDP socket is created and bound to port 5555, but messages are sent to port 5556. Which means that, in the code given at least, nothing is reading the messages sent. Is that your intention?

I will assume for now that this is indeed your intention, and that the example code is a cut down version of a larger piece of code that perhaps uses a pair of sockets for a 2-way conversation between port 5555 and port 5556.

Notes to follow.

Alan.
msg1868 (view) Author: Alan Kennedy (amak) Date: 2008-01-04.07:23:06
The problem is that the UDP support in the updated socket module has not been fully updated to use java.nio.

In the update of the socket module to support asynch sockets with java.nio, the creation of UDP sockets was updated to use the NIO API for creating an UDP socket, through the use of the DatagramChannel.open() method.

But the reading and writing of packets from/to the UDP socket still uses the old DatagramSocket API. Which in this case sees to cause a hang, when it appears that it shouldn't.

To illustrate the problem, I've rewritten Bob's java code, and atached it to this bug.

My version reads the command line arguments, looking for options, of which there are two.

By default, if no options are given, the code

1. Creates the UDP socket using the DatagramSocket API, which is the old java.net API.
2. Reads all messages through the DatagramSocket.receive() method.

This combination is analogous to the old jython (pre-asynch) socket module, and works as the OP expects, correctly I believe.

If the "create_nio" option is given on the command line

1. The UDP socket is created using the java.nio DatagramChannel.open() method 
2. All messages are read using the "old" java.net DatagramSocket.receive method()

This combination is analogous to the current status of the socket module, i.e. it hangs when two threads read and write on the socket at the same time.

If the "create_nio" and "read_nio" options are both given on the command line

1. The UDP socket is created using the java.nio DatagramChannel.open() method 
2. All messages are read using the "new" java.nio DatagramChannel.receive method()

This combination is analogous to the way the socket module *should* (and will) be. It uses only java.nio APIs, and exhibits the expected behaviour, i.e. it does not hang.

So the solution to this bug is to update the implementation of UDP sockets to use java.nio APIs for send and receive.

I will post an outline implementation on this bug report in a few days.

But I will not get to update the socket module, i.e. update the SVN repo, update the test suites, etc, for a week or so.

Alan.



File Added: JavaTestUDPNIO.java
msg1869 (view) Author: Bob Luse (rluse) Date: 2008-01-04.07:47:57
Hi,

I'll be posting notes about this bug in another message, but first just
wanted to point out that the example code is a little confusing.

The UDP socket is created and bound to port 5555, but messages are sent to
port 5556. Which means that, in the code given at least, nothing is reading
the messages sent. Is that your intention?

I will assume for now that this is indeed your intention, and that the
example code is a cut down version of a larger piece of code that perhaps
uses a pair of sockets for a 2-way conversation between port 5555 and port
5556.

Hi Alan,

I can't speak for the originator of this problem, but when I reduced his code to something that I understood, (the problem as I see it anyway), happens when the UDP socket is bound and listening for messages and then sends to any other socket.  Indeed a port will probably not send a message to itself.  In my tests, it worked just fine in Java, marginally worked in Python, and completely hung in Jython.  The example given is strange, but it seems to me that it should work, regardless.  So, as far as I am concerned, your assumption is correct.   

 - Bob
msg1870 (view) Author: Alan Kennedy (amak) Date: 2008-01-14.20:48:20
I've checked a fix for this bug into trunk, at revision 4028.

I may check it on the 2.2 maintenance branch as well.

I won't close the bug for now though.

msg3052 (view) Author: Alan Kennedy (amak) Date: 2008-02-27.13:04:41
Checked a fix for this issue into release_2_2_maint, at revision 4184;
backported from trunk. This issue should now be closed.
History
Date User Action Args
2008-02-28 19:00:03amaksetstatus: open -> closed
resolution: accepted -> fixed
versions: + 2.2.2
2008-02-27 13:04:42amaksetresolution: accepted
messages: + msg3052
2007-08-27 14:51:30wfouchecreate