Message10426

Jeff, this is very helpful in terms of using lsof to do this assessment. I too have noticed this problem, and it is because of using sockets.

I don't believe it's a fault in the socket module per se, which should be attempting to close out all Netty resources. But a lot of the tests rely on deterministic GC. So that's why test_socket, which previously would always reliably run out of files, uses the following function:

    def _assert_no_pending_threads(self, group, msg):
        # Ensure __del__ finalizers are called on sockets. Two things to note:
        # 1. It takes two collections for finalization to run.
        # 2. gc.collect() is only advisory to the JVM, never mandatory. Still 
        #    it usually seems to happen under light load.

        # Wait up to one second for there not to be pending threads

        for i in xrange(10):
            pending_threads = _check_threadpool_for_pending_threads(group)
            if len(pending_threads) == 0:
                break
            test_support.gc_collect()
            
        if pending_threads:
            print "Pending threads in Netty msg={} pool={}".format(msg, pprint.pformat(pending_threads))
        
Not running test_socket in the usual mix may be losing out on this attempt to do aggressive GC.

My expectation is that most Python code out there does the right thing and closes out resources like sockets. However, since the socket module does not support __enter__, __exit__, and I have code in the wild that does not do such closes, it's quite possible I'm wrongly hoping that!