Issue2530

Scenario: I use Jython to call the popular feedparser.py, which will sometimes go into an endless processing loop for some malformed news feeds

Current fix: I have to write code to "kill -9" the Java process and restart it every 10 minutes so that endless Jython execution threads don't block all available execution threads

Proposed fix: Implement some way to notify Jython to abort execution of whatever python it is running

Thanks!

Alternate fix: Implement a timeout parameter which instructs jython to abort execution after a certain amount of time has passed

Just to be clear: SIGTERM does not work for you?

Hi Jim, I didn't mean to imply that 'kill -9' was the only way to terminate the Java process from the command line. I'm sure I could have used SIGTERM instead of SIGKILL.

The issue is that where there is a PythonInterpreter.exec() in progress in a Java thread, there is no way for any other Java thread to halt execution in that thread.

Therefore, as it currently stands, the only way to stop a PythonInterpreter.exec() call which refuses to return is to kill the entire Java process on the command line, killing not only the long running PythonInterpreter.exec() but all other Java threads too.

What I'm suggesting is some kind of PythonInterpreter.abort() method so that out of control infinite loop python scripts being executed via PythonInterpreter.exec() can be stopped.

Without a feature like this, Jython is basically unusable in production if any non returning PythonInterpreter.exec() calls start to pile up.

Presumably we can implement PythonInterpreter#abort using a similar approach to org.python.util.jython#shutdownInterpreter, but to set the restart flag for a specific PySystemState, not the current state as defined by a thread local.

    public static void shutdownInterpreter() {
        // Stop all the active threads and signal the SystemRestart
        thread.interruptAllThreads();
        Py.getSystemState()._systemRestart = true;
        // Close all sockets -- not all of their operations are stopped by
        // Thread.interrupt (in particular pre-nio sockets)
        try {
            imp.load("socket").__findattr__("_closeActiveSockets").__call__();
        } catch (PyException pye) {
            // continue
        }
    }

Note that we should be able to do a better cleanup in  as of Jython 2.7 and recent Java - assuming well behaved resources.

Jim, thank you so much for your responses. I did not realise until you pointed it out that it was possible to interrupt a PythonInterpreter.exec() by simply doing a Thread.interrupt().

I was not aware of shutdownInterpreter(), and it seems to be the solution I was looking for. In fact, I didn't even realise from the Jython docs that I could use Thread.interrupt() to abort execution.

Thank you very much for helping me understand how to resolve this problem now. So although I know to fix my issue, a way to shutdown reset the PythonInterpreter rather than simply shut it down is still very welcome.

FYI, for anyone else coming across this page, here is the code that I didn't realise until now that I could have written to abort PythonInterpreter.exec():

PythonInterpreter pi = new PythonInterpreter();

    Thread jythonThread = new Thread(new Runnable() {
      @Override
      public void run() {
        pi.exec("while 1: print 'hello'");
      }
    });

    jythonThread.start();

    new Thread(new Runnable() {
      @Override
      public void run() {
        try {
          Thread.sleep(2000);
          jythonThread.interrupt();
        }
        catch (InterruptedException e) {
          e.printStackTrace();
        }
      }
    }).start();

Thread interruption from Python itself is also discussed here:
http://www.jython.org/jythonbook/en/1.0/Concurrency.html#interruption

We need to update the book to discuss functionality that has since become part of Jython 2.7.

This should also attempt a cleanup of the corresponding PySystemState in terms of any corresponding threads and files (including sockets).

It is possible that #abort should be on PySystemState itself, instead of the PythonInterpreter.

This is way worse a situation that I'd thought - I'd assumed so far that I could just call thread.interrupt and abort any PythonInterpreter.exec() and then just create new PythonInterpreters and carry on.

This is not the case - after a thread.interrupt(), the whole VM is broken and cannot exec any more python, even if new PythonInterpreters are instantiated!!

Sample code that demonstrates the problem:

    PythonInterpreter pi = new PythonInterpreter();

    Thread jythonThread = new Thread(() -> {
      try {
        pi.exec("while 1: print 'hello'");
      }
      catch (org.python.core.PyException e) {
        PythonInterpreter pi2 = new PythonInterpreter();
        pi2.exec("print 'hello again'"); // THROWS a org.python.core.PyException - WHY?!
      }
    });

    jythonThread.setName("jythonThread");
    jythonThread.start();

    new Thread(() -> {
      try {
        Thread.sleep(500);
        jythonThread.interrupt();
      }
      catch (InterruptedException e) {
        e.printStackTrace();
      }
    }).start();

For what it is worth: I had a similar problem and needed a possibility to abort execution of a Python program. I used the trace-function to achieve my goal, basically:
  Py.getThreadState().tracefunc = myTraceFunction;
"myTraceFunction" is an instance of a class extending "org.python.core.TraceFunction".

This way, before executing a line, I have Jython check if an "abort"-flag is set. If so, a simple exception is thrown, causing the interpreter to actually abort and report the exception -- which is, of course, caught by my code. I have never had any problems restarting another Python script in the same JVM afterwards (I am not talking about freeing all the resources here, just the basic concept).

If there was such a mechanism already built in, that would be a great help, though.