Issue2419

We'd like to have clean-running regression tests but have never (?) managed it.

We like using CPython tests by default, with necessary exclusions and variations. With those aims, we have added to regrtest.py a 'java' platform, extending the catalogue (_expectations) of tests it expects to raise unittest.SkipTest, and adding a catalogue (_failures) of tests it expects to fail.

But, the catalogues tagged simply 'java' in regrtest.py do not capture the variation in test success amongst OS platforms. In this scheme, if we want clean-running regression tests for everyone, we have to add to _failures, every test that fails on any OS platform. A recent shot in issue #2393 at getting clean regression tests on Windows ended by making these lists only correct for Windows (maybe just Cygwin). Some code in regrtest.py tries to accommodate OS variation, but it is hard to read and may be CPython-specific. This cannot be maintained reliably from any one OS.

I propose we add optional sections with keys like 'java.nt', where after the dot is taken from os._name, to catalogue those things applicable to that OS (taken from os._name). The key 'java' contains those things applicable to all Jython OSs. It is perhaps only necessary to do this to _failures. The OS-specific sections ought to be short, or something is wrong.

Good idea?

First: I really appreciate this topic gets some traction!
Your suggestion for OS-wise sections sounds good to me as a midterm solution. For short-term I would propose to more rigorously sort tests into stable vs. unstable, so that regrtests get reliable again (and quickly).
From that position we can step by step fix unstable tests or sort them into OS-specific sections, keeping regrtests reliable during the whole transition period.

I've done this now for Windows, adding a 'java.nt' key to _failures.

https://hg.python.org/jython/rev/7b1ad371113e

I don't know how we would "rigorously sort tests into stable vs. unstable". I saw test_glob fail yesterday, out of the blue, and I couldn't repeat it. Does that make it unstable? (It was just an unlink() failure, so I made that non-fatal.)

If we want repeatable tests we should err on the side of expecting as failures (or skipping in the module) those we find unreliable, but not without converting that choice to an issue.

See also CPython: https://docs.python.org/devguide/buildbots.html - mostly stable, but not always, and dependent on platform.

I think we are now in pretty good shape, with the exception of test_ssl which runs stably outside of the regrtest, but not in it. Something we can look at, maybe for 2.7.1 RC.

This is our common practice now we have the means in regrtest.py, so I'm closing as "fixed". However, it's worth saying that it is always better to skip individual failing tests than entire modules.