Message10417

We'd like to have clean-running regression tests but have never (?) managed it.

We like using CPython tests by default, with necessary exclusions and variations. With those aims, we have added to regrtest.py a 'java' platform, extending the catalogue (_expectations) of tests it expects to raise unittest.SkipTest, and adding a catalogue (_failures) of tests it expects to fail.

But, the catalogues tagged simply 'java' in regrtest.py do not capture the variation in test success amongst OS platforms. In this scheme, if we want clean-running regression tests for everyone, we have to add to _failures, every test that fails on any OS platform. A recent shot in issue #2393 at getting clean regression tests on Windows ended by making these lists only correct for Windows (maybe just Cygwin). Some code in regrtest.py tries to accommodate OS variation, but it is hard to read and may be CPython-specific. This cannot be maintained reliably from any one OS.

I propose we add optional sections with keys like 'java.nt', where after the dot is taken from os._name, to catalogue those things applicable to that OS (taken from os._name). The key 'java' contains those things applicable to all Jython OSs. It is perhaps only necessary to do this to _failures. The OS-specific sections ought to be short, or something is wrong.

Good idea?