MPS issue job003962

TitleRegular testing omits the MMQA test suite
Assigned userGareth Rees
DescriptionContinuous integration does not compile, run, and test the MMQA test suite example. This means that we risk breaking these tests and not finding out.
Analysis1. Make it easier to run the MMQA test suite by adding a make target.

2. Add the MMQA to the Travis continuous integration? The problem here is that the MMQA takes a very long time to run (on my laptop, 83 minutes) and this means not only that we are at risk of hitting the hard Travis time limit (120 minutes) but that we don't get timely results. It would be better to schedule the MMQA tests to run less often, for example once a day, for example using [2].

3. Install Jenkins locally and schedule regular MMQA builds there?

4. Speed up MMQA. Much of the runtime is overkill (do we really learn any more after 100,000 iterations than we did after 1,000?).

Here are all the test cases that take more than 10 seconds, in reverse order by how long they take:

206 1207 Reduce iterations; systematize configurations
171 1134 Reduce iterations so there aren't so many chunks
227 504 Reduce iterations
215 326 Reduce iterations
140 206 Exponentially reduce commit limit; stop when too small
125 176 Reduce arena extensions
78 168 Reduce iterations
55 166 Reduce iterations
164 160 Exponentially reduce commit limit; stop when too small
207 143 Exponentially reduce commit limit; stop when too small
45 122 Reduce iterations
129 107 Reduce iterations
114 81 Reduce iterations
204 71 Reduce iterations
205 70 Reduce iterations
203 70 Reduce iterations
200 63 Reduce iterations
109 53 More address space to avoid running out of zones
49 52 More address space to avoid running out of zones
70 51 Interactive test, no point in running unattended
35 45 Reduce iterations
113 39 Reduce iterations
61 37 Reduce iterations
226 34 Reduce iterations
12 31 Reduce iterations
48 29 Use smaller objects
134 28 Reduce iterations
128 27 Reduce iterations
127 27 Reduce iterations
223 26 Reduce iterations
124 25 Reduce iterations
77 22 Reduce iterations
60 21 Use smaller objects
30 21 Reduce iterations
44 20 Use smaller objects
97 19 Reduce iterations
96 19 Take bigger steps
29 18 Reduce iterations
25 17 Reduce iterations
43 16 Use smaller objects
42 15 Use smaller objects
104 14 Reduce iterations
41 13 Reduce iterations & use smaller objects
36 11 Use smaller objects & turn off debugging
How foundinspection
Evidence[1] <>
[2] <>
Created byGareth Rees
Created on2016-03-06 11:24:47
Last modified byGareth Rees
Last modified on2016-03-14 11:52:03
History2016-03-06 GDR Created based on [1].


Change Effect Date User Description
190027 open 2016-03-14 11:51:35 Gareth Rees Don't run MMQA on every Travis build: it has intermittent failures.
189957 closed 2016-03-13 12:57:02 Gareth Rees Regular testing of MMQA: "make test-make-build" at top level now runs MMQA, hence so does "make test" on Linux and FreeBSD. The whole test suite, including testci, testmmqa, testansi, and testpollnone, takes 26 minutes on my laptop.
MMQA "run" and "runset" commands now exit with non-zero status if any of the tests failed, so that test failures can be identifed by the calling program (for example, make).
Add missing license to files that state "See end of file for license."
189948 open 2016-03-13 11:42:53 Gareth Rees A few more MMQA speedups.
189947 open 2016-03-13 11:42:32 Gareth Rees Speed up more MMQA test cases. On my laptop, the whole MMQA test suite (or at least the passing subset) now takes 5 minutes in the hot variety and 11 minutes in the cool variety.
189674 open 2016-03-06 14:15:37 Gareth Rees Add make target for running the MMQA test suite.