| Title | Hard to tell which test case failed |
| Status | closed |
| Priority | optional |
| Assigned user | Gareth Rees |
| Organization | Ravenbrook |
| Description | When you run "make test" and the test suite fails, it's hard to tell which test case was the one that failed. For example, on Ubuntu (platform lii6gc) I just ran "make test" and it output nearly 3000 lines of test output followed by a failure. The last few lines of the output looked like this: Collection 77, 188375 objects. Collection 78, 190521 objects. Collection 79, 192662 objects. commit_limit_set before: 1 make[1]: *** [testrun] Error 1 make[1]: Leaving directory `/home/gdr/info.ravenbrook.com/project/mps/master/code' make: *** [test] Error 2 It is hard to tell from this which test case this was. Even looking backward through the output, there is no mention at the start of each test case's output of which test case it is. (By grepping through the code for "commit_limit_set before" I found that the test case must have been mpsicv, but this shouldn't be necessary. I reported the test failure as job003433.) |
| Analysis | I suggest the following improvements: 1. "make test" should report the name of the test case before running it, so that when you look back through the test output you can see which test case failed. 2. "make test" should collect the standard output from all the tests to a log file somewhere in /tmp, so that this does not clutter the user's terminal, and so that nothing is lost if the output exceeds the terminal's scrollback. 3. Each test case should print its success message ("Conclusion: Failed to find any defects.") to standard output (not standard error) so that these messages do no clutter up the terminal when running "make test". 4. Each test case should use its result code (not the printed message) to indicate whether it succeeded or failed. (They nearly all did.) 5. More of the diagnostic messages from the test cases should start by printing argv[0] so that it is easier to tell which test case was running. |
| How found | manual_test |
| Evidence | None |
| Observed in | 1.110.0 |
| Created by | Gareth Rees |
| Created on | 2013-03-07 10:48:48 |
| Last modified by | Gareth Rees |
| Last modified on | 2013-03-07 15:55:15 |
| History | 2013-03-07 GDR Created. |
| Change | Effect | Date | User | Description |
|---|---|---|---|---|
| 181071 | closed | 2013-03-07 13:13:32 | Gareth Rees | Improve the usability of the test suite: 1. "make test" now reports the name of the test case before running it, so that when you look back through the test output you can see which test case failed. 2. "make test" now collects the standard output from all the tests to a log file in /tmp, so that this does not clutter the user's terminal, and so that nothing is lost if the output exceeds the terminal's scrollback. 3. Each test case now prints a success message ("Conclusion: Failed to find any defects.") to standard output (not standard error) so that these messages do not clutter up the terminal when running "make test". 4. Each test case now uses its result code (not the printed message) to indicate whether it succeeded or failed. 5. More of the diagnostic messages from the test cases now start by printing argv[0] so that it is easier to tell which test case was running. |