How to run the MMQA system.
This short document is a 'howto' that shows you how to run the MMQA system, to do various tests on a set of MPS header files and libraries.
Readership: MPS developers. Confidential: no. Status: rough notes. Author: RHSK.
Background:
For a fuller introduction to MMQA:
For a look behind the curtain at what MMQA is doing:
NOTES
Each MMQA 'test' is simply a statement about the MPS, which may or may not be true for a given version of the MPS. It is not the case that all MMQA tests are expected to pass. Some have never passed. Some, perhaps, should never pass!
SUMMARY
>>> cd project/mps/master
>>> cd code
>>> make -f xcppgc.gmk
>>> cd ../test
>>> perl test/qa -i ../code -l ../code/xcppgc/ci/mps.a -nomail clib
>>> perl test/qa -i ../code -l ../code/xcppgc/ci/mps.a -nomail run function/80.c
DETAIL
>>> cd project/mps/master
# Make the MPS libraries, or a 'release', first:
>>> cd code
>>> make -f xcppgc.gmk # etc, eg. w3build.bat
# Now you can test it:
>>> cd ../test
>>> ls
argerr function test
conerr misc testsets
>>> perl test/qa help
Help topics:
clib compile test libraries
index list tests in a directory
install installing and invoking the MMQA test harness
naming Numbering of test programs
options display options and their current values
output Conventions for test output
problems known problems and how to report others you may find
run run tests, reporting results and verdict
runset run tests from a test-set
For help on a topic, type 'qa help '.
For general instructions on how to run tests, see guide.mmqa in
the MM Information database.
>>> perl test/qa options
------------------------------------------------------------------------
MMQA harness version 3.5
OPTION CODE CURRENT
DANGEROUS -[no]danger off
DATA_DIR -data test/../data
DEBUG_INFO -[no]debug off
LOG_DIR -g test/log
MAIL_RESULTS -[no]mail on
MAIL_SUBJECT -subj MMQA-log
MAIL_TO -m mm-qa-test-log
MPS_INCLUDE_DIR -i
MPS_LINK_OBJ -l
MPS_PRODUCT -prod
PARAMETERS -a
PLATFORM -p Darwin_8.3.0_powerpc__unix
STDIN -s
(Add MMQA_ to start of option to give environment variable)
------------------------------------------------------------------------
# Assemble your options, and use the "options" command again to check that your syntax is correct:
>>> perl test/qa -i ../code -l ../code/xcppgc/ci/mps.a -nomail options
# Command "clib" tells MMQA to inspect MPS headers and library, and build the test harness libraries:
>>> perl test/qa -i ../code -l ../code/xcppgc/ci/mps.a -nomail clib
Cleaning out old object files.
Checking settings.
Scrutinizing MPS header files.
../code/mpswin.h:19:21: error: windows.h: No such file or directory
../code/mpsw3.h:14:59: error: windows.h: No such file or directory
Compiling test libraries.
# Use the "options" command again; MPS_PRODUCT should now show the value "mps":
>>> perl test/qa -i ../code -l ../code/xcppgc/ci/mps.a -nomail options
------------------------------------------------------------------------
MMQA harness version 3.5
OPTION CODE CURRENT
DANGEROUS -[no]danger off
DATA_DIR -data test/../data
DEBUG_INFO -[no]debug off
LOG_DIR -g test/log
MAIL_RESULTS -[no]mail off
MAIL_SUBJECT -subj MMQA-log
MAIL_TO -m mm-qa-test-log
MPS_INCLUDE_DIR -i ../code
MPS_LINK_OBJ -l ../code/xcppgc/ci/mps.a
MPS_PRODUCT -prod mps
PARAMETERS -a
PLATFORM -p Darwin_8.3.0_powerpc__unix
STDIN -s
(Add MMQA_ to start of option to give environment variable)
------------------------------------------------------------------------
# Use "index" to list function/argerr/conerr/misc tests:
>>> perl test/qa -i ../code -l ../code/xcppgc/ci/mps.a -nomail index function
0.c: test that the mps header file is accepted by the compiler
1.c: create a space and then destroy it
2.c: test my format for format_a
3.c: allocate 100 items, throw away and repeat
4.c: allocate 100 items, throw away and repeat
5.c: (regression test) have an ambiguous reference to a reserved but not committed object, and then allocate lots more with another AP, to make it collect
...
...
226.c: test of mps_ld_merge
227.c: allocate in 2 arenas
500.txt: compare MM and boehm performance in Dylan
501.txt: MM works with Dylan
502.txt: MM works with Dylan debugger
503.txt: Compare variety.hi/he and variety.wi performance
504.txt: check release notes
# Use "run" to run a test:
>>> perl test/qa -i ../code -l ../code/xcppgc/ci/mps.a -nomail run function/80.c
% create arena: OK
% create table root: OK
% chain_create: OK
% create ap: OK
% create ap: OK
% 1 of 100.
% 2 of 100.
% 3 of 100.
...
...
% 99 of 100.
% Destroyed arena.
!completed=yes
-----------------------------------------------------------------------------
PASSED TEST function/80.c
*********|10s******|**
harness = 1.0
id = $HopeName: MMQA_test_function!80.c(trunk.3) $
language = c
link = testlib.o fastfmt.o
summary = AMC and itself performance
SPECIFIED RESULTS
completed = yes
MMQA OPTIONS
MPS_INCLUDE_DIR = ../code
MPS_LINK_OBJ = ../code/xcppgc/ci/mps.a
MPS_PRODUCT = mps
PLATFORM = Darwin_8.3.0_powerpc__unix
RESULTS
completed = yes
seconds = 36
CONCLUSION
PASS
# If you like, inspect the output, and the log:
>>> ls test/log/
last.log
B. DOCUMENT HISTORY
2005-11-30 RHSK Created.
C. COPYRIGHT AND LICENSE
Copyright (C) 2005 Ravenbrook Limited .
All rights reserved. This is an open source license. Contact
Ravenbrook for commercial licensing options.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
3. Redistributions in any form must be accompanied by information on how
to obtain complete source code for this software and any
accompanying software that uses this software. The source code must
either be included in the distribution or be available for no more than
the cost of distribution plus a nominal fee, and must be freely
redistributable under reasonable conditions. For an executable file,
complete source code means the source code for all modules it contains.
It does not include source code for modules or files that typically
accompany the major components of the operating system on which the
executable file runs.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
PURPOSE, OR NON-INFRINGEMENT, ARE DISCLAIMED. IN NO EVENT SHALL THE
COPYRIGHT HOLDERS AND CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
$Id: //info.ravenbrook.com/project/mps/doc/2005-11-30/mmqa-study/mmqa-howto.txt#1 $