The entry point to our testing suite is a Makefile that we have crafted with several options and rules to add flexibility and make your life easier. This section summarizes the different options and rules that area available to you.
The structure of the makefile command goes as follow:
make CC=ccompiler CXX=cppcompiler FC=fortrancompiler [OPTIONS] [RULE]
If no CC and CXX compilers are specified, gcc will be used by default, as this compiler is widely available in most of the systems. However, this assumes that gcc has openMP and offloading support
Depending on the rule you are applying to the make command, each option applies some modification. Some rules will change the compilation process, and some other will change how the tests are ran. Finally some rules depends on the system environment you are running on.
This is the list and a short description of all available options:
| Option | Description |
|---|---|
| VERBOSE=1 | Controls the verbosity of the make process. Enabling the output of the commands executed during the make process. |
| VERBOSE_TESTS=1 | Controls the verbosity of the tests outputs. By default tests only output PASS/FAIL. But this option adds extra information to the output. Enables extra information display in the tests |
| LOG=1 | Create the logs folder containing the log files with a defined format described in the reports section. |
| LOG_ALL=1 | Enables dump of the make process output, errors, and binary execution outputs into the log files. Implies LOG=1 |
| SYSTEM=sys_name | Includes the definitions for the requires modules and batch schedulers in the different systems. This definitions must be in sys/SYSTEM.def. (Do not include the .def extension). You can create new SYSTEMS following the instructions of the systems section |
| DEVICE_TYPE=dev_name | Specifies device type being used, either nvidia or amd, to change compiler flags as needed |
| ADD_BATCH_SCHED=1 | Add the batch scheduler command before the compilation or execution of the tests. This requires a SYSTEM.def file definition, and the use of the SYSTEM flag. |
| MODULE_LOAD=1 | Some SYSTEMS use environmental variables to enable different compiler options or necessary libraries. This option allows the execution of module load before each tests (consider that make will create a new environment for each command it executes, this could solve issues with environmental variables not being transferred to this new environment). If your modules software requires unloading or swapping modules manually, this command will not work. |
| NO_OFFLOADING=1 | (Support for this feature is still weak) Turn off offloading in the tests. Most of the tests will report if they were executed in the device or the hosts. This will remove/force the execution of the offloading reagions in the hosts. |
| SOURCES=file.(c,cpp,F90) | Specify the source file(s) that you want to apply the rule to. rule could be compile, run or all. Wildcards can be use to select many |
| OMP_VERSION=[e.g. 4.5] | This specifies which version of the specs we are targeting. This version should have its folder in tests/[version]. By default value is 4.5 WARNING: WHEN CHANGING VERSIONS START FROM A CLEAN BUILD. OTHERWISE BINARIES and LOG folders may collide. |
The difference between VERBOSE and VERBOSE_TESTS is that the later changes a binary that makes the tests themselves more verbose. This flag will include print statements that will spill extra information about the tests. The former changes the verbosity during the compilation process.
LOG enables the creation of logs that will be output in the logs folder. LOG_ALL changes how much information goes to these logs. If the output of the make process is placed in the logfiles or not.
SOURCES is used in the compilation process, it will select which tests to compile, compile and run, or only run. NO_OFFLOADING will change the compiler flags to either not include or disable the compilation for the target device (note: depending on the compiler, you have to explicitely disable offloading, or not include a flag to disable it).
MODULE_LOAD and ADD_BATCH_SCHED are two flags that depend on the sys/SYSTEM.def file. If no SYSTEM is used, then these flags will have no effect. To set up these flags for custom systems check out our Creating new systems section.
The different make rules allows to select what operation you want to perform. Run, compile the code, or compile results in different formats.
| Option | Explanation |
|---|---|
| compile | Compile tests using the CC and CXX compilers. If SOURCES option is specified, compile just the selected files, otherwise compile all the available tests. |
| run | run tests that have been previously compiled. If SOURCES option is specified, only run those tests which sources are the ones, otherwise run all OpenMP tests that are available within bin/ directory |
| all | Compile and run tests (applies both options above.) |
| clean | Remove all previously compiled tests from bin/ directory |
| tidy | Clean all previously compiled tests, as well as previous logs and result report. |
| compilers | Shows available compilers configuration (Not customizable yet) |
| report_json | This flag requires that tests have been compiled and run with the LOG option prior to its use. It will take the raw log files from the logs/ folder and it will create a json file containing the results. Please refer to reports for more information. |
| report_summary | This flag requires that tests have been compiled and run with the LOG option prior to its use. This will show a quick summary of the test that failed in the logs/ directory, and the reason of failure. Please refer to reports for more information. |
| report_csv | This rule requires that tests have been compiled and run with the LOG option prior to its use. It will take the raw log files from the logs/ folder and it will create a csv file containing the results. Please refer to reports for more information. |
| report_html | This flag requires that tests have been compiled and run with the LOG option prior to its use. It will take the raw log files from the logs/ folder and it will create an html based results report. This rule takes the json file and a pre-build HTML template, and it creates the results_report folder containing the report. This report allows filtering the results by system, compiler, and pass/fail result. For more information refer to reports |
| report_online | This flag requires that tests have been compiled and run with the LOG option prior to its use. It will take the raw log files from the logs/ folder and it will create an json based results report, then it will submit this report to our website for visualization purposes. This rule will output a link to your results visualization. It will create a tag that you can use to track down your results. For more information refer to reports |
Some simple examples and their explanation are shown here.
| Command | What it does |
|---|---|
| make CC=gcc CXX=g++ FC=gfortran all | ==> compile and run all test cases with GCC |
| make CC=gcc SOURCES=tests/a.c all | ==> compile and run tests/a.c with gcc |
| make CXX=g++ SOURCES=tests/target/* all | ==> compile and run all tests under tests/target with the default compilers |
| make CC=xlc CXX=xlc++ FC=xlf compile | ==> compile all test cases with IBM”s XL compiler |
| make run | ==> run all the cases that exist inside bin/ |
| make SOURCES=bin/myTestSourceFile run | ==> run myTestSourceFile if previously compiled |
| make CC=gcc SYSTEM=summit DEVICE_TYPE=nvidia SOURCES=tests/offloading_success.c VERBOSE=1 VERBOSE_TESTS=1 ADD_BATCH_SCHED=1 MODULE_LOAD=1 all | ==> include the sys/summit.def file which contains the modules and batch scheduler configuration parameters. Runs with offloading using Nvidia GPU. Compiles and run tests/offloading_success.c. Verbose make output and verbose tests. Adds modules and batch scheduler |
We do not support using multiple compilers for the same language with the same run. However, when evaluating a system, it is commont to want to evaluate it with multiple compilers. This is an example on how to run all the tests with all the compilers at once using a bash command. This example is for summitdev, but it can be easily changed, and you will need to change the values of the Ccomp, CPPcomp arrays to the compilers you desire. However, the pairs (e.g. clang and clang++) must match position of each array. The value of SYSTEM= to match the one under sys/SYSTEM.def
Ccomp=(clang xlc gcc); CPPcomp=(clang++ xlc++ g++);\
for (( i=0; i<${#Ccomp[@]}; i++)); \
do make clean; make CC=${Ccomp[$i]} CXX=${CPPcomp[$i]} \
SYSTEM=summitdev VERBOSE=1 LOG=1 LOG_ALL=1\
ADD_BATCH_SCHED=1 VERBOSE_TESTS=1 MODULE_LOAD=1 all;\
done;
Here is another example for summit:
Ccomp=(clang xlc); CPPcomp=(clang++ xlc++);\
for (( i=0; i<${#Ccomp[@]}; i++)); \
do make clean; make CC=${Ccomp[$i]} CXX=${CPPcomp[$i]} \
SYSTEM=summit VERBOSE=1 LOG=1 LOG_ALL=1\
ADD_BATCH_SCHED=1 VERBOSE_TESTS=1 MODULE_LOAD=1 all;\
done;
Here we list current issues, or unsupported features. If you find something else, please feel free to create a ticket in our Github issue tracking system.
target if tests requires the device to be availabe)