BigDFT build tree offers the possibility to have non-regression tests. When the user (or the devloper) types ``make check`` in the distribution, a number of tests are passed. Then a the ``fldiff_yaml.py`` script is checking that the logfiles of each run coincides with the reference results that are provided for each test. Ideally, each functionality of the code should have its own test, to ensure that further developments do not break it. It is therefore responsibility of the developer to provide new non-regression tests for each of the functionality. The optimal test should be - Reasonably fast (roughly less than 1 min on a monocore run) - Reasonably exhaustive (if the test passes this means that the functionality is indeed preserved - no false positive) Clearly, these two requirements contradict each other. The **art** of the developer is to find a good compromise to reassure: 1) the other developers not to slow down the ``make check`` execution, and 2) himself, to be sure that new developments still would preserve the functionality. Having said that, imagine that we have a set of files that we would like to use for non regression. There are different steps to include them in the test farm of BigDFT. As an example, we show the insertion of the test for the molecular dynamics in BigDFT. Prepare the directory --------------------- The tests should go in the ``tests`` directory (rather unexpectedly... ;)). The procedure described here is valid for four subdirectories: ``cubic``, ``linear``, ``overDFT`` and ``tutorials``. All the other directories have special Makefiles associated and have a less easy way to automatize the test procedure. As our examples are related to MD, we put our tests in the ``overDFT`` directory. We create the directory ``MD``: | ``cd tests/overDFT`` | ``bzr mkdir MD`` Once this is done, all the test files inside this directory should correspond to a particular naming scheme. For example, a test named ``foo`` have to be associated to two reference files: named ``MD_foo.out.ref``, containing the standard output (which essentially only contains one line with the name of the logfile), and ``MD_foo.out.ref.yaml`` which contains the lofgile associated to the test ``foo``. Clearly, also the input files ``foo.yaml`` and ``foo.xyz`` (or any other supported format have to be present, as if it was a normal running directory. THe presence of the two reference files will cause the check procedure to run ``$run_parallel ``\ \ ``/src/bigdft foo -l yes`` where ``run_parallel`` is the environment variable containing the desired command to run tests. These tests will be ran in the directory ``tests/overDFT/MD-test/`` of the **build** tree, to preserve untouched the input files and the references. The logfile of the run will be written in the file ``log-foo.yaml`` of the same directory. Let us imagine we would like to insert two tests, with codename ``LJ-NVE`` and ``LJ-NVT``. According to the above instruction, we therefore have to add the following files: | ``> ls MD/`` | ``> LJ-NVE.xyz  LJ-NVE.yaml LJ-NVT.xyz LJ-NVT.yaml  MD_LJ-NVE.out.ref  MD_LJ-NVE.out.ref.yaml  MD_LJ-NVT.out.ref  MD_LJ-NVT.out.ref.yaml  psppar.LJ`` The directory might contain also extra files, that are needed for the run, in this case ``psppar.LJ``. We should not froget to also add these files to the development branch: | ``> bzr add tests/overDFT/MD`` | ``adding tests/overDFT/MD/LJ-NVE.xyz`` | ``adding tests/overDFT/MD/LJ-NVE.yaml`` | ``adding tests/overDFT/MD/LJ-NVT.xyz`` | ``adding tests/overDFT/MD/LJ-NVT.yaml`` | ``adding tests/overDFT/MD/MD_LJ-NVE.out.ref`` | ``adding tests/overDFT/MD/MD_LJ-NVE.out.ref.yaml`` | ``adding tests/overDFT/MD/MD_LJ-NVT.out.ref`` | ``adding tests/overDFT/MD/MD_LJ-NVT.out.ref.yaml`` Insert the test in the ``make check`` command --------------------------------------------- Even if the directory is ready, the test is still not active. To activate it in the ``make check`` procedure, the developer has to tell the build system that ``MD`` is a valid test directory. To do that, it is enough to insert the name of the directory (``MD`` in our example) in the ``Makefile.am`` file of the ``tests/overDFT`` directory: .. code:: make # Give here the list of existing tests SHORT_TESTDIRS = FF-LENOSKY \ FF-LJ \ H2O-slab \ H2-freq \ MD #<<<< add it here... [...] LONG_TESTDIRS = $(SHORT_TESTDIRS) \ C6H6-freq \ GEOPT-all \ GEOPT-BFGS \ GEOPT-LBFGS \ [...] #<<< ...or here As it can be seen, the ``MD`` directory can be added either in the ``SHORT_TESTDIRS`` or in the ``LONG_TESTDIRS`` variable, but not in the two. Whether to choose one or another is a matter of the importance/robustness of the test: the tests in ``SHORT_TESTDIRS`` are ran by anybody typing ``make check`` on the released version whereas the others are only run by developers and expert users. Therefore a test in ``SHORT_TESTDIRS`` shoudl be.. short, and also should fail only when there are major instabilities in the code. Running the tests and prepare tolerances ---------------------------------------- Even though an output file is prepared, we cannot expect the output of the test to be bitwise identical to the provided output. It is therefore responsibility of the 'test forger' to provide a set of tolerances for any of the fields of the output file (which for this reason is in markup language). This require some running of the tests to have an idea of the correct tolerances. To run the test the first time, let us go to the **build** tree (not in the source tree and execute a check for the ``MD`` test: | ``> cd ``\ \ ``/tests/overDFT`` | ``> make MD.check`` This last command will, in the order: - Create the test directory ``MD-test`` in the build tree. - Copy the corresponding input files in such a directory. - Run the ``bigdft`` executable in the folder, with the command ``$run_parallel ``\ \ ``/src/bigdft ``\ \ `` -l yes`` where is each of the test runs (``NVE`` and ``NVT`` in the examples defined above). - Once the test is finished, the ``fldiff`` scripts are ran to compare the logfiles of the output witrh the given references. As all the log of the run are written in markup language, the different fields of the logfile might be compared with different tolerances. Such information is stored in the ``tests/tols-BigDFT.yaml`` file. Such file might be modified to adapt the tolerance of each field to the nature of the run.