To run user-mode emulation tests, we introduced the avocado_qemu.QemuUserTest which inherits from avocado_qemu.QemuBaseTest. System-mode emulation tests are based on the avocado_qemu.Test class, which also inherits avocado_qemu.QemuBaseTest. To avoid confusion, rename it as avocado_qemu.QemuSystemTest. Suggested-by: Wainer dos Santos Moschetta <wainersm@redhat.com> Reviewed-by: Willian Rampazzo <willianr@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Message-Id: <20211105143416.148332-7-f4bug@amsat.org>
1310 lines
44 KiB
ReStructuredText
1310 lines
44 KiB
ReStructuredText
Testing in QEMU
|
||
===============
|
||
|
||
This document describes the testing infrastructure in QEMU.
|
||
|
||
Testing with "make check"
|
||
-------------------------
|
||
|
||
The "make check" testing family includes most of the C based tests in QEMU. For
|
||
a quick help, run ``make check-help`` from the source tree.
|
||
|
||
The usual way to run these tests is:
|
||
|
||
.. code::
|
||
|
||
make check
|
||
|
||
which includes QAPI schema tests, unit tests, QTests and some iotests.
|
||
Different sub-types of "make check" tests will be explained below.
|
||
|
||
Before running tests, it is best to build QEMU programs first. Some tests
|
||
expect the executables to exist and will fail with obscure messages if they
|
||
cannot find them.
|
||
|
||
Unit tests
|
||
~~~~~~~~~~
|
||
|
||
Unit tests, which can be invoked with ``make check-unit``, are simple C tests
|
||
that typically link to individual QEMU object files and exercise them by
|
||
calling exported functions.
|
||
|
||
If you are writing new code in QEMU, consider adding a unit test, especially
|
||
for utility modules that are relatively stateless or have few dependencies. To
|
||
add a new unit test:
|
||
|
||
1. Create a new source file. For example, ``tests/unit/foo-test.c``.
|
||
|
||
2. Write the test. Normally you would include the header file which exports
|
||
the module API, then verify the interface behaves as expected from your
|
||
test. The test code should be organized with the glib testing framework.
|
||
Copying and modifying an existing test is usually a good idea.
|
||
|
||
3. Add the test to ``tests/unit/meson.build``. The unit tests are listed in a
|
||
dictionary called ``tests``. The values are any additional sources and
|
||
dependencies to be linked with the test. For a simple test whose source
|
||
is in ``tests/unit/foo-test.c``, it is enough to add an entry like::
|
||
|
||
{
|
||
...
|
||
'foo-test': [],
|
||
...
|
||
}
|
||
|
||
Since unit tests don't require environment variables, the simplest way to debug
|
||
a unit test failure is often directly invoking it or even running it under
|
||
``gdb``. However there can still be differences in behavior between ``make``
|
||
invocations and your manual run, due to ``$MALLOC_PERTURB_`` environment
|
||
variable (which affects memory reclamation and catches invalid pointers better)
|
||
and gtester options. If necessary, you can run
|
||
|
||
.. code::
|
||
|
||
make check-unit V=1
|
||
|
||
and copy the actual command line which executes the unit test, then run
|
||
it from the command line.
|
||
|
||
QTest
|
||
~~~~~
|
||
|
||
QTest is a device emulation testing framework. It can be very useful to test
|
||
device models; it could also control certain aspects of QEMU (such as virtual
|
||
clock stepping), with a special purpose "qtest" protocol. Refer to
|
||
:doc:`qtest` for more details.
|
||
|
||
QTest cases can be executed with
|
||
|
||
.. code::
|
||
|
||
make check-qtest
|
||
|
||
QAPI schema tests
|
||
~~~~~~~~~~~~~~~~~
|
||
|
||
The QAPI schema tests validate the QAPI parser used by QMP, by feeding
|
||
predefined input to the parser and comparing the result with the reference
|
||
output.
|
||
|
||
The input/output data is managed under the ``tests/qapi-schema`` directory.
|
||
Each test case includes four files that have a common base name:
|
||
|
||
* ``${casename}.json`` - the file contains the JSON input for feeding the
|
||
parser
|
||
* ``${casename}.out`` - the file contains the expected stdout from the parser
|
||
* ``${casename}.err`` - the file contains the expected stderr from the parser
|
||
* ``${casename}.exit`` - the expected error code
|
||
|
||
Consider adding a new QAPI schema test when you are making a change on the QAPI
|
||
parser (either fixing a bug or extending/modifying the syntax). To do this:
|
||
|
||
1. Add four files for the new case as explained above. For example:
|
||
|
||
``$EDITOR tests/qapi-schema/foo.{json,out,err,exit}``.
|
||
|
||
2. Add the new test in ``tests/Makefile.include``. For example:
|
||
|
||
``qapi-schema += foo.json``
|
||
|
||
check-block
|
||
~~~~~~~~~~~
|
||
|
||
``make check-block`` runs a subset of the block layer iotests (the tests that
|
||
are in the "auto" group).
|
||
See the "QEMU iotests" section below for more information.
|
||
|
||
QEMU iotests
|
||
------------
|
||
|
||
QEMU iotests, under the directory ``tests/qemu-iotests``, is the testing
|
||
framework widely used to test block layer related features. It is higher level
|
||
than "make check" tests and 99% of the code is written in bash or Python
|
||
scripts. The testing success criteria is golden output comparison, and the
|
||
test files are named with numbers.
|
||
|
||
To run iotests, make sure QEMU is built successfully, then switch to the
|
||
``tests/qemu-iotests`` directory under the build directory, and run ``./check``
|
||
with desired arguments from there.
|
||
|
||
By default, "raw" format and "file" protocol is used; all tests will be
|
||
executed, except the unsupported ones. You can override the format and protocol
|
||
with arguments:
|
||
|
||
.. code::
|
||
|
||
# test with qcow2 format
|
||
./check -qcow2
|
||
# or test a different protocol
|
||
./check -nbd
|
||
|
||
It's also possible to list test numbers explicitly:
|
||
|
||
.. code::
|
||
|
||
# run selected cases with qcow2 format
|
||
./check -qcow2 001 030 153
|
||
|
||
Cache mode can be selected with the "-c" option, which may help reveal bugs
|
||
that are specific to certain cache mode.
|
||
|
||
More options are supported by the ``./check`` script, run ``./check -h`` for
|
||
help.
|
||
|
||
Writing a new test case
|
||
~~~~~~~~~~~~~~~~~~~~~~~
|
||
|
||
Consider writing a tests case when you are making any changes to the block
|
||
layer. An iotest case is usually the choice for that. There are already many
|
||
test cases, so it is possible that extending one of them may achieve the goal
|
||
and save the boilerplate to create one. (Unfortunately, there isn't a 100%
|
||
reliable way to find a related one out of hundreds of tests. One approach is
|
||
using ``git grep``.)
|
||
|
||
Usually an iotest case consists of two files. One is an executable that
|
||
produces output to stdout and stderr, the other is the expected reference
|
||
output. They are given the same number in file names. E.g. Test script ``055``
|
||
and reference output ``055.out``.
|
||
|
||
In rare cases, when outputs differ between cache mode ``none`` and others, a
|
||
``.out.nocache`` file is added. In other cases, when outputs differ between
|
||
image formats, more than one ``.out`` files are created ending with the
|
||
respective format names, e.g. ``178.out.qcow2`` and ``178.out.raw``.
|
||
|
||
There isn't a hard rule about how to write a test script, but a new test is
|
||
usually a (copy and) modification of an existing case. There are a few
|
||
commonly used ways to create a test:
|
||
|
||
* A Bash script. It will make use of several environmental variables related
|
||
to the testing procedure, and could source a group of ``common.*`` libraries
|
||
for some common helper routines.
|
||
|
||
* A Python unittest script. Import ``iotests`` and create a subclass of
|
||
``iotests.QMPTestCase``, then call ``iotests.main`` method. The downside of
|
||
this approach is that the output is too scarce, and the script is considered
|
||
harder to debug.
|
||
|
||
* A simple Python script without using unittest module. This could also import
|
||
``iotests`` for launching QEMU and utilities etc, but it doesn't inherit
|
||
from ``iotests.QMPTestCase`` therefore doesn't use the Python unittest
|
||
execution. This is a combination of 1 and 2.
|
||
|
||
Pick the language per your preference since both Bash and Python have
|
||
comparable library support for invoking and interacting with QEMU programs. If
|
||
you opt for Python, it is strongly recommended to write Python 3 compatible
|
||
code.
|
||
|
||
Both Python and Bash frameworks in iotests provide helpers to manage test
|
||
images. They can be used to create and clean up images under the test
|
||
directory. If no I/O or any protocol specific feature is needed, it is often
|
||
more convenient to use the pseudo block driver, ``null-co://``, as the test
|
||
image, which doesn't require image creation or cleaning up. Avoid system-wide
|
||
devices or files whenever possible, such as ``/dev/null`` or ``/dev/zero``.
|
||
Otherwise, image locking implications have to be considered. For example,
|
||
another application on the host may have locked the file, possibly leading to a
|
||
test failure. If using such devices are explicitly desired, consider adding
|
||
``locking=off`` option to disable image locking.
|
||
|
||
Debugging a test case
|
||
~~~~~~~~~~~~~~~~~~~~~
|
||
|
||
The following options to the ``check`` script can be useful when debugging
|
||
a failing test:
|
||
|
||
* ``-gdb`` wraps every QEMU invocation in a ``gdbserver``, which waits for a
|
||
connection from a gdb client. The options given to ``gdbserver`` (e.g. the
|
||
address on which to listen for connections) are taken from the ``$GDB_OPTIONS``
|
||
environment variable. By default (if ``$GDB_OPTIONS`` is empty), it listens on
|
||
``localhost:12345``.
|
||
It is possible to connect to it for example with
|
||
``gdb -iex "target remote $addr"``, where ``$addr`` is the address
|
||
``gdbserver`` listens on.
|
||
If the ``-gdb`` option is not used, ``$GDB_OPTIONS`` is ignored,
|
||
regardless of whether it is set or not.
|
||
|
||
* ``-valgrind`` attaches a valgrind instance to QEMU. If it detects
|
||
warnings, it will print and save the log in
|
||
``$TEST_DIR/<valgrind_pid>.valgrind``.
|
||
The final command line will be ``valgrind --log-file=$TEST_DIR/
|
||
<valgrind_pid>.valgrind --error-exitcode=99 $QEMU ...``
|
||
|
||
* ``-d`` (debug) just increases the logging verbosity, showing
|
||
for example the QMP commands and answers.
|
||
|
||
* ``-p`` (print) redirects QEMU’s stdout and stderr to the test output,
|
||
instead of saving it into a log file in
|
||
``$TEST_DIR/qemu-machine-<random_string>``.
|
||
|
||
Test case groups
|
||
~~~~~~~~~~~~~~~~
|
||
|
||
"Tests may belong to one or more test groups, which are defined in the form
|
||
of a comment in the test source file. By convention, test groups are listed
|
||
in the second line of the test file, after the "#!/..." line, like this:
|
||
|
||
.. code::
|
||
|
||
#!/usr/bin/env python3
|
||
# group: auto quick
|
||
#
|
||
...
|
||
|
||
Another way of defining groups is creating the tests/qemu-iotests/group.local
|
||
file. This should be used only for downstream (this file should never appear
|
||
in upstream). This file may be used for defining some downstream test groups
|
||
or for temporarily disabling tests, like this:
|
||
|
||
.. code::
|
||
|
||
# groups for some company downstream process
|
||
#
|
||
# ci - tests to run on build
|
||
# down - our downstream tests, not for upstream
|
||
#
|
||
# Format of each line is:
|
||
# TEST_NAME TEST_GROUP [TEST_GROUP ]...
|
||
|
||
013 ci
|
||
210 disabled
|
||
215 disabled
|
||
our-ugly-workaround-test down ci
|
||
|
||
Note that the following group names have a special meaning:
|
||
|
||
- quick: Tests in this group should finish within a few seconds.
|
||
|
||
- auto: Tests in this group are used during "make check" and should be
|
||
runnable in any case. That means they should run with every QEMU binary
|
||
(also non-x86), with every QEMU configuration (i.e. must not fail if
|
||
an optional feature is not compiled in - but reporting a "skip" is ok),
|
||
work at least with the qcow2 file format, work with all kind of host
|
||
filesystems and users (e.g. "nobody" or "root") and must not take too
|
||
much memory and disk space (since CI pipelines tend to fail otherwise).
|
||
|
||
- disabled: Tests in this group are disabled and ignored by check.
|
||
|
||
.. _container-ref:
|
||
|
||
Container based tests
|
||
---------------------
|
||
|
||
Introduction
|
||
~~~~~~~~~~~~
|
||
|
||
The container testing framework in QEMU utilizes public images to
|
||
build and test QEMU in predefined and widely accessible Linux
|
||
environments. This makes it possible to expand the test coverage
|
||
across distros, toolchain flavors and library versions. The support
|
||
was originally written for Docker although we also support Podman as
|
||
an alternative container runtime. Although the many of the target
|
||
names and scripts are prefixed with "docker" the system will
|
||
automatically run on whichever is configured.
|
||
|
||
The container images are also used to augment the generation of tests
|
||
for testing TCG. See :ref:`checktcg-ref` for more details.
|
||
|
||
Docker Prerequisites
|
||
~~~~~~~~~~~~~~~~~~~~
|
||
|
||
Install "docker" with the system package manager and start the Docker service
|
||
on your development machine, then make sure you have the privilege to run
|
||
Docker commands. Typically it means setting up passwordless ``sudo docker``
|
||
command or login as root. For example:
|
||
|
||
.. code::
|
||
|
||
$ sudo yum install docker
|
||
$ # or `apt-get install docker` for Ubuntu, etc.
|
||
$ sudo systemctl start docker
|
||
$ sudo docker ps
|
||
|
||
The last command should print an empty table, to verify the system is ready.
|
||
|
||
An alternative method to set up permissions is by adding the current user to
|
||
"docker" group and making the docker daemon socket file (by default
|
||
``/var/run/docker.sock``) accessible to the group:
|
||
|
||
.. code::
|
||
|
||
$ sudo groupadd docker
|
||
$ sudo usermod $USER -a -G docker
|
||
$ sudo chown :docker /var/run/docker.sock
|
||
|
||
Note that any one of above configurations makes it possible for the user to
|
||
exploit the whole host with Docker bind mounting or other privileged
|
||
operations. So only do it on development machines.
|
||
|
||
Podman Prerequisites
|
||
~~~~~~~~~~~~~~~~~~~~
|
||
|
||
Install "podman" with the system package manager.
|
||
|
||
.. code::
|
||
|
||
$ sudo dnf install podman
|
||
$ podman ps
|
||
|
||
The last command should print an empty table, to verify the system is ready.
|
||
|
||
Quickstart
|
||
~~~~~~~~~~
|
||
|
||
From source tree, type ``make docker-help`` to see the help. Testing
|
||
can be started without configuring or building QEMU (``configure`` and
|
||
``make`` are done in the container, with parameters defined by the
|
||
make target):
|
||
|
||
.. code::
|
||
|
||
make docker-test-build@centos8
|
||
|
||
This will create a container instance using the ``centos8`` image (the image
|
||
is downloaded and initialized automatically), in which the ``test-build`` job
|
||
is executed.
|
||
|
||
Registry
|
||
~~~~~~~~
|
||
|
||
The QEMU project has a container registry hosted by GitLab at
|
||
``registry.gitlab.com/qemu-project/qemu`` which will automatically be
|
||
used to pull in pre-built layers. This avoids unnecessary strain on
|
||
the distro archives created by multiple developers running the same
|
||
container build steps over and over again. This can be overridden
|
||
locally by using the ``NOCACHE`` build option:
|
||
|
||
.. code::
|
||
|
||
make docker-image-debian10 NOCACHE=1
|
||
|
||
Images
|
||
~~~~~~
|
||
|
||
Along with many other images, the ``centos8`` image is defined in a Dockerfile
|
||
in ``tests/docker/dockerfiles/``, called ``centos8.docker``. ``make docker-help``
|
||
command will list all the available images.
|
||
|
||
To add a new image, simply create a new ``.docker`` file under the
|
||
``tests/docker/dockerfiles/`` directory.
|
||
|
||
A ``.pre`` script can be added beside the ``.docker`` file, which will be
|
||
executed before building the image under the build context directory. This is
|
||
mainly used to do necessary host side setup. One such setup is ``binfmt_misc``,
|
||
for example, to make qemu-user powered cross build containers work.
|
||
|
||
Tests
|
||
~~~~~
|
||
|
||
Different tests are added to cover various configurations to build and test
|
||
QEMU. Docker tests are the executables under ``tests/docker`` named
|
||
``test-*``. They are typically shell scripts and are built on top of a shell
|
||
library, ``tests/docker/common.rc``, which provides helpers to find the QEMU
|
||
source and build it.
|
||
|
||
The full list of tests is printed in the ``make docker-help`` help.
|
||
|
||
Debugging a Docker test failure
|
||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||
|
||
When CI tasks, maintainers or yourself report a Docker test failure, follow the
|
||
below steps to debug it:
|
||
|
||
1. Locally reproduce the failure with the reported command line. E.g. run
|
||
``make docker-test-mingw@fedora J=8``.
|
||
2. Add "V=1" to the command line, try again, to see the verbose output.
|
||
3. Further add "DEBUG=1" to the command line. This will pause in a shell prompt
|
||
in the container right before testing starts. You could either manually
|
||
build QEMU and run tests from there, or press Ctrl-D to let the Docker
|
||
testing continue.
|
||
4. If you press Ctrl-D, the same building and testing procedure will begin, and
|
||
will hopefully run into the error again. After that, you will be dropped to
|
||
the prompt for debug.
|
||
|
||
Options
|
||
~~~~~~~
|
||
|
||
Various options can be used to affect how Docker tests are done. The full
|
||
list is in the ``make docker`` help text. The frequently used ones are:
|
||
|
||
* ``V=1``: the same as in top level ``make``. It will be propagated to the
|
||
container and enable verbose output.
|
||
* ``J=$N``: the number of parallel tasks in make commands in the container,
|
||
similar to the ``-j $N`` option in top level ``make``. (The ``-j`` option in
|
||
top level ``make`` will not be propagated into the container.)
|
||
* ``DEBUG=1``: enables debug. See the previous "Debugging a Docker test
|
||
failure" section.
|
||
|
||
Thread Sanitizer
|
||
----------------
|
||
|
||
Thread Sanitizer (TSan) is a tool which can detect data races. QEMU supports
|
||
building and testing with this tool.
|
||
|
||
For more information on TSan:
|
||
|
||
https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual
|
||
|
||
Thread Sanitizer in Docker
|
||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||
TSan is currently supported in the ubuntu2004 docker.
|
||
|
||
The test-tsan test will build using TSan and then run make check.
|
||
|
||
.. code::
|
||
|
||
make docker-test-tsan@ubuntu2004
|
||
|
||
TSan warnings under docker are placed in files located at build/tsan/.
|
||
|
||
We recommend using DEBUG=1 to allow launching the test from inside the docker,
|
||
and to allow review of the warnings generated by TSan.
|
||
|
||
Building and Testing with TSan
|
||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||
|
||
It is possible to build and test with TSan, with a few additional steps.
|
||
These steps are normally done automatically in the docker.
|
||
|
||
There is a one time patch needed in clang-9 or clang-10 at this time:
|
||
|
||
.. code::
|
||
|
||
sed -i 's/^const/static const/g' \
|
||
/usr/lib/llvm-10/lib/clang/10.0.0/include/sanitizer/tsan_interface.h
|
||
|
||
To configure the build for TSan:
|
||
|
||
.. code::
|
||
|
||
../configure --enable-tsan --cc=clang-10 --cxx=clang++-10 \
|
||
--disable-werror --extra-cflags="-O0"
|
||
|
||
The runtime behavior of TSAN is controlled by the TSAN_OPTIONS environment
|
||
variable.
|
||
|
||
More information on the TSAN_OPTIONS can be found here:
|
||
|
||
https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags
|
||
|
||
For example:
|
||
|
||
.. code::
|
||
|
||
export TSAN_OPTIONS=suppressions=<path to qemu>/tests/tsan/suppressions.tsan \
|
||
detect_deadlocks=false history_size=7 exitcode=0 \
|
||
log_path=<build path>/tsan/tsan_warning
|
||
|
||
The above exitcode=0 has TSan continue without error if any warnings are found.
|
||
This allows for running the test and then checking the warnings afterwards.
|
||
If you want TSan to stop and exit with error on warnings, use exitcode=66.
|
||
|
||
TSan Suppressions
|
||
~~~~~~~~~~~~~~~~~
|
||
Keep in mind that for any data race warning, although there might be a data race
|
||
detected by TSan, there might be no actual bug here. TSan provides several
|
||
different mechanisms for suppressing warnings. In general it is recommended
|
||
to fix the code if possible to eliminate the data race rather than suppress
|
||
the warning.
|
||
|
||
A few important files for suppressing warnings are:
|
||
|
||
tests/tsan/suppressions.tsan - Has TSan warnings we wish to suppress at runtime.
|
||
The comment on each suppression will typically indicate why we are
|
||
suppressing it. More information on the file format can be found here:
|
||
|
||
https://github.com/google/sanitizers/wiki/ThreadSanitizerSuppressions
|
||
|
||
tests/tsan/blacklist.tsan - Has TSan warnings we wish to disable
|
||
at compile time for test or debug.
|
||
Add flags to configure to enable:
|
||
|
||
"--extra-cflags=-fsanitize-blacklist=<src path>/tests/tsan/blacklist.tsan"
|
||
|
||
More information on the file format can be found here under "Blacklist Format":
|
||
|
||
https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags
|
||
|
||
TSan Annotations
|
||
~~~~~~~~~~~~~~~~
|
||
include/qemu/tsan.h defines annotations. See this file for more descriptions
|
||
of the annotations themselves. Annotations can be used to suppress
|
||
TSan warnings or give TSan more information so that it can detect proper
|
||
relationships between accesses of data.
|
||
|
||
Annotation examples can be found here:
|
||
|
||
https://github.com/llvm/llvm-project/tree/master/compiler-rt/test/tsan/
|
||
|
||
Good files to start with are: annotate_happens_before.cpp and ignore_race.cpp
|
||
|
||
The full set of annotations can be found here:
|
||
|
||
https://github.com/llvm/llvm-project/blob/master/compiler-rt/lib/tsan/rtl/tsan_interface_ann.cpp
|
||
|
||
VM testing
|
||
----------
|
||
|
||
This test suite contains scripts that bootstrap various guest images that have
|
||
necessary packages to build QEMU. The basic usage is documented in ``Makefile``
|
||
help which is displayed with ``make vm-help``.
|
||
|
||
Quickstart
|
||
~~~~~~~~~~
|
||
|
||
Run ``make vm-help`` to list available make targets. Invoke a specific make
|
||
command to run build test in an image. For example, ``make vm-build-freebsd``
|
||
will build the source tree in the FreeBSD image. The command can be executed
|
||
from either the source tree or the build dir; if the former, ``./configure`` is
|
||
not needed. The command will then generate the test image in ``./tests/vm/``
|
||
under the working directory.
|
||
|
||
Note: images created by the scripts accept a well-known RSA key pair for SSH
|
||
access, so they SHOULD NOT be exposed to external interfaces if you are
|
||
concerned about attackers taking control of the guest and potentially
|
||
exploiting a QEMU security bug to compromise the host.
|
||
|
||
QEMU binaries
|
||
~~~~~~~~~~~~~
|
||
|
||
By default, qemu-system-x86_64 is searched in $PATH to run the guest. If there
|
||
isn't one, or if it is older than 2.10, the test won't work. In this case,
|
||
provide the QEMU binary in env var: ``QEMU=/path/to/qemu-2.10+``.
|
||
|
||
Likewise the path to qemu-img can be set in QEMU_IMG environment variable.
|
||
|
||
Make jobs
|
||
~~~~~~~~~
|
||
|
||
The ``-j$X`` option in the make command line is not propagated into the VM,
|
||
specify ``J=$X`` to control the make jobs in the guest.
|
||
|
||
Debugging
|
||
~~~~~~~~~
|
||
|
||
Add ``DEBUG=1`` and/or ``V=1`` to the make command to allow interactive
|
||
debugging and verbose output. If this is not enough, see the next section.
|
||
``V=1`` will be propagated down into the make jobs in the guest.
|
||
|
||
Manual invocation
|
||
~~~~~~~~~~~~~~~~~
|
||
|
||
Each guest script is an executable script with the same command line options.
|
||
For example to work with the netbsd guest, use ``$QEMU_SRC/tests/vm/netbsd``:
|
||
|
||
.. code::
|
||
|
||
$ cd $QEMU_SRC/tests/vm
|
||
|
||
# To bootstrap the image
|
||
$ ./netbsd --build-image --image /var/tmp/netbsd.img
|
||
<...>
|
||
|
||
# To run an arbitrary command in guest (the output will not be echoed unless
|
||
# --debug is added)
|
||
$ ./netbsd --debug --image /var/tmp/netbsd.img uname -a
|
||
|
||
# To build QEMU in guest
|
||
$ ./netbsd --debug --image /var/tmp/netbsd.img --build-qemu $QEMU_SRC
|
||
|
||
# To get to an interactive shell
|
||
$ ./netbsd --interactive --image /var/tmp/netbsd.img sh
|
||
|
||
Adding new guests
|
||
~~~~~~~~~~~~~~~~~
|
||
|
||
Please look at existing guest scripts for how to add new guests.
|
||
|
||
Most importantly, create a subclass of BaseVM and implement ``build_image()``
|
||
method and define ``BUILD_SCRIPT``, then finally call ``basevm.main()`` from
|
||
the script's ``main()``.
|
||
|
||
* Usually in ``build_image()``, a template image is downloaded from a
|
||
predefined URL. ``BaseVM._download_with_cache()`` takes care of the cache and
|
||
the checksum, so consider using it.
|
||
|
||
* Once the image is downloaded, users, SSH server and QEMU build deps should
|
||
be set up:
|
||
|
||
- Root password set to ``BaseVM.ROOT_PASS``
|
||
- User ``BaseVM.GUEST_USER`` is created, and password set to
|
||
``BaseVM.GUEST_PASS``
|
||
- SSH service is enabled and started on boot,
|
||
``$QEMU_SRC/tests/keys/id_rsa.pub`` is added to ssh's ``authorized_keys``
|
||
file of both root and the normal user
|
||
- DHCP client service is enabled and started on boot, so that it can
|
||
automatically configure the virtio-net-pci NIC and communicate with QEMU
|
||
user net (10.0.2.2)
|
||
- Necessary packages are installed to untar the source tarball and build
|
||
QEMU
|
||
|
||
* Write a proper ``BUILD_SCRIPT`` template, which should be a shell script that
|
||
untars a raw virtio-blk block device, which is the tarball data blob of the
|
||
QEMU source tree, then configure/build it. Running "make check" is also
|
||
recommended.
|
||
|
||
Image fuzzer testing
|
||
--------------------
|
||
|
||
An image fuzzer was added to exercise format drivers. Currently only qcow2 is
|
||
supported. To start the fuzzer, run
|
||
|
||
.. code::
|
||
|
||
tests/image-fuzzer/runner.py -c '[["qemu-img", "info", "$test_img"]]' /tmp/test qcow2
|
||
|
||
Alternatively, some command different from "qemu-img info" can be tested, by
|
||
changing the ``-c`` option.
|
||
|
||
Integration tests using the Avocado Framework
|
||
---------------------------------------------
|
||
|
||
The ``tests/avocado`` directory hosts integration tests. They're usually
|
||
higher level tests, and may interact with external resources and with
|
||
various guest operating systems.
|
||
|
||
These tests are written using the Avocado Testing Framework (which must
|
||
be installed separately) in conjunction with a the ``avocado_qemu.Test``
|
||
class, implemented at ``tests/avocado/avocado_qemu``.
|
||
|
||
Tests based on ``avocado_qemu.Test`` can easily:
|
||
|
||
* Customize the command line arguments given to the convenience
|
||
``self.vm`` attribute (a QEMUMachine instance)
|
||
|
||
* Interact with the QEMU monitor, send QMP commands and check
|
||
their results
|
||
|
||
* Interact with the guest OS, using the convenience console device
|
||
(which may be useful to assert the effectiveness and correctness of
|
||
command line arguments or QMP commands)
|
||
|
||
* Interact with external data files that accompany the test itself
|
||
(see ``self.get_data()``)
|
||
|
||
* Download (and cache) remote data files, such as firmware and kernel
|
||
images
|
||
|
||
* Have access to a library of guest OS images (by means of the
|
||
``avocado.utils.vmimage`` library)
|
||
|
||
* Make use of various other test related utilities available at the
|
||
test class itself and at the utility library:
|
||
|
||
- http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test
|
||
- http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html
|
||
|
||
Running tests
|
||
~~~~~~~~~~~~~
|
||
|
||
You can run the avocado tests simply by executing:
|
||
|
||
.. code::
|
||
|
||
make check-avocado
|
||
|
||
This involves the automatic creation of Python virtual environment
|
||
within the build tree (at ``tests/venv``) which will have all the
|
||
right dependencies, and will save tests results also within the
|
||
build tree (at ``tests/results``).
|
||
|
||
Note: the build environment must be using a Python 3 stack, and have
|
||
the ``venv`` and ``pip`` packages installed. If necessary, make sure
|
||
``configure`` is called with ``--python=`` and that those modules are
|
||
available. On Debian and Ubuntu based systems, depending on the
|
||
specific version, they may be on packages named ``python3-venv`` and
|
||
``python3-pip``.
|
||
|
||
It is also possible to run tests based on tags using the
|
||
``make check-avocado`` command and the ``AVOCADO_TAGS`` environment
|
||
variable:
|
||
|
||
.. code::
|
||
|
||
make check-avocado AVOCADO_TAGS=quick
|
||
|
||
Note that tags separated with commas have an AND behavior, while tags
|
||
separated by spaces have an OR behavior. For more information on Avocado
|
||
tags, see:
|
||
|
||
https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/tags.html
|
||
|
||
To run a single test file, a couple of them, or a test within a file
|
||
using the ``make check-avocado`` command, set the ``AVOCADO_TESTS``
|
||
environment variable with the test files or test names. To run all
|
||
tests from a single file, use:
|
||
|
||
.. code::
|
||
|
||
make check-avocado AVOCADO_TESTS=$FILEPATH
|
||
|
||
The same is valid to run tests from multiple test files:
|
||
|
||
.. code::
|
||
|
||
make check-avocado AVOCADO_TESTS='$FILEPATH1 $FILEPATH2'
|
||
|
||
To run a single test within a file, use:
|
||
|
||
.. code::
|
||
|
||
make check-avocado AVOCADO_TESTS=$FILEPATH:$TESTCLASS.$TESTNAME
|
||
|
||
The same is valid to run single tests from multiple test files:
|
||
|
||
.. code::
|
||
|
||
make check-avocado AVOCADO_TESTS='$FILEPATH1:$TESTCLASS1.$TESTNAME1 $FILEPATH2:$TESTCLASS2.$TESTNAME2'
|
||
|
||
The scripts installed inside the virtual environment may be used
|
||
without an "activation". For instance, the Avocado test runner
|
||
may be invoked by running:
|
||
|
||
.. code::
|
||
|
||
tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/avocado/
|
||
|
||
Note that if ``make check-avocado`` was not executed before, it is
|
||
possible to create the Python virtual environment with the dependencies
|
||
needed running:
|
||
|
||
.. code::
|
||
|
||
make check-venv
|
||
|
||
It is also possible to run tests from a single file or a single test within
|
||
a test file. To run tests from a single file within the build tree, use:
|
||
|
||
.. code::
|
||
|
||
tests/venv/bin/avocado run tests/avocado/$TESTFILE
|
||
|
||
To run a single test within a test file, use:
|
||
|
||
.. code::
|
||
|
||
tests/venv/bin/avocado run tests/avocado/$TESTFILE:$TESTCLASS.$TESTNAME
|
||
|
||
Valid test names are visible in the output from any previous execution
|
||
of Avocado or ``make check-avocado``, and can also be queried using:
|
||
|
||
.. code::
|
||
|
||
tests/venv/bin/avocado list tests/avocado
|
||
|
||
Manual Installation
|
||
~~~~~~~~~~~~~~~~~~~
|
||
|
||
To manually install Avocado and its dependencies, run:
|
||
|
||
.. code::
|
||
|
||
pip install --user avocado-framework
|
||
|
||
Alternatively, follow the instructions on this link:
|
||
|
||
https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/installing.html
|
||
|
||
Overview
|
||
~~~~~~~~
|
||
|
||
The ``tests/avocado/avocado_qemu`` directory provides the
|
||
``avocado_qemu`` Python module, containing the ``avocado_qemu.Test``
|
||
class. Here's a simple usage example:
|
||
|
||
.. code::
|
||
|
||
from avocado_qemu import QemuSystemTest
|
||
|
||
|
||
class Version(QemuSystemTest):
|
||
"""
|
||
:avocado: tags=quick
|
||
"""
|
||
def test_qmp_human_info_version(self):
|
||
self.vm.launch()
|
||
res = self.vm.command('human-monitor-command',
|
||
command_line='info version')
|
||
self.assertRegexpMatches(res, r'^(\d+\.\d+\.\d)')
|
||
|
||
To execute your test, run:
|
||
|
||
.. code::
|
||
|
||
avocado run version.py
|
||
|
||
Tests may be classified according to a convention by using docstring
|
||
directives such as ``:avocado: tags=TAG1,TAG2``. To run all tests
|
||
in the current directory, tagged as "quick", run:
|
||
|
||
.. code::
|
||
|
||
avocado run -t quick .
|
||
|
||
The ``avocado_qemu.Test`` base test class
|
||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
|
||
The ``avocado_qemu.Test`` class has a number of characteristics that
|
||
are worth being mentioned right away.
|
||
|
||
First of all, it attempts to give each test a ready to use QEMUMachine
|
||
instance, available at ``self.vm``. Because many tests will tweak the
|
||
QEMU command line, launching the QEMUMachine (by using ``self.vm.launch()``)
|
||
is left to the test writer.
|
||
|
||
The base test class has also support for tests with more than one
|
||
QEMUMachine. The way to get machines is through the ``self.get_vm()``
|
||
method which will return a QEMUMachine instance. The ``self.get_vm()``
|
||
method accepts arguments that will be passed to the QEMUMachine creation
|
||
and also an optional ``name`` attribute so you can identify a specific
|
||
machine and get it more than once through the tests methods. A simple
|
||
and hypothetical example follows:
|
||
|
||
.. code::
|
||
|
||
from avocado_qemu import QemuSystemTest
|
||
|
||
|
||
class MultipleMachines(QemuSystemTest):
|
||
def test_multiple_machines(self):
|
||
first_machine = self.get_vm()
|
||
second_machine = self.get_vm()
|
||
self.get_vm(name='third_machine').launch()
|
||
|
||
first_machine.launch()
|
||
second_machine.launch()
|
||
|
||
first_res = first_machine.command(
|
||
'human-monitor-command',
|
||
command_line='info version')
|
||
|
||
second_res = second_machine.command(
|
||
'human-monitor-command',
|
||
command_line='info version')
|
||
|
||
third_res = self.get_vm(name='third_machine').command(
|
||
'human-monitor-command',
|
||
command_line='info version')
|
||
|
||
self.assertEquals(first_res, second_res, third_res)
|
||
|
||
At test "tear down", ``avocado_qemu.Test`` handles all the QEMUMachines
|
||
shutdown.
|
||
|
||
The ``avocado_qemu.LinuxTest`` base test class
|
||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
|
||
The ``avocado_qemu.LinuxTest`` is further specialization of the
|
||
``avocado_qemu.Test`` class, so it contains all the characteristics of
|
||
the later plus some extra features.
|
||
|
||
First of all, this base class is intended for tests that need to
|
||
interact with a fully booted and operational Linux guest. At this
|
||
time, it uses a Fedora 31 guest image. The most basic example looks
|
||
like this:
|
||
|
||
.. code::
|
||
|
||
from avocado_qemu import LinuxTest
|
||
|
||
|
||
class SomeTest(LinuxTest):
|
||
|
||
def test(self):
|
||
self.launch_and_wait()
|
||
self.ssh_command('some_command_to_be_run_in_the_guest')
|
||
|
||
Please refer to tests that use ``avocado_qemu.LinuxTest`` under
|
||
``tests/avocado`` for more examples.
|
||
|
||
QEMUMachine
|
||
~~~~~~~~~~~
|
||
|
||
The QEMUMachine API is already widely used in the Python iotests,
|
||
device-crash-test and other Python scripts. It's a wrapper around the
|
||
execution of a QEMU binary, giving its users:
|
||
|
||
* the ability to set command line arguments to be given to the QEMU
|
||
binary
|
||
|
||
* a ready to use QMP connection and interface, which can be used to
|
||
send commands and inspect its results, as well as asynchronous
|
||
events
|
||
|
||
* convenience methods to set commonly used command line arguments in
|
||
a more succinct and intuitive way
|
||
|
||
QEMU binary selection
|
||
^^^^^^^^^^^^^^^^^^^^^
|
||
|
||
The QEMU binary used for the ``self.vm`` QEMUMachine instance will
|
||
primarily depend on the value of the ``qemu_bin`` parameter. If it's
|
||
not explicitly set, its default value will be the result of a dynamic
|
||
probe in the same source tree. A suitable binary will be one that
|
||
targets the architecture matching host machine.
|
||
|
||
Based on this description, test writers will usually rely on one of
|
||
the following approaches:
|
||
|
||
1) Set ``qemu_bin``, and use the given binary
|
||
|
||
2) Do not set ``qemu_bin``, and use a QEMU binary named like
|
||
"qemu-system-${arch}", either in the current
|
||
working directory, or in the current source tree.
|
||
|
||
The resulting ``qemu_bin`` value will be preserved in the
|
||
``avocado_qemu.Test`` as an attribute with the same name.
|
||
|
||
Attribute reference
|
||
~~~~~~~~~~~~~~~~~~~
|
||
|
||
Test
|
||
^^^^
|
||
|
||
Besides the attributes and methods that are part of the base
|
||
``avocado.Test`` class, the following attributes are available on any
|
||
``avocado_qemu.Test`` instance.
|
||
|
||
vm
|
||
''
|
||
|
||
A QEMUMachine instance, initially configured according to the given
|
||
``qemu_bin`` parameter.
|
||
|
||
arch
|
||
''''
|
||
|
||
The architecture can be used on different levels of the stack, e.g. by
|
||
the framework or by the test itself. At the framework level, it will
|
||
currently influence the selection of a QEMU binary (when one is not
|
||
explicitly given).
|
||
|
||
Tests are also free to use this attribute value, for their own needs.
|
||
A test may, for instance, use the same value when selecting the
|
||
architecture of a kernel or disk image to boot a VM with.
|
||
|
||
The ``arch`` attribute will be set to the test parameter of the same
|
||
name. If one is not given explicitly, it will either be set to
|
||
``None``, or, if the test is tagged with one (and only one)
|
||
``:avocado: tags=arch:VALUE`` tag, it will be set to ``VALUE``.
|
||
|
||
cpu
|
||
'''
|
||
|
||
The cpu model that will be set to all QEMUMachine instances created
|
||
by the test.
|
||
|
||
The ``cpu`` attribute will be set to the test parameter of the same
|
||
name. If one is not given explicitly, it will either be set to
|
||
``None ``, or, if the test is tagged with one (and only one)
|
||
``:avocado: tags=cpu:VALUE`` tag, it will be set to ``VALUE``.
|
||
|
||
machine
|
||
'''''''
|
||
|
||
The machine type that will be set to all QEMUMachine instances created
|
||
by the test.
|
||
|
||
The ``machine`` attribute will be set to the test parameter of the same
|
||
name. If one is not given explicitly, it will either be set to
|
||
``None``, or, if the test is tagged with one (and only one)
|
||
``:avocado: tags=machine:VALUE`` tag, it will be set to ``VALUE``.
|
||
|
||
qemu_bin
|
||
''''''''
|
||
|
||
The preserved value of the ``qemu_bin`` parameter or the result of the
|
||
dynamic probe for a QEMU binary in the current working directory or
|
||
source tree.
|
||
|
||
LinuxTest
|
||
^^^^^^^^^
|
||
|
||
Besides the attributes present on the ``avocado_qemu.Test`` base
|
||
class, the ``avocado_qemu.LinuxTest`` adds the following attributes:
|
||
|
||
distro
|
||
''''''
|
||
|
||
The name of the Linux distribution used as the guest image for the
|
||
test. The name should match the **Provider** column on the list
|
||
of images supported by the avocado.utils.vmimage library:
|
||
|
||
https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
|
||
|
||
distro_version
|
||
''''''''''''''
|
||
|
||
The version of the Linux distribution as the guest image for the
|
||
test. The name should match the **Version** column on the list
|
||
of images supported by the avocado.utils.vmimage library:
|
||
|
||
https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
|
||
|
||
distro_checksum
|
||
'''''''''''''''
|
||
|
||
The sha256 hash of the guest image file used for the test.
|
||
|
||
If this value is not set in the code or by a test parameter (with the
|
||
same name), no validation on the integrity of the image will be
|
||
performed.
|
||
|
||
Parameter reference
|
||
~~~~~~~~~~~~~~~~~~~
|
||
|
||
To understand how Avocado parameters are accessed by tests, and how
|
||
they can be passed to tests, please refer to::
|
||
|
||
https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#accessing-test-parameters
|
||
|
||
Parameter values can be easily seen in the log files, and will look
|
||
like the following:
|
||
|
||
.. code::
|
||
|
||
PARAMS (key=qemu_bin, path=*, default=./qemu-system-x86_64) => './qemu-system-x86_64
|
||
|
||
Test
|
||
^^^^
|
||
|
||
arch
|
||
''''
|
||
|
||
The architecture that will influence the selection of a QEMU binary
|
||
(when one is not explicitly given).
|
||
|
||
Tests are also free to use this parameter value, for their own needs.
|
||
A test may, for instance, use the same value when selecting the
|
||
architecture of a kernel or disk image to boot a VM with.
|
||
|
||
This parameter has a direct relation with the ``arch`` attribute. If
|
||
not given, it will default to None.
|
||
|
||
cpu
|
||
'''
|
||
|
||
The cpu model that will be set to all QEMUMachine instances created
|
||
by the test.
|
||
|
||
machine
|
||
'''''''
|
||
|
||
The machine type that will be set to all QEMUMachine instances created
|
||
by the test.
|
||
|
||
qemu_bin
|
||
''''''''
|
||
|
||
The exact QEMU binary to be used on QEMUMachine.
|
||
|
||
LinuxTest
|
||
^^^^^^^^^
|
||
|
||
Besides the parameters present on the ``avocado_qemu.Test`` base
|
||
class, the ``avocado_qemu.LinuxTest`` adds the following parameters:
|
||
|
||
distro
|
||
''''''
|
||
|
||
The name of the Linux distribution used as the guest image for the
|
||
test. The name should match the **Provider** column on the list
|
||
of images supported by the avocado.utils.vmimage library:
|
||
|
||
https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
|
||
|
||
distro_version
|
||
''''''''''''''
|
||
|
||
The version of the Linux distribution as the guest image for the
|
||
test. The name should match the **Version** column on the list
|
||
of images supported by the avocado.utils.vmimage library:
|
||
|
||
https://avocado-framework.readthedocs.io/en/latest/guides/writer/libs/vmimage.html#supported-images
|
||
|
||
distro_checksum
|
||
'''''''''''''''
|
||
|
||
The sha256 hash of the guest image file used for the test.
|
||
|
||
If this value is not set in the code or by this parameter no
|
||
validation on the integrity of the image will be performed.
|
||
|
||
Skipping tests
|
||
~~~~~~~~~~~~~~
|
||
|
||
The Avocado framework provides Python decorators which allow for easily skip
|
||
tests running under certain conditions. For example, on the lack of a binary
|
||
on the test system or when the running environment is a CI system. For further
|
||
information about those decorators, please refer to::
|
||
|
||
https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#skipping-tests
|
||
|
||
While the conditions for skipping tests are often specifics of each one, there
|
||
are recurring scenarios identified by the QEMU developers and the use of
|
||
environment variables became a kind of standard way to enable/disable tests.
|
||
|
||
Here is a list of the most used variables:
|
||
|
||
AVOCADO_ALLOW_LARGE_STORAGE
|
||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
Tests which are going to fetch or produce assets considered *large* are not
|
||
going to run unless that ``AVOCADO_ALLOW_LARGE_STORAGE=1`` is exported on
|
||
the environment.
|
||
|
||
The definition of *large* is a bit arbitrary here, but it usually means an
|
||
asset which occupies at least 1GB of size on disk when uncompressed.
|
||
|
||
AVOCADO_ALLOW_UNTRUSTED_CODE
|
||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||
There are tests which will boot a kernel image or firmware that can be
|
||
considered not safe to run on the developer's workstation, thus they are
|
||
skipped by default. The definition of *not safe* is also arbitrary but
|
||
usually it means a blob which either its source or build process aren't
|
||
public available.
|
||
|
||
You should export ``AVOCADO_ALLOW_UNTRUSTED_CODE=1`` on the environment in
|
||
order to allow tests which make use of those kind of assets.
|
||
|
||
AVOCADO_TIMEOUT_EXPECTED
|
||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||
The Avocado framework has a timeout mechanism which interrupts tests to avoid the
|
||
test suite of getting stuck. The timeout value can be set via test parameter or
|
||
property defined in the test class, for further details::
|
||
|
||
https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#setting-a-test-timeout
|
||
|
||
Even though the timeout can be set by the test developer, there are some tests
|
||
that may not have a well-defined limit of time to finish under certain
|
||
conditions. For example, tests that take longer to execute when QEMU is
|
||
compiled with debug flags. Therefore, the ``AVOCADO_TIMEOUT_EXPECTED`` variable
|
||
has been used to determine whether those tests should run or not.
|
||
|
||
GITLAB_CI
|
||
^^^^^^^^^
|
||
A number of tests are flagged to not run on the GitLab CI. Usually because
|
||
they proved to the flaky or there are constraints on the CI environment which
|
||
would make them fail. If you encounter a similar situation then use that
|
||
variable as shown on the code snippet below to skip the test:
|
||
|
||
.. code::
|
||
|
||
@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
|
||
def test(self):
|
||
do_something()
|
||
|
||
Uninstalling Avocado
|
||
~~~~~~~~~~~~~~~~~~~~
|
||
|
||
If you've followed the manual installation instructions above, you can
|
||
easily uninstall Avocado. Start by listing the packages you have
|
||
installed::
|
||
|
||
pip list --user
|
||
|
||
And remove any package you want with::
|
||
|
||
pip uninstall <package_name>
|
||
|
||
If you've used ``make check-avocado``, the Python virtual environment where
|
||
Avocado is installed will be cleaned up as part of ``make check-clean``.
|
||
|
||
.. _checktcg-ref:
|
||
|
||
Testing with "make check-tcg"
|
||
-----------------------------
|
||
|
||
The check-tcg tests are intended for simple smoke tests of both
|
||
linux-user and softmmu TCG functionality. However to build test
|
||
programs for guest targets you need to have cross compilers available.
|
||
If your distribution supports cross compilers you can do something as
|
||
simple as::
|
||
|
||
apt install gcc-aarch64-linux-gnu
|
||
|
||
The configure script will automatically pick up their presence.
|
||
Sometimes compilers have slightly odd names so the availability of
|
||
them can be prompted by passing in the appropriate configure option
|
||
for the architecture in question, for example::
|
||
|
||
$(configure) --cross-cc-aarch64=aarch64-cc
|
||
|
||
There is also a ``--cross-cc-flags-ARCH`` flag in case additional
|
||
compiler flags are needed to build for a given target.
|
||
|
||
If you have the ability to run containers as the user the build system
|
||
will automatically use them where no system compiler is available. For
|
||
architectures where we also support building QEMU we will generally
|
||
use the same container to build tests. However there are a number of
|
||
additional containers defined that have a minimal cross-build
|
||
environment that is only suitable for building test cases. Sometimes
|
||
we may use a bleeding edge distribution for compiler features needed
|
||
for test cases that aren't yet in the LTS distros we support for QEMU
|
||
itself.
|
||
|
||
See :ref:`container-ref` for more details.
|
||
|
||
Running subset of tests
|
||
~~~~~~~~~~~~~~~~~~~~~~~
|
||
|
||
You can build the tests for one architecture::
|
||
|
||
make build-tcg-tests-$TARGET
|
||
|
||
And run with::
|
||
|
||
make run-tcg-tests-$TARGET
|
||
|
||
Adding ``V=1`` to the invocation will show the details of how to
|
||
invoke QEMU for the test which is useful for debugging tests.
|
||
|
||
TCG test dependencies
|
||
~~~~~~~~~~~~~~~~~~~~~
|
||
|
||
The TCG tests are deliberately very light on dependencies and are
|
||
either totally bare with minimal gcc lib support (for softmmu tests)
|
||
or just glibc (for linux-user tests). This is because getting a cross
|
||
compiler to work with additional libraries can be challenging.
|
||
|
||
Other TCG Tests
|
||
---------------
|
||
|
||
There are a number of out-of-tree test suites that are used for more
|
||
extensive testing of processor features.
|
||
|
||
KVM Unit Tests
|
||
~~~~~~~~~~~~~~
|
||
|
||
The KVM unit tests are designed to run as a Guest OS under KVM but
|
||
there is no reason why they can't exercise the TCG as well. It
|
||
provides a minimal OS kernel with hooks for enabling the MMU as well
|
||
as reporting test results via a special device::
|
||
|
||
https://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git
|
||
|
||
Linux Test Project
|
||
~~~~~~~~~~~~~~~~~~
|
||
|
||
The LTP is focused on exercising the syscall interface of a Linux
|
||
kernel. It checks that syscalls behave as documented and strives to
|
||
exercise as many corner cases as possible. It is a useful test suite
|
||
to run to exercise QEMU's linux-user code::
|
||
|
||
https://linux-test-project.github.io/
|
||
|
||
GCC gcov support
|
||
----------------
|
||
|
||
``gcov`` is a GCC tool to analyze the testing coverage by
|
||
instrumenting the tested code. To use it, configure QEMU with
|
||
``--enable-gcov`` option and build. Then run the tests as usual.
|
||
|
||
If you want to gather coverage information on a single test the ``make
|
||
clean-gcda`` target can be used to delete any existing coverage
|
||
information before running a single test.
|
||
|
||
You can generate a HTML coverage report by executing ``make
|
||
coverage-html`` which will create
|
||
``meson-logs/coveragereport/index.html``.
|
||
|
||
Further analysis can be conducted by running the ``gcov`` command
|
||
directly on the various .gcda output files. Please read the ``gcov``
|
||
documentation for more information.
|