f0842a27c5
Adds an config attribute to the Depot Autopilot component:
:<config repeat>:
Can be one of
"false" - process the given test list only once,
"until_forever" - endlessly repeat processing the given test list,
"until_failed" - repeat processing the given test list until it fails.
Adds an environment variable to the Depot Autopilot Run script:
:TEST_REPEAT:
Same as the <config repeat> attribute of the Depot Autopilot.
This is useful when having to debug very sporadic errors during one test
or a series of tests.
391 lines
14 KiB
Plaintext
391 lines
14 KiB
Plaintext
The Depot Autopilot can be used to successively evaluate a given set of test
|
|
packages and produce an overview which makes it easy to see the most important
|
|
findings for each test. This is a brief overview of the features thereby
|
|
provided:
|
|
|
|
* Execute multiple tests in a sub-init successively in a given order
|
|
* Each test can define multiple log patterns and timeouts that render it
|
|
either failed or succeeded
|
|
* A tests definition and ingredients come in the form of a Genode package
|
|
archive that gets queried by the Autopilot on demand
|
|
* Forward the log of each test to the log session of the Autopilot prefixed
|
|
with a timestamp and the test-local label
|
|
* After the last test, print an overview that lists for each test the test
|
|
result, why this result was assumed, and the test duration
|
|
* Consider that tests might crash the whole system. Thus, the Autopilot
|
|
provides information on when to consider a system reboot. And it makes it
|
|
possible to gather all findings from before a system reboot to hand them
|
|
in at the next run.
|
|
* Consider tests that are "not supported" so the overview tables of different
|
|
platforms are comparable
|
|
|
|
|
|
The Depot Autopilot is accompanied by the Run script
|
|
'repos/gems/run/depot_autopilot.run' which doesn't describe a classical test
|
|
but rather helps integrating the Autopilot into a Linux-based test
|
|
environment. The run script provides the following features:
|
|
|
|
* Creates a complete Autopilot scenario from a list of local test packages
|
|
* Optional platform dependencies can be defined for each test package
|
|
* Simple interface for selectively using test ingredients built from source
|
|
* Can be combined with a list of source packages and gcov to show test coverage
|
|
* Exits with 0 if all tests are either "ok" or "skipped", otherwise with < 0
|
|
* The target is re-booted if the scenario gets stuck
|
|
* On each re-boot the Autopilot is configured only with the remaining tests
|
|
* Tests that caused a re-boot are listed as "failed" with cause "reboot"
|
|
* The final Autopilot result overview reflects all results from all boots
|
|
* If a complete result overview can't be obtained, a partial one is simulated
|
|
|
|
|
|
Depot Autopilot component
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Configuration
|
|
-------------
|
|
|
|
This is an example configuration of the Depot Autopilot:
|
|
|
|
! <config arch="x86_64" children_label_prefix="dynamic -> " repeat="false">
|
|
! <static>
|
|
! <parent-provides>
|
|
! <service name="ROM"/>
|
|
! <service name="CPU"/>
|
|
! ...
|
|
! </parent-provides>
|
|
! </static>
|
|
! <common_routes>
|
|
! <service name="ROM" label_last="ld.lib.so"> <parent/> </service>
|
|
! <service name="ROM" label_last="init"> <parent/> </service>
|
|
! <service name="CPU"> <parent/> </service>
|
|
! ...
|
|
! </common_routes>
|
|
!
|
|
! <previous-results time_sec="3"
|
|
! succeeded="0"
|
|
! skipped="1"
|
|
! failed="1"> test-rm_fault_no_nox skipped
|
|
! test-rm_nested failed 0.653 log "Error: "</previous-results>
|
|
!
|
|
! <start name="test-mmio" pkg="genodelabs/pkg/test-mmio/2018-10-30"/>
|
|
! <start name="test-rtc" skip="true"/>
|
|
! <start name="test-xml_node" pkg="genodelabs/pkg/test-xml_node/2018-10-30"/>
|
|
! ...
|
|
! </config>
|
|
|
|
:<config arch>:
|
|
|
|
Defines the architecture that the Autopilot is running on for querying the
|
|
appropriate variant of the test packages. Must be one of "x86_64", "x86_32",
|
|
"arm_v7a".
|
|
|
|
:<config children_label_prefix>:
|
|
|
|
Label prefix of LOG sessions of the components of a test. This is required
|
|
to relate incoming LOG-session request to a running test.
|
|
|
|
:<config repeat>:
|
|
|
|
Can be one of
|
|
|
|
"false" - process the given test list only once,
|
|
"until_forever" - endlessly repeat processing the given test list,
|
|
"until_failed" - repeat processing the given test list until it fails.
|
|
|
|
:<config><static>:
|
|
|
|
Contains Init configuration that shall always be added to the configuration
|
|
of the target Init that runs the test scenarios. For information about the
|
|
format of the content of this tag see 'repos/os/src/init/config.xsd'
|
|
|
|
:<config><common_routes>:
|
|
|
|
Contains a routing configuration like in a <route> tag of an Init <start>
|
|
node that shall be added to the routing configuration of the runtime-Init
|
|
of each test.
|
|
|
|
:<config><previous-results>:
|
|
|
|
Contains a string that shall preceed the overview table. Each line of the
|
|
string is expected to have the format:
|
|
" $NAME $SEC.$MS $RESULT $ADDITIONAL_INFO" where $NAME is the name of the
|
|
test package right-padded with whitespaces to a size of 32, $SEC are the
|
|
seconds of the test duration left-padded with whitespaces to a size of 3,
|
|
$MS are the milliseconds of the test duration left-padded with zeros to a
|
|
size of 3, $RESULT is one of "ok", "failed", "skipped" right-padded with
|
|
whitespaces to a size of 7, and $ADDITIONAL_INFO is an arbitrary string.
|
|
This format is the same as in the result line printed out by the Autopilot
|
|
after each single test. The string is taken as is and should not contain '<'
|
|
or additional tabulators, whitespaces or newlines. It is suggested that all
|
|
occurences of '<' are replaced with "<", so, they can be sanitized in the
|
|
future.
|
|
|
|
:<config><previous-results time_sec>:
|
|
|
|
Contains a time in seconds that shall be added to the total time shown in
|
|
header of the overview.
|
|
|
|
:<config><previous-results succeeded>:
|
|
:<config><previous-results failed>:
|
|
:<config><previous-results skipped>:
|
|
|
|
A non-negative number that denotes how many tests are already
|
|
succeeded/failed/skipped. The number is added to the number of
|
|
succeeded/failed/skipped tests in the result overview.
|
|
|
|
:<config><start>:
|
|
|
|
Each <start> node stands for a test that shall be evaluated by the Autopilot.
|
|
The tests are evaluated in the order that the start nodes state. Also the
|
|
result overview is ordered this way.
|
|
|
|
:<config><start name>:
|
|
|
|
This string is used to identify the test in the log output of the Autopilot
|
|
and to name the runtime component of the test. It should be unique amongst
|
|
all <start> nodes.
|
|
|
|
:<config><start pkg>:
|
|
|
|
Defines the Genode package path of the test package in the format
|
|
"$DEPOT_USER/pkg/$NAME/$VERSION".
|
|
|
|
:<config><start skip>:
|
|
|
|
A boolean value. Its default value is false. If set to true, the test is not
|
|
evaluated at all, no package or "pkg" attribute is needed. The test only
|
|
appears in the overview with the result "skipped".
|
|
|
|
|
|
Format of test packages
|
|
-----------------------
|
|
|
|
Besides the mandatory package content, a test package is expected to provide a
|
|
'runtime' file. This is an example runtime file:
|
|
|
|
! <runtime ram="32M" caps="1000" binary="init">
|
|
!
|
|
! <requires> <timer/> <nic/> </requires>
|
|
!
|
|
! <events>
|
|
! <timeout meaning="failed" sec="20" />
|
|
! <log meaning="succeeded">Ignores tabs
|
|
! and newlines but no whitespaces !
|
|
! Some XML: <my-node my_attr="123"/>
|
|
! [init -> test-example] Wildcarded *text *literal star * ampersand &
|
|
! </log>
|
|
! </events>
|
|
!
|
|
! <content>
|
|
! <rom label="ld.lib.so"/> <rom label="test-example"/>
|
|
! </content>
|
|
!
|
|
! <config>
|
|
! <parent-provides>
|
|
! ...
|
|
! </parent-provides>
|
|
! <default-route>
|
|
! ...
|
|
! </default-route>
|
|
! <default caps="100"/>
|
|
! <start name="test-example" caps="500">
|
|
! <resource name="RAM" quantum="10M"/>
|
|
! </start>
|
|
! </config>
|
|
! </runtime>
|
|
|
|
:<runtime ram>:
|
|
|
|
Amount of RAM quota that is donated to the tests root component.
|
|
|
|
:<runtime caps>:
|
|
|
|
Amount of CAP quota that is donated to the tests root component.
|
|
|
|
:<runtime binary>:
|
|
|
|
Name of the binary of the tests root component.
|
|
|
|
:<requires>:
|
|
|
|
Not examined yet. Should contain a sub-node for every individual resource
|
|
that the test expects to be provided by the world outside its sub-tree. This
|
|
exists mainly for the annotation of required drivers. Normally, drivers
|
|
should not be part of a test sub-tree as we don't want to get into the
|
|
situation of restarting them.
|
|
|
|
:<events>:
|
|
|
|
Lists events that may occur during the test and that shall imply a reaction
|
|
by the Autopilot.
|
|
|
|
:<events><timeout sec>:
|
|
|
|
This event occurs when the test execution lasts the given time in seconds.
|
|
|
|
:<events><timeout meaning>:
|
|
|
|
One of "succeeded", "failed". Both imply the test to be treated as finished
|
|
(gets terminated) with the test result set to the given value.
|
|
|
|
:<events><log>:
|
|
|
|
Contains a string pattern. The event occurs as soon as the pattern could be
|
|
completely matched against the LOG-session output of the test. Tabs and
|
|
newlines are ignored in the pattern as well as in the test output. Literal
|
|
characters '<', '&', '*' in the pattern must be escaped as "<", "&",
|
|
"*". A character '*' in the pattern is treated as non-greedy wildcard.
|
|
|
|
:<events><log meaning>:
|
|
|
|
See <events><timeout meaning>.
|
|
|
|
:<content>:
|
|
|
|
Lists required files from the test-package build besides the root-component
|
|
binary.
|
|
|
|
:<content><rom label>:
|
|
|
|
Requirement for a file with the given name.
|
|
|
|
:<config>:
|
|
|
|
Contains the configuration for the tests root component.
|
|
|
|
|
|
Integration
|
|
-----------
|
|
|
|
:Required sessions:
|
|
|
|
* A Timer session for realizing timeout events stated in the test
|
|
runtime-files and providing timing information in the LOG output.
|
|
|
|
* A Report session labeled "query". Through this report, the Depot Autopilot
|
|
requests the blueprint for the test-package required next. This is an
|
|
example content of this report:
|
|
|
|
! <query arch="x86_64">
|
|
! <blueprint pkg="genodelabs/pkg/test-ada/2018-11-02"/>
|
|
! </query>
|
|
|
|
The format is compliant to that of the "query" ROM required by the Depot
|
|
Query component in 'repos/gems/src/app/depot_query'.
|
|
|
|
* A ROM session labeled "blueprint". The blueprint ROM is expected to contain
|
|
the information requested via the above stated "query" Report. It may look
|
|
like this:
|
|
|
|
! <blueprint>
|
|
! <pkg name="test-ada" path="genodelabs/pkg/test-ada/2018-11-02">
|
|
! <rom label="ada.lib.so" path="genodelabs/bin/x86_64/ada/2018-11-01/ada.lib.so"/>
|
|
! <missing_rom label="ld.lib.so"/>
|
|
! ... # further <rom> and <missing_rom> tags
|
|
!
|
|
! <runtime ram="32M" caps="1000" binary="init">
|
|
! ... # content of the tests runtime file
|
|
! </runtime>
|
|
! </pkg>
|
|
! </blueprint>
|
|
|
|
The format is compliant to that of the "blueprint" Report provided by the
|
|
Depot Query component in 'repos/gems/src/app/depot_query'.
|
|
|
|
* A Report session labeled "init.config". Through this report a configuration
|
|
for an Init component is provided in a way that it serves as parent for the
|
|
root components of the tests. It may look like this:
|
|
|
|
! <config>
|
|
! ... # content of <static> tag in Autopilot configuration
|
|
!
|
|
! <start name="test-ada" caps="1000">
|
|
! <binary name="init"/>
|
|
! <resource name="RAM" quantum="32M"/>
|
|
!
|
|
! <config ...> ... <config> # from the tests runtime file
|
|
!
|
|
! <route>
|
|
! ... # content of <common_routes> tag in Autopilot configuration
|
|
!
|
|
! # routes for each <content><rom> tags in the tests runtime file
|
|
! <service name="ROM" label_last="test-ada">
|
|
! <parent label="genodelabs/bin/x86_64/test-ada/2018-11-02/test-ada"/>
|
|
! </service>
|
|
! ...
|
|
!
|
|
! </route>
|
|
! </start>
|
|
! </config>
|
|
|
|
:Provided services:
|
|
|
|
* A LOG service for matching LOG-event patterns stated in the test runtime-
|
|
files against the LOG output of the test components. Thus, you should take
|
|
care that all LOG session requests of test components are routed to the
|
|
Depot Autopilot. You can still receive the whole LOG output of the test
|
|
components as part of the LOG output of the Depot Autopilot.
|
|
|
|
|
|
Depot Autopilot Run script
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
By default, the Run script tries to evaluate all tests that are internally
|
|
configured, that is, normally, all that are available as package recipes in
|
|
the main repositories. This includes one test that is prepared for the later
|
|
examination by gcov as well as gcov itself, which is executed like a test at
|
|
the end of the test list to examine the above mentioned adapted test. The test
|
|
list is processed only once and all test ingredients are used from packages,
|
|
assuming the depot user is 'genodelabs'. For this default behavior just call
|
|
the script like any "normal" Run script:
|
|
|
|
! make run/depot_autopilot KERNEL="nova"
|
|
|
|
However, this is normally a pretty time intensive process and meant for the
|
|
integration into automatic test infrastructures rather than manual evaluation.
|
|
This is why you might want to adapt the scenario, for instance, for debugging
|
|
one or more tests that failed during your automated testing. For this purpose,
|
|
a user interface is given through specific environment variables. This is an
|
|
example containing all of these variables:
|
|
|
|
! make run/depot_autopilot \
|
|
! > TEST_PKGS="test-libc_vfs test-log" \
|
|
! > TEST_SRCS="test-xml_generator test-xml_node" \
|
|
! > TEST_BUILDS="server/ram_fs test/libc_vfs" \
|
|
! > TEST_MODULES="ram_fs test-libc_vfs vfs.lib.so" \
|
|
! > TEST_REPEAT="until_failed" \
|
|
! > KERNEL="nova"
|
|
|
|
:TEST_PKGS:
|
|
|
|
List of test package-archives that supersedes the internal list if set.
|
|
|
|
:TEST_SRCS:
|
|
|
|
List of test source-archives that supersedes the internal list if set.
|
|
|
|
:TEST_BUILDS:
|
|
|
|
List of test build-targets that be build from the local repositories.
|
|
|
|
:TEST_MODULES:
|
|
|
|
List of test boot-modules that overlay the contents of the package-archives.
|
|
|
|
:TEST_REPEAT:
|
|
|
|
See the <config repeat> attribute of the Depot Autopilot.
|
|
|
|
To get a hint which build components (TEST_BUILDS) and which boot modules
|
|
(TEST_MODULES) you may want to enter for a given test package, you may have
|
|
a look at the package recipe:
|
|
|
|
! repos/<REPO>/recipes/pkg/<TEST_PKG>/archives
|
|
! repos/<REPO>/recipes/pkg/<TEST_PKG>/runtime (<content> tag)
|
|
|
|
|
|
Examples
|
|
--------
|
|
|
|
Please see the run script 'repos/gems/run/depot_autopilot.run' for a
|
|
comprehensive example of how to use the Depot Autopilot.
|