Some tests use external tools (executables) during their operation. Add
a test.py mark to indicate this. This allows those tests to be skipped if
the required tool is not present.
Signed-off-by: Stephen Warren <swarren@nvidia.com>
When skipping tests, explicitly mention the board type or config option
that caused the skip. This will help people understand/fix any issues.
Signed-off-by: Stephen Warren <swarren@nvidia.com>
If a test uses a fixture which is expensive to setup, the fixture can
possibly created with session or module scope. As u_boot_console has
function scope, it can not be used in this case.
Signed-off-by: Stefan Brüns <stefan.bruens@rwth-aachen.de>
Acked-by: Stephen Warren <swarren@wwwdotorg.org>
In pytest 3, runtestprotocol() may not call pytest_runtest_setup() if
the test is skipped. That call is required to create a section for the
test in the log file. If this is skipped, the call to log.end_section()
at the tail of pytest_runtest_protocol() will throw an exception. This
patch ensures that a log section always exists, both to avoid the
exception and to ensure that a consistently structured log file is
always created.
Cc: Stefan Brüns <stefan.bruens@rwth-aachen.de>
Reported-by: Stefan Brüns <stefan.bruens@rwth-aachen.de>
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Tested-by: Tom Rini <trini@konsulko.com>
This board can sometimes be used for tests. Handle it the same way as
sandbox.
Note: I plan to drop the sandbox_spl board at some point and merge its
features into sandbox. So this commit may not be necessary.
Signed-off-by: Simon Glass <sjg@chromium.org>
Normally tests will run with the test.dtb file designed for this purpose.
However, the verified boot tests need to run with their own device-tree
file, containing a public key.
Make the device-tree file a config option so that it can be adjusted by
tests. The default is to keep the current behaviour.
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Teddy Reed <teddy.reed@gmail.com>
Use lists rather than sets to record the status of tests. This causes
the test summary in the HTML file to be generated in the same order as
the tests are (or would have been) run. This makes it easier to locate
the first failed test. The log for this test might have interesting
first clues re: interaction with the environment (e.g. hardware flashing,
serial console, ...) and may help tracking down external issues.
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
The code replaced pexpect with custom code long ago. Don't import the
unused module so it doesn't need to be installed.
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
Invoke each "ut"-based unit test as a separate pytest.
Now that the DM unit test runs under test/py, remove the manual shell
script that invokes it.
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org> # v2, on sandbox
Implement three improvements to the HTML log file:
- Ability to expand/contract sections. All passing sections are contracted
at file load time so the user can concentrate on issues requiring
action.
- The overall status report is copied to the top of the log for easy
access.
- Add links from the status report to the test logs, for easy navigation.
This all relies on Javascript and the jquery library. If the user doesn't
have Javascript enabled, or jquery can't be downloaded, the log should
look and behave identically to how it did before this patch.
A few notes on the diff:
- A few more 'with log.section("xxx")' were added, so that all stream
blocks are kept within a section block for consistent HTML entity
nesting structure. This changed indentation in a few places, making
the diff look slightly larger.
- HTML entity IDs are cleaned up. We assign simple incrementing integer
IDs now, rather than using mangled test names which were possibly
invalid.
- Sections and streams now use common CSS class names (in addition to the
current separate class names) to more easily share the new behaviour.
This also reduces the CSS file size since rules don't need to be
duplicated.
- An "OK" status is logged after some external command executions so that
make and flash steps are auto-contracted at log file load time, assuming
they passed.
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Implement command--line option --gdbserver COMM, which does two things:
a) Run the sandbox process under gdbserver, using COMM as gdbserver's
communication channel.
b) Disables all timeouts, so that if U-Boot is halted under the debugger,
tests don't fail. If the user gives up in the middle of a debugging
session, they can simply CTRL-C the test script to abort it.
This allows easy debugging of test failures without having to manually
re-create the failure conditions. Usage is:
Window 1:
./test/py/test.py --bd sandbox --gdbserver localhost:1234
Window 2:
gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234'
When using this option, it likely makes sense to use pytest's -k option
to limit the set of tests that are executed.
Simply running U-Boot directly under gdb (rather than gdbserver) was
also considered. However, this was rejected because:
a) gdb's output would then be processed by the test script, and likely
confuse it causing false failures.
b) pytest by default hides stdout from tests, which would prevent the
user from interacting with gdb.
While gdb can be told to redirect the debugee's stdio to a separate
PTY, this would appear to leave gdb's stdio directed at the test
scripts and the debugee's stdio directed elsewhere, which is the
opposite of the desired effect. Perhaps some complicated PTY muxing
and process hierarchy could invert this. However, the current scheme
is simple to implement and use, so it doesn't seem worth complicating
matters.
c) Using gdbserver allows arbitrary debuggers to be used, even those with
a GUI. If the test scripts invoked the debugger themselves, they'd have
to know how to execute arbitary applications. While the user could hide
this all in a wrapper script, this feels like extra complication.
An interesting future idea might be a --gdb-screen option, which could
spawn both U-Boot and gdb separately, and spawn the screen into a newly
created window under screen. Similar options could be envisaged for
creating a new xterm/... too.
--gdbserver currently only supports sandbox, and not real hardware.
That's primarily because the test hooks are responsible for all aspects of
hardware control, so there's nothing for the test scripts themselves can
do to enable gdbserver on real hardware. We might consider introducing a
separate --disable-timeouts option to support use of debuggers on real
hardware, and having --gdbserver imply that option.
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Tests can complete in passed, skipped, xpass, xfailed, or failed, states.
Currently the U-Boot log generation code doesn't handle the xfailed or
xpass states since they aren't used. Add support for the remaining states.
Without this, tests that xfail end up being reported as skipped.
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
When pytest generates the name for parametrized tests, simple parameter
values (ints, strings) get used directly, but more complex values such
as dicts are not handled. This yields test names such as:
dfu[env__usb_dev_port0-env__dfu_config0]
dfu[env__usb_dev_port0-env__dfu_config1]
Add some code to extract a custom fixture ID from the fixture values, so
that we end up with meaningful names such as:
dfu[micro_b-emmc]
dfu[devport2-ram]
If the boardenv file doesn't define custom names, the code falls back to
the old algorithm.
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Python's coding style docs indicate to use " not ' for docstrings.
test/py has other violations of the coding style docs, since the docs
specify a stranger style than I would expect, but nobody has complained
about those yet:-)
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Tests may fail for a number of reasons, and in particular for reasons
other than a timeout waiting for U-Boot to print expected data. If the
last operation that a failed test performs is not waiting for U-Boot to
print something, then any trailing output from U-Boot during that test's
operation will not be logged as part of that test, but rather either
along with the next test, or even thrown away, potentiall hiding clues
re: the test failure reason.
Solve this by explicitly draining (and hence logging) the U-Boot output
in the case of failed tests.
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
Prior to this change, U-Boot was lazilly (re-)spawned if/when a test
attempted to interact with it, and no active connection existed. This
approach was simple, yet had the disadvantage that U-Boot might be
spawned in the middle of a test function, e.g. after the test had already
performed actions such as creating data files, etc. In that case, this
could cause the log to contain the sequence (1) some test logs, (2)
U-Boot's boot process, (3) the rest of that test's logs. This isn't
optimally readable. This issue will affect the upcoming DFU and enhanced
UMS tests.
This change converts u_boot_console to be a function-scoped fixture, so
that pytest attempts to re-create the object for each test invocation.
This allows the fixture factory function to ensure that U-Boot is spawned
prior to every test. In practice, the same object is returned each time
so there is essentially no additional overhead due to this change.
This allows us to remove:
- The explicit ensure_spawned() call from test_sleep, since the core now
ensures that the spawn happens before the test code is executed.
- The laxy calls to ensure_spawned() in the u_boot_console_*
implementations.
The one downside is that test_env's "state_ttest_env" fixture must be
converted to a function-scoped fixture too, since a module-scoped fixture
cannot use a function-scoped fixture. To avoid overhead, we use the same
trick of returning the same object each time.
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
This tool aims to test U-Boot by executing U-Boot shell commands using the
console interface. A single top-level script exists to execute or attach
to the U-Boot console, run the entire script of tests against it, and
summarize the results. Advantages of this approach are:
- Testing is performed in the same way a user or script would interact
with U-Boot; there can be no disconnect.
- There is no need to write or embed test-related code into U-Boot itself.
It is asserted that writing test-related code in Python is simpler and
more flexible that writing it all in C.
- It is reasonably simple to interact with U-Boot in this way.
A few simple tests are provided as examples. Soon, we should convert as
many as possible of the other tests in test/* and test/cmd_ut.c too.
The hook scripts, relay control utilities, and udev rules I use for my
own HW setup are published at https://github.com/swarren/uboot-test-hooks.
See README.md for more details!
Signed-off-by: Stephen Warren <swarren@wwwdotorg.org>
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Tested-by: Michal Simek <michal.simek@xilinx.com>
Tested-by: Simon Glass <sjg@chromium.org>
Acked-by: Simon Glass <sjg@chromium.org> #v3
Stephen Warren
Stefan Brüns
Simon Glass
Stephen Warren