While running tests for a python tool, the tests' outputs get printed in
whatever order they happen to run, without any indication as to which
output belongs to which test. Unittest supports capturing these outputs
and printing them as part of the test summaries, but when a failure or
error occurs it switches back to printing as the tests run. Testtools
and subunit tests can do the same as their parts inherit from unittest,
but they don't outright expose this functionality.
On the unittest side, enable output buffering for the custom test result
class. Try to avoid ugly outputs by not printing stdout/stderr before
the test summary for low verbosity levels and for successful tests.
On the subunit side, implement a custom TestProtocolClient that enables
the same underlying functionality and injects the captured streams as
additional test details. This causes them to be merged into their test's
error traceback message, which is later rebuilt into an exception and
passed to our unittest report class.
Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
By default, unittest test summaries only print extended info about tests
that failed or couldn't run due to an error. Use a custom text result
class to print info about more cases: skipped tests, expected failures
and unexpected successes.
Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
The python tools' test utilities handle printing test results, but the
output is quite bare compared to an ordinary unittest run. Delegate
printing the results to a unittest text runner, which gives us niceties
like clear separation between each test's result and how long it took to
run the test suite.
Unfortunately it does not print info for skipped tests by default, but
this can be handled later by a custom test result subclass. It also does
not print the tool name; manually print a heading that includes the
toolname so that the outputs of each tool's tests are distinguishable in
the CI output.
Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
It's possible to request a specific test to run when trying to run a
python tool's tests. If we request a nonexistent test, the unittest
loaders generate a fake test that reports this as an error. However, we
get these fake tests even when the test exists, because test_util can
load tests from multiple places one by one and the test we want only
exists in one.
The test_util helpers currently remove these fake tests when printing
test results, but that's more of a workaround than a proper solution.
Instead, don't even try to load the missing tests.
Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
When printing a python tool's test results, the entire list of failed
tests and their tracebacks are reprinted for every failed test. This
makes the test output quite unreadable. Fix the loop to print failures
and tracebacks one at a time.
Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
At present this function does not run the doctests. Allow the caller to
pass these modules in as strings.
Update patman to use this.
Signed-off-by: Simon Glass <sjg@chromium.org>
With the change to absolute imports the concurrent tests feature
unfortunately broke. Fix it.
We cannot easy add a warning, since the output messes up tests which check
the output.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present binman's test coverage runs without a toolpath set. This means
that the system tools will be used. That may not be correct if they are
out of date or missing and this can result in a reduction in test coverage
below 100%.
Provide the toolpath to binman in this case.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present we use --test to indicate that tests should be run. It is
better to use a subcommand for list, like binman. Change it and adjust
the existing code to fit under a 'send' subcommand, the default.
Give this subcommand the same default arguments as the others.
Signed-off-by: Simon Glass <sjg@chromium.org>
With the change to absolute imports the concurrent tests feature
unfortunately broke. Fix it.
We cannot easy add a warning, since the output messes up tests which check
the output.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present binman's test coverage runs without a toolpath set. This means
that the system tools will be used. That may not be correct if they are
out of date or missing and this can result in a reduction in test coverage
below 100%.
Provide the toolpath to binman in this case.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present we use --test to indicate that tests should be run. It is
better to use a subcommand for list, like binman. Change it and adjust
the existing code to fit under a 'send' subcommand, the default.
Give this subcommand the same default arguments as the others.
Signed-off-by: Simon Glass <sjg@chromium.org>
This module doesn't need to import itself. It causes problems on
very old Python 3 (e.g. 3.4.0). Drop it.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present patman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.
Move patman to use absolute imports. This requires changes in tools which
use the patman libraries (which is most of them).
Signed-off-by: Simon Glass <sjg@chromium.org>
Python does not like the module name being the same as the module
directory. To allow buildman modules to be used from other tools, rename
it.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present entry modules can only be accessed using Entry.Lookup() or
Entry.Create(). Most of the time this is fine, but sometimes a module
needs to provide constants or helper functions useful to other modules.
It is easier in this case to use 'import'.
Add an __init__ file to permit this.
Signed-off-by: Simon Glass <sjg@chromium.org>
This class is the new way to handle arguments in Python. Convert binman
over to use it. At the same time, introduce commands so that we can
separate out the different parts of binman functionality.
Signed-off-by: Simon Glass <sjg@chromium.org>
We need slightly different commands to run code coverage with Python 3.
Update the RunTestCoverage() function to handle this.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present the tests run one after the other using a single CPU. This is
not very efficient. Bring in the concurrencytest module and run the tests
concurrently, using one process for each CPU by default. A -P option
allows this to be overridden, which is necessary for code-coverage to
function correctly.
This requires fixing a few tests which are currently not fully
independent.
At some point we might consider doing this across all pytests in U-Boot.
There is a pytest version that supports specifying the number of processes
to use, but it did not work for me.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present only binman has the logic for determining Python test coverage
but this is useful for other tools also. Move it out into a separate file
so it can be used by other tools.
Signed-off-by: Simon Glass <sjg@chromium.org>
