Commit graph

240 commits

Author SHA1 Message Date
Simon Glass
f8025b58f8 spi: Re-enable the SPI flash tests
These are working correctly again, so re-enable them.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Jagan Teki <jteki@openedev.com>
Tested-by: Jagan Teki <jteki@openedev.com>
2016-02-26 08:53:10 -07:00
Simon Glass
f91281b1eb trace: Improve the trace test number recognition
The awk tool can be confused by return character (ASCII 13) in its input
since it thinks there is a separate field. These can appear if the terminal
is in raw mode, perhaps due to a previous U-Boot crash with sandbox. This
is very confusing. Remove these so that the trace test passes.

Signed-off-by: Simon Glass <sjg@chromium.org>
2016-02-26 08:53:10 -07:00
Simon Glass
0edd82e244 image: Fix FIT and vboot tests to exit sandbox correctly
When used with a device tree, sandbox now requires a 'reset' controller. Add
this to the device trees so that reset works and the tests can complete.

Signed-off-by: Simon Glass <sjg@chromium.org>
Fixes: 5010d98f (sandbox: Use the reset driver to handle reset)
2016-02-26 08:53:10 -07:00
Stephen Warren
24862c640e test/py: skip tests that require large CONFIG_SYS_MAXARGS
test_hush_if_test.py executes commands that require large values of
CONFIG_SYS_MAXARGS. Detect cases where the configured value is too low
and skip those tests.

Ideally, this would be implemented inside console.run_command(). However,
the command passed to that function is already a completely formed string,
and determining its argument count usage would require splitting commands
at ;, handling quoting to deal with arguments containing spaces, etc. Even
passing the command as a list wouldn't solve all these issues, since we'd
still need to split commands on ; and deal with cases like "if test ..."
which consumes 0 of the argument count.

Signed-off-by: Stephen Warren <swarren@wwwdotorg.org>
Acked-by: Simon Glass <sjg@chromium.org>
2016-02-26 08:42:12 -05:00
Stephen Warren
38831ca3be test/py: use space to interrupt autoboot
Sending CTRL-C to QEMU's stdin aborts the process, even if stdin is being
used as a serial port (at least in the raspi2 machine with "qemu -serial
stdin"). Avoid sending CTRL-C to U-Boot to prevent it exiting.

I'd originally used CTRL-C to make sure that if the character used to
abort autoboot ended up being treated as part of a command as well, it'd
abort command entry and return the prompt to a known state. However, this
is not needed, since aborting the autoboot eats the character used to do
that.

Signed-off-by: Stephen Warren <swarren@wwwdotorg.org>
Acked-by: Simon Glass <sjg@chromium.org>
2016-02-26 08:42:11 -05:00
Michal Simek
299e5bb7c2 test/py: Add option to skip SPL signature checking
Provide user option to skip SPL signature verification for cases where
u-boot is build with SPL support but full U-Boot is also verified
without SPL.

If you want to support this feature please add env__spl_skipped = True
to your boardenv configuration file.

For example Xilinx Zynq is using this feature where the same U-Boot
binary is checked with SPL and without SPL(with FSBL).

Signed-off-by: Michal Simek <michal.simek@xilinx.com>
2016-02-25 10:21:19 -05:00
Heiko Schocher
b1309a23e0 test/py: only check for SPL signature if SPL uses serial output
check for U-Boot SPL signature only if SPL really has a serial output.
So check if CONFIG_SPL_SERIAL_SUPPORT is active in board config.

Signed-off-by: Heiko Schocher <hs@denx.de>
Tested-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Stephen Warren <swarren@nvidia.com>
2016-02-24 18:44:00 -05:00
Stephen Warren
9725543843 test/py: put "Starting U-Boot" into separate log section
The initial boot of U-Boot happens within the context of the first test
that needs to access the U-Boot console when there is no existing
connection. This keeps all activity nestled within test execution, which
fits well into the pytest model. However, this mingles the U-Boot startup
logs with the execution of some test(s), which hides find the boundary
between the two.

To solve this, wrap the "Starting U-Boot" logic into a separate log
section. If the user wishes, they can simply collapse this log section
when viewing the HTML log, to concentrate purely on the test's own
interaction.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-02-15 20:58:29 +00:00
Stephen Warren
93134e18e8 test/py: handle exceptions in console creation
u_boot_console.exec_attach.get_spawn() performs two steps:
1) Spawn a process to communicate with the serial console.
2) Reset the board so that U-Boot starts running from scratch.

Currently, if an exception happens in step (2), no cleanup is performed on
the process created in step (1). That process stays running and may e.g.
hold serial port locks, or simply continue to read data from the serial
port, thus preventing it from reaching any other process that attempts to
read from the same serial port later. While there is error cleanup code in
u_boot_console_base.ensure_spawned(), this is not triggered since the
exception prevents assignment to self.p there, and hence the exception
handler has no object to operate upon in cleanup_spawn().

Solve this by enhancing u_boot_console.exec_attach.get_spawn() to clean
up any objects it has created.

In theory, u_boot_spawn.Spawn's constructor has a similar issue, so fix
this too.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-02-15 20:58:29 +00:00
Stephen Warren
1326022c2e test/py: print summary in test order
Use lists rather than sets to record the status of tests. This causes
the test summary in the HTML file to be generated in the same order as
the tests are (or would have been) run. This makes it easier to locate
the first failed test. The log for this test might have interesting
first clues re: interaction with the environment (e.g. hardware flashing,
serial console, ...) and may help tracking down external issues.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-02-15 20:58:28 +00:00
Stephen Warren
1235c79182 test/py: fix CONFIG_SPL test
The Python ini file parser that's used to parse .config converts all keys
to lower-case. Hence, all queries against the results must use lower-case.
Fix u_boot_console.ensure_spawned() to test CONFIG_SPL correctly, or the
connection will fail for boards that have SPL.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-02-15 20:58:28 +00:00
Stephen Warren
06088b0473 test/py: don't import pexpect
The code replaced pexpect with custom code long ago. Don't import the
unused module so it doesn't need to be installed.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-02-15 20:58:27 +00:00
Stephen Warren
d70facf89c test/py: add docs for gdbserver and pytest options
Add documentation describing the new --gdbserver feature, and some common
pytest options.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-02-15 20:58:27 +00:00
Stephen Warren
1cd85f571d test/py: run all "ut" subtests
Invoke each "ut"-based unit test as a separate pytest.

Now that the DM unit test runs under test/py, remove the manual shell
script that invokes it.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org> # v2, on sandbox
2016-02-15 20:58:26 +00:00
Stephen Warren
c82ce04a3f test/py: capture the entire U-Boot version at boot
The existing regex simply ensures that the captured version string doesn't
go past the end of a line. We really want to grab as much as possible. Do
this by explicitly including a ) character at the end of the regex to
match the last character of the version test.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
2016-02-09 15:41:19 -07:00
Stephen Warren
d8926811fd test/py: fix off-by-one error in spawn matching code
A regex match object's .end() value is already the index after the match,
not the index of the last character in the match, so there's no need to
add 1 to point past the match.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
2016-02-09 15:41:19 -07:00
Stephen Warren
83357fd5c2 test/py: HTML awesome!
Implement three improvements to the HTML log file:
- Ability to expand/contract sections. All passing sections are contracted
  at file load time so the user can concentrate on issues requiring
  action.
- The overall status report is copied to the top of the log for easy
  access.
- Add links from the status report to the test logs, for easy navigation.

This all relies on Javascript and the jquery library. If the user doesn't
have Javascript enabled, or jquery can't be downloaded, the log should
look and behave identically to how it did before this patch.

A few notes on the diff:

- A few more 'with log.section("xxx")' were added, so that all stream
  blocks are kept within a section block for consistent HTML entity
  nesting structure. This changed indentation in a few places, making
  the diff look slightly larger.
- HTML entity IDs are cleaned up. We assign simple incrementing integer
  IDs now, rather than using mangled test names which were possibly
  invalid.
- Sections and streams now use common CSS class names (in addition to the
  current separate class names) to more easily share the new behaviour.
  This also reduces the CSS file size since rules don't need to be
  duplicated.
- An "OK" status is logged after some external command executions so that
  make and flash steps are auto-contracted at log file load time, assuming
  they passed.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
2016-02-09 15:41:19 -07:00
Stephen Warren
ac99831b7d test/py: exit(1) if there are problems running py.test
The test/py/test.py wrapper script catches exceptions thrown when
exec()ing py.test in order to print a helpful error message. However,
the exception handling code squashes the exception and so the script
exits with a non-zero exit code, leading callers to believe that it
passed. Fix this.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
2016-02-09 15:41:19 -07:00
Stephen Warren
89ab841088 test/py: support running sandbox under gdbserver
Implement command--line option --gdbserver COMM, which does two things:

a) Run the sandbox process under gdbserver, using COMM as gdbserver's
   communication channel.

b) Disables all timeouts, so that if U-Boot is halted under the debugger,
   tests don't fail. If the user gives up in the middle of a debugging
   session, they can simply CTRL-C the test script to abort it.

This allows easy debugging of test failures without having to manually
re-create the failure conditions. Usage is:

Window 1:
./test/py/test.py --bd sandbox --gdbserver localhost:1234

Window 2:
gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234'

When using this option, it likely makes sense to use pytest's -k option
to limit the set of tests that are executed.

Simply running U-Boot directly under gdb (rather than gdbserver) was
also considered. However, this was rejected because:

a) gdb's output would then be processed by the test script, and likely
   confuse it causing false failures.

b) pytest by default hides stdout from tests, which would prevent the
   user from interacting with gdb.

   While gdb can be told to redirect the debugee's stdio to a separate
   PTY, this would appear to leave gdb's stdio directed at the test
   scripts and the debugee's stdio directed elsewhere, which is the
   opposite of the desired effect. Perhaps some complicated PTY muxing
   and process hierarchy could invert this. However, the current scheme
   is simple to implement and use, so it doesn't seem worth complicating
   matters.

c) Using gdbserver allows arbitrary debuggers to be used, even those with
   a GUI. If the test scripts invoked the debugger themselves, they'd have
   to know how to execute arbitary applications. While the user could hide
   this all in a wrapper script, this feels like extra complication.

An interesting future idea might be a --gdb-screen option, which could
spawn both U-Boot and gdb separately, and spawn the screen into a newly
created window under screen. Similar options could be envisaged for
creating a new xterm/... too.

--gdbserver  currently only supports sandbox, and not real hardware.
That's primarily because the test hooks are responsible for all aspects of
hardware control, so there's nothing for the test scripts themselves can
do to enable gdbserver on real hardware. We might consider introducing a
separate --disable-timeouts option to support use of debuggers on real
hardware, and having --gdbserver imply that option.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
2016-02-08 10:22:39 -05:00
Simon Glass
a108082d4d video: test: Adjust order of file closure
Close the file earlier to hopefully fix a Coverity error.

Reported-by: Coverity (CID: 134901)

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>
2016-02-06 13:57:15 +01:00
Simon Glass
5674ead7d2 video: test: Add console tests for truetype
This adds tests for the different character types, line wrap, scrolling and
backspace.

Signed-off-by: Simon Glass <sjg@chromium.org>
2016-01-30 10:58:37 +01:00
Simon Glass
8df8dad528 video: sandbox: Enable truetype fonts for sandbox
Enable this feature so that truetype fonts can be used on the sandbox
console. Update the tests to select the normal/rotated console when needed.

Signed-off-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Anatolij Gustschin <agust@denx.de>
2016-01-30 10:58:26 +01:00
Simon Glass
5508f10ac9 video: Handle the 'bell' character
This can be sent when to many characters are entered. Make sure it is
ignored and does not cause a character to be displayed.

Signed-off-by: Simon Glass <sjg@chromium.org>
2016-01-30 10:55:02 +01:00
Simon Glass
f266178698 video: Use fractional units for X coordinates
With anti-aliased fonts we need a more fine-grained horizontal position
than a single pixel. Characters can be positioned to start part-way through
a pixel, with anti-aliasing (greyscale edges) taking care of the visual
effect.

To cope with this, use fractional units (1/256 pixel) for horizontal
positions in the text console.

Signed-off-by: Simon Glass <sjg@chromium.org>
[agust: rebased]
Signed-off-by: Anatolij Gustschin <agust@denx.de>
2016-01-30 10:53:26 +01:00
Stephen Warren
26db3a617b test/py: dfu: allow boardenv to specify test sizes
Allow the env__dfu_configs boardenv data to specify the set of DFU
transfer sizes to test. Manually specifying test sizes is useful if you
wish to test multiple DFU configurations (e.g. SD card ext4 filesystem, SD
card whole raw partition, RAM, etc.), but don't want to test every
single transfer size on each, to avoid bloating the overall time taken by
testing. If the boardenv doesn't specify a set of sizes, the built-in list
is used as a default, preserving backwards-compatibility.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:24 -07:00
Stephen Warren
db261f0076 test/py: fix a couple typos in comments
s/updata/update/.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:24 -07:00
Stephen Warren
d27f2fc1e1 test/py: run sandbox in source directory
Some unit tests expect the cwd of the sandbox process to be the root
of the source tree. Ensure that requirement is met.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:24 -07:00
Stephen Warren
77bcb22d77 test/py: pass test DTB to sandbox
This is required for at least "ut dm" to operate correctly.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:24 -07:00
Stephen Warren
78b39cc3e1 test/py: correctly log xfail/xpass tests
Tests can complete in passed, skipped, xpass, xfailed, or failed, states.
Currently the U-Boot log generation code doesn't handle the xfailed or
xpass states since they aren't used. Add support for the remaining states.
Without this, tests that xfail end up being reported as skipped.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:24 -07:00
Stephen Warren
9129d9f5fd test/py: detect another "bad pattern" in console output
Many error situations in U-Boot print the message:
    ### ERROR ### Please RESET the board ###

Add this to the list of bad patterns the test system detects. One
practical advantage of this change is to detect the case where sandbox
is told to use a particular DTB file, and the file cannot be opened.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:24 -07:00
Stephen Warren
0c6189b5d6 test/py: check for bad patterns everywhere we wait
Currently, bad patterns are only honored when executing a shell command.
Other cases, such as the initial boot-up of U-Boot or when interacting
with command output rather than gathering all output prior to the shell
prompt, do not currently look for bad patterns in console output. This
patch makes sure that bad patterns are honored everywhere.

One benefit of this change is that if U-Boot sandbox fails to start up,
the error message it emits can be caught immediately, rather than relying
on a (long) timeout when waiting for the expected signon message and/or
command prompt.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:24 -07:00
Stephen Warren
e4119ebb6d test.py: calculate bad patterns on change only
A future patch will use the bad_patterns array in multiple places. Rather
than duplicating the code to calculate it, or even sharing it in a
function and simply calling it redundantly when nothing has changed, only
re-calculate the list when some change is made to it. This reduces work.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:24 -07:00
Stephen Warren
44ac762b14 test/py: fix spawn.expect multiple match handling
Multiple patterns may be passed to spawn.expect(). The pattern which
matches at the earliest position should be designated as the match. This
aspect works correctly. When multiple patterns match at the same position,
priority should be given the the earliest entry in the list of patterns.
This aspect does not work correctly. This patch fixes it.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:24 -07:00
Stephen Warren
26e1beccbe test/dm: clear unit test failure count each run
The ut command prints a test failure count each time it is executed.
This is stored in a global variable which is never reset. Consequently,
the printed failure count accumulates across runs. Fix this by clearing
the counter each time "ut" is invoked.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:24 -07:00
Stephen Warren
d20e5e976f test/py: Provide custom IDs when parametrizing tests
When pytest generates the name for parametrized tests, simple parameter
values (ints, strings) get used directly, but more complex values such
as dicts are not handled. This yields test names such as:

    dfu[env__usb_dev_port0-env__dfu_config0]
    dfu[env__usb_dev_port0-env__dfu_config1]

Add some code to extract a custom fixture ID from the fixture values, so
that we end up with meaningful names such as:

    dfu[micro_b-emmc]
    dfu[devport2-ram]

If the boardenv file doesn't define custom names, the code falls back to
the old algorithm.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:24 -07:00
Stephen Warren
a2ec560647 test/py: Quote consistency
When converting test/py from " to ', I missed a few places (or added a
few inconsistencies later). Fix these.

Note that only quotes in code are converted; double-quotes in comments
and HTML are left as-is, since English and HTML use " not '.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:24 -07:00
Stephen Warren
e8debf394f test/py: use " for docstrings
Python's coding style docs indicate to use " not ' for docstrings.

test/py has other violations of the coding style docs, since the docs
specify a stranger style than I would expect, but nobody has complained
about those yet:-)

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:24 -07:00
Stephen Warren
56382a81f3 test/py: make net test aware of USB and PCI enumeration
The existing net test executes a list of commands supplied by boardenv
variable env__net_pre_commands. The idea was that boardenv would know
whether the Ethernet device was attached to USB, PCI, ... and hence was
the best place to put any commands required to probe the device.

However, this approach doesn't scale well when attempting to use a single
boardenv across multiple branches of U-Boot, some of which require "pci
enum" to enumerate PCI and others of which don't, or don't /yet/ simply
because various upstream changes haven't been merged down.

This patch updates the test to require that the boardenv state which HW
features are required for Ethernet to work, and lets the test itself map
that knowledge to the set of commands to execute. Since this mapping is
part of the test script, which is part of the U-Boot code/branch, this
approach is more scalable. It also feels cleaner, since again boardenv
is only providing data, rather than test logic.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:23 -07:00
Stephen Warren
be1df82656 test/py: dfu: error out if USB device already exists
The DFU test requests U-Boot configure its USB controller in device mode,
then waits for the host machine to enumerate the USB device and create a
device node for it. However, this wait can be fooled if the USB device
node already exists before the test starts, e.g. if some previous software
stack already configured the USB controller into device mode and never
de-configured it. This "previous software stack" could even be another
test/py test, if U-Boot's own USB teardown does not operate correctly. If
this happens, dfu-util may be run before U-Boot is ready to serve DFU
commands, which may cause false test failures.

Enhance the dfu test to fail if the device node exists before it is
expected to.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:23 -07:00
Stephen Warren
e787a58fe2 test/py: make crash detection more robust
test/py contains logic to detect the target crashing and rebooting by
searching the console output for a U-Boot signon message, which will
presumably be emitted when the system boots after the crash/reset.

Currently, this logic only searches for the exact signon message that
was printed by the U-Boot version under test, upon the assumption that
binary is written into flash, and hence will be the version booted after
any reset. However, this is not a valid assumption; some test setups
download the U-Boot-under-test into RAM and boot it from there, and in
such a scenario an arbitrary U-Boot version may be located in flash and
hence run after any reset.

Fix the reset detection logic to match any U-Boot signon message. This
prevents false negatives.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:23 -07:00
Stephen Warren
e5bb279f82 test/py: add a networking test
This tests:
- dhcp (if indicated by boardenv file).
- Static IP network setup (if provided by boardenv file).
- Ping.
- TFTP get.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:23 -07:00
Stephen Warren
0526610334 test/py: move find_ram_base() into u_boot_utils
find_ram_base() is a shared utility function, not a core part of the
U-Boot console interaction.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:23 -07:00
Stephen Warren
f5d196d03e test/py: add DFU test
Add a test of DFU functionality to the Python test suite. The test
starts DFU in U-Boot, waits for USB device enumeration on the host,
executes dfu-util multiple times to test various transfer sizes, many
of which trigger USB driver edge cases, and finally aborts the DFU
command in U-Boot.

This test mirrors the functionality previously available via the shell
scripts in test/dfu, and hence those are removed too.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Lukasz Majewski <l.majewski@samsung.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:23 -07:00
Stephen Warren
d054f4c2cb test/py: ums: add filesystem-based testing
Enhance the UMS test to optionally mount a partition and read/write a file
to it, validating that the content written and read back are identical.

This enhancement is backwards-compatible; old boardenv contents that don't
define the new configuration data will cause the test code to perform as
before.

test/ums/ is deleted since the Python test now performs the same testing
that it did.

The code is also re-written to make use of the recently added utility
module, and split it up into nested functions so the overall logic of
the test process can be followed more easily without the details
cluttering the code.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Lukasz Majewski <l.majewski@samsung.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:23 -07:00
Stephen Warren
76b4693928 test/py: add various utility code
Add various common utility functions. These will be used by a forthcoming
re-written UMS test, and a brand-new DFU test.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:23 -07:00
Stephen Warren
3f2faf7327 test/py: optionally ignore errors from shell commands
Sometimes it's useful to run shell commands and ignore any errors. One
example might be cleanup logic; if a test-case experiences an error, the
cleanup logic might experience an error too, and we don't want that error
to mask the original error, so we want to ignore the subsequent error.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:23 -07:00
Stephen Warren
783cbcd360 test/py: log when tests send CTRL-C
Write a note to the log file when a test sends CTRL-C to U-Boot. This
makes it easier to follow what's happening in the logs, especially since
U-Boot doesn't echo the character back to its output, so there's no other
signal of what's going on.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:23 -07:00
Stephen Warren
c10eb9d39f test/py: drain console log at the end of any failed test
Tests may fail for a number of reasons, and in particular for reasons
other than a timeout waiting for U-Boot to print expected data. If the
last operation that a failed test performs is not waiting for U-Boot to
print something, then any trailing output from U-Boot during that test's
operation will not be logged as part of that test, but rather either
along with the next test, or even thrown away, potentiall hiding clues
re: the test failure reason.

Solve this by explicitly draining (and hence logging) the U-Boot output
in the case of failed tests.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:23 -07:00
Stephen Warren
636f38d83a test/py: move U-Boot respawn trigger to the test core
Prior to this change, U-Boot was lazilly (re-)spawned if/when a test
attempted to interact with it, and no active connection existed. This
approach was simple, yet had the disadvantage that U-Boot might be
spawned in the middle of a test function, e.g. after the test had already
performed actions such as creating data files, etc. In that case, this
could cause the log to contain the sequence (1) some test logs, (2)
U-Boot's boot process, (3) the rest of that test's logs. This isn't
optimally readable. This issue will affect the upcoming DFU and enhanced
UMS tests.

This change converts u_boot_console to be a function-scoped fixture, so
that pytest attempts to re-create the object for each test invocation.
This allows the fixture factory function to ensure that U-Boot is spawned
prior to every test. In practice, the same object is returned each time
so there is essentially no additional overhead due to this change.

This allows us to remove:

- The explicit ensure_spawned() call from test_sleep, since the core now
ensures that the spawn happens before the test code is executed.

- The laxy calls to ensure_spawned() in the u_boot_console_*
implementations.

The one downside is that test_env's "state_ttest_env" fixture must be
converted to a function-scoped fixture too, since a module-scoped fixture
cannot use a function-scoped fixture. To avoid overhead, we use the same
trick of returning the same object each time.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:22 -07:00
Stephen Warren
d314e247e1 test/py: fix timeout to be absolute
Currently, Spawn.expect() imposes its timeout solely upon receipt of new
data, not on its overall operation. In theory, this could cause the
timeout not to fire if U-Boot continually generated output that did not
match the expected patterns.

Fix the code to additionally impose a timeout on overall operation, which
is the intended mode of operation.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Lukasz Majewski <l.majewski@samsung.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:22 -07:00