Commit graph

111 commits

Author SHA1 Message Date
Heinrich Schuchardt
38b1b79021 test/py: add a test calling the EFI selftest
A Python test script is provided that runs the EFI selftest
if CONFIG_CMD_EFI_SELFTEST=y.

Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
Signed-off-by: Alexander Graf <agraf@suse.de>
2017-09-18 23:53:57 +02:00
Stephen Warren
110ba62519 test/py: gpt: make use of infra-structure
Make various changes to the GPT test:

1) Reference the disk image using an absolute path in all cases. This
allows test/py to operate correctly if it's run from a directory other
than the root of the U-Boot source tree.

2) Store the disk image in the teswt/py persistent data directory. This
removes the need to re-generate it every time the tests are run.

3) Execute sgdisk using u_boot_utils.run_and_log() so that its output is
captured in the test log. This allows debugging any problems running it.

4) Make the disk image a test fixture. This removes the requirement to
always run all GPT tests, and run them in order. The current code doesn't
create the disk image if e.g. just test_gpt_uuid() is executed via the
test.py -k command-line option.

5) Use @pytest.mark.buildconfigspec for all feature dependencies, rather
than manually implementing some of them.

6) Make all tests depend on sandbox, since they use the sandbox-specific
host command.

Fixes: a2f422555f ("add pytests for 'gpt guid' command in sandbox")
Fixes: c5772188ed ("add pytests for 'gpt rename' and 'gpt swap'")
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Tested-by: Tom Rini <trini@konsulko.com>
2017-09-15 22:33:41 -04:00
Alison Chaiken
c5772188ed add pytests for 'gpt rename' and 'gpt swap'
Add unit tests for the 'gpt rename' and 'gpt swap' commands that
rely on the block device created by test/py/make_test_disk.py.
Add CONFIG_CMD_GPT_RENAME to the sandbox_defconfig.  Remove the
testdisk.raw test device at the end of the tests.

Signed-off-by: Alison Chaiken <alison@peloton-tech.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2017-09-14 21:32:59 -04:00
Alison Chaiken
a2f422555f add pytests for 'gpt guid' command in sandbox
Run unit tests for the 'gpt guid' command, making use of the block
device created by test/py/make_test_disk.py.  Remove this device at
the end of the tests.

Signed-off-by: Alison Chaiken <alison@peloton-tech.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2017-09-14 21:32:57 -04:00
Simon Glass
3a37aee30f test: Move the FIT test into the correct place
Move this test so that it will run when 'make tests' is used.

Signed-off-by: Simon Glass <sjg@chromium.org>
2017-08-13 15:17:36 -04:00
Tom Rini
845e10d165 tests: test_dfu.py: Add example udev rule for host_usb_dev_node
If one does not already have a rule to create a custom device node when
a given device enumerates it can be useful to have udev create a
bus path based node to the entry in /dev/bus/usb that was just
enumerated.  Given that DFU itself does not require a /dev entry it is a
good idea to provide a rule that will generate one.

Signed-off-by: Tom Rini <trini@konsulko.com>
2017-07-22 15:36:16 -04:00
Simon Glass
2028cc59f7 sandbox: Stop printing platdata at the start of SPL
Currently we have code which prints out platform data at the start of SPL.
Now that we have tests for dtoc this is probably not necessary. Drop it.
Update test_ofplatdata to check for empty output since it is useful to
check that sandbox_spl works as expected.

Signed-off-by: Simon Glass <sjg@chromium.org>
2017-07-11 10:08:20 -06:00
Michal Simek
5bdb317065 test: py: hush: Add echo dependency
Some tests depends on echo command to be present.

Reported-by: Stephen Warren <swarren@nvidia.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>
2017-06-05 14:13:02 -04:00
Michal Simek
a5b548138b test: py: Use global pytestmark for hush tests
All tests in test_hush_if_test depends on hush parser to be
present. This patch simplify test dependencies by using global
pytestmark.

Reported-by: Stephen Warren <swarren@nvidia.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Acked-by: Stephen Warren <swarren@nvidia.com>
2017-06-05 14:13:02 -04:00
Michal Simek
6b83c38d7a test: py: Add cmd_echo dependency
There is missing dependency on echo command. Mark tests which requires
echo.

Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Reviewed-by: Stephen Warren <swarren@nvidia.com>
2017-05-22 07:29:55 -04:00
Alexander Graf
0e4e38ae38 travis: Add efi_loader grub2 test
We have all the building blocks now to run arbitrary efi applications
in travis. The most important one out there is grub2, so let's add
a simple test to verify that grub2 still comes up.

Signed-off-by: Alexander Graf <agraf@suse.de>
2016-11-27 09:53:40 -05:00
Alexander Graf
95b62b2e28 efi_loader: Allow to compile helloworld.efi w/o bundling it
Today we can compile a self-contained hello world efi test binary that
allows us to quickly verify whether the EFI loader framwork works.

We can use that binary outside of the self-contained test case though,
by providing it to a to-be-tested system via tftp.

This patch separates compilation of the helloworld.efi file from
including it in the u-boot binary for "bootefi hello". It also modifies
the efi_loader test case to enable travis to pick up the compiled file.
Because we're now no longer bloating the resulting u-boot binary, we
can enable compilation always, giving us good travis test coverage.

Signed-off-by: Alexander Graf <agraf@suse.de>
Reviewed-by: Tom Rini <trini@konsulko.com>
2016-11-27 09:53:39 -05:00
Alexander Graf
4ca4b265ad tests: Add efi_loader hello world test
Now that we have working network tests and a hello world efi application
built inside our tree, we can automatically test that efi binary running
inside of U-Boot.

Signed-off-by: Alexander Graf <agraf@suse.de>
Reviewed-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>
2016-11-26 15:50:53 -05:00
Alexander Graf
1bce3ad5f3 tests: net: Offset downloads to 4MB
The network test currently downloads files at 0MB offset of RAM start.
This works for most ARM systems, but x86 has weird memory layout constraints
on the first MB of RAM.

To not get caught into any of these, let's add a 4MB pad from start
of RAM to the default memory offset.

Signed-off-by: Alexander Graf <agraf@suse.de>
Reviewed-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>
2016-11-26 15:50:52 -05:00
Tom Rini
2b2c6e51e7 test/py/tests/test_sleep.py: Add check for CONFIG_CMD_MISC
We can only run this command if the sleep command is enabled and that
depends on CONFIG_CMD_MISC

Signed-off-by: Tom Rini <trini@konsulko.com>
2016-10-14 19:12:31 -04:00
Guillaume GARDET
6a2981a713 test: add NFS download test
Add a NFS download test, based on TFTP test.
Tested on i.MX6 SabreLite board.

Signed-off-by: Guillaume GARDET <guillaume.gardet@free.fr>

Cc: Tom Rini <trini@konsulko.com>
Cc: Joe Hershberger <joe.hershberger@ni.com>
Cc: Stephen Warren <swarren@wwwdotorg.org>
Cc: Simon Glass <sjg@chromium.org>
Reviewed-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Joe Hershberger <joe.hershberger@ni.com>
2016-10-13 12:25:36 -05:00
Tom Rini
de4be9ec17 test/py/tests/test_vboot.py: Add check that we boot the image
Make sure that when we're telling bootm to boot an image, and we expect
the image to boot we get the output from sandbox that we attempted to
run Linux and that U-Boot completed its job.

Cc: Simon Glass <sjg@chromium.org>
Cc: Stephen Warren <swarren@nvidia.com>
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Acked-by: Stephen Warren <swarren@nvidia.com>
2016-09-22 11:34:58 -04:00
Simon Glass
f6d34651d8 test: Adjust run_command_list() to return a list of strings
Return one string for each command that was executed. This seems cleaner.

Suggested-by: Teddy Reed <teddy.reed@gmail.com>
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Stephen Warren <swarren@nvidia.com>
2016-08-05 20:55:23 -04:00
Simon Glass
27c087d58a test: Add a function to restart U-Boot
Add a proper function for this rather than using internal functions. Use it
in the single call site.

Also, do a restart at the end of the vboot test to reset to the normal
device tree.

Signed-off-by: Simon Glass <sjg@chromium.org>
Suggested-by: Stephen Warren <swarren@nvidia.com>
2016-08-05 20:55:22 -04:00
Simon Glass
851271a71a test: vboot: Put each test variant in its own section
Use 'cons.log.section' feature to split up the test output. This makes it
easier to read.

Suggested-by: Stephen Warren <swarren@nvidia.com>

Signed-off-by: Simon Glass <sjg@chromium.org>
2016-08-05 20:55:22 -04:00
Simon Glass
ac9a23cffc test: Rename sha to sha_algo and pass it around
Rename this argument and pass it to each function that needs it, instead of
making it global.

Suggested-by: Stephen Warren <swarren@nvidia.com>
Suggested-by: Teddy Reed <teddy.reed@gmail.com>

Signed-off-by: Simon Glass <sjg@chromium.org>
2016-08-05 20:55:21 -04:00
Simon Glass
ec70f8a911 test: Drop the cmd() function
Instead of this, use the existing run_and_log() function, enhanced to
support a command string as well as a list of arguments.

Suggested-by: Stephen Warren <swarren@nvidia.com>

Signed-off-by: Simon Glass <sjg@chromium.org>
2016-08-05 20:55:20 -04:00
Simon Glass
72f5226894 test: Fix typos in comments
Fix some typos in various files introduced with the vboot test conversion.

Reported-by: Teddy Reed <teddy.reed@gmail.com>

Signed-off-by: Simon Glass <sjg@chromium.org>
2016-08-05 20:55:20 -04:00
Simon Glass
bcbd0c8fe1 test: Fix typos and tidy up
Fix review comments that were missed at the time. Also explain why we need
to regenerate the device tree for each test.

Reported-by: Teddy Reed <teddy.reed@gmail.com>
Suggested-by: Stephen Warren <swarren@nvidia.com>
Signed-off-by: Simon Glass <sjg@chromium.org>
Fixes: f6349c3c (test: Add a README)
2016-08-05 20:55:19 -04:00
Michal Simek
04a4786c7c test/py: vboot can be run only at Sandbox
Getting this error:
Zynq> sb load hostfs - 100
/home/monstr/data/disk/u-boot/build-zynq_zc706/test.fit
Unknown command 'sb' - try 'help'

because sb command is present only for Sandbox
obj-$(CONFIG_SANDBOX) += host.o

that's why mark this test to be run only at Sandbox

Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-07-22 14:52:04 -04:00
Stephen Warren
c9ba60c438 test/py: use absolute dts path in vboot test
Without this, the test fails if the test is run with a cwd other than the
root of the U-Boot source tree.

Fixes: 8729d58259 ("test: Convert the vboot test to test/py")
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2016-07-22 14:46:24 -04:00
Simon Glass
f60d0603ed test: Adjust the of-platdata test run condition
This should be spl_of_platdata, since otherwise it will try to run on boards
that don't support of-platdata.

Signed-off-by: Simon Glass <sjg@chromium.org>
2016-07-16 21:01:44 -04:00
Tom Rini
ebe621d5fb Merge git://git.denx.de/u-boot-dm 2016-07-15 08:06:22 -04:00
Simon Glass
b979d3d4c5 dm: Add a test for of-platdata
Add a simple test which checks that the of-platdata system is working
correctly. The sequence is as follows:

- SPL starts up and probes all the UCLASS_MISC drivers
- There are 3 of these in sandbox.dts
- Therefore there should be 3 U_BOOT_DEVICE() declarations in dt-platdata.c
- These should produce 3 sandbox_spl_test devices
- Each device prints out its platform data when probed
- This test checks for this output and compares it against expectations

Signed-off-by: Simon Glass <sjg@chromium.org>
2016-07-14 20:40:24 -06:00
Simon Glass
8729d58259 test: Convert the vboot test to test/py
Now that we have a suitable test framework we should move all tests into it.
The vboot test is a suitable candidate. Rewrite it in Python and move the
data files into an appropriate directory.

Signed-off-by: Simon Glass <sjg@chromium.org>
2016-07-14 18:22:40 -04:00
Stephen Warren
a82642f398 test/py: fix CONFIG_ tests
Some CONFIG_ variables were recently renamed, but test/py wasn't updated
to match. This causes some tests to be skipped. Fix test/py so the tests
are run.

Fixes: 1163625898 ("Rename reset to sysreset")
Fixes: f1f9d4fac5 ("hush: complete renaming CONFIG_SYS_HUSH_PARSER to CONFIG_HUSH_PARSER")
Cc: Masahiro Yamada <yamada.masahiro@socionext.com>
Signed-off-by: Stephen Warren <swarren@nvidia.com>
2016-07-08 12:47:58 -04:00
Stephen Warren
7a8f886558 test/py: fix printenv signon message disable code
CONFIG_VERSION_VARIABLE isn't always defined, so we can't simply look up
its value directly, or an exception will occur if it isn't defined.
Instead, we must use .get() to supply a default value if the variable
isn't defined.

Fixes: da37f006e7 ("tests: py: disable main_signon check for printenv cmd")
Acked-by: Heiko Schocher <hs@denx.de>
Signed-off-by: Stephen Warren <swarren@nvidia.com>
2016-06-19 09:50:56 -04:00
Heiko Schocher
da37f006e7 tests: py: disable main_signon check for printenv cmd
if CONFIG_VERSION_VARIABLE is set, the U-Boot environment
contains a "vers" variable with the current U-Boot version
string. If now "printenv" is called, test/py fails as it
detects the main_sign string, which is in this case correct.

So check only the main_sign as an error, if CONFIG_VERSION_VARIABLE
is not set.

Signed-off-by: Heiko Schocher <hs@denx.de>
2016-06-09 13:53:05 -04:00
Stephen Warren
daa69f5f5d test/py: dfu: wait for USB device to go away at boot
It can take a while for a host machine to notice that a USB device has
disconnected, and process the change. At the end of the DFU test, we wait
up to 10 seconds for this to happen. This change makes the test wait the
same (up to) 10 seconds at the start of the test for any previously active
USB device-mode session to be cleaned up. Such as session might have been
used to download U-Boot into memory for example; this is certainly true
on my Tegra test systems. This changes should solve the DFU test
intermittency issues I've been seeing on some Tegra devices.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
2016-05-06 10:10:53 -04:00
Lukasz Majewski
f3a87f5b79 tests: py: dfu: Provide functionality to set test and dummy files alt settings
After concatenation of "dfu_alt_info" variable from "dfu_alt_boot" and
"dfu_alt_system" it may happen that test and dummy files alt settings
are different than default 0 and 1.

This patch provides the ability to set different values for them.

Signed-off-by: Lukasz Majewski <l.majewski@samsung.com>
Acked-by: Stephen Warren <swarren@nvidia.com>
---
Changes for v3:
- replace variables declarations with ones read from configuration file
- remove not necessary str() conversion at DFU host command generation

Changes for v2:
- generate "alt_info" automatically
- use file names as alt settings instead of numerical values
- extend in-code documentation
2016-04-25 17:56:30 +02:00
Lukasz Majewski
8eb3752446 tests: py: dfu: Add functionality to set different u-boot's dfu env variable
By default (on almost all systems) the dfu env variable, which defines
available alt settings, is named as "dfu_alt_info".

However on some platforms (i.e. Odroid XU3), the 'dfu_alt_info' is concatenated
from other variables - namely 'dfu_alt_boot' and 'dfu_alt_system' at run time
(when one types 'dfu 0 mmc 0' for first time).

'dfu_alt_boot' describes alt settings which depend on boot medium - for example
boot loader's LBA sectors which are different on eMMC and SD card because of e.g.
MBR/GPT.

'dfu_alt_system' describes board agnostic alt settings - like rootfs, kernel.
On such system we can only append/modify this env variable.

Because of the above, we must have way to modify other than "dfu_ale_info"
variable to perform tests.

Signed-off-by: Lukasz Majewski <l.majewski@samsung.com>
Acked-by: Stephen Warren <swarren@nvidia.com>
---
Changes for v3:
- None

Changes for v2:
- Rewrite of "alt_info_env_name" variable description
- Use of get() method on python's dictionary to easily obtain default
  value
2016-04-25 17:56:30 +02:00
Lukasz Majewski
c6eb899c4d tests: py: dfu: Add variables to store dfu alt numbers for test and dummy files
This patch replaces hardcoded (i.e. 0 and 1) values passed to dfu_{read|write}
with variables.

Signed-off-by: Lukasz Majewski <l.majewski@samsung.com>
Acked-by: Stephen Warren <swarren@nvidia.com>
---
Changes for v3:
- Replace per module global variables with ones defined inside a function
Changes for v2:
- None
2016-04-25 17:56:29 +02:00
Michal Simek
3ba1352b97 test/py: Add support for loading image via tftp to specified location
For example this setting:

env__net_tftp_readable_file = {
    "fn": "ep108/image.ub",
    "addr": 0x10000000,
    "size": 25846296,
    "crc32": "b726f9de",
}

Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Acked-by: Stephen Warren <swarren@nvidia.com>
2016-04-11 20:48:22 -04:00
Stephen Warren
24862c640e test/py: skip tests that require large CONFIG_SYS_MAXARGS
test_hush_if_test.py executes commands that require large values of
CONFIG_SYS_MAXARGS. Detect cases where the configured value is too low
and skip those tests.

Ideally, this would be implemented inside console.run_command(). However,
the command passed to that function is already a completely formed string,
and determining its argument count usage would require splitting commands
at ;, handling quoting to deal with arguments containing spaces, etc. Even
passing the command as a list wouldn't solve all these issues, since we'd
still need to split commands on ; and deal with cases like "if test ..."
which consumes 0 of the argument count.

Signed-off-by: Stephen Warren <swarren@wwwdotorg.org>
Acked-by: Simon Glass <sjg@chromium.org>
2016-02-26 08:42:12 -05:00
Stephen Warren
1cd85f571d test/py: run all "ut" subtests
Invoke each "ut"-based unit test as a separate pytest.

Now that the DM unit test runs under test/py, remove the manual shell
script that invokes it.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org> # v2, on sandbox
2016-02-15 20:58:26 +00:00
Stephen Warren
89ab841088 test/py: support running sandbox under gdbserver
Implement command--line option --gdbserver COMM, which does two things:

a) Run the sandbox process under gdbserver, using COMM as gdbserver's
   communication channel.

b) Disables all timeouts, so that if U-Boot is halted under the debugger,
   tests don't fail. If the user gives up in the middle of a debugging
   session, they can simply CTRL-C the test script to abort it.

This allows easy debugging of test failures without having to manually
re-create the failure conditions. Usage is:

Window 1:
./test/py/test.py --bd sandbox --gdbserver localhost:1234

Window 2:
gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234'

When using this option, it likely makes sense to use pytest's -k option
to limit the set of tests that are executed.

Simply running U-Boot directly under gdb (rather than gdbserver) was
also considered. However, this was rejected because:

a) gdb's output would then be processed by the test script, and likely
   confuse it causing false failures.

b) pytest by default hides stdout from tests, which would prevent the
   user from interacting with gdb.

   While gdb can be told to redirect the debugee's stdio to a separate
   PTY, this would appear to leave gdb's stdio directed at the test
   scripts and the debugee's stdio directed elsewhere, which is the
   opposite of the desired effect. Perhaps some complicated PTY muxing
   and process hierarchy could invert this. However, the current scheme
   is simple to implement and use, so it doesn't seem worth complicating
   matters.

c) Using gdbserver allows arbitrary debuggers to be used, even those with
   a GUI. If the test scripts invoked the debugger themselves, they'd have
   to know how to execute arbitary applications. While the user could hide
   this all in a wrapper script, this feels like extra complication.

An interesting future idea might be a --gdb-screen option, which could
spawn both U-Boot and gdb separately, and spawn the screen into a newly
created window under screen. Similar options could be envisaged for
creating a new xterm/... too.

--gdbserver  currently only supports sandbox, and not real hardware.
That's primarily because the test hooks are responsible for all aspects of
hardware control, so there's nothing for the test scripts themselves can
do to enable gdbserver on real hardware. We might consider introducing a
separate --disable-timeouts option to support use of debuggers on real
hardware, and having --gdbserver imply that option.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
2016-02-08 10:22:39 -05:00
Stephen Warren
26db3a617b test/py: dfu: allow boardenv to specify test sizes
Allow the env__dfu_configs boardenv data to specify the set of DFU
transfer sizes to test. Manually specifying test sizes is useful if you
wish to test multiple DFU configurations (e.g. SD card ext4 filesystem, SD
card whole raw partition, RAM, etc.), but don't want to test every
single transfer size on each, to avoid bloating the overall time taken by
testing. If the boardenv doesn't specify a set of sizes, the built-in list
is used as a default, preserving backwards-compatibility.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:24 -07:00
Stephen Warren
db261f0076 test/py: fix a couple typos in comments
s/updata/update/.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:24 -07:00
Stephen Warren
d20e5e976f test/py: Provide custom IDs when parametrizing tests
When pytest generates the name for parametrized tests, simple parameter
values (ints, strings) get used directly, but more complex values such
as dicts are not handled. This yields test names such as:

    dfu[env__usb_dev_port0-env__dfu_config0]
    dfu[env__usb_dev_port0-env__dfu_config1]

Add some code to extract a custom fixture ID from the fixture values, so
that we end up with meaningful names such as:

    dfu[micro_b-emmc]
    dfu[devport2-ram]

If the boardenv file doesn't define custom names, the code falls back to
the old algorithm.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:24 -07:00
Stephen Warren
a2ec560647 test/py: Quote consistency
When converting test/py from " to ', I missed a few places (or added a
few inconsistencies later). Fix these.

Note that only quotes in code are converted; double-quotes in comments
and HTML are left as-is, since English and HTML use " not '.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:24 -07:00
Stephen Warren
e8debf394f test/py: use " for docstrings
Python's coding style docs indicate to use " not ' for docstrings.

test/py has other violations of the coding style docs, since the docs
specify a stranger style than I would expect, but nobody has complained
about those yet:-)

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:24 -07:00
Stephen Warren
56382a81f3 test/py: make net test aware of USB and PCI enumeration
The existing net test executes a list of commands supplied by boardenv
variable env__net_pre_commands. The idea was that boardenv would know
whether the Ethernet device was attached to USB, PCI, ... and hence was
the best place to put any commands required to probe the device.

However, this approach doesn't scale well when attempting to use a single
boardenv across multiple branches of U-Boot, some of which require "pci
enum" to enumerate PCI and others of which don't, or don't /yet/ simply
because various upstream changes haven't been merged down.

This patch updates the test to require that the boardenv state which HW
features are required for Ethernet to work, and lets the test itself map
that knowledge to the set of commands to execute. Since this mapping is
part of the test script, which is part of the U-Boot code/branch, this
approach is more scalable. It also feels cleaner, since again boardenv
is only providing data, rather than test logic.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:23 -07:00
Stephen Warren
be1df82656 test/py: dfu: error out if USB device already exists
The DFU test requests U-Boot configure its USB controller in device mode,
then waits for the host machine to enumerate the USB device and create a
device node for it. However, this wait can be fooled if the USB device
node already exists before the test starts, e.g. if some previous software
stack already configured the USB controller into device mode and never
de-configured it. This "previous software stack" could even be another
test/py test, if U-Boot's own USB teardown does not operate correctly. If
this happens, dfu-util may be run before U-Boot is ready to serve DFU
commands, which may cause false test failures.

Enhance the dfu test to fail if the device node exists before it is
expected to.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:23 -07:00
Stephen Warren
e5bb279f82 test/py: add a networking test
This tests:
- dhcp (if indicated by boardenv file).
- Static IP network setup (if provided by boardenv file).
- Ping.
- TFTP get.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:23 -07:00
Stephen Warren
0526610334 test/py: move find_ram_base() into u_boot_utils
find_ram_base() is a shared utility function, not a core part of the
U-Boot console interaction.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:23 -07:00
Stephen Warren
f5d196d03e test/py: add DFU test
Add a test of DFU functionality to the Python test suite. The test
starts DFU in U-Boot, waits for USB device enumeration on the host,
executes dfu-util multiple times to test various transfer sizes, many
of which trigger USB driver edge cases, and finally aborts the DFU
command in U-Boot.

This test mirrors the functionality previously available via the shell
scripts in test/dfu, and hence those are removed too.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Lukasz Majewski <l.majewski@samsung.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:23 -07:00
Stephen Warren
d054f4c2cb test/py: ums: add filesystem-based testing
Enhance the UMS test to optionally mount a partition and read/write a file
to it, validating that the content written and read back are identical.

This enhancement is backwards-compatible; old boardenv contents that don't
define the new configuration data will cause the test code to perform as
before.

test/ums/ is deleted since the Python test now performs the same testing
that it did.

The code is also re-written to make use of the recently added utility
module, and split it up into nested functions so the overall logic of
the test process can be followed more easily without the details
cluttering the code.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Lukasz Majewski <l.majewski@samsung.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:23 -07:00
Stephen Warren
636f38d83a test/py: move U-Boot respawn trigger to the test core
Prior to this change, U-Boot was lazilly (re-)spawned if/when a test
attempted to interact with it, and no active connection existed. This
approach was simple, yet had the disadvantage that U-Boot might be
spawned in the middle of a test function, e.g. after the test had already
performed actions such as creating data files, etc. In that case, this
could cause the log to contain the sequence (1) some test logs, (2)
U-Boot's boot process, (3) the rest of that test's logs. This isn't
optimally readable. This issue will affect the upcoming DFU and enhanced
UMS tests.

This change converts u_boot_console to be a function-scoped fixture, so
that pytest attempts to re-create the object for each test invocation.
This allows the fixture factory function to ensure that U-Boot is spawned
prior to every test. In practice, the same object is returned each time
so there is essentially no additional overhead due to this change.

This allows us to remove:

- The explicit ensure_spawned() call from test_sleep, since the core now
ensures that the spawn happens before the test code is executed.

- The laxy calls to ensure_spawned() in the u_boot_console_*
implementations.

The one downside is that test_env's "state_ttest_env" fixture must be
converted to a function-scoped fixture too, since a module-scoped fixture
cannot use a function-scoped fixture. To avoid overhead, we use the same
trick of returning the same object each time.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-28 21:01:22 -07:00
Stephen Warren
1c8b4d5f4f test/py: add a test for the sleep command
Execute "sleep", and validate that it sleeps for approximately the correct
amount of time.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-20 19:06:24 -07:00
Stephen Warren
3045d7f043 test/py: test the ums command
This test invokes the "ums" command in U-Boot, and validates that a USB
storage device is enumerated on the test host system, and can be read
from.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-20 19:06:24 -07:00
Stephen Warren
cc156f3fc0 test/py: test the shell if command
Migrate all most tests from command_ut.c into the Python test system.
This allows the tests to be run against any U-Boot binary that supports
the if command (i.e. where hush is enabled) without requiring that
binary to be permanently bloated with the code from command_ut.

Some tests in command_ut.c can only be executed from C code, since they
test internal (more unit-level) features of various U-Boot APIs. The
migrated tests can all operate directly from the U-Boot console.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-20 19:06:23 -07:00
Stephen Warren
8b86c609b8 test/py: add test of basic shell functionality
This tests whether the following features of the U-Boot shell:
- Execution of a directly entered command.
- Compound commands (; delimiter).
- Quoting of arguments containing spaces.
- Executing commands from environment variables.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-20 19:06:23 -07:00
Stephen Warren
98cee89b55 test/py: test the md/mw commands
This tests whether md/mw work, and affect each-other.

Command repeat is also tested.

test/cmd_repeat.sh is removed, since the new Python-based test does
everything it used to.

Signed-off-by: Stephen Warren <swarren@wwwdotorg.org>
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-20 19:06:23 -07:00
Stephen Warren
5ab097abad test/py: add test of setenv/printenv/echo
This tests basic environment variable functionality.

Signed-off-by: Stephen Warren <swarren@wwwdotorg.org>
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-20 19:06:23 -07:00
Stephen Warren
472040d5cb test/py: test that sandbox exits when asked
Test the sandbox port's implementation of the reset command and SIGHUP
handling. These should both cause the U-Boot process to exit gracefully.

Signed-off-by: Stephen Warren <swarren@wwwdotorg.org>
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Acked-by: Simon Glass <sjg@chromium.org>
2016-01-20 19:06:23 -07:00
Stephen Warren
d201506cca test/py: Implement pytest infrastructure
This tool aims to test U-Boot by executing U-Boot shell commands using the
console interface. A single top-level script exists to execute or attach
to the U-Boot console, run the entire script of tests against it, and
summarize the results. Advantages of this approach are:

- Testing is performed in the same way a user or script would interact
  with U-Boot; there can be no disconnect.
- There is no need to write or embed test-related code into U-Boot itself.
  It is asserted that writing test-related code in Python is simpler and
  more flexible that writing it all in C.
- It is reasonably simple to interact with U-Boot in this way.

A few simple tests are provided as examples. Soon, we should convert as
many as possible of the other tests in test/* and test/cmd_ut.c too.

The hook scripts, relay control utilities, and udev rules I use for my
own HW setup are published at https://github.com/swarren/uboot-test-hooks.

See README.md for more details!

Signed-off-by: Stephen Warren <swarren@wwwdotorg.org>
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Tested-by: Michal Simek <michal.simek@xilinx.com>
Tested-by: Simon Glass <sjg@chromium.org>
Acked-by: Simon Glass <sjg@chromium.org> #v3
2016-01-20 19:06:23 -07:00