Commit graph

42 commits

Author SHA1 Message Date
Alper Nebi Yasak
880e4768c2 tools: Fix default target compile tools in Python tools
In commit 1e4687aa47 ("binman: Use target-specific tools when
cross-compiling"), a utility function was implemented to get preferred
compilation tools using environment variables like CC and CROSS_COMPILE.
Although it intended to provide custom default tools (same as those in
the global Makefile) when no relevant variables were set (for example
using "gcc" for "cc"), it is only doing so when CROSS_COMPILE is set and
returning the literal name of the tool otherwise.

Remove the check for an empty CROSS_COMPILE, which makes the function
use it as an empty prefix to the custom defaults and return the intended
executables.

Fixes: 1e4687aa47 ("binman: Use target-specific tools when cross-compiling")
Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
2021-07-06 10:38:03 -06:00
Simon Glass
5187b80617 binman: Use standard filenames for SPL/TPL devicetree
At present, before any entry expansion is done (such as a 'files' entry
expanding out to individual entries for each file it contains), we check
the binman definition (i.e. '/binman' node) to find out what devicetree
files are used in the images.

This is a pain, since the definition may change during expansion. For
example if there is no u-boot-spl-dtb entry in the definition at the start,
we assume that the SPL devicetree is not used. But if an entry later
expands to include this, then we don't notice.

In fact the flexibility provided by the current approach of checking the
definition is not really useful. We know that we can have SPL and TPL
devicetrees. We know the pathname to each, so we can simply check if the
files are present. If they are present, we can prepare them and update
them regardless of whether they are actually used. If they are not present,
we cannot prepare/update them anyway, i.e. an error will be generated.

Simplify state.Prepare() so it uses a hard-coded list of devicetree files.

Note that state.PrepareFromLoadedData() is left untouched, since in that
case we have a complete definition from the loaded file, but cannot of
course rely on the devicetree files that created it still being present.
So in that case we still check the image defitions.

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-03-26 17:03:09 +13:00
Simon Glass
6deff872cf patman: Correct lz4 compression parameters
At present on large files, lz4 uses a larger block size (e.g. 256KB) than
the 64KB supported by the U-Boot decompression implementation. Also it is
optimised for maximum compression speed, producing larger output than we
would like.

Update the parameters to correct these problems.

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-01-30 14:25:41 -07:00
Tom Rini
b11f634b1c Driver model: make some udevice fields private
Driver model: Rename U_BOOT_DEVICE et al.
 dtoc: Tidy up and add more tests
 ns16550 code clean-up
 x86 and sandbox minor fixes for of-platdata
 dtoc prepration for adding build-time instantiation
 -----BEGIN PGP SIGNATURE-----
 
 iQFFBAABCgAvFiEEslwAIq+Gp8wWVbYnfxc6PpAIreYFAl/09LURHHNqZ0BjaHJv
 bWl1bS5vcmcACgkQfxc6PpAIrebjwwf/fHRjYsAY/Yj/+y1xgo3L3sphIvQUqTDF
 KkLl+kHdV5r8W/HJULxLQcF2r7pcPEI6TAQxuj3qQ5SUvm2HviS8GHGPawDEwyht
 HgBp9VD56+HUadMfnbG//DVS73ycbL4XSKlYqpkINEejtnlttsCIawUXX5cTyGM/
 59VkgnKrKvJQRUXvYLa8MTugTs4fkPJGDqhActBk/7SP1SImj+rfalNSqA2/dx6y
 2RnPCSzB1x2231KSj+B1NgGlR3Xb8P8zgh20ijcEU/hrlXBTZyi7K7f4SJR30Efu
 LYkkuj4VbxcV/25RozR0fmknqCs0QyAI+/dql6TNtbTSPC/jAfj0jQ==
 =9kN3
 -----END PGP SIGNATURE-----

Merge tag 'dm-pull-5jan21' of git://git.denx.de/u-boot-dm into next

Driver model: make some udevice fields private
Driver model: Rename U_BOOT_DEVICE et al.
dtoc: Tidy up and add more tests
ns16550 code clean-up
x86 and sandbox minor fixes for of-platdata
dtoc prepration for adding build-time instantiation
2021-01-05 22:34:43 -05:00
Tom Rini
720620e691 Prepare v2021.01-rc5
-----BEGIN PGP SIGNATURE-----
 
 iQGzBAABCgAdFiEEGjx/cOCPqxcHgJu/FHw5/5Y0tywFAl/0YVIACgkQFHw5/5Y0
 tywtEwv/cJWlKgcSnYjuJrxwuJdauUTfXdbUgtCxOtBw/BP4dsKkbGTJPw5q5M+4
 LJJSKyksmJVTX26h1dpkzQjOpWtTDnWqm5CTIxD52oQD7pxK+zCQ9T6S+QbQD0Se
 ogHmZluzFoluxbNgo8tiO52xvMhDO3TVAzxsNDdGfkd5/tAXOHClPc34RmAkdRHU
 VsR89AKdT2q543fiUfrRZYDzdctaNWhRGXMDcJ4+QU/8hQhrpcr8EtHbF+3mWX4K
 pA01pDz150Rn4UI6S2xKEWrjSTHe55fxVj/Qj0rq9z2E/+NqGXemf5s13AR0G/z3
 PqHdVLHzDe64pbOvmyU1pVQ0aMb8vMJUnqx68SQZY3On2c+MjRWQ+7aVVaKOcPGp
 uatk6QMrggHp3Li+3yZrLBE0qPr/sNMVb7mUesdZb6lFd2VIs8siwhfeGXMS+nDI
 xePzsR43Fnn5Q5KIqqvcWUb+TTTqUDUff0wyAU8NBgCaIBIZK8h2ppS1jjnbms0I
 mr8Er2vb
 =Dfum
 -----END PGP SIGNATURE-----

Merge tag 'v2021.01-rc5' into next

Prepare v2021.01-rc5

Signed-off-by: Tom Rini <trini@konsulko.com>
2021-01-05 16:20:26 -05:00
Simon Glass
10cbd3b7da dtoc: Add an 'all' command
With upcoming changes, dtoc will output several files for different
of-platdata components.

Add a way to output all ava!ilable files at once ('all'), to the
appropriate directories, without needing to specify each one invidually.

This puts the commands in alphabetical order, so update the tests
accordingly.

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-01-05 12:26:35 -07:00
Simon Glass
479dd30f4a patman: Drop tools.ToChar() and ToChars()
This is useful anymore, since we always want to call chr() in Python 3.
Drop it and adjust callers to use chr().

Also drop ToChars() which is no-longer used.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-12-13 07:58:17 -07:00
Simon Glass
6a4ccad8e0 patman: Drop tools.ToByte()
This is not needed in Python 3. Drop it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-12-13 07:58:17 -07:00
Simon Glass
fc0056e8d5 patman: Drop unicode helper functions
We don't need these now that everything uses Python 3. Remove them and
the extra code in GetBytes() and ToBytes() too.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-12-13 07:58:17 -07:00
Simon Glass
f31e83d6cf binman: Handle tool paths containing '~' correctly
At present if CROSS_COMPILE contains a tilde, such as
~/.buildman-toolchains/gcc-7.3.0-nolibc/i386-linux/bin/i386-linux-gcc
then binman gives a confusing error:

   binman: Error 255 running '~/..buildman-toolchains/gcc-7.3.0- ...

Fix this by expanding it out before running the tool.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-11-30 13:14:43 -07:00
Alper Nebi Yasak
29cc091842 binman: Allow resolving host-specific tools from env vars
This patch lets tools.Run() use host-specific versions with the
for_host keyword argument, based on the host-specific environment
variables (HOSTCC, HOSTOBJCOPY, HOSTSTRIP, etc.).

Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2020-09-22 12:54:13 -06:00
Alper Nebi Yasak
1e4687aa47 binman: Use target-specific tools when cross-compiling
Currently, binman always runs the compile tools like cc, objcopy, strip,
etc. using their literal name. Instead, this patch makes it use the
target-specific versions by default, derived from the tool-specific
environment variables (CC, OBJCOPY, STRIP, etc.) or from the
CROSS_COMPILE environment variable.

For example, the u-boot-elf etype directly uses 'strip'. Trying to run
the tests with 'CROSS_COMPILE=i686-linux-gnu- binman test' on an arm64
host results in the '097_elf_strip.dts' test to fail as the arm64
version of 'strip' can't understand the format of the x86 ELF file.

This also adjusts some command.Output() calls that caused test errors or
failures to use the target versions of the tools they call. After this,
patch, an arm64 host can run all tests with no errors or failures using
a correct CROSS_COMPILE value.

Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2020-09-22 12:54:13 -06:00
Simon Glass
4f9f1056ec binman: Allow external binaries to be missing
Sometimes it is useful to build an image even though external binaries are
not present. This allows the build system to continue to function without
these files, albeit not producing valid images.

U-Boot does with with ATF (ARM Trusted Firmware) today.

Add a new flag to binman to request this behaviour.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
2020-07-25 14:46:57 -06:00
Simon Glass
fd70986a62 patman: Add a test that uses gitpython
It is convenient to use gitpython to create a real git repo for testing
patman's operation. Add a test for this. So far it just checks that patman
produces the right number of patches for a branch.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-24 19:25:15 -06:00
Tom Rini
7208396bbf Revert "Merge tag 'dm-pull-20jul20' of git://git.denx.de/u-boot-dm"
This reverts commit 5d3a21df66, reversing
changes made to 56d37f1c56.

Unfortunately this is causing CI failures:
https://travis-ci.org/github/trini/u-boot/jobs/711313649

Signed-off-by: Tom Rini <trini@konsulko.com>
2020-07-24 08:42:06 -04:00
Simon Glass
5f850fb9a6 binman: Allow external binaries to be missing
Sometimes it is useful to build an image even though external binaries are
not present. This allows the build system to continue to function without
these files, albeit not producing valid images.

U-Boot does with with ATF (ARM Trusted Firmware) today.

Add a new flag to binman to request this behaviour.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
2020-07-20 11:37:47 -06:00
Simon Glass
5d597584db patman: Add a test that uses gitpython
It is convenient to use gitpython to create a real git repo for testing
patman's operation. Add a test for this. So far it just checks that patman
produces the right number of patches for a branch.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-20 11:37:46 -06:00
Simon Glass
bf776679a7 patman: Move to absolute imports
At present patman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move patman to use absolute imports. This requires changes in tools which
use the patman libraries (which is most of them).

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-26 14:25:21 -06:00
Simon Glass
83a4518771 patman: Drop references to __future__
We don't need these now that the tools using Python 3. Drop them.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-26 14:25:21 -06:00
Simon Glass
3b3e3c0f6c patman: Adjust 'command' to return strings instead of bytes
At present all the 'command' methods return bytes. Most of the time we
actually want strings, so change this. We still need to keep the internal
representation as bytes since otherwise unicode strings might break over
a read() boundary (e.g. 4KB), causing errors. But we can convert the end
result to strings.

Add a 'binary' parameter to cover the few cases where bytes are needed.

Signed-off-by: Simon Glass <sjg@chromium.org>
2019-11-04 18:15:32 -07:00
Simon Glass
f514d8f23d binman: Use the Makefile for u_boot_ucode_ptr
Remove this file from git and instead build it using the Makefile.

Update tools.GetInputFilename() to support reading files from an absolute
path, so that we can read the Elf test files easily. Also make sure that
the temp directory is report in ELF tests as this was commented out.

Signed-off-by: Simon Glass <sjg@chromium.org>
2019-10-15 08:40:02 -06:00
Simon Glass
6eace39807 patman: Update command.Run() to handle failure better
At present tools are not expected to fail. If they do an exception is
raised but there is no detail about what went wrong. This makes it hard
to debug if something does actually go wrong.

Fix this by outputting both stderr and stdout on failure.

Signed-off-by: Simon Glass <sjg@chromium.org>
2019-10-15 08:40:02 -06:00
Simon Glass
3b1c0b09c9 patman: Drop binary parameter
Since cros_subprocess use bytestrings now, this feature not needed. Drop
it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2019-10-15 08:40:02 -06:00
Simon Glass
313533018d patman: Reset the output directory when it is removed
At present outdir remains set ever after the output directory has been
removed. Fix this to avoid trying to access it when it is not present.

Signed-off-by: Simon Glass <sjg@chromium.org>
2019-07-29 09:38:06 -06:00
Simon Glass
eb0f4a4cb4 binman: Support replacing data in a cbfs
At present binman cannot replace data within a CBFS since it does not
allow rewriting of the files in that CBFS. Implement this by using the
new WriteData() method to handle the case.

Add a header to compressed data so that the amount of compressed data can
be determined without reference to the size of the containing entry. This
allows the entry to be larger that the contents, without causing errors in
decompression. This is necessary to cope with a compressed device tree
being updated in such a way that it shrinks after the entry size is
already set (an obscure case). It is not used with CBFS since it has its
own metadata for this. Increase the number of passes allowed to resolve
the position of entries, to handle this case.

Add a test for this new logic.

Signed-off-by: Simon Glass <sjg@chromium.org>
2019-07-29 09:38:06 -06:00
Simon Glass
9f297b09c0 binman: Add a bit of logging in entries when packing
Use the new logging feature to log information about progress with
packing. This is useful to see how binman is figuring things out.

Also update elf.py to use the same feature.

Signed-off-by: Simon Glass <sjg@chromium.org>
2019-07-29 09:38:05 -06:00
Simon Glass
1cfdfc064a patman: Add a function to write ifwitool
This tool has quite a few arguments and options, so put the functionality
in a function so that we call it from one place and hopefully get it
right.

Signed-off-by: Simon Glass <sjg@chromium.org>
2019-07-24 12:53:46 -07:00
Simon Glass
07d9e70bf9 patman: Add functions to compress and decompress data
Add utility functions to compress and decompress using lz4 and lzma
algorithms. In the latter case these use the legacy lzma support favoured
by coreboot's CBFS.

No tests are provided as these functions will be tested by the CBFS
tests in a separate patch.

Signed-off-by: Simon Glass <sjg@chromium.org>
2019-07-23 20:27:57 -07:00
Simon Glass
c22b8cfc1d patman: Add a way to set the search path for tools
Sometimes tools can be located by looking in other locations. Add a way to
direct the search.

Signed-off-by: Simon Glass <sjg@chromium.org>
2019-07-23 20:27:57 -07:00
Simon Glass
3c47e4105e patman: Allow reading files in text mode
While reading files in binary mode is the norm, sometimes we want to use
text mode. Add an optional parameter to handle this.

Signed-off-by: Simon Glass <sjg@chromium.org>
2019-07-10 16:52:58 -06:00
Simon Glass
f6b64815dd dtoc: Use byte type instead of str in fdt
In Python 3 bytes and str are separate types. Use bytes to ensure that
the code functions correctly with Python 3.

Signed-off-by: Simon Glass <sjg@chromium.org>
2019-07-10 16:52:58 -06:00
Simon Glass
2b6ed5e92e dtoc: Updates BytesToValue() for Python 3
The difference between the bytes and str types in Python 3 requires a
number of minor changes to this function. Update it to handle the input
data using the 'bytes' type. Create two useful helper functions which can
be used by other modules too.

Signed-off-by: Simon Glass <sjg@chromium.org>
2019-07-10 16:52:58 -06:00
Simon Glass
513eace47d patman: Move unicode helpers to tools
Create helper functions in the tools module to deal with the differences
between unicode in Python 2 (where we use the 'unicode' type) and Python 3
(where we use the 'str' type).

Signed-off-by: Simon Glass <sjg@chromium.org>
2019-07-10 16:52:58 -06:00
Simon Glass
e6d85ff9f2 binman: Handle repeated bytes for Python 3
The method of multiplying a character by a number works well for creating
a repeated string in Python 2. But in Python 3 we need to use bytes()
instead, to avoid unicode problems, since 'bytes' is no-longer just an
alias of 'str'.

Create a function to handle this detail and call it from the relevant
places in binman.

Signed-off-by: Simon Glass <sjg@chromium.org>
2019-07-10 16:52:58 -06:00
Simon Glass
a92939a4aa patman: Provide a way to get program output in binary mode
At present cros_subprocess and the tools library use a string to obtain
stdout from a program. This works fine on Python 2. With Python 3 we end
up with unicode errors in some cases. Fix this by providing a binary mode,
which returns the data as bytes() instead of a string.

Signed-off-by: Simon Glass <sjg@chromium.org>
2019-07-10 16:52:57 -06:00
Simon Glass
1fda18205f tools: Set an initial value for indir
This variable is not documented or set up in the module. Fix this.

Signed-off-by: Simon Glass <sjg@chromium.org>
2018-10-08 07:34:34 -06:00
Simon Glass
0a98b28b06 binman: Support adding files
In some cases it is useful to add a group of files to the image and be
able to access them at run-time. Of course it is possible to generate
the binman config file with a set of blobs each with a filename. But for
convenience, add an entry type which can do this.

Add required support (for adding nodes and string properties) into the
state module.

Signed-off-by: Simon Glass <sjg@chromium.org>
2018-09-29 11:49:35 -06:00
Simon Glass
04187a845c patman: Detect missing tools and report them
When tools are needed but not present, at present we just get an error
which can be confusing for the user. Try to be helpful by reporting the
tool as missing and suggesting a possible remedy.

Also update the Run() method to support this.

Signed-off-by: Simon Glass <sjg@chromium.org>
2018-09-28 11:09:01 -06:00
Simon Glass
4f5dea4543 patman: Show the current directory in GetInputFilename()
When this fails it is useful to see the current directory, since U-Boot's
build system will typically change into the output directory during the
build. Add this information to the error.

Signed-off-by: Simon Glass <sjg@chromium.org>
2018-08-01 16:30:48 -06:00
Simon Glass
aeffc5e956 patman: Add a few more helpers to the tools library
Add functions to read and write a file, looking through a list of search
paths to find it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2018-08-01 16:30:48 -06:00
Tom Rini
83d290c56f SPDX: Convert all of our single license tags to Linux Kernel style
When U-Boot started using SPDX tags we were among the early adopters and
there weren't a lot of other examples to borrow from.  So we picked the
area of the file that usually had a full license text and replaced it
with an appropriate SPDX-License-Identifier: entry.  Since then, the
Linux Kernel has adopted SPDX tags and they place it as the very first
line in a file (except where shebangs are used, then it's second line)
and with slightly different comment styles than us.

In part due to community overlap, in part due to better tag visibility
and in part for other minor reasons, switch over to that style.

This commit changes all instances where we have a single declared
license in the tag as both the before and after are identical in tag
contents.  There's also a few places where I found we did not have a tag
and have introduced one.

Signed-off-by: Tom Rini <trini@konsulko.com>
2018-05-07 09:34:12 -04:00
Simon Glass
1f1864b408 patman: Add a tools library for using temporary files
For tools which want to use input files and temporary output, it is useful
to have the handling of these dealt with in one place. Add a new library
which allows input files to be read, and output files to be written, all
based on a common directory structure.

Signed-off-by: Simon Glass <sjg@chromium.org>
2016-09-18 21:04:38 -06:00