This is probably a little more efficient and it allows passing the object
to another function to write data. Convert config_out to use a string I/O
device.
Signed-off-by: Simon Glass <sjg@chromium.org>
Move more of the argument-building code into this function. Fix a missing
assignment for out_rel_dir too.
Rename the function since it now builds all the arguments.
Signed-off-by: Simon Glass <sjg@chromium.org>
This variable has a different meaning in the outer scope. Use a different
name to avoid confusion, or bugs.
Signed-off-by: Simon Glass <sjg@chromium.org>
The host compiler is not failing on warnings at present, when the
-E flag is used in buildman. Add the required flag to fix this.
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Tom Rini <trini@konsulko.com>
The patman directory has a number of modules which are used by other tools
in U-Boot. This makes it hard to package the tools using pypi since the
common files must be copied along with the tool that uses them.
To address this, move these files into a new u_boot_pylib library. This
can be packaged separately and listed as a dependency of each tool.
Signed-off-by: Simon Glass <sjg@chromium.org>
This is quite a useful thing to use when building since it avoids small
size changes between commits. Add a -r flag for it.
Also undefine CONFIG_LOCALVERSION_AUTO since this appends the git hash
to the version string, causing every build to be slightly different.
Signed-off-by: Simon Glass <sjg@chromium.org>
This cuts down build performance considerably and is not always needed,
when checking for build errors, etc.
Add a flag to disable it.
Signed-off-by: Simon Glass <sjg@chromium.org>
It is sometimes useful to see the exact 'make' command used by buildman
for a commit. Add an output file for this.
Signed-off-by: Simon Glass <sjg@chromium.org>
Add a new flag to buildman so that we will in turn pass
BINMAN_ALLOW_MISSING=1 to 'make'. Make use of this flag in CI.
Allow the settings file to control this.
Cc: Rasmus Villemoes <rasmus.villemoes@prevas.dk>
Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Signed-off-by: Simon Glass <sjg@chromium.org>
Buildman should consider a build as a success (with warnings) if missing
blobs have been dealt with by binman, even though buildman itself returns
and error code overall. This is how other warnings are dealt with.
We cannot easily access the 103 exit code, so detect the problem in the
output.
With this change, missing blobs result in an exit code of 101, although
they still indicate failure.
Signed-off-by: Simon Glass <sjg@chromium.org>
We have a module called 'board'. Sometimes buildman uses 'brd' as an
instance variable but sometimes it uses 'board', which is confusing and
can mess with the module handling. Update the code to use 'brd'
consistently, making it easier for tools to determine when the module
is being referenced.
Signed-off-by: Simon Glass <sjg@chromium.org>
Add a -a option to specify changes to the config before the build
commences. For example
buildman -a ~CONFIG_CMDLINE
disables CONFIG_CMDLINE before doing the build.
This makes it easier to try things out as well as to write tests without
creating a new board or manually manging the .config file.
Signed-off-by: Simon Glass <sjg@chromium.org>
If a thread crashes it is helpful to try the operation again with
threading disabled. Add a hint about that.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present buildman does not write any output (to the 'out' and 'err)
files if the build terminates with a fatal error. This is to avoid adding
lots of spam to the logs.
However there are times when this is actually useful, such as when the
build fails for an obscure reason such as a Kconfig loop.
Update the logic to always write the output, so that the user gets a clue
as to what is happening.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present we sometimes see problems in gitlab where the environment has
0x80 characters or sequences which are not valid UTF-8.
Avoid this by using bytes for the environment, both internal to buildman
and when writing out the 'env' file. Add a test to make sure this works
as expected.
Reported-by: Marek Vasut <marex@denx.de>
Fixes: e5fc79ea71 ("buildman: Write the environment out to an 'env' file")
Signed-off-by: Simon Glass <sjg@chromium.org>
There have been at least a few cases where an exception has occurred in a
thread and resulted in buildman hanging: running out of disk space and
getting a unicode error.
Handle these by collecting a list of exceptions, printing them out and
reporting failure if any are found. Add a test for this.
Signed-off-by: Simon Glass <sjg@chromium.org>
The environment may contain some unicode characters. At least that is what
seemed to happen on one commit:
Building current source for 1 boards (0 threads, 64 jobs per thread)
0 0 0 /1 -1 (starting)
Traceback (most recent call last):
File ".../tools/buildman/buildman", line 64, in <module>
ret_code = control.DoBuildman(options, args)
File "tools/buildman/control.py", line 372, in DoBuildman
options.keep_outputs, options.verbose)
File ".../tools/buildman/builder.py", line 1704, in BuildBoards
results = self._single_builder.RunJob(job)
File ".../tools/buildman/builderthread.py", line 526, in RunJob
self._WriteResult(result, job.keep_outputs, job.work_in_output)
File ".../tools//buildman/builderthread.py", line 349, in _WriteResult
print('%s="%s"' % (var, env[var]), file=fd)
UnicodeEncodeError: 'ascii' codec can't encode characters in position
311-312: ordinal not in range(128)
The problem defies repetition with any change at all to buildman. But
let's set an encoding in any case.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present even if only a single thread is in use, buildman still uses
threading.
For some debugging it is helpful to do everything in the main process.
Allow -T0 to support this.
Signed-off-by: Simon Glass <sjg@chromium.org>
Buildman reuses build directories from previous builds to avoid the cost
of 'make mrproper' for every build. If the previous build produced an SPL
image but the current one does not, the SPL image will remain and buildman
will think it is a result of building the current board.
Remove these files before building, to avoid this problem.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present patman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.
Move patman to use absolute imports. This requires changes in tools which
use the patman libraries (which is most of them).
Signed-off-by: Simon Glass <sjg@chromium.org>
At present buildman does not write its own output files (err, done, the
environment) when using -w. However this is useful for when the build is
run with -s to check it.
In fact ProduceResultSummary() reads the result from those files rather
than using the 'result' info directly. So ProcessResult() does not work
with -w at present. It does not print any output.
Fix this by writing output files even when -w is used.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present the environment used by U-Boot is written to the 'env'
directory. This is fine when the output directory is not the same as the
source directory, but when it is (as with -w) it conflicts with the source
directory of the same name.
Rename 'env' to 'out-env' to fix this.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present buildman defaults to running 'mrproper' on every thread before
it starts building commits for each board. This can add a delay of about 5
seconds to the start of the process, since the tools and other invariants
must be rebuilt.
In particular, a build without '-b', to build current source, runs much
slower without -I, since any existing build is removed, thus losing the
possibility of an incremental build.
Partly this behaviour was to avoid strange build-system problems caused by
running 'make defconfig' for one board and then one with a different
architecture. But these problems were fixed quite a while ago.
The -I option (which disabled mrproper) was introduced four years ago and
does not seem to cause any problems with builds.
So make -I the default and deprecate the option. To allow use of
'mrproper', add a new -m flag.
Signed-off-by: Simon Glass <sjg@chromium.org>
It is useful to run a simple build and put all the output in a single
directory. Add a -w option to support this.
Signed-off-by: Simon Glass <sjg@chromium.org>
This class has a few more members now. Add documentation for them and fix
a nit in the 'commits' comment.
Signed-off-by: Simon Glass <sjg@chromium.org>
Sometimes it is useful to see the environment that was used to build
U-Boot. Write this out to a file in the build directory.
Signed-off-by: Simon Glass <sjg@chromium.org>
The 'done' files created by buildman may end up being empty if buildman
runs out of disk space while writing them. This error is then persistent,
since even if disk space is reclaimed and the build retries, the empty
file causes an exception in the builder thread.
Deal with this silently by doing a rebuild.
Signed-off-by: Simon Glass <sjg@chromium.org>
As we're building the boards, extract the default U-Boot environment to
uboot.env so we can interrogate it later.
Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
When the U-Boot base directory happens to have the same name as the branch
that buildman is directed to use via the '-b' option and no output
directory is specified with '-o', buildman happily starts removing the
whole U-Boot sources eventually only stopped with the error message:
OSError: [Errno 20] Not a directory: '../<branch-name>/boards.cfg
Add a check to avoid this and also deal with the case where '-o' points
to the source directory, or any subdirectory of it.
Finally, tidy up the confusing logic for removing the old tree when using
-b. This is only done when building a branch.
Signed-off-by: Lothar Waßmann <LW@KARO-electronics.de>
Signed-off-by: Simon Glass <sjg@chromium.org>
Tested-by: Lothar Waßmann <LW@KARO-electronics.de>
When U-Boot started using SPDX tags we were among the early adopters and
there weren't a lot of other examples to borrow from. So we picked the
area of the file that usually had a full license text and replaced it
with an appropriate SPDX-License-Identifier: entry. Since then, the
Linux Kernel has adopted SPDX tags and they place it as the very first
line in a file (except where shebangs are used, then it's second line)
and with slightly different comment styles than us.
In part due to community overlap, in part due to better tag visibility
and in part for other minor reasons, switch over to that style.
This commit changes all instances where we have a single declared
license in the tag as both the before and after are identical in tag
contents. There's also a few places where I found we did not have a tag
and have introduced one.
Signed-off-by: Tom Rini <trini@konsulko.com>
Add a new option '-E' for treating all compiler warnings as errors.
Eventually this will pass 'KCFLAGS=-Werror' to Kbuild.
Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>
The build output can still produce unicode encoded output. But in
the buildman's log and err files we only want plain ASCII characters.
To handle all situations with unicode and non-unicode output, encode
the stdout and stderr strings to UTF-8 and afterwards to ASCII with
replacing all special characters.
Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>
When writing out some of our results we may now have UTF-8 characters
in there as well. Translate these to latin-1 and ignore any errors (as
this is for diagnostic and given the githash anything else can be
reconstructed by the user.
Reviewed-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
