Commit graph

313 commits

Author SHA1 Message Date
Simon Glass
7bf83a5d7b buildman: Detect Kconfig loops
Hex and int Kconfig options are supposed to have defaults. This is so we
can configure U-Boot without having to enter particular values for the
items that don't have specific values in the board's defconfig file.

If this rule is not followed, then introducing a new Kconfig can produce
a loop like this:

   Break things (BREAK_ME) [] (NEW)
   Error in reading or end of file.

   Break things (BREAK_ME) [] (NEW)
   Error in reading or end of file.

The continues forever since buildman passes /dev/null to 'conf', and
the build system just tries again. Eventually there is so much output that
buildman runs out of memory.

We can detect this situation by looking for a symbol (like 'BREAK_ME')
which has no default (the '[]' above) and is marked as new. If this
appears multiple times in the output, we know something is wrong.

Add a filter function for the output which detects this situation. Allow
it to return True to terminate the process. Implement this termination in
cros_subprocess.

With this we get a nice message:

   buildman --board sandbox -T0
   Building current source for 1 boards (0 threads, 32 jobs per thread)
      sandbox:  w+   sandbox
   +.config:66:warning: symbol value '' invalid for BREAK_ME
   +
   +Error in reading or end of file.
   +make[3]: *** [scripts/kconfig/Makefile:75: syncconfig] Terminated
   +make[2]: *** [Makefile:569: syncconfig] Terminated
   +make: *** [Makefile:177: sub-make] Terminated
   +(** did you define an int/hex Kconfig with no default? **)

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-10-20 10:59:55 +02:00
Simon Glass
bafdeb4546 buildman: Write output even on fatal error
At present buildman does not write any output (to the 'out' and 'err)
files if the build terminates with a fatal error. This is to avoid adding
lots of spam to the logs.

However there are times when this is actually useful, such as when the
build fails for an obscure reason such as a Kconfig loop.

Update the logic to always write the output, so that the user gets a clue
as to what is happening.

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-10-20 10:59:54 +02:00
Paul Barker
5fe50f9a40 tools: Refactor full help printing
Collect the code for printing the full help message of patman, buildman
and binman into a single function in patman.tools.

Signed-off-by: Paul Barker <paul.barker@sancloud.com>
2021-09-24 14:30:46 -04:00
Simon Glass
5ed16a9511 net: Rename SPL_NET_SUPPORT to SPL_NET
Rename this option so that CONFIG_IS_ENABLED can be used with it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-09-04 12:48:53 -04:00
Simon Glass
103c5f1806 mmc: Rename MMC_SUPPORT to MMC
Rename these options so that CONFIG_IS_ENABLED can be used with them.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Jaehoon Chung <jh80.chung@samsung.com>
[trini: Fixup some incorrect renames]
Signed-off-by: Tom Rini <trini@konsulko.com>
2021-09-04 11:42:41 -04:00
Simon Glass
333e4a621d Rename SPL_USB_HOST_SUPPORT to SPL_USB_HOST
Rename this option so that CONFIG_IS_ENABLED can be used with it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-07-28 14:27:54 -04:00
Simon Glass
f1a83abe60 buildman: Use bytes for the environment
At present we sometimes see problems in gitlab where the environment has
0x80 characters or sequences which are not valid UTF-8.

Avoid this by using bytes for the environment, both internal to buildman
and when writing out the 'env' file. Add a test to make sure this works
as expected.

Reported-by: Marek Vasut <marex@denx.de>
Fixes: e5fc79ea71 ("buildman: Write the environment out to an 'env' file")
Signed-off-by: Simon Glass <sjg@chromium.org>
2021-04-29 03:23:39 -07:00
Simon Glass
8116c78ffd buildman: Handle exceptions in threads gracefully
There have been at least a few cases where an exception has occurred in a
thread and resulted in buildman hanging: running out of disk space and
getting a unicode error.

Handle these by collecting a list of exceptions, printing them out and
reporting failure if any are found. Add a test for this.

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-04-29 03:23:39 -07:00
Simon Glass
ab9b4f35e3 buildman: Use common code to send an result
At present the code to report a build result is duplicated. Put it in a
common function to avoid this.

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-04-29 03:23:39 -07:00
Simon Glass
249933136f buildman: Tidy up a few comments
Add some function comments which are missing, or missing arguments.

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-04-29 03:23:39 -07:00
Simon Glass
5f86454b3d buildman: Add an encoding to the out-env file
The environment may contain some unicode characters. At least that is what
seemed to happen on one commit:

Building current source for 1 boards (0 threads, 64 jobs per thread)
    0    0    0 /1       -1      (starting)
Traceback (most recent call last):
  File ".../tools/buildman/buildman", line 64, in <module>
    ret_code = control.DoBuildman(options, args)
  File "tools/buildman/control.py", line 372, in DoBuildman
    options.keep_outputs, options.verbose)
  File ".../tools/buildman/builder.py", line 1704, in BuildBoards
    results = self._single_builder.RunJob(job)
  File ".../tools/buildman/builderthread.py", line 526, in RunJob
    self._WriteResult(result, job.keep_outputs, job.work_in_output)
  File ".../tools//buildman/builderthread.py", line 349, in _WriteResult
    print('%s="%s"' % (var, env[var]), file=fd)
UnicodeEncodeError: 'ascii' codec can't encode characters in position
	311-312: ordinal not in range(128)

The problem defies repetition with any change at all to buildman. But
let's set an encoding in any case.

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-04-06 16:33:19 +12:00
Simon Glass
b82492bbcc buildman: Support single-threaded operation
At present even if only a single thread is in use, buildman still uses
threading.

For some debugging it is helpful to do everything in the main process.
Allow -T0 to support this.

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-03-04 17:51:43 -05:00
Heinrich Schuchardt
fd434f47d4 buildman: 'Thread' object has no attribute 'isAlive'
The isAlive() method was deprecated in Python 3.8 and has been removed in
Python 3.9. See https://bugs.python.org/issue37804. Use is_alive() instead.

Since Python 2.6 is_alive() has been a synonym for isAlive(). So there
should be no problems for users using elder Python 3 versions.

Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
Reviewed-by: Simon Glass <sjg@chromium.org>
2021-02-14 11:32:04 +01:00
Simon Glass
73da3d2cff buildman: Remove output binaries before building
Buildman reuses build directories from previous builds to avoid the cost
of 'make mrproper' for every build. If the previous build produced an SPL
image but the current one does not, the SPL image will remain and buildman
will think it is a result of building the current board.

Remove these files before building, to avoid this problem.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-12-22 20:39:26 -07:00
Simon Glass
d93720e138 patman: Rename functions in patchstream
Rename these functions to lower case as per PEP8.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-11-05 09:11:31 -07:00
Alper Nebi Yasak
76de29fc4f buildman: Use git worktrees instead of git clones when possible
This patch makes buildman create linked working trees instead of clones
of the source repository, but keeps updating the older clones of the
repository that might already exist. These worktrees share "everything
except working directory specific files such as HEAD, index, etc." with
the source repository. See the git-worktree(1) manual page for more
information.

If git-worktree isn't available, silently falls back to cloning the
repository.

Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org>
2020-09-22 12:50:43 -06:00
Simon Glass
2b522f1e79 tools: Drop unnecessary use of __file__
There are few places where the path of the current modules is calculated
but not used. Drop them.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-09-22 12:50:43 -06:00
Simon Glass
38f159c05b buildman: Show the build rate at the end
It is interesting to note the number of builds completed per second to
track machine performance and build speed. Add a 'rate' value at the end
of the build to show this.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-28 19:30:39 -06:00
Simon Glass
3918dfaa91 buildman: Correct the testOutputDir() unit test
This current fails with an error. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Fixes: 7664b03ffc ("buildman: Remove _of_#_ from results directory paths")
2020-07-28 19:30:39 -06:00
Simon Glass
5a910b92bc buildman: Allow using older versions of genboardscfg.py
Older versions of this script don't support the -q flag. Since buildman
runs this script from when it starts, we may get the old version.

Fix this in two ways:

1. Use the version from the same tree as buildman is run from, if
available
2. Failing that, allow the -q flag to be missing

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-28 19:30:39 -06:00
Tom Rini
ddf34e7d3c kconfiglib: Update to the 14.1.0 release
A large number of changes have happened upstream since our last sync
in commit 65e05ddc1a ("kconfiglib: Update to the 12.14.0 release").

The big motivation for this sync is support for user defined macros
within Kconfig.

Cc: Masahiro Yamada <yamada.masahiro@socionext.com>
Signed-off-by: Tom Rini <trini@konsulko.com>
2020-07-01 10:11:03 -04:00
Ovidiu Panait
7664b03ffc buildman: Remove _of_#_ from results directory paths
Currently, the following scenario will rebuild the first commit even
though it is not really necessary - the commit sha or the position in the
patchset did not change:

$ git am <local-patch-0001>
$ tools/buildman/buildman -P -E -W -b master mx6
<do some more development work>
$ git am <local-patch-0002>
$ tools/buildman/buildman -P -E -W -b master mx6 <- will rebuild the first
						    commit as well, even
						    though nothing has
						    changed about it.

This is due to the fact that previous results directories get removed
when the number of commits change. By removing the _of_#_ part of the
directory path, the commits will be rebuilt only if the commit sha or the
position in the patchset changes. Also, update the testcase to reflect this
change.

Signed-off-by: Ovidiu Panait <ovidiu.panait@windriver.com>
2020-06-11 20:52:11 -06:00
Simon Glass
b4fa94959d patman: Tidy up sys.path changes
Now that we are using absolute paths we can remove some of the sys.path
mangling that appears in the tools.

We only need to add the path to 'tools/' so that everything can find
modules relative to that directory.

The special paths for finding pylibfdt remain.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-26 14:25:21 -06:00
Simon Glass
bf776679a7 patman: Move to absolute imports
At present patman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move patman to use absolute imports. This requires changes in tools which
use the patman libraries (which is most of them).

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-26 14:25:21 -06:00
Simon Glass
0ede00fdaf buildman: Move to absolute imports
At present buildman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move buildman to use absolute imports. Also adjust moveconfig.py too since
it uses some buildman modules and cannot work without this.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-26 14:25:21 -06:00
Simon Glass
7ec3dc57b3 buildman: Rename the main module
Python does not like the module name being the same as the module
directory. To allow buildman modules to be used from other tools, rename
it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-26 14:25:21 -06:00
Simon Glass
83a4518771 patman: Drop references to __future__
We don't need these now that the tools using Python 3. Drop them.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-26 14:25:21 -06:00
Simon Glass
60b285f8c3 buildman: Write output files when using -w
At present buildman does not write its own output files (err, done, the
environment) when using -w. However this is useful for when the build is
run with -s to check it.

In fact ProduceResultSummary() reads the result from those files rather
than using the 'result' info directly. So ProcessResult() does not work
with -w at present. It does not print any output.

Fix this by writing output files even when -w is used.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-26 14:24:08 -06:00
Simon Glass
166a98a426 buildman: Use out-env for environment output
At present the environment used by U-Boot is written to the 'env'
directory. This is fine when the output directory is not the same as the
source directory, but when it is (as with -w) it conflicts with the source
directory of the same name.

Rename 'env' to 'out-env' to fix this.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-26 14:24:08 -06:00
Simon Glass
88daaef19f buildman: Make sure that -o is given with -w
It is a bad idea to use the default output directory ('..') with -w since
it does a build in that directory and writes various files these.

Require that -o is given to avoid this problem.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-26 14:24:08 -06:00
Simon Glass
97944d3f7d buildman: Correct operation of -A flag
This was broken when -a was removed and unfortunately there are no tests
for this. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-26 14:24:08 -06:00
Simon Glass
55a98d96bd buildman: Fix test for new 9.2 kernel
The naming is slightly different on kernel.org now. Update the regex so
that the test still passes.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-26 14:24:08 -06:00
Simon Glass
b1e5e6d224 buildman: Change the exit codes
The current exit codes of 128 and 129 are useful in that they do not
conflict with those returned by tools, but they are not actually valid.
It seems better to pick some codes which work with 'bit bisect run'.

Update them to 100 (for errors) and 101 (for warnings).

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-21 06:33:47 -06:00
Simon Glass
a38930e227 buildman: Update the TODO items
A few of these have been done. Drop those and add some new ideas.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-21 06:33:47 -06:00
Simon Glass
113a8a5ff3 buildman: Add an option to ignore migration warnings
These are becoming more common now. They cause boards to show warnings
which can be mistaking for compiler warnings.

Add a buildman option to ignore them. This option works only with the
summary option (-s). It does not affect the build process.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-21 06:33:47 -06:00
Simon Glass
174592b964 buildman: Add an option to ignore device-tree warnings
Unfortunately the plague of device-tree warnings has not lifted. These
warnings infiltrate almost every build, adding noise and confusion.

Add a buildman option to ignore them. This option works only with the
summary option (-s). It does not affect the build process.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-21 06:33:47 -06:00
Simon Glass
eb70a2c059 buildman: Make -I the default
At present buildman defaults to running 'mrproper' on every thread before
it starts building commits for each board. This can add a delay of about 5
seconds to the start of the process, since the tools and other invariants
must be rebuilt.

In particular, a build without '-b', to build current source, runs much
slower without -I, since any existing build is removed, thus losing the
possibility of an incremental build.

Partly this behaviour was to avoid strange build-system problems caused by
running 'make defconfig' for one board and then one with a different
architecture. But these problems were fixed quite a while ago.

The -I option (which disabled mrproper) was introduced four years ago and
does not seem to cause any problems with builds.

So make -I the default and deprecate the option. To allow use of
'mrproper', add a new -m flag.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-21 06:33:47 -06:00
Simon Glass
ea09fb5bf1 buildman: Update workflow documentation with more detail
Make a few additions and change some wording in the workflow
documentation.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-21 06:33:47 -06:00
Simon Glass
9e040b5746 buildman: Add the abbreviation for --boards
This option may be frequency used, so mention that it can be abbreviated
to --bo

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-21 06:33:47 -06:00
Simon Glass
f972836593 buildman: Update the 'theory of operation' a little
Make a few updates to this important section of the documentation, to
make things clearer.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-21 06:33:47 -06:00
Simon Glass
7b33f21804 buildman: Show a summary of the build result
When buildman finishes it leaves the last summary line visible, which
shows the number of successful builds, builds with warnings and builds
with errors.

It is useful also to see how many builds were done in total along with
the time taken. Show these on a separate line before buildman finishes.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-21 06:33:47 -06:00
Simon Glass
95ed0a2dbf buildman: Limit the length of progress messages
If a progress message is longer than the terminal line it will scroll the
terminal. Limit the messages to the terminal width.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-21 06:33:47 -06:00
Simon Glass
6eb76cac1c buildman: Show the number of builds remaining
It is nice to see the actual number of builds remaining to complete. Add
this in the progress message, using a different colour.

Drop the unnecessary 'name' variable while we are here.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-21 06:33:47 -06:00
Simon Glass
edae4ad3bc buildman: Drop unused output code
The commit counter is a hangover from when buildman processed each board
for a commit. Now buildman processes each commit for a board, so this
output is never triggered.

Delete it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-21 06:33:47 -06:00
Simon Glass
212c0b8170 buildman: Show a message when fetching a repo
Fetching updated versions of a repo can take time. At present buildman
gives no indication that it is doing this.

Add a message to explain the delay.

Tidy up a few other messages while we are here.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-21 06:33:47 -06:00
Simon Glass
102969bb2c buildman: Drop the line-clearing code in Builder
The new feature in terminal can be used by buildman. Update the Builder
class accordingly.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-21 06:33:47 -06:00
Simon Glass
9ef0ceb730 buildman: Use spaces in the board list
At present the board names shown with -l are separated by commas. This
makes it hard to double-click to select a particular board. Also it is not
possible to select all boards and paste them as arguments to a subsequent
buildman run, since buildman requires spaces to separate the list on the
command line, not commas.

Change the output format to use spaces instead of commas.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-21 06:33:47 -06:00
Simon Glass
8c9a2674ee buildman: Show the list of boards in magenta
It is quite hard to see the list of board for each error line since the
colour is the same as the actual error line. Show the board list in
magenta so that it is easier to distinguish them.

There is no point in checking the colour of the overall line, since there
are now multiple colours. So drop those tests.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-21 06:33:47 -06:00
Simon Glass
35d696dbe5 buildman: Use an object to hold error lines
At present the string for each error line is created in _CalcErrorDelta()
and used to create the summary output. This is inflexible since all the
information (error/warning character, error line, list of boards with that
error line) is munged together in a string.

Create an object to hold this information and only convert it to a string
when printing the actual output.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-21 06:33:47 -06:00
Simon Glass
5627bd9d96 buildman: Use yellow consistently for warning lines
At present warnings are shown in yellow in the summary (-s) but magenta in
the detail listing (-e). Use yellow in both.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-21 06:33:47 -06:00
Simon Glass
e631a2b48a buildman: Test the output with --list-error-boards
Add a test to cover this flag, which adds the name of each board to each
error/warning line.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-21 06:33:47 -06:00
Simon Glass
c9dd80b3d4 buildman: Add a test helper for creating a line prefix
The split/join code is repeated in a lot of places. Add a function to
handle this.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-21 06:33:47 -06:00
Simon Glass
ce558db34b buildman: Split out testOutput() into separate functions
We want to add a few more tests similar to testOutput(). Split its logic
into a function which runs buildman to get the output and another which
checks the output. This will make it easier to reuse the code.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-21 06:33:47 -06:00
Simon Glass
af43065f8d buildman: Create temp directory in test setup
Rather than having a few tests handle this themselves, create the
temporary directory in the setUp() method and remove it in tearDown().
This will make it easier to add more tests.

Only testOutput and testGit() actually need it, but it doesn't add to the
test time noticeably to do this for all tests in this file.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-21 06:33:47 -06:00
Simon Glass
c3bc4f1fca buildman: Use an iterator to check test output
Rather than using the absolute array index, use an interator to work
through the expected output lines. This is easier to follow.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-21 06:33:47 -06:00
Simon Glass
3c339bbd4c buildman: Add test coverage for error/warning colour
Buildman should output the right colours for each error/warning line. Some
of these checks are missing. Add them.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-21 06:33:47 -06:00
Simon Glass
b206d87d52 buildman: Refactor error-line output int a function
Reduce the amount of repeated code by creating an _OutputErrLines()
function to hold this code.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-21 06:33:47 -06:00
Simon Glass
4e9162d519 buildman: Drop the -a option
There is no point in setting the ARCH environment variable since the
U-Boot build system no-longer uses it.

It seems safe to drop this feature since it was only recently added.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>
2020-04-10 21:21:06 -04:00
Simon Glass
b2d89bc538 buildman: Allow building within a subdir of the current dir
This is useful in some situations, in particular with -w and when building
in-tree. Now that we are more careful about what we remove in
_PrepareOutputSpace(), it should be safe to relax this restriction.

Update the progress information also so it is clear what buildman is
doing. Remove files can take a long time.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-10 21:21:06 -04:00
Simon Glass
925f6adfa5 buildman: Be more selective about which directories to remove
At present buildman removes any directory it doesn't intend to write
output into. This is overly expansive since if the output directory
happens to be somewhere with existing files, they may be removed. Using
an existing directory for buildman is not a good practice, but since the
result might be catastrophic, it is best to guard against it.

A previous commit[1] fixed this by refusing to write to a subdirectory
of the current directory, assumed to have U-Boot source code. But we can
do better by only removing directories that look like the ones buildman
creates.

Update the code to do this and add a test.

Signed-off-by: Simon Glass <sjg@chromium.org>

[1] 409fc029c4 tools: buildman: Don't use the working dir as build dir
2020-04-10 21:21:06 -04:00
Simon Glass
7beb43c980 buildman: Allow ignoring warnings in the return code
Sometimes we don't want buildman to return failure if it seems warnings.
Add a -W option to support this. If buildman detects warnings (and no
errors) it will return an exit code of 0 (success).

Note that the definition of 'warnings' includes the migration warnings
produced by U-Boot, such as:

    ===================== WARNING ======================
    This board does not use CONFIG_DM_MMC. Please update
    ...
    ====================================================

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>
2020-04-10 21:21:06 -04:00
Simon Glass
f9c094bbce buildman: Update help for -d
This help is a bit ambiguous. It only does anything if asked to show size
changes with -S. Update the help and the function comments.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-10 21:21:06 -04:00
Simon Glass
d829f1217c bulidman: Add support for a simple build
It is useful to run a simple build and put all the output in a single
directory. Add a -w option to support this.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-10 21:21:06 -04:00
Simon Glass
e9fbbf633e buildman: Document the members of BuilderJob
This class has a few more members now. Add documentation for them and fix
a nit in the 'commits' comment.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-10 21:21:06 -04:00
Bin Meng
205254eab8 buildman: Support fetching gcc 9.2.0
This adds support to fetch gcc 9.2.0 toolchains.

Signed-off-by: Bin Meng <bmeng.cn@gmail.com>
Reviewed-by: Tom Rini <trini@konsulko.com>
2020-04-10 15:54:16 -04:00
Matthias Brugger
2f7c53cbd3 buildman: Enable buildman on aarch64 hosts
At kernel.org aarch64 toolchains are published in folder
arm64. Fix the URL for that case, so that we can fetch
toolchains on aarch64 machines.

Signed-off-by: Matthias Brugger <mbrugger@suse.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2020-02-05 19:33:46 -07:00
Flavio Suligoi
9de5c397b2 tools: buildman: fix typo
Signed-off-by: Flavio Suligoi <f.suligoi@asem.it>
Reviewed-by: Simon Glass <sjg@chromium.org>
2020-01-30 13:30:35 -05:00
Tom Rini
d08c38c32f buildman: Ignore blank lines during size checks
Today when parsing the .sizes files we get a warning about an invalid
line in the file as it's blank.  Solve this by checking that we have a
non-blank line prior to processing.

Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2019-12-10 21:11:31 -07:00
Simon Glass
57cb9d5239 buildman: Add options to get the arch and toolchain info
Sometimes it is useful for external tools to use buildman to provide the
toolchain information. Add an -a option which shows the value to use for
the ARCH environment variable, and -A which does the same for
CROSS_COMPILE

Signed-off-by: Simon Glass <sjg@chromium.org>
2019-12-10 21:11:31 -07:00
Simon Glass
7c66ead452 buildman: Figure out boards before commits
At present buildman looks at toolchains, then commits and then boards.
Move the board processing up above the commit processing, since it relates
to the toolchain code. This will make it easier to check the toolchains
needed for a board without processing commits first.

Signed-off-by: Simon Glass <sjg@chromium.org>
2019-12-10 21:11:31 -07:00
Simon Glass
6a3fc91ea7 buildman: Ask genboardscfg to be quiet
Now that this tool has a 'quiet' flag, use it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2019-12-10 21:11:31 -07:00
Cristian Ciocaltea
4251fbc6fb buildman: Improve [make-flags] section parser to allow quoted strings
The parser responsible for the '[make-flags]' section in
the '.buildman' settings file is currently not able to
handle quoted strings, as given in the sample bellow:

[make-flags]
qemu_arm=HOSTCC="cc -isystem /add/include" HOSTLDFLAGS="-L/add/lib"

This patch replaces the simple string splitter based on the <space>
delimiter with a regex tokenizer that preserves spaces inside double
quoted strings.

Signed-off-by: Cristian Ciocaltea <cristian.ciocaltea@gmail.com>
2019-12-10 05:54:55 -07:00
Tom Rini
70f6813985 buildman: Fix problem with non-existent output directories
Now that we have buildman telling genboards.cfg to use an output
directory we need to ensure that it exists.

Cc: Bin Meng <bmeng.cn@gmail.com>
Cc: Simon Glass <sjg@chromium.org>
Fixes: bc750bca12 ("tools: buildman: Honor output directory when generating boards.cfg")
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
2019-11-23 14:53:48 -05:00
Simon Glass
c05aa03642 buildman: Convert to Python 3
Convert buildman to Python 3 and make it use that, to meet the 2020
deadline.

Signed-off-by: Simon Glass <sjg@chromium.org>
2019-11-04 18:15:32 -07:00
Bin Meng
578b637203 tools: buildman: Remove useless mkdir() in Make() in test.py
In the 'Make' function, the codes tries to create a directory
if current stage is 'build'. But the directory isn't used at
all anywhere.

Signed-off-by: Bin Meng <bmeng.cn@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2019-10-29 16:17:39 -04:00
Bin Meng
bc750bca12 tools: buildman: Honor output directory when generating boards.cfg
buildman always generates boards.cfg in the U-Boot source tree.
When '-o' is given, we should generate boards.cfg to the given
output directory.

Signed-off-by: Bin Meng <bmeng.cn@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2019-10-29 16:17:36 -04:00
Tom Rini
aae62584a6 buildman: Perform tests in a temporary directory
We may not always be able to write to the default output directory so
have a temporary directory for our output be created.

Cc: Simon Glass <sjg@chromium.org>
Reviewed-by: Stephen Warren <swarren@wwwdotorg.org>
Suggested-by: Stephen Warren <swarren@wwwdotorg.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
2019-10-08 12:10:38 -04:00
Tom Rini
65e05ddc1a kconfiglib: Update to the 12.14.0 release
A large number of changes have happened upstream since our last sync
which was to 375506d.  The reason to do the upgrade at this point is for
improved Python 3 support.

As part of this upgrade we need to update moveconfig.py and
genboardscfg.py the current API.  This is:
- Change "kconfiglib.Config" calls to "kconfiglib.Kconfig"
- Change get_symbol() calls to syms.get().
- Change get_value() to str_value.

Cc: Masahiro Yamada <yamada.masahiro@socionext.com>
Signed-off-by: Tom Rini <trini@konsulko.com>
2019-10-08 08:37:29 -04:00
Baruch Siach
2429613601 buildman: fix invocation examples typos
Also, make command line example indentation consistent.

Signed-off-by: Baruch Siach <baruch@tkos.co.il>
2019-07-24 12:54:08 -07:00
Simon Glass
347ea0b63e buildman: Deal more nicely with invalid build-status file
The 'done' files created by buildman may end up being empty if buildman
runs out of disk space while writing them. At present buildman dies with
an exception when using -s to check the build status. Fix this.

Seriesl-cc: trini

Signed-off-by: Simon Glass <sjg@chromium.org>
2019-05-21 17:33:23 -06:00
Chris Packham
58804b8cf2 buildman: fix typo
Fix a typo in the error message from CheckOutputDir().

Signed-off-by: Chris Packham <judge.packham@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2019-02-09 12:50:22 -07:00
Simon Glass
e5fc79ea71 buildman: Write the environment out to an 'env' file
Sometimes it is useful to see the environment that was used to build
U-Boot. Write this out to a file in the build directory.

Signed-off-by: Simon Glass <sjg@chromium.org>
2019-02-09 12:50:22 -07:00
Simon Glass
ccd2979a8d buildman: Fix tabs in GetWrapper()
This function has tabs instead of spaces. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2019-01-14 17:47:13 -07:00
Simon Glass
00beb2485f buildman: Add support for building with clang
Add a -O option which allows building with clang.

Signed-off-by: Simon Glass <sjg@chromium.org>
2019-01-14 17:47:13 -07:00
Simon Glass
e74429bb17 buildman: Deal nicely with invalid build-status file
The 'done' files created by buildman may end up being empty if buildman
runs out of disk space while writing them. This error is then persistent,
since even if disk space is reclaimed and the build retries, the empty
file causes an exception in the builder thread.

Deal with this silently by doing a rebuild.

Signed-off-by: Simon Glass <sjg@chromium.org>
2019-01-14 17:47:13 -07:00
Simon Glass
90a29fcc1f buildman: Drop comment about Ctrl-C problem
This bug is now fixed, so drop this comment.

Signed-off-by: Simon Glass <sjg@chromium.org>
2019-01-14 17:47:13 -07:00
Trevor Woerner
b11f12642f buildman/toolchain.py: handle inconsistent tarball names
Unfortunately, for some releases the kernel.org toolchain tarball names adhere
to the following pattern:

	<hostarch>-gcc-<ver>-nolib-<targetarch>-<type>.tar.xz

e.g.:
	x86_64-gcc-8.1.0-nolibc-aarch64-linux.tar.xz

while others use the following pattern:

	<hostarch>-gcc-<ver>-nolib_<targetarch>-<type>.tar.xz

e.g.:

	x86_64-gcc-7.3.0-nolibc_aarch64-linux.tar.xz

Notice that the first pattern has dashes throughout, while the second has
dashes throughout except just before the target architecture which has an
underscore.

The "dash throughout" versions from kernel.org are:

	8.1.0, 6.4.0, 5.5.0, 4.9.4, 4.8.5, 4.6.1

while the "dash and underscore" versions from kernel.org are:

	7.3.0, 4.9.0, 4.8.0, 4.7.3, 4.6.3, 4.6.2, 4.5.1, 4.2.4

This tweak allows the code to handle both versions. Note that this tweak also
causes the architecture parsing to get confused and find the following two
bogus architectures, "2.0" and "64", which are explicitly checked for, and
removed.

Signed-off-by: Trevor Woerner <trevor@toganlabs.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Change single quotes to double quotes:
Signed-off-by: Simon Glass <sjg@chromium.org>
2018-12-05 06:06:30 -07:00
Trevor Woerner
d82f539ab9 buildman/toolchain.py: fix toolchain directory
The hexagon toolchain (4.6.1) from kernel.org, for example, was packaged in
a way that is different from most toolchains. The first entry when unpacking
most toolchain tarballs is:

	gcc-<version>-nolib/<targetarch>-<system>

e.g.:

	gcc-8.1.0-nolibc/aarch64-linux/

The first entry of the hexagon toolchain, however, is:

	gcc-4.6.1-nolibc/

This causes the buildman logic in toolchain.py::ScanPath() to not be able to
find the "*gcc" executable since it looks in gcc-4.6.1-nolib/{.|bin|usr/bin}
instead of gcc-4.6.1/hexagon-linux/{.|bin|usr/bin}. Therefore when buildman
tries to download a set of toolchains that includes hexagon, the script fails.

This update takes the second line of the tarball unpacking (which works for
all the toolchains I've tested from kernel.org) and parses it to take the
first two elements, separated by '/'. It makes this logic a bit more robust.

Signed-off-by: Trevor Woerner <trevor@toganlabs.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2018-12-05 06:01:35 -07:00
Simon Glass
6af7101b75 buildman: Show boards with warning with w+
At present we should boards with warnings in the same way as those with
errors. This is not ideal. Add a new 'warn' state and show these listed
in yellow to match the actual warning lines printing with -e.

Signed-off-by: Simon Glass <sjg@chromium.org>
2018-11-20 19:14:22 -07:00
Simon Glass
4cf2b221c6 buildman: Rename the good, better, worse variables
At present we don't distinguish between errors and warnings when printing
the architecture summary. Rename the variables to better describe their
purpose.

'Worse' at present means we got an error, so use that as the name.

Signed-off-by: Simon Glass <sjg@chromium.org>
2018-11-20 19:14:22 -07:00
Simon Glass
2d48333e44 buildman: Detect dtc warnings
At present messages from the device-tree compiler like this:

  arch/arm/dts/socfpga_arria10_socdk_sdmmc.dtb: Warning
     (avoid_unnecessary_addr_size): /clocks: unnecessary
     #address-cells/#size-cells without "ranges" or child "reg" property

are detected as errors since they don't match the gcc warning regex. Add a
new one for dtc to fix this.

Signed-off-by: Simon Glass <sjg@chromium.org>
2018-11-20 19:14:22 -07:00
Simon Glass
40232c91d7 buildman: Only print toolchain probing with -v
At present --list-tool-chains prints a lot of information about the
toolchain-probing process. This is generally not very interesting.
Update buildman to print this only if --list-tool-chains is given
with -v.

Signed-off-by: Simon Glass <sjg@chromium.org>
2018-11-20 19:14:22 -07:00
Simon Glass
0689036a35 buildman: Add a --boards option to specify particular boards to build
At present 'buildman sandbox' will build all 5 boards for the sandbox
architecture rather than the single board 'sandbox'. The only current way
to exclude sandbox_spl, sandbox_noblk, etc. is to use -x which is a bit
clumbsy.

Add a --boards option to allow individual build targets to be specified.

Signed-off-by: Simon Glass <sjg@chromium.org>
2018-11-14 09:16:27 -08:00
Simon Glass
4b4bc06ef4 buildman: dtoc: Suppress unwanted output from test
There are a few test cases which print output. Suppress this so that tests
can run silently in the normal case.

Signed-off-by: Simon Glass <sjg@chromium.org>
2018-10-08 07:34:34 -06:00
Simon Glass
da753e35cb buildman: Make the toolchain test more forgiving
The filenames of the toolchains on kernel.org changes every now and then.
Fix it for the current change, and make the test use a regex so that it
has a better chance of passing with future changes too.

Signed-off-by: Simon Glass <sjg@chromium.org>
2018-10-08 07:34:34 -06:00
Simon Glass
e62a24ce27 buildman: Avoid hanging when the config changes
Something has changed in the last several month such that when buildman
builds U-Boot incrementally and a new CONFIG option has been added to the
Kconfig, the build hanges waiting for input:

    Test new config (NEW_CONFIG) [N/y/?] (NEW)

Since binamn does not connect the build's stdin to anything this waits on
stdin to the build thread, which never comes. Eventually I suspect all the
threads end up in this state and the build does not progress.

Fix this by passing /dev/null as input to the build. That way, if there is
a new CONFIG, the build will stop (and fail):

    Test new config (NEW_CONFIG) [N/y/?] (NEW)

    Error in reading or end of file.

Signed-off-by: Simon Glass <sjg@chromium.org>
2018-09-29 11:49:35 -06:00
Masahiro Yamada
9865543ae6 Remove CONFIG_USE_STDINT
You do not need to use the typedefs provided by compiler.

Our compilers are either IPL32 or LP64.  Hence, U-Boot can/should
always use int-ll64.h typedefs like Linux kernel, whatever the
typedefs the compiler internally uses.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-09-10 20:48:16 -04:00
Alex Kiernan
48ae412424 buildman: Add support for environment delta in summary
When summarising the builds, add the -U option to emit delta lines for
the default environment built into U-Boot at each commit.

Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2018-06-07 11:25:07 -08:00
Alex Kiernan
0ddc510ea3 buildman: Extract environment as part of each build
As we're building the boards, extract the default U-Boot environment to
uboot.env so we can interrogate it later.

Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
2018-06-07 11:25:07 -08:00
Daniel Schwierzeck
4c58d273e0 buildman: support newer gcc versions from kernel.org
Add support for gcc versions 7.3.0, 6.4.0 and 4.9.4.

Also use a regex for matching the tarball names. Some gcc versions
use '-ARCH-' instead of '_ARCH-'.

As part of this, we switch TravisCI to also using these toolchains for
all platforms.

Signed-off-by: Daniel Schwierzeck <daniel.schwierzeck@gmail.com>
Signed-off-by: Tom Rini <trini@konsulko.com>
2018-05-22 22:08:57 -04:00