Commit graph

279 commits

Author SHA1 Message Date
Simon Glass
098b10fb34 patman: Convert camel case in terminal.py
Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-02-09 12:30:13 -07:00
Simon Glass
252ac58996 patman: Rename Color() method to build()
This method has the same name as its class which is confusing. It is also
annoying when searching the code.

It builds a string with a colour, so rename it to build().

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-02-09 12:30:13 -07:00
Simon Glass
82e0e732ee patman: Rename Print() to Tprint()
Rename this function so that when we convert it to snake case it will not
conflict with the built-in print() function.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-02-09 12:30:13 -07:00
Simon Glass
5e2ab40172 patman: Convert camel case in test_util.py
Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-02-09 12:30:12 -07:00
Simon Glass
0157b187f4 patman: Convert camel case in gitutil.py
Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-02-09 12:26:12 -07:00
Simon Glass
d98006997c patman: Convert camel case in command.py
Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-02-09 12:26:12 -07:00
Simon Glass
c1aa66e75d patman: Convert camel case in tools.py
Convert this file to snake case and update all files which use it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-02-09 12:26:12 -07:00
Simon Glass
2b4806e481 buildman: Allow adjusting board config on the fly
Add a -a option to specify changes to the config before the build
commences. For example

   buildman -a ~CONFIG_CMDLINE

disables CONFIG_CMDLINE before doing the build.

This makes it easier to try things out as well as to write tests without
creating a new board or manually manging the .config file.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-02-08 23:07:59 -05:00
Simon Glass
8ca0931a91 buildman: Provide a hint on how to debug thread crashes
If a thread crashes it is helpful to try the operation again with
threading disabled. Add a hint about that.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-02-08 23:07:59 -05:00
Simon Glass
19133b7184 buildman: Add helper functions for updating .config files
At present the only straightforward way to write tests that need a
slightly different configuration is to create a new board with its own
configuration. This is cumbersome.

It would be useful if buildman could adjust the configuration of a build
on the fly. In preparation for this, add a utility library which can
modify a .config file according to various parameters passed to it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-02-08 23:07:59 -05:00
Simon Glass
d10dc40283 buildman: Make use of test_util
Use test_util to run the tests, with the ability to select a single test
to run, if desired.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-02-08 23:07:59 -05:00
Simon Glass
433fa549e1 buildman: Add a flag to control the traceback
At present the full horror of the Python traceback is shown by default. It
is normally only useful for debugging. Turn it off by default and add a
--debug flag to enable it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-02-08 23:07:59 -05:00
Simon Glass
5b7968693f patman: Tidy up the download function a little
Reverse the order of the return tuple, so that the filename is first.
This seems more obvious than putting the temporary directory first.

Correct a bug that leaves a space on the final line.

Allow the caller to control the name of the temporary directory.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-01-25 12:36:11 -07:00
Simon Glass
8ea6d23ffb buildman: Move the download function to tools
This function is handy for binman as well. Move it into the shared 'tools'
module.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-01-25 12:36:10 -07:00
Simon Glass
c761cf778f tools: Add init files for Python tools
Add some empty __init__ files for binman, buildman and dtoc so that
pylint is able to recognise these as Python modules and produce more
useful pylint output.

Signed-off-by: Simon Glass <sjg@chromium.org>
2022-01-24 16:03:27 -05:00
Tom Rini
f7832ee552 buildman: Add gcc-11.1.0 to the directory list
While CI has been using gcc-11.1.0 for a long time, we have not updated
buildman to match.  Correct this omission.

Signed-off-by: Tom Rini <trini@konsulko.com>
2021-10-21 12:50:48 -06:00
Simon Glass
7bf83a5d7b buildman: Detect Kconfig loops
Hex and int Kconfig options are supposed to have defaults. This is so we
can configure U-Boot without having to enter particular values for the
items that don't have specific values in the board's defconfig file.

If this rule is not followed, then introducing a new Kconfig can produce
a loop like this:

   Break things (BREAK_ME) [] (NEW)
   Error in reading or end of file.

   Break things (BREAK_ME) [] (NEW)
   Error in reading or end of file.

The continues forever since buildman passes /dev/null to 'conf', and
the build system just tries again. Eventually there is so much output that
buildman runs out of memory.

We can detect this situation by looking for a symbol (like 'BREAK_ME')
which has no default (the '[]' above) and is marked as new. If this
appears multiple times in the output, we know something is wrong.

Add a filter function for the output which detects this situation. Allow
it to return True to terminate the process. Implement this termination in
cros_subprocess.

With this we get a nice message:

   buildman --board sandbox -T0
   Building current source for 1 boards (0 threads, 32 jobs per thread)
      sandbox:  w+   sandbox
   +.config:66:warning: symbol value '' invalid for BREAK_ME
   +
   +Error in reading or end of file.
   +make[3]: *** [scripts/kconfig/Makefile:75: syncconfig] Terminated
   +make[2]: *** [Makefile:569: syncconfig] Terminated
   +make: *** [Makefile:177: sub-make] Terminated
   +(** did you define an int/hex Kconfig with no default? **)

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-10-20 10:59:55 +02:00
Simon Glass
bafdeb4546 buildman: Write output even on fatal error
At present buildman does not write any output (to the 'out' and 'err)
files if the build terminates with a fatal error. This is to avoid adding
lots of spam to the logs.

However there are times when this is actually useful, such as when the
build fails for an obscure reason such as a Kconfig loop.

Update the logic to always write the output, so that the user gets a clue
as to what is happening.

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-10-20 10:59:54 +02:00
Paul Barker
5fe50f9a40 tools: Refactor full help printing
Collect the code for printing the full help message of patman, buildman
and binman into a single function in patman.tools.

Signed-off-by: Paul Barker <paul.barker@sancloud.com>
2021-09-24 14:30:46 -04:00
Simon Glass
5ed16a9511 net: Rename SPL_NET_SUPPORT to SPL_NET
Rename this option so that CONFIG_IS_ENABLED can be used with it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-09-04 12:48:53 -04:00
Simon Glass
103c5f1806 mmc: Rename MMC_SUPPORT to MMC
Rename these options so that CONFIG_IS_ENABLED can be used with them.

Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Jaehoon Chung <jh80.chung@samsung.com>
[trini: Fixup some incorrect renames]
Signed-off-by: Tom Rini <trini@konsulko.com>
2021-09-04 11:42:41 -04:00
Simon Glass
333e4a621d Rename SPL_USB_HOST_SUPPORT to SPL_USB_HOST
Rename this option so that CONFIG_IS_ENABLED can be used with it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-07-28 14:27:54 -04:00
Simon Glass
f1a83abe60 buildman: Use bytes for the environment
At present we sometimes see problems in gitlab where the environment has
0x80 characters or sequences which are not valid UTF-8.

Avoid this by using bytes for the environment, both internal to buildman
and when writing out the 'env' file. Add a test to make sure this works
as expected.

Reported-by: Marek Vasut <marex@denx.de>
Fixes: e5fc79ea71 ("buildman: Write the environment out to an 'env' file")
Signed-off-by: Simon Glass <sjg@chromium.org>
2021-04-29 03:23:39 -07:00
Simon Glass
8116c78ffd buildman: Handle exceptions in threads gracefully
There have been at least a few cases where an exception has occurred in a
thread and resulted in buildman hanging: running out of disk space and
getting a unicode error.

Handle these by collecting a list of exceptions, printing them out and
reporting failure if any are found. Add a test for this.

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-04-29 03:23:39 -07:00
Simon Glass
ab9b4f35e3 buildman: Use common code to send an result
At present the code to report a build result is duplicated. Put it in a
common function to avoid this.

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-04-29 03:23:39 -07:00
Simon Glass
249933136f buildman: Tidy up a few comments
Add some function comments which are missing, or missing arguments.

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-04-29 03:23:39 -07:00
Simon Glass
5f86454b3d buildman: Add an encoding to the out-env file
The environment may contain some unicode characters. At least that is what
seemed to happen on one commit:

Building current source for 1 boards (0 threads, 64 jobs per thread)
    0    0    0 /1       -1      (starting)
Traceback (most recent call last):
  File ".../tools/buildman/buildman", line 64, in <module>
    ret_code = control.DoBuildman(options, args)
  File "tools/buildman/control.py", line 372, in DoBuildman
    options.keep_outputs, options.verbose)
  File ".../tools/buildman/builder.py", line 1704, in BuildBoards
    results = self._single_builder.RunJob(job)
  File ".../tools/buildman/builderthread.py", line 526, in RunJob
    self._WriteResult(result, job.keep_outputs, job.work_in_output)
  File ".../tools//buildman/builderthread.py", line 349, in _WriteResult
    print('%s="%s"' % (var, env[var]), file=fd)
UnicodeEncodeError: 'ascii' codec can't encode characters in position
	311-312: ordinal not in range(128)

The problem defies repetition with any change at all to buildman. But
let's set an encoding in any case.

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-04-06 16:33:19 +12:00
Simon Glass
b82492bbcc buildman: Support single-threaded operation
At present even if only a single thread is in use, buildman still uses
threading.

For some debugging it is helpful to do everything in the main process.
Allow -T0 to support this.

Signed-off-by: Simon Glass <sjg@chromium.org>
2021-03-04 17:51:43 -05:00
Heinrich Schuchardt
fd434f47d4 buildman: 'Thread' object has no attribute 'isAlive'
The isAlive() method was deprecated in Python 3.8 and has been removed in
Python 3.9. See https://bugs.python.org/issue37804. Use is_alive() instead.

Since Python 2.6 is_alive() has been a synonym for isAlive(). So there
should be no problems for users using elder Python 3 versions.

Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
Reviewed-by: Simon Glass <sjg@chromium.org>
2021-02-14 11:32:04 +01:00
Simon Glass
73da3d2cff buildman: Remove output binaries before building
Buildman reuses build directories from previous builds to avoid the cost
of 'make mrproper' for every build. If the previous build produced an SPL
image but the current one does not, the SPL image will remain and buildman
will think it is a result of building the current board.

Remove these files before building, to avoid this problem.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-12-22 20:39:26 -07:00
Simon Glass
d93720e138 patman: Rename functions in patchstream
Rename these functions to lower case as per PEP8.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-11-05 09:11:31 -07:00
Alper Nebi Yasak
76de29fc4f buildman: Use git worktrees instead of git clones when possible
This patch makes buildman create linked working trees instead of clones
of the source repository, but keeps updating the older clones of the
repository that might already exist. These worktrees share "everything
except working directory specific files such as HEAD, index, etc." with
the source repository. See the git-worktree(1) manual page for more
information.

If git-worktree isn't available, silently falls back to cloning the
repository.

Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org>
2020-09-22 12:50:43 -06:00
Simon Glass
2b522f1e79 tools: Drop unnecessary use of __file__
There are few places where the path of the current modules is calculated
but not used. Drop them.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-09-22 12:50:43 -06:00
Simon Glass
38f159c05b buildman: Show the build rate at the end
It is interesting to note the number of builds completed per second to
track machine performance and build speed. Add a 'rate' value at the end
of the build to show this.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-28 19:30:39 -06:00
Simon Glass
3918dfaa91 buildman: Correct the testOutputDir() unit test
This current fails with an error. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>
Fixes: 7664b03ffc ("buildman: Remove _of_#_ from results directory paths")
2020-07-28 19:30:39 -06:00
Simon Glass
5a910b92bc buildman: Allow using older versions of genboardscfg.py
Older versions of this script don't support the -q flag. Since buildman
runs this script from when it starts, we may get the old version.

Fix this in two ways:

1. Use the version from the same tree as buildman is run from, if
available
2. Failing that, allow the -q flag to be missing

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-07-28 19:30:39 -06:00
Tom Rini
ddf34e7d3c kconfiglib: Update to the 14.1.0 release
A large number of changes have happened upstream since our last sync
in commit 65e05ddc1a ("kconfiglib: Update to the 12.14.0 release").

The big motivation for this sync is support for user defined macros
within Kconfig.

Cc: Masahiro Yamada <yamada.masahiro@socionext.com>
Signed-off-by: Tom Rini <trini@konsulko.com>
2020-07-01 10:11:03 -04:00
Ovidiu Panait
7664b03ffc buildman: Remove _of_#_ from results directory paths
Currently, the following scenario will rebuild the first commit even
though it is not really necessary - the commit sha or the position in the
patchset did not change:

$ git am <local-patch-0001>
$ tools/buildman/buildman -P -E -W -b master mx6
<do some more development work>
$ git am <local-patch-0002>
$ tools/buildman/buildman -P -E -W -b master mx6 <- will rebuild the first
						    commit as well, even
						    though nothing has
						    changed about it.

This is due to the fact that previous results directories get removed
when the number of commits change. By removing the _of_#_ part of the
directory path, the commits will be rebuilt only if the commit sha or the
position in the patchset changes. Also, update the testcase to reflect this
change.

Signed-off-by: Ovidiu Panait <ovidiu.panait@windriver.com>
2020-06-11 20:52:11 -06:00
Simon Glass
b4fa94959d patman: Tidy up sys.path changes
Now that we are using absolute paths we can remove some of the sys.path
mangling that appears in the tools.

We only need to add the path to 'tools/' so that everything can find
modules relative to that directory.

The special paths for finding pylibfdt remain.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-26 14:25:21 -06:00
Simon Glass
bf776679a7 patman: Move to absolute imports
At present patman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move patman to use absolute imports. This requires changes in tools which
use the patman libraries (which is most of them).

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-26 14:25:21 -06:00
Simon Glass
0ede00fdaf buildman: Move to absolute imports
At present buildman sets the python path on startup so that it can access
the libraries it needs. If we convert to use absolute imports this is not
necessary.

Move buildman to use absolute imports. Also adjust moveconfig.py too since
it uses some buildman modules and cannot work without this.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-26 14:25:21 -06:00
Simon Glass
7ec3dc57b3 buildman: Rename the main module
Python does not like the module name being the same as the module
directory. To allow buildman modules to be used from other tools, rename
it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-26 14:25:21 -06:00
Simon Glass
83a4518771 patman: Drop references to __future__
We don't need these now that the tools using Python 3. Drop them.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-26 14:25:21 -06:00
Simon Glass
60b285f8c3 buildman: Write output files when using -w
At present buildman does not write its own output files (err, done, the
environment) when using -w. However this is useful for when the build is
run with -s to check it.

In fact ProduceResultSummary() reads the result from those files rather
than using the 'result' info directly. So ProcessResult() does not work
with -w at present. It does not print any output.

Fix this by writing output files even when -w is used.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-26 14:24:08 -06:00
Simon Glass
166a98a426 buildman: Use out-env for environment output
At present the environment used by U-Boot is written to the 'env'
directory. This is fine when the output directory is not the same as the
source directory, but when it is (as with -w) it conflicts with the source
directory of the same name.

Rename 'env' to 'out-env' to fix this.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-26 14:24:08 -06:00
Simon Glass
88daaef19f buildman: Make sure that -o is given with -w
It is a bad idea to use the default output directory ('..') with -w since
it does a build in that directory and writes various files these.

Require that -o is given to avoid this problem.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-26 14:24:08 -06:00
Simon Glass
97944d3f7d buildman: Correct operation of -A flag
This was broken when -a was removed and unfortunately there are no tests
for this. Fix it.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-26 14:24:08 -06:00
Simon Glass
55a98d96bd buildman: Fix test for new 9.2 kernel
The naming is slightly different on kernel.org now. Update the regex so
that the test still passes.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-26 14:24:08 -06:00
Simon Glass
b1e5e6d224 buildman: Change the exit codes
The current exit codes of 128 and 129 are useful in that they do not
conflict with those returned by tools, but they are not actually valid.
It seems better to pick some codes which work with 'bit bisect run'.

Update them to 100 (for errors) and 101 (for warnings).

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-21 06:33:47 -06:00
Simon Glass
a38930e227 buildman: Update the TODO items
A few of these have been done. Drop those and add some new ideas.

Signed-off-by: Simon Glass <sjg@chromium.org>
2020-04-21 06:33:47 -06:00