At present this function does not run the doctests. Allow the caller to
pass these modules in as strings.
Update patman to use this.
Signed-off-by: Simon Glass <sjg@chromium.org>
The compression functions are not actually used by patman, so we don't
need then in the tools module. Also we want to change them to use
bintools, which patman will not support.
Move these into a new comp_util module, within binman.
Signed-off-by: Simon Glass <sjg@chromium.org>
Update the ifwi entry type to use this bintool, instead of running
ifwitool directly. This simplifies the code and provides more
consistency as well as supporting missing bintools.
Signed-off-by: Simon Glass <sjg@chromium.org>
The Run() function automatically uses the PATH variable to locate a tool
when running it. Add a function that does this manually, so we don't have
to run a tool to find out if it is present.
This is needed by the new Bintool class, which wants to check which tools
are present.
Signed-off-by: Simon Glass <sjg@chromium.org>
Reverse the order of the return tuple, so that the filename is first.
This seems more obvious than putting the temporary directory first.
Correct a bug that leaves a space on the final line.
Allow the caller to control the name of the temporary directory.
Signed-off-by: Simon Glass <sjg@chromium.org>
Add a new function which returns the entire result from running a tool,
not just stdout. Update Run() to use this and to return stdout on error,
if stderr is empty, since some unfortunate tools write their error
output to stdout rather than stderr.
Move building of the PATH to a separate function.
Make the exception catching more specific, to catch just ValueError, since
broad exceptions are a pain to debug.
Signed-off-by: Simon Glass <sjg@chromium.org>
Python doesn't naturally support tilde (~) as a user-home marker in
paths, but git-config does. So we need to resolve it before continuing.
We also shouldn't blindly join the top-level tree with the aliasesfile
path, because it might be an absolute path.
This resolves warnings like the following:
Warning: Cannot find alias file '/path/to/source/tree/~/.git-email'
Seen when git-config is like:
$ git config sendemail.aliasesfile
~/.git-email
Signed-off-by: Brian Norris <briannorris@chromium.org>
Reviewed-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Otavio Salvador <otavio@ossystems.com.br>
The Exception base class is a very vague and could be confusing to the
test system. Use the more specific ValueError exception instead.
Signed-off-by: Simon Glass <sjg@chromium.org>
In some communities, it may be necessary to append something after PATCH
in the subject line. For example, the Linux networking subsystem
expects [1] patch subject prefixes like [RFC PATCH net-next 0/99]. This
adds support for such "postfix"s to patman. Although entirely cosmetic,
it is still nice to have.
[1] https://www.kernel.org/doc/html/latest/networking/netdev-FAQ.html#how-do-i-indicate-which-tree-net-vs-net-next-my-patch-should-be-in
Signed-off-by: Sean Anderson <sean.anderson@seco.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Hex and int Kconfig options are supposed to have defaults. This is so we
can configure U-Boot without having to enter particular values for the
items that don't have specific values in the board's defconfig file.
If this rule is not followed, then introducing a new Kconfig can produce
a loop like this:
Break things (BREAK_ME) [] (NEW)
Error in reading or end of file.
Break things (BREAK_ME) [] (NEW)
Error in reading or end of file.
The continues forever since buildman passes /dev/null to 'conf', and
the build system just tries again. Eventually there is so much output that
buildman runs out of memory.
We can detect this situation by looking for a symbol (like 'BREAK_ME')
which has no default (the '[]' above) and is marked as new. If this
appears multiple times in the output, we know something is wrong.
Add a filter function for the output which detects this situation. Allow
it to return True to terminate the process. Implement this termination in
cros_subprocess.
With this we get a nice message:
buildman --board sandbox -T0
Building current source for 1 boards (0 threads, 32 jobs per thread)
sandbox: w+ sandbox
+.config:66:warning: symbol value '' invalid for BREAK_ME
+
+Error in reading or end of file.
+make[3]: *** [scripts/kconfig/Makefile:75: syncconfig] Terminated
+make[2]: *** [Makefile:569: syncconfig] Terminated
+make: *** [Makefile:177: sub-make] Terminated
+(** did you define an int/hex Kconfig with no default? **)
Signed-off-by: Simon Glass <sjg@chromium.org>
When printing full help output from a tool, we should be able to handle
a PAGER variable which includes arguments, e.g. PAGER='less -F'.
Signed-off-by: Paul Barker <paul.barker@sancloud.com>
Collect the code for printing the full help message of patman, buildman
and binman into a single function in patman.tools.
Signed-off-by: Paul Barker <paul.barker@sancloud.com>
In some cases 'patman status' leaves a blank line between the sign-off
and the tags it collects from patchwork. Fix this and add a test.
Signed-off-by: Simon Glass <sjg@chromium.org>
Add a error in patman tool when the commit message contents an invalid
tag "Serie-.*" instead of "Series-.*".
Signed-off-by: Patrick Delaunay <patrick.delaunay@foss.st.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
If the process outputs a lot of data on stdout this can be quite slow,
since the bytestring is regenerated each time. Use a bytearray instead.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present compression uses the same temporary file for all invocations.
With multithreading this causes the data to become corrupted. Use a
different filename each time.
Signed-off-by: Simon Glass <sjg@chromium.org>
In commit 1e4687aa47 ("binman: Use target-specific tools when
cross-compiling"), a utility function was implemented to get preferred
compilation tools using environment variables like CC and CROSS_COMPILE.
Although it intended to provide custom default tools (same as those in
the global Makefile) when no relevant variables were set (for example
using "gcc" for "cc"), it is only doing so when CROSS_COMPILE is set and
returning the literal name of the tool otherwise.
Remove the check for an empty CROSS_COMPILE, which makes the function
use it as an empty prefix to the custom defaults and return the intended
executables.
Fixes: 1e4687aa47 ("binman: Use target-specific tools when cross-compiling")
Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
Parse each empty-line-delimited message separately. This saves having to
deal with all the different line content styles, we only care about the
header ERROR | WARNING | NOTE...
Also make checkpatch print line information for a uboot specific
warning.
Signed-off-by: Evan Benn <evanbenn@chromium.org>
Reviewed-by: Simon Glass <sjg@chromium.org>
Given that we have tests that require pygit2 and it can be installed
like any other python module, fail much more loudly if it is missing.
Cc: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
strn(cat|cpy) has a bad habit of not nul-terminating the destination,
resulting in constructions like
strncpy(foo, bar, sizeof(foo) - 1);
foo[sizeof(foo) - 1] = '\0';
However, it is very easy to forget about this behavior and accidentally
leave a string unterminated. This has shown up in some recent coverity
scans [1, 2] (including code recently touched by yours truly).
Fortunately, the guys at OpenBSD came up with strl(cat|cpy), which always
nul-terminate strings. These functions are already in U-Boot, so we should
encourage new code to use them instead of strn(cat|cpy).
[1] https://lists.denx.de/pipermail/u-boot/2021-March/442888.html
[2] https://lists.denx.de/pipermail/u-boot/2021-January/438073.html
Signed-off-by: Sean Anderson <seanga2@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
While we cannot know which commit the warning relates to, this should not
be fatal. Print the warning and carry on.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present, before any entry expansion is done (such as a 'files' entry
expanding out to individual entries for each file it contains), we check
the binman definition (i.e. '/binman' node) to find out what devicetree
files are used in the images.
This is a pain, since the definition may change during expansion. For
example if there is no u-boot-spl-dtb entry in the definition at the start,
we assume that the SPL devicetree is not used. But if an entry later
expands to include this, then we don't notice.
In fact the flexibility provided by the current approach of checking the
definition is not really useful. We know that we can have SPL and TPL
devicetrees. We know the pathname to each, so we can simply check if the
files are present. If they are present, we can prepare them and update
them regardless of whether they are actually used. If they are not present,
we cannot prepare/update them anyway, i.e. an error will be generated.
Simplify state.Prepare() so it uses a hard-coded list of devicetree files.
Note that state.PrepareFromLoadedData() is left untouched, since in that
case we have a complete definition from the loaded file, but cannot of
course rely on the devicetree files that created it still being present.
So in that case we still check the image defitions.
Signed-off-by: Simon Glass <sjg@chromium.org>
It's convenient to be able to scroll up in `patman -H`.
Signed-off-by: Nicolas Boichat <drinkcat@chromium.org>
Reviewed-by: Simon Glass <sjg@chromium.org>
When a tag is used in a patch subject (e.g. "tag: rest of message") and
it cannot be found as an alias, patman currently reports a fatal error,
unless -t is provided, in which case it reports a warning.
Experience suggest that the fatal error is not very useful. Instead,
default to reporting a warning, with -t tell patman to ignore it
altogether.
Signed-off-by: Simon Glass <sjg@chromium.org>
With the subcommands some of the documentation examples are no-longer
correct. Fix all of them, so it is consistent.
Signed-off-by: Simon Glass <sjg@chromium.org>
patman's --limit-cc option parses its argument to an integer and uses
that to trim the list of CC recipients to a particular maximum. but that
only works if the cc variable is a list, which it is not.
Signed-off-by: Bernhard Kirchen <bernhard.kirchen@mbconnectline.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
At present on large files, lz4 uses a larger block size (e.g. 256KB) than
the 64KB supported by the U-Boot decompression implementation. Also it is
optimised for maximum compression speed, producing larger output than we
would like.
Update the parameters to correct these problems.
Signed-off-by: Simon Glass <sjg@chromium.org>
Driver model: Rename U_BOOT_DEVICE et al.
dtoc: Tidy up and add more tests
ns16550 code clean-up
x86 and sandbox minor fixes for of-platdata
dtoc prepration for adding build-time instantiation
-----BEGIN PGP SIGNATURE-----
iQFFBAABCgAvFiEEslwAIq+Gp8wWVbYnfxc6PpAIreYFAl/09LURHHNqZ0BjaHJv
bWl1bS5vcmcACgkQfxc6PpAIrebjwwf/fHRjYsAY/Yj/+y1xgo3L3sphIvQUqTDF
KkLl+kHdV5r8W/HJULxLQcF2r7pcPEI6TAQxuj3qQ5SUvm2HviS8GHGPawDEwyht
HgBp9VD56+HUadMfnbG//DVS73ycbL4XSKlYqpkINEejtnlttsCIawUXX5cTyGM/
59VkgnKrKvJQRUXvYLa8MTugTs4fkPJGDqhActBk/7SP1SImj+rfalNSqA2/dx6y
2RnPCSzB1x2231KSj+B1NgGlR3Xb8P8zgh20ijcEU/hrlXBTZyi7K7f4SJR30Efu
LYkkuj4VbxcV/25RozR0fmknqCs0QyAI+/dql6TNtbTSPC/jAfj0jQ==
=9kN3
-----END PGP SIGNATURE-----
Merge tag 'dm-pull-5jan21' of git://git.denx.de/u-boot-dm into next
Driver model: make some udevice fields private
Driver model: Rename U_BOOT_DEVICE et al.
dtoc: Tidy up and add more tests
ns16550 code clean-up
x86 and sandbox minor fixes for of-platdata
dtoc prepration for adding build-time instantiation
With upcoming changes, dtoc will output several files for different
of-platdata components.
Add a way to output all ava!ilable files at once ('all'), to the
appropriate directories, without needing to specify each one invidually.
This puts the commands in alphabetical order, so update the tests
accordingly.
Signed-off-by: Simon Glass <sjg@chromium.org>
Add defaults for FSF/GNU projects, such as gcc, that provide sensible
settings for those projects.
Signed-off-by: Philipp Tomsich <philipp.tomsich@vrull.eu>
Reviewed-by: Simon Glass <sjg@chromium.org>
To enable use of patman with FSF/GNU projects, such as GCC or
Binutils, no Signed-off-by may be added. This adds a command
line flag '--no-signoff' to suppress adding signoffs in patman
when processing commits.
Signed-off-by: Philipp Tomsich <philipp.tomsich@vrull.eu>
Reviewed-by: Simon Glass <sjg@chromium.org>
Fix patman testBranch() test:
Signed-off-by: Simon Glass <sjg@chromium.org>
As a way of keeping the driver declarations more consistent, add a warning
if the struct used does not end with _priv or _plat.
Signed-off-by: Simon Glass <sjg@chromium.org>
This is useful anymore, since we always want to call chr() in Python 3.
Drop it and adjust callers to use chr().
Also drop ToChars() which is no-longer used.
Signed-off-by: Simon Glass <sjg@chromium.org>
We don't need these now that everything uses Python 3. Remove them and
the extra code in GetBytes() and ToBytes() too.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present if CROSS_COMPILE contains a tilde, such as
~/.buildman-toolchains/gcc-7.3.0-nolibc/i386-linux/bin/i386-linux-gcc
then binman gives a confusing error:
binman: Error 255 running '~/..buildman-toolchains/gcc-7.3.0- ...
Fix this by expanding it out before running the tool.
Signed-off-by: Simon Glass <sjg@chromium.org>
Add a commit tag to allow the Patchwork URL to be specified in a commit.
This can be handy for when you submit code to multiple projects but don't
want to use the -p option.
Signed-off-by: Simon Glass <sjg@chromium.org>
Add an argument to allow specifying the the patchwork URL. This also adds
this feature to the settings file, either globally, or on a per-project
basis.
Signed-off-by: Simon Glass <sjg@chromium.org>
Add a new argument to allow the URL of the patchwork server to be
speciified. For now this is hard-coded in the main file, but future
patches will move it to the settings file.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present values from the settings file are only applied to the main
parser. With the new parser structure this means that some settings are
ignored.
Update the implementation to set defaults across the main parser and all
subparsers. Also fix up the comments, since ArgumentParser is being used
now.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present patman tries to assume a default subcommand of 'send', to
maintain backwards compatibility. However it does not cope with
arguments added to the default command, so for example 'patman -t'
does not work.
Update the logic to handle this. Also update the CC command to use 'send'
explicitly, since otherwise patman gets confused with the patch-filename
argument.
Signed-off-by: Simon Glass <sjg@chromium.org>
While reviewing feedback it is helpful to see the review comments on the
command line to check that each has been addressed. Add an option to
support that.
Update the workflow documentation to describe the new features.
Signed-off-by: Simon Glass <sjg@chromium.org>
Add support for parsing the contents of a patchwork 'patch' web page
containing comments received from reviewers. This allows patman to show
these comments in a simple 'snippets' format.
A snippet is some quoted code plus some unquoted comments below it. Each
review is from a unique person/email and can produce multiple snippets,
one for each part of the code that attracts a comment.
Show the file and line-number info at the top of each snippet if
available.
Signed-off-by: Simon Glass <sjg@chromium.org>
It is tedious to add review tags into the local branch and errors can
sometimes be made. Add an option to create a new branch with the review
tags obtained from patchwork.
Signed-off-by: Simon Glass <sjg@chromium.org>
Before sending out a new version of a series for review, it is important
to add any review tags (e.g. Reviewed-by, Acked-by) collected by
patchwork. Otherwise people waste time reviewing the same patch
repeatedly, become frustrated and stop reviewing your patches.
To help with this, add a new 'status' subcommand that checks patchwork
for review tags, showing those which are not present in the local branch.
This allows users to see what new review tags have been received and then
add them.
Sample output:
$ patman status
1 Subject 1
Reviewed-by: Joe Bloggs <joe@napierwallies.co.nz>
2 Subject 2
Tested-by: Lord Edmund Blackaddër <weasel@blackadder.org>
Reviewed-by: Fred Bloggs <f.bloggs@napier.net>
+ Reviewed-by: Mary Bloggs <mary@napierwallies.co.nz>
1 new response available in patchwork
The '+' indicates a new tag. Colours are used to make it easier to read.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present if we fail to find the upstream then the error output is piped
to wc, resulting in bogus results. Avoid the pipe and check the output
directly.
Signed-off-by: Simon Glass <sjg@chromium.org>
Sometimes warnings are associated with a file and sometimes with the
patch as a whole. Update the regular expression to handle both cases,
even in emacs mode. Also add support for detecting new files.
Signed-off-by: Simon Glass <sjg@chromium.org>
These lines can indicate a continuation of an error and should not be
ignored. Fix this.
Fixes: 666eb15e92 ("patman: Handle checkpatch output with notes and code")
Signed-off-by: Simon Glass <sjg@chromium.org>
On balance it is easier to use an iterator here, particularly if we need
to insert lines due to new functionality. The only niggle is the need to
keep the previous iterator value around in one case.
Convert this test to use iter().
Signed-off-by: Simon Glass <sjg@chromium.org>
The current functional tests run most of patman. Add a smaller test that
just checks tag handling with the PatchStream class.
Signed-off-by: Simon Glass <sjg@chromium.org>
If the Series-xxx tag is not recognised patman currently reports a fatal
error. This is inconvenient if a new feature is later added to patman that
an earlier version does not support.
Report a warning instead, to allow the user to take action if needed, but
still allow operation to proceed.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present warnings are produced across the whole set of patches when
parsing them. It is more useful to associate each warning with the patch
(or commit) that generated it.
Attach warnings to the Commit object and move them out of PatchStream.
Also avoid generating duplicate warnings for the same commit.
Signed-off-by: Simon Glass <sjg@chromium.org>
Add a new function in PatchStream to collect the warnings generated while
parsing the stream. This will allow us to adjust the logic, such as
dealing with per-commit warnings.
Two of the warnings are in fact internal errors, so change them to raise
and exception.
Signed-off-by: Simon Glass <sjg@chromium.org>
Add a new Series-links tag to tell patman how to find the series in
patchwork. Each item is the series ID optionally preceded by the series
version that the link refers to. An empty version indicates this is the
latest series.
For example:
Series-links: 209816 1:203302
Documentation is added in a later patch.
Signed-off-by: Simon Glass <sjg@chromium.org>
One test still uses its own function for capturing output. Modify it to
use the standard one in test_util
Signed-off-by: Simon Glass <sjg@chromium.org>
This operation was unfortunately broken by a recent change. It is now
necessary to use -i in addition to -n, if there are errors or warnings in
the patches.
Correct this by always showing the summary information.
Fixes: f365375975 ("patman: Move main code out to a control module")
Signed-off-by: Simon Glass <sjg@chromium.org>
CONFIG_IS_ENABLED() takes the kconfig name without the CONFIG_ prefix,
e.g. CONFIG_IS_ENABLED(CLK) for CONFIG_CLK. Make including the prefix
an error in checkpatch.pl so calls in the wrong format aren't
accidentally reintroduced.
Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
This patch lets tools.Run() use host-specific versions with the
for_host keyword argument, based on the host-specific environment
variables (HOSTCC, HOSTOBJCOPY, HOSTSTRIP, etc.).
Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Currently, binman always runs the compile tools like cc, objcopy, strip,
etc. using their literal name. Instead, this patch makes it use the
target-specific versions by default, derived from the tool-specific
environment variables (CC, OBJCOPY, STRIP, etc.) or from the
CROSS_COMPILE environment variable.
For example, the u-boot-elf etype directly uses 'strip'. Trying to run
the tests with 'CROSS_COMPILE=i686-linux-gnu- binman test' on an arm64
host results in the '097_elf_strip.dts' test to fail as the arm64
version of 'strip' can't understand the format of the x86 ELF file.
This also adjusts some command.Output() calls that caused test errors or
failures to use the target versions of the tools they call. After this,
patch, an arm64 host can run all tests with no errors or failures using
a correct CROSS_COMPILE value.
Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
This patch makes buildman create linked working trees instead of clones
of the source repository, but keeps updating the older clones of the
repository that might already exist. These worktrees share "everything
except working directory specific files such as HEAD, index, etc." with
the source repository. See the git-worktree(1) manual page for more
information.
If git-worktree isn't available, silently falls back to cloning the
repository.
Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Simon Glass <sjg@chromium.org>
With the change to absolute imports the concurrent tests feature
unfortunately broke. Fix it.
We cannot easy add a warning, since the output messes up tests which check
the output.
Signed-off-by: Simon Glass <sjg@chromium.org>
The user can either count the number of patches, or provide a
tracking branch.
Signed-off-by: Nicolas Boichat <drinkcat@chromium.org>
Reviewed-by: Simon Glass <sjg@chromium.org>
Setting sendemail.suppresscc to all or cccmd leads to --cc-cmd
parameter being ignored, and emails going either nowhere, or
just to the To: line maintainer.
Signed-off-by: Nicolas Boichat <drinkcat@chromium.org>
Reviewed-by: Simon Glass <sjg@chromium.org>
The unsupported Commit-xxx option are silently skipped
and removed as 're_remove=Commit-\w*', this patch adds
warning message in this case to detect misspelled issue
for the 2 supported options:
Commit-notes:
Commit-changes:
For example: the final 's' is missing (Commit-note:)
NB: no issue for Series-xxx option as only the supported
options are accepted (see valid_series in series.py)
Signed-off-by: Patrick Delaunay <patrick.delaunay@st.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Detect unexpected 'END' line when a section is not detected.
This patch detect issue when tag name for section start is misspelled,
for example 'Commit-note:' for 'Commit-notes:'
Commit-note:
....
END
Then 'Commit-note:' is removed silently by re_remove = "Commit-\w*:"
but 'END' is kept in commit message.
Signed-off-by: Patrick Delaunay <patrick.delaunay@st.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
When warnings and errors are produced by tools they should be written to
stderr. Update the tout implementation to handle this.
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
Sometimes it is useful to build an image even though external binaries are
not present. This allows the build system to continue to function without
these files, albeit not producing valid images.
U-Boot does with with ATF (ARM Trusted Firmware) today.
Add a new flag to binman to request this behaviour.
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
