Update the zstd implementation to match Linux zstd 1.5.2 from commit
2aa14b1ab2.
This was motivated by running into decompression corruption issues when
trying to uncompress files compressed with newer versions of zstd. zstd
users also claim significantly improved decompression times with newer
zstd versions which is a side benefit.
Original zstd code was copied from Linux commit 2aa14b1ab2 which is a
custom-built implementation based on zstd 1.3.1. Linux switched to an
implementation that is a copy of the upstream zstd code in Linux commit
e0c1b49f5b, this results in a large code diff. However this should make
future updates easier along with other benefits[1].
This commit is a straight mirror of the Linux zstd code, except to:
- update a few #include that do not translate cleanly
- linux/swab.h -> asm/byteorder.h
- linux/limits.h -> linux/kernel.h
- linux/module.h -> linux/compat.h
- remove assert() from debug.h so it doesn't conflict with u-boot's
assert()
- strip out the compressor code as was done in the previous u-boot zstd
- update existing zstd users to the new Linux zstd API
- change the #define for MEM_STATIC to use INLINE_KEYWORD for codesize
- add a new KConfig option that sets zstd build options to minify code
based on zstd's ZSTD_LIB_MINIFY[2].
These changes were tested by booting a zstd 1.5.2 compressed kernel inside a
FIT. And the squashfs changes by loading a file from zstd compressed squashfs
with sqfsload. buildman was used to compile test other boards and check for
binary bloat, as follows:
> $ buildman -b zstd2 --boards dh_imx6,m53menlo,mvebu_espressobin-88f3720,sandbox,sandbox64,stm32mp15_dhcom_basic,stm32mp15_dhcor_basic,turris_mox,turris_omnia -sS
> Summary of 6 commits for 9 boards (8 threads, 1 job per thread)
> 01: Merge branch '2023-01-10-platform-updates'
> arm: w+ m53menlo dh_imx6
> 02: lib: zstd: update to latest Linux zstd 1.5.2
> aarch64: (for 2/2 boards) all -3186.0 rodata +920.0 text -4106.0
> arm: (for 5/5 boards) all +1254.4 rodata +940.0 text +314.4
> sandbox: (for 2/2 boards) all -4452.0 data -16.0 rodata +640.0 text -5076.0
[1] e0c1b49f5b
[2] f302ad8811/lib/libzstd.mk (L31)
Signed-off-by: Brandon Maier <brandon.maier@collins.com>
[trini: Set ret to -EINVAL for the error of "failed to detect
compressed" to fix warning, drop ZSTD_SRCSIZEHINT_MAX for non-Linux host
tool builds]
Signed-off-by: Tom Rini <trini@konsulko.com>
[BUG]
Since btrfs supports single device RAID0 at mkfs time after btrfs-progs
v5.14, if we create a single device raid0 btrfs, and created a file
crossing stripe boundary:
# mkfs.btrfs -m dup -d raid0 test.img
# mount test.img mnt
# xfs_io -f -c "pwrite 0 128K" mnt/file
# umount mnt
Since btrfs is using 64K as stripe length, above 128K data write is
definitely going to cross at least one stripe boundary.
Then u-boot would fail to read above 128K file:
=> host bind 0 /home/adam/test.img
=> ls host 0
< > 131072 Fri Dec 30 00:18:25 2022 file
=> load host 0 0 file
BTRFS: An error occurred while reading file file
Failed to load 'file'
[CAUSE]
Unlike tree blocks read, data extent reads doesn't consider cases in which
one data extent can cross stripe boundary.
In read_data_extent(), we just call btrfs_map_block() once and read the
first mapped range.
And if the first mapped range is smaller than the desired range, it
would return error.
But since even single device btrfs can utilize RAID0 profiles, the first
mapped range can only be at most 64K for RAID0 profiles, and cause false
error.
[FIX]
Just like read_whole_eb(), we should call btrfs_map_block() in a loop
until we read all data.
Since we're here, also add extra error messages for the following cases:
- btrfs_map_block() failure
We already have the error message for it.
- Missing device
This should not happen, as we only support single device for now.
- __btrfs_devread() failure
With this bug fixed, btrfs driver of u-boot can properly read the above
128K file, and have the correct content:
=> host bind 0 /home/adam/test.img
=> ls host 0
< > 131072 Fri Dec 30 00:18:25 2022 file
=> load host 0 0 file
131072 bytes read in 0 ms
=> md5sum 0 0x20000
md5 for 00000000 ... 0001ffff ==> d48858312a922db7eb86377f638dbc9f
^^^ Above md5sum also matches.
Reported-by: Sam Winchenbach <swichenbach@tethers.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
For a squashfs filesystem, the fragment table is followed by
the following tables: NFS export table, ID table, xattr table.
The export and xattr tables are both completely optional, but
the ID table is mandatory. The Linux implementation refuses to
mount the image if the ID table is missing. Tables that are no
present have their location in the super block set
to 0xFFFFFFFFFFFFFFFF.
The u-boot implementation previously assumed that it can always
rely on the export table location as an upper bound for the fragment
table, trying (and failing) to read past filesystem bounds if it
is not present.
This patch changes the driver to use the ID table instead and only
use the export table location if it lies between the two.
Signed-off-by: David Oberhollenzer <goliath@infraroot.at>
Reviewed-by: Miquel Raynal <miquel.raynal@bootlin.com>
When compling for x86:
u-boot/fs/squashfs/sqfs.c:90: undefined reference to `__udivmoddi4'
Signed-off-by: Kasper Revsbech <kasper.revsbech.ext@siemensgamesa.com>
Tested-by: Sean Nyekjaer <sean@geanix.com>
The 'depth_dirname', 'ptr', 'parent_inode' and 'first_inode' pointers
may be null. Thus, it is necessary to check them before using free() to
avoid free(NULL) cases.
Fixes: 934b14f2bb ("ext4: free allocations by parse_path()")
Signed-off-by: Mikhail Ilin <ilin.mikhail.ol@gmail.com>
Debug dump logs are not always required. Add a new config option
UBIFS_SILENCE_DEBUG_DUMP to silence all debug dumps. On powerpc/mpc85xx
when enabled this will decrease size of U-Boot binary by 11 kB.
Signed-off-by: Pali Rohár <pali@kernel.org>
Tested-by: Tony Dinh <mibodhi@gmail.com>
Move the symbol SYS_JFFS2_SORT_FRAGMENTS to Kconfig and use the only
remaining part of doc/README.JFFS2 that is still relevant and useful to
the help for this option.
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
When looking for a filesystem on a partition we should do so quietly. At
present if the filesystem is very small (e.g. 512 bytes) we get a host of
messages.
Update these to only show when debugging.
Signed-off-by: Simon Glass <sjg@chromium.org>
This line probably got in by mistake as there is no fs_mutex member in
the btrfs_fs_info struct.
Signed-off-by: Pankaj Raghav <p.raghav@samsung.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Globally replace all occurances of WATCHDOG_RESET() with schedule(),
which handles the HW_WATCHDOG functionality and the cyclic
infrastructure.
Signed-off-by: Stefan Roese <sr@denx.de>
Reviewed-by: Simon Glass <sjg@chromium.org>
Tested-by: Tom Rini <trini@konsulko.com> [am335x_evm, mx6cuboxi, rpi_3,dra7xx_evm, pine64_plus, am65x_evm, j721e_evm]
At present we use HAVE_BLOCK_DEVICE to indicate when block devices are
available.
This is a very strange option, since it partially duplicates the BLK
option used by driver model. It also covers both U-Boot proper and SPL,
even though one might have block devices and another not.
As a first step towards correcting this, create a new inline function
called blk_enabled() which indicates if block devices are available.
This cannot be used in Makefiles, or #if clauses, but can be used in C
code.
A function is useful because we cannot use CONFIG_IS_ENABLED(BLK) to
decide if block devices are needed, since we must consider the legacy
block interface, enabled by HAVE_BLOCK_DEVICE
Update a few places where it can be used and drop some unnecessary #if
checks around some functions in disk/part.c - rely on the compiler's
dead-code elimination instead.
Signed-off-by: Simon Glass <sjg@chromium.org>
This patch integrates the main function responsible for calling all
registered cyclic functions cyclic_run() into the common WATCHDOG_RESET
macro. This guarantees that cyclic_run() is executed very often, which
is necessary for the cyclic functions to get scheduled and executed at
their configured periods.
If CONFIG_WATCHDOG is not enabled, only cyclic_run() without calling
watchdog_reset(). This guarantees that the cyclic functionality does not
rely on CONFIG_WATCHDOG being enabled.
Signed-off-by: Stefan Roese <sr@denx.de>
Reviewed-by: Simon Glass <sjg@chromium.org>
fs_set_blk_dev() probes all file-systems until it finds one that matches
the volume. We do not expect any console output for non-matching
file-systems.
Convert error messages in erofs_read_superblock() to debug output.
Fixes: 830613f8f5 ("fs/erofs: add erofs filesystem support")
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Huang Jianan <jnhuang95@gmail.com>
A large number of files include <flash.h> as it used to be how various
SPI flash related functions were found, or for other reasons entirely.
In order to migrate some further CONFIG symbols to Kconfig we need to
not include flash.h in cases where we don't have a NOR flash of some
sort enabled. Furthermore, in cases where we are in common code and it
doesn't make sense to try and further refactor the code itself in to new
files we need to guard this inclusion.
Signed-off-by: Tom Rini <trini@konsulko.com>
Fix diacritics in some instances of my name and change my e-mail address
to kabel@kernel.org.
Add corresponding .mailmap entries.
Signed-off-by: Marek Behún <kabel@kernel.org>
Reviewed-by: Stefan Roese <sr@denx.de>
U-Boot does not implement down_write_trylock() and its stub always returns
true that lock was acquired. Therefore ubifs_assert_cmt_locked() assert
currently always fails.
Fix this issue by redefining ubifs_assert_cmt_locked() to just empty stub
as there is nothing to assert.
Signed-off-by: Pali Rohár <pali@kernel.org>
Original ubifs code was designed that after ubifs_umount() call it is
required to also call ubi_close_volume() which closes underlying UBI
volume. But U-Boot ubifs modification have not implemented it properly
which caused that ubifsumount command contains resource leak. It can be
observed by calling simple sequence of commands:
=> ubi part mtd2
ubi0: attaching mtd2
...
=> ubifsmount ubi0
=> ubifsumount
Unmounting UBIFS volume rootfs!
=> ubi detach
ubi0 error: ubi_detach_mtd_dev: ubi0 reference count 1, destroy anyway
ubi0: detaching mtd2
ubi0: mtd2 is detached
Fix this issue by calling ubi_close_volume() and mutex_unlock() in
directly in ubifs_umount() function before freeing U-Boot's global
ubifs_sb. And remove duplicate calls of these two functions in remaining
places. Note that when ubifs_umount() is not called then during error
handling is still needed to call ubi_close_volume() and mutex_unlock.
With this change ubifsumount command does not throw that error anymore:
=> ubi part rootfs
ubi0: attaching mtd2
...
=> ubifsmount ubi0
=> ubifsumount
Unmounting UBIFS volume rootfs!
=> ubi detach
ubi0: detaching mtd2
ubi0: mtd2 is detached
Signed-off-by: Pali Rohár <pali@kernel.org>
Tighten up symbol dependencies in a number of places. Ensure that a SPL
specific option has at least a direct dependency on SPL. In places
where it's clear that we depend on something more specific, use that
dependency instead. This means in a very small number of places we can
drop redundant dependencies.
Reported-by: Pali Rohár <pali@kernel.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
A crafted squashfs image could embed a huge number of empty metadata
blocks in order to make the amount of malloc()'d memory overflow and be
much smaller than expected. Because of this flaw, any random code
positioned at the right location in the squashfs image could be memcpy'd
from the squashfs structures into U-Boot code location while trying to
access the rearmost blocks, before being executed.
In order to prevent this vulnerability from being exploited in eg. a
secure boot environment, let's add a check over the amount of data
that is going to be allocated. Such a check could look like:
if (!elem_size || n > SIZE_MAX / elem_size)
return NULL;
The right way to do it would be to enhance the calloc() implementation
but this is quite an impacting change for such a small fix. Another
solution would be to add the check before the malloc call in the
squashfs implementation, but this does not look right. So for now, let's
use the kcalloc() compatibility function from Linux, which has this
check.
Fixes: c510061303 ("fs/squashfs: new filesystem")
Reported-by: Tatsuhiko Yasumatsu <Tatsuhiko.Yasumatsu@sony.com>
Signed-off-by: Miquel Raynal <miquel.raynal@bootlin.com>
Tested-by: Tatsuhiko Yasumatsu <Tatsuhiko.Yasumatsu@sony.com>
Following Jincheng's report, an out-of-band write leading to arbitrary
code execution is possible because on one side the squashfs logic
accepts directory names up to 65535 bytes (u16), while U-Boot fs logic
accepts directory names up to 255 bytes long.
Prevent such an exploit from happening by capping directory name sizes
to 255. Use a define for this purpose so that developers can link the
limitation to its source and eventually kill it some day by dynamically
allocating this array (if ever desired).
Link: https://lore.kernel.org/all/CALO=DHFB+yBoXxVr5KcsK0iFdg+e7ywko4-e+72kjbcS8JBfPw@mail.gmail.com
Reported-by: Jincheng Wang <jc.w4ng@gmail.com>
Signed-off-by: Miquel Raynal <miquel.raynal@bootlin.com>
Tested-by: Jincheng Wang <jc.w4ng@gmail.com>
Setting sblk = NULL has no effect on the caller.
We want to set *sblk = NULL if an error occurrs to avoid usage after free.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
After returning if ret <= 0 we know that ret > 0. No need to check it.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: Anand Jain <anand.jain>
Commit b1a14f8a1c ("UBIFS: Change ubifsload to not read beyond the
requested size") added optimization to do not read more bytes than it is
really needed. But this commit introduced incorrect handling of the hole at
the end of file. This logic cause U-Boot to crash or lockup when trying to
read from the ubifs filesystem.
When read_block() call returns -ENOENT error (not an error, but the hole)
then dn-> structure is not filled and contain garbage. So using of dn->size
for memcpy() argument cause that U-Boot tries to copy unspecified amount of
bytes from possible unmapped memory. Which randomly cause lockup of P2020
CPU.
Fix this issue by copying UBIFS_BLOCK_SIZE bytes from read buffer when
dn->size is not available. UBIFS_BLOCK_SIZE is the size of the buffer
itself and read_block() fills buffer by zeros when it returns -ENOENT.
This patch fixes ubifsload on P2020.
Fixes: b1a14f8a1c ("UBIFS: Change ubifsload to not read beyond the requested size")
Signed-off-by: Pali Rohár <pali@kernel.org>
Reviewed-by: Stefan Roese <sr@denx.de>
When compling for x86:
ld.bfd: fs/squashfs/sqfs.o: in function `sqfs_read':
u-boot/fs/squashfs/sqfs.c:1443: undefined reference to `__udivmoddi4'
ld.bfd: u-boot/fs/squashfs/sqfs.c:1521: undefined reference to `__udivmoddi4'
Signed-off-by: Sean Nyekjaer <sean.nyekjaer.ext@siemensgamesa.com>
Reviewed-by: Miquel Raynal <miquel.raynal@bootlin.com>
Reviewed-by: Pali Rohár <pali@kernel.org>
It is helpful to be able to try out bootstd on sandbox, using host files.
This is easier than using a block device, which must have a filesystem,
partition table, etc.
Add a new driver which provides this feature. For now it is not used in
tests, but it is likely to be useful.
Add notes in the devicetree also, but don't disturb the tests.
Signed-off-by: Simon Glass <sjg@chromium.org>
When sandbox is used with hostfs we won't have a block device, but still
must set up the filesystem type before any filesystem operation, such as
loading a file. Add a function to handle this.
Signed-off-by: Simon Glass <sjg@chromium.org>
Implementation in linux/crc16.h provides standard CRC-16 algorithm with
polynomial x^16 + x^15 + x^2 + 1. Use it and remove duplicate ext4 CRC-16
specific code.
Signed-off-by: Pali Rohár <pali@kernel.org>
Reviewed-by: Stefan Roese <sr@denx.de>
File fs/ubifs/crc16.h is standard linux's crc16.h include file. So move it
from fs/ubifs to include/linux where are also other linux include files.
Signed-off-by: Pali Rohár <pali@kernel.org>
Reviewed-by: Stefan Roese <sr@denx.de>
Currently there is no btrfs support in SPL. But macro CONFIG_FS_BTRFS is
defined also when building SPL. When both FS_BTRFS and SPL are enabled
then build process throw compile error.
Fix check for btrfs code in fstypes[] to allow compiling FS_BTRFS only in
proper U-Boot.
Signed-off-by: Pali Rohár <pali@kernel.org>
Fix following two compile errors on big endian systems:
CC fs/btrfs/btrfs.o
In file included from include/linux/byteorder/big_endian.h:107,
from ./arch/powerpc/include/asm/byteorder.h:82,
from ./arch/powerpc/include/asm/bitops.h:8,
from include/linux/bitops.h:152,
from include/uuid.h:9,
from fs/btrfs/btrfs.c:10:
fs/btrfs/conv-funcs.h: In function ‘btrfs_key_to_disk’:
include/linux/byteorder/generic.h:90:21: error: ‘__cpu_to_le16’ undeclared (first use in this function); did you mean ‘__cpu_to_le16p’?
#define cpu_to_le16 __cpu_to_le16
^~~~~~~~~~~~~
fs/btrfs/conv-funcs.h:79:10: note: in expansion of macro ‘cpu_to_le16’
__u16: cpu_to_le16, \
^~~~~~~~~~~
CC fs/btrfs/compression.o
In file included from ./arch/powerpc/include/asm/unaligned.h:9,
from fs/btrfs/compression.c:16:
include/linux/unaligned/access_ok.h:6:19: error: redefinition of ‘get_unaligned_le16’
static inline u16 get_unaligned_le16(const void *p)
^~~~~~~~~~~~~~~~~~
In file included from fs/btrfs/ctree.h:16,
from fs/btrfs/btrfs.h:12,
from fs/btrfs/compression.c:8:
include/linux/unaligned/le_byteshift.h:40:19: note: previous definition of ‘get_unaligned_le16’ was here
static inline u16 get_unaligned_le16(const void *p)
^~~~~~~~~~~~~~~~~~
Include file asm/unaligned.h contains arch specific macros and functions
for unaligned access as opposite to linux/unaligned le_byteshift.h which
contains macros and functions specific to little endian systems only.
Signed-off-by: Pali Rohár <pali@kernel.org>
Reviewed-by: Marek Behún <marek.behun@nic.cz>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Currently there is no UBIFS support in SPL. But macro CONFIG_CMD_UBIFS is
defined also when building SPL. when both CMD_UBIFS and SPL are enabled
then build process throw compile error.
Fix check for ubifs code in fstypes[] to allow compiling CMD_UBIFS only in
proper U-Boot.
Signed-off-by: Pali Rohár <pali@kernel.org>
Backport commit 1cb51a15b576 ("ubifs: Fix journal replay wrt. xattr
nodes") from the Linux Kernel, which has the following Signed-off-by
line:
Signed-off-by: Richard Weinberger <richard@nod.at>
For U-Boot, after comapring with the upstream commit:
Signed-off-by: Tom Rini <trini@konsulko.com>
This adds a filesystem which is backed by the host's filesystem. It is
modeled off of sandboxfs, which has very similar aims. Semihosting
doesn't support listing directories (except with SYS_SYSTEM), so neither
do we. it's possible to optimize a bit for the common case of reading a
whole file by omitting a call to smh_seek, but this is left as a future
optimization.
Signed-off-by: Sean Anderson <sean.anderson@seco.com>
[BUG]
When passing a btrfs with NO_HOLE feature to U-boot, and if one file
contains holes, then the hash of the file is not correct in U-boot:
# mkfs.btrfs -f test.img # Since v5.15, mkfs defaults to NO_HOLES
# mount test.img /mnt/btrfs
# xfs_io -f -c "pwrite 0 4k" -c "pwrite 8k 4k" /mnt/btrfs/file
# md5sum /mnt/btrfs/file
277f3840b275c74d01e979ea9d75ac19 /mnt/btrfs/file
# umount /mnt/btrfs
# ./u-boot
=> host bind 0 /home/adam/test.img
=> ls host 0
< > 12288 Mon Dec 27 05:35:23 2021 file
=> load host 0 0x1000000 file
12288 bytes read in 0 ms
=> md5sum 0x1000000 0x3000
md5 for 01000000 ... 01002fff ==> 855ffdbe4d0ccc5acab92e1b5330e4c1
The md5sum doesn't match at all.
[CAUSE]
In U-boot btrfs implementation, the function btrfs_read_file() has the
following iteration for file extent iteration:
/* Read the aligned part */
while (cur < aligned_end) {
ret = lookup_data_extent(root, &path, ino, cur, &next_offset);
if (ret < 0)
goto out;
if (ret > 0) {
/* No next, direct exit */
if (!next_offset) {
ret = 0;
goto out;
}
}
/* Read file extent */
But for NO_HOLES features, hole extents will not have any extent item
for it.
Thus if @cur is at a hole, lookup_data_extent() will just return >0, and
update @next_offset.
But we still believe there is some data to read for @cur for ret > 0
case, causing we read extent data from the next file extent.
This means, what we do for above NO_HOLES btrfs is:
- Read 4K data from disk to file offset [0, 4K)
So far the data is still correct
- Read 4K data from disk to file offset [4K, 8K)
We didn't skip the 4K hole, but read the data at file offset [8K, 12K)
into file offset [4K, 8K).
This causes the checksum mismatch.
[FIX]
Add extra check to skip to the next non-hole range after
lookup_data_extent().
Signed-off-by: Qu Wenruo <wqu@suse.com>
