Some recent arm64 cores have a facility that allows the page
table walker to track the dirty state of a page. This makes it
really efficient to perform CMOs by VA as we only need to look
at dirty pages.
Signed-off-by: Marc Zyngier <maz@kernel.org>
[ Paul: pick from the Android tree. Rebase to the upstream ]
Signed-off-by: Ying-Chun Liu (PaulLiu) <paul.liu@linaro.org>
Cc: Tom Rini <trini@konsulko.com>
Link: 3c433724e6
count_required_pts()'s complexity is high if mappings are not using the
largest possible block size (due to some other requirement such as tracking
dirty pages, for example).
Let's switch to a method that follows the pattern established with
the add_map() helper, and make it almost instantaneous instead of
taking a large amount of time if 2MB mappings are in use instead of
1GB.
Signed-off-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Pierre-Clément Tosi <ptosi@google.com>
[ Paul: pick from the Android tree. Fixup Pierre's commit. Rebase to the
upstream ]
Signed-off-by: Ying-Chun Liu (PaulLiu) <paul.liu@linaro.org>
Cc: Tom Rini <trini@konsulko.com>
Link: 5d756d147e
Link: 6be9330601
In the add_map() function, for each level it populates, it iterates from
the root of the PT tree, making it ineficient if a mapping needs to occur
past level 1.
Instead, replace it with a recursive (and much simpler) algorithm
that keeps the complexity as low as possible. With this, mapping
512GB at level 2 goes from several seconds down to not measurable
on an A55 machine.
We keep the block mappings at level 1 for now though.
Signed-off-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Pierre-Clément Tosi <ptosi@google.com>
[ Paul: pick from the Android tree. Fixup Pierre's commit. Rebase to the
upstream ]
Signed-off-by: Ying-Chun Liu (PaulLiu) <paul.liu@linaro.org>
Cc: Tom Rini <trini@konsulko.com>
Link: 96ad729cf4
Link: 6be9330601
Exposing set/way cache maintenance to a virtual machine is unsafe, not
least because the instructions are not permission-checked but also
because they are not broadcast between CPUs. Consequently, KVM traps and
emulates such maintenance in the host kernel using by-VA operations and
looping over the stage-2 page-tables. However, when running under
protected KVM, these instructions are not able to be emulated and will
instead result in an exception being delivered to the guest.
Introduce CONFIG_CMO_BY_VA_ONLY so that virtual platforms can select
this option and perform by-VA cache maintenance instead of using the
set/way instructions.
Signed-off-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Will Deacon <willdeacon@google.com>
Signed-off-by: Pierre-Clément Tosi <ptosi@google.com>
[ Paul: pick from the Android tree. Fixup Pierre's commit. And fix some
checkpatch warnings. Rebased to upstream. ]
Signed-off-by: Ying-Chun Liu (PaulLiu) <paul.liu@linaro.org>
Cc: Tom Rini <trini@konsulko.com>
Link: db5507f47f
Link: 2baf54e743
When MMU is already enabled then dcache_enable() does not call mmu_setup()
and so setup_all_pgtables() is also never called.
In this situation when some driver calls mmu_set_region_dcache_behaviour()
function then U-Boot crashes with error message:
Emergency page table not setup.
Fix this issue by explicitly calling setup_all_pgtables() in dcache_enable()
function near condition for mmu_setup().
This change fixes chainloading U-Boot from U-Boot on Turris Mox board which
uses mvneta ethernet driver which calls mmu_set_region_dcache_behaviour().
Signed-off-by: Pali Rohár <pali@kernel.org>
Reviewed-by: Marek Behún <kabel@kernel.org>
Reviewed-by: Stefan Roese <sr@denx.de>
Currently get_tcr() takes an "el" parameter, to select the proper
version of the TCR_ELx system register.
This is problematic in case of the Apple M1, since it runs with
HCR_EL2.E2H fixed to 1, so TCR_EL2 is actually using the TCR_EL1 layout,
and we get the wrong version.
For U-Boot's purposes the only sensible choice here is the current
exception level, and indeed most callers treat it like that, so let's
remove that parameter and read the current EL inside the function.
This allows us to check for the E2H bit, and pretend it's EL1 in this
case.
There are two callers which don't care about the EL, and they pass 0,
which looks wrong, but is irrelevant in these two cases, since we don't
use the return value there. So the change cannot affect those two.
Signed-off-by: Andre Przywara <andre.przywara@arm.com>
Reviewed-by: Mark Kettenis <kettenis@openbsd.org>
Tested-by: Mark Kettenis <kettenis@openbsd.org>
At early U-Boot stage, before relocation, MMU is not yet configured
and disabled. DDR may not be configured with the correct memory
attributes (can be configured in MT_DEVICE instead of MT_MEMORY).
In this case, usage of memcpy_{from, to}io() may leads to synchronous
abort in AARCH64 in case the normal memory address is not 64Bits aligned.
To avoid such situation, forbid usage of normal memory cast to (u64 *) in
case MMU is not enabled.
Signed-off-by: Patrice Chotard <patrice.chotard@foss.st.com>
Reviewed-by: Patrick Delaunay <patrick.delaunay@foss.st.com>
Cc: mark.kettenis@xs4all.nl
Signed-off-by: Patrick Delaunay <patrick.delaunay@foss.st.com>
Move this out of the common header and include it only where needed. In
a number of cases this requires adding "struct udevice;" to avoid adding
another large header or in other cases replacing / adding missing header
files that had been pulled in, very indirectly. Finally, we have a few
cases where we did not need to include <asm/global_data.h> at all, so
remove that include.
Signed-off-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
The enum dcache_optoion contains a shift left 2 bits in the armv8 case
already. The PMD_ATTRINDX(option) macro will perform a left shift of 2
bits. Perform a right shift so that in the end we get the correct
value.
[trini: Reword the commit message]
Reviewed-by: Ye Li <ye.li@nxp.com>
Signed-off-by: Peng Fan <peng.fan@nxp.com>
Move this header out of the common header. Network support is used in
quite a few places but it still does not warrant blanket inclusion.
Note that this net.h header itself has quite a lot in it. It could be
split into the driver-mode support, functions, structures, checksumming,
etc.
Signed-off-by: Simon Glass <sjg@chromium.org>
At present panic() is in the vsprintf.h header file. That does not seem
like an obvious choice for hang(), even though it relates to panic(). So
let's put hang() in its own header.
Signed-off-by: Simon Glass <sjg@chromium.org>
[trini: Migrate a few more files]
Signed-off-by: Tom Rini <trini@konsulko.com>
These functions belong in cpu_func.h. Another option would be cache.h
but that code uses driver model and we have not moved these cache
functions to use driver model. Since they are CPU-related it seems
reasonable to put them here.
Move them over.
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>
While converting CONFIG_SYS_[DI]CACHE_OFF to Kconfig, there are instances
where these configuration items are conditional on SPL. This commit adds SPL
variants of these configuration items, uses CONFIG_IS_ENABLED(), and updates
the configurations as required.
Acked-by: Alexey Brodkin <abrodkin@synopsys.com>
Signed-off-by: Trevor Woerner <trevor@toganlabs.com>
[trini: Make the default depend on the setting for full U-Boot, update
more zynq hardware]
Signed-off-by: Tom Rini <trini@konsulko.com>
On AM654 SoC(arm64) which is IO coherent and has L3 Cache, cache
maintenance operations being done to support non-coherent platforms
causes issues.
For example, here is how U-Boot prepares/handles a buffer to receive
data from a device (DMA Write). This may vary slightly depending on the
driver framework:
Start DMA to write to destination buffer
Wait for DMA to be done (dma_receive()/dma_memcpy())
Invalidate destination buffer (invalidate_dcache_range())
Read from destination buffer
The invalidate after the DMA is needed in order to read latest data from
memory that’s updated by DMA write. Also, in case random prefetch has
pulled in buffer data during the “wait for DMA” before the DMA has
written to it. This works well for non-coherent architectures.
In case of coherent architecture with L3 cache, DMA write would directly
update L3 cache contents (assuming cacheline is present in L3) without
updating the DDR memory. So invalidate after “wait for DMA” in above
sequence would discard latest data and read will cause stale data to be
fetched from DDR. Therefore invalidate after “wait for DMA” is not
always correct on coherent architecture.
Therefore, provide a Kconfig option to disable cache maintenance ops on
coherent architectures. This has added benefit of improving the
performance of DMA transfers as we no longer need to invalidate/flush
individual cache lines(especially for buffer thats several KBs in size).
In order to facilitate use of same Kconfig across different
architecture, I have added the symbol to top level arch/Kconfig file.
Patch currently disables cache maintenance ops for arm64 only.
flush_dcache_all() and invalidate_dcache_all() are exclusively used
during enabling/disabling dcache and hence are not disabled.
Signed-off-by: Vignesh Raghavendra <vigneshr@ti.com>
When U-Boot started using SPDX tags we were among the early adopters and
there weren't a lot of other examples to borrow from. So we picked the
area of the file that usually had a full license text and replaced it
with an appropriate SPDX-License-Identifier: entry. Since then, the
Linux Kernel has adopted SPDX tags and they place it as the very first
line in a file (except where shebangs are used, then it's second line)
and with slightly different comment styles than us.
In part due to community overlap, in part due to better tag visibility
and in part for other minor reasons, switch over to that style.
This commit changes all instances where we have a single declared
license in the tag as both the before and after are identical in tag
contents. There's also a few places where I found we did not have a tag
and have introduced one.
Signed-off-by: Tom Rini <trini@konsulko.com>
To page mapping the lowest 2 bits needs to be 0x3.
If not fix this, the final lowest 3 bits for page mapping is 0x1
which is marked as reserved.
Signed-off-by: Peng Fan <peng.fan@nxp.com>
Reviewed-by: York Sun <york.sun@nxp.com>
At present there is not operation to invalidate a cache range. This seems
to be needed to fill out the cache operations. Add an implementation based
on the flush operation.
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: York Sun <york.sun@nxp.com>
Function mmu_change_region_attr() is added to change existing mapping
with updated PXN, UXN and memory type. This is a break-before-make
process during which the mapping becomes fault (invalid) before final
attributres are set.
Signed-off-by: York Sun <york.sun@nxp.com>
SoC-specific logic may be required for all forms of cache-wide
operations; invalidate and flush of both dcache and icache (note that
only 3 of the 4 possible combinations make sense, since the icache never
contains dirty lines). This patch adds an optional hook for all
implemented cache-wide operations, and renames the one existing hook to
better represent exactly which operation it is implementing. A dummy
no-op implementation of each hook is provided.
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Warren <twarren@nvidia.com>
We create 2 sets of page tables: One for normal operation, one for
emergency (used while modifying the former).
Because the page tables grow dynamically, we have code that checks
for overflow. Unfortunately we didn't adjust the available space
variable while creating the emergency tables, so potentially someone
might run into an overflow there (not seen in real world yet though!).
Fix it by properly adjusting the size as well as the base offset in
emergency page table creation.
Reported-by: York Sun <york.sun@nxp.com>
Signed-off-by: Alexander Graf <agraf@suse.de>
Reviewed-by: York Sun <york.sun@nxp.com>
Introduce virtual and physical addresses in the mapping table. This change
have no impact on existing boards because they all use idential mapping.
Signed-off-by: York Sun <york.sun@nxp.com>
When page tables are created, allow later table to be created on
previous block entry. Splitting block feature is already working
with current code. This patch only rearranges the code order and
adds one condition to call split_block().
Signed-off-by: York Sun <york.sun@nxp.com>
Make setup_pgtages() and get_tcr() available for platform code to
customize MMU tables.
Remove unintentional call of create_table().
Signed-off-by: York Sun <york.sun@nxp.com>
With commit 7985cdf we converted all systems except for the Layerscape
SoCs to the generic descriptor table based page table setup.
On the Layerscape SoCs however, we just provide an empty table stub
and do the setup ourselves. To reserve enough memory for the tables,
we need to override the default counting mechanism which would end up
with an empty table because we have no maps.
Fixes: 7985cdf
Reported-by: York Sun <york.sun@nxp.com>
CC: Alison Wang <alison.wang@nxp.com>
CC: Prabhakar Kushwaha <prabhakar.kushwaha@nxp.com>
Signed-off-by: Alexander Graf <agraf@suse.de>
Tested-by: York Sun <york.sun@nxp.com>
Reviewed-by: York Sun <york.sun@nxp.com>
Now that we have an easy way to describe memory regions and enable the MMU,
there really shouldn't be anything holding people back from running with
caches enabled on AArch64. To make sure people catch early if they're missing
on the caching fun, give them a compile error.
Signed-off-by: Alexander Graf <agraf@suse.de>
By now the code to only have a single page table level with 64k page
size and 42 bit address space is no longer used by any board in tree,
so we can safely remove it.
To clean up code, move the layerscape mmu code to the new defines,
removing redundant field definitions.
Signed-off-by: Alexander Graf <agraf@suse.de>
The MMU range table can vary depending on things we may only find
out at runtime. While the very simple ThunderX variant does not
change, other boards will, so move the definition from a static
entry in a header file to the board file.
Signed-off-by: Alexander Graf <agraf@suse.de>
The idea to generate our pages tables from an array of memory ranges
is very sound. However, instead of hard coding the code to create up
to 2 levels of 64k granule page tables, we really should just create
normal 4k page tables that allow us to set caching attributes on 2M
or 4k level later on.
So this patch moves the full_va mapping code to 4k page size and
makes it fully flexible to dynamically create as many levels as
necessary for a map (including dynamic 1G/2M pages). It also adds
support to dynamically split a large map into smaller ones when
some code wants to set dcache attributes.
With all this in place, there is very little reason to create your
own page tables in board specific files.
Signed-off-by: Alexander Graf <agraf@suse.de>
When running in EL1, AArch64 knows two page table maps. One with addresses
that start with all zeros (TTBR0) and one with addresses that start with all
ones (TTBR1).
In U-Boot we don't care about the high up maps, so just disable them to ensure
we don't walk an invalid page table by accident.
Reported-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Alexander Graf <agraf@suse.de>
Based on the memory map we can determine a lot of hard coded fields of
TCR, like the maximum VA and max PA we want to support. Calculate those
dynamically to reduce the chance for pit falls.
Signed-off-by: Alexander Graf <agraf@suse.de>
This patch adds code which sets up 2-level page tables on ARM64 thus
extending available VA space. CPUs implementing 64k translation
granule are able to use direct PA-VA mapping of the whole 48 bit
address space.
It also adds the ability to reset the SCTRL register at the very beginning
of execution to avoid interference from stale mappings set up by early
firmware/loaders/etc.
Signed-off-by: Sergey Temerkhanov <s.temerkhanov@gmail.com>
Signed-off-by: Radha Mohan Chintakuntla <rchintakuntla@cavium.com>
For most device addresses excution shouldn't be allowed. Revise
the MMU table to enforce execute-never bits. OCRAM, DDR and IFC
are allowed for excution.
Signed-off-by: York Sun <yorksun@freescale.com>
Signed-off-by: Alison Wang <alison.wang@freescale.com>
Reported-by: Zhichun Hua <zhichun.hua@freescale.com>
In order for noncached_init() to operate correctly, SoCs must set up a
custom page table with fine-grained (2MiB) sections, which can be
configured from noncached_init().
This is currently performed by arch/arm/cpu/armv8/{fsl-lsch3,zynqmp}/cpu.c
by cut/pasting and re-implementing mmu_setup, enable_caches(), etc. There
are some other reasons for the duplication there though, such as enabling
icache early, and enabling dcaching earlier with a different configuration.
This change makes mmu_setup() a weak implementation, so that the MMU setup
code can be replaced without having to duplicate other code that calls it.
Signed-off-by: Stephen Warren <swarren@nvidia.com>
For EL3 and EL2, the documentation says that bits 31 and 23 are reserved
but should be written as 1.
For EL1, only bit 23 is not reserved, so only write bit 31 as 1.
Cc: Albert Aribaud <albert.u.boot@aribaud.net>
Cc: Marc Zyngier <marc.zyngier@arm.com>
Signed-off-by: Thierry Reding <treding@nvidia.com>
This patch rewrites MMU translation table entries. To start, all table
entries are written as "invalid", then "device-ngnrnr" and "normal" are
written to the entries to enable access to specific addresses.
Signed-off-by: Alison Wang <alison.wang@freescale.com>
Signed-off-by: York Sun <yorksun@freescale.com>
Current many cpu use the same flush_cache() function, which just call
the flush_dcache_range().
So implement a weak flush_cache() for all the cpus to use.
In original weak flush_cache() in arch/arm/lib/cache.c, there has some
code for ARM1136 & ARM926ejs. But in the arch/arm/cpu/arm1136/cpu.c and
arch/arm/cpu/arm926ejs/cache.c, there implements a real flush_cache()
function as well. That means the original code for ARM1136 & ARM926ejs
in weak flush_cache() of arch/arm/lib/cache.c is totally useless.
So in this patch remove such code in flush_cache() and only call
flush_dcache_range().
Signed-off-by: Josh Wu <josh.wu@atmel.com>
Since some driver like ohci, lcd used dcache functions. But some ARM
cpu don't implement the invalidate_dcache_range()/flush_dcache_range()
functions.
To avoid compiling errors this patch adds an weak empty stub function
for all ARM cpu in arch/arm/lib/cache.c.
And ARM cpu still can implemnt its own cache functions on the cpu folder.
Signed-off-by: Josh Wu <josh.wu@atmel.com>
Reviewed-by: York Sun <yorksun@freescale.com>
Added routine mmu_set_region_dcache_behaviour() to set a
particular region as non cacheable.
Define dummy routine for mmu_set_region_dcache_behaviour()
to handle incase of dcache off.
Signed-off-by: Siva Durga Prasad Paladugu <sivadur@xilinx.com>
Acked-by: Michal Simek <michal.simek@xilinx.com>
While generating the page tables, a running integer index is shifted by
SECTION_SHIFT (29) and causes overflow for any integer bigger than 7.
The page tables therefore alias to the same 8 sections and cause U-Boot
to hang once the MMU is enabled.
Fix this by making the index a 64-bit unsigned integer and so avoid the
overflow.
swarren notes: currently "i" ranges from 0..8191 on all ARM64 boards, and
"j" varies depending on RAM size; from 4 to 11 for a board with 4GB at
physical address 2GB, as some Tegra boards have.
Signed-off-by: Thierry Reding <treding@nvidia.com>
Signed-off-by: Tom Warren <twarren@nvidia.com>
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Flushing L3 cache in CCN-504 requries d-cache to be disabled. Using
assembly function to guarantee stack is not used before flushing is
completed. Timeout is needed for simualtor on which CCN-504 is not
implemented. Return value can be checked for timeout situation.
Change bootm.c to disable dcache instead of simply flushing, required
by flushing L3.
Signed-off-by: York Sun <yorksun@freescale.com>
Freescale LayerScape with Chassis Generation 3 is a set of SoCs with
ARMv8 cores and 3rd generation of Chassis. We use different MMU setup
to support memory map and cache attribute for these SoCs. MMU and cache
are enabled very early to bootst performance, especially for early
development on emulators. After u-boot relocates to DDR, a new MMU
table with QBMan cache access is created in DDR. SMMU pagesize is set
in SMMU_sACR register. Both DDR3 and DDR4 are supported.
Signed-off-by: York Sun <yorksun@freescale.com>
Signed-off-by: Varun Sethi <Varun.Sethi@freescale.com>
Signed-off-by: Arnab Basu <arnab.basu@freescale.com>
Make MMU function reusable. Platform code can setup its own MMU tables.
Signed-off-by: York Sun <yorksun@freescale.com>
CC: David Feng <fenghua@phytium.com.cn>
When SoC first boots up, we should invalidate the cache but not flush it.
We can use the same function for invalid and flush mostly, with a wrapper.
Invalidating large cache can ben slow on emulator, so we postpone doing
so until I-cache is enabled, and before enabling D-cache.
Signed-off-by: York Sun <yorksun@freescale.com>
CC: David Feng <fenghua@phytium.com.cn>
Move setting for MAIR and TCR to cache_v8.c, to avoid conflict with
sub-architecture.
Signed-off-by: York Sun <yorksun@freescale.com>
CC: David Feng <fenghua@phytium.com.cn>
Relocation code based on a patch by Scott Wood, which is:
Signed-off-by: Scott Wood <scottwood@freescale.com>
Signed-off-by: David Feng <fenghua@phytium.com.cn>
