arm64: mm: Don't wait for completion of TLB invalidation when page aging
authorAlex Van Brunt <avanbrunt@nvidia.com>
Mon, 29 Oct 2018 09:25:58 +0000 (14:55 +0530)
committerWill Deacon <will.deacon@arm.com>
Mon, 26 Nov 2018 16:59:46 +0000 (16:59 +0000)
commit3403e56b41c176f6531a2a6d77d85b46fa34169c
treed1250d81b892500f84ce53775ec44879f7be6c5c
parentc8ebf64eab743130fe404dc6679c2ff0cbc01615
arm64: mm: Don't wait for completion of TLB invalidation when page aging

When transitioning a PTE from young to old as part of page aging, we
can avoid waiting for the TLB invalidation to complete and therefore
drop the subsequent DSB instruction. Whilst this opens up a race with
page reclaim, where a PTE in active use via a stale, young TLB entry
does not update the underlying descriptor, the worst thing that happens
is that the page is reclaimed and then immediately faulted back in.

Given that we have a DSB in our context-switch path, the window for a
spurious reclaim is fairly limited and eliding the barrier claims to
boost NVMe/SSD accesses by over 10% on some platforms.

A similar optimisation was made for x86 in commit b13b1d2d8692 ("x86/mm:
In the PTE swapout page reclaim case clear the accessed bit instead of
flushing the TLB").

Signed-off-by: Alex Van Brunt <avanbrunt@nvidia.com>
Signed-off-by: Ashish Mhetre <amhetre@nvidia.com>
[will: rewrote patch]
Signed-off-by: Will Deacon <will.deacon@arm.com>
arch/arm64/include/asm/pgtable.h
arch/arm64/include/asm/tlbflush.h