Lines Matching refs:memory
10 (*) Abstract memory access model.
15 (*) What are memory barriers?
17 - Varieties of memory barrier.
18 - What may not be assumed about memory barriers?
22 - Examples of memory barrier sequences.
23 - Read memory barriers vs load speculation.
29 - CPU memory barriers.
32 (*) Implicit kernel memory barriers.
41 - Locks vs memory accesses.
44 (*) Where are memory barriers needed?
101 Each CPU executes a program that generates memory access operations. In the
102 abstract CPU, memory operation ordering is very relaxed, and a CPU may actually
103 perform the memory operations in any order it likes, provided program causality
108 So in the above diagram, the effects of the memory operations performed by a
121 The set of accesses as seen by the memory system in the middle can be arranged
142 Furthermore, the stores committed by a CPU to the memory system may not be
170 Some devices present their control interfaces as collections of memory
194 (*) On any given CPU, dependent memory accesses will be issued in order, with
199 the CPU will issue the following memory operations:
214 the CPU will only issue the following sequence of memory operations:
227 memory).
232 with memory references that are not protected by READ_ONCE() and
251 (*) It _must_ be assumed that overlapping memory accesses may be merged or
295 defines "memory location" as follows:
297 memory location
302 separate memory locations without interfering with
306 are in separate memory locations. The same applies
321 As can be seen above, independent memory operations are effectively performed
327 ordering over the memory operations on either side of the barrier.
331 deferral and combination of memory operations; speculative loads; speculative
342 (1) Write (or store) memory barriers.
344 A write memory barrier gives a guarantee that all the STORE operations
353 memory system as time progresses. All stores before a write barrier will
374 committing sequences of stores to the memory system that the CPU being
382 See the "Examples of memory barrier sequences" subsection for diagrams
396 (3) Read (or load) memory barriers.
406 Read memory barriers imply data dependency barriers, and so can substitute
413 (4) General memory barriers.
415 A general memory barrier gives a guarantee that all the LOAD and STORE
420 A general memory barrier is a partial ordering over both loads and stores.
422 General memory barriers imply both read and write memory barriers, and so
430 This acts as a one-way permeable barrier. It guarantees that all memory
446 memory operations before the RELEASE operation will appear to happen
455 for other sorts of memory barrier (but note the exceptions mentioned in
457 pair is -not- guaranteed to act as a full memory barrier. However, after
458 an ACQUIRE on a given variable, all memory accesses preceding any prior
471 memory barriers are unnecessary in that piece of code.
482 There are certain things that the Linux kernel memory barriers do not guarantee:
484 (*) There is no guarantee that any of the memory accesses specified before a
485 memory barrier will be _complete_ by the completion of a memory barrier
489 (*) There is no guarantee that issuing a memory barrier on one CPU will have
495 from a second CPU's accesses, even _if_ the second CPU uses a memory
496 barrier, unless the first CPU _also_ uses a matching memory barrier (see
500 hardware[*] will not reorder the memory accesses. CPU cache coherency
501 mechanisms should propagate the indirect effects of a memory barrier
569 number is read from memory and then used to calculate the index for an array
595 A load-load control dependency requires a full read memory barrier, not
674 memory barriers, for example, smp_store_release():
685 In contrast, without explicit memory barriers, two-legged-if control
828 When dealing with CPU-CPU interactions, certain types of memory barrier should
902 This sequence of events is committed to the memory coherence system in an order
924 | memory system by CPU 1
1162 item from memory, and they find a time where they're not using the bus for any
1212 speculated memory location, then the speculated value will just be used:
1281 In the Linux kernel, use of general memory barriers guarantees
1321 (*) CPU memory barriers.
1330 compiler from moving the memory accesses either side of it to the other side:
1477 (*) The compiler is within its rights to reorder memory accesses unless
1536 indicated memory locations, while with barrier() the compiler must
1537 discard the value of all memory locations that it has currented
1572 (*) For aligned memory locations whose size allows them to be accessed
1573 with a single memory-reference instruction, prevents "load tearing"
1632 The Linux kernel has eight basic CPU memory barriers:
1642 All memory barriers except the data dependency barriers imply a compiler
1655 SMP memory barriers are reduced to compiler barriers on uniprocessor compiled
1659 [!] Note that SMP memory barriers _must_ be used to control the ordering of
1660 references to shared memory on SMP systems, though the use of locking instead
1665 used to control MMIO effects on accesses through relaxed memory I/O windows.
1667 memory operations appear to a device by prohibiting both the compiler and the
1675 This assigns the value to the variable and then inserts a full memory
1685 reference counting. These functions do not imply memory barriers.
1718 These are for use with consistent memory to guarantee the ordering
1719 of writes or reads of shared memory accessible to both the CPU and a
1722 For example, consider a device driver that shares memory with a device
1741 /* force memory to sync before notifying device via MMIO */
1752 cache coherent memory writes have completed before attempting a write to
1755 See Documentation/DMA-API.txt for more information on consistent memory.
1760 The Linux kernel also has a special barrier for use with memory-mapped I/O
1776 Some of the other functions in the linux kernel imply memory barriers, amongst
1838 An ACQUIRE followed by a RELEASE may not be assumed to be full memory barrier
1856 RELEASE may -not- be assumed to be a full memory barrier.
1859 not imply a full memory barrier. Therefore, the CPU's execution of the
1894 a memory barrier, which will force the earlier unlock operation
1937 (RELEASE equivalent) will act as compiler barriers only. So if memory or I/O
1961 A general memory barrier is interpolated automatically by set_current_state()
1977 which therefore also imply a general memory barrier after setting the state.
1979 interpolate the memory barrier in the right place:
2001 A write memory barrier is implied by wake_up() and co. if and only if they wake
2013 To repeat, this write memory barrier is present if and only if something
2022 wake_up(); load from Y sees 1, no memory barrier
2047 [!] Note that the memory barriers implied by the sleeper and the waker do _not_
2066 code on both sides must interpolate its own memory barriers between the
2088 (*) schedule() and similar imply full memory barriers.
2096 that does affect memory access ordering on other CPUs, within the context of
2137 read memory barriers.
2202 Under normal operation, memory operation reordering is generally not going to
2296 The way to deal with this is to insert a general SMP memory barrier:
2305 In this case, the barrier makes a guarantee that all memory accesses before the
2306 barrier will appear to happen before all the memory accesses after the barrier
2308 the memory accesses before the barrier will be complete by the time the barrier
2321 operations are noted specially as some of them imply full memory barriers and
2325 Any atomic operation that modifies some state in memory and returns information
2326 about the state (old or new) implies an SMP-conditional general memory barrier
2351 such the implicit memory barrier effects are necessary.
2354 The following operations are potential problems as they do _not_ imply memory
2363 With these the appropriate explicit memory barrier should be used if necessary
2367 The following also do _not_ imply memory barriers, and so may require explicit
2368 memory barriers under some circumstances (smp_mb__before_atomic() for
2376 If they're used for statistics generation, then they probably don't need memory
2380 they probably don't need memory barriers because either the reference count
2382 sufficient references to make the lock, and thus a memory barrier unnecessary.
2385 do need memory barriers as a lock primitive generally has to do things in a
2388 Basically, each usage case has to be carefully considered as to whether memory
2401 [!] Note that special memory barrier primitives are available for these
2402 situations because on some CPUs the atomic instructions used imply full memory
2412 Many devices can be memory mapped, and so appear to the CPU as if they're just
2413 a set of memory locations. To control such a device, the driver usually has to
2414 make the right memory accesses in exactly the right order.
2425 use of memory barriers unnecessary, there are a couple of situations where they
2432 (2) If the accessor functions are used to refer to an I/O memory window with
2433 relaxed memory access properties, then _mandatory_ memory barriers are
2492 When accessing I/O memory, drivers should use the appropriate accessor
2497 These are intended to talk to I/O space rather than memory space, but
2505 memory map, particularly on those CPUs that don't support alternate I/O
2515 memory and I/O operation.
2521 defined for the memory window through which they're accessing. On later
2537 Used with prefetchable I/O memory, an mmiowb() barrier may be required to
2545 These are similar to readX() and writeX(), but provide weaker memory
2547 respect to normal memory accesses (e.g. DMA buffers) nor do they guarantee
2577 condition codes, changing registers or changing memory - and different
2594 The way cached memory operations are perceived across the system is affected to
2595 a certain extent by the caches that lie between CPUs and memory, and by the
2596 memory coherence system that maintains the consistency of state in the system.
2599 caches goes, the memory system has to include the CPU's caches, and memory
2601 (memory barriers logically act on the dotted line in the following diagram):
2627 it will still appear as if the full memory access had taken place as far as the
2633 generate load and store operations which then go into the queue of memory
2638 What memory barriers are concerned with is controlling the order in which
2639 accesses cross from the CPU side of things to the memory side of things, and
2647 the properties of the memory window through which devices are accessed and/or
2687 resident in memory;
2690 resident in memory;
2717 The write memory barrier forces the other CPUs in the system to perceive that
2777 Whilst most CPUs do imply a data dependency barrier on the read when a memory
2781 cachelets for normal memory accesses. The semantics of the Alpha removes the
2782 need for coordination in the absence of memory barriers.
2809 Memory mapped I/O usually takes place through memory locations that are part of
2810 a window in the CPU's memory space that has different properties assigned than
2815 may, in effect, overtake accesses to cached memory that were emitted earlier.
2816 A memory barrier isn't sufficient in such a case, but rather the cache must be
2817 flushed between the cached memory write and the MMIO access if the two are in
2825 A programmer might take it for granted that the CPU will perform memory
2835 they would then expect that the CPU will complete the memory operation for each
2855 (*) the order of the memory accesses may be rearranged to promote better use
2859 memory or I/O hardware that can do batched accesses of adjacent locations,
2860 thus cutting down on transaction setup costs (memory and PCI devices may
2877 _own_ accesses appear to be correctly ordered, without the need for a memory
2895 The code above may cause the CPU to generate the full sequence of memory
2928 may, without a memory barrier or an READ_ONCE() and WRITE_ONCE(), be
2944 caches with the memory coherence system, thus making it seem like pointer
2947 The Alpha defines the Linux kernel's memory barrier model.