Lines Matching refs:memory
10 (*) Abstract memory access model.
15 (*) What are memory barriers?
17 - Varieties of memory barrier.
18 - What may not be assumed about memory barriers?
22 - Examples of memory barrier sequences.
23 - Read memory barriers vs load speculation.
29 - CPU memory barriers.
32 (*) Implicit kernel memory barriers.
41 - Locks vs memory accesses.
44 (*) Where are memory barriers needed?
101 Each CPU executes a program that generates memory access operations. In the
102 abstract CPU, memory operation ordering is very relaxed, and a CPU may actually
103 perform the memory operations in any order it likes, provided program causality
108 So in the above diagram, the effects of the memory operations performed by a
121 The set of accesses as seen by the memory system in the middle can be arranged
142 Furthermore, the stores committed by a CPU to the memory system may not be
170 Some devices present their control interfaces as collections of memory
194 (*) On any given CPU, dependent memory accesses will be issued in order, with
199 the CPU will issue the following memory operations:
214 the CPU will only issue the following sequence of memory operations:
227 memory).
232 memory references that are not protected by ACCESS_ONCE(). Without
251 (*) It _must_ be assumed that overlapping memory accesses may be merged or
295 defines "memory location" as follows:
297 memory location
302 separate memory locations without interfering with
306 are in separate memory locations. The same applies
321 As can be seen above, independent memory operations are effectively performed
327 ordering over the memory operations on either side of the barrier.
331 deferral and combination of memory operations; speculative loads; speculative
342 (1) Write (or store) memory barriers.
344 A write memory barrier gives a guarantee that all the STORE operations
353 memory system as time progresses. All stores before a write barrier will
374 committing sequences of stores to the memory system that the CPU being
382 See the "Examples of memory barrier sequences" subsection for diagrams
396 (3) Read (or load) memory barriers.
406 Read memory barriers imply data dependency barriers, and so can substitute
413 (4) General memory barriers.
415 A general memory barrier gives a guarantee that all the LOAD and STORE
420 A general memory barrier is a partial ordering over both loads and stores.
422 General memory barriers imply both read and write memory barriers, and so
430 This acts as a one-way permeable barrier. It guarantees that all memory
446 memory operations before the RELEASE operation will appear to happen
455 for other sorts of memory barrier (but note the exceptions mentioned in
457 pair is -not- guaranteed to act as a full memory barrier. However, after
458 an ACQUIRE on a given variable, all memory accesses preceding any prior
471 memory barriers are unnecessary in that piece of code.
482 There are certain things that the Linux kernel memory barriers do not guarantee:
484 (*) There is no guarantee that any of the memory accesses specified before a
485 memory barrier will be _complete_ by the completion of a memory barrier
489 (*) There is no guarantee that issuing a memory barrier on one CPU will have
495 from a second CPU's accesses, even _if_ the second CPU uses a memory
496 barrier, unless the first CPU _also_ uses a matching memory barrier (see
500 hardware[*] will not reorder the memory accesses. CPU cache coherency
501 mechanisms should propagate the indirect effects of a memory barrier
569 number is read from memory and then used to calculate the index for an array
595 A load-load control dependency requires a full read memory barrier, not
674 memory barriers, for example, smp_store_release():
685 In contrast, without explicit memory barriers, two-legged-if control
826 When dealing with CPU-CPU interactions, certain types of memory barrier should
900 This sequence of events is committed to the memory coherence system in an order
922 | memory system by CPU 1
1160 item from memory, and they find a time where they're not using the bus for any
1210 speculated memory location, then the speculated value will just be used:
1279 In the Linux kernel, use of general memory barriers guarantees
1319 (*) CPU memory barriers.
1328 compiler from moving the memory accesses either side of it to the other side:
1474 (*) The compiler is within its rights to reorder memory accesses unless
1530 the contents of the indicated memory locations, while with barrier()
1531 the compiler must discard the value of all memory locations that
1566 (*) For aligned memory locations whose size allows them to be accessed
1567 with a single memory-reference instruction, prevents "load tearing"
1625 The Linux kernel has eight basic CPU memory barriers:
1635 All memory barriers except the data dependency barriers imply a compiler
1647 SMP memory barriers are reduced to compiler barriers on uniprocessor compiled
1651 [!] Note that SMP memory barriers _must_ be used to control the ordering of
1652 references to shared memory on SMP systems, though the use of locking instead
1657 used to control MMIO effects on accesses through relaxed memory I/O windows.
1659 memory operations appear to a device by prohibiting both the compiler and the
1667 This assigns the value to the variable and then inserts a full memory
1677 reference counting. These functions do not imply memory barriers.
1699 These are for use with consistent memory to guarantee the ordering
1700 of writes or reads of shared memory accessible to both the CPU and a
1703 For example, consider a device driver that shares memory with a device
1722 /* force memory to sync before notifying device via MMIO */
1733 cache coherent memory writes have completed before attempting a write to
1736 See Documentation/DMA-API.txt for more information on consistent memory.
1741 The Linux kernel also has a special barrier for use with memory-mapped I/O
1757 Some of the other functions in the linux kernel imply memory barriers, amongst
1821 An ACQUIRE followed by a RELEASE may not be assumed to be full memory barrier
1839 RELEASE may -not- be assumed to be a full memory barrier.
1842 imply a full memory barrier. If it is necessary for a RELEASE-ACQUIRE
1882 a memory barrier, which will force the earlier unlock operation
1948 (RELEASE equivalent) will act as compiler barriers only. So if memory or I/O
1972 A general memory barrier is interpolated automatically by set_current_state()
1988 which therefore also imply a general memory barrier after setting the state.
1990 interpolate the memory barrier in the right place:
2012 A write memory barrier is implied by wake_up() and co. if and only if they wake
2024 To repeat, this write memory barrier is present if and only if something
2033 wake_up(); load from Y sees 1, no memory barrier
2058 [!] Note that the memory barriers implied by the sleeper and the waker do _not_
2077 code on both sides must interpolate its own memory barriers between the
2099 (*) schedule() and similar imply full memory barriers.
2107 that does affect memory access ordering on other CPUs, within the context of
2182 read memory barriers.
2247 Under normal operation, memory operation reordering is generally not going to
2341 The way to deal with this is to insert a general SMP memory barrier:
2350 In this case, the barrier makes a guarantee that all memory accesses before the
2351 barrier will appear to happen before all the memory accesses after the barrier
2353 the memory accesses before the barrier will be complete by the time the barrier
2366 operations are noted specially as some of them imply full memory barriers and
2370 Any atomic operation that modifies some state in memory and returns information
2371 about the state (old or new) implies an SMP-conditional general memory barrier
2396 such the implicit memory barrier effects are necessary.
2399 The following operations are potential problems as they do _not_ imply memory
2408 With these the appropriate explicit memory barrier should be used if necessary
2412 The following also do _not_ imply memory barriers, and so may require explicit
2413 memory barriers under some circumstances (smp_mb__before_atomic() for
2421 If they're used for statistics generation, then they probably don't need memory
2425 they probably don't need memory barriers because either the reference count
2427 sufficient references to make the lock, and thus a memory barrier unnecessary.
2430 do need memory barriers as a lock primitive generally has to do things in a
2433 Basically, each usage case has to be carefully considered as to whether memory
2446 [!] Note that special memory barrier primitives are available for these
2447 situations because on some CPUs the atomic instructions used imply full memory
2457 Many devices can be memory mapped, and so appear to the CPU as if they're just
2458 a set of memory locations. To control such a device, the driver usually has to
2459 make the right memory accesses in exactly the right order.
2470 use of memory barriers unnecessary, there are a couple of situations where they
2477 (2) If the accessor functions are used to refer to an I/O memory window with
2478 relaxed memory access properties, then _mandatory_ memory barriers are
2537 When accessing I/O memory, drivers should use the appropriate accessor
2542 These are intended to talk to I/O space rather than memory space, but
2550 memory map, particularly on those CPUs that don't support alternate I/O
2560 memory and I/O operation.
2566 defined for the memory window through which they're accessing. On later
2582 Used with prefetchable I/O memory, an mmiowb() barrier may be required to
2590 These are similar to readX() and writeX(), but provide weaker memory
2592 respect to normal memory accesses (e.g. DMA buffers) nor do they guarantee
2622 condition codes, changing registers or changing memory - and different
2639 The way cached memory operations are perceived across the system is affected to
2640 a certain extent by the caches that lie between CPUs and memory, and by the
2641 memory coherence system that maintains the consistency of state in the system.
2644 caches goes, the memory system has to include the CPU's caches, and memory
2646 (memory barriers logically act on the dotted line in the following diagram):
2672 it will still appear as if the full memory access had taken place as far as the
2678 generate load and store operations which then go into the queue of memory
2683 What memory barriers are concerned with is controlling the order in which
2684 accesses cross from the CPU side of things to the memory side of things, and
2692 the properties of the memory window through which devices are accessed and/or
2732 resident in memory;
2735 resident in memory;
2762 The write memory barrier forces the other CPUs in the system to perceive that
2822 Whilst most CPUs do imply a data dependency barrier on the read when a memory
2826 cachelets for normal memory accesses. The semantics of the Alpha removes the
2827 need for coordination in the absence of memory barriers.
2854 Memory mapped I/O usually takes place through memory locations that are part of
2855 a window in the CPU's memory space that has different properties assigned than
2860 may, in effect, overtake accesses to cached memory that were emitted earlier.
2861 A memory barrier isn't sufficient in such a case, but rather the cache must be
2862 flushed between the cached memory write and the MMIO access if the two are in
2870 A programmer might take it for granted that the CPU will perform memory
2880 they would then expect that the CPU will complete the memory operation for each
2900 (*) the order of the memory accesses may be rearranged to promote better use
2904 memory or I/O hardware that can do batched accesses of adjacent locations,
2905 thus cutting down on transaction setup costs (memory and PCI devices may
2922 _own_ accesses appear to be correctly ordered, without the need for a memory
2940 The code above may cause the CPU to generate the full sequence of memory
2972 may, without a memory barrier or an ACCESS_ONCE(), be reduced to:
2987 caches with the memory coherence system, thus making it seem like pointer
2990 The Alpha defines the Linux kernel's memory barrier model.