1Started Nov 1999 by Kanoj Sarcar <kanoj@sgi.com>
2
3What is NUMA?
4
5This question can be answered from a couple of perspectives:  the
6hardware view and the Linux software view.
7
8From the hardware perspective, a NUMA system is a computer platform that
9comprises multiple components or assemblies each of which may contain 0
10or more CPUs, local memory, and/or IO buses.  For brevity and to
11disambiguate the hardware view of these physical components/assemblies
12from the software abstraction thereof, we'll call the components/assemblies
13'cells' in this document.
14
15Each of the 'cells' may be viewed as an SMP [symmetric multi-processor] subset
16of the system--although some components necessary for a stand-alone SMP system
17may not be populated on any given cell.   The cells of the NUMA system are
18connected together with some sort of system interconnect--e.g., a crossbar or
19point-to-point link are common types of NUMA system interconnects.  Both of
20these types of interconnects can be aggregated to create NUMA platforms with
21cells at multiple distances from other cells.
22
23For Linux, the NUMA platforms of interest are primarily what is known as Cache
24Coherent NUMA or ccNUMA systems.   With ccNUMA systems, all memory is visible
25to and accessible from any CPU attached to any cell and cache coherency
26is handled in hardware by the processor caches and/or the system interconnect.
27
28Memory access time and effective memory bandwidth varies depending on how far
29away the cell containing the CPU or IO bus making the memory access is from the
30cell containing the target memory.  For example, access to memory by CPUs
31attached to the same cell will experience faster access times and higher
32bandwidths than accesses to memory on other, remote cells.  NUMA platforms
33can have cells at multiple remote distances from any given cell.
34
35Platform vendors don't build NUMA systems just to make software developers'
36lives interesting.  Rather, this architecture is a means to provide scalable
37memory bandwidth.  However, to achieve scalable memory bandwidth, system and
38application software must arrange for a large majority of the memory references
39[cache misses] to be to "local" memory--memory on the same cell, if any--or
40to the closest cell with memory.
41
42This leads to the Linux software view of a NUMA system:
43
44Linux divides the system's hardware resources into multiple software
45abstractions called "nodes".  Linux maps the nodes onto the physical cells
46of the hardware platform, abstracting away some of the details for some
47architectures.  As with physical cells, software nodes may contain 0 or more
48CPUs, memory and/or IO buses.  And, again, memory accesses to memory on
49"closer" nodes--nodes that map to closer cells--will generally experience
50faster access times and higher effective bandwidth than accesses to more
51remote cells.
52
53For some architectures, such as x86, Linux will "hide" any node representing a
54physical cell that has no memory attached, and reassign any CPUs attached to
55that cell to a node representing a cell that does have memory.  Thus, on
56these architectures, one cannot assume that all CPUs that Linux associates with
57a given node will see the same local memory access times and bandwidth.
58
59In addition, for some architectures, again x86 is an example, Linux supports
60the emulation of additional nodes.  For NUMA emulation, linux will carve up
61the existing nodes--or the system memory for non-NUMA platforms--into multiple
62nodes.  Each emulated node will manage a fraction of the underlying cells'
63physical memory.  NUMA emluation is useful for testing NUMA kernel and
64application features on non-NUMA platforms, and as a sort of memory resource
65management mechanism when used together with cpusets.
66[see Documentation/cgroups/cpusets.txt]
67
68For each node with memory, Linux constructs an independent memory management
69subsystem, complete with its own free page lists, in-use page lists, usage
70statistics and locks to mediate access.  In addition, Linux constructs for
71each memory zone [one or more of DMA, DMA32, NORMAL, HIGH_MEMORY, MOVABLE],
72an ordered "zonelist".  A zonelist specifies the zones/nodes to visit when a
73selected zone/node cannot satisfy the allocation request.  This situation,
74when a zone has no available memory to satisfy a request, is called
75"overflow" or "fallback".
76
77Because some nodes contain multiple zones containing different types of
78memory, Linux must decide whether to order the zonelists such that allocations
79fall back to the same zone type on a different node, or to a different zone
80type on the same node.  This is an important consideration because some zones,
81such as DMA or DMA32, represent relatively scarce resources.  Linux chooses
82a default zonelist order based on the sizes of the various zone types relative
83to the total memory of the node and the total memory of the system.  The
84default zonelist order may be overridden using the numa_zonelist_order kernel
85boot parameter or sysctl.  [see Documentation/kernel-parameters.txt and
86Documentation/sysctl/vm.txt]
87
88By default, Linux will attempt to satisfy memory allocation requests from the
89node to which the CPU that executes the request is assigned.  Specifically,
90Linux will attempt to allocate from the first node in the appropriate zonelist
91for the node where the request originates.  This is called "local allocation."
92If the "local" node cannot satisfy the request, the kernel will examine other
93nodes' zones in the selected zonelist looking for the first zone in the list
94that can satisfy the request.
95
96Local allocation will tend to keep subsequent access to the allocated memory
97"local" to the underlying physical resources and off the system interconnect--
98as long as the task on whose behalf the kernel allocated some memory does not
99later migrate away from that memory.  The Linux scheduler is aware of the
100NUMA topology of the platform--embodied in the "scheduling domains" data
101structures [see Documentation/scheduler/sched-domains.txt]--and the scheduler
102attempts to minimize task migration to distant scheduling domains.  However,
103the scheduler does not take a task's NUMA footprint into account directly.
104Thus, under sufficient imbalance, tasks can migrate between nodes, remote
105from their initial node and kernel data structures.
106
107System administrators and application designers can restrict a task's migration
108to improve NUMA locality using various CPU affinity command line interfaces,
109such as taskset(1) and numactl(1), and program interfaces such as
110sched_setaffinity(2).  Further, one can modify the kernel's default local
111allocation behavior using Linux NUMA memory policy.
112[see Documentation/vm/numa_memory_policy.txt.]
113
114System administrators can restrict the CPUs and nodes' memories that a non-
115privileged user can specify in the scheduling or NUMA commands and functions
116using control groups and CPUsets.  [see Documentation/cgroups/cpusets.txt]
117
118On architectures that do not hide memoryless nodes, Linux will include only
119zones [nodes] with memory in the zonelists.  This means that for a memoryless
120node the "local memory node"--the node of the first zone in CPU's node's
121zonelist--will not be the node itself.  Rather, it will be the node that the
122kernel selected as the nearest node with memory when it built the zonelists.
123So, default, local allocations will succeed with the kernel supplying the
124closest available memory.  This is a consequence of the same mechanism that
125allows such allocations to fallback to other nearby nodes when a node that
126does contain memory overflows.
127
128Some kernel allocations do not want or cannot tolerate this allocation fallback
129behavior.  Rather they want to be sure they get memory from the specified node
130or get notified that the node has no free memory.  This is usually the case when
131a subsystem allocates per CPU memory resources, for example.
132
133A typical model for making such an allocation is to obtain the node id of the
134node to which the "current CPU" is attached using one of the kernel's
135numa_node_id() or CPU_to_node() functions and then request memory from only
136the node id returned.  When such an allocation fails, the requesting subsystem
137may revert to its own fallback path.  The slab kernel memory allocator is an
138example of this.  Or, the subsystem may choose to disable or not to enable
139itself on allocation failure.  The kernel profiling subsystem is an example of
140this.
141
142If the architecture supports--does not hide--memoryless nodes, then CPUs
143attached to memoryless nodes would always incur the fallback path overhead
144or some subsystems would fail to initialize if they attempted to allocated
145memory exclusively from a node without memory.  To support such
146architectures transparently, kernel subsystems can use the numa_mem_id()
147or cpu_to_mem() function to locate the "local memory node" for the calling or
148specified CPU.  Again, this is the same node from which default, local page
149allocations will be attempted.
150