1Using numa=fake and CPUSets for Resource Management 2Written by David Rientjes <rientjes@cs.washington.edu> 3 4This document describes how the numa=fake x86_64 command-line option can be used 5in conjunction with cpusets for coarse memory management. Using this feature, 6you can create fake NUMA nodes that represent contiguous chunks of memory and 7assign them to cpusets and their attached tasks. This is a way of limiting the 8amount of system memory that are available to a certain class of tasks. 9 10For more information on the features of cpusets, see 11Documentation/cgroups/cpusets.txt. 12There are a number of different configurations you can use for your needs. For 13more information on the numa=fake command line option and its various ways of 14configuring fake nodes, see Documentation/x86/x86_64/boot-options.txt. 15 16For the purposes of this introduction, we'll assume a very primitive NUMA 17emulation setup of "numa=fake=4*512,". This will split our system memory into 18four equal chunks of 512M each that we can now use to assign to cpusets. As 19you become more familiar with using this combination for resource control, 20you'll determine a better setup to minimize the number of nodes you have to deal 21with. 22 23A machine may be split as follows with "numa=fake=4*512," as reported by dmesg: 24 25 Faking node 0 at 0000000000000000-0000000020000000 (512MB) 26 Faking node 1 at 0000000020000000-0000000040000000 (512MB) 27 Faking node 2 at 0000000040000000-0000000060000000 (512MB) 28 Faking node 3 at 0000000060000000-0000000080000000 (512MB) 29 ... 30 On node 0 totalpages: 130975 31 On node 1 totalpages: 131072 32 On node 2 totalpages: 131072 33 On node 3 totalpages: 131072 34 35Now following the instructions for mounting the cpusets filesystem from 36Documentation/cgroups/cpusets.txt, you can assign fake nodes (i.e. contiguous memory 37address spaces) to individual cpusets: 38 39 [root@xroads /]# mkdir exampleset 40 [root@xroads /]# mount -t cpuset none exampleset 41 [root@xroads /]# mkdir exampleset/ddset 42 [root@xroads /]# cd exampleset/ddset 43 [root@xroads /exampleset/ddset]# echo 0-1 > cpus 44 [root@xroads /exampleset/ddset]# echo 0-1 > mems 45 46Now this cpuset, 'ddset', will only allowed access to fake nodes 0 and 1 for 47memory allocations (1G). 48 49You can now assign tasks to these cpusets to limit the memory resources 50available to them according to the fake nodes assigned as mems: 51 52 [root@xroads /exampleset/ddset]# echo $$ > tasks 53 [root@xroads /exampleset/ddset]# dd if=/dev/zero of=tmp bs=1024 count=1G 54 [1] 13425 55 56Notice the difference between the system memory usage as reported by 57/proc/meminfo between the restricted cpuset case above and the unrestricted 58case (i.e. running the same 'dd' command without assigning it to a fake NUMA 59cpuset): 60 Unrestricted Restricted 61 MemTotal: 3091900 kB 3091900 kB 62 MemFree: 42113 kB 1513236 kB 63 64This allows for coarse memory management for the tasks you assign to particular 65cpusets. Since cpusets can form a hierarchy, you can create some pretty 66interesting combinations of use-cases for various classes of tasks for your 67memory management needs. 68