3.2. Linux Performance Tools: CPU and Memory

Here begins our discussion of performance tools that enable you to extract the memory performance information described previously.

3.2.1. vmstat (Virtual Memory Statistics) II

As you have seen before, vmstat can provide information about many different performance aspects of a system—although its primary purpose, as shown next, is to provide information about virtual memory system performance. In addition to the CPU performance statistics described in the previous chapter, it can also tell you the following:

  • How much swap is being used

  • How the physical memory is being used

  • How much memory is free

As you can see, vmstat provides (via the statistics it displays) a wealth of information about the health and performance of the system in a single line of text.

3.2.1.1 System-Wide Memory-Related Options

In addition to the CPU statistics vmstat can provide, you can invoke vmstat with the following command-line options when investigating memory statistics:

vmstat [-a] [-s] [-m]


As before, you can run vmstat in two modes: sample mode and average mode. The added command-line options enable you to get the performance statistics about how the Linux kernel is using memory. Table 3-1 describes the options that vmstat accepts.

Table 3-1. vmstat Command-Line Options

Option

Explanation

-a

This changes the default output of memory statistics to indicate the active/inactive amount of memory rather than information about buffer and cache usage.

-s (procps 3.2 or greater)

This prints out the vm table. This is a grab bag of differentstatistics about the system since it has booted. It cannot be run in sample mode. It contains both memory and CPU statistics.

-m (procps 3.2 or greater)

This prints out the kernel’s slab info. This is the same information that can be retrieved by typingcat/proc/slabinfo. This describes in detail how the kernel’s memory is allocated and can be helpful to determine what area of the kernel is consuming the most memory.


Table 3-2 provides a list of the memory statistics that vmstat can provide. As with the CPU statistics, when run in normal mode, the first line that vmstat provides is the average values for all the rate statistics (so and si) and the instantaneous value for all the numeric statistics (swpdfreebuffcacheactive, and inactive).

Table 3-2. Memory-Specific vmstat Output Statistics

Column

Explanation

swpd

The total amount of memory currently swapped to disk.

free

The amount of physical memory not being used by the operating system or applications.

buff

The size (in KB) of the system buffers, or memory used to store data waiting to be saved to disk. This memory allows an application to continue execution immediately after it has issued a write call to the Linux kernel (instead of waiting until the data has been committed to disk).

cache

The size (in KB) of the system cache or memory used to store data previously read from disk. If an application needs this data again, it allows the kernel to fetch it from memory rather than disk, thus increasing performance.

active

The amount of memory actively being used. The active/ inactive statistics are orthogonal to the buffer/cache; buffer and cache memory can be active and inactive.

inactive

The amount of inactive memory (in KB), or memory that has not been used for a while and is eligible to be swapped to disk.

si

The rate of memory (in KB/s) that has been swapped in from disk during the last sample.

so

The rate of memory (in KB/s) that has been swapped out to disk during the last sample.

pages paged in

The amount of memory (in pages) read from the disk(s) into the system buffers. (On most IA32 systems, a page is 4KB.)

pages paged out

The amount of memory (in pages) written to the disk(s) from the system cache. (On most IA32 systems, a page is 4KB.)

pages swapped in

The amount of memory (in pages) read from swap into system memory.

pages swapped in/out

The amount of memory (in pages) written from system memory to the swap.

used swap

The amount of swap currently being used by the Linux kernel.

free swap

The amount of swap currently available for use.

total swap

The total amount of swap that the system has; this is also the sum of used swap plus free swap.


vmstat provides a good overview of the current state of the virtual memory system for a given machine. Although it does not provide a complete and detailed list of every Linux memory performance statistic available, it does provide a compact output that can indicate how the system memory is being used overall.

3.2.1.2 Example Usage

In Listing 3.2, as you saw in the previous chapter, if vmstat is invoked without any command-line options, it displays average values for performance statistics since system boot (si and so), and it shows the instantaneous values for the rest of them (swpdfreebuff, and cache). In this case, we can see that the system has about 500MB of memory that has been swapped to disk. ~14MB of the system memory is free. ~4MB is used for buffers that contain data that has yet to be flushed to disk. ~627MB is used for the disk cache that contains data that has been read off the disk in the past.

Listing 3.2.
bash-2.05b$ vmstat

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so     bi   bo   in    cs us sy id wa

 0  0 511012  14840   4412 642072   33   31    204  247 1110  1548  8  5 73 14


In Listing 3.3, we ask vmstat to display information about the number of active and inactive pages. The amount of inactive pages indicates how much of the memory could be swapped to disk and how much is currently being used. In this case, we can see that 1310MB of memory is active, and only 78MB is considered inactive. This machine has a large amount of memory, and much of it is being actively used.

Listing 3.3.
bash-2.05b$ vmstat -a

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

 r  b   swpd   free  inact active   si   so    bi    bo   in    cs us sy id wa

 2  1 514004   5640 79816 1341208   33   31   204   247 1111  1548  8  5 73 14


Next, in Listing 3.4, we look at a different system, one that is actively swapping data in and out of memory. The si column indicates that swap data has been read in at a rate of 480KB, 832KB, 764KB, 344KB, and 512KB during each of those sample periods. The so column indicates that memory data has been written to swap at a rate of 9KB, 0KB, 916KB, 0KB, 1068KB, 444KB, 792KB, during each of the samples. These results could indicate that the system does not have enough memory to handle all the running processes. A simultaneously high swap-in and swap-out rate can occur when a process’s memory is being saved to make way for an application that had been previously swapped to disk. This can be disastrous if two running programs both need more memory than the system can provide. For example, if two processes are using a large amount of memory, and both are trying to run simultaneously, each can cause the other’s memory to be written to swap. When one of the programs needs a piece of memory, it kicks out one that the other applications needs. When the other application starts to run, it kicks out a piece of memory that the original program was using, and waits until its memory is loaded from the swap. This can cause both applications to grind to a halt while they wait for their memory to be retrieved from swap before they can continue execution. As soon as one makes a little bit of progress, it swaps out memory that the other process was using and causes the other program to slow down. This is called thrashing. When this happens, the system spends most of its time reading memory to and from the swap, and system performance dramatically slows down.

In this particular case, the swapping eventually stopped, so most likely the memory that was swapped to disk was not immediately needed by the original process. This means the swap usage was effective, the contents of the memory that was not being used was written to disk, and the memory was then given to the process that needed it.

Listing 3.4.
[ezolt@localhost book]$ vmstat 1 100

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa

 2  1 131560   2320   8640  53036    1    9   107    69 1137   426 10  7 74  9

 0  1 131560   2244   8640  53076  480    0   716     0 1048   207  6  1  0 93

 1  2 132476   3424   8592  53272  832  916  1356   916 1259   692 11  4  0 85

 1  0 132476   2400   8600  53280  764    0  1040    40 1288   762 14  5  0 81

 0  1 133544   2656   8624  53392  344 1068  1096  1068 1217   436  8  3  5 84

 0  1 133988   2300   8620  54288  512  444  1796   444 1090   230  5  1  2 92

 0  0 134780   3148   8612  53688    0  792     0   792 1040   166  5  1 92  2

 0  0 134780   3148   8612  53688    0    0     0     0 1050   158  4  1 95  0

 0  0 134780   3148   8612  53688    0    0     0     0 1148   451  7  2 91  0

 0  0 134780   3148   8620  53680    0    0     0    12 1196   477  8  2 78 12

 ....


As shown in Listing 3.5, as you saw in the previous chapter, vmstat can show a vast array of different system statistics. Now as we look at it, we can see some of the same statistics that were present in some of the other output modes, such as active, inactive, buffer, cache, and used swap. However, it also has a few new statistics, such as total memory, which indicates that this system has a total of 1516MB of memory, and total swap, which indicates that this system has a total of 2048MB of swap. It can be helpful to know the system totals when trying to figure out what percentage of the swap and memory is currently being used. Another interesting statistic is the pages paged in, which indicates the total number of pages that were read from the disk. This statistic includes the pages that are read starting an application and those that the application itself may be using.

Listing 3.5.
bash-2.05b$ vmstat -s

      1552528  total memory

      1546692  used memory

      1410448  active memory

        11100  inactive memory

         5836  free memory

         2676  buffer memory

       645864  swap cache

      2097096  total swap

       526280  used swap

      1570816  free swap

     20293225 non-nice user cpu ticks

     18284715 nice user cpu ticks

     17687435 system cpu ticks

    357314699 idle cpu ticks

     67673539 IO-wait cpu ticks

       352225 IRQ cpu ticks

      4872449 softirq cpu ticks

    495248623 pages paged in

    600129070 pages paged out

     19877382 pages swapped in

     18874460 pages swapped out

   2702803833 interrupts

   3763550322 CPU context switches

   1094067854 boot time

     20158151 forks


Finally, in Listing 3.6, we see that vmstat can provide information about how the Linux kernel allocates its memory. As previously described, the Linux kernel has a series of “slabs” to hold its dynamic data structures. vmstat displays each of the slabs (Cache), shows how many of the elements are being used (Num), shows how many are allocated (Total), shows the size of each element (Size), and shows the amount of memory in pages (Pages) that the total slab is using. This can be helpful when tracking down exactly how the kernel is using its memory.

Listing 3.6.
bash-2.05b$ vmstat -m

Cache                          Num   Total   Size   Pages

udf_inode_cache                  0       0    416       9

fib6_nodes                       7     113     32     113

ip6_dst_cache                    9      17    224      17

ndisc_cache                      1      24    160      24

raw6_sock                        0       0    672       6

udp6_sock                        0       0    640       6

tcp6_sock                      404     441   1120       7

ip_fib_hash                     39     202     16     202

ext3_inode_cache              1714    3632    512       8

...


vmstat provides an easy way to extract large amounts of information about the Linux memory subsystem. When combined with the other information provided on the default output screen, it provides a fair picture of the health and resource usage of the system.

3.2.2. top (2.x and 3.x)

As discussed in the previous chapter, top presents both system-wide and process-specific performance statistics. By default, top presents a list, in decreasing order, of the top CPU-consuming processes, but it can also be adjusted to sort by the total memory usage, enabling you to track down which process is using the most memory.

3.2.2.1 Memory Performance-Related Options

top does not have any special command-line options that manipulate its display of memory statistics. It is invoked with the following command line:

top


However, once running, top enables you to select whether system-wide memory information displays, and whether processes are sorted by memory usage. Sorting by memory consumption proves particularly useful to determine which process is consuming the most memory. Table 3-3 describes the different memory-related output toggles.

Table 3-3. top Runtime Toggles

Option

Explanation

m

This toggles whether information about the system memory usage will be shown on the screen.

M

Sorts the tasks by the amount of memory they are using. Because processes may have allocated more memory than they are using, this sorts by resident set size. Resident set size is the amount the processes are actually using rather than what they have simply asked for.


Table 3-4 describes the memory performance statistics that top can provide for both the entire system and individual processes. top has two different versions, 2.x and 3.x, which have slightly different names for output statistics. Table 3-4 describes the names for both versions.

Table 3-4. top Memory Performance Statistics

Option

Explanation

%MEM

This is the percentage of the system’s physical memory that this process is using.

SIZE (v 2.x)

VIRT (v 3.x)

This is the total size of the process’s virtual memory usage. This includes all the memory that the application has allocated but is not using.

SWAP

This is the amount of swap (in KB) that the process is using.

RSS (v 2.x)

RES (v 3.x)

This is the amount of physical memory that the application is actually using.

TRS (v 2.x)

CODE (v 3.x)

The total amount of physical memory (in KB) that the process’s executable code is using.

DSIZE (v 2.x)

DATA (v 3.x)

The total amount of memory (in KB) dedicated to a process’s data and stack.

SHARE (v 2.x)

SHR (v 3.x)

The total amount of memory (in KB) that can be shared with other processors.

D (v 2.x)

nDRT (v 3.x)

The number of pages that are dirty and need to be flushed to disk.

Mem:

total, used, free

Of the physical memory, this indicates the total amount, the used amount, and the free amount.

swap: total, used, free

Of the swap, this indicates the total amount, the used amount, and the free amount.

active (v 2.x)

The amount of physical memory currently active.

inactive (v 2.x)

The amount of physical memory that is inactive and hasn’t been used in a while.

buffers

The total amount of physical memory (in KB) used to buffer values to be written to disk.


top provides a large amount of memory information about the different running processes. As discussed in later chapters, you can use this information to determine exactly how an application allocates and uses memory.

3.2.2.2 Example Usage

Listing 3.7 is similar to the example run of top shown in the previous chapter. However, in this example, notice that in the buffers, we have a total amount of 1,024MB of physical memory, of which ~84MB is free.

Listing 3.7.
[ezolt@wintermute doc]$ top

top - 15:47:03 up 24 days,  4:32, 15 users,  load average: 1.17, 1.19, 1.17

Tasks: 151 total,   1 running, 138 sleeping,  11 stopped,   1 zombie

Cpu(s):  1.2% us,  0.7% sy,  0.0% ni, 93.0% id,  4.9% wa,  0.0% hi,  0.1% si

Mem:   1034320k total,   948336k used,    85984k free,    32840k buffers

Swap:  2040244k total,   276796k used,  1763448k free,   460864k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

26364 root      16   0  405m  71m 321m S  4.0  7.1 462:25.50 X

17345 ezolt     16   0  176m  73m  98m S  2.6  7.3   1:17.48 soffice.bin

18316 ezolt     16   0  2756 1096 1784 R  0.7  0.1   0:05.54 top

26429 ezolt     16   0 65588  52m  12m S  0.3  5.2  16:16.77 metacity

26510 ezolt     16   0 19728 5660  16m S  0.3  0.5  27:57.87 clock-applet

26737 ezolt     16   0 70224  35m  20m S  0.3  3.5   8:32.28 gnome-terminal

     1 root     16   0  2396  448 1316 S  0.0  0.0   0:01.72 init

     2 root     RT   0     0    0    0 S  0.0  0.0   0:00.88 migration/0

     3 root     34  19     0    0    0 S  0.0  0.0   0:00.01 ksoftirqd/0

     4 root     RT   0     0    0    0 S  0.0  0.0   0:00.35 migration/1

     5 root     34  19     0    0    0 S  0.0  0.0   0:00.01 ksoftirqd/1

     6 root     RT   0     0    0    0 S  0.0  0.0   0:34.20 migration/2

     7 root     34  19     0    0    0 S  0.0  0.0   0:00.01 ksoftirqd/2


Again, top can be customized to display only what you are interested in observing. Listing 3.8 shows a highly configured output screen that shows only memory performance statistics.

Listing 3.8.
Mem:   1034320k total,   948336k used,    85984k free,    33024k buffers

Swap:  2040244k total,   276796k used,  1763448k free,   460680k cached

 VIRT  RES  SHR %MEM SWAP CODE DATA nFLT nDRT COMMAND

 405m  71m 321m  7.1 333m 1696 403m 4328    0 X

70224  35m  20m  3.5  33m  280  68m 3898    0 gnome-terminal

 2756 1104 1784  0.1 1652   52 2704    0    0 top

19728 5660  16m  0.5  13m   44  19m   17    0 clock-applet

 2396  448 1316  0.0 1948   36 2360   16    0 init

    0    0    0  0.0    0    0    0    0    0 migration/0

    0    0    0  0.0    0    0    0    0    0 ksoftirqd/0

    0    0    0  0.0    0    0    0    0    0 migration/1

    0    0    0  0.0    0    0    0    0    0 ksoftirqd/1

    0    0    0  0.0    0    0    0    0    0 migration/2

    0    0    0  0.0    0    0    0    0    0 ksoftirqd/2

    0    0    0  0.0    0    0    0    0    0 migration/3


top provides a real-time update of memory statistics and a display of which processes are using which types of memory. This information becomes useful as we investigate application memory usage.

3.2.3. procinfo II

As we’ve seen before, procinfo provides a view of the system-wide performance characteristics. In addition to the statistics described in the previous chapter, procinfo provides a few memory statistics, similar to vmstat and top, that indicate how the current memory is being used.

3.2.3.1 Memory Performance-Related Options

procinfo does not have any options that change the output of the memory statistics displayed and, as a result, is invoked with the following command:

procinfo


procinfo displays the basic memory system memory statistics, similar to top and vmstat. These are shown in Table 3-5.

Table 3-5. procinfo CPU Statistics

Option

Explanation

Total

This is the total amount of physical memory.

Used

This is the amount of physical memory in use.

Free

This is the amount of unused physical memory.

Shared

This is an obsolete value and should be ignored.

Buffers

This is the amount of physical memory used as buffers for disk writes.

Page in

The number of blocks (usually 1KB) read from disk. (This is broken on 2.6.x kernels.)

Page out

The number of blocks (usually 1KB) written to disk. (This is broken on 2.6.x kernels.)

Swap in

The number of memory pages read in from swap. (This statistic is broken on 2.6.x kernels.)

Swap out

The number of memory pages written to swap. (This statistic is broken on 2.6.x kernels.)


Much like vmstat or topprocinfo is a low-overhead command that is good to leave running in a console or window on the screen. It gives a good indication of a system’s health and performance.

3.2.3.2 Example Usage

Listing 3.9 is a typical output for procinfo. As you can see, it reports summary information about how the system is using virtual memory. In this case, the system has a total of 312MB of memory; 301MB is in use by the kernel and applications, 11MB is used by system buffers, and 11MB is not used at all.

Listing 3.9.
[ezolt@localhost procinfo-18]$ ./procinfo

Linux 2.6.6-1.435.2.3smp (bhcompile@tweety.build.redhat.com) (gcc 3.3.3 20040412 )

#1

1CPU [localhost]

Memory:      Total        Used        Free      Shared      Buffers

Mem:        320468      308776       11692           0        11604

Swap:       655192      220696      434496

Bootup: Sun Oct 24 10:03:43 2004    Load average: 0.44 0.53 0.51 3/110 32243

user  :       0:57:58.92   9.0%  page in :        0

nice  :       0:02:51.09   0.4%  page out:        0

system:       0:20:18.43   3.2%  swap in :        0

idle  :       8:47:31.54  81.9%  swap out:        0

uptime:      10:44:01.94         context : 13368094

irq  0:  38645994 timer                 irq  7:         2

irq  1:     90516 i8042                 irq  8:         1 rtc

irq  2:         0 cascade [4]           irq  9:         2

irq  3:    742857 prism2_cs             irq 10:         2

irq  4:         6                       irq 11:    562551 uhci_hcd, yenta, yen

irq  5:         2                       irq 12:   1000803 i8042

irq  6:         8                       irq 14:    207681 ide0


procinfo provides information about system performance in a single screen of information. Although it provides a few of the important statistics for memory, vmstat or top is much better suited for investigating system-wide memory usage.

3.2.4. gnome-system-monitor (II)

gnome-system-monitor is in many ways a graphical counterpart of top. It enables you to monitor individual process and observe the load on the system based on the graphs that it displays. It also provides rudimentary graphs of CPU and memory usage.

3.2.4.1 Memory Performance-Related Options

gnome-system-monitor can be invoked from the Gnome menu. (Under Red Hat 9 and higher, this is in System Tools > System Monitor.) However, it can also be invoked using the following command:

gnome-system-monitor


gnome-system-monitor has no relevant command-line options that affect the memory performance measurements.

3.2.4.2 Example Usage

When you launch gnome-system-monitor and select the System Monitor tab, you see a window similar to Figure 3-1. This window enables you to glance at the graph and see how much physical memory and swap is currently being used, and how its usage has changed over time. In this case, we see that 969MB of a total of 1,007MB is currently being used. The memory usage has been relatively flat for a while.

 

Figure 3-1.

 


The graphical view of data provided by gnome-system-monitor can make it easier and faster; however, most of the details, such as how the memory is being used, are missing.

3.2.5. free

free provides an overall view of how your system is using your memory, including the amount of free memory. Although the free command may show that a particular system does not have much free memory, this is not necessarily bad. Instead of letting memory sit unused, the Linux kernel puts it to work as a cache for disk reads and as a buffer for disk writes. This can dramatically increase system performance. Because these cache and buffers can always be discarded and the memory can be used if needed by applications, free shows you the amount of free memory plus or minus these buffers.

3.2.5.1 Memory Performance-Related Options

free can be invoked using the following command line:

free [-l] [-t] [-s delay ] [-c count ]


Table 3-6 describes the parameters that modify the types of statistics that free displays. Much like vmstatfree can periodically display updated memory statistics.

Table 3-6. free Command-Line Options

Option

Explanation

-s delay

This option causes free to print out new memory statistics every delay seconds.

-c count

This option causes free to print out new statistics for count times.

-l

This option shows you how much high memory and how much low memory are being used.


free actually displays some of the most complete memory statistics of any of the memory statistic tools. The statistics that it displays are shown in Table 3-7.

Table 3-7. free Memory Statistics

Statistic

Explanation

Total

This is the total amount of physical memory and swap.

Used

This is the amount of physical memory and swap in use.

Free

This is the amount of unused physical memory and swap.

Shared

This is an obsolete value and should be ignored.

Buffers

This is the amount of physical memory used as buffers for disk writes.

Cached

This is the amount of physical memory used as cache for disk reads.

-/+ buffers/cache

In the Used column, this shows the amount of memory that would be used if buffers/cache were not counted as used memory. In the Free column, this shows the amount of memory that would be free if buffers/cache were counted as free memory.

Low

The total amount of low memory or memory directly accessible by the kernel.

High

The total amount of high memory or memory not directly accessible by the kernel.

Totals

This shows the combination of physical memory and swap for the TotalUsed, and Free columns.


free provides information about the system-wide memory usage of Linux. It is a fairly complete range of memory statistics.

3.2.5.2 Example Usage

Calling free without any command options gives you an overall view of the memory subsystem.

As mentioned previously, Linux tries to use all the available memory if possible to cache data and applications. In Listing 3.10free tells us that we are currently using 234,720 bytes of memory; however, if you ignore the buffers and cache, we are only using 122,772 bytes of memory. The opposite is true of the free column. We currently have 150,428 bytes of memory free; if you also count the buffers and cached memory (which you can, because Linux throws away those buffers if the memory is needed), however, we have 262,376 bytes of memory free.

Listing 3.10.
[ezolt@wintermute procps-3.2.0]$ free

             total       used       free     shared    buffers      cached

Mem:        385148     234720     150428          0       8016      103932

-/+ buffers/cache:     122772     262376

Swap:       394080      81756     312324


Although you could just total the columns yourself, the -t flag shown in Listing 3.11 tells you the totals when adding both swap and real memory. In this case, the system had 376MB of physical memory and 384MB of swap. The total amount of memory available on the system is 376MB plus 384MB, or ~760MB. The total amount of free memory was 134MB of physical memory plus 259MB of swap, yielding a total of 393MB of free memory.

Listing 3.11.
[ezolt@wintermute procps-3.2.0]$ free -t

             total       used       free     shared    buffers      cached

Mem:        385148     247088     138060          0       9052      115024

-/+ buffers/cache:     123012     262136

Swap:       394080      81756     312324

Total:      779228     328844     450384


Finally, free tells you the amount of high and low memory that the system is using. This is mainly useful on 32-bit machines (such as IA32) with 1GB or more of physical memory. (32-bit machines are the only machines that will have high memory.) Listing 3.12 shows a system with a very small amount of free memory, 6MB in total. It shows a system with 876MB of low memory and 640MB of high memory. This system also has much more cached memory than buffer memory, suggesting that it may be aggressively writing data to disk rather than leaving it in the buffer cache a long time.

Listing 3.12.
fas% free -l

             total       used       free     shared    buffers      cached

Mem:       1552528    1546472       6056          0       7544      701408

Low:        897192     892800       4392

High:       655336     653672       1664

-/+ buffers/cache:     837520     715008

Swap:      2097096     566316    1530780


free gives a good idea of how the system memory is being used. It may take a little while to get used to output format, but it contains all the important memory statistics.

3.2.6. slabtop

slabtop is similar to top, but instead of displaying information about the CPU and memory usage of the processes in the system, slabtop shows in real-time how the kernel is allocating its various caches and how full they are. Internally, the kernel has a series of caches that are made up of one or more slabs. Each slab consists of a set of one or more objects. These objects can be active (or used) or inactive (unused). slabtop shows you the status of the different slabs. It shows you how full they are and how much memory they are using.

3.2.6.1 Memory Performance-Related Options

slabtop is invoked using the following command line:

slabtop [--delay n –sort={a | b | c | l | v | n | o | p | s | u}


The command-line options for slabtop are described in Table 3-8.

Table 3-8. slabtop Command-Line Options

Option

Explanation

--delay

This specifies how long slabtop waits between updates.

--sort {order}

This specifies the order of the output. order can be one of the following:

a

This sorts by the number of active objects in each slab.

b

This sorts by the total number of objects (active and inactive) in each slab for a particular cache.

c

This sorts by the total amount of memory each cache is using.

l

This sorts by the number of slabs used by each cache.

v

This sorts by the number of active slabs used by each cache.

n

This sorts by the name of the cache.

o

This sorts by the number of objects in the particular cache.

P

This sorts by the number of pages used per slab.

s

This sorts by the size of the objects in the cache.


slabtop provides a glimpse into the data structures of the Linux kernel. Each of these slab types is tied closed to the Linux kernel, and a description of each of these slabs is beyond the scope of this book. If a particular slab is using a large amount of kernel memory, reading the Linux kernel source code and searching the Web are two great ways to figure out what these slabs are used for.

3.2.6.2 Example Usage

As shown in Listing 3.13, by default, slabtop fills the entire console and continually updates the statistics every three seconds. In this example, you can see that the size-64 slab has the most objects, only half of which are active.

Listing 3.13.
[ezolt@wintermute proc]$ slabtop

 Active / Total Objects (% used)    : 185642 / 242415 (76.6%)

 Active / Total Slabs (% used)      : 12586 / 12597 (99.9%)

 Active / Total Caches (% used)     : 88 / 134 (65.7%)

 Active / Total Size (% used)       : 42826.23K / 50334.67K (85.1%)

 Minimum / Average / Maximum Object : 0.01K / 0.21K / 128.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME

 66124  34395  52%    0.06K   1084       61      4336K size-64

 38700  35699  92%    0.05K    516       75      2064K buffer_head

 30992  30046  96%    0.15K   1192       26      4768K dentry_cache

 21910  21867  99%    0.27K   1565       14      6260K radix_tree_node

 20648  20626  99%    0.50K   2581        8     10324K ext3_inode_cache

 11781   7430  63%    0.03K     99      119       396K size-32

  9675   8356  86%    0.09K    215       45       860K vm_area_struct

  6024   2064  34%    0.62K   1004        6      4016K ntfs_big_inode_cache

  4520   3633  80%    0.02K     20      226        80K anon_vma

  4515   3891  86%    0.25K    301       15      1204K filp

  4464   1648  36%    0.12K    144       31       576K size-128

  3010   3010 100%    0.38K    301       10      1204K proc_inode_cache

  2344    587  25%    0.50K    293        8      1172K size-512

  2250   2204  97%    0.38K    225       10       900K inode_cache

  2100    699  33%    0.25K    140       15       560K size-256

  1692   1687  99%    0.62K    282        6      1128K nfs_inode_cache

  1141   1141 100%    4.00K   1141        1      4564K size-4096


Because the information provided by slabtop is updated periodically, it is a great way to see how the Linux kernel’s memory usage changes in response to different workloads.

3.2.7. sar (II)

All the advantages of using sar as a CPU performance tool, such as the easy recording of samples, extraction to multiple output formats, and time stamping of each sample, are still present when monitoring memory statistics. sarprovides similar information to the other memory statistics tools, such as the current values for the amount of free memory, buffers, cached, and swap amount. However, it also provides the rate at which these value change and provides information about the percentage of physical memory and swap that is currently being consumed.

3.2.7.1 Memory Performance-Related Options

sar can be invoked with the following command line:

sar [-B] [-r] [-R]


By default, sar displays only CPU performance statistics; so to retrieve any of the memory subsystem statistics, you must use the options described in Table 3-9.

Table 3-9. sar Command-Line Options

Option

Explanation

-B

This reports information about the number of blocks that the kernel swapped to and from disk. In addition, for kernel versions after v2.5, it reports information about the number of page faults.

-W

This reports the number of pages of swap that are brought in and out of the system.

-r

This reports information about the memory being used in the system. It includes information about the total free memory, swap, cache, and buffers being used.


sar provides a fairly complete view of the Linux memory subsystem. One advantage that sar provides over other tools is that it shows the rate of change for many of the important values in addition to the absolute values. You can use these values to see exactly how memory usage is changing over time without having to figure out the differences between values at each sample. Table 3-10 shows the memory statistics that sar provides.

Table 3-10. sar Memory Statistics

Statistic

Explanation

pgpgin/s

The amount of memory (in KB) that the kernel paged in from disk.

pgpgout/s

The amount of memory (in KB) that the kernel paged out to disk.

fault/s

The total number of faults that that the memory subsystem needed to fill. These may or may not have required a disk access.

majflt/s

The total number of faults that the memory subsystem needed to fill and required a disk access.

pswpin/s

The amount of swap (in pages) that the system brought into memory.

pswpout/s

The amount of memory (in pages) that the system wrote to swap.

kbmemfree

This is the total physical memory (in KB) that is currently free or not being used.

kbmemused

This is the total amount of physical memory (in KB) currently being used.

%memused

This is the percentage of the total physical memory being used.

kbbuffers

This is the amount of physical memory used as buffers for disk writes.

kbcached

This is the amount of physical memory used as cache for disk reads.

kbswpfree

This is the amount of swap (in KB) currently free.

kbswpused

This is the amount of swap (in KB) currently used.

%swpused

This is the percentage of the swap being used.

kbswpcad

This is memory that is both swapped to disk and present in memory. If the memory is needed, it can be immediately reused because the data is already present in the swap area.

frmpg/s

The rate that the system is freeing memory pages. A negative number means the system is allocating them.

bufpg/s

The rate that the system is using new memory pages as buffers. A negative number means that number of buffers is shrinking, and the system is using less of them.


Although sar misses the high and low memory statistics, it provides nearly every other memory statistic. The fact that it can also record network CPU and disk I/O statistics makes it very powerful.

3.2.7.2 Example Usage

Listing 3.14 shows sar providing information about the current state of the memory subsystem. From the results, we can see that the amount of memory that the system used varies from 98.87 percent to 99.25 percent of the total memory. Over the course of the observation, the amount of free memory drops from 11MB to 7MB. The percentage of swap that is used hovers around ~11 percent. The system has ~266MB of data cache and about 12MB in buffers that can be written to disk.

Listing 3.14.

[ezolt@scrffy manuscript]$ sar -r 1 5

Linux 2.6.8-1.521smp (scrffy)   10/25/2004

09:45:30 AM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  

%swpused  kbswpcad

09:45:31 AM     11732   1022588     98.87     12636    272284   1816140    224104     10

.98     66080

09:45:32 AM     10068   1024252     99.03     12660    273300   1816140    224104     10

.98     66080

09:45:33 AM      5348   1028972     99.48     12748    275292   1816140    224104     10

.98     66080

09:45:35 AM      4932   1029388     99.52     12732    273748   1816140    224104     10

.98     66080

09:45:36 AM      6968   1027352     99.33     12724    271876   1815560    224684     11

.01     66660

Average:         7810   1026510     99.25     12700    273300   1816024    224220     10

.99     66196


Listing 3.15 shows that the system is consuming the free memory at a rate of ~82 pages per second during the first sample. Later, it frees ~16 pages, and then consumes ~20 pages. The number pages of buffers grew only once during the observation, at a rate of 2.02 pages per second. Finally, the pages of cache shrunk by 2.02, but in the end, they expanded by 64.36 pages/second.

Listing 3.15.
[ezolt@scrffy manuscript]$ sar -R 1 5

Linux 2.6.8-1.521smp (scrffy)   10/25/2004

09:57:22 AM   frmpg/s   bufpg/s    campg/s

09:57:23 AM    -81.19      0.00       0.00

09:57:24 AM      8.00      0.00       0.00

09:57:25 AM      0.00      2.02      -2.02

09:57:26 AM     15.84      0.00       0.00

09:57:27 AM    -19.80      0.00      64.36

Average:       -15.54      0.40      12.55


Listing 3.16 shows that the system wrote ~53 pages of memory to disk during the third sample. The system has a relatively high fault count, which means that memory pages are being allocated and used. Fortunately, none of these are major faults, meaning that the system does not have to go to disk to fulfill the fault.

Listing 3.16.
[ezolt@scrffy dvi]$ sar -B 1 5

Linux 2.6.8-1.521smp (scrffy)   10/25/2004

09:58:34 AM  pgpgin/s pgpgout/s   fault/s   majflt/s

09:58:35 AM      0.00      0.00   1328.28       0.00

09:58:36 AM      0.00      0.00    782.18       0.00

09:58:37 AM      0.00     53.06    678.57       0.00

09:58:38 AM      0.00      0.00    709.80       0.00

09:58:39 AM      0.00      0.00    717.17       0.00

Average:         0.00     10.42    842.48       0.00


As you can see, sar is a powerful tool that enhances the functionality of the other system memory performance tools by adding the ability to archive, time stamp, and simultaneously collect many different types of statistics.

3.2.8. /proc/meminfo

The Linux kernel provides a user-readable text file called /proc/meminfo that displays current system-wide memory performance statistics. It provides a superset of the system-wide memory statistics that can be acquired with vmstat,topfree, and procinfo; but it is a little more difficult to use. If you want periodic updates, you must write a script or some code for such. If you want to store the memory performance information or coordinate it with CPU statistics, you must create a new tool or write a script. However, /proc/meminfo provides the most complete view of system memory usage.

3.2.8.1 Memory Performance-Related Options

The information in /proc/meminfo can be retrieved with the following command line:

cat /proc/meminfo


This command displays the array of statistics described in Table 3-11.

Table 3-11. /proc/meminfo Memory Statistics (All in KB)

Statistic

Explanation

MemTotal

The total amount of system physical memory.

MemFree

The total amount of free physical memory.

Buffers

The amount of memory used for pending disk writes.

Cached

The amount of memory used to cache disk reads.

SwapCached

The amount of memory that exists in both the swap and the physical memory.

Active

The amount of memory currently active in the system.

Inactive

The amount of memory currently inactive and a candidate for swap.

HighTotal

The amount of high memory (in KB).

HighFree

The amount of free high memory (in KB).

LowTotal

The amount of low memory (in KB).

LowFree

The amount of free low memory (in KB).

SwapTotal

The amount of swap memory (in KB).

SwapFree

The amount of free swap memory (in KB).

Dirty

Memory waiting to be written to disk.

Writeback

Memory currently being written to disk.

Mapped

The total amount of memory brought into a process’s virtual address space using mmap.

Slab

The total amount of kernel slab memory (in KB).

Committed_AS

The amount of memory required to almost never run out of memory with the current workload. Normally, the kernel hands out more memory than it actually has with the hope that applications overallocate. If all the applications were to use what they allocate, this is the amount of physical memory you would need.

PageTables

The amount of memory reserved for kernel page tables.

VmallocTotal

The amount of kernel memory usable for vmalloc.

VmallocUsed

The amount of kernel memory used for vmalloc.

VmallocChunk

The largest contiguous chunk of vmalloc‘able memory.

HugePages_Total

The total size of all HugePages.

HugePages_Free

The total amount of free HugePages.


/proc/meminfo provides a wealth of information about the current state of the Linux memory subsystem.

3.8.2.2 Example Usage

Listing 3.17 shows an example output of /proc/meminfo. It shows some memory statistics that are similar to those we have seen with other tools. However, some statistics show new information. One, Dirty, shows that the system currently has 24KB of data waiting to be written to disk. Another, Commited_AS, shows that we need a little more memory (needed: 1068MB versus total: 1024MB) to avoid a potential out-of-memory situation.

Listing 3.17.
[ezolt@scrffy /tmp]$ cat /proc/meminfo

MemTotal:      1034320 kB

MemFree:         10788 kB

Buffers:         29692 kB

Cached:         359496 kB

SwapCached:     113912 kB

Active:         704928 kB

Inactive:       222764 kB

HighTotal:           0 kB

HighFree:            0 kB

LowTotal:      1034320 kB

LowFree:         10788 kB

SwapTotal:     2040244 kB

SwapFree:      1756832 kB

Dirty:              24 kB

Writeback:           0 kB

Mapped:         604248 kB

Slab:            51352 kB

Committed_AS:  1093856 kB

PageTables:       9560 kB

VmallocTotal:  3088376 kB

VmallocUsed:     26600 kB

VmallocChunk:  3058872 kB

HugePages_Total:     0

HugePages_Free:      0

Hugepagesize:     2048 kB


/proc/meminfo is the most complete place to gather system-wide Linux memory statistics. Because you can treat it as a text file, any custom script or program can easily extract the statistics.