3.2. Linux Performance Tools: CPU and Memory

Here begins our discussion of performance tools that enable you to extract the memory performance information described previously.

3.2.1. vmstat (Virtual Memory Statistics) II

As you have seen before, vmstat can provide information about many different performance aspects of a system—although its primary purpose, as shown next, is to provide information about virtual memory system performance. In addition to the CPU performance statistics described in the previous chapter, it can also tell you the following:

• How much swap is being used

• How the physical memory is being used

• How much memory is free

As you can see, vmstat provides (via the statistics it displays) a wealth of information about the health and performance of the system in a single line of text.

3.2.1.1 System-Wide Memory-Related Options

In addition to the CPU statistics vmstat can provide, you can invoke vmstat with the following command-line options when investigating memory statistics:

As before, you can run vmstat in two modes: sample mode and average mode. The added command-line options enable you to get the performance statistics about how the Linux kernel is using memory. Table 3-1 describes the options that vmstat accepts.

Table 3-1. vmstat Command-Line Options

Option	Explanation
-a	This changes the default output of memory statistics to indicate the active/inactive amount of memory rather than information about buffer and cache usage.
-s (procps 3.2 or greater)	This prints out the vm table. This is a grab bag of differentstatistics about the system since it has booted. It cannot be run in sample mode. It contains both memory and CPU statistics.
-m (procps 3.2 or greater)	This prints out the kernel’s slab info. This is the same information that can be retrieved by typingcat/proc/slabinfo. This describes in detail how the kernel’s memory is allocated and can be helpful to determine what area of the kernel is consuming the most memory.

Table 3-2 provides a list of the memory statistics that vmstat can provide. As with the CPU statistics, when run in normal mode, the first line that vmstat provides is the average values for all the rate statistics (so and si) and the instantaneous value for all the numeric statistics (swpd, free, buff, cache, active, and inactive).

Table 3-2. Memory-Specific vmstat Output Statistics

Column	Explanation
swpd	The total amount of memory currently swapped to disk.
free	The amount of physical memory not being used by the operating system or applications.
buff	The size (in KB) of the system buffers, or memory used to store data waiting to be saved to disk. This memory allows an application to continue execution immediately after it has issued a write call to the Linux kernel (instead of waiting until the data has been committed to disk).
cache	The size (in KB) of the system cache or memory used to store data previously read from disk. If an application needs this data again, it allows the kernel to fetch it from memory rather than disk, thus increasing performance.
active	The amount of memory actively being used. The active/ inactive statistics are orthogonal to the buffer/cache; buffer and cache memory can be active and inactive.
inactive	The amount of inactive memory (in KB), or memory that has not been used for a while and is eligible to be swapped to disk.
si	The rate of memory (in KB/s) that has been swapped in from disk during the last sample.
so	The rate of memory (in KB/s) that has been swapped out to disk during the last sample.
pages paged in	The amount of memory (in pages) read from the disk(s) into the system buffers. (On most IA32 systems, a page is 4KB.)
pages paged out	The amount of memory (in pages) written to the disk(s) from the system cache. (On most IA32 systems, a page is 4KB.)
pages swapped in	The amount of memory (in pages) read from swap into system memory.
pages swapped in/out	The amount of memory (in pages) written from system memory to the swap.
used swap	The amount of swap currently being used by the Linux kernel.
free swap	The amount of swap currently available for use.
total swap	The total amount of swap that the system has; this is also the sum of used swap plus free swap.

vmstat provides a good overview of the current state of the virtual memory system for a given machine. Although it does not provide a complete and detailed list of every Linux memory performance statistic available, it does provide a compact output that can indicate how the system memory is being used overall.

3.2.1.2 Example Usage

In Listing 3.2, as you saw in the previous chapter, if vmstat is invoked without any command-line options, it displays average values for performance statistics since system boot (si and so), and it shows the instantaneous values for the rest of them (swpd, free, buff, and cache). In this case, we can see that the system has about 500MB of memory that has been swapped to disk. ~14MB of the system memory is free. ~4MB is used for buffers that contain data that has yet to be flushed to disk. ~627MB is used for the disk cache that contains data that has been read off the disk in the past.

Listing 3.2.

In Listing 3.3, we ask vmstat to display information about the number of active and inactive pages. The amount of inactive pages indicates how much of the memory could be swapped to disk and how much is currently being used. In this case, we can see that 1310MB of memory is active, and only 78MB is considered inactive. This machine has a large amount of memory, and much of it is being actively used.

Listing 3.3.

Next, in Listing 3.4, we look at a different system, one that is actively swapping data in and out of memory. The si column indicates that swap data has been read in at a rate of 480KB, 832KB, 764KB, 344KB, and 512KB during each of those sample periods. The so column indicates that memory data has been written to swap at a rate of 9KB, 0KB, 916KB, 0KB, 1068KB, 444KB, 792KB, during each of the samples. These results could indicate that the system does not have enough memory to handle all the running processes. A simultaneously high swap-in and swap-out rate can occur when a process’s memory is being saved to make way for an application that had been previously swapped to disk. This can be disastrous if two running programs both need more memory than the system can provide. For example, if two processes are using a large amount of memory, and both are trying to run simultaneously, each can cause the other’s memory to be written to swap. When one of the programs needs a piece of memory, it kicks out one that the other applications needs. When the other application starts to run, it kicks out a piece of memory that the original program was using, and waits until its memory is loaded from the swap. This can cause both applications to grind to a halt while they wait for their memory to be retrieved from swap before they can continue execution. As soon as one makes a little bit of progress, it swaps out memory that the other process was using and causes the other program to slow down. This is called thrashing. When this happens, the system spends most of its time reading memory to and from the swap, and system performance dramatically slows down.

In this particular case, the swapping eventually stopped, so most likely the memory that was swapped to disk was not immediately needed by the original process. This means the swap usage was effective, the contents of the memory that was not being used was written to disk, and the memory was then given to the process that needed it.

Listing 3.4.

As shown in Listing 3.5, as you saw in the previous chapter, vmstat can show a vast array of different system statistics. Now as we look at it, we can see some of the same statistics that were present in some of the other output modes, such as active, inactive, buffer, cache, and used swap. However, it also has a few new statistics, such as total memory, which indicates that this system has a total of 1516MB of memory, and total swap, which indicates that this system has a total of 2048MB of swap. It can be helpful to know the system totals when trying to figure out what percentage of the swap and memory is currently being used. Another interesting statistic is the pages paged in, which indicates the total number of pages that were read from the disk. This statistic includes the pages that are read starting an application and those that the application itself may be using.

Listing 3.5.

Finally, in Listing 3.6, we see that vmstat can provide information about how the Linux kernel allocates its memory. As previously described, the Linux kernel has a series of “slabs” to hold its dynamic data structures. vmstat displays each of the slabs (Cache), shows how many of the elements are being used (Num), shows how many are allocated (Total), shows the size of each element (Size), and shows the amount of memory in pages (Pages) that the total slab is using. This can be helpful when tracking down exactly how the kernel is using its memory.

Listing 3.6.

vmstat provides an easy way to extract large amounts of information about the Linux memory subsystem. When combined with the other information provided on the default output screen, it provides a fair picture of the health and resource usage of the system.

3.2.2. top (2.x and 3.x)

As discussed in the previous chapter, top presents both system-wide and process-specific performance statistics. By default, top presents a list, in decreasing order, of the top CPU-consuming processes, but it can also be adjusted to sort by the total memory usage, enabling you to track down which process is using the most memory.

3.2.2.1 Memory Performance-Related Options

top does not have any special command-line options that manipulate its display of memory statistics. It is invoked with the following command line:

However, once running, top enables you to select whether system-wide memory information displays, and whether processes are sorted by memory usage. Sorting by memory consumption proves particularly useful to determine which process is consuming the most memory. Table 3-3 describes the different memory-related output toggles.

Table 3-3. top Runtime Toggles

Option	Explanation
m	This toggles whether information about the system memory usage will be shown on the screen.
M	Sorts the tasks by the amount of memory they are using. Because processes may have allocated more memory than they are using, this sorts by resident set size. Resident set size is the amount the processes are actually using rather than what they have simply asked for.

Table 3-4 describes the memory performance statistics that top can provide for both the entire system and individual processes. top has two different versions, 2.x and 3.x, which have slightly different names for output statistics. Table 3-4 describes the names for both versions.

Table 3-4. top Memory Performance Statistics

Option	Explanation
%MEM	This is the percentage of the system’s physical memory that this process is using.
SIZE (v 2.x) VIRT (v 3.x)	This is the total size of the process’s virtual memory usage. This includes all the memory that the application has allocated but is not using.
SWAP	This is the amount of swap (in KB) that the process is using.
RSS (v 2.x) RES (v 3.x)	This is the amount of physical memory that the application is actually using.
TRS (v 2.x) CODE (v 3.x)	The total amount of physical memory (in KB) that the process’s executable code is using.
DSIZE (v 2.x) DATA (v 3.x)	The total amount of memory (in KB) dedicated to a process’s data and stack.
SHARE (v 2.x) SHR (v 3.x)	The total amount of memory (in KB) that can be shared with other processors.
D (v 2.x) nDRT (v 3.x)	The number of pages that are dirty and need to be flushed to disk.
Mem: total, used, free	Of the physical memory, this indicates the total amount, the used amount, and the free amount.
swap: total, used, free	Of the swap, this indicates the total amount, the used amount, and the free amount.
active (v 2.x)	The amount of physical memory currently active.
inactive (v 2.x)	The amount of physical memory that is inactive and hasn’t been used in a while.
buffers	The total amount of physical memory (in KB) used to buffer values to be written to disk.

top provides a large amount of memory information about the different running processes. As discussed in later chapters, you can use this information to determine exactly how an application allocates and uses memory.

3.2.2.2 Example Usage

Listing 3.7 is similar to the example run of top shown in the previous chapter. However, in this example, notice that in the buffers, we have a total amount of 1,024MB of physical memory, of which ~84MB is free.

Listing 3.7.

Again, top can be customized to display only what you are interested in observing. Listing 3.8 shows a highly configured output screen that shows only memory performance statistics.

Listing 3.8.

top provides a real-time update of memory statistics and a display of which processes are using which types of memory. This information becomes useful as we investigate application memory usage.

3.2.3. procinfo II

As we’ve seen before, procinfo provides a view of the system-wide performance characteristics. In addition to the statistics described in the previous chapter, procinfo provides a few memory statistics, similar to vmstat and top, that indicate how the current memory is being used.

3.2.3.1 Memory Performance-Related Options

procinfo does not have any options that change the output of the memory statistics displayed and, as a result, is invoked with the following command:

procinfo displays the basic memory system memory statistics, similar to top and vmstat. These are shown in Table 3-5.

Table 3-5. procinfo CPU Statistics

Option	Explanation
Total	This is the total amount of physical memory.
Used	This is the amount of physical memory in use.
Free	This is the amount of unused physical memory.
Shared	This is an obsolete value and should be ignored.
Buffers	This is the amount of physical memory used as buffers for disk writes.
Page in	The number of blocks (usually 1KB) read from disk. (This is broken on 2.6.x kernels.)
Page out	The number of blocks (usually 1KB) written to disk. (This is broken on 2.6.x kernels.)
Swap in	The number of memory pages read in from swap. (This statistic is broken on 2.6.x kernels.)
Swap out	The number of memory pages written to swap. (This statistic is broken on 2.6.x kernels.)

Much like vmstat or top, procinfo is a low-overhead command that is good to leave running in a console or window on the screen. It gives a good indication of a system’s health and performance.

3.2.3.2 Example Usage

Listing 3.9 is a typical output for procinfo. As you can see, it reports summary information about how the system is using virtual memory. In this case, the system has a total of 312MB of memory; 301MB is in use by the kernel and applications, 11MB is used by system buffers, and 11MB is not used at all.

Listing 3.9.

procinfo provides information about system performance in a single screen of information. Although it provides a few of the important statistics for memory, vmstat or top is much better suited for investigating system-wide memory usage.

3.2.4. gnome-system-monitor (II)

gnome-system-monitor is in many ways a graphical counterpart of top. It enables you to monitor individual process and observe the load on the system based on the graphs that it displays. It also provides rudimentary graphs of CPU and memory usage.

3.2.4.1 Memory Performance-Related Options

gnome-system-monitor can be invoked from the Gnome menu. (Under Red Hat 9 and higher, this is in System Tools > System Monitor.) However, it can also be invoked using the following command:

gnome-system-monitor has no relevant command-line options that affect the memory performance measurements.

3.2.4.2 Example Usage

When you launch gnome-system-monitor and select the System Monitor tab, you see a window similar to Figure 3-1. This window enables you to glance at the graph and see how much physical memory and swap is currently being used, and how its usage has changed over time. In this case, we see that 969MB of a total of 1,007MB is currently being used. The memory usage has been relatively flat for a while.

 

Figure 3-1.

 

The graphical view of data provided by gnome-system-monitor can make it easier and faster; however, most of the details, such as how the memory is being used, are missing.

3.2.5. free

free provides an overall view of how your system is using your memory, including the amount of free memory. Although the free command may show that a particular system does not have much free memory, this is not necessarily bad. Instead of letting memory sit unused, the Linux kernel puts it to work as a cache for disk reads and as a buffer for disk writes. This can dramatically increase system performance. Because these cache and buffers can always be discarded and the memory can be used if needed by applications, free shows you the amount of free memory plus or minus these buffers.

3.2.5.1 Memory Performance-Related Options

free can be invoked using the following command line:

Table 3-6 describes the parameters that modify the types of statistics that free displays. Much like vmstat, free can periodically display updated memory statistics.

Table 3-6. free Command-Line Options

Option	Explanation
-s delay	This option causes free to print out new memory statistics every delay seconds.
-c count	This option causes free to print out new statistics for count times.
-l	This option shows you how much high memory and how much low memory are being used.

free actually displays some of the most complete memory statistics of any of the memory statistic tools. The statistics that it displays are shown in Table 3-7.

Table 3-7. free Memory Statistics

Statistic	Explanation
Total	This is the total amount of physical memory and swap.
Used	This is the amount of physical memory and swap in use.
Free	This is the amount of unused physical memory and swap.
Shared	This is an obsolete value and should be ignored.
Buffers	This is the amount of physical memory used as buffers for disk writes.
Cached	This is the amount of physical memory used as cache for disk reads.
-/+ buffers/cache	In the Used column, this shows the amount of memory that would be used if buffers/cache were not counted as used memory. In the Free column, this shows the amount of memory that would be free if buffers/cache were counted as free memory.
Low	The total amount of low memory or memory directly accessible by the kernel.
High	The total amount of high memory or memory not directly accessible by the kernel.
Totals	This shows the combination of physical memory and swap for the Total, Used, and Free columns.

free provides information about the system-wide memory usage of Linux. It is a fairly complete range of memory statistics.

3.2.5.2 Example Usage

Calling free without any command options gives you an overall view of the memory subsystem.

As mentioned previously, Linux tries to use all the available memory if possible to cache data and applications. In Listing 3.10, free tells us that we are currently using 234,720 bytes of memory; however, if you ignore the buffers and cache, we are only using 122,772 bytes of memory. The opposite is true of the free column. We currently have 150,428 bytes of memory free; if you also count the buffers and cached memory (which you can, because Linux throws away those buffers if the memory is needed), however, we have 262,376 bytes of memory free.

Listing 3.10.

Although you could just total the columns yourself, the -t flag shown in Listing 3.11 tells you the totals when adding both swap and real memory. In this case, the system had 376MB of physical memory and 384MB of swap. The total amount of memory available on the system is 376MB plus 384MB, or ~760MB. The total amount of free memory was 134MB of physical memory plus 259MB of swap, yielding a total of 393MB of free memory.

Listing 3.11.

Finally, free tells you the amount of high and low memory that the system is using. This is mainly useful on 32-bit machines (such as IA32) with 1GB or more of physical memory. (32-bit machines are the only machines that will have high memory.) Listing 3.12 shows a system with a very small amount of free memory, 6MB in total. It shows a system with 876MB of low memory and 640MB of high memory. This system also has much more cached memory than buffer memory, suggesting that it may be aggressively writing data to disk rather than leaving it in the buffer cache a long time.

Listing 3.12.

free gives a good idea of how the system memory is being used. It may take a little while to get used to output format, but it contains all the important memory statistics.

3.2.6. slabtop

slabtop is similar to top, but instead of displaying information about the CPU and memory usage of the processes in the system, slabtop shows in real-time how the kernel is allocating its various caches and how full they are. Internally, the kernel has a series of caches that are made up of one or more slabs. Each slab consists of a set of one or more objects. These objects can be active (or used) or inactive (unused). slabtop shows you the status of the different slabs. It shows you how full they are and how much memory they are using.

3.2.6.1 Memory Performance-Related Options

slabtop is invoked using the following command line:

The command-line options for slabtop are described in Table 3-8.

Table 3-8. slabtop Command-Line Options

Option	Explanation
--delay	This specifies how long slabtop waits between updates.
--sort {order}	This specifies the order of the output. order can be one of the following:
	a	This sorts by the number of active objects in each slab.
	b	This sorts by the total number of objects (active and inactive) in each slab for a particular cache.
	c	This sorts by the total amount of memory each cache is using.
	l	This sorts by the number of slabs used by each cache.
	v	This sorts by the number of active slabs used by each cache.
	n	This sorts by the name of the cache.
	o	This sorts by the number of objects in the particular cache.
	P	This sorts by the number of pages used per slab.
	s	This sorts by the size of the objects in the cache.

slabtop provides a glimpse into the data structures of the Linux kernel. Each of these slab types is tied closed to the Linux kernel, and a description of each of these slabs is beyond the scope of this book. If a particular slab is using a large amount of kernel memory, reading the Linux kernel source code and searching the Web are two great ways to figure out what these slabs are used for.

3.2.6.2 Example Usage

As shown in Listing 3.13, by default, slabtop fills the entire console and continually updates the statistics every three seconds. In this example, you can see that the size-64 slab has the most objects, only half of which are active.

Listing 3.13.

Because the information provided by slabtop is updated periodically, it is a great way to see how the Linux kernel’s memory usage changes in response to different workloads.

3.2.7. sar (II)

All the advantages of using sar as a CPU performance tool, such as the easy recording of samples, extraction to multiple output formats, and time stamping of each sample, are still present when monitoring memory statistics. sarprovides similar information to the other memory statistics tools, such as the current values for the amount of free memory, buffers, cached, and swap amount. However, it also provides the rate at which these value change and provides information about the percentage of physical memory and swap that is currently being consumed.

3.2.7.1 Memory Performance-Related Options

sar can be invoked with the following command line:

By default, sar displays only CPU performance statistics; so to retrieve any of the memory subsystem statistics, you must use the options described in Table 3-9.

Table 3-9. sar Command-Line Options

Option	Explanation
-B	This reports information about the number of blocks that the kernel swapped to and from disk. In addition, for kernel versions after v2.5, it reports information about the number of page faults.
-W	This reports the number of pages of swap that are brought in and out of the system.
-r	This reports information about the memory being used in the system. It includes information about the total free memory, swap, cache, and buffers being used.

sar provides a fairly complete view of the Linux memory subsystem. One advantage that sar provides over other tools is that it shows the rate of change for many of the important values in addition to the absolute values. You can use these values to see exactly how memory usage is changing over time without having to figure out the differences between values at each sample. Table 3-10 shows the memory statistics that sar provides.

Table 3-10. sar Memory Statistics

Statistic	Explanation
pgpgin/s	The amount of memory (in KB) that the kernel paged in from disk.
pgpgout/s	The amount of memory (in KB) that the kernel paged out to disk.
fault/s	The total number of faults that that the memory subsystem needed to fill. These may or may not have required a disk access.
majflt/s	The total number of faults that the memory subsystem needed to fill and required a disk access.
pswpin/s	The amount of swap (in pages) that the system brought into memory.
pswpout/s	The amount of memory (in pages) that the system wrote to swap.
kbmemfree	This is the total physical memory (in KB) that is currently free or not being used.
kbmemused	This is the total amount of physical memory (in KB) currently being used.
%memused	This is the percentage of the total physical memory being used.
kbbuffers	This is the amount of physical memory used as buffers for disk writes.
kbcached	This is the amount of physical memory used as cache for disk reads.
kbswpfree	This is the amount of swap (in KB) currently free.
kbswpused	This is the amount of swap (in KB) currently used.
%swpused	This is the percentage of the swap being used.
kbswpcad	This is memory that is both swapped to disk and present in memory. If the memory is needed, it can be immediately reused because the data is already present in the swap area.
frmpg/s	The rate that the system is freeing memory pages. A negative number means the system is allocating them.
bufpg/s	The rate that the system is using new memory pages as buffers. A negative number means that number of buffers is shrinking, and the system is using less of them.

Although sar misses the high and low memory statistics, it provides nearly every other memory statistic. The fact that it can also record network CPU and disk I/O statistics makes it very powerful.

3.2.7.2 Example Usage

Listing 3.14 shows sar providing information about the current state of the memory subsystem. From the results, we can see that the amount of memory that the system used varies from 98.87 percent to 99.25 percent of the total memory. Over the course of the observation, the amount of free memory drops from 11MB to 7MB. The percentage of swap that is used hovers around ~11 percent. The system has ~266MB of data cache and about 12MB in buffers that can be written to disk.

Listing 3.14.

Listing 3.15 shows that the system is consuming the free memory at a rate of ~82 pages per second during the first sample. Later, it frees ~16 pages, and then consumes ~20 pages. The number pages of buffers grew only once during the observation, at a rate of 2.02 pages per second. Finally, the pages of cache shrunk by 2.02, but in the end, they expanded by 64.36 pages/second.

Listing 3.15.

Listing 3.16 shows that the system wrote ~53 pages of memory to disk during the third sample. The system has a relatively high fault count, which means that memory pages are being allocated and used. Fortunately, none of these are major faults, meaning that the system does not have to go to disk to fulfill the fault.

Listing 3.16.

As you can see, sar is a powerful tool that enhances the functionality of the other system memory performance tools by adding the ability to archive, time stamp, and simultaneously collect many different types of statistics.

3.2.8. /proc/meminfo

The Linux kernel provides a user-readable text file called /proc/meminfo that displays current system-wide memory performance statistics. It provides a superset of the system-wide memory statistics that can be acquired with vmstat,top, free, and procinfo; but it is a little more difficult to use. If you want periodic updates, you must write a script or some code for such. If you want to store the memory performance information or coordinate it with CPU statistics, you must create a new tool or write a script. However, /proc/meminfo provides the most complete view of system memory usage.

3.2.8.1 Memory Performance-Related Options

The information in /proc/meminfo can be retrieved with the following command line:

This command displays the array of statistics described in Table 3-11.

Table 3-11. /proc/meminfo Memory Statistics (All in KB)

Statistic	Explanation
MemTotal	The total amount of system physical memory.
MemFree	The total amount of free physical memory.
Buffers	The amount of memory used for pending disk writes.
Cached	The amount of memory used to cache disk reads.
SwapCached	The amount of memory that exists in both the swap and the physical memory.
Active	The amount of memory currently active in the system.
Inactive	The amount of memory currently inactive and a candidate for swap.
HighTotal	The amount of high memory (in KB).
HighFree	The amount of free high memory (in KB).
LowTotal	The amount of low memory (in KB).
LowFree	The amount of free low memory (in KB).
SwapTotal	The amount of swap memory (in KB).
SwapFree	The amount of free swap memory (in KB).
Dirty	Memory waiting to be written to disk.
Writeback	Memory currently being written to disk.
Mapped	The total amount of memory brought into a process’s virtual address space using mmap.
Slab	The total amount of kernel slab memory (in KB).
Committed_AS	The amount of memory required to almost never run out of memory with the current workload. Normally, the kernel hands out more memory than it actually has with the hope that applications overallocate. If all the applications were to use what they allocate, this is the amount of physical memory you would need.
PageTables	The amount of memory reserved for kernel page tables.
VmallocTotal	The amount of kernel memory usable for vmalloc.
VmallocUsed	The amount of kernel memory used for vmalloc.
VmallocChunk	The largest contiguous chunk of vmalloc‘able memory.
HugePages_Total	The total size of all HugePages.
HugePages_Free	The total amount of free HugePages.

/proc/meminfo provides a wealth of information about the current state of the Linux memory subsystem.

3.8.2.2 Example Usage

Listing 3.17 shows an example output of /proc/meminfo. It shows some memory statistics that are similar to those we have seen with other tools. However, some statistics show new information. One, Dirty, shows that the system currently has 24KB of data waiting to be written to disk. Another, Commited_AS, shows that we need a little more memory (needed: 1068MB versus total: 1024MB) to avoid a potential out-of-memory situation.

Listing 3.17.

/proc/meminfo is the most complete place to gather system-wide Linux memory statistics. Because you can treat it as a text file, any custom script or program can easily extract the statistics.