Monday, June 8, 2009

High memory utilization and cache

Many people complains about high memory utilization in the output of free command, which in most situations caused by misreading of free command.

If we are taking following free output for instance,



total used free shared buffers cached
Mem: 8054896 7193712 861184 0 188348 5286732
-/+ buffers/cache: 1718632 6336264
Swap: 2096440 144 2096296



It is seen that most of the memory in the system is used by 'cached' (5286732kb) and buffers (188348kb). The system over time increases the size of the buffers and cache in its RAM. The cache stores various pieces of data from the hard disk such as data from files. When reading from the hard disk, the kernel first looks into cache for that data. This is more efficient, as reading from memory is much faster than reading from the disk. This memory is returned whenever it is required by other programs and should be calculated as being free along with the buffers.

This is an expected behaviour by Linux kernel to speed up the system.

Memory utilization need not be an issue unless there is an Out of Memory or a performance issue.

--

In RHEL5, If cache memory need to be flushed /proc/sys/vm/drop_caches could be used.To free pagecache:


# echo 1 > /proc/sys/vm/drop_caches


To free dentries and inodes:

# echo 2 > /proc/sys/vm/drop_caches


To free pagecache, dentries and inodes:

# echo 3 > /proc/sys/vm/drop_caches


This could cause a hike in I/O in some situations, as kernel will try to write all data to disk.

"sync" command should be run before the above commands.

In RHEL4 /proc/sys/vm/drop_caches is introduced in 2.6.9-67* kernel.

There is also a known bug in which the system hangs when cache is flushed using drop_caches. The bug is only reproduced in large systems with at least 64GB RAM and 8 cpus. The bug is fixed in 2.6.9-78* kernel. Please refer following bugzilla.

https://bugzilla.redhat.com/show_bug.cgi?id=449381

--

Cached memory is good for most of the work loads.

If you want to specifically opt for less cache usage, increasing the value of /proc/sys/vm/swappiness could help.

For Example:

# echo 100 > /proc/sys/vm/swappiness


You can make the values persistent over reboot by adding following to /etc/sysctl.conf.

vm.swappiness=


Please remember that this will increase the usage of swap, which might affect system performance.

--

Following are some other kernel tuning parameters that helps to manage cache.

vm.dirty_expire_centisecs=2000


It determines the amount of time for a page to be considered old enough for flushing to disk via the pdflush daemon. Expressed in 100'ths of a second.

vm.dirty_writeback_centisecs=400


It determines the interval at which the pdflush daemon wakes up to write dirty data to disk.. Expressed in 100'ths of a second.


vm.dirty_background_ratio=5


It determines the percentage of total system memory at which the pdflush background writeback daemon will start writing out dirty data.


vm.dirty_ratio=20


It determines percentage of total system memory at which a process which is generating disk writes will itself start writing out dirty
data.

kernel panic on boot and lvm filters

Problem:

Updated the RHEL5 system to 2.6.18-92.1.13.e15 kernel. When he rebooted the system, system panicked with following errors.


Unable to access resume device (/dev/VolGroup00/LogVol01)
mount: could not find filesystem '/dev/root'
setuproot: moving /dev failed: No such file or directory
switchroot: mount failed: No such file or directory
Kernel panic - not syncing: Attempted to kill init!


Troubleshooting:

I saw / mounted on lvm.

/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)


But in the 'vgdisplay' command output, I was not able to see the logical volume !!!!

Interestingly following entry was in /etc/lvm/lvm.conf.

filter = [ "r/disk/", "r/sd./", "a/./" ]


The filter will exclude all /dev/disk/* and /dev/sd* devices.


Solution:

Edit /etc/lvm/lvm.conf and replace following entry

filter = [ "r/disk/", "r/sd.*/", "a/.*/" ]


with

filter = [ "a/dev/mapper/.*/", "a/dev/sda2$/", "r/.*/" ]


This filter will enable all /dev/mpath/* devices and /dev/sda2 device where the / was mounted.

* Clear /etc/lvm/.cache.

# > /etc/lvm/.cache


* Run vgscan command

# vgscan


* Reinstall the latest kernel rpm.

# rpm -e kernel-2.6.18-128.1.10.el5
# yum install kernel


After following the suggestions, the system booted fine with the new kernel.

Rebuilding the initrd would have also worked.

How can I improve the performance of the syslog daemon in Red Hat Enterprise Linux?

http://kbase.redhat.com/faq/docs/DOC-17098