byobu - new funky wrapper for screen command

screen command was always been a very helpful tool for system admins.

byobu is a wrapper script over screen.

Best part is that it looks good.

You can also configure status notifications which will be displayed in the bottom screen. This is useful in monitoring the system while working in the terminal.

In Fedora, you can install byobu using following command.

# yum install byobu

Some useful shortcuts from man byobu

F2 - Create a new window

F3 - Move to previous window

F4 - Move to next window

F5 - Reload profile

F6 - Detach from this session

F7 - Enter copy/scrollback mode

F8 - Re-title a window

F9 - Configuration Menu

F12 - Lock this terminal

Run 'byobu' command to start the script.

How do you find out which cpu a process is running on?

The output of the ps command can be changed to a user defined format by using the -o option.

The following command can be used to display the CPU and the processes assigned.

# ps -eo pid,args,psr

Options:

pid - The process ID of the process.

args - Command with all its arguments as a string.

psr - processor currently assigned to the process.

Go to the man pages for more information on the ps command.

What is the logic behind killing processes during an Out of Memory situation?

As per kernel source code, following is OOM-killer logic,

A function called badness() is defined to calculate points for each processes.

* Points will be added to following processes.

Processes with high memory size.
Niced processes.

* Points will be reduced from following processes.

Processes which were running for long time.
Processes which were started by superusers.
Process with direct hardware access.

The process with the highest number of point, will be killed, unless it is already in the midst of freeing up memory on its own.

Then the system will wait for sometime to see if enough memory is freed. If enough memory is not freed after killing one process, the above steps will continue.

As per select_bad_process function, if a processes is having 0 or less points it could not be killed. The oom kills will be continued until there is no candidate processes left to kill. If the system is not able to find a candidate process to kill, it panics.

static unsigned long badness(struct task_struct *p, unsigned long uptime)
{
unsigned long points, cpu_time, run_time, s;

if (!p->mm)
return 0;

if (p->flags & PF_MEMDIE)
return 0;
/*
* The memory size of the process is the basis for the badness.
*/
points = p->mm->total_vm;

/*
* CPU time is in tens of seconds and run time is in thousands
* of seconds. There is no particular reason for this other than
* that it turned out to work very well in practice.
*/
cpu_time = (p->utime + p->stime) >> (SHIFT_HZ + 3);

if (uptime >= p->start_time.tv_sec)
run_time = (uptime - p->start_time.tv_sec) >> 10;
else
run_time = 0;

s = int_sqrt(cpu_time);
if (s)
points /= s;
s = int_sqrt(int_sqrt(run_time));
if (s)
points /= s;

/*
* Niced processes are most likely less important, so double
* their badness points.
*/
*/
if (task_nice(p) > 0)
points *= 2;

/*
* Superuser processes are usually more important, so we make it
* less likely that we kill those.
*/
if (cap_t(p->cap_effective) & CAP_TO_MASK(CAP_SYS_ADMIN) ||
p->uid == 0 || p->euid == 0)
points /= 4;

/*
* We don't want to kill a process with direct hardware access.
* Not only could that mess up the hardware, but usually users
* tend to only have this flag set on applications they think
* of as important.
*/
if (cap_t(p->cap_effective) & CAP_TO_MASK(CAP_SYS_RAWIO))
points /= 4;
#ifdef DEBUG
printk(KERN_DEBUG "OOMkill: task %d (%s) got %d points\n",
p->pid, p->comm, points);
#endif
return points;
}

What is kipmi? Why is it taking too much cpu in my Red Hat Enterprise Linux system?

If your IPMI hardware interface does not support interrupts and is a KCS or SMIC interface, the IPMI driver will start a kernel thread for the interface to help speed things up. This is a low-priority kernel thread that constantly polls the IPMI driver while an IPMI operation is in progress.

kipmi is that low-priority kernel thread. If the system does large number of IPMI operations, there is a possibility of kipmi using too much cpu time.

The kipmi thread is a workaround to fix the hardware disability so there is no real fix. Sometimes firmware updates can fix this though.

As a workaround, you can disable kipmi using following steps. This may decrease the speed of IPMI operations though.
* Edit /etc/modprobe.conf and add following entry.

options ipmi_si force_kipmid=0

* Restart the ipmi using following command.

# service ipmi restart

skype and twitux

Tested Skype and Twitux in my Fedora 11 X86_64 system.

Used following repo for skype..

[skype]
name=Skype Repository
baseurl=http://download.skype.com/linux/repos/fedora/updates/i586/
enabled=1
gpgkey=http://www.skype.com/products/skype/linux/rpm-public-key.asc
gpgcheck=0

Twitux is a gnome client for twitter. Simple and nice..
http://sourceforge.net/projects/twitux/
http://twitter.com

tape tricks

* display contents of a tape

# tar tvf /dev/tape

* tape rewind

# mt -f /dev/tape rewind

High memory utilization and cache

Many people complains about high memory utilization in the output of free command, which in most situations caused by misreading of free command.

If we are taking following free output for instance,

total used free shared buffers cached
Mem: 8054896 7193712 861184 0 188348 5286732
-/+ buffers/cache: 1718632 6336264
Swap: 2096440 144 2096296

It is seen that most of the memory in the system is used by 'cached' (5286732kb) and buffers (188348kb). The system over time increases the size of the buffers and cache in its RAM. The cache stores various pieces of data from the hard disk such as data from files. When reading from the hard disk, the kernel first looks into cache for that data. This is more efficient, as reading from memory is much faster than reading from the disk. This memory is returned whenever it is required by other programs and should be calculated as being free along with the buffers.

This is an expected behaviour by Linux kernel to speed up the system.

Memory utilization need not be an issue unless there is an Out of Memory or a performance issue.

--

In RHEL5, If cache memory need to be flushed /proc/sys/vm/drop_caches could be used.To free pagecache:

# echo 1 > /proc/sys/vm/drop_caches

To free dentries and inodes:

# echo 2 > /proc/sys/vm/drop_caches

To free pagecache, dentries and inodes:

# echo 3 > /proc/sys/vm/drop_caches

This could cause a hike in I/O in some situations, as kernel will try to write all data to disk.

"sync" command should be run before the above commands.

In RHEL4 /proc/sys/vm/drop_caches is introduced in 2.6.9-67* kernel.

There is also a known bug in which the system hangs when cache is flushed using drop_caches. The bug is only reproduced in large systems with at least 64GB RAM and 8 cpus. The bug is fixed in 2.6.9-78* kernel. Please refer following bugzilla.

https://bugzilla.redhat.com/show_bug.cgi?id=449381

--

Cached memory is good for most of the work loads.

If you want to specifically opt for less cache usage, increasing the value of /proc/sys/vm/swappiness could help.

For Example:

# echo 100 > /proc/sys/vm/swappiness

You can make the values persistent over reboot by adding following to /etc/sysctl.conf.

vm.swappiness=

Please remember that this will increase the usage of swap, which might affect system performance.

--

Following are some other kernel tuning parameters that helps to manage cache.

vm.dirty_expire_centisecs=2000

It determines the amount of time for a page to be considered old enough for flushing to disk via the pdflush daemon. Expressed in 100'ths of a second.

vm.dirty_writeback_centisecs=400

It determines the interval at which the pdflush daemon wakes up to write dirty data to disk.. Expressed in 100'ths of a second.

vm.dirty_background_ratio=5

It determines the percentage of total system memory at which the pdflush background writeback daemon will start writing out dirty data.

vm.dirty_ratio=20

It determines percentage of total system memory at which a process which is generating disk writes will itself start writing out dirty
data.