Linux memory utilitization and disk caching

People often come to me and say, we are running out of memory or I'm not sure who is using all the memory on this box especially on a linux box.

I think, it confuses people more when they run (top command) and see 90% of the system memory is in use.

Here are some of mis-conception/interpretation of results we see in top command.
Snapshot of top command:
top - 10:20:43 up 12 days, 23:08, 2 users, load average: 0.06, 0.21, 0.18
Tasks: 90 total, 1 running, 89 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.8% us, 0.1% sy, 0.0% ni, 98.8% id, 0.1% wa, 0.1% hi, 0.1% si
Mem: 16414396k total, 10941732k used, 5472664k free, 261608k buffers
Swap: 8388600k total, 0k used, 8388600k free, 7092524k cached
21900 app_user 16 0 5976m 2.0g 80m S 22 12.5 931:56.60 WebSphere
21964 app_user 16 0 785m 675m 4236 S 16 4.2 107:04.09 mysqld

Total memory: 16GB
Used memory: 10GB+
Free memory:5GB+

In the above example, Out of 16GB, all 10GB is not used by the application.
It also includes all memory that the kernel uses for block caching and stuff.

Here's the explaination of caching and how it works by one linux guru:
Suppose that you run "ls /". What the kernel does then (after some processing), is that it requests the hard disk sectors that make up the root directory, and then it makes the ls process sleep until those sectors are received. While the hard drive is looking up those sectors, the system does some other things that don't require hard drive access. The hard drive then puts the sectors that it found in your RAM, and reports back to the kernel that the operation completed. Then the kernel continues with the ls process, which outputs what you wanted to see. After that, the kernel has two choices:

1. It could discard the sectors loaded from the hard drive.
2. It could keep the sectors in memory.

Now, if we look carefully at option number 1, which at the first glance might seem like a good thing, since it apparently makes that memory usable again, we see that it would require those sectors to be loaded once again whenever a process wants to look in the root directory. Since many files that are opened are opened with absolute path names, there are many root directory accesses in the system. If the sectors would have to be loaded each time a process accesses the root directory, the system would be slow, since the hard drive is slow. Therefore, the kernel keeps the sectors in memory. The memory used this way is what you see in the "cached" entry. That allows for extremely fast disk access, since all sectors that have once been accessed don't have to be reloaded from the physical disk.

Now, you might think that that's an extreme waste of memory, but if you do, you're very wrong. You see, the block caches are low-priority pages, which essentially means that they are free to be used be processes, should they need the memory. Every time a process needs an extra memory page and there isn't already one free, a page used for block buffering is freed by the kernel and put into the process' addressing space for the process to use in whatever way it see fit. Together with some intelligently designed algorithms for choosing which cache page to be freed, this all allows for really fast disk access.

Now, that we figured out the total 10GB in use also includes kernel caching.
The question still remains about what is the actual application memory usage?

Try (free commands)
free -m -t
free -m -s 5
free -g -t
total used free shared buffers cached
Mem: 15 10 5 0 0 6
-/+ buffers/cache: 3 12
Swap: 7 0 7
Total: 23 10 13

We still see total 15/16GB, and 10GB in use.

You will notice, that the top row 'used' (10) value will almost always nearly match the top row mem 'total' value (15). Since Linux likes to use any spare memory to cache disk blocks shown in top row 'cached' value (6).

The key used figure to look at is the buffers/cache row used value (3). This is how much space your applications are currently using. For best performance, this number should be less than your total (15) memory. To prevent out of memory errors, it needs to be less than the total memory (15) and swap space (7).

If you wish to quickly see how much memory is free look at the buffers/cache row free value (12).
This is the total memory (15)
The actual used (3)
Free memory (15 - 3) = 12

Ok, Now we also know that actually the application is only using 3GB out of 16GB, instead of 10GB reported by top. We may need to find out who is using that 3GB memory.

Try ps command
ps aux
ps aux | awk '{print $4"\t"$1"\t"$11}' | sort -nr

snapshot of ps:
%MEM USER COMMAND
12.6 app_user startServer.sh
4.2 app_user /data/App_Server/mysql/bin/mysqld

To see only the memory resources occupied by each category of processes, such as Apache httpd, MySQL mysqld or Java, use the following command:

ps aux | awk '{print $4"\t"$11}' | sort | uniq -c | awk '{print $2" "$1" "$3}' | sort -nr

Now, if we calculate again. We have startServer.sh using 12.6% of 16GB is 2.01GB.
Similarly, mysql is taking about 4.2% of 16GB, is 670MB, as shown above by the top command.