Friday, November 22, 2013

Recovering from QFS corruption, prepare before you run samfsck on huge file system

Whenever you face a file system corruption issue, we probably know what commands to run to fix the corruption.
But, its a good idea to prepare yourself before you start running the repair commands.

So, far I've experienced at least 3 file corruption issues related to QFS. And its a very painful experience.

I just wanted to recommend some tips you should consider if you are dealing with huge file system indicating inconsistency. Frankly QFS samfsck is not very reliable, less self healing and cannot deal with too many inodes repair, again it depends on how much is too much.

In my experience if you are recovering about 10K inodes, consider creating a new file system and restore from backup. The samfsck runs completely and still cannot make the file system clean. It gets into the loop to repair the inode number again and again even if it has marked it damaged. Sometimes it will wipe off the entire lost+found.

If you cannot get a clean file system after samfsck, consider not using it.

Here are some estimates:
10TG file system with 7.2TB of data, it takes about 8hrs. to complete.


Inodes processed: 10983424

total data kilobytes       = 10737376960
total data kilobytes free  = 2979166784
INFO:  FS online_data repaired:
        start:  Fri May 03 23:18:27 2013
        finish: Sat May 04 06:56:03 2013


Oracle support will say they have never seen this problem before and you have run sammkfs and restore.

You may have run samfsck multiple times to repair the file system.

Every time you run samfsck.

# umount

#Make sure the file system you are repairing has a clean lost+found and big enough to hold all corrupted inodes. In my experience if you are dealing with, let's say 5-10TB file system, it takes about 6-8 hrs. to scan thru all the inode numbers and then complains at the end that lost+found is not big enough. Please rerun samfsck again.

Here is the step
cd /filesystem/lost+found

N=0
while [ $N -lt 102400 ]; do
touch TMPFILE$N
N=`expr $N + 1`
done
rm TMPFILE*


It will create more than 100K files, and then rm will fail.

We have to manually remove all the TMPFILE files from lost+found, If you do this step correctly it will save you lot of time. By default it can handle around 28mb. 

With this step lost+found can handle around 250MB

/opt/SUNWsamfs/sbin/samfsck -F -V | tee /var/tmp/samfsck.`date '+%Y%m%d.%H%M%S'`
# After samfsck completes, try to mount and try to write and read from the file system to confirm data
Keep monitoring QFS logs /var/adm/messages or configured otherwise for error related to SAM QFS .

-----------------------------------------------------------------------------------------------------
Oracle KB 1006526.1 describes how to increase the lost+found size
-----------------------------------------------------------------------------------------------------

Sun QFS and Storage Archive Manager (SAM): samfsck Complains "Orphan processing stopped due to full lost+found" [ID 1006526.1]

Applies to:

Sun QFS and Storage Archive Manager (SAM) - Version 4.0 and later
All Platforms
***Checked for relevance on 18-Nov-2011***


Symptoms:
When attempting to repair your SAM-FS or SAM-QFS file system with samfsck(1M), it complains it is unable to process all of the orphan inodes due to a full lost+found directory.
Ex :

samfsck: NOTICE: Filesystem samfs2 requires fsck
name:     samfs2       version:     2
First pass
Second pass
Third pass
NOTICE: ino 2155175.3,  Repaired link count from 4 to 1
NOTICE: Orphan ino:        ino 2155175 moved to lost+found
NOTICE: Orphan ino:        ino 2155404 moved to lost+found
NOTICE: Orphan ino:        ino 2155689 moved to lost+found
...
NOTICE: Orphan ino:        ino 2155852 moved to lost+found
NOTICE: Orphan ino:        ino 2155853 moved to lost+found
NOTICE: Orphan processing stopped due to full lost+found.
Increase lost+found and rerun.
NOTICE: ino 9717769.5,  Repaired link count from 0 to 8
NOTICE: ino 9718942.3,  Repaired link count from 9 to 10
...
NOTICE: ino 9718949.1,  Repaired link count from 0 to 1
NOTICE: ino 9718950.1,  Repaired link count from 0 to 1
Inodes processed: 9808384
total data kilobytes       = 272121856
total data kilobytes free  = 77938544
NOTICE: Reclaimed 8126464 bytes

Cause

During normal operation, when directories are created within a SAM-QFS file system, it is able to accommodate X number of files.

When the X+1 file is created, the directory size is expanded to accommodate an additional X files (X+X). And this trend continues. Since samfsck(1M) is performing fixes on the file system while it is unmounted, it's unable to 'expand' the directory to make room, so if the existing space is filled up, it cannot proceed. As such, it may become necessary to create additional file entries within the directory when the file system is online, and remove these files to make room for orphan inodes.

As files are removed, the directory size is not decreased. Depending on the number of orphans, you may have to use a value larger than 1,024. Please adjust this accordingly. The 'samfsck -V' command (without the  -F option) can be used to find out how many orphan inodes exist. This command can be used while the file system is mounted or unmounted. Once these commands are executed, the lost+found directory should have sufficient room for the orphan inodes processed by samfsck(1M).

Solution
The following is taken from the samfsck(1M) man page:

     If there are files encountered that are not  attached  to  a
     parent    directory,    they    will   be   moved   to   the
     /mount_point/lost+found directory.  If this  directory  does
     not  exist, you must create this directory first and make it
     sufficently large to hold the  expected  number  of  discon-
     nected  files if you wish this to happen.  Here is how to do
     this in the Bourne shell for a SAM file  system  mounted  on
     /sam:

     /bin/mkdir /sam/lost+found
     cd /sam/lost+found
     N=0
     while [ $N -lt 1024 ]; do
         touch TMPFILE$N
         N=`expr $N + 1`
     done
     rm TMPFILE*

Additional Information

Directory Size (bytes) Maximum number of files
4096 144 (initial size)
8192 289 (+145)
12288 434 (+145)
16384 579 (+145)

The directory size will be (N/145 + 1)*4096 bytes, where N is the number of files in the directory and integer division is used(no fraction).

For example, for 870 files:
(870/145 + 1)*4096 (6 + 1)*4096 = 28672 bytes

Error sam_putapage: sparse block

Errors in /var/adm/messages:

Mar 28 18:31:12 dam-app1 samfs: [ID 247537 kern.warning] WARNING: SAM-QFS: online_3par: sam_putapage: sparse block, ip=30087ca1568, ino=9820091.1, pp=7001 03acc00, off=0
Mar 28 18:31:12 dam-app1 samfs: [ID 756621 kern.warning] WARNING: SAM-QFS: online_3par: inode 0x95d7bb has a SAM-QFS: sam_putapage: sparse block error


File system message indicating that there is an error in flushing disk blocks for the given inode. This is sometimes seen from a client node that is unable to flush its disk block pages because the server claims there is a problem with the range.

This is not always a file system problem, it sometimes indicates that the client has stale cached information and the server is blocking the request. If this is happening on a MDS system, then it is possible it indicates a file system corruption problem.

If on a client, the file system may have to be forcibly unmounted, when the file system is remounted on the client, the correct information is obtained from the server and these messages should go away.
If these messages are on the MDS server, a samfsck would be recommended.