But, its a good idea to prepare yourself before you start running the repair commands.
So, far I've experienced at least 3 file corruption issues related to QFS. And its a very painful experience.
I just wanted to recommend some tips you should consider if you are dealing with huge file system indicating inconsistency. Frankly QFS samfsck is not very reliable, less self healing and cannot deal with too many inodes repair, again it depends on how much is too much.
In my experience if you are recovering about 10K inodes, consider creating a new file system and restore from backup. The samfsck runs completely and still cannot make the file system clean. It gets into the loop to repair the inode number again and again even if it has marked it damaged. Sometimes it will wipe off the entire lost+found.
If you cannot get a clean file system after samfsck, consider not using it.
Here are some estimates:
10TG file system with 7.2TB of data, it takes about 8hrs. to complete.
Inodes processed: 10983424
total data kilobytes = 10737376960
total data kilobytes free = 2979166784
INFO: FS online_data repaired:
start: Fri May 03 23:18:27 2013
finish: Sat May 04 06:56:03 2013
Oracle support will say they have never seen this problem before and you have run sammkfs and restore.
You may have run samfsck multiple times to repair the file system.
Every time you run samfsck.
# umount
#Make sure the file system you are repairing has a clean lost+found and big enough to hold all corrupted inodes. In my experience if you are dealing with, let's say 5-10TB file system, it takes about 6-8 hrs. to scan thru all the inode numbers and then complains at the end that lost+found is not big enough. Please rerun samfsck again.
Here is the step
cd /filesystem/lost+found
N=0
while [ $N -lt 102400 ]; do
touch TMPFILE$N
N=`expr $N + 1`
done
rm TMPFILE*
It will create more than 100K files, and then rm will fail.
We have to manually remove all the TMPFILE files from lost+found, If you do this step correctly it will save you lot of time. By default it can handle around 28mb.
With this step lost+found can handle around 250MB
# /opt/SUNWsamfs/sbin/samfsck -F -V
# After samfsck completes, try to mount
# Keep monitoring QFS logs /var/adm/messages or configured otherwise for error related to SAM QFS .
-----------------------------------------------------------------------------------------------------
Oracle KB 1006526.1 describes how to increase the lost+found size
-----------------------------------------------------------------------------------------------------
Sun QFS and Storage Archive Manager (SAM): samfsck Complains "Orphan processing stopped due to full lost+found" [ID 1006526.1]
Applies to:
Sun QFS and Storage Archive Manager (SAM) - Version 4.0 and later
All Platforms
***Checked for relevance on 18-Nov-2011***
Symptoms:
When attempting to repair your SAM-FS or SAM-QFS file system with samfsck(1M), it complains it is unable to process all of the orphan inodes due to a full lost+found directory.
Ex :
samfsck: NOTICE: Filesystem samfs2 requires fsck
name: samfs2 version: 2
First pass
Second pass
Third pass
NOTICE: ino 2155175.3, Repaired link count from 4 to 1
NOTICE: Orphan ino: ino 2155175 moved to lost+found
NOTICE: Orphan ino: ino 2155404 moved to lost+found
NOTICE: Orphan ino: ino 2155689 moved to lost+found
...
NOTICE: Orphan ino: ino 2155852 moved to lost+found
NOTICE: Orphan ino: ino 2155853 moved to lost+found
NOTICE: Orphan processing stopped due to full lost+found.
Increase lost+found and rerun.
NOTICE: ino 9717769.5, Repaired link count from 0 to 8
NOTICE: ino 9718942.3, Repaired link count from 9 to 10
...
NOTICE: ino 9718949.1, Repaired link count from 0 to 1
NOTICE: ino 9718950.1, Repaired link count from 0 to 1
Inodes processed: 9808384
total data kilobytes = 272121856
total data kilobytes free = 77938544
NOTICE: Reclaimed 8126464 bytes
Cause
During normal operation, when directories are created within a SAM-QFS file system, it is able to accommodate X number of files.
When the X+1 file is created, the directory size is expanded to accommodate an additional X files (X+X). And this trend continues. Since samfsck(1M) is performing fixes on the file system while it is unmounted, it's unable to 'expand' the directory to make room, so if the existing space is filled up, it cannot proceed. As such, it may become necessary to create additional file entries within the directory when the file system is online, and remove these files to make room for orphan inodes.
As files are removed, the directory size is not decreased. Depending on the number of orphans, you may have to use a value larger than 1,024. Please adjust this accordingly. The 'samfsck -V' command (without the -F option) can be used to find out how many orphan inodes exist. This command can be used while the file system is mounted or unmounted. Once these commands are executed, the lost+found directory should have sufficient room for the orphan inodes processed by samfsck(1M).
Solution
The following is taken from the samfsck(1M) man page:
If there are files encountered that are not attached to a
parent directory, they will be moved to the
/mount_point/lost+found directory. If this directory does
not exist, you must create this directory first and make it
sufficently large to hold the expected number of discon-
nected files if you wish this to happen. Here is how to do
this in the Bourne shell for a SAM file system mounted on
/sam:
/bin/mkdir /sam/lost+found
cd /sam/lost+found
N=0
while [ $N -lt 1024 ]; do
touch TMPFILE$N
N=`expr $N + 1`
done
rm TMPFILE*
Additional Information
Directory Size (bytes) Maximum number of files
4096 144 (initial size)
8192 289 (+145)
12288 434 (+145)
16384 579 (+145)
The directory size will be (N/145 + 1)*4096 bytes, where N is the number of files in the directory and integer division is used(no fraction).
For example, for 870 files:
(870/145 + 1)*4096 (6 + 1)*4096 = 28672 bytes
No comments:
Post a Comment