Tuesday, December 23, 2008

Setting up Solaris RAID 5 metadevice using solstice vol mgr

Setting up Solaris RAID 5 metadevice

These are available 7 slices of 926GB each:
3. c3t5006048AD5312D07d34
/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0/ssd@w5006048ad5312d07,22
4. c3t5006048AD5312D07d35
/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0/ssd@w5006048ad5312d07,23
5. c3t5006048AD5312D07d36
/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0/ssd@w5006048ad5312d07,24
6. c3t5006048AD5312D07d37
/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0/ssd@w5006048ad5312d07,25
7. c3t5006048AD5312D07d38
/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0/ssd@w5006048ad5312d07,26
8. c3t5006048AD5312D07d39
/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0/ssd@w5006048ad5312d07,27
9. c3t5006048AD5312D07d40
/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0/ssd@w5006048ad5312d07,28


Each device looks like this:
partition> p
Current partition table (original):
Total disk cylinders available: 65533 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks
0 root wm 0 - 8 130.39MB (9/0/0) 267030
1 swap wu 9 - 17 130.39MB (9/0/0) 267030
2 backup wu 0 - 65532 927.15GB (65533/0/0) 1944364110
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 0 0 (0/0/0) 0
6 usr wm 18 - 65532 926.89GB (65515/0/0) 1943830050
7 unassigned wm 0 0 (0/0/0) 0


Slice 0 has 130M for metadb
And then using slice 6 for 926GB


Step one: Creating metadevice database
metadb -a -f -c2 c3t5006048AD5312D07d34s0 c3t5006048AD5312D07d35s0 c3t5006048AD5312D07d36s0 c3t5006048AD5312D07d37s0 c3t5006048AD5312D07d38s0 c3t5006048AD5312D07d39s0 c3t5006048AD5312D07d40s0

This will create a RAID database replicas on all 7 slices listed in the above command. These database contains state and configuration information.
The -a switch tells metadb to attach a new database device, and modifies /etc/system to tell the system to reattach the devices at boot-time.
The -f switch is used to create the initial state database.
The -c switch is used to determine the number of database replicas that will be created on each of the specified slices.

# List the metabd just created:
bash-3.00# metadb -i
flags first blk block count
a u 16 8192 /dev/dsk/c3t5006048AD5312D07d34s0
a u 8208 8192 /dev/dsk/c3t5006048AD5312D07d34s0
a u 16 8192 /dev/dsk/c3t5006048AD5312D07d35s0
a u 8208 8192 /dev/dsk/c3t5006048AD5312D07d35s0
a u 16 8192 /dev/dsk/c3t5006048AD5312D07d36s0
a u 8208 8192 /dev/dsk/c3t5006048AD5312D07d36s0
a u 16 8192 /dev/dsk/c3t5006048AD5312D07d37s0
a u 8208 8192 /dev/dsk/c3t5006048AD5312D07d37s0
a u 16 8192 /dev/dsk/c3t5006048AD5312D07d38s0
a u 8208 8192 /dev/dsk/c3t5006048AD5312D07d38s0
a u 16 8192 /dev/dsk/c3t5006048AD5312D07d39s0
a u 8208 8192 /dev/dsk/c3t5006048AD5312D07d39s0
a u 16 8192 /dev/dsk/c3t5006048AD5312D07d40s0
a u 8208 8192 /dev/dsk/c3t5006048AD5312D07d40s0

Step Two: Creating RAID5 devices
We need to define RAID-5 metadevice by name, slices participating in the metadevice, and stripe width

metainit d6 -r c3t5006048AD5312D07d34s6 c3t5006048AD5312D07d35s6 c3t5006048AD5312D07d36s6 c3t5006048AD5312D07d37s6 c3t5006048AD5312D07d38s6 c3t5006048AD5312D07d39s6 c3t5006048AD5312D07d40s6 -i 65k

>>output
d6: RAID is setup


In this example, d6 is the name of the volume, -r designates this metadevice as a RAID-5 metadevice, and the parameters are the slices that participate in the metadevice. In this case, we’ve chosen slice 6 on each disk. One could choose other slices, so long as each slice is the same size. -i 65k defines a stripe interlace size of 65KB (the default is 16KB), which will give us a stripe size of 260KB (7 disks * 65KB per disk). Others have empirically determined that 256KB-512KB is the optimum stripe width for a general-purpose RAID-5 volume, because most writes will fit into a single stripe, minimizing the numbers of reads and writes that must occur for parity calculations[1]. If you know your average file size, then you should tailor the stripe interlace size accordingly.

Note that if your stripe size is a power of 2, there’s a good chance that all of your superblocks and inodes will end up on the same physical disk, which will negatively impact performance. That’s why the example uses 65KB as an interlace size instead of 64KB.


Step Three: Starting Volume Manager
bash-3.00# sh /etc/init.d/volmgt start
>>output
volume management starting.

To be able to use our metadevice as if it were a physical device, we have to start the logical volume manager (our RAID-5 metadevice is a logical volume)


Now, you must wait until the metadevice is initialized before proceeding to create a new filesystem on the metadevice.
You can watch the progress of metadevice initialization via repeated invocations of the command
#metastat -i

>>output while intializing
d6: RAID
State: Initializing
Initialization in progress: 1.6% done
Interlace: 130 blocks
Size: 11662950630 blocks (5.4 TB)
Original device:
Size: 11662971840 blocks (5.4 TB)
Device Start Block Dbase State Reloc Hot Spare
c3t5006048AD5312D07d34s6 1310 No Initializing Yes
c3t5006048AD5312D07d35s6 1310 No Initializing Yes
c3t5006048AD5312D07d36s6 1310 No Initializing Yes
c3t5006048AD5312D07d37s6 1310 No Initializing Yes
c3t5006048AD5312D07d38s6 1310 No Initializing Yes
c3t5006048AD5312D07d39s6 1310 No Initializing Yes
c3t5006048AD5312D07d40s6 1310 No Initializing Yes


>>output after initialization finished
d6: RAID
State: Okay
Interlace: 130 blocks
Size: 11662950630 blocks (5.4 TB)
Original device:
Size: 11662971840 blocks (5.4 TB)
Device Start Block Dbase State Reloc Hot Spare
c3t5006048AD5312D07d34s6 1310 No Okay Yes
c3t5006048AD5312D07d35s6 1310 No Okay Yes
c3t5006048AD5312D07d36s6 1310 No Okay Yes
c3t5006048AD5312D07d37s6 1310 No Okay Yes
c3t5006048AD5312D07d38s6 1310 No Okay Yes
c3t5006048AD5312D07d39s6 1310 No Okay Yes
c3t5006048AD5312D07d40s6 1310 No Okay Yes


Step Four: Creating filesytem on the metadevice
newfs -c 256 -i 8192 -m 8 -C 65 /dev/md/rdsk/d6
>>output
/dev/md/rdsk/d6: Unable to find Media type. Proceeding with system determined parameters.
Warning: cylinders/group is obsolete for this device and will be ignored.
newfs: construct a new file system /dev/md/rdsk/d6: (y/n)? y
Warning: 1824 sector(s) in last cylinder unallocated
/dev/md/rdsk/d6: 11662950624 sectors in 1898267 cylinders of 48 tracks, 128 sectors
5694800.0MB in 13275 cyl groups (143 c/g, 429.00MB/g, 448 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
32, 878752, 1757472, 2636192, 3514912, 4393632, 5272352, 6151072, 7029792,
7908512,
Initializing cylinder groups:
...............................................................................
...............................................................................
...............................................................................
............................
super-block backups for last 10 cylinder groups at:
11654525088, 11655403808, 11656282528, 11657161248, 11658039968, 11658918688,
11659797408, 11660676128, 11661554848, 11662433568



Step five: mount the new file system
bash-3.00# mkdir newmount
bash-3.00# mount /dev/md/dsk/d6 /newmount

No comments: