Thursday, August 18, 2011

3PAR storage on Solaris 10 using native MPxIO

This page discusses configuring storage exported from 3PAR to Solaris 10 using native MPxIO

We have 2 dual port HBA connected to 3PAR T800 using QLogic switches

Hardwares:
SPARC-Enterprise-T5220
Emulex LPe11002-S
3PAR T800
QLogic SANBox 5802 FC Switch


Multiplathing on Solaris 10 update 5 works with FC and SCSI


3PAR LUNs exported as below:
This is just to show that you have completed your zoning and exported luns to the solaris host

LUN,Domain,Virtual Volume,RAID Type,Host,Port,Host WWN/iSCSI Name,Exported Size (GiB),Type
0,--,VV-mdc2ps006-R5-NL1,RAID 5,mdc2ps006,4:5:1,10000000C97D2C71,1 024.000 GiB,Host Sees
0,--,VV-mdc2ps006-R5-NL1,RAID 5,mdc2ps006,6:5:4,10000000C97D2C71,1 024.000 GiB,Host Sees
0,--,VV-mdc2ps006-R5-NL1,RAID 5,mdc2ps006,5:5:1,10000000C97BFB4D,1 024.000 GiB,Host Sees
0,--,VV-mdc2ps006-R5-NL1,RAID 5,mdc2ps006,7:5:4,10000000C97BFB4D,1 024.000 GiB,Host Sees



Confirmation of zoning and connectivity:

bash-3.00# cfgadm -al -o show_FCP_dev
Ap_Id Type Receptacle Occupant Condition
c2 fc connected unconfigured unknown
c3 fc-fabric connected configured unknown <-- HBA connection

on c3
c3::25510002ac000ae2,0 disk connected configured unknown <-- 3PAR LUN on c3
c3::25510002ac000ae2,254 ESI connected configured unknown
c3::27540002ac000ae2,0 disk connected configured unknown <-- 3PAR LUN on c3
c3::27540002ac000ae2,254 ESI connected configured unknown
c4 fc connected unconfigured unknown
c5 fc-fabric connected configured unknown <-- HBA connection

on c5
c5::24510002ac000ae2,0 disk connected configured unknown <-- 3PAR LUN on c5
c5::24510002ac000ae2,254 ESI connected configured unknown
c5::26540002ac000ae2,0 disk connected configured unknown
c5::26540002ac000ae2,254 ESI connected configured unknown <-- 3PAR LUN on c5


connection Topology:
HBA1 -> Qlogic1 -> 3PAR1-C1/3PAR-C2
HBA2 -> QLogic2 -> 3PAR1-C3/3PAR-C4


bash-3.00# cfgadm -al
Ap_Id Type Receptacle Occupant Condition
c1 scsi-sata connected configured unknown
c1::dsk/c1t0d0 disk connected configured unknown
c1::dsk/c1t1d0 disk connected configured unknown
c1::dsk/c1t2d0 disk connected configured unknown
c1::dsk/c1t3d0 disk connected configured unknown
c1::dsk/c1t4d0 disk connected configured unknown
c1::dsk/c1t5d0 disk connected configured unknown
c2 fc connected unconfigured unknown
c3 fc-fabric connected configured unknown
c3::25510002ac000ae2 disk connected configured unknown <--3PAR LUN
c3::27540002ac000ae2 disk connected configured unknown <--3PAR LUN
c4 fc connected unconfigured unknown
c5 fc-fabric connected configured unknown
c5::24510002ac000ae2 disk connected configured unknown <--3PAR LUN
c5::26540002ac000ae2 disk connected configured unknown <--3PAR LUN


bash-3.00# luxadm probe
No Network Array enclosures found in /dev/es

Found Fibre Channel device(s):
Node WWN:2ff70002ac000ae2 Device Type:Disk device
Logical Path:/dev/rdsk/c3t25510002AC000AE2d0s2
Logical Path:/dev/rdsk/c3t27540002AC000AE2d0s2
Logical Path:/dev/rdsk/c5t24510002AC000AE2d0s2
Logical Path:/dev/rdsk/c5t26540002AC000AE2d0s2


bash-3.00# format
Searching for disks...done

c3t25510002AC000AE2d0: configured with capacity of 2048.00GB
c3t27540002AC000AE2d0: configured with capacity of 2048.00GB
c5t24510002AC000AE2d0: configured with capacity of 2048.00GB
c5t26540002AC000AE2d0: configured with capacity of 2048.00GB


AVAILABLE DISK SELECTIONS:
0. c1t0d0
/pci@0/pci@0/pci@2/scsi@0/sd@0,0
1. c1t1d0
/pci@0/pci@0/pci@2/scsi@0/sd@1,0
2. c1t2d0
/pci@0/pci@0/pci@2/scsi@0/sd@2,0
3. c1t3d0
/pci@0/pci@0/pci@2/scsi@0/sd@3,0
4. c1t4d0
/pci@0/pci@0/pci@2/scsi@0/sd@4,0
5. c1t5d0
/pci@0/pci@0/pci@2/scsi@0/sd@5,0
3PAR LUNs on c3 and c5
6. c3t25510002AC000AE2d0 <3PARdata-VV-0000-2.00TB>
/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1/fp@0,0/ssd@w25510002ac000ae2,0
7. c3t27540002AC000AE2d0 <3PARdata-VV-0000-2.00TB>
/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1/fp@0,0/ssd@w27540002ac000ae2,0
8. c5t24510002AC000AE2d0 <3PARdata-VV-0000-2.00TB>
/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0/ssd@w24510002ac000ae2,0
9. c5t26540002AC000AE2d0 <3PARdata-VV-0000-2.00TB>
/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0/ssd@w26540002ac000ae2,0




Important:

These four devices listed above (6,7,8,9) refer to the same LUN on the 3PAR storage known as (LUN

0). This is determined by the 'd0' at the end of the device name. The problem here is that we are

looking at these different paths to the same device.

First path to LUN 0 is over the 255 interface (c3t25510002AC000AE2d0)
Second path to LUN 0 is over the 275 interface (c3t27540002AC000AE2d0)
Third path to LUN 0 is over the 245 interface (c5t24510002AC000AE2d0)
Fourth path to LUN 0 is over the 265 interface (c5t26540002AC000AE2d0)


LUN Owner
As we see, we have 4 disk devices instead of one. We need to mount only one of these paths of the

LUN. The path that we need to choose is the path that is the 'owner' of the LUN.
For example, if the owner is the 255 interface, we should mount c3t25510002AC000AE2d0.
However, if the 255 interface has issues and we lose that path, we will lose access to the LUN.

We need a layer to abstract these 4 paths into a single device that we use. This layer would

present a new virtual/psuedo device that could be mounted. Underneath this virtual device would

be the two paths to the LUN, the owner and the non-owner. This resolves the issue of what device

to use if an interface to the LUN fails.

The multipathing layer would simply start accessing the device over the alternate path and would

not require any changes to what device the OS accesses the filesystem over.


In Solaris 10, MPXIO provides this layer of abstraction to solve this issue.
Solaris I/O multipathing gives you the ability to set up multiple redundant paths to a storage

system and gives you the benefits of load balancing and failover.



Enable MPxIO
Solaris 10 is the easier, because the mpxio capability is built-in. You just need to turn it on!


Step #1.
vi /kernel/drv/fp.conf
mpxio-disable="no";

Enabling mpxio enables it everywhere.
However, You can enable mulipathing for each device

First list all the FC devices

bash-3.00# ls -l /dev/fc
total 8
lrwxrwxrwx 1 root root 70 May 13 15:41 fp0 ->

../../devices/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0/fp@0,0:devctl
lrwxrwxrwx 1 root root 72 May 13 15:41 fp1 ->

../../devices/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1/fp@0,0:devctl
lrwxrwxrwx 1 root root 70 May 13 15:41 fp2 ->

../../devices/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0/fp@0,0:devctl
lrwxrwxrwx 1 root root 72 May 13 15:41 fp3 ->

../../devices/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0:devctl

or

bash-3.00# luxadm -e port
/devices/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0/fp@0,0:devctl NOT CONNECTED -> Port 0
/devices/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1/fp@0,0:devctl CONNECTED -> Port 1
/devices/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0:devctl CONNECTED -> Port 1
/devices/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0/fp@0,0:devctl NOT CONNECTED -> Port 0

HBA is connected to port 1


We are interested in the path after '/devices' and before '/fp@0,0...'
In this case, the path we are interested in is: '/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0'.
This path is associated with port 0 of the fp driver (from the '0' in fp0).


mpxio-disable="yes";
name="fp" parent="/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0" port=1 mpxio-disable="no";
name="fp" parent="/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1" port=1 mpxio-disable="no";
name="fp" parent="/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0" port=1 mpxio-disable="no";
name="fp" parent="/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1" port=1 mpxio-disable="no";

This disables MPXIO globally and enables it for the above device.
HBA is connected to port 1

name="fp" parent="parent name" port=port-number mpxio-disable="no/yes"
where parent name is the Qlogic device path

ie: /pci@9,600000/pci@2/SUNW,qlc@4
and port-number is the port number of the HBA



Step #2.

Edit /kernel/drv/scsi_vhci.conf
The scsi_vhci driver is responsible for hiding the owner and non-owner paths to the LUNs.

To enable this capability, the driver must know what type of device that the HBA is connected to.
To do this, you need to set the following in /kernel/drv/scsi_vhci.conf:

device-type-scsi-options-list="3PARdataVV", "symmetric-option";
symmetric-option = 0x1000000;

We knew to set the option to "3PARdata VV" from the output of format:
6. c3t25510002AC000AE2d0 <3PARdata-VV-0000-2.00TB>

Since 3PAR is symmetric luns which basically means active/active between all the paths the luns sees. We have to tweak the MPxIO to make it work.

Important:
In case of 3PAR /kernel/drv/scsi_vhci.conf doesn't like space between 3PARdata and VV :-)
device-type-scsi-options-list="3PARdata VV", "symmetric-option";

make sure it is 3PARdataVV as below
device-type-scsi-options-list="3PARdataVV", "symmetric-option";


format -e
select the lun

format> i
Vendor: 3PARdata
Product: VV
Revision: 0000


Vendor ID has 8 characters
Product ID has 16 characters


Step #3.

Now, if you are manually updated

/kernel/drv/fp.conf
and
/kernel/drv/scsi_vhci.conf


then you can simply use stmsboot update and it will udpate the multipathing and vfstab
#stmsboot -u

you have reboot again with this option
reboot -- -rv


Below is the steps to show stmsboot options

For Solaris 10 (SPARC)
run the stmsboot command and confirm the reboot:
# stmsboot –e
another stmsboot commands:
To Disable = stmsboot -d
To update = stmsboot -u


Output of -e
bash-3.00# stmsboot -e

WARNING: stmsboot operates on each supported multipath-capable controller
detected in a host. In your system, these controllers are

/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0/fp@0,0
/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1/fp@0,0
/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0/fp@0,0
/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0
/pci@0/pci@0/pci@2/scsi@0

If you do NOT wish to operate on these controllers, please quit stmsboot
and re-invoke with -D { fp | mpt | mpt_sas} to specify which controllers you wish
to modify your multipathing configuration for.

Do you wish to continue? [y/n] (default: y)
WARNING: This operation will require a reboot.
Do you want to continue ? [y/n] (default: y)
The changes will come into effect after rebooting the system.
Reboot the system now ? [y/n] (default: y)
updating /platform/sun4v/boot_archive


This will also udpate /kernel/drv/fp.conf with mpxio-disable="no";



bash-3.00# stmsboot -L
non-STMS device name STMS device name
------------------------------------------------------------------
/dev/rdsk/c1t0d0 /dev/rdsk/c6t5000C500091B90B3d0
/dev/rdsk/c1t1d0 /dev/rdsk/c6t5000C500091B7D93d0
/dev/rdsk/c1t2d0 /dev/rdsk/c6t5000CCA00A350350d0
/dev/rdsk/c1t3d0 /dev/rdsk/c6t5000CCA00A33F218d0
/dev/rdsk/c1t4d0 /dev/rdsk/c6t5000CCA00A3468E0d0
/dev/rdsk/c1t5d0 /dev/rdsk/c6t5000CCA00A34E950d0


This is when we enable multipathing on all devices
say yes to reboot


To enable Solaris I/O Multipathing on multipath-capable Fibrechannel controller ports,

enter:

bash-3.00# stmsboot -D fp -e
WARNING: This operation will require a reboot.
Do you want to continue ? [y/n] (default: y)
The changes will come into effect after rebooting the system.
Reboot the system now ? [y/n] (default: y)




Before multipathing, you would see four copies of each disk in format. Afterwards, you'll just

see the one copy.

It assigns the next available controller ID, and makes up some horrendously long target number.
For example:

Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c6t600C0FF000000000086AB238B2AF0600d0s5 697942398 20825341 670137634 4% /test



At this point we see 4 devices ( 6-9 as listed above)


Before MPxIO
bash-3.00# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
0. c1t0d0
/pci@0/pci@0/pci@2/scsi@0/sd@0,0
1. c1t1d0
/pci@0/pci@0/pci@2/scsi@0/sd@1,0
2. c1t2d0
/pci@0/pci@0/pci@2/scsi@0/sd@2,0
3. c1t3d0
/pci@0/pci@0/pci@2/scsi@0/sd@3,0
4. c1t4d0
/pci@0/pci@0/pci@2/scsi@0/sd@4,0
5. c1t5d0
/pci@0/pci@0/pci@2/scsi@0/sd@5,0
3PAR lists 4 devices
6. c3t25510002AC000AE2d0 <3PARdata-VV-0000-2.00TB>
/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1/fp@0,0/ssd@w25510002ac000ae2,0
7. c3t27540002AC000AE2d0 <3PARdata-VV-0000-2.00TB>
/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1/fp@0,0/ssd@w27540002ac000ae2,0
8. c5t24510002AC000AE2d0 <3PARdata-VV-0000-2.00TB>
/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0/ssd@w24510002ac000ae2,0
9. c5t26540002AC000AE2d0 <3PARdata-VV-0000-2.00TB>
/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0/ssd@w26540002ac000ae2,0


After Enabling MPxIO


bash-3.00# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
0. c6t5000C500091B7D93d0
/scsi_vhci/disk@g5000c500091b7d93
1. c6t5000C500091B90B3d0
/scsi_vhci/disk@g5000c500091b90b3
2. c6t5000CCA00A33F218d0
/scsi_vhci/disk@g5000cca00a33f218
3. c6t5000CCA00A34E950d0
/scsi_vhci/disk@g5000cca00a34e950
4. c6t5000CCA00A3468E0d0
/scsi_vhci/disk@g5000cca00a3468e0
5. c6t5000CCA00A350350d0
/scsi_vhci/disk@g5000cca00a350350
3PAR list one device instead of 4 as above
6. c6t50002AC000220AE2d0 <3PARdata-VV-0000-2.00TB>
/scsi_vhci/ssd@g50002ac000220ae2


You will notice all the above devices changed to /scsi_vhci/XXXXX instead of /pci@0/pci@0/


Also compare your previous probe output to one below

bash-3.00# luxadm probe
No Network Array enclosures found in /dev/es

Found Fibre Channel device(s):
Node WWN:2ff70002ac000ae2 Device Type:Disk device
Logical Path:/dev/rdsk/c6t50002AC000220AE2d0s2


Below stmsboot output shows you how all interfaces 245, 255, 265 and 275 has one pseudo name as

50002AC000220AE2


bash-3.00# stmsboot -L
non-STMS device name STMS device name
------------------------------------------------------------------
/dev/rdsk/c1t0d0 /dev/rdsk/c6t5000C500091B90B3d0
/dev/rdsk/c1t1d0 /dev/rdsk/c6t5000C500091B7D93d0
/dev/rdsk/c1t2d0 /dev/rdsk/c6t5000CCA00A350350d0
/dev/rdsk/c1t3d0 /dev/rdsk/c6t5000CCA00A33F218d0
/dev/rdsk/c1t4d0 /dev/rdsk/c6t5000CCA00A3468E0d0
/dev/rdsk/c1t5d0 /dev/rdsk/c6t5000CCA00A34E950d0
/dev/rdsk/c5t26540002AC000AE2d0 /dev/rdsk/c6t50002AC000220AE2d0
/dev/rdsk/c5t24510002AC000AE2d0 /dev/rdsk/c6t50002AC000220AE2d0
/dev/rdsk/c3t27540002AC000AE2d0 /dev/rdsk/c6t50002AC000220AE2d0
/dev/rdsk/c3t25510002AC000AE2d0 /dev/rdsk/c6t50002AC000220AE2d0





Step #4.

using mpathadm commands
To List all your initiators
bash-3.00# mpathadm list initiator-port
Initiator Port: 508002000081b570
Initiator Port: ,4000002a00ff
Initiator Port: 10000000c97bfb4c
Initiator Port: 10000000c97d2c70
Initiator Port: 10000000c97bfb4d -> connected HBA
Initiator Port: 10000000c97d2c71 -> connected HBA


bash-3.00# mpathadm list lu
List all LUNs
bash-3.00# mpathadm list lu
/dev/rdsk/c6t5000C500091B90B3d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t5000C500091B7D93d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t5000CCA00A350350d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t5000CCA00A33F218d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t5000CCA00A3468E0d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t5000CCA00A34E950d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t50002AC000220AE2d0s2 <--- pesudo LUN we are interested in
Total Path Count: 4
Operational Path Count: 4

This shows how this pesudo device using the different path underneath

bash-3.00# mpathadm show lu /dev/rdsk/c6t50002AC000220AE2d0s2
Logical Unit: /dev/rdsk/c6t50002AC000220AE2d0s2
mpath-support: libmpscsi_vhci.so
Vendor: 3PARdata
Product: VV
Revision: 0000
Name Type: unknown type
Name: 50002ac000220ae2
Asymmetric: no
Current Load Balance: round-robin
Logical Unit Group ID: NA
Auto Failback: on
Auto Probing: NA

Paths:
Initiator Port Name: 10000000c97d2c71
Target Port Name: 26540002ac000ae2
Override Path: NA
Path State: OK
Disabled: no

Initiator Port Name: 10000000c97d2c71
Target Port Name: 24510002ac000ae2
Override Path: NA
Path State: OK
Disabled: no

Initiator Port Name: 10000000c97bfb4d
Target Port Name: 27540002ac000ae2
Override Path: NA
Path State: OK
Disabled: no

Initiator Port Name: 10000000c97bfb4d
Target Port Name: 25510002ac000ae2
Override Path: NA
Path State: OK
Disabled: no

Target Ports:
Name: 26540002ac000ae2
Relative ID: 0

Name: 24510002ac000ae2
Relative ID: 0

Name: 27540002ac000ae2
Relative ID: 0

Name: 25510002ac000ae2
Relative ID: 0



3 comments:

Unknown said...

You seem to have some practical experience with 3PAR and Solaris 10. Have you found any documentation or done any experimentation with the load balancing algorithm that MPxIO uses?

Pankaj Gautam said...

Stephen, what are you trying to resolve?
Here are some configuration for solaris multipath
http://docs.oracle.com/cd/E23824_01/html/E23097/agkap.html#agkar

Gabriel said...

This was incredibly helpful. Thank you so very much!!!