This page discusses configuring storage exported from 3PAR to Solaris 10 using native MPxIO
We have 2 dual port HBA connected to 3PAR T800 using QLogic switches
Hardwares:
SPARC-Enterprise-T5220
Emulex LPe11002-S
3PAR T800
QLogic SANBox 5802 FC Switch
Multiplathing on Solaris 10 update 5 works with FC and SCSI
3PAR LUNs exported as below:
This is just to show that you have completed your zoning and exported luns to the solaris host
LUN,Domain,Virtual Volume,RAID Type,Host,Port,Host WWN/iSCSI Name,Exported Size (GiB),Type
0,--,VV-mdc2ps006-R5-NL1,RAID 5,mdc2ps006,4:5:1,10000000C97D2C71,1 024.000 GiB,Host Sees
0,--,VV-mdc2ps006-R5-NL1,RAID 5,mdc2ps006,6:5:4,10000000C97D2C71,1 024.000 GiB,Host Sees
0,--,VV-mdc2ps006-R5-NL1,RAID 5,mdc2ps006,5:5:1,10000000C97BFB4D,1 024.000 GiB,Host Sees
0,--,VV-mdc2ps006-R5-NL1,RAID 5,mdc2ps006,7:5:4,10000000C97BFB4D,1 024.000 GiB,Host Sees
Confirmation of zoning and connectivity:
bash-3.00# cfgadm -al -o show_FCP_dev
Ap_Id Type Receptacle Occupant Condition
c2 fc connected unconfigured unknown
c3 fc-fabric connected configured unknown <-- HBA connection
on c3
c3::25510002ac000ae2,0 disk connected configured unknown <-- 3PAR LUN on c3
c3::25510002ac000ae2,254 ESI connected configured unknown
c3::27540002ac000ae2,0 disk connected configured unknown <-- 3PAR LUN on c3
c3::27540002ac000ae2,254 ESI connected configured unknown
c4 fc connected unconfigured unknown
c5 fc-fabric connected configured unknown <-- HBA connection
on c5
c5::24510002ac000ae2,0 disk connected configured unknown <-- 3PAR LUN on c5
c5::24510002ac000ae2,254 ESI connected configured unknown
c5::26540002ac000ae2,0 disk connected configured unknown
c5::26540002ac000ae2,254 ESI connected configured unknown <-- 3PAR LUN on c5
connection Topology:
HBA1 -> Qlogic1 -> 3PAR1-C1/3PAR-C2
HBA2 -> QLogic2 -> 3PAR1-C3/3PAR-C4
bash-3.00# cfgadm -al
Ap_Id Type Receptacle Occupant Condition
c1 scsi-sata connected configured unknown
c1::dsk/c1t0d0 disk connected configured unknown
c1::dsk/c1t1d0 disk connected configured unknown
c1::dsk/c1t2d0 disk connected configured unknown
c1::dsk/c1t3d0 disk connected configured unknown
c1::dsk/c1t4d0 disk connected configured unknown
c1::dsk/c1t5d0 disk connected configured unknown
c2 fc connected unconfigured unknown
c3 fc-fabric connected configured unknown
c3::25510002ac000ae2 disk connected configured unknown <--3PAR LUN
c3::27540002ac000ae2 disk connected configured unknown <--3PAR LUN
c4 fc connected unconfigured unknown
c5 fc-fabric connected configured unknown
c5::24510002ac000ae2 disk connected configured unknown <--3PAR LUN
c5::26540002ac000ae2 disk connected configured unknown <--3PAR LUN
bash-3.00# luxadm probe
No Network Array enclosures found in /dev/es
Found Fibre Channel device(s):
Node WWN:2ff70002ac000ae2 Device Type:Disk device
Logical Path:/dev/rdsk/c3t25510002AC000AE2d0s2
Logical Path:/dev/rdsk/c3t27540002AC000AE2d0s2
Logical Path:/dev/rdsk/c5t24510002AC000AE2d0s2
Logical Path:/dev/rdsk/c5t26540002AC000AE2d0s2
bash-3.00# format
Searching for disks...done
c3t25510002AC000AE2d0: configured with capacity of 2048.00GB
c3t27540002AC000AE2d0: configured with capacity of 2048.00GB
c5t24510002AC000AE2d0: configured with capacity of 2048.00GB
c5t26540002AC000AE2d0: configured with capacity of 2048.00GB
AVAILABLE DISK SELECTIONS:
0. c1t0d0
/pci@0/pci@0/pci@2/scsi@0/sd@0,0
1. c1t1d0
/pci@0/pci@0/pci@2/scsi@0/sd@1,0
2. c1t2d0
/pci@0/pci@0/pci@2/scsi@0/sd@2,0
3. c1t3d0
/pci@0/pci@0/pci@2/scsi@0/sd@3,0
4. c1t4d0
/pci@0/pci@0/pci@2/scsi@0/sd@4,0
5. c1t5d0
/pci@0/pci@0/pci@2/scsi@0/sd@5,0
3PAR LUNs on c3 and c5
6. c3t25510002AC000AE2d0 <3PARdata-VV-0000-2.00TB>
/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1/fp@0,0/ssd@w25510002ac000ae2,0
7. c3t27540002AC000AE2d0 <3PARdata-VV-0000-2.00TB>
/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1/fp@0,0/ssd@w27540002ac000ae2,0
8. c5t24510002AC000AE2d0 <3PARdata-VV-0000-2.00TB>
/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0/ssd@w24510002ac000ae2,0
9. c5t26540002AC000AE2d0 <3PARdata-VV-0000-2.00TB>
/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0/ssd@w26540002ac000ae2,0
Important:
These four devices listed above (6,7,8,9) refer to the same LUN on the 3PAR storage known as (LUN
0). This is determined by the 'd0' at the end of the device name. The problem here is that we are
looking at these different paths to the same device.
First path to LUN 0 is over the 255 interface (c3t25510002AC000AE2d0)
Second path to LUN 0 is over the 275 interface (c3t27540002AC000AE2d0)
Third path to LUN 0 is over the 245 interface (c5t24510002AC000AE2d0)
Fourth path to LUN 0 is over the 265 interface (c5t26540002AC000AE2d0)
LUN Owner
As we see, we have 4 disk devices instead of one. We need to mount only one of these paths of the
LUN. The path that we need to choose is the path that is the 'owner' of the LUN.
For example, if the owner is the 255 interface, we should mount c3t25510002AC000AE2d0.
However, if the 255 interface has issues and we lose that path, we will lose access to the LUN.
We need a layer to abstract these 4 paths into a single device that we use. This layer would
present a new virtual/psuedo device that could be mounted. Underneath this virtual device would
be the two paths to the LUN, the owner and the non-owner. This resolves the issue of what device
to use if an interface to the LUN fails.
The multipathing layer would simply start accessing the device over the alternate path and would
not require any changes to what device the OS accesses the filesystem over.
In Solaris 10, MPXIO provides this layer of abstraction to solve this issue.
Solaris I/O multipathing gives you the ability to set up multiple redundant paths to a storage
system and gives you the benefits of load balancing and failover.
Enable MPxIO
Solaris 10 is the easier, because the mpxio capability is built-in. You just need to turn it on!
Step #1.
vi /kernel/drv/fp.conf
mpxio-disable="no";
Enabling mpxio enables it everywhere.
However, You can enable mulipathing for each device
First list all the FC devices
bash-3.00# ls -l /dev/fc
total 8
lrwxrwxrwx 1 root root 70 May 13 15:41 fp0 ->
../../devices/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0/fp@0,0:devctl
lrwxrwxrwx 1 root root 72 May 13 15:41 fp1 ->
../../devices/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1/fp@0,0:devctl
lrwxrwxrwx 1 root root 70 May 13 15:41 fp2 ->
../../devices/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0/fp@0,0:devctl
lrwxrwxrwx 1 root root 72 May 13 15:41 fp3 ->
../../devices/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0:devctl
or
bash-3.00# luxadm -e port
/devices/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0/fp@0,0:devctl NOT CONNECTED -> Port 0
/devices/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1/fp@0,0:devctl CONNECTED -> Port 1
/devices/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0:devctl CONNECTED -> Port 1
/devices/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0/fp@0,0:devctl NOT CONNECTED -> Port 0
HBA is connected to port 1
We are interested in the path after '/devices' and before '/fp@0,0...'
In this case, the path we are interested in is: '/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0'.
This path is associated with port 0 of the fp driver (from the '0' in fp0).
mpxio-disable="yes";
name="fp" parent="/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0" port=1 mpxio-disable="no";
name="fp" parent="/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1" port=1 mpxio-disable="no";
name="fp" parent="/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0" port=1 mpxio-disable="no";
name="fp" parent="/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1" port=1 mpxio-disable="no";
This disables MPXIO globally and enables it for the above device.
HBA is connected to port 1
name="fp" parent="parent name" port=port-number mpxio-disable="no/yes"
where parent name is the Qlogic device path
ie: /pci@9,600000/pci@2/SUNW,qlc@4
and port-number is the port number of the HBA
Step #2.
Edit /kernel/drv/scsi_vhci.conf
The scsi_vhci driver is responsible for hiding the owner and non-owner paths to the LUNs.
To enable this capability, the driver must know what type of device that the HBA is connected to.
To do this, you need to set the following in /kernel/drv/scsi_vhci.conf:
device-type-scsi-options-list="3PARdataVV", "symmetric-option";
symmetric-option = 0x1000000;
We knew to set the option to "3PARdata VV" from the output of format:
6. c3t25510002AC000AE2d0 <3PARdata-VV-0000-2.00TB>
Since 3PAR is symmetric luns which basically means active/active between all the paths the luns sees. We have to tweak the MPxIO to make it work.
Important:
In case of 3PAR /kernel/drv/scsi_vhci.conf doesn't like space between 3PARdata and VV :-)
device-type-scsi-options-list="3PARdata VV", "symmetric-option";
make sure it is 3PARdataVV as below
device-type-scsi-options-list="3PARdataVV", "symmetric-option";
format -e
select the lun
format> i
Vendor: 3PARdata
Product: VV
Revision: 0000
Vendor ID has 8 characters
Product ID has 16 characters
Step #3.
Now, if you are manually updated
/kernel/drv/fp.conf
and
/kernel/drv/scsi_vhci.conf
then you can simply use stmsboot update and it will udpate the multipathing and vfstab
#stmsboot -u
you have reboot again with this option
reboot -- -rv
Below is the steps to show stmsboot options
For Solaris 10 (SPARC)
run the stmsboot command and confirm the reboot:
# stmsboot –e
another stmsboot commands:
To Disable = stmsboot -d
To update = stmsboot -u
Output of -e
bash-3.00# stmsboot -e
WARNING: stmsboot operates on each supported multipath-capable controller
detected in a host. In your system, these controllers are
/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0/fp@0,0
/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1/fp@0,0
/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0/fp@0,0
/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0
/pci@0/pci@0/pci@2/scsi@0
If you do NOT wish to operate on these controllers, please quit stmsboot
and re-invoke with -D { fp | mpt | mpt_sas} to specify which controllers you wish
to modify your multipathing configuration for.
Do you wish to continue? [y/n] (default: y)
WARNING: This operation will require a reboot.
Do you want to continue ? [y/n] (default: y)
The changes will come into effect after rebooting the system.
Reboot the system now ? [y/n] (default: y)
updating /platform/sun4v/boot_archive
This will also udpate /kernel/drv/fp.conf with mpxio-disable="no";
bash-3.00# stmsboot -L
non-STMS device name STMS device name
------------------------------------------------------------------
/dev/rdsk/c1t0d0 /dev/rdsk/c6t5000C500091B90B3d0
/dev/rdsk/c1t1d0 /dev/rdsk/c6t5000C500091B7D93d0
/dev/rdsk/c1t2d0 /dev/rdsk/c6t5000CCA00A350350d0
/dev/rdsk/c1t3d0 /dev/rdsk/c6t5000CCA00A33F218d0
/dev/rdsk/c1t4d0 /dev/rdsk/c6t5000CCA00A3468E0d0
/dev/rdsk/c1t5d0 /dev/rdsk/c6t5000CCA00A34E950d0
This is when we enable multipathing on all devices
say yes to reboot
To enable Solaris I/O Multipathing on multipath-capable Fibrechannel controller ports,
enter:
bash-3.00# stmsboot -D fp -e
WARNING: This operation will require a reboot.
Do you want to continue ? [y/n] (default: y)
The changes will come into effect after rebooting the system.
Reboot the system now ? [y/n] (default: y)
Before multipathing, you would see four copies of each disk in format. Afterwards, you'll just
see the one copy.
It assigns the next available controller ID, and makes up some horrendously long target number.
For example:
Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c6t600C0FF000000000086AB238B2AF0600d0s5 697942398 20825341 670137634 4% /test
At this point we see 4 devices ( 6-9 as listed above)
Before MPxIO
bash-3.00# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c1t0d0
/pci@0/pci@0/pci@2/scsi@0/sd@0,0
1. c1t1d0
/pci@0/pci@0/pci@2/scsi@0/sd@1,0
2. c1t2d0
/pci@0/pci@0/pci@2/scsi@0/sd@2,0
3. c1t3d0
/pci@0/pci@0/pci@2/scsi@0/sd@3,0
4. c1t4d0
/pci@0/pci@0/pci@2/scsi@0/sd@4,0
5. c1t5d0
/pci@0/pci@0/pci@2/scsi@0/sd@5,0
3PAR lists 4 devices
6. c3t25510002AC000AE2d0 <3PARdata-VV-0000-2.00TB>
/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1/fp@0,0/ssd@w25510002ac000ae2,0
7. c3t27540002AC000AE2d0 <3PARdata-VV-0000-2.00TB>
/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1/fp@0,0/ssd@w27540002ac000ae2,0
8. c5t24510002AC000AE2d0 <3PARdata-VV-0000-2.00TB>
/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0/ssd@w24510002ac000ae2,0
9. c5t26540002AC000AE2d0 <3PARdata-VV-0000-2.00TB>
/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0/ssd@w26540002ac000ae2,0
After Enabling MPxIO
bash-3.00# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c6t5000C500091B7D93d0
/scsi_vhci/disk@g5000c500091b7d93
1. c6t5000C500091B90B3d0
/scsi_vhci/disk@g5000c500091b90b3
2. c6t5000CCA00A33F218d0
/scsi_vhci/disk@g5000cca00a33f218
3. c6t5000CCA00A34E950d0
/scsi_vhci/disk@g5000cca00a34e950
4. c6t5000CCA00A3468E0d0
/scsi_vhci/disk@g5000cca00a3468e0
5. c6t5000CCA00A350350d0
/scsi_vhci/disk@g5000cca00a350350
3PAR list one device instead of 4 as above
6. c6t50002AC000220AE2d0 <3PARdata-VV-0000-2.00TB>
/scsi_vhci/ssd@g50002ac000220ae2
You will notice all the above devices changed to /scsi_vhci/XXXXX instead of /pci@0/pci@0/
Also compare your previous probe output to one below
bash-3.00# luxadm probe
No Network Array enclosures found in /dev/es
Found Fibre Channel device(s):
Node WWN:2ff70002ac000ae2 Device Type:Disk device
Logical Path:/dev/rdsk/c6t50002AC000220AE2d0s2
Below stmsboot output shows you how all interfaces 245, 255, 265 and 275 has one pseudo name as
50002AC000220AE2
bash-3.00# stmsboot -L
non-STMS device name STMS device name
------------------------------------------------------------------
/dev/rdsk/c1t0d0 /dev/rdsk/c6t5000C500091B90B3d0
/dev/rdsk/c1t1d0 /dev/rdsk/c6t5000C500091B7D93d0
/dev/rdsk/c1t2d0 /dev/rdsk/c6t5000CCA00A350350d0
/dev/rdsk/c1t3d0 /dev/rdsk/c6t5000CCA00A33F218d0
/dev/rdsk/c1t4d0 /dev/rdsk/c6t5000CCA00A3468E0d0
/dev/rdsk/c1t5d0 /dev/rdsk/c6t5000CCA00A34E950d0
/dev/rdsk/c5t26540002AC000AE2d0 /dev/rdsk/c6t50002AC000220AE2d0
/dev/rdsk/c5t24510002AC000AE2d0 /dev/rdsk/c6t50002AC000220AE2d0
/dev/rdsk/c3t27540002AC000AE2d0 /dev/rdsk/c6t50002AC000220AE2d0
/dev/rdsk/c3t25510002AC000AE2d0 /dev/rdsk/c6t50002AC000220AE2d0
Step #4.
using mpathadm commands
To List all your initiators
bash-3.00# mpathadm list initiator-port
Initiator Port: 508002000081b570
Initiator Port: ,4000002a00ff
Initiator Port: 10000000c97bfb4c
Initiator Port: 10000000c97d2c70
Initiator Port: 10000000c97bfb4d -> connected HBA
Initiator Port: 10000000c97d2c71 -> connected HBA
bash-3.00# mpathadm list lu
List all LUNs
bash-3.00# mpathadm list lu
/dev/rdsk/c6t5000C500091B90B3d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t5000C500091B7D93d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t5000CCA00A350350d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t5000CCA00A33F218d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t5000CCA00A3468E0d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t5000CCA00A34E950d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t50002AC000220AE2d0s2 <--- pesudo LUN we are interested in
Total Path Count: 4
Operational Path Count: 4
This shows how this pesudo device using the different path underneath
bash-3.00# mpathadm show lu /dev/rdsk/c6t50002AC000220AE2d0s2
Logical Unit: /dev/rdsk/c6t50002AC000220AE2d0s2
mpath-support: libmpscsi_vhci.so
Vendor: 3PARdata
Product: VV
Revision: 0000
Name Type: unknown type
Name: 50002ac000220ae2
Asymmetric: no
Current Load Balance: round-robin
Logical Unit Group ID: NA
Auto Failback: on
Auto Probing: NA
Paths:
Initiator Port Name: 10000000c97d2c71
Target Port Name: 26540002ac000ae2
Override Path: NA
Path State: OK
Disabled: no
Initiator Port Name: 10000000c97d2c71
Target Port Name: 24510002ac000ae2
Override Path: NA
Path State: OK
Disabled: no
Initiator Port Name: 10000000c97bfb4d
Target Port Name: 27540002ac000ae2
Override Path: NA
Path State: OK
Disabled: no
Initiator Port Name: 10000000c97bfb4d
Target Port Name: 25510002ac000ae2
Override Path: NA
Path State: OK
Disabled: no
Target Ports:
Name: 26540002ac000ae2
Relative ID: 0
Name: 24510002ac000ae2
Relative ID: 0
Name: 27540002ac000ae2
Relative ID: 0
Name: 25510002ac000ae2
Relative ID: 0
