3PAR storage on Solaris 10 using native MPxIO

This page discusses configuring storage exported from 3PAR to Solaris 10 using native MPxIO

We have 2 dual port HBA connected to 3PAR T800 using QLogic switches

Hardwares:

SPARC-Enterprise-T5220

Emulex LPe11002-S

3PAR T800

QLogic SANBox 5802 FC Switch

Multiplathing on Solaris 10 update 5 works with FC and SCSI

3PAR LUNs exported as below:

This is just to show that you have completed your zoning and exported luns to the solaris host

LUN,Domain,Virtual Volume,RAID Type,Host,Port,Host WWN/iSCSI Name,Exported Size (GiB),Type

0,--,VV-mdc2ps006-R5-NL1,RAID 5,mdc2ps006,4:5:1,10000000C97D2C71,1 024.000 GiB,Host Sees

0,--,VV-mdc2ps006-R5-NL1,RAID 5,mdc2ps006,6:5:4,10000000C97D2C71,1 024.000 GiB,Host Sees

0,--,VV-mdc2ps006-R5-NL1,RAID 5,mdc2ps006,5:5:1,10000000C97BFB4D,1 024.000 GiB,Host Sees

0,--,VV-mdc2ps006-R5-NL1,RAID 5,mdc2ps006,7:5:4,10000000C97BFB4D,1 024.000 GiB,Host Sees

Confirmation of zoning and connectivity:

bash-3.00# cfgadm -al -o show_FCP_dev

Ap_Id Type Receptacle Occupant Condition

c2 fc connected unconfigured unknown

c3 fc-fabric connected configured unknown <-- HBA connection

on c3

c3::25510002ac000ae2,0 disk connected configured unknown <-- 3PAR LUN on c3

c3::25510002ac000ae2,254 ESI connected configured unknown

c3::27540002ac000ae2,0 disk connected configured unknown <-- 3PAR LUN on c3

c3::27540002ac000ae2,254 ESI connected configured unknown

c4 fc connected unconfigured unknown

c5 fc-fabric connected configured unknown <-- HBA connection

on c5

c5::24510002ac000ae2,0 disk connected configured unknown <-- 3PAR LUN on c5

c5::24510002ac000ae2,254 ESI connected configured unknown

c5::26540002ac000ae2,0 disk connected configured unknown

c5::26540002ac000ae2,254 ESI connected configured unknown <-- 3PAR LUN on c5

connection Topology:

HBA1 -> Qlogic1 -> 3PAR1-C1/3PAR-C2

HBA2 -> QLogic2 -> 3PAR1-C3/3PAR-C4

bash-3.00# cfgadm -al

Ap_Id Type Receptacle Occupant Condition

c1 scsi-sata connected configured unknown

c1::dsk/c1t0d0 disk connected configured unknown

c1::dsk/c1t1d0 disk connected configured unknown

c1::dsk/c1t2d0 disk connected configured unknown

c1::dsk/c1t3d0 disk connected configured unknown

c1::dsk/c1t4d0 disk connected configured unknown

c1::dsk/c1t5d0 disk connected configured unknown

c2 fc connected unconfigured unknown

c3 fc-fabric connected configured unknown

c3::25510002ac000ae2 disk connected configured unknown <--3PAR LUN

c3::27540002ac000ae2 disk connected configured unknown <--3PAR LUN

c4 fc connected unconfigured unknown

c5 fc-fabric connected configured unknown

c5::24510002ac000ae2 disk connected configured unknown <--3PAR LUN

c5::26540002ac000ae2 disk connected configured unknown <--3PAR LUN

bash-3.00# luxadm probe

No Network Array enclosures found in /dev/es

Found Fibre Channel device(s):

Node WWN:2ff70002ac000ae2 Device Type:Disk device

Logical Path:/dev/rdsk/c3t25510002AC000AE2d0s2

Logical Path:/dev/rdsk/c3t27540002AC000AE2d0s2

Logical Path:/dev/rdsk/c5t24510002AC000AE2d0s2

Logical Path:/dev/rdsk/c5t26540002AC000AE2d0s2

bash-3.00# format

Searching for disks...done

c3t25510002AC000AE2d0: configured with capacity of 2048.00GB

c3t27540002AC000AE2d0: configured with capacity of 2048.00GB

c5t24510002AC000AE2d0: configured with capacity of 2048.00GB

c5t26540002AC000AE2d0: configured with capacity of 2048.00GB

AVAILABLE DISK SELECTIONS:

0. c1t0d0

/pci@0/pci@0/pci@2/scsi@0/sd@0,0

1. c1t1d0

/pci@0/pci@0/pci@2/scsi@0/sd@1,0

2. c1t2d0

/pci@0/pci@0/pci@2/scsi@0/sd@2,0

3. c1t3d0

/pci@0/pci@0/pci@2/scsi@0/sd@3,0

4. c1t4d0

/pci@0/pci@0/pci@2/scsi@0/sd@4,0

5. c1t5d0

/pci@0/pci@0/pci@2/scsi@0/sd@5,0

3PAR LUNs on c3 and c5

6. c3t25510002AC000AE2d0 <3PARdata-VV-0000-2.00TB>

/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1/fp@0,0/ssd@w25510002ac000ae2,0

7. c3t27540002AC000AE2d0 <3PARdata-VV-0000-2.00TB>

/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1/fp@0,0/ssd@w27540002ac000ae2,0

8. c5t24510002AC000AE2d0 <3PARdata-VV-0000-2.00TB>

/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0/ssd@w24510002ac000ae2,0

9. c5t26540002AC000AE2d0 <3PARdata-VV-0000-2.00TB>

/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0/ssd@w26540002ac000ae2,0

Important:

These four devices listed above (6,7,8,9) refer to the same LUN on the 3PAR storage known as (LUN

0). This is determined by the 'd0' at the end of the device name. The problem here is that we are

looking at these different paths to the same device.

First path to LUN 0 is over the 255 interface (c3t25510002AC000AE2d0)

Second path to LUN 0 is over the 275 interface (c3t27540002AC000AE2d0)

Third path to LUN 0 is over the 245 interface (c5t24510002AC000AE2d0)

Fourth path to LUN 0 is over the 265 interface (c5t26540002AC000AE2d0)

LUN Owner

As we see, we have 4 disk devices instead of one. We need to mount only one of these paths of the

LUN. The path that we need to choose is the path that is the 'owner' of the LUN.

For example, if the owner is the 255 interface, we should mount c3t25510002AC000AE2d0.

However, if the 255 interface has issues and we lose that path, we will lose access to the LUN.

We need a layer to abstract these 4 paths into a single device that we use. This layer would

present a new virtual/psuedo device that could be mounted. Underneath this virtual device would

be the two paths to the LUN, the owner and the non-owner. This resolves the issue of what device

to use if an interface to the LUN fails.

The multipathing layer would simply start accessing the device over the alternate path and would

not require any changes to what device the OS accesses the filesystem over.

In Solaris 10, MPXIO provides this layer of abstraction to solve this issue.

Solaris I/O multipathing gives you the ability to set up multiple redundant paths to a storage

system and gives you the benefits of load balancing and failover.

Enable MPxIO

Solaris 10 is the easier, because the mpxio capability is built-in. You just need to turn it on!

Step #1.

vi /kernel/drv/fp.conf

mpxio-disable="no";

Enabling mpxio enables it everywhere.

However, You can enable mulipathing for each device

First list all the FC devices

bash-3.00# ls -l /dev/fc

total 8

lrwxrwxrwx 1 root root 70 May 13 15:41 fp0 ->

../../devices/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0/fp@0,0:devctl

lrwxrwxrwx 1 root root 72 May 13 15:41 fp1 ->

../../devices/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1/fp@0,0:devctl

lrwxrwxrwx 1 root root 70 May 13 15:41 fp2 ->

../../devices/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0/fp@0,0:devctl

lrwxrwxrwx 1 root root 72 May 13 15:41 fp3 ->

../../devices/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0:devctl

or

bash-3.00# luxadm -e port

/devices/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0/fp@0,0:devctl NOT CONNECTED -> Port 0

/devices/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1/fp@0,0:devctl CONNECTED -> Port 1

/devices/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0:devctl CONNECTED -> Port 1

/devices/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0/fp@0,0:devctl NOT CONNECTED -> Port 0

HBA is connected to port 1

We are interested in the path after '/devices' and before '/fp@0,0...'

In this case, the path we are interested in is: '/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0'.

This path is associated with port 0 of the fp driver (from the '0' in fp0).

mpxio-disable="yes";

name="fp" parent="/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0" port=1 mpxio-disable="no";

name="fp" parent="/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1" port=1 mpxio-disable="no";

name="fp" parent="/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0" port=1 mpxio-disable="no";

name="fp" parent="/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1" port=1 mpxio-disable="no";

This disables MPXIO globally and enables it for the above device.

HBA is connected to port 1

name="fp" parent="parent name" port=port-number mpxio-disable="no/yes"

where parent name is the Qlogic device path

ie: /pci@9,600000/pci@2/SUNW,qlc@4

and port-number is the port number of the HBA

Step #2.

Edit /kernel/drv/scsi_vhci.conf

The scsi_vhci driver is responsible for hiding the owner and non-owner paths to the LUNs.

To enable this capability, the driver must know what type of device that the HBA is connected to.

To do this, you need to set the following in /kernel/drv/scsi_vhci.conf:

device-type-scsi-options-list="3PARdataVV", "symmetric-option";

symmetric-option = 0x1000000;

We knew to set the option to "3PARdata VV" from the output of format:

6. c3t25510002AC000AE2d0 <3PARdata-VV-0000-2.00TB>

Since 3PAR is symmetric luns which basically means active/active between all the paths the luns sees. We have to tweak the MPxIO to make it work.

Important:

In case of 3PAR /kernel/drv/scsi_vhci.conf doesn't like space between 3PARdata and VV :-)

device-type-scsi-options-list="3PARdata VV", "symmetric-option";

make sure it is 3PARdataVV as below

device-type-scsi-options-list="3PARdataVV", "symmetric-option";

format -e

select the lun

format> i

Vendor: 3PARdata

Product: VV

Revision: 0000

Vendor ID has 8 characters

Product ID has 16 characters

Step #3.

Now, if you are manually updated

/kernel/drv/fp.conf

and

/kernel/drv/scsi_vhci.conf

then you can simply use stmsboot update and it will udpate the multipathing and vfstab

#stmsboot -u

you have reboot again with this option

reboot -- -rv

Below is the steps to show stmsboot options

For Solaris 10 (SPARC)

run the stmsboot command and confirm the reboot:

# stmsboot –e

another stmsboot commands:

To Disable = stmsboot -d

To update = stmsboot -u

Output of -e

bash-3.00# stmsboot -e

WARNING: stmsboot operates on each supported multipath-capable controller

detected in a host. In your system, these controllers are

/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0/fp@0,0

/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1/fp@0,0

/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0/fp@0,0

/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0

/pci@0/pci@0/pci@2/scsi@0

If you do NOT wish to operate on these controllers, please quit stmsboot

and re-invoke with -D { fp | mpt | mpt_sas} to specify which controllers you wish

to modify your multipathing configuration for.

Do you wish to continue? [y/n] (default: y)

WARNING: This operation will require a reboot.

Do you want to continue ? [y/n] (default: y)

The changes will come into effect after rebooting the system.

Reboot the system now ? [y/n] (default: y)

updating /platform/sun4v/boot_archive

This will also udpate /kernel/drv/fp.conf with mpxio-disable="no";

bash-3.00# stmsboot -L

non-STMS device name STMS device name

------------------------------------------------------------------

/dev/rdsk/c1t0d0 /dev/rdsk/c6t5000C500091B90B3d0

/dev/rdsk/c1t1d0 /dev/rdsk/c6t5000C500091B7D93d0

/dev/rdsk/c1t2d0 /dev/rdsk/c6t5000CCA00A350350d0

/dev/rdsk/c1t3d0 /dev/rdsk/c6t5000CCA00A33F218d0

/dev/rdsk/c1t4d0 /dev/rdsk/c6t5000CCA00A3468E0d0

/dev/rdsk/c1t5d0 /dev/rdsk/c6t5000CCA00A34E950d0

This is when we enable multipathing on all devices

say yes to reboot

To enable Solaris I/O Multipathing on multipath-capable Fibrechannel controller ports,

enter:

bash-3.00# stmsboot -D fp -e

WARNING: This operation will require a reboot.

Do you want to continue ? [y/n] (default: y)

The changes will come into effect after rebooting the system.

Reboot the system now ? [y/n] (default: y)

Before multipathing, you would see four copies of each disk in format. Afterwards, you'll just

see the one copy.

It assigns the next available controller ID, and makes up some horrendously long target number.

For example:

Filesystem kbytes used avail capacity Mounted on

/dev/dsk/c6t600C0FF000000000086AB238B2AF0600d0s5 697942398 20825341 670137634 4% /test

At this point we see 4 devices ( 6-9 as listed above)

Before MPxIO

bash-3.00# format

Searching for disks...done

AVAILABLE DISK SELECTIONS:

0. c1t0d0

/pci@0/pci@0/pci@2/scsi@0/sd@0,0

1. c1t1d0

/pci@0/pci@0/pci@2/scsi@0/sd@1,0

2. c1t2d0

/pci@0/pci@0/pci@2/scsi@0/sd@2,0

3. c1t3d0

/pci@0/pci@0/pci@2/scsi@0/sd@3,0

4. c1t4d0

/pci@0/pci@0/pci@2/scsi@0/sd@4,0

5. c1t5d0

/pci@0/pci@0/pci@2/scsi@0/sd@5,0

3PAR lists 4 devices

6. c3t25510002AC000AE2d0 <3PARdata-VV-0000-2.00TB>

/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1/fp@0,0/ssd@w25510002ac000ae2,0

7. c3t27540002AC000AE2d0 <3PARdata-VV-0000-2.00TB>

/pci@0/pci@0/pci@8/pci@0/pci@2/SUNW,emlxs@0,1/fp@0,0/ssd@w27540002ac000ae2,0

8. c5t24510002AC000AE2d0 <3PARdata-VV-0000-2.00TB>

/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0/ssd@w24510002ac000ae2,0

9. c5t26540002AC000AE2d0 <3PARdata-VV-0000-2.00TB>

/pci@0/pci@0/pci@8/pci@0/pci@a/SUNW,emlxs@0,1/fp@0,0/ssd@w26540002ac000ae2,0

After Enabling MPxIO

bash-3.00# format

Searching for disks...done

AVAILABLE DISK SELECTIONS:

0. c6t5000C500091B7D93d0

/scsi_vhci/disk@g5000c500091b7d93

1. c6t5000C500091B90B3d0

/scsi_vhci/disk@g5000c500091b90b3

2. c6t5000CCA00A33F218d0

/scsi_vhci/disk@g5000cca00a33f218

3. c6t5000CCA00A34E950d0

/scsi_vhci/disk@g5000cca00a34e950

4. c6t5000CCA00A3468E0d0

/scsi_vhci/disk@g5000cca00a3468e0

5. c6t5000CCA00A350350d0

/scsi_vhci/disk@g5000cca00a350350

3PAR list one device instead of 4 as above

6. c6t50002AC000220AE2d0 <3PARdata-VV-0000-2.00TB>

/scsi_vhci/ssd@g50002ac000220ae2

You will notice all the above devices changed to /scsi_vhci/XXXXX instead of /pci@0/pci@0/

Also compare your previous probe output to one below

bash-3.00# luxadm probe

No Network Array enclosures found in /dev/es

Found Fibre Channel device(s):

Node WWN:2ff70002ac000ae2 Device Type:Disk device

Logical Path:/dev/rdsk/c6t50002AC000220AE2d0s2

Below stmsboot output shows you how all interfaces 245, 255, 265 and 275 has one pseudo name as

50002AC000220AE2

bash-3.00# stmsboot -L

non-STMS device name STMS device name

------------------------------------------------------------------

/dev/rdsk/c1t0d0 /dev/rdsk/c6t5000C500091B90B3d0

/dev/rdsk/c1t1d0 /dev/rdsk/c6t5000C500091B7D93d0

/dev/rdsk/c1t2d0 /dev/rdsk/c6t5000CCA00A350350d0

/dev/rdsk/c1t3d0 /dev/rdsk/c6t5000CCA00A33F218d0

/dev/rdsk/c1t4d0 /dev/rdsk/c6t5000CCA00A3468E0d0

/dev/rdsk/c1t5d0 /dev/rdsk/c6t5000CCA00A34E950d0

/dev/rdsk/c5t26540002AC000AE2d0 /dev/rdsk/c6t50002AC000220AE2d0

/dev/rdsk/c5t24510002AC000AE2d0 /dev/rdsk/c6t50002AC000220AE2d0

/dev/rdsk/c3t27540002AC000AE2d0 /dev/rdsk/c6t50002AC000220AE2d0

/dev/rdsk/c3t25510002AC000AE2d0 /dev/rdsk/c6t50002AC000220AE2d0

Step #4.

using mpathadm commands

To List all your initiators

bash-3.00# mpathadm list initiator-port

Initiator Port: 508002000081b570

Initiator Port: ,4000002a00ff

Initiator Port: 10000000c97bfb4c

Initiator Port: 10000000c97d2c70

Initiator Port: 10000000c97bfb4d -> connected HBA

Initiator Port: 10000000c97d2c71 -> connected HBA

bash-3.00# mpathadm list lu

List all LUNs

bash-3.00# mpathadm list lu

/dev/rdsk/c6t5000C500091B90B3d0s2

Total Path Count: 1

Operational Path Count: 1

/dev/rdsk/c6t5000C500091B7D93d0s2

Total Path Count: 1

Operational Path Count: 1

/dev/rdsk/c6t5000CCA00A350350d0s2

Total Path Count: 1

Operational Path Count: 1

/dev/rdsk/c6t5000CCA00A33F218d0s2

Total Path Count: 1

Operational Path Count: 1

/dev/rdsk/c6t5000CCA00A3468E0d0s2

Total Path Count: 1

Operational Path Count: 1

/dev/rdsk/c6t5000CCA00A34E950d0s2

Total Path Count: 1

Operational Path Count: 1

/dev/rdsk/c6t50002AC000220AE2d0s2 <--- pesudo LUN we are interested in

Total Path Count: 4

Operational Path Count: 4

This shows how this pesudo device using the different path underneath

bash-3.00# mpathadm show lu /dev/rdsk/c6t50002AC000220AE2d0s2

Logical Unit: /dev/rdsk/c6t50002AC000220AE2d0s2

mpath-support: libmpscsi_vhci.so

Vendor: 3PARdata

Product: VV

Revision: 0000

Name Type: unknown type

Name: 50002ac000220ae2

Asymmetric: no

Current Load Balance: round-robin

Logical Unit Group ID: NA

Auto Failback: on

Auto Probing: NA

Paths:

Initiator Port Name: 10000000c97d2c71

Target Port Name: 26540002ac000ae2

Override Path: NA

Path State: OK

Disabled: no

Initiator Port Name: 10000000c97d2c71

Target Port Name: 24510002ac000ae2

Override Path: NA

Path State: OK

Disabled: no

Initiator Port Name: 10000000c97bfb4d

Target Port Name: 27540002ac000ae2

Override Path: NA

Path State: OK

Disabled: no

Initiator Port Name: 10000000c97bfb4d

Target Port Name: 25510002ac000ae2

Override Path: NA

Path State: OK

Disabled: no

Target Ports:

Name: 26540002ac000ae2

Relative ID: 0

Name: 24510002ac000ae2

Relative ID: 0

Name: 27540002ac000ae2

Relative ID: 0

Name: 25510002ac000ae2

Relative ID: 0