3PAR remote copy setup

Remote copy status and troubleshooting

before starting remote copy steps:

Setup remotecopy ip interface and licenses

DataCenter1 remote copy port and IP

2:6:1 - 19.162.27.7

3:6:1 - 19.162.27.8

3par2 cli% showport -rcip

N:S:P State ---HwAddr---     IPAddr         Netmask     Gateway  MTU  Rate Duplex AutoNeg

2:6:1 ready 0002AC69249B 19.162.27.7 255.255.255.240 19.162.27.14 1500 1Gbps   Full     Yes

3:6:1 ready 0002AC6A2867 19.162.27.8 255.255.255.240 19.162.27.14 1500 1Gbps   Full     Yes

DataCenter2 remote copy port and IP

2:6:1 - 19.220.33.145

3:6:1 - 19.220.33.146

3par1 cli% showport -rcip

N:S:P   State ---HwAddr---        IPAddr         Netmask       Gateway  MTU  Rate Duplex AutoNeg

0:6:1 offline 0002AC5410C7             -               -             -    -   n/a    n/a     n/a

1:6:1 offline 0002AC540DFC             -               -             -    -   n/a    n/a     n/a

2:6:1   ready 0002AC540CE6 19.220.33.145 255.255.255.240 19.220.33.158 1500 1Gbps   Full     Yes

3:6:1   ready 0002AC540C23 19.220.33.146 255.255.255.240 19.220.33.158 1500 

1Gbps   Full     Yes

F400 ports:

T800 ports:

7 steps to setup 3PAR remote copy

Step #1. Starting remotecopy

Step #2. Setting up copy target

Step #3. Checking the links

Step #4. Creating Volume Groups for Synchronous Long Distance Remote Copy

Step #5. Add luns to the remote volume group

Step #6. Status

Step #7. Start replication

Please review 3PAR Remote Copy Software User's Guide 3.1.2 under attachment for more details.

Step #1. Starting remotecopy

Starting rcopy on source

3par1 cli% startrcopy

Starting rcopy on target

3par2 cli% startrcopy

Step #2. Setting up copy target

Setting Up the Primary System

creatercopytarget IP : :

3par1 cli% creatercopytarget 3par2 IP 2:6:1:19.220.33.145 3:6:1:19.220.33.146

Setting Up the Backup Systems

creatercopytarget IP : :

3par2 cli% creatercopytarget 3par1 IP 2:6:1:19.162.27.7 3:6:1:19.162.27.8

Step #3. Checking the links

Checking the Links between Systems

3par1 cli% showrcopy

Remote Copy System Information

Status: Started, Normal

Target Information

Name; ID Type Status Options Policy

3par2 1 IP; ready; mirror_config

Link Information

Target Node Address Status Options

3par2 2:6:1 19.220.33.145 Up

3par2 3:6:1 19.220.33.146 Up

receive 2:6:1 receive Up

receive 3:6:1 receive Up

3par2 cli% showrcopyRemote Copy System Information

Status: Started, Normal

Target Information

Name               ID Type Status Options Policy

3par1 1  IP   ready          mirror_config

Link Information

Target             Node  Address    Status Options

3par1 2:6:1 19.162.27.7 Up

3par1 3:6:1 19.162.27.8 Up

receive            2:6:1 receive    Up

receive            3:6:1 receive    Up

Step #4. Creating Volume Groups for Synchronous Long Distance Remote Copy

creatercopygroup :

automatic synchronization.sync command

3par1 cli% creatercopygroup group-remotecopy1 3par2:sync

3par1 cli% creatercopygroup group-bcom-nearline 3par2:periodicsync| periodic

(for synchrouns and async mode)

(Optional).

If you are creating a volume group that uses periodic modeperiod commands

setrcopygroup period s|m|h|d

or

setrcopygroup period s|m|h|d -pat setrcopygroup period 1h 3par2 group-remotecopy1

setrcopygroup period 1h 3par2 -pat  

Auto recover setting:

setrcopygroup pol auto_recover

setrcopygroup pol auto_recover -pat

Step #5. Add luns to the remote volume group

On the source, add the pre-existing virtual volume(s) to the volume group

admitrcopyvv 

admitrcopyvv -pat

admitrcopyvv VV-NL-RCOPYTEST4-R5 group-remotecopy1 3par2:VV-NL-RCOPYTEST1-R5

admitrcopyvv -pat VV-NL-RCOPYTEST1-R5 group-remotecopy1 3par2:VV-NL-RCOPYTEST1-R5 

3par1 cli% admitrcopyvv VV-NL-RCOPYTEST4-R5 group-remotecopy1 3par2:VV-NL-RCOPYTEST1-R5

3par1 cli% admitrcopyvv VV-NL-RCOPYTEST5-R5 group-remotecopy1 3par2:VV-NL-RCOPYTEST2-R5

3par1 cli% admitrcopyvv VV-NL-RCOPYTEST6-R5 group-remotecopy1 3par2:VV-NL-RCOPYTEST3-R5 

Step #6. Status

3par1 cli% showrcopy

Remote Copy System Information

Status: Started, Normal

Target Information

Name   ID Type Status Options Policy

3par2 1  IP   ready          mirror_config

Link Information

Target  Node  Address       Status Options

3par2  2:6:1 19.220.33.145 Up

3par2  3:6:1 19.220.33.146 Up

receive 2:6:1 receive       Up

receive 3:6:1 receive       Up

Group Information

Name              Target     Status   Role       Mode     Options

group-remotecopy1 3par2     New      Primary    Sync

  LocalVV             ID   RemoteVV            ID   SyncStatus    LastSyncTime

  VV-NL-RCOPYTEST4-R5 29   VV-NL-RCOPYTEST1-R5 125  New           NA

  VV-NL-RCOPYTEST5-R5 30   VV-NL-RCOPYTEST2-R5 167  New           NA

  VV-NL-RCOPYTEST6-R5 31   VV-NL-RCOPYTEST3-R5 168  New           NA 

3par1 cli% showrcopy

Remote Copy System Information

Status: Started, Normal

Target Information

Name   ID Type Status Options Policy

3par2 1  IP   ready          mirror_config

Link Information

Target  Node  Address       Status Options

3par2  2:6:1 19.220.33.145 Up

3par2  3:6:1 19.220.33.146 Up

receive 2:6:1 receive       Up

receive 3:6:1 receive       Up

Group Information

Name              Target     Status   Role       Mode     Options

group-remotecopy1 3par2     Started  Primary    Sync

  LocalVV             ID   RemoteVV            ID   SyncStatus    LastSyncTime

  VV-NL-RCOPYTEST4-R5 29   VV-NL-RCOPYTEST1-R5 125  Syncing (0%)  NA

  VV-NL-RCOPYTEST5-R5 30   VV-NL-RCOPYTEST2-R5 167  Syncing (0%)  NA

  VV-NL-RCOPYTEST6-R5 31   VV-NL-RCOPYTEST3-R5 168  Syncing (0%)  NA

3par1 cli% showtask

  Id Type             Name                Status Phase   Step ----StartTime------ -FinishTime- -Priority-

9404 remote_copy_sync VV-NL-RCOPYTEST6-R5 active   2/3 0/1024 2012-09-27 19:24:15 EDT -            n/a

9405 remote_copy_sync VV-NL-RCOPYTEST5-R5 active   2/3 0/1024 2012-09-27 19:24:15 EDT -            n/a

9406 remote_copy_sync VV-NL-RCOPYTEST4-R5 active   2/3 0/1024 2012-09-27 19:24:15 EDT -            n/a 

3par1 cli% showtask

  Id Type             Name                Status Phase Step ----StartTime---- ---FinishTime------ -Priority-

9404 remote_copy_sync VV-NL-RCOPYTEST6-R5 done     --  -- 2012-09-27 19:24:15 EDT 2012-09-27 19:27:37 EDT n/a

9405 remote_copy_sync VV-NL-RCOPYTEST5-R5 done     --  -- 2012-09-27 19:24:15 EDT 2012-09-27 19:27:43 EDT n/a

9406 remote_copy_sync VV-NL-RCOPYTEST4-R5 done     --  -- 2012-09-27 19:24:15 EDT 2012-09-27 19:27:43 EDT n/a

Step #7. Start replication

Starting Initial Replication: Copying Data Directly from Primary Volume Groups

startrcopygroup 

startrcopygroup group-remotecopy1

Is there way to find dd status on Solaris?

Well, this is all started when we had to move around 24TB data between the 2 data centers and we decided to use dd command. I guess it is much easier on Linux where you have a way to can use progress bar etc. and dd standard out gives you percentage finished. Unfortunately I couldn't find anything fancy with Solaris.

If you are using a dd command as below and you want find out when it will finish.What options do you have?
dd if=/share/disks/disk-402.img | ssh root@server "dd of=/dev/dsk/c6t50002AC000AE0AE2d0s0"


One of the option is this which is I found on net, but it didn't work for me. It does sends a user signal 1 to the process but it kills the dd process, that means the process take a standard/default kill instead of USR1

Printing dd status

I recently used dd to zero out some hard drives on my Fedora Core workstation, and found that this operation takes a good deal of time (even when large blocksizes are used, it still takes a while). The dd utility doesn’t report status information by default, but when fed a SIGUSR1 signal it will dump the status of the current operation:

$ dd if=/dev/zero of=/dev/hda1 bs=512 &

$ kill -SIGUSR1 1749
1038465+0 records in
1038465+0 records out
531694080 bytes (532 MB) copied, 11.6338 seconds, 45.7 MB/s

watch -n 10 kill -USR1

It still amazes me how much stuff I have left to learn about the utilities I use daily.

The other option was to see if there is any way we can from the storage side what is total blocks written so far and how much time it will take to write the whole 24TB on that lun. The problem with fully provisioned lun is that the storage allocates all the blocks to the lun as soon as its provisioned. And you cannot see anything under that block, there may be a way to see this if its thin provisioned.



The easiest option is to use iostat and see how fast you are writing to the device and do the math

-bash-3.00#  iostat -xnmMpz 1
                    extended device statistics
    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    1.0    0.0    0.0  0.0  0.0    0.0    5.5   0   1 c1t0d0
    0.0    1.0    0.0    0.0  0.0  0.0    0.0    5.5   0   1 c1t0d0s0 (/)
  982.5  982.5    7.7    7.7  0.0  1.1    0.0    0.5   1  62 c6t50002AC000B10AE2d0
  982.5  982.5    7.7    7.7  0.0  1.1    0.0    0.5   1  62 c6t50002AC000B10AE2d0s0
                    extended device statistics
    r/s    w/s   Mr/s   Mw/s wait actv wsvc_t asvc_t  %w  %b device
  989.6  989.6    7.7    7.7  0.0  1.1    0.0    0.5   1  61 c6t50002AC000B10AE2d0
  989.6  989.6    7.7    7.7  0.0  1.1    0.0    0.5   1  61 c6t50002AC000B10AE2d0s0


As we see here we are writing at 7.7MB/s.
Now if we are writing 1TB it should take around 36hrs and 24TB would take around 865 hrs.

How to rename QFS fsname?

1: umount /new

2: vi /etc/vfstab     change new to new4
3: vi mcf      change new to new4 in all places.

cat /etc/opt/SUNWsamfs/mcf

new4                                            500    ms      new4   on        shared
/dev/dsk/c6t600A0B80004715700000074D47FD082Ed0s0 501    md      new4   on
/dev/dsk/c6t600A0B8000471570000007D247FD0A22d0s0 502    md      new4   on
/dev/dsk/c6t600A0B80004715700000086F49F1D551d0s0 504    md      new4   on


4: samd config
5: samfsck -F -R new4          (this will change the file system name to new4)

Example:
bash-3.00# samfsck -F -R new4
name:     bcombackups       version:     2A    shared
First pass
ALERT:  ino 1743721017.967461289, Object flag set, should be clear, meta_flag 1, size 8581431105406506118
ALERT:  ino -796177020.1556009446, Object flag set, should be clear, meta_flag 0, size 3201155233938255577
ALERT:  ino 22367104.671481968, Object flag set, should be clear, meta_flag 0
ALERT:  ino 700653835.-1820544200, Object flag set, should be clear, meta_fla
ALERT:  ino -2107630336.-937590376, Object flag set, should be clear, meta_fl
ALERT:  ino 738287680.101681192, Object flag set, should be clear, meta_flag
ALERT:  ino 503702500.2047090706, Object flag set, should be clear, meta_flag
ALERT:  ino -445008776.-1635288612, Object flag set, should be clear, meta_fl
ALERT:
Second pass
Third pass
ALERT:  Invalid block:        ino 1853 marked damaged
NOTICE: ino 2214.1,     Repaired link count from 688 to 674
ALERT:  Invalid block:        ino 4218 marked damaged
ALERT:  Invalid block:        ino 5893 marked damaged
ALERT:  Invalid block:        ino 8836 marked damaged
ALERT:  Invalid block:        ino 12528 marked damaged
ALERT:  Invalid block:        ino 16594 marked damaged
ALERT:  Invalid block:        ino 18916 marked damaged

Inodes processed: 8066560

total data kilobytes       = 5858959296
total data kilobytes free  = 1617932544
NOTICE: Reclaimed 332916064256 bytes

Another example:

root@dam-app2 # samfsck -F -R new4

name:     opsbackup       version:     2A    shared

First pass

Second pass

Third pass

Inodes processed: 2048

total data kilobytes       = 3382270208

total data kilobytes free  = 3382259904

6: mount new4

new4                  4.7T    34G   4.7T     1%    /nearline3

Netgear N600 and DOCSIS 3.0

I've been in IT Infrastructure and Operations world for more 15 years now, helping companies and folks to resolve all sorts of performance issues. Here is my horrible experience with internet connection @ home :)

I have been using ATT for almost a decade and finally decided to go to Comcast.
I upgraded to a DOCSIS 3.0 Motorola modem SB6120 router and also bought a Netgear N600 wireless dual band router for Comcast internet to avoid any monthly charges at all.

Configured and customized router to whatever I wanted, internet all good. By the way I have CAT 6 structured wiring at my home covers almost all the rooms.

The problem I was having was that when connecting to the internet through the wifi,
there was a lag and websites would load way slower.

Symptoms:
Frankly I didn't notice any lags, however my wife started complaining that her vpn drops connection after we switched to Comcast. And its horribly slow compared to AT&T. We were ATT Pro which was 3mbps and Comcast promised at least 10 mbps. Speedtest confirm its double the speed, it never resulted in less than 22mbps. Now, mathematically there is no way internet will be slow. So, I ignore the problem. The complain continues so finally I decided to spend some time with issue.

I still cannot see the difference visually with google.com or yahoo.com

But, yes it can be better there is some lag of 3-5 seconds while pulling the site which may be annoying to some.
The problem became very obvious when I started a download, it was 70KB/s, that means my internet speed is only 560kbps


Troubleshooting steps:
#Packet loss:
The first thing i noticed was the packet loss. While pinging the router after every 25 packets of 1ms latency there are 4 packets which is 100ms. Called Netgear, spend couple of hrs. with tech support, they tried lots of combination with no security, changing to different channels, (I believe 6 and 11 are reliable :-)), I had already downloaded inSSIDer to see if there is any interference with channels neighbors are using, recycle the modem, re flash the modem etc. finally changed default fragmentation length from 2346 to 2345 and there was some improvement in packet loss. But ultimately gave up.

#Wireless speed test
Tried a speedtest using wireless, it says 22Mbps however download sucks (50-100 KB/s)

#Wired speedtest
Tried a speedtest directly connecting to router, it says 22Mbps however download still sucks (50-100 KB/s)

#Connected directly to modem
Now connected directly to modem, speedtest is good 27mbps and so is the download speed (2.4 MB/s)
so now the problem boils down to router.

I google'd the Netgear problem with Comcast and found that we have disable WWM for Netgear. It was already disabled when I had initially configured it. Again played with few things no change. There were lot of suggestion to change the router and go with Comcast router.

However, I decided to reset the router again. used all the default option to connect wirelessly.

What's the download speed 4MB/s !!

Now I have backup of N600 old config and new config which works with expected speed. The problem is the netgear.cfg is encrypted and so do we want to solve this problem now. May be .. whenever time permits next time..

I took me more than a month to solve my home internet speed :(