Storage system administrators will often come across situations like performance degrade and other related issues. Though we have sophisticated tools to analyze the storage array, the best method is to use the native commands to understand the IO load and port statistics. In this article we will try to explore the possible options of HP 3PAR StoreServ performance troubleshooting.
All performance concerns are related to IOPS and throughput(MB/s), hence we should follow a clear sequence while performing performance troubleshooting on 3PAR systems. Basically it is categorized in to 6 areas and they are,- Host Vlun properties and statistics
- Host port performance
- Node chache settings
- Virtual volume settings and performance
- Disk port settings and characteristics
- Physical disks
The commands used generally are statvlun, statport, statcmp, statvv and statpd. In the following section, we will explore the usage and options for each commands.
Statvlun is used for collecting the performance statistics for virtual LUN which are exported to host. statport collect the statistics for Fibre Channel ports available in the storage array. Statcmp gives the statistics of cache memory pages available in the nodes. Statpd gives the statistics of physical disks available in the storage system and statvv gives the statistics of virtual volumes which are present in a Storeserv storage system.
Basically, stat command category holds all the commands which are essential for collecting the performance statistics of 3PAR StoreServ arrays.
All Stats commands continue to run until the return key is pressed or the -iter option is specified.
The statvlun command displays performance statistics for virtual luns which are attached to hosts.
The most common options used along with this commands are,
-host (Shows only VLUNs exported to the specified host(s))
-rw (Specifies that the display includes separate read and write data.)
-iter ( Specifies that the statistics are to stop after the indicated number of iterations)
-ni ( Specifies that statistics for only non-idle devices are displayed.)
mylab3par cli% statvlun -ni -rw
22:04:31 03/14/2016 r/w I/O per second KBytes per sec Svt ms IOSz KB
Lun VVname Host Port Cur Avg Max Cur Avg Max Cur Avg Cur Avg Qlen
1 DATA_NR_CP_CLxx_VMFS001 mylabesx01 0:1:1 r 11 11 11 546 546 546 1.34 1.34 49.9 49.9 -
1 DATA_NR_CP_CLxx_VMFS001 mylabesx01 0:1:1 w 1 1 1 2 2 2 0.26 0.26 2.3 2.3 -
1 DATA_NR_CP_CLxx_VMFS001 mylabesx01 0:1:1 t 12 12 12 549 549 549 1.25 1.25 45.9 45.9 0
3 DATA_NR_CP_CLxx_VMFS003 mylabesx01 0:1:1 r 0 0 0 0 0 0 0.00 0.00 0.0 0.0 -
3 DATA_NR_CP_CLxx_VMFS003 mylabesx01 0:1:1 w 1 1 1 4 4 4 0.24 0.24 2.9 2.9 -
3 DATA_NR_CP_CLxx_VMFS003 mylabesx01 0:1:1 t 1 1 1 4 4 4 0.24 0.24 2.9 2.9 0
2 DATA_NR_CP_CLxx_VMFS002 mylabesx01 1:1:1 r 0 0 0 0 0 0 0.00 0.00 0.0 0.0 -
2 DATA_NR_CP_CLxx_VMFS002 mylabesx01 1:1:1 w 2 2 2 40 40 40 0.27 0.27 16.0 16.0 -
2 DATA_NR_CP_CLxx_VMFS002 mylabesx01 1:1:1 t 2 2 2 40 40 40 0.27 0.27 16.0 16.0 0
2 DATA_NR_CP_CLxx_VMFS002 mylabesx02 1:1:1 r 29 29 29 1009 1009 1009 0.92 0.92 34.4 34.4 -
2 DATA_NR_CP_CLxx_VMFS002 mylabesx02 1:1:1 w 20 20 20 344 344 344 0.29 0.29 16.9 16.9 -
2 DATA_NR_CP_CLxx_VMFS002 mylabesx02 1:1:1 t 50 50 50 1354 1354 1354 0.66 0.66 27.2 27.2 0
3 DATA_NR_CP_CLxx_VMFS003 mylabesx02 1:1:1 r 0 0 0 0 0 0 0.00 0.00 0.0 0.0 -
3 DATA_NR_CP_CLxx_VMFS003 mylabesx02 1:1:1 w 1 1 1 41 41 41 0.27 0.27 27.5 27.5 -
3 DATA_NR_CP_CLxx_VMFS003 mylabesx02 1:1:1 t 1 1 1 41 41 41 0.27 0.27 27.5 27.5 0
1 DATA_NR_CP_CLxx_VMFS001 mylabesx02 1:1:2 r 3 3 3 37 37 37 3.76 3.76 12.3 12.3 -
1 DATA_NR_CP_CLxx_VMFS001 mylabesx02 1:1:2 w 45 45 45 268 268 268 0.34 0.34 5.9 5.9 -
1 DATA_NR_CP_CLxx_VMFS001 mylabesx02 1:1:2 t 48 48 48 305 305 305 0.55 0.55 6.3 6.3 0
---------------------------------------------------------------------------------------------------------
6 r 43 43 1592 1592 1.22 1.22 36.8 36.8 -
6 w 72 72 700 700 0.32 0.32 9.7 9.7 -
6 t 115 115 2293 2293 0.66 0.66 19.9 19.9 0
Press the enter key to stop...
From the about output, you can easily read Service time in milliseconds, I/O size in KBytes and Queue length. This helps you to identify where exactly the performance hit is happening.
statport command shows storeserv port statistics for all the available ports in the system. The most common options used are,
-host|-disk|-rcfc|-peer (Specifies to display only host ports or only disk ports
or Fibre Channel Remote Copy configured ports, or only Fibre Channel ports for Data Migration.)
-both | -ctl | -data (Specifies that both control and data transfers are displayed(-both) or only control transfers are displayed (-ctl), or only data transfers are displayed (-data). If this option is not specified, only data transfers are displayed.)
mylab3par cli% statport -iter 1
22:20:06 03/14/2016 r/w I/O per second KBytes per sec Svt ms IOSz KB
Port D/C Cur Avg Max Cur Avg Max Cur Avg Cur Avg Qlen
0:0:1 Data t 307 307 307 5612 5612 5612 6.77 6.77 18.3 18.3 0
0:0:2 Data t 323 323 323 6611 6611 6611 6.60 6.60 20.5 20.5 0
0:1:1 Data t 48 48 48 1311 1311 1311 1.05 1.05 27.1 27.1 0
0:1:2 Data t 48 48 48 2782 2782 2782 0.41 0.41 58.2 58.2 0
1:0:1 Data t 175 175 175 2604 2604 2604 6.19 6.19 14.9 14.9 0
1:0:2 Data t 183 183 183 2527 2527 2527 6.18 6.18 13.8 13.8 0
1:1:1 Data t 74 74 74 3800 3800 3800 0.74 0.74 51.5 51.5 0
1:1:2 Data t 116 116 116 2997 2997 2997 0.83 0.83 25.9 25.9 0
--------------------------------------------------------------------------------
8 Data t 1274 1274 28243 28243 5.22 5.22 22.2 22.2 0
mylab3par cli% statport -both -iter 1
22:21:50 03/14/2016 r/w I/O per second KBytes per sec Svt ms IOSz KB
Port D/C Cur Avg Max Cur Avg Max Cur Avg Cur Avg Qlen
0:0:1 Data t 312 312 312 6697 6697 6697 6.24 6.24 21.5 21.5 0
0:0:1 Ctrl t 0 0 0 0 0 0 0.00 0.00 0.0 0.0 0
0:0:2 Data t 306 306 306 6830 6830 6830 6.40 6.40 22.3 22.3 0
0:0:2 Ctrl t 0 0 0 0 0 0 0.00 0.00 0.0 0.0 0
0:1:1 Data t 105 105 105 801 801 801 0.49 0.49 7.7 7.7 0
0:1:1 Ctrl t 0 0 0 0 0 0 0.08 0.08 0.0 0.0 0
0:1:2 Data t 162 162 162 3422 3422 3422 1.02 1.02 21.1 21.1 0
0:1:2 Ctrl t 0 0 0 0 0 0 0.00 0.00 0.0 0.0 0
1:0:1 Data t 296 296 296 5441 5441 5441 8.16 8.16 18.4 18.4 0
1:0:1 Ctrl t 0 0 0 0 0 0 0.00 0.00 0.0 0.0 0
1:0:2 Data t 308 308 308 5934 5934 5934 7.76 7.76 19.3 19.3 0
1:0:2 Ctrl t 0 0 0 0 0 0 0.00 0.00 0.0 0.0 0
1:1:1 Data t 1 1 1 2 2 2 0.24 0.24 1.5 1.5 0
1:1:1 Ctrl t 0 0 0 0 0 0 0.00 0.00 0.0 0.0 0
1:1:2 Data t 66 66 66 444 444 444 1.47 1.47 6.7 6.7 0
1:1:2 Ctrl t 0 0 0 0 0 0 0.00 0.00 0.0 0.0 0
--------------------------------------------------------------------------------
8 Data t 1556 1556 29572 29572 5.80 5.80 19.0 19.0 0
8 Ctrl t 0 0 0 0 0.08 0.08 0.0 0.0 0
Click here for part 2...
No comments :
Post a Comment