I used our EqualLogic iSCSI SAN for well over a year without any effective way to monitor the array’s performance. We have virtualized over 30 servers, including some fairly high-I/O servers like Lotus Domino, and while the virtual platform usually performs very well it was difficult to see what was going on when a slowdown occured. Enter EqualLogic SAN HeadQuarters.
SAN HeadQuarters installs on an application server and logs information received via SNMP from the EqualLogic Group. Firmware 3.3.3 or newer is required to use this software, and 32 and 64 bit versions of Windows XP, Windows Vista, Windows Server 2003, and Windows Server 2008 are supported. For monitoring from your workstation, a client-only installation allows you to view the logged information without accessing the console of the server where you have installed SAN HeadQuarters.
We only have a single-array SAN for now so there are features in this application that I will not cover, like statistics on Inbound Replicas. The main application is layed out in a familiar three-pane format, with a hierarchical menu on the left, timeline and horizontal menus at the top, and statistics content in the main pane. Clicking a Group from the left menu brings you to a general dashboard, showing the current IOPS and other stats like latency and throughput. Drilling down further into the group, you can view Capacity, Combined Graphs, Hardware/Firmware info, I/O, and Network statistics. From each of those views you can narrow down by Pool, Member, Volumes, Inbound Replicas, Network Ports, and Disks. This flexibility is particularly handy when your array’s IOPS is through the roof, because you can open the I/O view from the left menu, then open the Volumes filter from the horizontal menu, and then sort by Avg. I/O. In just a few seconds, I’ve narrowed down the issue and have the information I need to to balance the disk I/O load if required. Additional screenshots at the end of this article.
I’ve been using this for only one week, and in conjunction with Storage vMotion I’ve already been able to isolate and remediate two significant performance bottlenecks in our system. Most of the time I use the Combined Graphs view because it gives me all of the important statistics on one screen, including avg. I/O size (KB), avg. IOPS, avg. latency (ms), avg I/O throughput (MB/sec), % retransmit, and network throughput (MB/sec.). This one screen paints a clear picture of my storage environment, and drilling down to view specific stats is just a few mouse clicks away.
I’m inclined to say that this little piece of software is as great as the EqualLogic platform itself. If you’re not using this yet, you should be. If you’re not at firmware v3.3.3 or higher you should upgrade because it’s easy, and when you’re done you can start using SAN HeadQuarters.
SAN HeadQuarters can be downloaded from the EqualLogic Support Site, but you will have to register/login.




Twitter
Josh-
Can you highlight what the two performance bottlenecks were that you found? We use a similar setup and have not started using SANHQ yet, although we plan to start soon.
Hi Rob,
The bottlenecks were pretty basic issues but I probably wouldn’t have seen them without SANHQ. We have several volumes with multiple VMs on each, which under high I/O conditions can lead to network contention and slow things down on the disk side. We had this particular issue with one higher-I/O VM and just moved it to another volume to balance the I/O load. The other issue we had was that two higher-I/O VMs on the same box were saturating the bandwidth on the HBA/Network side of the host they were on. It became clear that these two VMs should not be on the same host/iSCSI channel, so I set a DRS rule to keep them on different hosts. Problem solved.
These issues were nailed down in a matter of 5 minutes with the help of SANHQ, where it would have taken me quite a bit longer to get things ironed out using only the performance graphs in vCenter. I hope this helps!
Yes, that does help, and a good demonstration of why SANHQ is needed. Thanks!
Josh – thanks for this. I’m an analyst currently writing a research profile on SAN HQ and would love to ask you a few questions about your experience with the product, if you’d care to share. Anonymously, of course
Dave Bartoletti, dave at tanejagroup