Comments on: VMWare VI and iSCSI, a match made in…wait, what? http://joshcurrier.com/2008/09/vmware-vi-and-iscsi-a-match-made-inwait-what/ Blogging about stuff. Wed, 14 Jul 2010 13:47:33 +0000 hourly 1 http://wordpress.org/?v=3.0 By: Josh http://joshcurrier.com/2008/09/vmware-vi-and-iscsi-a-match-made-inwait-what/comment-page-1/#comment-3932 Josh Mon, 05 Oct 2009 17:26:54 +0000 http://joshcurrier.com/?p=17#comment-3932 Hi Shawn, I haven't done much testing since getting the HBAs. The performance is better but not what I would like to see from this system. I do plan to test further but I'm spread pretty thin these days and it might be a while before this particular issue finds its way to the top of the list again. Thanks for the insight on Server 2008 and Win7, I'll keep that in mind when I get back to testing! Hi Shawn,
I haven’t done much testing since getting the HBAs. The performance is better but not what I would like to see from this system. I do plan to test further but I’m spread pretty thin these days and it might be a while before this particular issue finds its way to the top of the list again. Thanks for the insight on Server 2008 and Win7, I’ll keep that in mind when I get back to testing!

]]> By: Shawn Lebbon http://joshcurrier.com/2008/09/vmware-vi-and-iscsi-a-match-made-inwait-what/comment-page-1/#comment-3928 Shawn Lebbon Sat, 03 Oct 2009 13:05:30 +0000 http://joshcurrier.com/?p=17#comment-3928 I'd be curious if you ever did any more testing or trying to isolate this issue? I have a nearly identical setup to yours before you got the Qlogic HBAs, (VMware ESX 3.5, Equallogic san, both Broadcom built-in and pci-e intel gig-e cards, etc) except we have old dell gigabit switches (although we don't think they're the main performance problem), even right down to having the Dell servers. We notice similar issues never getting more than 20MB/s or so of speeds from our virtualized, SAN-attached servers. Although general performance for small random I/O has still been OK, as we have exchange, SQL, file servers, etc all running off this setup concurrently and it's all still a littlle faster than it used to be before SANs and Virtulization...although maybe it's supposed to be A LOT faster and we're just not seeing it? However there's one key factor we discovered in testing...Windows. We're still mostly an XP/2003 shop, but have just started playing with some Win7/Svr 2008 stuff. To an XP machine from 2003 file server, I can't get more than 20MB-27MB/s download of a large file. Even trying to eliminate the local disk, by downloading to a RAM-disk or opening CAD files direct from the server. Funny issue though is we know our Backups pull in close to 80MB/s, which is really starting to show good utilization of a gigabit pipe...and they're running as clients on the servers, sending out to the tape library server over the LAN links, so we kind of figure the SAN/VMware infrastructure isn't likely at fault... Then we tested with Windows 7 on an older Laptop. And got much closer to expected speeds! 30-40MB/s coping large file to the local drive, and even higher opening/loading large file into memory... So I wonder if the problem isn't more on the client (server?) OS side? I'd be interested in hearing what you think. Thanks. I’d be curious if you ever did any more testing or trying to isolate this issue? I have a nearly identical setup to yours before you got the Qlogic HBAs, (VMware ESX 3.5, Equallogic san, both Broadcom built-in and pci-e intel gig-e cards, etc) except we have old dell gigabit switches (although we don’t think they’re the main performance problem), even right down to having the Dell servers.

We notice similar issues never getting more than 20MB/s or so of speeds from our virtualized, SAN-attached servers. Although general performance for small random I/O has still been OK, as we have exchange, SQL, file servers, etc all running off this setup concurrently and it’s all still a littlle faster than it used to be before SANs and Virtulization…although maybe it’s supposed to be A LOT faster and we’re just not seeing it?

However there’s one key factor we discovered in testing…Windows. We’re still mostly an XP/2003 shop, but have just started playing with some Win7/Svr 2008 stuff. To an XP machine from 2003 file server, I can’t get more than 20MB-27MB/s download of a large file. Even trying to eliminate the local disk, by downloading to a RAM-disk or opening CAD files direct from the server. Funny issue though is we know our Backups pull in close to 80MB/s, which is really starting to show good utilization of a gigabit pipe…and they’re running as clients on the servers, sending out to the tape library server over the LAN links, so we kind of figure the SAN/VMware infrastructure isn’t likely at fault… Then we tested with Windows 7 on an older Laptop. And got much closer to expected speeds! 30-40MB/s coping large file to the local drive, and even higher opening/loading large file into memory… So I wonder if the problem isn’t more on the client (server?) OS side?

I’d be interested in hearing what you think. Thanks.

]]> By: Capacity Planning for Virtualization | Josh Currier http://joshcurrier.com/2008/09/vmware-vi-and-iscsi-a-match-made-inwait-what/comment-page-1/#comment-3917 Capacity Planning for Virtualization | Josh Currier Mon, 28 Sep 2009 18:28:57 +0000 http://joshcurrier.com/?p=17#comment-3917 [...] transition went great!  I had some network performance issues early on but we got things smoothed out and were back on track.  What I didn’t see coming [...] [...] transition went great!  I had some network performance issues early on but we got things smoothed out and were back on track.  What I didn’t see coming [...]

]]> By: Josh http://joshcurrier.com/2008/09/vmware-vi-and-iscsi-a-match-made-inwait-what/comment-page-1/#comment-3845 Josh Tue, 15 Sep 2009 21:03:45 +0000 http://joshcurrier.com/?p=17#comment-3845 I haven't done a direct comparison no, but thanks for the tip! I will do some testing with this next week. I haven’t done a direct comparison no, but thanks for the tip! I will do some testing with this next week.

]]> By: Joord http://joshcurrier.com/2008/09/vmware-vi-and-iscsi-a-match-made-inwait-what/comment-page-1/#comment-3844 Joord Tue, 15 Sep 2009 20:56:57 +0000 http://joshcurrier.com/?p=17#comment-3844 Hi, You might want to check teaming settings. I have noticed that the teaming settings "route based on..." are by far slower than the "use explicit failover..". Hope it helps, Hi,

You might want to check teaming settings.

I have noticed that the teaming settings “route based on…” are by far slower than the “use explicit failover..”.

Hope it helps,

]]> By: Josh http://joshcurrier.com/2008/09/vmware-vi-and-iscsi-a-match-made-inwait-what/comment-page-1/#comment-3747 Josh Wed, 12 Aug 2009 20:46:24 +0000 http://joshcurrier.com/?p=17#comment-3747 Hi Will, As it turns out, none of my performance issues were caused byt he 2810 switches. As recommended by a 'storage expert' I upgraded to a Procurve 2900 because it supported jumbo frames and flow control simultaneously, but it didn't fix the problem. In light of that, yep I was more than happy with the 2810 switch. I hear that software iSCSI has been much improved in ESX4 as well, though I haven' had the opportunity to work with it yet. Hi Will,
As it turns out, none of my performance issues were caused byt he 2810 switches. As recommended by a ‘storage expert’ I upgraded to a Procurve 2900 because it supported jumbo frames and flow control simultaneously, but it didn’t fix the problem. In light of that, yep I was more than happy with the 2810 switch.

I hear that software iSCSI has been much improved in ESX4 as well, though I haven’ had the opportunity to work with it yet.

]]> By: huberw http://joshcurrier.com/2008/09/vmware-vi-and-iscsi-a-match-made-inwait-what/comment-page-1/#comment-3746 huberw Wed, 12 Aug 2009 20:23:47 +0000 http://joshcurrier.com/?p=17#comment-3746 Josh, Are you happy with the 2810 switches for iSCSI? We are having a hard time deciding on an HP iSCSI network switch. It's either the 1800-24G or the 2810-24G. Thanks, Will Josh,

Are you happy with the 2810 switches for iSCSI? We are having a hard time deciding on an HP iSCSI network switch. It’s either the 1800-24G or the 2810-24G.

Thanks,

Will

]]> By: Brian http://joshcurrier.com/2008/09/vmware-vi-and-iscsi-a-match-made-inwait-what/comment-page-1/#comment-338 Brian Fri, 27 Mar 2009 15:06:57 +0000 http://joshcurrier.com/?p=17#comment-338 I think this may be the solution I was looking for: I tested IOmeter on a new Win2003 host I created from scratch and it's virtual disk throughput was great (80MB/sec @ 1MB, 50/50 RW, 50/50 random) to our iSCSI DataCore storage. I tested IOmeter on an old P2V converted host and got poor performance (15MB/sec @ 1MB, 50/50 RW, 50/50 random). I also did more of a real-world test to see how the IO was rather than just disk bandwidth (~280 iops, 1MB/sec @ 4K 50/50 RW, 75/25 random, 64 command queue). I then took the same P2V converted host and changed the virtual SCSI controller used from Buslogic to LSI, rebooted, let it install drivers, rebooted again, ran IOmeter, and got really good performance (50MB/sec @ 1MB, 50/50 RW, 50/50 random), (~1300 iops, 5MB/sec @ 4K 50/50 RW, 75/25 random, 64 command queue). This made my day!!! Now I just have to go back and switch to LSI SCSI all my P2V'ed hosts. This doesn't clear up my poor disk throughput when I am copying large vmdk files using VI Client, but at least my guest VMs will be zippy now! Cross-posted to: http://communities.vmware.com/post!reply.jspa?messageID=1191348 I think this may be the solution I was looking for:

I tested IOmeter on a new Win2003 host I created from scratch and it’s virtual disk throughput was great (80MB/sec @ 1MB, 50/50 RW, 50/50 random) to our iSCSI DataCore storage.

I tested IOmeter on an old P2V converted host and got poor performance (15MB/sec @ 1MB, 50/50 RW, 50/50 random). I also did more of a real-world test to see how the IO was rather than just disk bandwidth (~280 iops, 1MB/sec @ 4K 50/50 RW, 75/25 random, 64 command queue).

I then took the same P2V converted host and changed the virtual SCSI controller used from Buslogic to LSI, rebooted, let it install drivers, rebooted again, ran IOmeter, and got really good performance (50MB/sec @ 1MB, 50/50 RW, 50/50 random), (~1300 iops, 5MB/sec @ 4K 50/50 RW, 75/25 random, 64 command queue).

This made my day!!! Now I just have to go back and switch to LSI SCSI all my P2V’ed hosts. This doesn’t clear up my poor disk throughput when I am copying large vmdk files using VI Client, but at least my guest VMs will be zippy now!

Cross-posted to: http://communities.vmware.com/post!reply.jspa?messageID=1191348

]]> By: Josh http://joshcurrier.com/2008/09/vmware-vi-and-iscsi-a-match-made-inwait-what/comment-page-1/#comment-151 Josh Mon, 23 Feb 2009 19:40:17 +0000 http://joshcurrier.com/?p=17#comment-151 Long story short, VMWare software iSCSI initiator is crap. Last week I got 4 QLogic 4062c iSCSI HBAs, and wouldn't you know it I'm getting 500-700 Mbps throughput on file copy tests across the network. I knew that software iSCSI would be a little slower than hardware, but that's a little ridiculous. Long story short, VMWare software iSCSI initiator is crap. Last week I got 4 QLogic 4062c iSCSI HBAs, and wouldn’t you know it I’m getting 500-700 Mbps throughput on file copy tests across the network. I knew that software iSCSI would be a little slower than hardware, but that’s a little ridiculous.

]]> By: Brian http://joshcurrier.com/2008/09/vmware-vi-and-iscsi-a-match-made-inwait-what/comment-page-1/#comment-16 Brian Fri, 30 Jan 2009 19:37:56 +0000 http://joshcurrier.com/?p=17#comment-16 I am working through the same issue on some HP DL360 G4 and HP DL360 G5 servers using ESXi 3.5 Update3 with two Cisco 3650G switches. And in my case installing the MS initiator inside a virtual guest is fast (~100MBytes/sec) just like on a physical host, but letting VMware ESXi's software iSCSI handle the I/O I only get around 7MBytes/sec each direction. Please update here and on your VMware community post if you find out anything, and I will do the same. Thanks! I am working through the same issue on some HP DL360 G4 and HP DL360 G5 servers using ESXi 3.5 Update3 with two Cisco 3650G switches. And in my case installing the MS initiator inside a virtual guest is fast (~100MBytes/sec) just like on a physical host, but letting VMware ESXi’s software iSCSI handle the I/O I only get around 7MBytes/sec each direction.

Please update here and on your VMware community post if you find out anything, and I will do the same.

Thanks!

]]> By: Josh http://joshcurrier.com/2008/09/vmware-vi-and-iscsi-a-match-made-inwait-what/comment-page-1/#comment-15 Josh Tue, 27 Jan 2009 13:53:02 +0000 http://joshcurrier.com/?p=17#comment-15 I haven't tried that, no...but all indications point to the ESX network side as the culprit. I'll do some testing with the MS initiator though, thanks for the suggestion! I haven’t tried that, no…but all indications point to the ESX network side as the culprit. I’ll do some testing with the MS initiator though, thanks for the suggestion!

]]> By: ben http://joshcurrier.com/2008/09/vmware-vi-and-iscsi-a-match-made-inwait-what/comment-page-1/#comment-14 ben Tue, 27 Jan 2009 10:47:35 +0000 http://joshcurrier.com/?p=17#comment-14 Have you tried a Microsoft iSCSI initiator in the actual VM? I have heard that this is actually faster than the ESX iSCSI initiator if you do not have hardware HBA in the ESX server.... Have you tried a Microsoft iSCSI initiator in the actual VM? I have heard that this is actually faster than the ESX iSCSI initiator if you do not have hardware HBA in the ESX server….

]]>