FreeBSD Based Guests in ESXi with VMXNET3?

Zarathustra[H]

Zarathustra[H]

Extremely [H]
Joined
Oct 29, 2000
Messages
38,082
Hey all,

I stumbled across this strange behavior in another thread.

At first I thought it was a problem with my setup, but MySongRanHills had the same issue.


So the issue is this:

VMXNet3 works fine for other guests. registering ~13Gbit/s as measured in iPerf between them, until I test with FreeNAS (9.3)

I get ~21Gbit/s when Ubuntu is the server (iperf -s) and FreeNAS is the client (iperf -c) but only ~3.2Gbit/s when FreeNAS is the server and Ubuntu is the client.

iperf output:

Two Iperf tests with FreeNAS as a server (iperf -s) and Ubuntu Server as client (iperf -c):

Code: 
------------------------------------------------------------
Client connecting to xxx.xxx.xxx.xxx, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local xxx.xxx.xxx.xxx port 33485 connected with xxx.xxx.xxx.xxx port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  3.79 GBytes  3.25 Gbits/sec

------------------------------------------------------------
Client connecting to xxx.xxx.xxx.xxx, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local xxx.xxx.xxx.xxx port 43405 connected with xxx.xxx.xxx.xxx port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  3.69 GBytes  3.17 Gbits/sec

Two Iperf tests with Ubuntu Server as a server (iperf -s) and FreeNAS as client (iperf -c):

Code: 
------------------------------------------------------------
Client connecting to xxx.xxx.xxx.xxx, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[  3] local xxx.xxx.xxx.xxx port 53941 connected with xxx.xxx.xxx.xxx port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  24.9 GBytes  21.4 Gbits/sec

------------------------------------------------------------
Client connecting to xxx.xxx.xxx.xxx, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[  3] local xxx.xxx.xxx.xxx port 64555 connected with xxx.xxx.xxx.xxx port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  25.5 GBytes  21.9 Gbits/sec


I recently got a couple of Brocades to get better transfer speeds between FreeNAS and my workstation, but this issue is holding me back.

Any ideas what might be going on, and how I can fix it?
 
Sorry for posting on such an old thread... couldn't find anyone others with my exact issue.


Thanks for your post!

I was experiencing similar issues running a FreeNAS VM on top of ESXi 5.5. I eventually setup a second virtual switch and added another interface to my FreeNAS and Ubuntu VMs to create a storage network. I enabled Jumbo Frames on this second network. My speeds between Ubuntu VMs jumped from ~ 9 Gbits/sec to ~ 14 Gbits/sec. The best part though, is that my Ubuntu->FreeNAS iperf results went from ~ 3-4Gbits/sec to ~ 10 Gbits/sec.



Code: 
Last login: Sun Jul  9 13:44:59 2017 from 192.168.30.91
FreeBSD 10.3-STABLE (FreeNAS.amd64) #0 r295946+21897e6695f(HEAD): Thu Jun  8 20:02:12 UTC 2017

    FreeNAS (c) 2009-2016, The FreeNAS Development Team
    All rights reserved.
    FreeNAS is released under the modified BSD license.

    For more information, documentation, help or support, go here:
     http://freenas.org
Welcome to FreeNAS
[root@freenas] ~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 192.168.40.3 port 5001 connected with 192.168.40.4 port 44688
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  12.1 GBytes  10.4 Gbits/sec
^C[root@freenas] ~# iperf -c 192.168.40.4
------------------------------------------------------------
Client connecting to 192.168.40.4, TCP port 5001
TCP window size: 35.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.40.3 port 39684 connected with 192.168.40.4 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  27.3 GBytes  23.5 Gbits/sec
[root@freenas] ~#

I am still perplexed with how FreeNASVM->UbuntuVM is 23Gbits/sec.. That's 10Gbits/sec faster than my UbuntuVM<->UbuntuVM results and 13Gbits/sec faster than UbuntuVM->FreeNASVM.
 
What are the results on the E1000 virtual cards instead? there have been issues with VMXNET on some of the BSD and oddball OSes (solaris).
 
lopoetve said:
What are the results on the E1000 virtual cards instead? there have been issues with VMXNET on some of the BSD and oddball OSes (solaris).
Click to expand...


e1000 adapter in my FreeNAS VM. My Ubuntu iPerf test VMs always ran the VMXNET3. No Jumbo Frames:


Code: 
iPerfTestClient (VMXNET 3) -> FreeNAS (e1000) - ~ 1.7Gbits/sec
brian@iPerfTestClient:~$ iperf -c 192.168.30.101
------------------------------------------------------------
Client connecting to 192.168.30.101, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.30.239 port 46434 connected with 192.168.30.101 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  2.02 GBytes  1.74 Gbits/sec


FreeNAS (e1000) -> iPerfTestServer (VMXNET 3) - ~ 2.5 Gbits/sec
[root@freenas ~]# iperf -c 192.168.30.244                                                                                           
------------------------------------------------------------                                                                       
Client connecting to 192.168.30.244, TCP port 5001                                                                                 
TCP window size: 32.5 KByte (default)                                                                                               
------------------------------------------------------------                                                                       
[  6] local 192.168.30.101 port 15085 connected with 192.168.30.244 port 5001                                                       
[ ID] Interval       Transfer     Bandwidth                                                                                         
[  6]  0.0-10.0 sec  3.02 GBytes  2.59 Gbits/sec                                                                                   
[root@freenas ~]# iperf -c 192.168.30.244                                                                                           
------------------------------------------------------------                                                                       
Client connecting to 192.168.30.244, TCP port 5001                                                                                 
TCP window size: 32.5 KByte (default)                                                                                               
------------------------------------------------------------                                                                       
[  6] local 192.168.30.101 port 63030 connected with 192.168.30.244 port 5001                                                       
[ ID] Interval       Transfer     Bandwidth                                                                                         
[  6]  0.0-10.0 sec  2.91 GBytes  2.50 Gbits/sec

In the end, I chalked it up to issues with FreeBSD and the VM NIC. Hoping that once FreeNAS 11 is stable enough to upgrade to this'll be resolved. I need to find time to setup a VM to test the RCs with.

Thanks,
Brian
 
You must log in or register to reply here.
Back
Top