10GbE speeds

classic Classic list List threaded Threaded
13 messages Options
ari
Reply | Threaded
Open this post in threaded view
|

10GbE speeds

ari
I have a project with 3 workstations all needing high speed access to  
about 6Tb of storage. In the past I've installed technology such as  
fibre channel connected SAN storage (using Apple's xsan) for up to a  
dozen workstations, but with only three workstations for this project  
I'm thinking about 10GbE. The workstations will be OSX and the server  
FreeBSD 7 with a bunch of disks in a RAID 5 or RAID 10 configuration.

Are 10GbE NICs and drivers significantly mature enough under FreeBSD  
to accomplish this? I'd need to achieve about 60MB/s transfer rate  
which is theoretically quite doable, as long as the drive array can  
keep up with three streams of that speed. I'd use netatalk, samba or  
nfs to share files depending on which I can eek the best speeds out of.

Alternatively I could populate the server with 1GbE NICs, one per  
workstation and use cross over cable. That way there is absolutely no  
contention on the network.

Any thoughts about the viability of this? Is 10GbE in production use  
with FreeBSD and does it scale well?


Cheers
Ari Maniatis




-------------------------->
ish
http://www.ish.com.au
Level 1, 30 Wilson Street Newtown 2042 Australia
phone +61 2 9550 5001   fax +61 2 9550 4001
GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A


_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 10GbE speeds

Kip Macy
>  Any thoughts about the viability of this? Is 10GbE in production use with
> FreeBSD and does it scale well?

It seems reasonable to me. Vendors are shipping products with some
flavor of freebsd with cxgb and mxge and products are planned for
ixgbe.

 -Kip

>
>
>  Cheers
>  Ari Maniatis
>
>
>
>
>  -------------------------->
>  ish
>  http://www.ish.com.au
>  Level 1, 30 Wilson Street Newtown 2042 Australia
>  phone +61 2 9550 5001   fax +61 2 9550 4001
>  GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
>
>
>  _______________________________________________
>  [hidden email] mailing list
>  http://lists.freebsd.org/mailman/listinfo/freebsd-performance
>  To unsubscribe, send any mail to
> "[hidden email]"
>
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

RE: 10GbE speeds

Muhammad Shafiq
In reply to this post by ari
Neterion is shipping 10G (XFRAME series) adapters, along with FreeBSD
driver ("nxge"), for quite some time. Please refer to
http://www.neterion.com/support/xframe_customers.html for latest FreeBSD
driver(s) and other support info.

Shafiq

 

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Kip Macy
Sent: Friday, April 11, 2008 4:40 PM
To: Aristedes Maniatis
Cc: [hidden email]
Subject: Re: 10GbE speeds

>  Any thoughts about the viability of this? Is 10GbE in production use
with
> FreeBSD and does it scale well?

It seems reasonable to me. Vendors are shipping products with some
flavor of freebsd with cxgb and mxge and products are planned for
ixgbe.

 -Kip

>
>
>  Cheers
>  Ari Maniatis
>
>
>
>
>  -------------------------->
>  ish
>  http://www.ish.com.au
>  Level 1, 30 Wilson Street Newtown 2042 Australia
>  phone +61 2 9550 5001   fax +61 2 9550 4001
>  GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
>
>
>  _______________________________________________
>  [hidden email] mailing list
>  http://lists.freebsd.org/mailman/listinfo/freebsd-performance
>  To unsubscribe, send any mail to
> "[hidden email]"
>
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to
"[hidden email]"
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 10GbE speeds

George Neville-Neil-3
In reply to this post by ari
At Sat, 12 Apr 2008 09:08:13 +1000,
Aristedes Maniatis wrote:

>
> I have a project with 3 workstations all needing high speed access to  
> about 6Tb of storage. In the past I've installed technology such as  
> fibre channel connected SAN storage (using Apple's xsan) for up to a  
> dozen workstations, but with only three workstations for this project  
> I'm thinking about 10GbE. The workstations will be OSX and the server  
> FreeBSD 7 with a bunch of disks in a RAID 5 or RAID 10 configuration.
>
> Are 10GbE NICs and drivers significantly mature enough under FreeBSD  
> to accomplish this? I'd need to achieve about 60MB/s transfer rate  
> which is theoretically quite doable, as long as the drive array can  
> keep up with three streams of that speed. I'd use netatalk, samba or  
> nfs to share files depending on which I can eek the best speeds out of.
>
> Alternatively I could populate the server with 1GbE NICs, one per  
> workstation and use cross over cable. That way there is absolutely no  
> contention on the network.
>
> Any thoughts about the viability of this? Is 10GbE in production use  
> with FreeBSD and does it scale well?

I am working with the Chelsio hardware and it seems to be well
supported on FreeBSD.  All the machines in that system are FreeBSD
though.  How do you intend to get the OSX systems to be 10GE?

Best,
George
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
ari
Reply | Threaded
Open this post in threaded view
|

Re: 10GbE speeds

ari

On 15/04/2008, at 10:54 PM, [hidden email] wrote:
> I am working with the Chelsio hardware and it seems to be well
> supported on FreeBSD.  All the machines in that system are FreeBSD
> though.  How do you intend to get the OSX systems to be 10GE?


What sort of throughput are you getting with that setup? Are you using  
NFS between the systems?

Good question about OSX, and I hadn't got to that part yet :-)  But I  
was hoping that some OSX drivers existed. My fail back plan is to put  
3 x 1GbE NICs into the server and just use crossover cable between the  
3 workstations and the server to avoid any contention within the  
ethernet network.

Ari Maniatis


-------------------------->
ish
http://www.ish.com.au
Level 1, 30 Wilson Street Newtown 2042 Australia
phone +61 2 9550 5001   fax +61 2 9550 4001
GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A


_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 10GbE speeds

Kip Macy
Myricom has OS X support, Chelsio has support in the works. I don't
know about Neterion or Intel.

    -Kip



On 4/15/08, Aristedes Maniatis <[hidden email]> wrote:

>
> On 15/04/2008, at 10:54 PM, [hidden email] wrote:
> > I am working with the Chelsio hardware and it seems to be well
> > supported on FreeBSD.  All the machines in that system are FreeBSD
> > though.  How do you intend to get the OSX systems to be 10GE?
>
>
> What sort of throughput are you getting with that setup? Are you using
> NFS between the systems?
>
> Good question about OSX, and I hadn't got to that part yet :-)  But I
> was hoping that some OSX drivers existed. My fail back plan is to put
> 3 x 1GbE NICs into the server and just use crossover cable between the
> 3 workstations and the server to avoid any contention within the
> ethernet network.
>
> Ari Maniatis
>
>
> -------------------------->
> ish
> http://www.ish.com.au
> Level 1, 30 Wilson Street Newtown 2042 Australia
> phone +61 2 9550 5001   fax +61 2 9550 4001
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
>
>
> _______________________________________________
> [hidden email] mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
> To unsubscribe, send any mail to
> "[hidden email]"
>
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 10GbE speeds

Benjeman J. Meekhof
In reply to this post by ari
Hi Aristedes,

We are/were testing FreeBSD on a Dell PE2950 with a Myricom 10GB
PCI-Express copper CX card.  The driver seems mature.  In tests out of
the box, I only saw about 3Gbps from iperf (testing against a linux
system...and maybe there are other issues with our environment/that
system to tune up yet...i wouldn't take that number too seriously).

I think some tuning could get it up to the maximum, anyways.  It
certainly took a little work to get our Linux systems up to the max, so
I would say that in general the defaults with 10G drivers on any system
need some work to hit the maximum bandwidth.

-Ben

Aristedes Maniatis wrote:

> I have a project with 3 workstations all needing high speed access to
> about 6Tb of storage. In the past I've installed technology such as
> fibre channel connected SAN storage (using Apple's xsan) for up to a
> dozen workstations, but with only three workstations for this project
> I'm thinking about 10GbE. The workstations will be OSX and the server
> FreeBSD 7 with a bunch of disks in a RAID 5 or RAID 10 configuration.
>
> Are 10GbE NICs and drivers significantly mature enough under FreeBSD to
> accomplish this? I'd need to achieve about 60MB/s transfer rate which is
> theoretically quite doable, as long as the drive array can keep up with
> three streams of that speed. I'd use netatalk, samba or nfs to share
> files depending on which I can eek the best speeds out of.
>
> Alternatively I could populate the server with 1GbE NICs, one per
> workstation and use cross over cable. That way there is absolutely no
> contention on the network.
>
> Any thoughts about the viability of this? Is 10GbE in production use
> with FreeBSD and does it scale well?
>
>
> Cheers
> Ari Maniatis
>
>
>
>
> -------------------------->
> ish
> http://www.ish.com.au
> Level 1, 30 Wilson Street Newtown 2042 Australia
> phone +61 2 9550 5001   fax +61 2 9550 4001
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
>
>
> _______________________________________________
> [hidden email] mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
> To unsubscribe, send any mail to
> "[hidden email]"
>
>

--
Benjeman Meekhof - UM ATLAS/AGLT2 Computing
office: 734-764-3450 cell: 734-417-6312

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 10GbE speeds

George Neville-Neil-3
In reply to this post by ari
At Wed, 16 Apr 2008 10:02:33 +1000,
Aristedes Maniatis wrote:

>
>
> On 15/04/2008, at 10:54 PM, [hidden email] wrote:
> > I am working with the Chelsio hardware and it seems to be well
> > supported on FreeBSD.  All the machines in that system are FreeBSD
> > though.  How do you intend to get the OSX systems to be 10GE?
>
>
> What sort of throughput are you getting with that setup? Are you using  
> NFS between the systems?

I haven't done any performance testing of bandwidth as yet and we are
not using these for NFS.  

> Good question about OSX, and I hadn't got to that part yet :-) But I
> was hoping that some OSX drivers existed. My fail back plan is to
> put 3 x 1GbE NICs into the server and just use crossover cable
> between the 3 workstations and the server to avoid any contention
> within the ethernet network.

I think you want either a pure 10GbE network or a pure 1GbE network
because my understanding is that 10GbE switches have issues with mixed
links.  I have not tested that first hand yet though.

Best,
George
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 10GbE speeds

Stefan Lambrev-2
In reply to this post by Benjeman J. Meekhof
Greetings,

Benjeman J. Meekhof wrote:
> Hi Aristedes,
>
> We are/were testing FreeBSD on a Dell PE2950 with a Myricom 10GB
> PCI-Express copper CX card.  The driver seems mature.  In tests out of
> the box, I only saw about 3Gbps from iperf (testing against a linux
> system...and maybe there are other issues with our environment/that
> system to tune up yet...i wouldn't take that number too seriously).
If I remember correctly there is a problem with iperf, because it
utilize too much CPU. I saw patches flying around, and part of them are
in FreeBSD ports collection I think.
Personally I prefer netperf for doing network tests :)

>
> I think some tuning could get it up to the maximum, anyways.  It
> certainly took a little work to get our Linux systems up to the max,
> so I would say that in general the defaults with 10G drivers on any
> system need some work to hit the maximum bandwidth.
>
> -Ben
>
> Aristedes Maniatis wrote:
>> I have a project with 3 workstations all needing high speed access to
>> about 6Tb of storage. In the past I've installed technology such as
>> fibre channel connected SAN storage (using Apple's xsan) for up to a
>> dozen workstations, but with only three workstations for this project
>> I'm thinking about 10GbE. The workstations will be OSX and the server
>> FreeBSD 7 with a bunch of disks in a RAID 5 or RAID 10 configuration.
>>
>> Are 10GbE NICs and drivers significantly mature enough under FreeBSD
>> to accomplish this? I'd need to achieve about 60MB/s transfer rate
>> which is theoretically quite doable, as long as the drive array can
>> keep up with three streams of that speed. I'd use netatalk, samba or
>> nfs to share files depending on which I can eek the best speeds out of.
>>
>> Alternatively I could populate the server with 1GbE NICs, one per
>> workstation and use cross over cable. That way there is absolutely no
>> contention on the network.
>>
>> Any thoughts about the viability of this? Is 10GbE in production use
>> with FreeBSD and does it scale well?
>>
>>
>> Cheers
>> Ari Maniatis
>>
>>
>>
>>
>> -------------------------->
>> ish
>> http://www.ish.com.au
>> Level 1, 30 Wilson Street Newtown 2042 Australia
>> phone +61 2 9550 5001   fax +61 2 9550 4001
>> GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
>>
>>
>> _______________________________________________
>> [hidden email] mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
>> To unsubscribe, send any mail to
>> "[hidden email]"
>>
>>
>

--

Best Wishes,
Stefan Lambrev
ICQ# 24134177

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 10GbE speeds

Kris Kennaway-3
In reply to this post by ari
On Wed, Apr 16, 2008 at 10:02:33AM +1000, Aristedes Maniatis wrote:
>
> On 15/04/2008, at 10:54 PM, [hidden email] wrote:
> >I am working with the Chelsio hardware and it seems to be well
> >supported on FreeBSD.  All the machines in that system are FreeBSD
> >though.  How do you intend to get the OSX systems to be 10GE?
>
>
> What sort of throughput are you getting with that setup? Are you using  
> NFS between the systems?

I get about 150 MB/sec NFS random write throughput between chelsio
NICs.  We are still in the process of optimizing NFS at the high end.
For bulk packet throughput it is not difficult to saturate it.

Kris

--
In God we Trust -- all others must submit an X.509 certificate.
    -- Charles Forsythe <[hidden email]>
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

RE: 10GbE speeds

Leonid Grossman
In reply to this post by Kip Macy
Neterion supports OS X on 10GbE NICs, I think Intel does as well (via
third party).
Leonid

> -----Original Message-----
> From: [hidden email] [mailto:owner-freebsd-
> [hidden email]] On Behalf Of Kip Macy
> Sent: Tuesday, April 15, 2008 5:09 PM
> To: Aristedes Maniatis; [hidden email];
[hidden email]

> Subject: Re: 10GbE speeds
>
> Myricom has OS X support, Chelsio has support in the works. I don't
> know about Neterion or Intel.
>
>     -Kip
>
>
>
> On 4/15/08, Aristedes Maniatis <[hidden email]> wrote:
> >
> > On 15/04/2008, at 10:54 PM, [hidden email] wrote:
> > > I am working with the Chelsio hardware and it seems to be well
> > > supported on FreeBSD.  All the machines in that system are FreeBSD
> > > though.  How do you intend to get the OSX systems to be 10GE?
> >
> >
> > What sort of throughput are you getting with that setup? Are you
using
> > NFS between the systems?
> >
> > Good question about OSX, and I hadn't got to that part yet :-)  But
I
> > was hoping that some OSX drivers existed. My fail back plan is to
put
> > 3 x 1GbE NICs into the server and just use crossover cable between
the

> > 3 workstations and the server to avoid any contention within the
> > ethernet network.
> >
> > Ari Maniatis
> >
> >
> > -------------------------->
> > ish
> > http://www.ish.com.au
> > Level 1, 30 Wilson Street Newtown 2042 Australia
> > phone +61 2 9550 5001   fax +61 2 9550 4001
> > GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
> >
> >
> > _______________________________________________
> > [hidden email] mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-performance
> > To unsubscribe, send any mail to
> > "[hidden email]"
> >
> _______________________________________________
> [hidden email] mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
> To unsubscribe, send any mail to "freebsd-performance-
> [hidden email]"
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

NFS performance

Markus Klaschka-2
In reply to this post by Kris Kennaway-3
That's interesting cause heavy reading from NFS brought me a loadavg of
70 and more if there were a lot of small files to read.
I thought this is a normal issue about NFS...
By the way, are all Realtek Cards for the bin or only the 8139...the
server has a 'RTL8169 Gigabit Ethernet Adapter'
I mean, you all know this, if not read the comments in that file ;)

    root@kalium:~ > grep worst /usr/src/sys/pci/if_rl.c
    * probably the worst PCI ethernet controller ever made, with the
    possible

What could be configured wrong?
What's the best way to test bandwidth if I only got one well connected
server? pathchar?

Cheers
Markus

Kris Kennaway schrieb:

> On Wed, Apr 16, 2008 at 10:02:33AM +1000, Aristedes Maniatis wrote:
>  
>> On 15/04/2008, at 10:54 PM, [hidden email] wrote:
>>    
>>> I am working with the Chelsio hardware and it seems to be well
>>> supported on FreeBSD.  All the machines in that system are FreeBSD
>>> though.  How do you intend to get the OSX systems to be 10GE?
>>>      
>> What sort of throughput are you getting with that setup? Are you using  
>> NFS between the systems?
>>    
>
> I get about 150 MB/sec NFS random write throughput between chelsio
> NICs.  We are still in the process of optimizing NFS at the high end.
> For bulk packet throughput it is not difficult to saturate it.
>
> Kris
>
> --
> In God we Trust -- all others must submit an X.509 certificate.
>     -- Charles Forsythe <[hidden email]>
> _______________________________________________
> [hidden email] mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
> To unsubscribe, send any mail to "[hidden email]"
>  

--
Markus Klaschka
MKDev - Markus Klaschka Development
http://www.mkdev.eu

Spain: 0034 - 63 747 23 07
UK:
        0044 - 750 910 2718
Mail: [hidden email] <mailto:[hidden email]>
Skype: mark-use
IRC: mark-use @ irc.freenode.net : #freebsd, ##security, #freebsd-src,
#bsdforen.de, #bsdgroup.de


_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: NFS performance

Kris Kennaway-3
On Thu, Apr 17, 2008 at 10:47:15AM +0200, Markus Klaschka wrote:

> That's interesting cause heavy reading from NFS brought me a loadavg of
> 70 and more if there were a lot of small files to read.
> I thought this is a normal issue about NFS...
> By the way, are all Realtek Cards for the bin or only the 8139...the
> server has a 'RTL8169 Gigabit Ethernet Adapter'
> I mean, you all know this, if not read the comments in that file ;)
>
>    root@kalium:~ > grep worst /usr/src/sys/pci/if_rl.c
>    * probably the worst PCI ethernet controller ever made, with the
>    possible
>
> What could be configured wrong?
> What's the best way to test bandwidth if I only got one well connected
> server? pathchar?

NFS performance is limited by several things:

* server disk I/O.  With low end disk hardware you are not going to
get good performance at high load.

* network bandwidth.  Ditto.

* NFS client and server implementation

There have been important relevant improvements in 8.0 that improve
the client performance with many concurrent processes doing NFS I/O.
Also 7.0 has much better performance than 6.x.

Kris

P.S. Load average doesn't tell you if your system is performing badly,
it tells you that the system is running many processes.

--
In God we Trust -- all others must submit an X.509 certificate.
    -- Charles Forsythe <[hidden email]>
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"