Tuning: 100mbit faster, gbit slower.

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Tuning: 100mbit faster, gbit slower.

Daniel Andersson-9
Hey!

I was trying to milk the most out of my 100/100. What I
ended up with was something, to me, quite odd. When I
hadn't done anything I could ftp things from my server
box at 50mb/s and run rtorrent at about 9-10 mb/s at most.
After my "tuning" I can only ftp at a very "choppy"
30-40mb/s, but rtorrent runs at about 11mb/s.
This is what I did:

kern.ipc.maxsockbuf=16777216
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216

according to
http://dsd.lbl.gov/TCP-tuning/FreeBSD.html
every other setting there was default
I believe.

I also set these:

net.inet.tcp.recvspace: 262144
net.inet.tcp.sendspace: 262144

dmesg:
http://pastebin.org/24780

Am I just imagining that rtorrent runs faster?
Can't ftp handle high buffers or did I mess
something up? Is there something else I
could do to make it faster? Setting up
polling perhaps?

Cheers,
Daniel Andersson
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Tuning: 100mbit faster, gbit slower.

Jin Guojun [VFFS]
You cannot do blind tuning according those numbers. They are for
1-10Gbps pipe.
The proper number is "pipe diamter" x "pipe length" = capacity

In your case, the maximum number = 100Mbps x "delay from your machine to
server"

-Jin

Daniel Andersson wrote:

>Hey!
>
>I was trying to milk the most out of my 100/100. What I
>ended up with was something, to me, quite odd. When I
>hadn't done anything I could ftp things from my server
>box at 50mb/s and run rtorrent at about 9-10 mb/s at most.
>After my "tuning" I can only ftp at a very "choppy"
>30-40mb/s, but rtorrent runs at about 11mb/s.
>This is what I did:
>
>kern.ipc.maxsockbuf=16777216
>net.inet.tcp.sendbuf_max=16777216
>net.inet.tcp.recvbuf_max=16777216
>
>according to
>http://dsd.lbl.gov/TCP-tuning/FreeBSD.html
>every other setting there was default
>I believe.
>
>I also set these:
>
>net.inet.tcp.recvspace: 262144
>net.inet.tcp.sendspace: 262144
>
>dmesg:
>http://pastebin.org/24780
>
>Am I just imagining that rtorrent runs faster?
>Can't ftp handle high buffers or did I mess
>something up? Is there something else I
>could do to make it faster? Setting up
>polling perhaps?
>
>Cheers,
>Daniel Andersson
>  
>
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Tuning: 100mbit faster, gbit slower.

Josh Paetzel
In reply to this post by Daniel Andersson-9
On Sunday 23 March 2008 05:21:48 pm Daniel Andersson wrote:

> Hey!
>
> I was trying to milk the most out of my 100/100. What I
> ended up with was something, to me, quite odd. When I
> hadn't done anything I could ftp things from my server
> box at 50mb/s and run rtorrent at about 9-10 mb/s at most.
> After my "tuning" I can only ftp at a very "choppy"
> 30-40mb/s, but rtorrent runs at about 11mb/s.
> This is what I did:
>
> kern.ipc.maxsockbuf=16777216
> net.inet.tcp.sendbuf_max=16777216
> net.inet.tcp.recvbuf_max=16777216
>
> according to
> http://dsd.lbl.gov/TCP-tuning/FreeBSD.html
> every other setting there was default
> I believe.
>
> I also set these:
>
> net.inet.tcp.recvspace: 262144
> net.inet.tcp.sendspace: 262144
>
> dmesg:
> http://pastebin.org/24780
>
> Am I just imagining that rtorrent runs faster?
> Can't ftp handle high buffers or did I mess
> something up? Is there something else I
> could do to make it faster? Setting up
> polling perhaps?
>
> Cheers,
> Daniel Andersson
The stock settings are more than enough to saturate 100TX with even relatively
ancient hardware.  And by ancient I mean Pentium 2 class machines.

The biggest tuning you can do is use intel (fxp) or 3com (xl) NICS and a
halfway decent switch.

If your server box can't saturate 100TX ethernet with the defaults then
something is amiss.  Perhaps provide a dmesg from the server and a client and
a tcpdump from an FTP session between them?

--
Thanks,

Josh Paetzel

PGP: 8A48 EF36 5E9F 4EDA 5A8C 11B4 26F9 01F1 27AF AECB

signature.asc (202 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Tuning: 100mbit faster, gbit slower.

Stefan Lambrev-2
In reply to this post by Daniel Andersson-9
Daniel Andersson wrote:
> Hey!
>
> I was trying to milk the most out of my 100/100. What I
> ended up with was something, to me, quite odd. When I
> hadn't done anything I could ftp things from my server
> box at 50mb/s and run rtorrent at about 9-10 mb/s at most.
> After my "tuning" I can only ftp at a very "choppy"
> 30-40mb/s, but rtorrent runs at about 11mb/s.
>  
Are you sure the problem is in the network ?
Sounds like the bottleneck is your HDD.
You can run netperf to check this :)

> This is what I did:
>
> kern.ipc.maxsockbuf=16777216
> net.inet.tcp.sendbuf_max=16777216
> net.inet.tcp.recvbuf_max=16777216
>
> according to
> http://dsd.lbl.gov/TCP-tuning/FreeBSD.html
> every other setting there was default
> I believe.
>
> I also set these:
>
> net.inet.tcp.recvspace: 262144
> net.inet.tcp.sendspace: 262144
>
> dmesg:
> http://pastebin.org/24780
>
> Am I just imagining that rtorrent runs faster?
> Can't ftp handle high buffers or did I mess
> something up? Is there something else I
> could do to make it faster? Setting up
> polling perhaps?
>
> Cheers,
> Daniel Andersson
> _______________________________________________
> [hidden email] mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
> To unsubscribe, send any mail to "[hidden email]"
>  

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Tuning: 100mbit faster, gbit slower.

Daniel Andersson-9
In reply to this post by Josh Paetzel
Thanks for the reply! Maybe I should have been more clear.
My setup looks like this:
internet - em1(server box) em0 - windoze desktop.

The internet(em1) 100mbit seems to do fine, even better than
before, I get about 11mb/s with rtorrent(uploading). It's the
internal gbit connection that's weird with ftp. It not as fast nor
as smooth as it was before I did the "tuning". I doesn't have
any trouble running ftping at 30mb/s after the tuning so it is
definately capabel of delivering 100mbit?

$ifconfig -a
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:1b:21:0a:1d:87
        inet 192.168.0.10 netmask 0xffffff00 broadcast 192.168.0.255
        media: Ethernet autoselect (1000baseTX <full-duplex>)
        status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:1b:21:0c:d1:b3
        inet external.ip.goes.here netmask 0xfffffc00 broadcast
xx.xxx.147.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
plip0: flags=108810<POINTOPOINT,SIMPLEX,MULTICAST,NEEDSGIANT> metric 0 mtu
1500
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet6 ::1 prefixlen 128
        inet 127.0.0.1 netmask 0xff000000
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33204

tcpdump as requested:
engy# tcpdump -vv -i em0 portrange 40000-42000 or portrange 20-21 |
cat > /usr/home/engy/tcpdump

http://pastebin.org/25081

If you had something else in mind, let me know. When I transferred large
files I sometimes got about 600 packages dropped by kernel. That
can't be good. Also it seemed that I got a lot of ack packages, don't
know if that's normal.

Cheers,
Daniel


>The stock settings are more than enough to saturate 100TX with even
> relatively
> >ancient hardware.  And by ancient I mean Pentium 2 class machines.
> >
> >The biggest tuning you can do is use intel (fxp) or 3com (xl) NICS and a
> >halfway decent switch.
> >
> >If your server box can't saturate 100TX ethernet with the defaults then
> >something is amiss.  Perhaps provide a dmesg from the server and a client
> and
> >a tcpdump from an FTP session between them?
> >
> >--
> >Thanks,
> >
> >Josh Paetzel
>
> >PGP: 8A48 EF36 5E9F 4EDA 5A8C 11B4 26F9 01F1 27AF AECB
>
>
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Tuning: 100mbit faster, gbit slower.

Daniel Andersson-9
In reply to this post by Stefan Lambrev-2
Bloody hell! You were right. I transferred a big file
to the system disk and ftped it from there to my
desktop and it topped at 59mb/s, Sorry guys!

It's quite odd though. It's my newest disk:
ad10: 476940MB <Seagate ST3500630AS 3.AAE> at
ata5-master SATA150

It's connected to:
atapci2: <Promise PDC20579 SATA150 controller> port
0xd200-0xd27f,0xd300-0xd3ff mem 0xee0c0000-0xee0c0fff,0xee080000-0xee09ffff
irq 18 at device 11.0 on pci2
atapci2: [ITHREAD]atapci2: [ITHREAD]

I recall having trouble installing on that controller, it would just reboot.

I just checked the 7.0 Hardware notes and can't find it there. That sucks.

Guess I'll have to buy a PCI controller then. Any good ones you
can recommend?

Cheers,
Daniel


>Are you sure the problem is in the network ?
> >Sounds like the bottleneck is your HDD.
> >You can run netperf to check this :)
>
>
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Tuning: 100mbit faster, gbit slower.

Daniel Andersson-9
In reply to this post by Daniel Andersson-9
I might have spoken too soon. I ran the transfer a few times
and got this once:

215653 packets captured
379235 packets received by filter
97650 packets dropped by kernel

The transfer wouldn't even start. Buggy windoze client perhaps?

Other times it would jump anywhere from 26-61mb/s. That could
be disk bottleneck I guess.
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Tuning: 100mbit faster, gbit slower.

Josh Paetzel
In reply to this post by Daniel Andersson-9
On Monday 24 March 2008 06:22:12 am Daniel Andersson wrote:

> Thanks for the reply! Maybe I should have been more clear.
> My setup looks like this:
> internet - em1(server box) em0 - windoze desktop.
>
> The internet(em1) 100mbit seems to do fine, even better than
> before, I get about 11mb/s with rtorrent(uploading). It's the
> internal gbit connection that's weird with ftp. It not as fast nor
> as smooth as it was before I did the "tuning". I doesn't have
> any trouble running ftping at 30mb/s after the tuning so it is
> definately capabel of delivering 100mbit?
>
<snip>

I think we are having a terminology problem here.  Are you meaning 30mb/s as
in 30 megabits per second (30% of 100TX speed) or do you mean 30 MB/s as in
30 Megabytes/sec?  I think in rereading you post you are meaning Megabytes
but using the terminology for megabits.

So here's what I've done to nearly saturate gig-e.  Keep in mind that I have
15k SAS drives and intel gig-e adapters that aren't sitting in 33mhz 32bit
PCI slots, single IDE/SATA drives are going to be a bottleneck as are 33mhz
32bit PCI NICs.

This is on RELENG_6_3

net.inet.tcp.sendspace=262144
net.inet.tcp.recvspace=262144
kern.ipc.maxsockbuf=1048576
ifconfig em0 mtu 9014 (You'll need a switch that supports jumbo frames to do
this)

iperf shows wire traffic around 969 mbps and FTP runs at 110 Megs/sec
scp/sftp appears to be cpu bound at 45 Megs/sec, and NFS with TCP mounts and
send/receive packets set to 16384 manages about 90 Megs/sec.

--
Thanks,

Josh Paetzel

PGP: 8A48 EF36 5E9F 4EDA 5A8C 11B4 26F9 01F1 27AF AECB

signature.asc (202 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Tuning: 100mbit faster, gbit slower.

Daniel Andersson-9
> >I think we are having a terminology problem here.  Are you meaning 30mb/s
> as
> >in 30 megabits per second (30% of 100TX speed) or do you mean 30 MB/s as
> in
> >30 Megabytes/sec?  I think in rereading you post you are meaning
> Megabytes
> >but using the terminology for megabits.


 Ah, yes, sorry! I ment 30MB/s.

>So here's what I've done to nearly saturate gig-e.  Keep in mind that I
> have
> >15k SAS drives and intel gig-e adapters that aren't sitting in 33mhz
> 32bit
> >PCI slots, single IDE/SATA drives are going to be a bottleneck as are
> 33mhz
> >32bit PCI NICs.
> >
> >This is on RELENG_6_3
> >
> >
> >net.inet.tcp.sendspace=262144
> >net.inet.tcp.recvspace=262144
> >
> >kern.ipc.maxsockbuf=1048576
> >ifconfig em0 mtu 9014 (You'll need a switch that supports jumbo frames to
> do
> >this)
> >
> >iperf shows wire traffic around 969 mbps and FTP runs at 110 Megs/sec
> >scp/sftp appears to be cpu bound at 45 Megs/sec, and NFS with TCP mounts
> and
> >send/receive packets set to 16384 manages about 90 Megs/sec.
> >--
> >Thanks,
> >Josh Paetzel
> >PGP: 8A48 EF36 5E9F 4EDA 5A8C 11B4 26F9 01F1 27AF AECB
>
> I don't really need the extra speed on the internal network. I just want
the
steady 50MB/s back, but also the increased performance of rtorrent I got
from my "tuning". I'm also planning on getting more disks and setting up
either zfs or softraid to lessen the hdd bottleneck. One thing at a time
though.

Maybe setting kern.ipc.maxsockbuf=1M+ and leaving everything else default
will solve it. Since rtorrent has it's own setting for buffer sizes.

Thanks for the replies and sorry for the mixup!
Daniel
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Tuning: 100mbit faster, gbit slower.

Tom Judge
In reply to this post by Josh Paetzel
Josh Paetzel wrote:
<SNIP>
>
> This is on RELENG_6_3
>
> net.inet.tcp.sendspace=262144
> net.inet.tcp.recvspace=262144
> kern.ipc.maxsockbuf=1048576
> ifconfig em0 mtu 9014 (You'll need a switch that supports jumbo frames to do
> this)
>

Daniel, you should note that all devices on your network also need to
have the same MTU configured, and that Jumbo frames are only supported
on gige connections so if you have any 100/10 devices you can not use
jumbo frames.


> iperf shows wire traffic around 969 mbps and FTP runs at 110 Megs/sec
> scp/sftp appears to be cpu bound at 45 Megs/sec, and NFS with TCP mounts and
> send/receive packets set to 16384 manages about 90 Megs/sec.
>

Hi Josh,

If you are running SCP/SFTP over your internal lan and are not worried
about the security of the data in the session only the authentication
then you man want to check out the HPN patches for openssh (I belive
they are available as an option in the openssh port) which re-enable the
cypher 'none'.  Also the latest patch set introduces multi threaded
crypto so that ssh is not bound by the performance of a single cpu in
multi cpu/core systems.

We run the HPN patches here (the older set with no multi threading
support) and we can saturate gige with a single scp/sftp/ssh connection
between two reasonably spec'd boxes (Our mtu is 8192 as this is more
compatiable with most hardware [nics and switchs] most of the hardware
wie have will only do 9000 byte jumbo frames).

Tom

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"