tuning for high connection rates

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

tuning for high connection rates

Philipp Wuensche-2
Hi,

we are running a FreeBSD 7-BETA4 with SCHED_4BSD on a Intel Core2Dual
E6600 2.4GHz system for our bittorrent Opentracker.

The system handles about 20Kpps (18Mbit/s) incoming and 15kpps (22
Mbit/s) outgoing traffic serving 4000 connections/sec using TCP. The
connections are very short-living, all answered within one packet.

You can find the system stats at
http://outpost.h3q.com/stalker/munin/opentracker/opentracker.html

We are now running into some limits at peak time, system is up to 100%
and em0 takes about 80% on one CPU while the Opentracker software only
takes 10-15% CPU. The system is still responsible and answers all the
requests, but we are worried what will happen if the tracker grows at
the current rate.

Currently we are out of ideas for tuning, so we kindly ask for ideas on
tuning the system to bring down the CPU usage from the em and the system
CPU usage. We tried tuning the em int_delay and abs_int_delay but
without success.

We have updated to the latest em driver:

em0: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port
0x4000-0x401f mem 0xe8000000-0xe801ffff irq 16 at device 0.0 on pci13
em0: Using MSI interrupt
em0: Ethernet address: 00:30:48:92:06:5f
em0: [FILTER]

The debug output of em0 looks like this:

em0: CTRL = 0x40140248 RCTL = 0x8002
em0: Packet buffer = Tx=20k Rx=12k
em0: Flow control watermarks high = 10240 low = 8740
em0: tx_int_delay = 66, tx_abs_int_delay = 66
em0: rx_int_delay = 32, rx_abs_int_delay = 66
em0: fifo workaround = 0, fifo_reset_count = 0
em0: hw tdh = 183, hw tdt = 183
em0: hw rdh = 139, hw rdt = 139
em0: Num Tx descriptors avail = 223
em0: Tx Descriptors not avail1 = 6225
em0: Tx Descriptors not avail2 = 3
em0: Std mbuf failed = 0
em0: Std mbuf cluster failed = 0
em0: Driver dropped packets = 0
em0: Driver tx dma failure in encap = 0

We did some tuning already and our current sysctl.conf looks like this:

kern.ipc.somaxconn=32768
net.inet.icmp.icmplim=3000
kern.ipc.maxsockets=300000
net.inet.tcp.delayed_ack=1
net.inet.tcp.finwait2_timeout=15000
net.inet.tcp.fast_finwait2_recycle=1
net.inet.tcp.maxtcptw=196607
dev.em.0.rx_processing_limit=-1

greetings,
cryx

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: tuning for high connection rates

Philipp Wuensche-2
Philipp Wuensche wrote:
> Hi,
>
> we are running a FreeBSD 7-BETA4 with SCHED_4BSD on a Intel Core2Dual
> E6600 2.4GHz system for our bittorrent Opentracker.

I forgot to mention, its FreeBSD amd64.

greetings,
cryx

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: tuning for high connection rates

Bill Moran-2
In reply to this post by Philipp Wuensche-2
Philipp Wuensche <[hidden email]> wrote:

>
> Hi,
>
> we are running a FreeBSD 7-BETA4 with SCHED_4BSD on a Intel Core2Dual
> E6600 2.4GHz system for our bittorrent Opentracker.
>
> The system handles about 20Kpps (18Mbit/s) incoming and 15kpps (22
> Mbit/s) outgoing traffic serving 4000 connections/sec using TCP. The
> connections are very short-living, all answered within one packet.
>
> You can find the system stats at
> http://outpost.h3q.com/stalker/munin/opentracker/opentracker.html
>
> We are now running into some limits at peak time, system is up to 100%
> and em0 takes about 80% on one CPU while the Opentracker software only
> takes 10-15% CPU. The system is still responsible and answers all the
> requests, but we are worried what will happen if the tracker grows at
> the current rate.
>
> Currently we are out of ideas for tuning, so we kindly ask for ideas on
> tuning the system to bring down the CPU usage from the em and the system
> CPU usage. We tried tuning the em int_delay and abs_int_delay but
> without success.

Enable polling on the interface and see if that helps.  See man polling.

--
Bill Moran
Collaborative Fusion Inc.

[hidden email]
Phone: 412-422-3463x4023
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: tuning for high connection rates

Philipp Wuensche-2
Bill Moran wrote:
>
> Enable polling on the interface and see if that helps.  See man polling.

We tried polling already and it didn't help at all.

As I understand it, and correct me if I'm wrong, polling helps against
high interrupt rates but for that intel gigabit cards have interrupt
moderation. We don't have a problem with interrupts (20% CPU) at the
moment but with system (100% CPU) as you can see in the system
monitoring graphs. Interrupts sometimes peak at, but usually are under,
the 2k interrupts/sec limit.

greetings,
cryx

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: tuning for high connection rates

Robert N. M. Watson-2
In reply to this post by Philipp Wuensche-2

On Wed, 5 Dec 2007, Philipp Wuensche wrote:

> we are running a FreeBSD 7-BETA4 with SCHED_4BSD on a Intel Core2Dual E6600
> 2.4GHz system for our bittorrent Opentracker.
>
> The system handles about 20Kpps (18Mbit/s) incoming and 15kpps (22 Mbit/s)
> outgoing traffic serving 4000 connections/sec using TCP. The connections are
> very short-living, all answered within one packet.
>
> You can find the system stats at
> http://outpost.h3q.com/stalker/munin/opentracker/opentracker.html
>
> We are now running into some limits at peak time, system is up to 100% and
> em0 takes about 80% on one CPU while the Opentracker software only takes
> 10-15% CPU. The system is still responsible and answers all the requests,
> but we are worried what will happen if the tracker grows at the current
> rate.
>
> Currently we are out of ideas for tuning, so we kindly ask for ideas on
> tuning the system to bring down the CPU usage from the em and the system CPU
> usage. We tried tuning the em int_delay and abs_int_delay but without
> success.

Could you show us the output from "top -S" left running for a few minutes in
the steady state.

Could you try setting the sysctl net.isr.direct to 0, and see how that affects
performance, CPU time reports, and "top -S" output?

Robert N M Watson
Computer Laboratory
University of Cambridge

>
> We have updated to the latest em driver:
>
> em0: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port
> 0x4000-0x401f mem 0xe8000000-0xe801ffff irq 16 at device 0.0 on pci13
> em0: Using MSI interrupt
> em0: Ethernet address: 00:30:48:92:06:5f
> em0: [FILTER]
>
> The debug output of em0 looks like this:
>
> em0: CTRL = 0x40140248 RCTL = 0x8002
> em0: Packet buffer = Tx=20k Rx=12k
> em0: Flow control watermarks high = 10240 low = 8740
> em0: tx_int_delay = 66, tx_abs_int_delay = 66
> em0: rx_int_delay = 32, rx_abs_int_delay = 66
> em0: fifo workaround = 0, fifo_reset_count = 0
> em0: hw tdh = 183, hw tdt = 183
> em0: hw rdh = 139, hw rdt = 139
> em0: Num Tx descriptors avail = 223
> em0: Tx Descriptors not avail1 = 6225
> em0: Tx Descriptors not avail2 = 3
> em0: Std mbuf failed = 0
> em0: Std mbuf cluster failed = 0
> em0: Driver dropped packets = 0
> em0: Driver tx dma failure in encap = 0
>
> We did some tuning already and our current sysctl.conf looks like this:
>
> kern.ipc.somaxconn=32768
> net.inet.icmp.icmplim=3000
> kern.ipc.maxsockets=300000
> net.inet.tcp.delayed_ack=1
> net.inet.tcp.finwait2_timeout=15000
> net.inet.tcp.fast_finwait2_recycle=1
> net.inet.tcp.maxtcptw=196607
> dev.em.0.rx_processing_limit=-1
>
> greetings,
> cryx
>
> _______________________________________________
> [hidden email] mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
> To unsubscribe, send any mail to "[hidden email]"
>
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: tuning for high connection rates

Jin Guojun [VFFS]-2
In reply to this post by Bill Moran-2
Philipp Wuensche <[hidden email]> wrote:

>>Hi,
>>
>>we are running a FreeBSD 7-BETA4 with SCHED_4BSD on a Intel Core2Dual
>>E6600 2.4GHz system for our bittorrent Opentracker.
>>
>>The system handles about 20Kpps (18Mbit/s) incoming and 15kpps (22
>>Mbit/s) outgoing traffic serving 4000 connections/sec using TCP. The
>>connections are very short-living, all answered within one packet.
>>
>>You can find the system stats at
>>http://outpost.h3q.com/stalker/munin/opentracker/opentracker.html
>>
>>We are now running into some limits at peak time, system is up to 100%
>>and em0 takes about 80% on one CPU while the Opentracker software only
>>takes 10-15% CPU. The system is still responsible and answers all the
>>requests, but we are worried what will happen if the tracker grows at
>>the current rate.
>>
>>Currently we are out of ideas for tuning, so we kindly ask for ideas on
>>tuning the system to bring down the CPU usage from the em and the system
>>CPU usage. We tried tuning the em int_delay and abs_int_delay but
>>without success.
>>    
>>
Is there some data for the context switch time? It may tell something.

-Jin
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: tuning for high connection rates

Adrian Chadd-2
In reply to this post by Philipp Wuensche-2
On 05/12/2007, Philipp Wuensche <[hidden email]> wrote:

> As I understand it, and correct me if I'm wrong, polling helps against
> high interrupt rates but for that intel gigabit cards have interrupt
> moderation. We don't have a problem with interrupts (20% CPU) at the
> moment but with system (100% CPU) as you can see in the system
> monitoring graphs. Interrupts sometimes peak at, but usually are under,
> the 2k interrupts/sec limit.

Begin by reading up on the hardware profiling support (hwpmc, pmc,
etc) and see if you can get some system and process-specific profiling
information.

Kernel/System profiling will probably show you an interesting thing or
two. One thing I noticed was high in my high-TCP-transaction tests
(but not on hardware anywhere near as nice as yours!) was crypto calls
for, IIRC, syncookies.


Adrian

--
Adrian Chadd - [hidden email]
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: tuning for high connection rates

Philipp Wuensche-2
In reply to this post by Robert N. M. Watson-2
Robert Watson wrote:
>
> Could you show us the output from "top -S" left running for a few
> minutes in the steady state.
>
> Could you try setting the sysctl net.isr.direct to 0, and see how that
> affects performance, CPU time reports, and "top -S" output?

I first had too look up what net.isr.direct does and I found
http://lists.freebsd.org/pipermail/freebsd-performance/2005-October/001561.html

Interesting, seems like the CPU usage switches between system and
interrupt, "swi1: net" pops up with 65% CPU. Interrupts go up to 2k
interrupts/sec. But in general the system usage stays the same, as far
as we can tell in this short time.

We will keep the system running with net.isr.direct=0 for a day or so to
get a better picture how the system performs over the day.

With net.isr.direct=1

CPU states:  1.9% user,  0.0% nice, 43.3% system,  9.8% interrupt, 45.0%
idle
Mem: 163M Active, 139M Inact, 695M Wired, 44K Cache, 213M Buf, 975M Free
Swap: 2048M Total, 2048M Free

  PID USERNAME     THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
   23 root           1 -68    -     0K    16K CPU0   0  25.4H 66.46% em0
taskq
   11 root           1 171 ki31     0K    16K RUN    1  25.5H 47.22%
idle: cpu1
   12 root           1 171 ki31     0K    16K RUN    0  17.3H 38.92%
idle: cpu0
 7467 nobody         3  96    0   152M   146M ucond  1 153:18 24.61%
opentracke
   13 root           1 -32    -     0K    16K RUN    0 785:03 17.33%
swi4: cloc

With net.isr.direct=0

CPU states:  1.9% user,  0.0% nice, 11.1% system, 42.1% interrupt, 44.9%
idle
Mem: 151M Active, 139M Inact, 695M Wired, 44K Cache, 213M Buf, 987M Free
Swap: 2048M Total, 2048M Free

  PID USERNAME     THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
   15 root           1 -44    -     0K    16K CPU0   0   2:49 64.70%
swi1: net
   11 root           1 171 ki31     0K    16K RUN    1  25.5H 46.09%
idle: cpu1
   12 root           1 171 ki31     0K    16K RUN    0  17.3H 36.18%
idle: cpu0
 7467 nobody         3  96    0   137M   132M ucond  1 154:28 26.37%
opentracke
   13 root           1 -32    -     0K    16K WAIT   1 786:38 17.48%
swi4: cloc
   23 root           1 -68    -     0K    16K -      0  25.4H  2.98% em0
taskq

greetings,
cryx

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: tuning for high connection rates

Philipp Wuensche-2
In reply to this post by Adrian Chadd-2
Adrian Chadd wrote:

> On 05/12/2007, Philipp Wuensche <[hidden email]> wrote:
>
>> As I understand it, and correct me if I'm wrong, polling helps against
>> high interrupt rates but for that intel gigabit cards have interrupt
>> moderation. We don't have a problem with interrupts (20% CPU) at the
>> moment but with system (100% CPU) as you can see in the system
>> monitoring graphs. Interrupts sometimes peak at, but usually are under,
>> the 2k interrupts/sec limit.
>
> Begin by reading up on the hardware profiling support (hwpmc, pmc,
> etc) and see if you can get some system and process-specific profiling
> information.

Oh interesting stuff, I definitely have to take a look into that. Nice.

> Kernel/System profiling will probably show you an interesting thing or
> two. One thing I noticed was high in my high-TCP-transaction tests
> (but not on hardware anywhere near as nice as yours!) was crypto calls
> for, IIRC, syncookies.

We tried with syncookies enabled and disabled, no change at all. But as
you already said, crypto calls on this kind of hardware are not that
expensive ;-)

greetings,
cryx

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: tuning for high connection rates

mdtancsa
In reply to this post by Philipp Wuensche-2
At 07:14 PM 12/4/2007, Philipp Wuensche wrote:

>The debug output of em0 looks like this:
>
>em0: CTRL = 0x40140248 RCTL = 0x8002
>em0: Packet buffer = Tx=20k Rx=12k
>em0: Flow control watermarks high = 10240 low = 8740
>em0: tx_int_delay = 66, tx_abs_int_delay = 66
>em0: rx_int_delay = 32, rx_abs_int_delay = 66
>em0: fifo workaround = 0, fifo_reset_count = 0
>em0: hw tdh = 183, hw tdt = 183
>em0: hw rdh = 139, hw rdt = 139
>em0: Num Tx descriptors avail = 223
>em0: Tx Descriptors not avail1 = 6225
>em0: Tx Descriptors not avail2 = 3
>em0: Std mbuf failed = 0
>em0: Std mbuf cluster failed = 0
>em0: Driver dropped packets = 0
>em0: Driver tx dma failure in encap = 0

If you do a
sysctl -w dev.em.0.stats=1

It will spit the nic stats to syslog.  What are the results ?  Also,
what does ifconfig em0 look like (i.e. what options do you have set, speed etc)

         ---Mike

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: tuning for high connection rates

Philipp Wuensche-2
Mike Tancsa wrote:

> At 07:14 PM 12/4/2007, Philipp Wuensche wrote:
>
>> The debug output of em0 looks like this:
>>
>> em0: CTRL = 0x40140248 RCTL = 0x8002
>> em0: Packet buffer = Tx=20k Rx=12k
>> em0: Flow control watermarks high = 10240 low = 8740
>> em0: tx_int_delay = 66, tx_abs_int_delay = 66
>> em0: rx_int_delay = 32, rx_abs_int_delay = 66
>> em0: fifo workaround = 0, fifo_reset_count = 0
>> em0: hw tdh = 183, hw tdt = 183
>> em0: hw rdh = 139, hw rdt = 139
>> em0: Num Tx descriptors avail = 223
>> em0: Tx Descriptors not avail1 = 6225
>> em0: Tx Descriptors not avail2 = 3
>> em0: Std mbuf failed = 0
>> em0: Std mbuf cluster failed = 0
>> em0: Driver dropped packets = 0
>> em0: Driver tx dma failure in encap = 0
>
> If you do a
> sysctl -w dev.em.0.stats=1
>
> It will spit the nic stats to syslog.  What are the results ?

em0: Excessive collisions = 0
em0: Sequence errors = 0
em0: Defer count = 0
em0: Missed Packets = 12876719
em0: Receive No Buffers = 5950326
em0: Receive Length Errors = 0
em0: Receive errors = 0
em0: Crc errors = 0
em0: Alignment errors = 0
em0: Collision/Carrier extension errors = 0
em0: RX overruns = 56256
em0: watchdog timeouts = 0
em0: XON Rcvd = 0
em0: XON Xmtd = 0
em0: XOFF Rcvd = 0
em0: XOFF Xmtd = 0
em0: Good Packets Rcvd = 3384408375
em0: Good Packets Xmtd = 2657550034
em0: TSO Contexts Xmtd = 6925441
em0: TSO Contexts Failed = 0


> Also,
> what does ifconfig em0 look like (i.e. what options do you have set,
> speed etc)

Its on a 100Mbit/s switch, we haven't changed the options:

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
        media: Ethernet autoselect (100baseTX <full-duplex>)

greetings,
cryx

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: tuning for high connection rates

mdtancsa
At 12:23 PM 12/5/2007, Philipp Wuensche wrote:

>Mike Tancsa wrote:
> > At 07:14 PM 12/4/2007, Philipp Wuensche wrote:
> >
> >> The debug output of em0 looks like this:
> >>
> >> em0: CTRL = 0x40140248 RCTL = 0x8002
> >> em0: Packet buffer = Tx=20k Rx=12k
> >> em0: Flow control watermarks high = 10240 low = 8740
> >> em0: tx_int_delay = 66, tx_abs_int_delay = 66
> >> em0: rx_int_delay = 32, rx_abs_int_delay = 66
> >> em0: fifo workaround = 0, fifo_reset_count = 0
> >> em0: hw tdh = 183, hw tdt = 183
> >> em0: hw rdh = 139, hw rdt = 139
> >> em0: Num Tx descriptors avail = 223
> >> em0: Tx Descriptors not avail1 = 6225
> >> em0: Tx Descriptors not avail2 = 3
> >> em0: Std mbuf failed = 0
> >> em0: Std mbuf cluster failed = 0
> >> em0: Driver dropped packets = 0
> >> em0: Driver tx dma failure in encap = 0
> >
> > If you do a
> > sysctl -w dev.em.0.stats=1
> >
> > It will spit the nic stats to syslog.  What are the results ?
>
>em0: Excessive collisions = 0
>em0: Sequence errors = 0
>em0: Defer count = 0
>em0: Missed Packets = 12876719
>em0: Receive No Buffers = 5950326
>em0: Receive Length Errors = 0
>em0: Receive errors = 0
>em0: Crc errors = 0
>em0: Alignment errors = 0
>em0: Collision/Carrier extension errors = 0
>em0: RX overruns = 56256
>em0: watchdog timeouts = 0
>em0: XON Rcvd = 0
>em0: XON Xmtd = 0
>em0: XOFF Rcvd = 0
>em0: XOFF Xmtd = 0
>em0: Good Packets Rcvd = 3384408375
>em0: Good Packets Xmtd = 2657550034
>em0: TSO Contexts Xmtd = 6925441
>em0: TSO Contexts Failed = 0
>
>
> > Also,
> > what does ifconfig em0 look like (i.e. what options do you have set,
> > speed etc)
>
>Its on a 100Mbit/s switch, we haven't changed the options:
>
>em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>         options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
>         media: Ethernet autoselect (100baseTX <full-duplex>)

Some people have reported that TSO is a "bad thing" on 100Mb.  Can
you try disabling that ? Also, you seem to have a lot of RX overruns
and missed packets such that the nic cannot process things fast
enough.  I havent done any benchmarks yet, but the Yandex people
claim their modified EM driver can handle higher PPS rates than the
stock em driver.    Not sure if they have a RELENG_7 port or not but
they might have some insight.

         ---Mike


_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: tuning for high connection rates

Philipp Wuensche-2
Mike Tancsa wrote:

> At 12:23 PM 12/5/2007, Philipp Wuensche wrote:
>> Mike Tancsa wrote:
>> > At 07:14 PM 12/4/2007, Philipp Wuensche wrote:
>> >
>> >> The debug output of em0 looks like this:
>> >>
>> >> em0: CTRL = 0x40140248 RCTL = 0x8002
>> >> em0: Packet buffer = Tx=20k Rx=12k
>> >> em0: Flow control watermarks high = 10240 low = 8740
>> >> em0: tx_int_delay = 66, tx_abs_int_delay = 66
>> >> em0: rx_int_delay = 32, rx_abs_int_delay = 66
>> >> em0: fifo workaround = 0, fifo_reset_count = 0
>> >> em0: hw tdh = 183, hw tdt = 183
>> >> em0: hw rdh = 139, hw rdt = 139
>> >> em0: Num Tx descriptors avail = 223
>> >> em0: Tx Descriptors not avail1 = 6225
>> >> em0: Tx Descriptors not avail2 = 3
>> >> em0: Std mbuf failed = 0
>> >> em0: Std mbuf cluster failed = 0
>> >> em0: Driver dropped packets = 0
>> >> em0: Driver tx dma failure in encap = 0
>> >
>> > If you do a
>> > sysctl -w dev.em.0.stats=1
>> >
>> > It will spit the nic stats to syslog.  What are the results ?
>>
>> em0: Excessive collisions = 0
>> em0: Sequence errors = 0
>> em0: Defer count = 0
>> em0: Missed Packets = 12876719
>> em0: Receive No Buffers = 5950326
>> em0: Receive Length Errors = 0
>> em0: Receive errors = 0
>> em0: Crc errors = 0
>> em0: Alignment errors = 0
>> em0: Collision/Carrier extension errors = 0
>> em0: RX overruns = 56256
>> em0: watchdog timeouts = 0
>> em0: XON Rcvd = 0
>> em0: XON Xmtd = 0
>> em0: XOFF Rcvd = 0
>> em0: XOFF Xmtd = 0
>> em0: Good Packets Rcvd = 3384408375
>> em0: Good Packets Xmtd = 2657550034
>> em0: TSO Contexts Xmtd = 6925441
>> em0: TSO Contexts Failed = 0
>>
>>
>> > Also,
>> > what does ifconfig em0 look like (i.e. what options do you have set,
>> > speed etc)
>>
>> Its on a 100Mbit/s switch, we haven't changed the options:
>>
>> em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>>        
>> options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
>>         media: Ethernet autoselect (100baseTX <full-duplex>)
>
> Some people have reported that TSO is a "bad thing" on 100Mb.  Can you
> try disabling that ? Also, you seem to have a lot of RX overruns and
> missed packets such that the nic cannot process things fast enough.  I
> havent done any benchmarks yet, but the Yandex people claim their
> modified EM driver can handle higher PPS rates than the stock em
> driver.    Not sure if they have a RELENG_7 port or not but they might
> have some insight.

After switching to net.isr.direct=0 and 346609775 good packets later, RX
overruns haven't increased by one! Thats nice. Still interrupt is using
up the CPU. I'm not quite sure if polling would help now!?

current stats:
em0: Excessive collisions = 0
em0: Sequence errors = 0
em0: Defer count = 0
em0: Missed Packets = 12885592
em0: Receive No Buffers = 5953357
em0: Receive Length Errors = 0
em0: Receive errors = 0
em0: Crc errors = 0
em0: Alignment errors = 0
em0: Collision/Carrier extension errors = 0
em0: RX overruns = 56296
em0: watchdog timeouts = 0
em0: XON Rcvd = 0
em0: XON Xmtd = 0
em0: XOFF Rcvd = 0
em0: XOFF Xmtd = 0
em0: Good Packets Rcvd = 3731018150
em0: Good Packets Xmtd = 2936659631
em0: TSO Contexts Xmtd = 7734211
em0: TSO Contexts Failed = 0

We will try disabling TSO to see if anything changes.

greetings,
cryx

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: tuning for high connection rates

mdtancsa
At 05:19 PM 12/5/2007, Philipp Wuensche wrote:

>After switching to net.isr.direct=0 and 346609775 good packets later, RX
>overruns haven't increased by one! Thats nice. Still interrupt is using
>up the CPU. I'm not quite sure if polling would help now!?

Polling is helpful to prevent livelock. Not sure if thats happening
to you.  What firewall (if any) are you using ?  pf used to be a lot
slower than ipfw.

The Yandex driver is at
http://people.yandex-team.ru/~wawa/
but its against RELENG_6 only I think.


Another thing to try is to turn back on Fast Interrupt handling. I
think its currently disabled.

In if_em.h, try adding

#define EM_FAST_IRQ 1

and then recompile the kernel or just driver.


>We will try disabling TSO to see if anything changes.

If you have TCP in your app, it seems thats the thing to do according
to the Intel developer.

         ---Mike

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: tuning for high connection rates

Philipp Wuensche-2
Mike Tancsa wrote:
> At 05:19 PM 12/5/2007, Philipp Wuensche wrote:
>
>> After switching to net.isr.direct=0 and 346609775 good packets later, RX
>> overruns haven't increased by one! Thats nice. Still interrupt is using
>> up the CPU. I'm not quite sure if polling would help now!?
>
> Polling is helpful to prevent livelock. Not sure if thats happening to
> you.  

No problems with livelock, system is usable all the time.

> What firewall (if any) are you using ?  pf used to be a lot slower
> than ipfw.

We use pf. Disabling it at all gives no noticable performance boost
because instead performance drops due to connections from networks we
currently filter. Maybe ipfw is faster, we could try that but would like
to use pf furthermore.

> Another thing to try is to turn back on Fast Interrupt handling. I think
> its currently disabled.
>
> In if_em.h, try adding
>
> #define EM_FAST_IRQ 1
>
> and then recompile the kernel or just driver.

Seems to be enabled by default on freebsd7, from our if_em.h:

/* Set FAST handling on by default */
#if __FreeBSD_version > 700000
#define EM_FAST_IRQ
#endif

greetings,
cryx


_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: tuning for high connection rates

Stanislav Sedov-3
In reply to this post by Philipp Wuensche-2
On Wed, Dec 05, 2007 at 04:03:29PM +0100 Philipp Wuensche mentioned:

> Adrian Chadd wrote:
> > On 05/12/2007, Philipp Wuensche <[hidden email]> wrote:
> >
> >> As I understand it, and correct me if I'm wrong, polling helps against
> >> high interrupt rates but for that intel gigabit cards have interrupt
> >> moderation. We don't have a problem with interrupts (20% CPU) at the
> >> moment but with system (100% CPU) as you can see in the system
> >> monitoring graphs. Interrupts sometimes peak at, but usually are under,
> >> the 2k interrupts/sec limit.
> >
> > Begin by reading up on the hardware profiling support (hwpmc, pmc,
> > etc) and see if you can get some system and process-specific profiling
> > information.
>
> Oh interesting stuff, I definitely have to take a look into that. Nice.
>

You can find a good tutorial on hwpmc by Robert Watson here:
http://lists.freebsd.org/pipermail/freebsd-current/2006-February/061096.html

--
Stanislav Sedov
ST4096-RIPE
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: tuning for high connection rates

Kris Kennaway-3
In reply to this post by Philipp Wuensche-2
Philipp Wuensche wrote:

> Adrian Chadd wrote:
>> On 05/12/2007, Philipp Wuensche <[hidden email]> wrote:
>>
>>> As I understand it, and correct me if I'm wrong, polling helps against
>>> high interrupt rates but for that intel gigabit cards have interrupt
>>> moderation. We don't have a problem with interrupts (20% CPU) at the
>>> moment but with system (100% CPU) as you can see in the system
>>> monitoring graphs. Interrupts sometimes peak at, but usually are under,
>>> the 2k interrupts/sec limit.
>> Begin by reading up on the hardware profiling support (hwpmc, pmc,
>> etc) and see if you can get some system and process-specific profiling
>> information.
>
> Oh interesting stuff, I definitely have to take a look into that. Nice.

Did you make progress on this?  I'd like to see a profiling trace from
hwpmc, and also from LOCK_PROFILING.

Kris

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"