performance degradation in 6.2 when adding a second quad core chip

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

performance degradation in 6.2 when adding a second quad core chip

benjamin thielsen-2
hi folks-

we've been experiencing some interesting behavior on single quad core  
computers as compared to dual quad core computers.

it appears that adding a second processor to the system (leaving it  
otherwise untouched) actually decreases performance.  we've got a  
small rudimentary test process, built in house, that does postgresql  
queries (selects) via http requests (apache2/php5).

here is a small data set from what we've seen so far:

a user is one iteration of a curl statement and values listed in the  
cpu columns are average time before the page returns, in seconds.

users        1 cpu       2 cpu
-------------------------------
60            2.48        2.18
80            3.34        2.72
100           4.18        5.34
120           9.48       15.61
140           13.2       46.16
160           26.28      66.99

we're confident that nothing else has changed other than the addition  
of the chip, so i'm hoping for some insight on where we might look for  
clues.  the above results used 6.2-RELEASE.  we've started the same  
test using 7.0-RC2 and are seeing similar response times with both  
processors in place.  next on our list is doing a single processor  
iteration of the 7.0-RC2 test to corroborate that data, followed by a  
local test to query pg more directly, removing apache and friends from  
the equation, in hopes of getting some clarity.

below are the basics of this configuration - this is a new list for  
me, so i've been conservative, but i'm happy to provide as much detail  
as is helpful.

computer:
dell poweredge 2900
xeon quad core 2 ghz / 1333 mhz bus speed / 2 x 6mb l2 cache
4 gb 667 mhz memory

all of the software has been installed via ports, the 6.2 kernel is a  
custom kernel with (i believe) mostly rudimentary changes and support  
for the perc6 disk controller (that became available in 7) added, and  
the 7.0 kernel contains the same changes as the 6.2 kernel, excluding  
the custom perc6 support.

thanks!
-ben
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: performance degradation in 6.2 when adding a second quad core chip

Kris Kennaway-3
benjamin thielsen wrote:
> hi folks-
>
> we've been experiencing some interesting behavior on single quad core
> computers as compared to dual quad core computers.

Yes, this can happen when you run into concurrency bottlenecks in the
application or in the kernel.

> it appears that adding a second processor to the system (leaving it
> otherwise untouched) actually decreases performance.  we've got a small
> rudimentary test process, built in house, that does postgresql queries
> (selects) via http requests (apache2/php5).

7.0 will perform much better than 6.x on SMP workloads in general,
however TCP I/O is not yet at the point where it can make efficient use
of many processors (there has been a lot of work on TCP in 7.0, but it
is not yet at the stage where a performance payoff will be seen with
more than about 4 CPUs).  This is one of the projects that we will be
working on this year, so you can expect future releases to have improved
concurrent TCP performance.

There may be other issues, so if you like you can enable LOCK_PROFILING
and obtain a trace when your workload is running (see the manpage).  You
should also try the ULE scheduler on 7.0.

Kris

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: performance degradation in 6.2 when adding a second quad core chip

benjamin thielsen-2

On Feb 21, 2008, at 17.49, Kris Kennaway wrote:

> benjamin thielsen wrote:
>> hi folks-
>> we've been experiencing some interesting behavior on single quad  
>> core computers as compared to dual quad core computers.
>
> Yes, this can happen when you run into concurrency bottlenecks in  
> the application or in the kernel.
>
>> it appears that adding a second processor to the system (leaving it  
>> otherwise untouched) actually decreases performance.  we've got a  
>> small rudimentary test process, built in house, that does  
>> postgresql queries (selects) via http requests (apache2/php5).
>
> 7.0 will perform much better than 6.x on SMP workloads in general,  
> however TCP I/O is not yet at the point where it can make efficient  
> use of many processors (there has been a lot of work on TCP in 7.0,  
> but it is not yet at the stage where a performance payoff will be  
> seen with more than about 4 CPUs).  This is one of the projects that  
> we will be working on this year, so you can expect future releases  
> to have improved concurrent TCP performance.
>
> There may be other issues, so if you like you can enable  
> LOCK_PROFILING and obtain a trace when your workload is running (see  
> the manpage).  You should also try the ULE scheduler on 7.0.

i apologize - i neglected to mention that we are using ule on 7.0.

what do you guys generally endorse/recommend for local(non-network)  
load/performance testing?

-ben
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

RE: performance degradation in 6.2 when adding a second quad core chip

Noisex
So Benjamin you want to tell, that adding additional CPU decrease the
performance on 6.2? How did you made tests?
        I just want to know because in my plans to the next week is also add
2nd CPU (dual core) and some additional RAM to one of the my HP Proliant
DL360 (generaly mysql db server) because due to heavy MySQL usage nearly to
100% or even more sometimes some of my customers are not very happy with
service like that (long load time).

Server: HP proliant G4P DL360, CPU 1x3Ghz dual core, RAM: 4Gb (on system i
can see 3Gb :) ), SCSI 146Gbx2 (RAID 1+0 with smartarray controller p4600).
OS: FreeBSD 6.2, SMP

Current load of mysql process:

  PID USERNAME       THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU
COMMAND
70348 mysql           17  20    0   450M   234M kserel 1 111.4H 99.46%
mysqld

MySQL Version 5.0.51 i386
Uptime = 28 days 18 hrs 51 min 19 sec
Avg. qps = 567
Total Questions = 469973918
Threads Connected = 4

however that may be:
1) on next week i will upgrade to 6.3 but i dont know will it give some
performance or not but generaly i wait final relase of 7.0 with ULE
sheduller.
2) i will add +4GB RAM
3) i will add 2nd CPU

If it will not help, i will consider about mysql clustering...btw, is it
ok/done on freebsd?

p.s what you guys could recommend about enabled/disabled hyperthreading on
intel? Will it increase performance or over the left decrease?


-Gatis Rumbens



-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of benjamin
thielsen
Sent: Friday, February 22, 2008 4:36 PM
To: [hidden email]
Subject: Re: performance degradation in 6.2 when adding a second quad core
chip


On Feb 21, 2008, at 17.49, Kris Kennaway wrote:

> benjamin thielsen wrote:
>> hi folks-
>> we've been experiencing some interesting behavior on single quad  
>> core computers as compared to dual quad core computers.
>
> Yes, this can happen when you run into concurrency bottlenecks in  
> the application or in the kernel.
>
>> it appears that adding a second processor to the system (leaving it  
>> otherwise untouched) actually decreases performance.  we've got a  
>> small rudimentary test process, built in house, that does  
>> postgresql queries (selects) via http requests (apache2/php5).
>
> 7.0 will perform much better than 6.x on SMP workloads in general,  
> however TCP I/O is not yet at the point where it can make efficient  
> use of many processors (there has been a lot of work on TCP in 7.0,  
> but it is not yet at the stage where a performance payoff will be  
> seen with more than about 4 CPUs).  This is one of the projects that  
> we will be working on this year, so you can expect future releases  
> to have improved concurrent TCP performance.
>
> There may be other issues, so if you like you can enable  
> LOCK_PROFILING and obtain a trace when your workload is running (see  
> the manpage).  You should also try the ULE scheduler on 7.0.

i apologize - i neglected to mention that we are using ule on 7.0.

what do you guys generally endorse/recommend for local(non-network)  
load/performance testing?

-ben
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to
"[hidden email]"

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: performance degradation in 6.2 when adding a second quad core chip

Kris Kennaway-3
Noisex wrote:

> So Benjamin you want to tell, that adding additional CPU decrease the
> performance on 6.2? How did you made tests?
> I just want to know because in my plans to the next week is also add
> 2nd CPU (dual core) and some additional RAM to one of the my HP Proliant
> DL360 (generaly mysql db server) because due to heavy MySQL usage nearly to
> 100% or even more sometimes some of my customers are not very happy with
> service like that (long load time).
>
> Server: HP proliant G4P DL360, CPU 1x3Ghz dual core, RAM: 4Gb (on system i
> can see 3Gb :) ), SCSI 146Gbx2 (RAID 1+0 with smartarray controller p4600).
> OS: FreeBSD 6.2, SMP

I wouldn't bother with 8 CPUs if you are running 6.x.  As I mentioned,
7.0 SMP performance is much better.

   http://people.freebsd.org/~kris/scaling/7.0%20Preview.pdf

> p.s what you guys could recommend about enabled/disabled hyperthreading on
> intel? Will it increase performance or over the left decrease?

It depends on your workload, it may or may not give a performance
benefit.  ULE tends to perform better than 4BSD though.

Kris

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"