Apparent performance regression 8.3@ -> 8.4@r255966?

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Apparent performance regression 8.3@ -> 8.4@r255966?

David Wolfskill
At work, we have a bunch of machines that developers use to build some
software.  The machines presently run FreeBSD/amd64 8.3-STABLE @rxxxxxx
(with a few local patches, which have since been committed to stable/8),
and the software is built within a 32-bit jail.

The hardware includes 2 packages of 6 physical cores each @3.47GHz
(Intel X5690); SMT is enabled (so the scheduler sees hw.ncpu ==
24).  The memory on the machines was recently increased from 6GB
to 96GB.

I am trying to set up a replacement host environment on my test machine;
the current environment there is FreeBSD/amd64 8.4-STABLE @r255966; this
environment achieves a couple of objectives:

* It has no local patches.
* The known problems (e.g., with mfiutil failing to report battery
  status accurately) are believed to be addressed appropriately.

However: when I do comparison software builds, the new environment is
taking about 12% longer to perform the same work (comparing against a
fair sample of the deployed machines):


Now, when I do these builds, I do so under /usr/bin/time, as well
as using a bit of "scaffolding" I cobbled up (a few years back)
that basically samples a bunch of sysctl OIDs periodically (by
default, every 10 seconds).  Once the build is done, I grab the
file that has the sampled OID data and bring it to my desktop machine
to post-process it; I generate graphs showing (aggregate and per-core)
CPU utilization, as well as Load Averages over the course of the
build.  I can also generate graphs that show how the memory statistics
that "top" displays vary during the course of the build, as well as just
about any univariate OID, and quite a few simple multivariate OIDs
(e.g., kern.cp_time, kern.cp_times, and vm.loadavg).

After seeing the above results and poking around looking for
somewhat-recent tuning information, I ran across a suggestion that the
default of 2MB for vfs.ufs.dirhash_maxmem was probably on the low side.
So I started sampling both vfs.ufs.dirhash_maxmem (mostly to make
documentation of the configuration for a test run easier) and
vfs.ufs.dirhash_mem (to see what we were actually using).  And I tried
quadrupling vfs.ufs.dirhash_maxmem (to 8MB).

The next time I tried a test build, I found that vfs.ufs.dirhash_mem
started at about 3.8MB, climbed fairly steadily, then "clipped" at
8MB, so I quadrupled it again (to 32MB), and found that it climbed
to almost 12MB, then dropped precipitously to about 400KB (and
oscillated between about 400KB & 20MB for the rest of the build,
which appears to be the "packaging" phase).

Despite that increase in vfs.ufs.dirhash_maxmem, this does not
appear to have measurably affected the build times.

In examining the CPU utilization graphs, the CPU generally looks
about 5% busy for the first 15 minutes; this would be bmake determining
dependency graphs, I expect. For the next 1:20, CPU is about 70%
busy (~15% system; ~65% user/nice) for about 20 minutes, then drops
to about 45% busy (~25% system; ~20% user/nice) for the next 20
minutes, and that pattern repeats once.

We then see overall CPU use climb to about 60% (~20% system; ~40%
user/nice) for about 1:20.

Then there's a period of about 2:00 where overall CPU is at about 40%
(~30% system; ~10% user/nice).

Based on earlier work I did, where I was able to do a similar build in a
native FreeBSD/i386 (no PAE) enviroment on the same hardware (but when
it still only had 6GB RAM), and I managed to get the build done in 2:47,
I believe that getting more work done in parallel in this 2:00 period is
a key to improving performance: the 2:47 result showed that period to be
a very busy one for the CPU.

But I am at a loss to understand what might be preventing the work form
getting done (in a timely fashion).

I believe that there were some commits made to stable/9 (MFCed from
head) a few months ago to significantly reduce the overhead of using
jails or using nullfs (or both).  And I'm looking forward to being able
to test that -- but I need to get a "fixed" 8.x environment deployed
first, and a 12% increase in build times is not something that is likely
to be well-received.

Help?

Peace,
david
--
David H. Wolfskill [hidden email]
Taliban: Evil cowards with guns afraid of truth from a 14-year old girl.

See http://www.catwhisker.org/~david/publickey.gpg for my public key.

attachment0 (203 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Apparent performance regression 8.3@ -> 8.4@r255966?

Adrian Chadd-2
On Oct 7, 2013 1:28 PM, "David Wolfskill" <[hidden email]> wrote:
>
> At work, we have a bunch of machines that developers use to build some
> software.  The machines presently run FreeBSD/amd64 8.3-STABLE @rxxxxxx
> (with a few local patches, which have since been committed to stable/8),
> and the software is built within a 32-bit jail.

Well, whats the start commit? Xxxxxx isnt all that helpful. :-)

Once you can establish that, we xan go over the commit logs to see what
changed.

Youre building exactly the same software on both, right?

-adrian

> The hardware includes 2 packages of 6 physical cores each @3.47GHz
> (Intel X5690); SMT is enabled (so the scheduler sees hw.ncpu ==
> 24).  The memory on the machines was recently increased from 6GB
> to 96GB.
>
> I am trying to set up a replacement host environment on my test machine;
> the current environment there is FreeBSD/amd64 8.4-STABLE @r255966; this
> environment achieves a couple of objectives:
>
> * It has no local patches.
> * The known problems (e.g., with mfiutil failing to report battery
>   status accurately) are believed to be addressed appropriately.
>
> However: when I do comparison software builds, the new environment is
> taking about 12% longer to perform the same work (comparing against a
> fair sample of the deployed machines):
>
>
> Now, when I do these builds, I do so under /usr/bin/time, as well
> as using a bit of "scaffolding" I cobbled up (a few years back)
> that basically samples a bunch of sysctl OIDs periodically (by
> default, every 10 seconds).  Once the build is done, I grab the
> file that has the sampled OID data and bring it to my desktop machine
> to post-process it; I generate graphs showing (aggregate and per-core)
> CPU utilization, as well as Load Averages over the course of the
> build.  I can also generate graphs that show how the memory statistics
> that "top" displays vary during the course of the build, as well as just
> about any univariate OID, and quite a few simple multivariate OIDs
> (e.g., kern.cp_time, kern.cp_times, and vm.loadavg).
>
> After seeing the above results and poking around looking for
> somewhat-recent tuning information, I ran across a suggestion that the
> default of 2MB for vfs.ufs.dirhash_maxmem was probably on the low side.
> So I started sampling both vfs.ufs.dirhash_maxmem (mostly to make
> documentation of the configuration for a test run easier) and
> vfs.ufs.dirhash_mem (to see what we were actually using).  And I tried
> quadrupling vfs.ufs.dirhash_maxmem (to 8MB).
>
> The next time I tried a test build, I found that vfs.ufs.dirhash_mem
> started at about 3.8MB, climbed fairly steadily, then "clipped" at
> 8MB, so I quadrupled it again (to 32MB), and found that it climbed
> to almost 12MB, then dropped precipitously to about 400KB (and
> oscillated between about 400KB & 20MB for the rest of the build,
> which appears to be the "packaging" phase).
>
> Despite that increase in vfs.ufs.dirhash_maxmem, this does not
> appear to have measurably affected the build times.
>
> In examining the CPU utilization graphs, the CPU generally looks
> about 5% busy for the first 15 minutes; this would be bmake determining
> dependency graphs, I expect. For the next 1:20, CPU is about 70%
> busy (~15% system; ~65% user/nice) for about 20 minutes, then drops
> to about 45% busy (~25% system; ~20% user/nice) for the next 20
> minutes, and that pattern repeats once.
>
> We then see overall CPU use climb to about 60% (~20% system; ~40%
> user/nice) for about 1:20.
>
> Then there's a period of about 2:00 where overall CPU is at about 40%
> (~30% system; ~10% user/nice).
>
> Based on earlier work I did, where I was able to do a similar build in a
> native FreeBSD/i386 (no PAE) enviroment on the same hardware (but when
> it still only had 6GB RAM), and I managed to get the build done in 2:47,
> I believe that getting more work done in parallel in this 2:00 period is
> a key to improving performance: the 2:47 result showed that period to be
> a very busy one for the CPU.
>
> But I am at a loss to understand what might be preventing the work form
> getting done (in a timely fashion).
>
> I believe that there were some commits made to stable/9 (MFCed from
> head) a few months ago to significantly reduce the overhead of using
> jails or using nullfs (or both).  And I'm looking forward to being able
> to test that -- but I need to get a "fixed" 8.x environment deployed
> first, and a 12% increase in build times is not something that is likely
> to be well-received.
>
> Help?
>
> Peace,
> david
> --
> David H. Wolfskill                              [hidden email]
> Taliban: Evil cowards with guns afraid of truth from a 14-year old girl.
>
> See http://www.catwhisker.org/~david/publickey.gpg for my public key.
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Apparent performance regression 8.3@ -> 8.4@r255966?

David Wolfskill
On Mon, Oct 07, 2013 at 10:32:57AM -0700, Adrian Chadd wrote:
> On Oct 7, 2013 1:28 PM, "David Wolfskill" <[hidden email]> wrote:
> >
> > At work, we have a bunch of machines that developers use to build some
> > software.  The machines presently run FreeBSD/amd64 8.3-STABLE @rxxxxxx
> > (with a few local patches, which have since been committed to stable/8),
> > and the software is built within a 32-bit jail.
>
> Well, whats the start commit? Xxxxxx isnt all that helpful. :-)

r238914M -- sorry; I got impatient & sent th emessage off before I went
through & plugged in the stuff I meant to plug in later.... :-(

> Once you can establish that, we xan go over the commit logs to see what
> changed.
>
> Youre building exactly the same software on both, right?

Yes: I have a script that checks out a "sandbox" from the repo, then
tars it up.  Each test iteration then:

* Ensures there is nothing of an old sandbox (directory) around.

* Unpacks the pristine copy of the sandbox.

* Dives into the sandbox & performs a measured build.

* Exits the sandbox (and removes it).

> ....

Here's the other bit I neglected to include:

dwolf-fbsd(9.2-S)[30] /usr/bin/ministat -s baseline test_env
x baseline
+ test_env
+------------------------------------------------------------------------------+
|   xxxxx xx    xxxxxxxx x  xx      x                     +                    |
|xxxxxxxx xx    xxxxxxxx x  xx xxxx x      x    +x        *+ + ++             x|
|       |___________A____________|                                             |
|                                                     |____A____|              |
+------------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x  75       4.91961       6.16419       5.22432     5.2311616    0.20113192
+   7       5.68327       5.94078       5.86111     5.8563686   0.086090721
Difference at 95.0% confidence
        0.625207 +/- 0.153262
        11.9516% +/- 2.92979%
        (Student's t, pooled s = 0.194874)



Peace,
david
--
David H. Wolfskill [hidden email]
Taliban: Evil cowards with guns afraid of truth from a 14-year old girl.

See http://www.catwhisker.org/~david/publickey.gpg for my public key.

attachment0 (203 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Apparent performance regression 8.3@ -> 8.4@r255966?

Erik Cederstrand-3
In reply to this post by David Wolfskill

Den 07/10/2013 kl. 19.28 skrev David Wolfskill <[hidden email]>:

> In examining the CPU utilization graphs, the CPU generally looks
> about 5% busy for the first 15 minutes; this would be bmake determining
> dependency graphs, I expect.

Is that one process using 100% of one core, or many processes using 5% total?

> Based on earlier work I did, where I was able to do a similar build in a
> native FreeBSD/i386 (no PAE) enviroment on the same hardware (but when
> it still only had 6GB RAM), and I managed to get the build done in 2:47,
> I believe that getting more work done in parallel in this 2:00 period is
> a key to improving performance: the 2:47 result showed that period to be
> a very busy one for the CPU.

You need to know where your bottlenecks are during the build. Since you have lots of RAM, you could try to rule out differences in filesystem access by placing your jail on an mfs and building your software off that. If that improves build times, then you're IO bound at least some of the time. You should be logging disk access along with CPU and memory during the build.

Erik
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Apparent performance regression 8.3@ -> 8.4@r255966?

David Wolfskill
On Mon, Oct 07, 2013 at 09:19:38PM +0200, Erik Cederstrand wrote:
> ...
> > In examining the CPU utilization graphs, the CPU generally looks
> > about 5% busy for the first 15 minutes; this would be bmake determining
> > dependency graphs, I expect.
>
> Is that one process using 100% of one core, or many processes using 5% total?

It's closer to the first, but "which core" varies a bit over time.  (In
one specific instance, core 17, then 13 are at near 100% within that
first 15-minute interval.)

> > Based on earlier work I did, where I was able to do a similar build in a
> > native FreeBSD/i386 (no PAE) enviroment on the same hardware (but when
> > it still only had 6GB RAM), and I managed to get the build done in 2:47,
> > I believe that getting more work done in parallel in this 2:00 period is
> > a key to improving performance: the 2:47 result showed that period to be
> > a very busy one for the CPU.
>
> You need to know where your bottlenecks are during the build.

Indeed.  :-}

> Since you have lots of RAM, you could try to rule out differences in filesystem access by placing your jail on an mfs and building your software off that.

That's an experiment that I've been advocating, but 96GB isn't enough.

> If that improves build times, then you're IO bound at least some of the time. You should be logging disk access along with CPU and memory during the build.

FWIW, I'm seeing very little CPU time spent in interrupt mode.

I'll see if I can find a reasonable way to munge the output of iostat to
be usable for the purpose, I suppose; thanks.

Peace,
david
--
David H. Wolfskill [hidden email]
Taliban: Evil cowards with guns afraid of truth from a 14-year old girl.

See http://www.catwhisker.org/~david/publickey.gpg for my public key.

attachment0 (203 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Apparent performance regression 8.3@ -> 8.4@r255966?

Ivan Voras
In reply to this post by David Wolfskill
On 07/10/2013 19:28, David Wolfskill wrote:> At work, we have a bunch of
machines that developers use to build some

> software.  The machines presently run FreeBSD/amd64 8.3-STABLE @rxxxxxx
> (with a few local patches, which have since been committed to stable/8),
> and the software is built within a 32-bit jail.
>
> The hardware includes 2 packages of 6 physical cores each @3.47GHz
> (Intel X5690); SMT is enabled (so the scheduler sees hw.ncpu ==
> 24).  The memory on the machines was recently increased from 6GB
> to 96GB.
>
> I am trying to set up a replacement host environment on my test machine;
> the current environment there is FreeBSD/amd64 8.4-STABLE @r255966; this
> environment achieves a couple of objectives:
>
> * It has no local patches.
> * The known problems (e.g., with mfiutil failing to report battery
>   status accurately) are believed to be addressed appropriately.
>
> However: when I do comparison software builds, the new environment is
> taking about 12% longer to perform the same work (comparing against a
> fair sample of the deployed machines):
So, the test machine is exactly the same as the old machines? Does the
hardware upgrade coincide with 8.4-STABLE upgrade?

At a guess, you also might be hitting a problem with either NUMA (which
would mean the difference you encountered is pretty much random,
depending on how the memory from your processes was allocated), or a
generic scheduler issue (IIRC, FreeBSD 9 series was found to be much
more scalable for > 16 CPUs).

Just a thought - you *could* set up an 8-STABLE jail in a 9-STABLE
environment if you need the 8-STABLE libraries for your software.




signature.asc (401 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Apparent performance regression 8.3@ -> 8.4@r255966?

Julian Elischer
On 10/10/13 10:02 PM, Ivan Voras wrote:

> On 07/10/2013 19:28, David Wolfskill wrote:> At work, we have a bunch of
> machines that developers use to build some
>> software.  The machines presently run FreeBSD/amd64 8.3-STABLE @rxxxxxx
>> (with a few local patches, which have since been committed to stable/8),
>> and the software is built within a 32-bit jail.
>>
>> The hardware includes 2 packages of 6 physical cores each @3.47GHz
>> (Intel X5690); SMT is enabled (so the scheduler sees hw.ncpu ==
>> 24).  The memory on the machines was recently increased from 6GB
>> to 96GB.
>>
>> I am trying to set up a replacement host environment on my test machine;
>> the current environment there is FreeBSD/amd64 8.4-STABLE @r255966; this
>> environment achieves a couple of objectives:
>>
>> * It has no local patches.
>> * The known problems (e.g., with mfiutil failing to report battery
>>    status accurately) are believed to be addressed appropriately.
>>
>> However: when I do comparison software builds, the new environment is
>> taking about 12% longer to perform the same work (comparing against a
>> fair sample of the deployed machines):
> So, the test machine is exactly the same as the old machines? Does the
> hardware upgrade coincide with 8.4-STABLE upgrade?
>
> At a guess, you also might be hitting a problem with either NUMA (which
> would mean the difference you encountered is pretty much random,
> depending on how the memory from your processes was allocated), or a
> generic scheduler issue (IIRC, FreeBSD 9 series was found to be much
> more scalable for > 16 CPUs).
>
> Just a thought - you *could* set up an 8-STABLE jail in a 9-STABLE
> environment if you need the 8-STABLE libraries for your software.
>
>
>
OR,
take the new kernel and use it to boot the old system
then compare times.

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Apparent performance regression 8.3@ -> 8.4@r255966?

David Wolfskill
In reply to this post by Ivan Voras
On Thu, Oct 10, 2013 at 04:02:47PM +0200, Ivan Voras wrote:
> On 07/10/2013 19:28, David Wolfskill wrote:> At work, we have a bunch of
> machines that developers use to build some
> > software.  The machines presently run FreeBSD/amd64 8.3-STABLE @rxxxxxx
> > (with a few local patches, which have since been committed to stable/8),
> > and the software is built within a 32-bit jail.
> ....
> So, the test machine is exactly the same as the old machines? Does the
> hardware upgrade coincide with 8.4-STABLE upgrade?

The only hardware upgrade was to increase RAM from 6GB to 96GB,
which was done for all of the machines being discussed.  The machine
I'm using for testing configurations now is the one I used to
test configurations before we bought the machines we've now deployed.

I cite 8.4 as a (vague) reference point; more specifically, I used a
snapshot of stable/8.  More precisely, it's stab le/8 @r255978.

> At a guess, you also might be hitting a problem with either NUMA (which
> would mean the difference you encountered is pretty much random,
> depending on how the memory from your processes was allocated), or a
> generic scheduler issue (IIRC, FreeBSD 9 series was found to be much
> more scalable for > 16 CPUs).

I'm hoping to demonstrate that -- but first, I need to get demonstrate
something that fixes some known bugs while not saddling the developers
with a 12% increase in build times.  And I've been trying (when my
test machine has been available to me) to get this done for nearly
a year, now.  (There is also a perception that "jumping" from 8.x
to 9.x is "scary".  I am reminded of the "It's just a leaf!" sequence
near the beginning of "A Bug's Life.")

> Just a thought - you *could* set up an 8-STABLE jail in a 9-STABLE
> environment if you need the 8-STABLE libraries for your software.

The "32-bit jail" is actually a 7.1-R [+ a few patches] environment as
it is.

Thanks for responding. :-}

Peace,
david
--
David H. Wolfskill [hidden email]
Taliban: Evil cowards with guns afraid of truth from a 14-year old girl.

See http://www.catwhisker.org/~david/publickey.gpg for my public key.

attachment0 (203 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Apparent performance regression 8.3@ -> 8.4@r255966?

Adrian Chadd-2
In reply to this post by David Wolfskill
Hi David,

How about just picking a revision of stable/8 half way between the good and
not good version, then re-test?

It'd help to narrow down the range of commits that could've caused problems.



-adrian
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: Apparent performance regression 8.3@ -> 8.4@r255966?

Anton Yuzhaninov
In reply to this post by David Wolfskill
On Fri, 11 Oct 2013 08:57:54, David Wolfskill wrote:
DW> The only hardware upgrade was to increase RAM from 6GB to 96GB,
DW> which was done for all of the machines being discussed.

FYI: RAM size increase can cause performance regression, probably due to
decreased CPU TLB cache hit rate.

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"