PostgreSQL performance on FreeBSD

classic Classic list List threaded Threaded
36 messages Options
12
Reply | Threaded
Open this post in threaded view
|

PostgreSQL performance on FreeBSD

Konstantin Belousov
Hi,
I did some measurements and hacks to see about the performance and
scalability of PostgreSQL 9.3 on FreeBSD, sponsored by The FreeBSD
Foundation.

The results are described in https://kib.kiev.ua/kib/pgsql_perf.pdf.
The uncommitted patches, referenced in the article, are available as
https://kib.kiev.ua/kib/pig1.patch.txt
https://kib.kiev.ua/kib/patch-2

attachment0 (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: PostgreSQL performance on FreeBSD

Mark Felder
June 27 2014 7:56 AM, "Konstantin Belousov"  wrote:

> Hi,
> I did some measurements and hacks to see about the performance and
> scalability of PostgreSQL 9.3 on FreeBSD, sponsored by The FreeBSD
> Foundation.
>
> The results are described in https://kib.kiev.ua/kib/pgsql_perf.pdf.
> The uncommitted patches, referenced in the article, are available as
> https://kib.kiev.ua/kib/pig1.patch.txt
> https://kib.kiev.ua/kib/patch-2
>

Thank you for taking the time to do this benchmark.
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PostgreSQL performance on FreeBSD

John Baldwin
In reply to this post by Konstantin Belousov
On Friday, June 27, 2014 8:56:13 am Konstantin Belousov wrote:
> Hi,
> I did some measurements and hacks to see about the performance and
> scalability of PostgreSQL 9.3 on FreeBSD, sponsored by The FreeBSD
> Foundation.
>
> The results are described in https://kib.kiev.ua/kib/pgsql_perf.pdf.
> The uncommitted patches, referenced in the article, are available as
> https://kib.kiev.ua/kib/pig1.patch.txt
> https://kib.kiev.ua/kib/patch-2

Did you run the same benchmark on the same hardware with any other OS's to
compare results?

--
John Baldwin
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PostgreSQL performance on FreeBSD

Tobias Grosser-4
In reply to this post by Konstantin Belousov
On 27/06/2014 14:56, Konstantin Belousov wrote:
> Hi,
> I did some measurements and hacks to see about the performance and
> scalability of PostgreSQL 9.3 on FreeBSD, sponsored by The FreeBSD
> Foundation.
>
> The results are described in https://kib.kiev.ua/kib/pgsql_perf.pdf.
> The uncommitted patches, referenced in the article, are available as
> https://kib.kiev.ua/kib/pig1.patch.txt
> https://kib.kiev.ua/kib/patch-2

Interesting. Did you report the clang bug upstream?

Tobias

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PostgreSQL performance on FreeBSD

Konstantin Belousov
In reply to this post by John Baldwin
On Fri, Jun 27, 2014 at 10:57:53AM -0400, John Baldwin wrote:

> On Friday, June 27, 2014 8:56:13 am Konstantin Belousov wrote:
> > Hi,
> > I did some measurements and hacks to see about the performance and
> > scalability of PostgreSQL 9.3 on FreeBSD, sponsored by The FreeBSD
> > Foundation.
> >
> > The results are described in https://kib.kiev.ua/kib/pgsql_perf.pdf.
> > The uncommitted patches, referenced in the article, are available as
> > https://kib.kiev.ua/kib/pig1.patch.txt
> > https://kib.kiev.ua/kib/patch-2
>
> Did you run the same benchmark on the same hardware with any other OS's to
> compare results?
No.

FWIW, before the failing after the 30 clients is corrected, I do not
think it is much interesting to do such comparision.

attachment0 (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: PostgreSQL performance on FreeBSD

Palle Girgensohn-2


27 jun 2014 kl. 18:34 skrev Konstantin Belousov <[hidden email]>:

> On Fri, Jun 27, 2014 at 10:57:53AM -0400, John Baldwin wrote:
>> On Friday, June 27, 2014 8:56:13 am Konstantin Belousov wrote:
>>> Hi,
>>> I did some measurements and hacks to see about the performance and
>>> scalability of PostgreSQL 9.3 on FreeBSD, sponsored by The FreeBSD
>>> Foundation.
>>>
>>> The results are described in https://kib.kiev.ua/kib/pgsql_perf.pdf.
>>> The uncommitted patches, referenced in the article, are available as
>>> https://kib.kiev.ua/kib/pig1.patch.txt
>>> https://kib.kiev.ua/kib/patch-2
>>
>> Did you run the same benchmark on the same hardware with any other OS's to
>> compare results?
>
> No.
>
> FWIW, before the failing after the 30 clients is corrected, I do not
> think it is much interesting to do such comparision.

This is great work!

Does anybody know how far back in FreeBSD versions using posix semaphore instead of sysv would make a difference?  It seems we need a rather current version? 8.x did not support it at all, at some point at lest, and in 9 it was buggy. I could add he patch-2 to the port, but I reckon it needs a conditional based on FreeBSD version?

The clang bug should go upstreams, right?

I have seen similar curves, presented by Greg Smith (PostgreSQL hacker) where he concluded that there is no point in running more than 50 concurrent connections. This was for Linux. In your measures, the knee is at 30. That's said, FreeBSD could and should do better, but probably there is a limit where there will be a knee in the graph and performance will drop. It should be more than 30, though, as you rightly commented.

Do you any ideas to pursue this further apart from complicated rewrites like DragonFly?

Palle
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PostgreSQL performance on FreeBSD

Konstantin Belousov
On Sat, Jun 28, 2014 at 12:08:39PM +0200, Palle Girgensohn wrote:

>
>
> 27 jun 2014 kl. 18:34 skrev Konstantin Belousov <[hidden email]>:
>
> > On Fri, Jun 27, 2014 at 10:57:53AM -0400, John Baldwin wrote:
> >> On Friday, June 27, 2014 8:56:13 am Konstantin Belousov wrote:
> >>> Hi,
> >>> I did some measurements and hacks to see about the performance and
> >>> scalability of PostgreSQL 9.3 on FreeBSD, sponsored by The FreeBSD
> >>> Foundation.
> >>>
> >>> The results are described in https://kib.kiev.ua/kib/pgsql_perf.pdf.
> >>> The uncommitted patches, referenced in the article, are available as
> >>> https://kib.kiev.ua/kib/pig1.patch.txt
> >>> https://kib.kiev.ua/kib/patch-2
> >>
> >> Did you run the same benchmark on the same hardware with any other OS's to
> >> compare results?
> >
> > No.
> >
> > FWIW, before the failing after the 30 clients is corrected, I do not
> > think it is much interesting to do such comparision.
>
> This is great work!
>
> Does anybody know how far back in FreeBSD versions using posix semaphore instead of sysv would make a difference?  It seems we need a rather current version? 8.x did not support it at all, at some point at lest, and in 9 it was buggy. I could add he patch-2 to the port, but I reckon it needs a conditional based on FreeBSD version?
>
I recommend to add it as an option.  The currently supported versions
of stable/9 and higher have new posix semaphores implementation.
The stable/8 also has posix semaphores, but there it is kernel-based
interface, I do not plan to evaluate it in any way.


> The clang bug should go upstreams, right?
I believe there is already some activity about it.  I do not follow
clang development.

>
> I have seen similar curves, presented by Greg Smith (PostgreSQL
> hacker) where he concluded that there is no point in running more
> than 50 concurrent connections. This was for Linux. In your measures,
> the knee is at 30. That's said, FreeBSD could and should do better,
> but probably there is a limit where there will be a knee in the graph
> and performance will drop. It should be more than 30, though, as you
> rightly commented.
>
> Do you any ideas to pursue this further apart from complicated
> rewrites like DragonFly?
>
I do.

The scope of the current work was done to obtain understanding where do we stay
and, if possible, evaluate ideas, possibly in the hackish way. I hope
and almost sure that this will be continued, but cannot provide any time
estimation.

attachment0 (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: PostgreSQL performance on FreeBSD

Palle Girgensohn-2


> 28 jun 2014 kl. 12:21 skrev Konstantin Belousov <[hidden email]>:
>
>> On Sat, Jun 28, 2014 at 12:08:39PM +0200, Palle Girgensohn wrote:
>>
>>
>>> 27 jun 2014 kl. 18:34 skrev Konstantin Belousov <[hidden email]>:
>>>
>>>> On Fri, Jun 27, 2014 at 10:57:53AM -0400, John Baldwin wrote:
>>>>> On Friday, June 27, 2014 8:56:13 am Konstantin Belousov wrote:
>>>>> Hi,
>>>>> I did some measurements and hacks to see about the performance and
>>>>> scalability of PostgreSQL 9.3 on FreeBSD, sponsored by The FreeBSD
>>>>> Foundation.
>>>>>
>>>>> The results are described in https://kib.kiev.ua/kib/pgsql_perf.pdf.
>>>>> The uncommitted patches, referenced in the article, are available as
>>>>> https://kib.kiev.ua/kib/pig1.patch.txt
>>>>> https://kib.kiev.ua/kib/patch-2
>>>>
>>>> Did you run the same benchmark on the same hardware with any other OS's to
>>>> compare results?
>>>
>>> No.
>>>
>>> FWIW, before the failing after the 30 clients is corrected, I do not
>>> think it is much interesting to do such comparision.
>>
>> This is great work!
>>
>> Does anybody know how far back in FreeBSD versions using posix semaphore instead of sysv would make a difference?  It seems we need a rather current version? 8.x did not support it at all, at some point at lest, and in 9 it was buggy. I could add he patch-2 to the port, but I reckon it needs a conditional based on FreeBSD version?
> I recommend to add it as an option.  The currently supported versions
> of stable/9 and higher have new posix semaphores implementation.
> The stable/8 also has posix semaphores, but there it is kernel-based
> interface, I do not plan to evaluate it in any way.

According to one source, posix semaphores uses O(N^2) file descriptors, where N is the number of connections. Do you know if this is true? (I'll try it, naturally, just checking).

>
>
>> The clang bug should go upstreams, right?
> I believe there is already some activity about it.  I do not follow
> clang development.

Sounds good enough.

>
>>
>> I have seen similar curves, presented by Greg Smith (PostgreSQL
>> hacker) where he concluded that there is no point in running more
>> than 50 concurrent connections. This was for Linux. In your measures,
>> the knee is at 30. That's said, FreeBSD could and should do better,
>> but probably there is a limit where there will be a knee in the graph
>> and performance will drop. It should be more than 30, though, as you
>> rightly commented.
>>
>> Do you any ideas to pursue this further apart from complicated
>> rewrites like DragonFly?
> I do.
>
> The scope of the current work was done to obtain understanding where do we stay
> and, if possible, evaluate ideas, possibly in the hackish way. I hope
> and almost sure that this will be continued, but cannot provide any time
> estimation.

Great. If you need help testing, I might be able to help.

Cheers,
Palle
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PostgreSQL performance on FreeBSD

Konstantin Belousov
On Sat, Jun 28, 2014 at 01:37:20PM +0200, Palle Girgensohn wrote:

>
>
> > 28 jun 2014 kl. 12:21 skrev Konstantin Belousov <[hidden email]>:
> >
> >> On Sat, Jun 28, 2014 at 12:08:39PM +0200, Palle Girgensohn wrote:
> >>
> >>
> >>> 27 jun 2014 kl. 18:34 skrev Konstantin Belousov <[hidden email]>:
> >>>
> >>>> On Fri, Jun 27, 2014 at 10:57:53AM -0400, John Baldwin wrote:
> >>>>> On Friday, June 27, 2014 8:56:13 am Konstantin Belousov wrote:
> >>>>> Hi,
> >>>>> I did some measurements and hacks to see about the performance and
> >>>>> scalability of PostgreSQL 9.3 on FreeBSD, sponsored by The FreeBSD
> >>>>> Foundation.
> >>>>>
> >>>>> The results are described in https://kib.kiev.ua/kib/pgsql_perf.pdf.
> >>>>> The uncommitted patches, referenced in the article, are available as
> >>>>> https://kib.kiev.ua/kib/pig1.patch.txt
> >>>>> https://kib.kiev.ua/kib/patch-2
> >>>>
> >>>> Did you run the same benchmark on the same hardware with any other OS's to
> >>>> compare results?
> >>>
> >>> No.
> >>>
> >>> FWIW, before the failing after the 30 clients is corrected, I do not
> >>> think it is much interesting to do such comparision.
> >>
> >> This is great work!
> >>
> >> Does anybody know how far back in FreeBSD versions using posix semaphore instead of sysv would make a difference?  It seems we need a rather current version? 8.x did not support it at all, at some point at lest, and in 9 it was buggy. I could add he patch-2 to the port, but I reckon it needs a conditional based on FreeBSD version?
> > I recommend to add it as an option.  The currently supported versions
> > of stable/9 and higher have new posix semaphores implementation.
> > The stable/8 also has posix semaphores, but there it is kernel-based
> > interface, I do not plan to evaluate it in any way.
>
> According to one source, posix semaphores uses O(N^2) file descriptors, where N is the number of connections. Do you know if this is true? (I'll try it, naturally, just checking).
>
(New) posix semaphores implementation, done by David Xu, opens a file
descriptor during the sem_open(3), which is used to mmap the area carrying
the lock word, and is immediately closed afterward in sem_open().  In
other words, if you have N semaphores and M processes, there would
be N*M open(2)/close(2) pairs, and N files, each mmaped to M processes.
New implementation does not use file descriptor during semaphore use,
and does not keep the backing file open.

> >
> >
> >> The clang bug should go upstreams, right?
> > I believe there is already some activity about it.  I do not follow
> > clang development.
>
> Sounds good enough.
>
> >
> >>
> >> I have seen similar curves, presented by Greg Smith (PostgreSQL
> >> hacker) where he concluded that there is no point in running more
> >> than 50 concurrent connections. This was for Linux. In your measures,
> >> the knee is at 30. That's said, FreeBSD could and should do better,
> >> but probably there is a limit where there will be a knee in the graph
> >> and performance will drop. It should be more than 30, though, as you
> >> rightly commented.
> >>
> >> Do you any ideas to pursue this further apart from complicated
> >> rewrites like DragonFly?
> > I do.
> >
> > The scope of the current work was done to obtain understanding where do we stay
> > and, if possible, evaluate ideas, possibly in the hackish way. I hope
> > and almost sure that this will be continued, but cannot provide any time
> > estimation.
>
> Great. If you need help testing, I might be able to help.
I have the test set up and the graphing mostly automated, although
the repeat of the configuration would be quite laborous.

On the other hand, if you get access to zoo, replication could be easier.

attachment0 (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: PostgreSQL performance on FreeBSD

Rang, Anton
In reply to this post by Konstantin Belousov
Thanks for this.

The cpu_search problem you reference came up here at Isilon as well.  Here's a patch which should get clang to do the "right thing" (inlining 3 specialized copies of cpu_search); I haven't checked to make sure it doesn't hurt gcc, though.

Anton

Index: sched_ule.c
===================================================================
--- sched_ule.c (revision 268043)
+++ sched_ule.c (working copy)
@@ -622,11 +622,11 @@
  for ((cpu) = 0; (cpu) <= mp_maxid; (cpu)++) \
  if (CPU_ISSET(cpu, &mask))
 
-static __inline int cpu_search(const struct cpu_group *cg, struct cpu_search *low,
+static __always_inline int cpu_search(const struct cpu_group *cg, struct cpu_search *low,
     struct cpu_search *high, const int match);
-int cpu_search_lowest(const struct cpu_group *cg, struct cpu_search *low);
-int cpu_search_highest(const struct cpu_group *cg, struct cpu_search *high);
-int cpu_search_both(const struct cpu_group *cg, struct cpu_search *low,
+int __noinline cpu_search_lowest(const struct cpu_group *cg, struct cpu_search *low);
+int __noinline cpu_search_highest(const struct cpu_group *cg, struct cpu_search *high);
+int __noinline cpu_search_both(const struct cpu_group *cg, struct cpu_search *low,
     struct cpu_search *high);
 
 /*
@@ -640,7 +640,7 @@
  * match argument.  It is reduced to the minimum set for each case.  It is
  * also recursive to the depth of the tree.
  */
-static __inline int
+static __always_inline int
 cpu_search(const struct cpu_group *cg, struct cpu_search *low,
     struct cpu_search *high, const int match)
 {

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Konstantin Belousov
Sent: Friday, June 27, 2014 7:56 AM
To: [hidden email]
Cc: [hidden email]
Subject: PostgreSQL performance on FreeBSD

Hi,
I did some measurements and hacks to see about the performance and scalability of PostgreSQL 9.3 on FreeBSD, sponsored by The FreeBSD Foundation.

The results are described in https://kib.kiev.ua/kib/pgsql_perf.pdf.
The uncommitted patches, referenced in the article, are available as https://kib.kiev.ua/kib/pig1.patch.txt
https://kib.kiev.ua/kib/patch-2
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PostgreSQL performance on FreeBSD

Konstantin Belousov
In reply to this post by Konstantin Belousov
On Fri, Jun 27, 2014 at 03:56:13PM +0300, Konstantin Belousov wrote:
> Hi,
> I did some measurements and hacks to see about the performance and
> scalability of PostgreSQL 9.3 on FreeBSD, sponsored by The FreeBSD
> Foundation.
>
> The results are described in https://kib.kiev.ua/kib/pgsql_perf.pdf.
> The uncommitted patches, referenced in the article, are available as
> https://kib.kiev.ua/kib/pig1.patch.txt
> https://kib.kiev.ua/kib/patch-2

A followup to the original paper.

Most importantly, I identified the cause for the drop on the graph
after the 30 clients, which appeared to be the debugging version
of malloc(3) in libc.

Also there are some updates on the patches.

New version of the paper is available at
https://www.kib.kiev.ua/kib/pgsql_perf_v2.0.pdf
The changes are marked as 'update for version 2.0'.

attachment0 (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: PostgreSQL performance on FreeBSD

Adrian Chadd-2
Hi!


On 16 July 2014 06:29, Konstantin Belousov <[hidden email]> wrote:

> On Fri, Jun 27, 2014 at 03:56:13PM +0300, Konstantin Belousov wrote:
>> Hi,
>> I did some measurements and hacks to see about the performance and
>> scalability of PostgreSQL 9.3 on FreeBSD, sponsored by The FreeBSD
>> Foundation.
>>
>> The results are described in https://kib.kiev.ua/kib/pgsql_perf.pdf.
>> The uncommitted patches, referenced in the article, are available as
>> https://kib.kiev.ua/kib/pig1.patch.txt
>> https://kib.kiev.ua/kib/patch-2
>
> A followup to the original paper.
>
> Most importantly, I identified the cause for the drop on the graph
> after the 30 clients, which appeared to be the debugging version
> of malloc(3) in libc.
>
> Also there are some updates on the patches.
>
> New version of the paper is available at
> https://www.kib.kiev.ua/kib/pgsql_perf_v2.0.pdf
> The changes are marked as 'update for version 2.0'.

Would you mind trying a default (non-PRODUCTION) build, but with junk
filling turned off?

adrian@adrian-hackbox:~ % ls -l /etc/malloc.conf

lrwxr-xr-x  1 root  wheel  10 Jun 24 04:37 /etc/malloc.conf -> junk:false

That fixes almost all of the malloc debug performance issues that I
see without having to recompile.

I'd like to know if you see any after that.

Thanks!



-a
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PostgreSQL performance on FreeBSD

Hooman Fazaeli-3
In reply to this post by Konstantin Belousov
On 7/16/2014 5:59 PM, Konstantin Belousov wrote:

> On Fri, Jun 27, 2014 at 03:56:13PM +0300, Konstantin Belousov wrote:
>> Hi,
>> I did some measurements and hacks to see about the performance and
>> scalability of PostgreSQL 9.3 on FreeBSD, sponsored by The FreeBSD
>> Foundation.
>>
>> The results are described in https://kib.kiev.ua/kib/pgsql_perf.pdf.
>> The uncommitted patches, referenced in the article, are available as
>> https://kib.kiev.ua/kib/pig1.patch.txt
>> https://kib.kiev.ua/kib/patch-2
> A followup to the original paper.
>
> Most importantly, I identified the cause for the drop on the graph
> after the 30 clients, which appeared to be the debugging version
> of malloc(3) in libc.
>
> Also there are some updates on the patches.
>
> New version of the paper is available at
> https://www.kib.kiev.ua/kib/pgsql_perf_v2.0.pdf
> The changes are marked as 'update for version 2.0'.

Thanks for the great work!

Did you tested the effect of hyper-threading (on or off) on the results?


--

Best regards.
Hooman Fazaeli

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PostgreSQL performance on FreeBSD

O. Hartmann-4
Am Thu, 17 Jul 2014 20:15:39 +0430
Hooman Fazaeli <[hidden email]> schrieb:

> On 7/16/2014 5:59 PM, Konstantin Belousov wrote:
> > On Fri, Jun 27, 2014 at 03:56:13PM +0300, Konstantin Belousov wrote:
> >> Hi,
> >> I did some measurements and hacks to see about the performance and
> >> scalability of PostgreSQL 9.3 on FreeBSD, sponsored by The FreeBSD
> >> Foundation.
> >>
> >> The results are described in https://kib.kiev.ua/kib/pgsql_perf.pdf.
> >> The uncommitted patches, referenced in the article, are available as
> >> https://kib.kiev.ua/kib/pig1.patch.txt
> >> https://kib.kiev.ua/kib/patch-2
> > A followup to the original paper.
> >
> > Most importantly, I identified the cause for the drop on the graph
> > after the 30 clients, which appeared to be the debugging version
> > of malloc(3) in libc.
> >
> > Also there are some updates on the patches.
> >
> > New version of the paper is available at
> > https://www.kib.kiev.ua/kib/pgsql_perf_v2.0.pdf
> > The changes are marked as 'update for version 2.0'.
>
> Thanks for the great work!
>
> Did you tested the effect of hyper-threading (on or off) on the results?
>
>

A "naive" question besides:

Does this labor and effort only affects the work with the PostgreSQL 9.3 database and is
recent FreeBSD only optimized for this servicing puprpose or provides this also some
benefeits for other high-performance scenarios?

Oliver

signature.asc (484 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: PostgreSQL performance on FreeBSD

John Baldwin
In reply to this post by Adrian Chadd-2
On Wednesday, July 16, 2014 1:52:45 pm Adrian Chadd wrote:

> Hi!
>
>
> On 16 July 2014 06:29, Konstantin Belousov <[hidden email]> wrote:
> > On Fri, Jun 27, 2014 at 03:56:13PM +0300, Konstantin Belousov wrote:
> >> Hi,
> >> I did some measurements and hacks to see about the performance and
> >> scalability of PostgreSQL 9.3 on FreeBSD, sponsored by The FreeBSD
> >> Foundation.
> >>
> >> The results are described in https://kib.kiev.ua/kib/pgsql_perf.pdf.
> >> The uncommitted patches, referenced in the article, are available as
> >> https://kib.kiev.ua/kib/pig1.patch.txt
> >> https://kib.kiev.ua/kib/patch-2
> >
> > A followup to the original paper.
> >
> > Most importantly, I identified the cause for the drop on the graph
> > after the 30 clients, which appeared to be the debugging version
> > of malloc(3) in libc.
> >
> > Also there are some updates on the patches.
> >
> > New version of the paper is available at
> > https://www.kib.kiev.ua/kib/pgsql_perf_v2.0.pdf
> > The changes are marked as 'update for version 2.0'.
>
> Would you mind trying a default (non-PRODUCTION) build, but with junk
> filling turned off?
>
> adrian@adrian-hackbox:~ % ls -l /etc/malloc.conf
>
> lrwxr-xr-x  1 root  wheel  10 Jun 24 04:37 /etc/malloc.conf -> junk:false
>
> That fixes almost all of the malloc debug performance issues that I
> see without having to recompile.
>
> I'd like to know if you see any after that.

OTOH, I have actually seen junk profiling _improve_ performance in certain
cases as it forces promotion of allocated pages to superpages since all pages
are dirtied.  (I have a local hack that adds a new malloc option to explicitly
memset() new pages allocated via mmap() that gives the same benefit without
the junking overheadon each malloc() / free(), but it does increase physical
RAM usage.)

--
John Baldwin
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PostgreSQL performance on FreeBSD

Adrian Chadd-2
On 12 August 2014 11:09, John Baldwin <[hidden email]> wrote:

> On Wednesday, July 16, 2014 1:52:45 pm Adrian Chadd wrote:
>> Hi!
>>
>>
>> On 16 July 2014 06:29, Konstantin Belousov <[hidden email]> wrote:
>> > On Fri, Jun 27, 2014 at 03:56:13PM +0300, Konstantin Belousov wrote:
>> >> Hi,
>> >> I did some measurements and hacks to see about the performance and
>> >> scalability of PostgreSQL 9.3 on FreeBSD, sponsored by The FreeBSD
>> >> Foundation.
>> >>
>> >> The results are described in https://kib.kiev.ua/kib/pgsql_perf.pdf.
>> >> The uncommitted patches, referenced in the article, are available as
>> >> https://kib.kiev.ua/kib/pig1.patch.txt
>> >> https://kib.kiev.ua/kib/patch-2
>> >
>> > A followup to the original paper.
>> >
>> > Most importantly, I identified the cause for the drop on the graph
>> > after the 30 clients, which appeared to be the debugging version
>> > of malloc(3) in libc.
>> >
>> > Also there are some updates on the patches.
>> >
>> > New version of the paper is available at
>> > https://www.kib.kiev.ua/kib/pgsql_perf_v2.0.pdf
>> > The changes are marked as 'update for version 2.0'.
>>
>> Would you mind trying a default (non-PRODUCTION) build, but with junk
>> filling turned off?
>>
>> adrian@adrian-hackbox:~ % ls -l /etc/malloc.conf
>>
>> lrwxr-xr-x  1 root  wheel  10 Jun 24 04:37 /etc/malloc.conf -> junk:false
>>
>> That fixes almost all of the malloc debug performance issues that I
>> see without having to recompile.
>>
>> I'd like to know if you see any after that.
>
> OTOH, I have actually seen junk profiling _improve_ performance in certain
> cases as it forces promotion of allocated pages to superpages since all pages
> are dirtied.  (I have a local hack that adds a new malloc option to explicitly
> memset() new pages allocated via mmap() that gives the same benefit without
> the junking overheadon each malloc() / free(), but it does increase physical
> RAM usage.)

Hm. this isn't a jemalloc config option?


-a
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PostgreSQL performance on FreeBSD

David Chisnall-2
In reply to this post by John Baldwin
On 12 Aug 2014, at 19:09, John Baldwin <[hidden email]> wrote:

> OTOH, I have actually seen junk profiling _improve_ performance in certain
> cases as it forces promotion of allocated pages to superpages since all pages
> are dirtied.  (I have a local hack that adds a new malloc option to explicitly
> memset() new pages allocated via mmap() that gives the same benefit without
> the junking overheadon each malloc() / free(), but it does increase physical
> RAM usage.)

Do you get the same effect by adding MAP_ALIGNED_SUPER | MAP_PREFAULT_READ to the mmap() call in jemalloc?  I've been meaning to try the latter on BERI, as we spend a lot of time bouncing back and forth between user code and the TLB miss handlers.  Given that jemalloc asks for memory in 8MB chunks (I think via a single mmap call, although I'm not 100% certain), MAP_ALIGNED_SUPER should have little impact on any platform.  MAP_PREFAULT_READ may cause problems on machines with limited RAM and no swap (I don't know if the VM subsystem knows that it can safely discard a zero'd page that has been read but not written - I'd hope so, but it's been a while since I read that code).

It might be that we can make jemalloc autotune whether to use MAP_PREFAULT_READ depending on some heuristic.  I wonder if something as simple as 'turn it on after the first mmap call' would be enough: programs that don't use more than 8MB of RAM won't prefault, but after that the wasted physical memory becomes an increasingly small percentage.

David

_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PostgreSQL performance on FreeBSD

Alan Cox-9
On Wed, Aug 13, 2014 at 4:58 AM, David Chisnall <[hidden email]>
wrote:

> On 12 Aug 2014, at 19:09, John Baldwin <[hidden email]> wrote:
>
> > OTOH, I have actually seen junk profiling _improve_ performance in
> certain
> > cases as it forces promotion of allocated pages to superpages since all
> pages
> > are dirtied.  (I have a local hack that adds a new malloc option to
> explicitly
> > memset() new pages allocated via mmap() that gives the same benefit
> without
> > the junking overheadon each malloc() / free(), but it does increase
> physical
> > RAM usage.)
>
> Do you get the same effect by adding MAP_ALIGNED_SUPER | MAP_PREFAULT_READ
> to the mmap() call in jemalloc?



No.  MAP_PREFAULT_READ does not allocate physical pages.  It establishes
mappings to pages that are already allocated.



> I've been meaning to try the latter on BERI, as we spend a lot of time
> bouncing back and forth between user code and the TLB miss handlers.  Given
> that jemalloc asks for memory in 8MB chunks (I think via a single mmap
> call, although I'm not 100% certain), MAP_ALIGNED_SUPER should have little
> impact on any platform.  MAP_PREFAULT_READ may cause problems on machines
> with limited RAM and no swap (I don't know if the VM subsystem knows that
> it can safely discard a zero'd page that has been read but not written -
> I'd hope so, but it's been a while since I read that code).
>
> It might be that we can make jemalloc autotune whether to use
> MAP_PREFAULT_READ depending on some heuristic.  I wonder if something as
> simple as 'turn it on after the first mmap call' would be enough: programs
> that don't use more than 8MB of RAM won't prefault, but after that the
> wasted physical memory becomes an increasingly small percentage.
>
> David
>
> _______________________________________________
> [hidden email] mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "[hidden email]"
>
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PostgreSQL performance on FreeBSD

Alan Cox-9
In reply to this post by John Baldwin
On Tue, Aug 12, 2014 at 1:09 PM, John Baldwin <[hidden email]> wrote:

> On Wednesday, July 16, 2014 1:52:45 pm Adrian Chadd wrote:
> > Hi!
> >
> >
> > On 16 July 2014 06:29, Konstantin Belousov <[hidden email]> wrote:
> > > On Fri, Jun 27, 2014 at 03:56:13PM +0300, Konstantin Belousov wrote:
> > >> Hi,
> > >> I did some measurements and hacks to see about the performance and
> > >> scalability of PostgreSQL 9.3 on FreeBSD, sponsored by The FreeBSD
> > >> Foundation.
> > >>
> > >> The results are described in https://kib.kiev.ua/kib/pgsql_perf.pdf.
> > >> The uncommitted patches, referenced in the article, are available as
> > >> https://kib.kiev.ua/kib/pig1.patch.txt
> > >> https://kib.kiev.ua/kib/patch-2
> > >
> > > A followup to the original paper.
> > >
> > > Most importantly, I identified the cause for the drop on the graph
> > > after the 30 clients, which appeared to be the debugging version
> > > of malloc(3) in libc.
> > >
> > > Also there are some updates on the patches.
> > >
> > > New version of the paper is available at
> > > https://www.kib.kiev.ua/kib/pgsql_perf_v2.0.pdf
> > > The changes are marked as 'update for version 2.0'.
> >
> > Would you mind trying a default (non-PRODUCTION) build, but with junk
> > filling turned off?
> >
> > adrian@adrian-hackbox:~ % ls -l /etc/malloc.conf
> >
> > lrwxr-xr-x  1 root  wheel  10 Jun 24 04:37 /etc/malloc.conf -> junk:false
> >
> > That fixes almost all of the malloc debug performance issues that I
> > see without having to recompile.
> >
> > I'd like to know if you see any after that.
>
> OTOH, I have actually seen junk profiling _improve_ performance in certain
> cases as it forces promotion of allocated pages to superpages since all
> pages
> are dirtied.  (I have a local hack that adds a new malloc option to
> explicitly
> memset() new pages allocated via mmap() that gives the same benefit without
> the junking overheadon each malloc() / free(), but it does increase
> physical
> RAM usage.)
>
>

John,

A couple small steps have been taken toward eliminating the need for this
hack: the addition of the "page size index" field to struct vm_page and the
addition of a similarly named parameter to pmap_enter().  However, at the
moment, the only tangible effect is in the automatic prefaulting by
mmap(2).  Instead of establishing 96 4KB page mappings, the automatic
prefaulting establishes 96 page mappings whose size is determined by the
size of the physical pages that it finds in the vm object.  So, the
prefaulting overhead remains constant, but the coverage provided by the
automatic prefaulting will vary with the underlying page size.
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: PostgreSQL performance on FreeBSD

John Baldwin
On Wednesday, August 13, 2014 1:00:22 pm Alan Cox wrote:

> On Tue, Aug 12, 2014 at 1:09 PM, John Baldwin <[hidden email]> wrote:
>
> > On Wednesday, July 16, 2014 1:52:45 pm Adrian Chadd wrote:
> > > Hi!
> > >
> > >
> > > On 16 July 2014 06:29, Konstantin Belousov <[hidden email]> wrote:
> > > > On Fri, Jun 27, 2014 at 03:56:13PM +0300, Konstantin Belousov wrote:
> > > >> Hi,
> > > >> I did some measurements and hacks to see about the performance and
> > > >> scalability of PostgreSQL 9.3 on FreeBSD, sponsored by The FreeBSD
> > > >> Foundation.
> > > >>
> > > >> The results are described in https://kib.kiev.ua/kib/pgsql_perf.pdf.
> > > >> The uncommitted patches, referenced in the article, are available as
> > > >> https://kib.kiev.ua/kib/pig1.patch.txt
> > > >> https://kib.kiev.ua/kib/patch-2
> > > >
> > > > A followup to the original paper.
> > > >
> > > > Most importantly, I identified the cause for the drop on the graph
> > > > after the 30 clients, which appeared to be the debugging version
> > > > of malloc(3) in libc.
> > > >
> > > > Also there are some updates on the patches.
> > > >
> > > > New version of the paper is available at
> > > > https://www.kib.kiev.ua/kib/pgsql_perf_v2.0.pdf
> > > > The changes are marked as 'update for version 2.0'.
> > >
> > > Would you mind trying a default (non-PRODUCTION) build, but with junk
> > > filling turned off?
> > >
> > > adrian@adrian-hackbox:~ % ls -l /etc/malloc.conf
> > >
> > > lrwxr-xr-x  1 root  wheel  10 Jun 24 04:37 /etc/malloc.conf -> junk:false
> > >
> > > That fixes almost all of the malloc debug performance issues that I
> > > see without having to recompile.
> > >
> > > I'd like to know if you see any after that.
> >
> > OTOH, I have actually seen junk profiling _improve_ performance in certain
> > cases as it forces promotion of allocated pages to superpages since all
> > pages
> > are dirtied.  (I have a local hack that adds a new malloc option to
> > explicitly
> > memset() new pages allocated via mmap() that gives the same benefit without
> > the junking overheadon each malloc() / free(), but it does increase
> > physical
> > RAM usage.)
> >
> >
>
> John,
>
> A couple small steps have been taken toward eliminating the need for this
> hack: the addition of the "page size index" field to struct vm_page and the
> addition of a similarly named parameter to pmap_enter().  However, at the
> moment, the only tangible effect is in the automatic prefaulting by
> mmap(2).  Instead of establishing 96 4KB page mappings, the automatic
> prefaulting establishes 96 page mappings whose size is determined by the
> size of the physical pages that it finds in the vm object.  So, the
> prefaulting overhead remains constant, but the coverage provided by the
> automatic prefaulting will vary with the underlying page size.

Yes, I think what we might actually want is what I mentioned in person at
BSDCan: some sort of flag to mmap() that malloc() could use to assume that any
reservations are fully used when they are reserved.  This would avoid the need
to wait for all pages to be dirtied before promotion provides a superpage
mapping and would avoid demotions while still allowing the kernel to gracefully
fall back to regular pages if a reservation can't be made.

--
John Baldwin
_______________________________________________
[hidden email] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "[hidden email]"
12