RFC: Automatically Reloading /etc/resolv.conf

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

RFC: Automatically Reloading /etc/resolv.conf

Eric van Gyzen-5
I would like to change the libc resolver to automatically reload
/etc/resolv.conf when the latter changes.  I would like to hear opinions
about the implementation.  Broadly, I see two approaches.

== "stat" ==

When loading the file, record the mod time.  Before each query, stat()
the file to see if it has changed.

Advantage:  It uses no extra persistently allocated objects.

Disadvantage:  It incurs a stat() on every query.  I don't see this as a
major disadvantage, since the resolver already does a lot of work on
every query.  (For example, it creates and destroys a kqueue and a socket.)

OpenBSD uses this approach.  It also uses clock_gettime(CLOCK_MONOTONIC)
to rate-limit the stat() calls to one per several seconds.

== "kqueue" ==

When loading the file, open a kqueue and register for the appropriate
events.  Before each query, check for kevents.

Advantage:  The per-query overhead is fairly small.

Disadvantage:  This would persistently allocate an open file and a
kqueue for every thread that ever uses the resolver, for the life of the
thread.  This seems fairly expensive.

NetBSD uses this approach.  It mitigates most of the space-cost by using
a shared pool of res_state objects, instead of one per thread [that uses
the resolver].  On each query, a thread allocates/borrows a res_state
from the pool, uses it, and returns it.  So, the number of objects is
only the high water mark of the number of threads _concurrently_ issuing
resolver queries.

There are probably several variations on each theme, of course.  I would
appreciate your thoughts on these approaches and others I missed, as
well as variations and details.

FYI, I'm leaning toward the "stat" approach.

Cheers,

Eric
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Automatically Reloading /etc/resolv.conf

Jilles Tjoelker
On Wed, Oct 07, 2015 at 04:02:39PM -0500, Eric van Gyzen wrote:
> I would like to change the libc resolver to automatically reload
> /etc/resolv.conf when the latter changes.  I would like to hear opinions
> about the implementation.  Broadly, I see two approaches.

> == "stat" ==

> When loading the file, record the mod time.  Before each query, stat()
> the file to see if it has changed.

> Advantage:  It uses no extra persistently allocated objects.

> Disadvantage:  It incurs a stat() on every query.  I don't see this as a
> major disadvantage, since the resolver already does a lot of work on
> every query.  (For example, it creates and destroys a kqueue and a socket.)

> OpenBSD uses this approach.  It also uses clock_gettime(CLOCK_MONOTONIC)
> to rate-limit the stat() calls to one per several seconds.

This looks reasonable.

Nitpick: the resolver has used poll instead of kqueue for a while, but
that does not fundamentally change your argument.

Some glibc people think the extra stat may be too slow, though:
https://sourceware.org/bugzilla/show_bug.cgi?id=984

> == "kqueue" ==

> When loading the file, open a kqueue and register for the appropriate
> events.  Before each query, check for kevents.

> Advantage:  The per-query overhead is fairly small.

> Disadvantage:  This would persistently allocate an open file and a
> kqueue for every thread that ever uses the resolver, for the life of the
> thread.  This seems fairly expensive.

This sounds a bit scary in conjunction with code that bluntly closes
file descriptors it does not know about.

Also, kqueues do not inherit across fork, so the resolver needs some
sort of atfork handler.

> NetBSD uses this approach.  It mitigates most of the space-cost by using
> a shared pool of res_state objects, instead of one per thread [that uses
> the resolver].  On each query, a thread allocates/borrows a res_state
> from the pool, uses it, and returns it.  So, the number of objects is
> only the high water mark of the number of threads _concurrently_ issuing
> resolver queries.

Is there code that depends on implicit per-thread res_state objects? If
so, this will break it.

> There are probably several variations on each theme, of course.  I would
> appreciate your thoughts on these approaches and others I missed, as
> well as variations and details.

> FYI, I'm leaning toward the "stat" approach.

This seems safe, and being able to rely on automatic /etc/resolv.conf
reloading can simplify application code considerably.

The NextBSD people have a more efficient alternative notify(3). Using
that, resolvconf(8) can cause a counter in shared memory to be
incremented, which can be detected efficiently by the resolver in each
process. The stat() approach need not wait for this, though.

--
Jilles Tjoelker
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Automatically Reloading /etc/resolv.conf

Warner Losh
In reply to this post by Eric van Gyzen-5
On Wed, Oct 7, 2015 at 3:02 PM, Eric van Gyzen <[hidden email]> wrote:

> I would like to change the libc resolver to automatically reload
> /etc/resolv.conf when the latter changes.  I would like to hear opinions
> about the implementation.  Broadly, I see two approaches.
>
> == "stat" ==
>
> When loading the file, record the mod time.  Before each query, stat()
> the file to see if it has changed.
>
> Advantage:  It uses no extra persistently allocated objects.
>
> Disadvantage:  It incurs a stat() on every query.  I don't see this as a
> major disadvantage, since the resolver already does a lot of work on
> every query.  (For example, it creates and destroys a kqueue and a socket.)
>
> OpenBSD uses this approach.  It also uses clock_gettime(CLOCK_MONOTONIC)
> to rate-limit the stat() calls to one per several seconds.
>

Make sure you rate limit it. Calling stat at a high rate from multiple CPUs
exposes poor locking
scaleability and eats a lot of CPU. We had some setting wrong on NGINX at
work that lead to
large NGINX latency spikes because this contention was blocking the worker
threads. Correcting
the settings to cache the results of stat made this problem disappear.


> == "kqueue" ==
>
> When loading the file, open a kqueue and register for the appropriate
> events.  Before each query, check for kevents.
>
> Advantage:  The per-query overhead is fairly small.
>
> Disadvantage:  This would persistently allocate an open file and a
> kqueue for every thread that ever uses the resolver, for the life of the
> thread.  This seems fairly expensive.
>

Why does this follow? Can't you have a global one for the process?


> NetBSD uses this approach.  It mitigates most of the space-cost by using
> a shared pool of res_state objects, instead of one per thread [that uses
> the resolver].  On each query, a thread allocates/borrows a res_state
> from the pool, uses it, and returns it.  So, the number of objects is
> only the high water mark of the number of threads _concurrently_ issuing
> resolver queries.n
>

What's the locking scaleability here when resolv.conf changes?

There are probably several variations on each theme, of course.  I would
> appreciate your thoughts on these approaches and others I missed, as
> well as variations and details.
>
> FYI, I'm leaning toward the "stat" approach.
>

It sounds simpler to implement, but likely a higher overhead than the
kqueue approach. resolv.conf changes on an time scale measured in minutes
or hours for the mobile user, and on the scale of years for servers.
Polling on the scale of seconds (at least two orders of magnitude faster
than the rate of change) seems like a lot of extra work over the life of
resolv.conf.

Warner
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Automatically Reloading /etc/resolv.conf

Hajimu UMEMOTO
In reply to this post by Eric van Gyzen-5
Hi,

>>>>> On Wed, 7 Oct 2015 16:02:39 -0500
>>>>> Eric van Gyzen <[hidden email]> said:

vangyzen> I would like to change the libc resolver to automatically reload
vangyzen> /etc/resolv.conf when the latter changes.  I would like to hear opinions
vangyzen> about the implementation.

When I proposed it in the past, it ended up with dns/nss_resinit port.
Please take a look at it for reference.

Sincerely,

--
Hajimu UMEMOTO
[hidden email]  [hidden email]
http://www.mahoroba.org/~ume/
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Automatically Reloading /etc/resolv.conf

Eric van Gyzen-5
In reply to this post by Jilles Tjoelker
> Nitpick: the resolver has used poll instead of kqueue for a while, but
> that does not fundamentally change your argument.

I was looking at a 10.2-RELEASE system.  Indeed, I see it using poll on
head.  Thanks.

> Some glibc people think the extra stat may be too slow, though:
> https://sourceware.org/bugzilla/show_bug.cgi?id=984

I added simple code to call stat() on every query and ran a benchmark.
It reduces queries-per-second by 15.87%, so I definitely won't take that
approach.

>> When loading the file, open a kqueue and register for the appropriate
>> events.  Before each query, check for kevents.
>
> This sounds a bit scary in conjunction with code that bluntly closes
> file descriptors it does not know about.

That's true, but such code would break a variety of other facilities,
such as syslog.  I'll keep this in mind.

> Also, kqueues do not inherit across fork, so the resolver needs some
> sort of atfork handler.

I hadn't thought of that.  Thanks.

>> NetBSD uses this approach.  It mitigates most of the space-cost by using
>> a shared pool of res_state objects, instead of one per thread [that uses
>> the resolver].  On each query, a thread allocates/borrows a res_state
>> from the pool, uses it, and returns it.  So, the number of objects is
>> only the high water mark of the number of threads _concurrently_ issuing
>> resolver queries.
>
> Is there code that depends on implicit per-thread res_state objects? If
> so, this will break it.

I'm just describing NetBSD's current approach.  I don't plan to adopt
this in FreeBSD.  To answer your question, though, I don't know if such
code exists.

> The NextBSD people have a more efficient alternative notify(3). Using
> that, resolvconf(8) can cause a counter in shared memory to be
> incremented, which can be detected efficiently by the resolver in each
> process. The stat() approach need not wait for this, though.

I didn't know about this.  It sounds rather useful.  Thanks!

Eric
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Automatically Reloading /etc/resolv.conf

Eric van Gyzen-5
In reply to this post by Warner Losh
On 10/08/2015 11:13, Warner Losh wrote:
>
> Make sure you rate limit it.

Agreed.  As I just wrote in my other reply, calling stat() on every
query reduces queries-per-second by 15.87%.  Rate-limiting to one stat()
every ten seconds fixes it.  That is, by manually running three or four
benchmarks with and without my changes, I see no real difference.  This
was on amd64 where the cost of clock_gettime() is trivialized by vdso.

>> Disadvantage:  This would persistently allocate an open file and a
>> kqueue for every thread that ever uses the resolver, for the life of the
>> thread.  This seems fairly expensive.
>>
>
> Why does this follow? Can't you have a global one for the process?

Well, gee, that seems like a good compromise, now that you mention it.
;-)  Seriously, I simply didn't consider that.

Given that the rate-limited stat() approach is so cheap, I wonder if
it's worth the trouble to try the global kqueue approach.

>> NetBSD uses this approach.  It mitigates most of the space-cost by using
>> a shared pool of res_state objects, instead of one per thread [that uses
>> the resolver].  On each query, a thread allocates/borrows a res_state
>> from the pool, uses it, and returns it.  So, the number of objects is
>> only the high water mark of the number of threads _concurrently_ issuing
>> resolver queries.n
>>
>
> What's the locking scaleability here when resolv.conf changes?

The mutex is only held during one or two SLIST operations, so the
locking is independent of resolv.conf changing.  Maybe I misunderstood
your question?

>> FYI, I'm leaning toward the "stat" approach.
>>
>
> It sounds simpler to implement, but likely a higher overhead than the
> kqueue approach. resolv.conf changes on an time scale measured in minutes
> or hours for the mobile user, and on the scale of years for servers.
> Polling on the scale of seconds (at least two orders of magnitude faster
> than the rate of change) seems like a lot of extra work over the life of
> resolv.conf.

On the other hand, when changes _are_ needed, you don't want to restart
critical services in order to make them effective.  (This is my
motivation, in fact.)  It would be trivial to add a "no-reload" option
so performance-sensitive users can turn it off.  In fact, I half
expected this request from someone focused on embedded work, such as
yourself.  ;-)

Eric
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Automatically Reloading /etc/resolv.conf

Eric van Gyzen-5
In reply to this post by Hajimu UMEMOTO
On 10/08/2015 11:30, Hajimu UMEMOTO wrote:
> vangyzen> I would like to change the libc resolver to automatically reload
> vangyzen> /etc/resolv.conf when the latter changes.  I would like to hear opinions
> vangyzen> about the implementation.
>
> When I proposed it in the past, it ended up with dns/nss_resinit port.
> Please take a look at it for reference.

Nice.  That's exactly what I want to do in libc.  Doing it in nss is a
clever way to avoid changing libc.  Thanks for the pointer.

Eric
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: RFC: Automatically Reloading /etc/resolv.conf

Eric van Gyzen-5
In reply to this post by Eric van Gyzen-5
On 10/7/15 4:02 PM, Eric van Gyzen wrote:
> I would like to change the libc resolver to automatically reload
> /etc/resolv.conf when the latter changes.

I implemented and tested the rate-limited stat(2) approach.
I would be grateful for a review:

        https://reviews.freebsd.org/D3867

Eric
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "[hidden email]"