13-CURRENT: several GB swap being used despite plenty of free RAM

classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|

13-CURRENT: several GB swap being used despite plenty of free RAM

freebsd-hackers mailing list
I'm running 13-CURRENT from a few days ago. I noticed the system using several
GB of swap despite having 90GB RAM still free. I know FreeBSD will use *some*,
but 3450MB seems excessive when there's still 90GB RAM free.

CPU:  0.1% user,  0.0% nice,  0.0% system,  0.0% interrupt, 99.8% idle
Mem: 3392M Active, 5489M Inact, 390M Laundry, 25G Wired, 90G Free
ARC: 19G Total, 8080M MFU, 9242M MRU, 64K Anon, 197M Header, 1570M Other
     15G Compressed, 26G Uncompressed, 1.73:1 Ratio
Swap: 8192M Total, 3450M Used, 4742M Free, 42% Inuse, 36K In

Quitting firefox caused the swap usage to drop to just 460MB. I don't
understand why it would decide to swap out so much, especially since I have
vm.swap_idle_enabled set to 0.

--
Rebecca


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 13-CURRENT: several GB swap being used despite plenty of free RAM

Wojciech Puchar-8
freebsd will not swap with that lots of free ram.
but it's 90GB free NOW, how about before?

If something got swapped it isn't read back just because there is free
memory but when it will be needed.

On Sat, 17 Nov 2018, Rebecca Cran via freebsd-hackers wrote:

> I'm running 13-CURRENT from a few days ago. I noticed the system using several
> GB of swap despite having 90GB RAM still free. I know FreeBSD will use *some*,
> but 3450MB seems excessive when there's still 90GB RAM free.
>
> CPU:  0.1% user,  0.0% nice,  0.0% system,  0.0% interrupt, 99.8% idle
> Mem: 3392M Active, 5489M Inact, 390M Laundry, 25G Wired, 90G Free
> ARC: 19G Total, 8080M MFU, 9242M MRU, 64K Anon, 197M Header, 1570M Other
>     15G Compressed, 26G Uncompressed, 1.73:1 Ratio
> Swap: 8192M Total, 3450M Used, 4742M Free, 42% Inuse, 36K In
>
> Quitting firefox caused the swap usage to drop to just 460MB. I don't
> understand why it would decide to swap out so much, especially since I have
> vm.swap_idle_enabled set to 0.
>
> --
> Rebecca
>
>
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "[hidden email]"
>
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 13-CURRENT: several GB swap being used despite plenty of free RAM

freebsd-hackers mailing list
In reply to this post by freebsd-hackers mailing list
On Saturday, 17 November 2018 14:44:41 MST Rebecca Cran via freebsd-hackers
wrote:

> Quitting firefox caused the swap usage to drop to just 460MB. I don't
> understand why it would decide to swap out so much, especially since I have
> vm.swap_idle_enabled set to 0.

Of course it was because just before looking at top, I'd been running
something that used up all the memory. I'd run "grep -rHn" and it had wanted
to use over 120GB memory, which caused all the swapouts.

--
Rebecca


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 13-CURRENT: several GB swap being used despite plenty of free RAM

freebsd-hackers mailing list
In reply to this post by Wojciech Puchar-8
On Saturday, 17 November 2018 14:52:22 MST Wojciech Puchar wrote:
> freebsd will not swap with that lots of free ram.
> but it's 90GB free NOW, how about before?
>
> If something got swapped it isn't read back just because there is free
> memory but when it will be needed.

That was it: just before, I'd run grep and canceled it after it reached about
120GB memory usage!

--
Rebecca


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 13-CURRENT: several GB swap being used despite plenty of free RAM

Cy Schubert-4
In reply to this post by freebsd-hackers mailing list
In message <[hidden email]>, Rebecca Cran
via freeb
sd-hackers writes:

> I'm running 13-CURRENT from a few days ago. I noticed the system using severa
> l
> GB of swap despite having 90GB RAM still free. I know FreeBSD will use *some*
> ,
> but 3450MB seems excessive when there's still 90GB RAM free.
>
> CPU:  0.1% user,  0.0% nice,  0.0% system,  0.0% interrupt, 99.8% idle
> Mem: 3392M Active, 5489M Inact, 390M Laundry, 25G Wired, 90G Free
> ARC: 19G Total, 8080M MFU, 9242M MRU, 64K Anon, 197M Header, 1570M Other
>      15G Compressed, 26G Uncompressed, 1.73:1 Ratio
> Swap: 8192M Total, 3450M Used, 4742M Free, 42% Inuse, 36K In
>
> Quitting firefox caused the swap usage to drop to just 460MB. I don't
> understand why it would decide to swap out so much, especially since I have
> vm.swap_idle_enabled set to 0.

Were you by chance building any ports at the time? Or, possibly
extracting a tarball?


--
Cheers,
Cy Schubert <[hidden email]>
FreeBSD UNIX:  <[hidden email]>   Web:  http://www.FreeBSD.org

        The need of the many outweighs the greed of the few.


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 13-CURRENT: several GB swap being used despite plenty of free RAM

Ian Lepore-3
In reply to this post by Wojciech Puchar-8
On Sat, 2018-11-17 at 22:52 +0100, Wojciech Puchar wrote:
> freebsd will not swap with that lots of free ram.
> but it's 90GB free NOW, how about before?
>

Your information is outdated. For at least a couple years now (since
approximately the 10.1 - 10.2 timeframe is my vague estimate), freebsd
will page out application memory that hasn't been referenced for some
time, even when the system has no shortage of free memory at all.

The advice I was recently given to revert to the old behavior is:

  sysctl vm.pageout_update_period=0

I've been using it on a couple systems here for a few days now, and so
far results are promising, I am no longer seeing gratuitous swapfile
usage on systems that have so much free physical ram that they should
never need to page anything out. I haven't yet pushed one of those
systems hard enough to check what happens when they do need to start
proactively paging out inactive memory due to shortages -- it could be
that turning off the new behavior has downsides for some workloads.

-- Ian



> If something got swapped it isn't read back just because there is
> free 
> memory but when it will be needed.
>

On Sat, 17 Nov 2018, Rebecca Cran via freebsd-hackers wrote:

>
> >
> > I'm running 13-CURRENT from a few days ago. I noticed the system
> > using several
> > GB of swap despite having 90GB RAM still free. I know FreeBSD will
> > use *some*,
> > but 3450MB seems excessive when there's still 90GB RAM free.
> >
> > CPU:  0.1% user,  0.0% nice,  0.0% system,  0.0% interrupt, 99.8%
> > idle
> > Mem: 3392M Active, 5489M Inact, 390M Laundry, 25G Wired, 90G Free
> > ARC: 19G Total, 8080M MFU, 9242M MRU, 64K Anon, 197M Header, 1570M
> > Other
> >     15G Compressed, 26G Uncompressed, 1.73:1 Ratio
> > Swap: 8192M Total, 3450M Used, 4742M Free, 42% Inuse, 36K In
> >
> > Quitting firefox caused the swap usage to drop to just 460MB. I
> > don't
> > understand why it would decide to swap out so much, especially
> > since I have
> > vm.swap_idle_enabled set to 0.
> >
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 13-CURRENT: several GB swap being used despite plenty of free RAM

Mark Johnston-2
On Sat, Nov 17, 2018 at 04:59:48PM -0700, Ian Lepore wrote:
> On Sat, 2018-11-17 at 22:52 +0100, Wojciech Puchar wrote:
> > freebsd will not swap with that lots of free ram.
> > but it's 90GB free NOW, how about before?
> >
>
> Your information is outdated. For at least a couple years now (since
> approximately the 10.1 - 10.2 timeframe is my vague estimate), freebsd
> will page out application memory that hasn't been referenced for some
> time, even when the system has no shortage of free memory at all.

No, FreeBSD will only ever swap when there is a free page shortage.  The
difference is that we now slowly age unreferenced pages into the
inactive queue, which makes them candidates for pageout and subsequent
eviction.  With pageout_update_period=0, anonymous memory won't get
paged out unless there's a shortage of inactive pages, or an application
calls madvise(MADV_DONTNEED) on a range of memory (which moves any
backing pages to the inactive queue).

> The advice I was recently given to revert to the old behavior is:
>
>   sysctl vm.pageout_update_period=0
>
> I've been using it on a couple systems here for a few days now, and so
> far results are promising, I am no longer seeing gratuitous swapfile
> usage on systems that have so much free physical ram that they should
> never need to page anything out. I haven't yet pushed one of those
> systems hard enough to check what happens when they do need to start
> proactively paging out inactive memory due to shortages -- it could be
> that turning off the new behavior has downsides for some workloads.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 13-CURRENT: several GB swap being used despite plenty of free RAM

freebsd-hackers mailing list
On 2018-Nov-17, at 16:13, Mark Johnston <markj at freebsd.org> wrote:

>
> On Sat, Nov 17, 2018 at 04:59:48PM -0700, Ian Lepore wrote:
>> On Sat, 2018-11-17 at 22:52 +0100, Wojciech Puchar wrote:
>>> freebsd will not swap with that lots of free ram.
>>> but it's 90GB free NOW, how about before?
>>>
>>
>> Your information is outdated. For at least a couple years now (since
>> approximately the 10.1 - 10.2 timeframe is my vague estimate), freebsd
>> will page out application memory that hasn't been referenced for some
>> time, even when the system has no shortage of free memory at all.
>
> No, FreeBSD will only ever swap when there is a free page shortage.  The
> difference is that we now slowly age unreferenced pages into the
> inactive queue, which makes them candidates for pageout and subsequent
> eviction.  With pageout_update_period=0, anonymous memory won't get
> paged out unless there's a shortage of inactive pages, or an application
> calls madvise(MADV_DONTNEED) on a range of memory (which moves any
> backing pages to the inactive queue).

Swapping is built on top of paging as I understand. The system
can page without swapping but can not swap without (effectively)
paging to implement the swapping, if I understand right. If I
understand right, swapped-out means that kernel stacks have
been written out and have to be loaded back in RAM before the
process/threads can even run. (I might not understand.)

If I've got that right, are there distinctions here for
paging that is not part of swapping vs. actual swapping
(and its use of paging)? Saying that something does not
swap does not necessarily imply that it does not page:
it still could have paging activity that does not include
moving the kernel stacks for the process to backing media?

At times I have trouble interpreting when wording goes back
and forth between swapping and paging, both for the intended
meaning and for the technical implications.

>> The advice I was recently given to revert to the old behavior is:
>>
>>   sysctl vm.pageout_update_period=0
>>
>> I've been using it on a couple systems here for a few days now, and so
>> far results are promising, I am no longer seeing gratuitous swapfile
>> usage on systems that have so much free physical ram that they should
>> never need to page anything out. I haven't yet pushed one of those
>> systems hard enough to check what happens when they do need to start
>> proactively paging out inactive memory due to shortages -- it could be
>> that turning off the new behavior has downsides for some workloads.




===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 13-CURRENT: several GB swap being used despite plenty of free RAM

Mark Johnston-2
On Sat, Nov 17, 2018 at 04:41:55PM -0800, Mark Millard wrote:

> On 2018-Nov-17, at 16:13, Mark Johnston <markj at freebsd.org> wrote:
> >
> > On Sat, Nov 17, 2018 at 04:59:48PM -0700, Ian Lepore wrote:
> >> On Sat, 2018-11-17 at 22:52 +0100, Wojciech Puchar wrote:
> >>> freebsd will not swap with that lots of free ram.
> >>> but it's 90GB free NOW, how about before?
> >>>
> >>
> >> Your information is outdated. For at least a couple years now (since
> >> approximately the 10.1 - 10.2 timeframe is my vague estimate), freebsd
> >> will page out application memory that hasn't been referenced for some
> >> time, even when the system has no shortage of free memory at all.
> >
> > No, FreeBSD will only ever swap when there is a free page shortage.  The
> > difference is that we now slowly age unreferenced pages into the
> > inactive queue, which makes them candidates for pageout and subsequent
> > eviction.  With pageout_update_period=0, anonymous memory won't get
> > paged out unless there's a shortage of inactive pages, or an application
> > calls madvise(MADV_DONTNEED) on a range of memory (which moves any
> > backing pages to the inactive queue).
>
> Swapping is built on top of paging as I understand. The system
> can page without swapping but can not swap without (effectively)
> paging to implement the swapping, if I understand right.

Right.

> If I
> understand right, swapped-out means that kernel stacks have
> been written out and have to be loaded back in RAM before the
> process/threads can even run. (I might not understand.)

When free pages are scarce, one measure that the kernel may take to
address the shortage is to swap out the kernel stacks of the threads in
a process, thus allowing the pages backing the stacks to be reused for
some other purpose, but preventing that process from running on a CPU
until the stacks are swapped back in and "locked" (wired) into memory.

Most of the pages consumed by an application like firefox are not used
for kernel stacks.  Most of them are used for the application's heap
memory, and are thus private to that process.  In general, pieces of
such memory are subject to being paged out to the swap device,
particularly when they are not frequently referenced (read or written
to) by the application, in order to replenish the pool of free pages.
Such memory is often said to be swapped out.

As a side note, there are some system calls that modify this behaviour.
mlock(2) effectively prevents the kernel from swapping out pages backing
the specified virtual addresses; this guarantees that an access of the
virtual memory range will never incur the cost of an expensive page-in.
madvise(MADV_FREE) tells the kernel that the specified pages may be
freed without first being written to the swap device.  Thus, a
subsequent read of an affected page may return the page's previous
contents (if the page had not yet been reclaimed to make up for a
shortage), or all zeroes (if the page had been freed without saving its
contents to swap).

> If I've got that right, are there distinctions here for
> paging that is not part of swapping vs. actual swapping
> (and its use of paging)? Saying that something does not
> swap does not necessarily imply that it does not page:
> it still could have paging activity that does not include
> moving the kernel stacks for the process to backing media?

Indeed, as I tried to describe above, kernel stack swapouts usually
represent only a small portion of many applications' total swap
usage, to the point where I at least usually don't think much about
them.

> At times I have trouble interpreting when wording goes back
> and forth between swapping and paging, both for the intended
> meaning and for the technical implications.

When discussing paging activity related to the swap pager, I
typically use those terms interchangeably.  Sorry for the confusion.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 13-CURRENT: several GB swap being used despite plenty of free RAM

Cy Schubert-4
In reply to this post by freebsd-hackers mailing list
In message <[hidden email]>, Mark
Millard via f
reebsd-hackers writes:

> On 2018-Nov-17, at 16:13, Mark Johnston <markj at freebsd.org> wrote:
> >
> > On Sat, Nov 17, 2018 at 04:59:48PM -0700, Ian Lepore wrote:
> >> On Sat, 2018-11-17 at 22:52 +0100, Wojciech Puchar wrote:
> >>> freebsd will not swap with that lots of free ram.
> >>> but it's 90GB free NOW, how about before?
> >>>
> >>
> >> Your information is outdated. For at least a couple years now (since
> >> approximately the 10.1 - 10.2 timeframe is my vague estimate), freebsd
> >> will page out application memory that hasn't been referenced for some
> >> time, even when the system has no shortage of free memory at all.
> >
> > No, FreeBSD will only ever swap when there is a free page shortage.  The
> > difference is that we now slowly age unreferenced pages into the
> > inactive queue, which makes them candidates for pageout and subsequent
> > eviction.  With pageout_update_period=0, anonymous memory won't get
> > paged out unless there's a shortage of inactive pages, or an application
> > calls madvise(MADV_DONTNEED) on a range of memory (which moves any
> > backing pages to the inactive queue).
>
> Swapping is built on top of paging as I understand. The system
> can page without swapping but can not swap without (effectively)
> paging to implement the swapping, if I understand right. If I
> understand right, swapped-out means that kernel stacks have
> been written out and have to be loaded back in RAM before the
> process/threads can even run. (I might not understand.)
>
> If I've got that right, are there distinctions here for
> paging that is not part of swapping vs. actual swapping
> (and its use of paging)? Saying that something does not
> swap does not necessarily imply that it does not page:
> it still could have paging activity that does not include
> moving the kernel stacks for the process to backing media?

This is generally the old-school definition, IIRC third year comp sci,
original BSD definition, which was also the way IBM defined it for MVS.
(IBM also virtually swapped out address spaces, not written to DASD, as
a means to deny CPU cycles through the scheduler not given the chance
to consider the tasks (threads) in the address space).

You can disable swapping by setting vm.swap_enabled=0.

>
> At times I have trouble interpreting when wording goes back
> and forth between swapping and paging, both for the intended
> meaning and for the technical implications.
>
> >> The advice I was recently given to revert to the old behavior is:
> >>
> >>   sysctl vm.pageout_update_period=0
> >>
> >> I've been using it on a couple systems here for a few days now, and so
> >> far results are promising, I am no longer seeing gratuitous swapfile
> >> usage on systems that have so much free physical ram that they should
> >> never need to page anything out. I haven't yet pushed one of those
> >> systems hard enough to check what happens when they do need to start
> >> proactively paging out inactive memory due to shortages -- it could be
> >> that turning off the new behavior has downsides for some workloads.
>
>
>
>
> ===
> Mark Millard
> marklmi at yahoo.com
> ( dsl-only.net went
> away in early 2018-Mar)
>
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "[hidden email]"
>

--
Cheers,
Cy Schubert <[hidden email]>
FreeBSD UNIX:  <[hidden email]>   Web:  http://www.FreeBSD.org

        The need of the many outweighs the greed of the few.


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 13-CURRENT: several GB swap being used despite plenty of free RAM

Stefan Blachmann
The inconveniences that the new swapping strategy causes are a regular
topic in the FreeBSD forums.

Desktop users complain about lagginess, server users complain of long
delays because server processes intended to be kept in memory for
quick response times got swapped out and need to be swapped in again,
resulting in outrageously poor server performance in spite of plenty
of unused memory.

Turning off swap completely, as Cy Schubert suggests, is strongly
discouraged in the forums, as it can lead to kernel panicking because
of being unable to swap out in critical kernel memory shortage
situations, leading to the risk of  very serious filesystem
corruption.

However, Cy Schubert is probably right when stating that the new
swapping strategy resembles the 1960s-1980s industry's main swapping
strategy.
The bad thing is now, that nowadays memory is no longer scarce and
people can dimension their memory such that under normal circumstances
there will never be any need to swap.

So I guess the unwillingness of the developer team to add an option
like "NoPreemptiveSwapping", which disables swapping out as long as
there is free physical memory available, is of psychological nature.

Lacking such an option, there is still the possibility to use rctl to
disable swapping for particular users, processes, jails etc to
mitigate the problems caused by the new swapping strategy to some
degree.

On 11/18/18, Cy Schubert <[hidden email]> wrote:

> In message <[hidden email]>, Mark
> Millard via f
> reebsd-hackers writes:
>> On 2018-Nov-17, at 16:13, Mark Johnston <markj at freebsd.org> wrote:
>> >
>> > On Sat, Nov 17, 2018 at 04:59:48PM -0700, Ian Lepore wrote:
>> >> On Sat, 2018-11-17 at 22:52 +0100, Wojciech Puchar wrote:
>> >>> freebsd will not swap with that lots of free ram.
>> >>> but it's 90GB free NOW, how about before?
>> >>>
>> >>
>> >> Your information is outdated. For at least a couple years now (since
>> >> approximately the 10.1 - 10.2 timeframe is my vague estimate), freebsd
>> >> will page out application memory that hasn't been referenced for some
>> >> time, even when the system has no shortage of free memory at all.
>> >
>> > No, FreeBSD will only ever swap when there is a free page shortage.
>> > The
>> > difference is that we now slowly age unreferenced pages into the
>> > inactive queue, which makes them candidates for pageout and subsequent
>> > eviction.  With pageout_update_period=0, anonymous memory won't get
>> > paged out unless there's a shortage of inactive pages, or an
>> > application
>> > calls madvise(MADV_DONTNEED) on a range of memory (which moves any
>> > backing pages to the inactive queue).
>>
>> Swapping is built on top of paging as I understand. The system
>> can page without swapping but can not swap without (effectively)
>> paging to implement the swapping, if I understand right. If I
>> understand right, swapped-out means that kernel stacks have
>> been written out and have to be loaded back in RAM before the
>> process/threads can even run. (I might not understand.)
>>
>> If I've got that right, are there distinctions here for
>> paging that is not part of swapping vs. actual swapping
>> (and its use of paging)? Saying that something does not
>> swap does not necessarily imply that it does not page:
>> it still could have paging activity that does not include
>> moving the kernel stacks for the process to backing media?
>
> This is generally the old-school definition, IIRC third year comp sci,
> original BSD definition, which was also the way IBM defined it for MVS.
> (IBM also virtually swapped out address spaces, not written to DASD, as
> a means to deny CPU cycles through the scheduler not given the chance
> to consider the tasks (threads) in the address space).
>
> You can disable swapping by setting vm.swap_enabled=0.
>
>>
>> At times I have trouble interpreting when wording goes back
>> and forth between swapping and paging, both for the intended
>> meaning and for the technical implications.
>>
>> >> The advice I was recently given to revert to the old behavior is:
>> >>
>> >>   sysctl vm.pageout_update_period=0
>> >>
>> >> I've been using it on a couple systems here for a few days now, and so
>> >> far results are promising, I am no longer seeing gratuitous swapfile
>> >> usage on systems that have so much free physical ram that they should
>> >> never need to page anything out. I haven't yet pushed one of those
>> >> systems hard enough to check what happens when they do need to start
>> >> proactively paging out inactive memory due to shortages -- it could be
>> >> that turning off the new behavior has downsides for some workloads.
>>
>>
>>
>>
>> ===
>> Mark Millard
>> marklmi at yahoo.com
>> ( dsl-only.net went
>> away in early 2018-Mar)
>>
>> _______________________________________________
>> [hidden email] mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
>> To unsubscribe, send any mail to
>> "[hidden email]"
>>
>
> --
> Cheers,
> Cy Schubert <[hidden email]>
> FreeBSD UNIX:  <[hidden email]>   Web:  http://www.FreeBSD.org
>
> The need of the many outweighs the greed of the few.
>
>
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "[hidden email]"
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 13-CURRENT: several GB swap being used despite plenty of free RAM

George Mitchell
On 11/18/18 7:11 AM, Stefan Blachmann wrote:
> The inconveniences that the new swapping strategy causes are a regular
> topic in the FreeBSD forums.
>
> Desktop users complain about lagginess, server users complain of long
> delays because server processes intended to be kept in memory for
> quick response times got swapped out and need to be swapped in again,
> resulting in outrageously poor server performance in spite of plenty
> of unused memory.  [...]
Before necessarily blaming swapping, have you tried substituting
SCHED_4BSD for SCHED_ULE in the kernel?  SCHED_ULE is not the world
champion scheduler for interactive tasks.

(I'm ducking back into my rabbit hole now and no one needs to throw
any more brickbats at me.  Sorry for the digression.)      -- George


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: 13-CURRENT: several GB swap being used despite plenty of free RAM

Stefan Blachmann
Probably a good suggestion to try the alternative scheduler, thank you.
However, as people a) always see disk activity LED flashing up when PC
is "lagging", which can be clearly identified as swap I/O using the
various utilities, and b) these "lag" issues do not appear with swap
deactivated, I think I have reason to believe the performance issues
are not scheduler-related.

On 11/18/18, George Mitchell <[hidden email]> wrote:

> On 11/18/18 7:11 AM, Stefan Blachmann wrote:
>> The inconveniences that the new swapping strategy causes are a regular
>> topic in the FreeBSD forums.
>>
>> Desktop users complain about lagginess, server users complain of long
>> delays because server processes intended to be kept in memory for
>> quick response times got swapped out and need to be swapped in again,
>> resulting in outrageously poor server performance in spite of plenty
>> of unused memory.  [...]
> Before necessarily blaming swapping, have you tried substituting
> SCHED_4BSD for SCHED_ULE in the kernel?  SCHED_ULE is not the world
> champion scheduler for interactive tasks.
>
> (I'm ducking back into my rabbit hole now and no one needs to throw
> any more brickbats at me.  Sorry for the digression.)      -- George
>
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 13-CURRENT: several GB swap being used despite plenty of free RAM

Wojciech Puchar-8
In reply to this post by Stefan Blachmann
why someone changes WELL WORKING swapping algorithm?

On Sun, 18 Nov 2018, Stefan Blachmann wrote:

> So I guess the unwillingness of the developer team to add an option
> like "NoPreemptiveSwapping", which disables swapping out as long as
> there is free physical memory available, is of psychological nature.

in FReeBSD 12 there is

vm.swap_idle_threshold2: 10
vm.swap_idle_threshold1: 2
vm.swap_idle_enabled: 0


which works as expected and can be turned on or off as required.

what's wrong with it
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 13-CURRENT: several GB swap being used despite plenty of free RAM

freebsd-hackers mailing list
In reply to this post by Stefan Blachmann
On 2018-Nov-18, at 04:11, Stefan Blachmann <[hidden email]> wrote:

> The inconveniences that the new swapping strategy causes are a regular
> topic in the FreeBSD forums.
>
> Desktop users complain about lagginess, server users complain of long
> delays because server processes intended to be kept in memory for
> quick response times got swapped out and need to be swapped in again,
> resulting in outrageously poor server performance in spite of plenty
> of unused memory.

I've no clue of the variations in workloads involved for
various folks, but I only see swapping/paging once free
memory is low in my UFS based contexts. (I've not used
ZFS in a long time.) I do not see free RAM decreasing
without known, expected use of the RAM. Mostly I've had
to learn how late it may be before pages for cached
information explicitly moves back to free, no matter if
that involves paging/swapping out or not.

As I understand what Mark Johnston reported, he said that there is
no preemptive paging/swapping:

"FreeBSD will only ever swap when there is a free page shortage"

Moving between Active and Inactive does not of itself involve
paging/swapping. So that part of his description is a different
issue.

So the issue may trace back to why there ends up being a prior
free RAM shortage and/or when pages are moved back to RAM after
such a shortage.

I believe my report above is consistent with what Mark
Johnston reported about the modern algorithm(s) involved.

> Turning off swap completely, as Cy Schubert suggests, is strongly
> discouraged in the forums, as it can lead to kernel panicking because
> of being unable to swap out in critical kernel memory shortage
> situations, leading to the risk of  very serious filesystem
> corruption.
>
> However, Cy Schubert is probably right when stating that the new
> swapping strategy resembles the 1960s-1980s industry's main swapping
> strategy.

Important parts of "new" way seems to have been in place since after
4.4BSD (so BSD 5.0 and later) . . .

The book "The Design and Implementation of the FreeBSD Operating System"
(2nd edition) states (page labeled 296):

QUOTE:
The FreeBSD swap-out daemon will not select a runnable processes to swap
out. So, if the set of runnable processes do not fit in memory, the
machine will effectively deadlock. Current machines have enough memory
that this condition usually does not arise. If it does, FreeBSD avoids
deadlock by killing the largest process. If the condition begins to arise
in normal operation, the 4.4BSD algorithm will need to be restored.
END QUOTE.

(I've not found vm.pageout_oom_seq in the book. It is for controlling
how much effort is put into freeing RAM before kills start, and so,
indirectly, the elapsed time that happens before those kills start.)

I'm not sure what specific changes that are newer might be under
discussion.

> The bad thing is now, that nowadays memory is no longer scarce and
> people can dimension their memory such that under normal circumstances
> there will never be any need to swap.

My typical use is apparently not normal. I'd agree with that.

> So I guess the unwillingness of the developer team to add an option
> like "NoPreemptiveSwapping", which disables swapping out as long as
> there is free physical memory available, is of psychological nature.

Unsure how to interpret this given:

"FreeBSD will only ever swap when there is a free page shortage"

> Lacking such an option, there is still the possibility to use rctl to
> disable swapping for particular users, processes, jails etc to
> mitigate the problems caused by the new swapping strategy to some
> degree.

Whatever is going on for this, it seems a more technical identification
of what it is would be needed in order to get to the right context.
(It may be non-trivial to identify.)

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 13-CURRENT: several GB swap being used despite plenty of free RAM

freebsd-hackers mailing list
In reply to this post by Wojciech Puchar-8
On Sun, 18 Nov 2018 17:53:52 +0100 (CET)
Wojciech Puchar wrote:

> why someone changes WELL WORKING swapping algorithm?
>
> On Sun, 18 Nov 2018, Stefan Blachmann wrote:
>
> > So I guess the unwillingness of the developer team to add an option
> > like "NoPreemptiveSwapping", which disables swapping out as long as
> > there is free physical memory available, is of psychological
> > nature.  
>
> in FReeBSD 12 there is
>
> vm.swap_idle_threshold2: 10
> vm.swap_idle_threshold1: 2
> vm.swap_idle_enabled: 0
>
>
> which works as expected and can be turned on or off as required.

The above settings are about process level swapping, i.e. the
deactivation of whole processes, they don't control swapping in the
sense of paging out to the swap device.

By default process swapping is only used under extreme memory shortage,
setting vm.swap_idle_enabled=1 allows idle processes to be deactivated.
It's only really intended for some special cases like login servers,
which tend to have a lot of inactive shell and editor processes.

Changing those setting wont reduce swap usage, they may increase
it.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 13-CURRENT: several GB swap being used despite plenty of free RAM

Cy Schubert-4
In reply to this post by freebsd-hackers mailing list
In message <[hidden email]
il.com>
, Stefan Blachmann writes:
> The inconveniences that the new swapping strategy causes are a regular
> topic in the FreeBSD forums.
>
> Desktop users complain about lagginess, server users complain of long
> delays because server processes intended to be kept in memory for
> quick response times got swapped out and need to be swapped in again,
> resulting in outrageously poor server performance in spite of plenty
> of unused memory.

There are many factors for this. One of the biggest mistakes people
make is use UFS and ZFS on the same system. Limiting ZFS ARC max would
be a good place to start.

Unfortunately like many operating systems these days, FreeBSD's tuning
parameters are generally baked in. And, fencing (an IBM term to limit
address space memory use or to establish a minimum) isn't available.
So, tuning for a site specific workload is more difficult.

Making sure there is plenty of RAM installed and tuning down ZFS ARC
can go a long way. Having said that, it's been a while since I've had
to do this. The updates made to ZFS ARC and NUMA have allowed me to
rely on the algorithms baked into FreeBSD. My 8 GB systems have
performed rather well.

>
> Turning off swap completely, as Cy Schubert suggests, is strongly
> discouraged in the forums, as it can lead to kernel panicking because
> of being unable to swap out in critical kernel memory shortage
> situations, leading to the risk of  very serious filesystem
> corruption.

Actually, I didn't suggest it. I simply pointed out the option.
Furthermore, it was a mistake for Linux to remove swapping entirely
from their kernel. For those who want to try it, however, the option is
there.

>
> However, Cy Schubert is probably right when stating that the new
> swapping strategy resembles the 1960s-1980s industry's main swapping
> strategy.
> The bad thing is now, that nowadays memory is no longer scarce and
> people can dimension their memory such that under normal circumstances
> there will never be any need to swap.

Adding more RAM has been the general thinking over the last 15-20 years.

>
> So I guess the unwillingness of the developer team to add an option
> like "NoPreemptiveSwapping", which disables swapping out as long as
> there is free physical memory available, is of psychological nature.

How do you know processes are indeed swapped out rather than (many)
pages paged out. You can't tell and using top's w mode, sorted by swap
doesn't tell us anything because IMO it's bogus, incorrect numbers.

>
> Lacking such an option, there is still the possibility to use rctl to
> disable swapping for particular users, processes, jails etc to
> mitigate the problems caused by the new swapping strategy to some
> degree.

Our options are limited. I'd prefer something similar to what we had on
MVS, when I worked on it, however even IBM has dumbed down SRM (System
Resource Manager) such that knobs that were once there no longer are.
Tuning based upon analysis of site specific stats are a thing of the
past. It's 2018 and sadly one size fits all.


--
Cheers,
Cy Schubert <[hidden email]>
FreeBSD UNIX:  <[hidden email]>   Web:  http://www.FreeBSD.org

        The need of the many outweighs the greed of the few.


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 13-CURRENT: several GB swap being used despite plenty of free RAM

freebsd-hackers mailing list
On Sunday, 18 November 2018 15:05:04 MST Cy Schubert wrote:

> Unfortunately like many operating systems these days, FreeBSD's tuning
> parameters are generally baked in. And, fencing (an IBM term to limit
> address space memory use or to establish a minimum) isn't available.
> So, tuning for a site specific workload is more difficult.

Talking of tuning, this seems like a good resource for tuning for desktop use:
https://cooltrainer.org/a-freebsd-desktop-howto/ .

--
Rebecca


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 13-CURRENT: several GB swap being used despite plenty of free RAM

Wojciech Puchar-8
In reply to this post by freebsd-hackers mailing list
>
> The above settings are about process level swapping, i.e. the
> deactivation of whole processes, they don't control swapping in the
> sense of paging out to the swap device.
>
> By default process swapping is only used under extreme memory shortage,
> setting vm.swap_idle_enabled=1 allows idle processes to be deactivated.
> It's only really intended for some special cases like login servers,
> which tend to have a lot of inactive shell and editor processes.
>

and my servers that have LOTS of httpd servers each for one webpage which
are usually rarely visited.

> Changing those setting wont reduce swap usage, they may increase
> it.
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "[hidden email]"
>
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 13-CURRENT: several GB swap being used despite plenty of free RAM

Wojciech Puchar-8
In reply to this post by Cy Schubert-4
> can go a long way. Having said that, it's been a while since I've had
> to do this. The updates made to ZFS ARC and NUMA have allowed me to
> rely on the algorithms baked into FreeBSD. My 8 GB systems have
> performed rather well.

8GB is LOT LOT LOT of memory.

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "[hidden email]"
12