leaked swap?

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

leaked swap?

Andriy Gapon

First, a note that this was observed on a system that runs a fairly old current
(~ 1 year old) with a fairly long uptime (> 6 months).
I noticed that the system was nearly out of memory, 98% of swap was in use,
there was less than 1 GB of free memory, several GBs of each of active, inactive
and laundry memory, and many GBs of wired (mostly ZFS).
I decided to pro-actively reboot the system, but to speed that up I put the
system to the single-user mode (via shutdown) and then back to multi-user. So,
there was no real hardware reboot and the kernel kept running.  However, all
userland processes were terminated.

To my surprise, even while in the single-user mode the swap utilization didn't
go below 70%.  Also, laundry memory remained in multi-GB area, but let's ignore
this for now.

I think that the swap could be used only for anonymous memory, so I expected it
go to zero after the shutdown to the single user mode.
Does anyone have any ideas?
Maybe that's something that has already been fixed?
If not, any ideas on what to look for?
Thanks!

--
Andriy Gapon
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: leaked swap?

Cy Schubert-4
On March 18, 2019 8:20:35 AM PDT, Andriy Gapon <[hidden email]> wrote:

>
>First, a note that this was observed on a system that runs a fairly old
>current
>(~ 1 year old) with a fairly long uptime (> 6 months).
>I noticed that the system was nearly out of memory, 98% of swap was in
>use,
>there was less than 1 GB of free memory, several GBs of each of active,
>inactive
>and laundry memory, and many GBs of wired (mostly ZFS).
>I decided to pro-actively reboot the system, but to speed that up I put
>the
>system to the single-user mode (via shutdown) and then back to
>multi-user. So,
>there was no real hardware reboot and the kernel kept running.
>However, all
>userland processes were terminated.
>
>To my surprise, even while in the single-user mode the swap utilization
>didn't
>go below 70%.  Also, laundry memory remained in multi-GB area, but
>let's ignore
>this for now.
>
>I think that the swap could be used only for anonymous memory, so I
>expected it
>go to zero after the shutdown to the single user mode.
>Does anyone have any ideas?
>Maybe that's something that has already been fixed?
>If not, any ideas on what to look for?
>Thanks!

I've had a hunch of this but haven't gone down this rabbit hole to investigate. Related, yesterday I performed a git gc --aggressive. Top did not report any swap used by git and GB of swap were used. I think to help address this we need a reliable reporting tool. Obviously two separate symptoms, not sure if the same cause.




--
Pardon the typos and autocorrect, small keyboard in use.
Cheers,
Cy Schubert <[hidden email]>
FreeBSD UNIX: <[hidden email]> Web: http://www.FreeBSD.org

        The need of the many outweighs the greed of the few.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: leaked swap?

Konstantin Belousov
In reply to this post by Andriy Gapon
On Mon, Mar 18, 2019 at 05:20:35PM +0200, Andriy Gapon wrote:

>
> First, a note that this was observed on a system that runs a fairly old current
> (~ 1 year old) with a fairly long uptime (> 6 months).
> I noticed that the system was nearly out of memory, 98% of swap was in use,
> there was less than 1 GB of free memory, several GBs of each of active, inactive
> and laundry memory, and many GBs of wired (mostly ZFS).
> I decided to pro-actively reboot the system, but to speed that up I put the
> system to the single-user mode (via shutdown) and then back to multi-user. So,
> there was no real hardware reboot and the kernel kept running.  However, all
> userland processes were terminated.
>
> To my surprise, even while in the single-user mode the swap utilization didn't
> go below 70%.  Also, laundry memory remained in multi-GB area, but let's ignore
> this for now.
>
> I think that the swap could be used only for anonymous memory, so I expected it
> go to zero after the shutdown to the single user mode.
> Does anyone have any ideas?
> Maybe that's something that has already been fixed?
> If not, any ideas on what to look for?
tmpfs, swap-backed (or even memory backed) md, persistent posix shared
memory, SysV shared memory.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: leaked swap?

Andriy Gapon
In reply to this post by Cy Schubert-4
On 18/03/2019 17:30, Cy Schubert wrote:
> Top did not report any swap used by git and GB of swap were used.

Last time I checked top reported something very different as swap.
IIRC, its notion of swap usage comes from the age when the swap granularity was
a whole process.

--
Andriy Gapon
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: leaked swap?

Andriy Gapon
In reply to this post by Konstantin Belousov
On 18/03/2019 17:32, Konstantin Belousov wrote:

> On Mon, Mar 18, 2019 at 05:20:35PM +0200, Andriy Gapon wrote:
>>
>> First, a note that this was observed on a system that runs a fairly old current
>> (~ 1 year old) with a fairly long uptime (> 6 months).
>> I noticed that the system was nearly out of memory, 98% of swap was in use,
>> there was less than 1 GB of free memory, several GBs of each of active, inactive
>> and laundry memory, and many GBs of wired (mostly ZFS).
>> I decided to pro-actively reboot the system, but to speed that up I put the
>> system to the single-user mode (via shutdown) and then back to multi-user. So,
>> there was no real hardware reboot and the kernel kept running.  However, all
>> userland processes were terminated.
>>
>> To my surprise, even while in the single-user mode the swap utilization didn't
>> go below 70%.  Also, laundry memory remained in multi-GB area, but let's ignore
>> this for now.
>>
>> I think that the swap could be used only for anonymous memory, so I expected it
>> go to zero after the shutdown to the single user mode.
>> Does anyone have any ideas?
>> Maybe that's something that has already been fixed?
>> If not, any ideas on what to look for?
> tmpfs, swap-backed (or even memory backed) md, persistent posix shared
> memory, SysV shared memory.
>

Thank you.
There is a single tmpfs mount:
$ df -t tmpfs -h
Filesystem    Size    Used   Avail Capacity  Mounted on
tmpfs         1.0G    4.0K    1.0G     0%    /tmp/tmp

No md devices at all according to mdconfig.

Not sure how to check for the shared memory though.

--
Andriy Gapon
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: leaked swap?

Alan Somers-2
On Mon, Mar 18, 2019 at 9:38 AM Andriy Gapon <[hidden email]> wrote:

>
> On 18/03/2019 17:32, Konstantin Belousov wrote:
> > On Mon, Mar 18, 2019 at 05:20:35PM +0200, Andriy Gapon wrote:
> >>
> >> First, a note that this was observed on a system that runs a fairly old current
> >> (~ 1 year old) with a fairly long uptime (> 6 months).
> >> I noticed that the system was nearly out of memory, 98% of swap was in use,
> >> there was less than 1 GB of free memory, several GBs of each of active, inactive
> >> and laundry memory, and many GBs of wired (mostly ZFS).
> >> I decided to pro-actively reboot the system, but to speed that up I put the
> >> system to the single-user mode (via shutdown) and then back to multi-user. So,
> >> there was no real hardware reboot and the kernel kept running.  However, all
> >> userland processes were terminated.
> >>
> >> To my surprise, even while in the single-user mode the swap utilization didn't
> >> go below 70%.  Also, laundry memory remained in multi-GB area, but let's ignore
> >> this for now.
> >>
> >> I think that the swap could be used only for anonymous memory, so I expected it
> >> go to zero after the shutdown to the single user mode.
> >> Does anyone have any ideas?
> >> Maybe that's something that has already been fixed?
> >> If not, any ideas on what to look for?
> > tmpfs, swap-backed (or even memory backed) md, persistent posix shared
> > memory, SysV shared memory.
> >
>
> Thank you.
> There is a single tmpfs mount:
> $ df -t tmpfs -h
> Filesystem    Size    Used   Avail Capacity  Mounted on
> tmpfs         1.0G    4.0K    1.0G     0%    /tmp/tmp
>
> No md devices at all according to mdconfig.
>
> Not sure how to check for the shared memory though.

Try "ipcs -a"
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: leaked swap?

Andriy Gapon
On 18/03/2019 17:55, Alan Somers wrote:
> Try "ipcs -a"

Thank you. I will do it while in the single-user again. Right now it's too long
a list.

--
Andriy Gapon
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: leaked swap?

Cy Schubert-4
On March 18, 2019 8:57:03 AM PDT, Andriy Gapon <[hidden email]> wrote:
>On 18/03/2019 17:55, Alan Somers wrote:
>> Try "ipcs -a"
>
>Thank you. I will do it while in the single-user again. Right now it's
>too long
>a list.

Shared memory segments are not necessarily deleted  by applications.
--
Pardon the typos and autocorrect, small keyboard in use.
Cheers,
Cy Schubert <[hidden email]>
FreeBSD UNIX: <[hidden email]> Web: http://www.FreeBSD.org

        The need of the many outweighs the greed of the few.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: leaked swap?

Andrey Fesenko
In reply to this post by Andriy Gapon
On Mon, Mar 18, 2019 at 6:23 PM Andriy Gapon <[hidden email]> wrote:

>
>
> First, a note that this was observed on a system that runs a fairly old current
> (~ 1 year old) with a fairly long uptime (> 6 months).
> I noticed that the system was nearly out of memory, 98% of swap was in use,
> there was less than 1 GB of free memory, several GBs of each of active, inactive
> and laundry memory, and many GBs of wired (mostly ZFS).
> I decided to pro-actively reboot the system, but to speed that up I put the
> system to the single-user mode (via shutdown) and then back to multi-user. So,
> there was no real hardware reboot and the kernel kept running.  However, all
> userland processes were terminated.
>
> To my surprise, even while in the single-user mode the swap utilization didn't
> go below 70%.  Also, laundry memory remained in multi-GB area, but let's ignore
> this for now.
>
> I think that the swap could be used only for anonymous memory, so I expected it
> go to zero after the shutdown to the single user mode.
> Does anyone have any ideas?
> Maybe that's something that has already been fixed?
> If not, any ideas on what to look for?
> Thanks!
>

Not this? ZFS use wired and not clean only reboot?
https://reviews.freebsd.org/D7538?id=25108
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: leaked swap?

Andriy Gapon
On 18/03/2019 20:01, Andrey Fesenko wrote:
> Not this? ZFS use wired and not clean only reboot?
> https://reviews.freebsd.org/D7538?id=25108

Wired memory surely has nothing to do with swap.

--
Andriy Gapon
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: leaked swap?

Cy Schubert-4
In reply to this post by Andriy Gapon
In message <[hidden email]>, Andriy Gapon writes:
> On 18/03/2019 17:30, Cy Schubert wrote:
> > Top did not report any swap used by git and GB of swap were used.
>
> Last time I checked top reported something very different as swap.
> IIRC, its notion of swap usage comes from the age when the swap granularity was
> a whole process.

OK, in the strictest sense. The BSD definition of swap is the same as IBM's z/OS,
a whole address space.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: leaked swap?

Slawa Olhovchenkov
In reply to this post by Andriy Gapon
On Mon, Mar 18, 2019 at 11:00:10PM +0200, Andriy Gapon wrote:

> On 18/03/2019 20:01, Andrey Fesenko wrote:
> > Not this? ZFS use wired and not clean only reboot?
> > https://reviews.freebsd.org/D7538?id=25108
>
> Wired memory surely has nothing to do with swap.

Wired memory can pressure to swapable memory
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: leaked swap?

Trond Endrestøl
On Tue, 19 Mar 2019 16:02+0300, Slawa Olhovchenkov wrote:

> On Mon, Mar 18, 2019 at 11:00:10PM +0200, Andriy Gapon wrote:
>
> > On 18/03/2019 20:01, Andrey Fesenko wrote:
> > > Not this? ZFS use wired and not clean only reboot?
> > > https://reviews.freebsd.org/D7538?id=25108
> >
> > Wired memory surely has nothing to do with swap.
>
> Wired memory can pressure to swapable memory

I've noticed setting vm.pageout_update_period=0 in /etc/sysctl.conf
eliminated excessive swapping, effectively disabling r334154.

Maybe this is irrelevant. I waited a while before mentioning this.
Response times of idle services skyrocketed in my case.

--
Trond.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: leaked swap?

Andriy Gapon
In reply to this post by Slawa Olhovchenkov
On 19/03/2019 15:02, Slawa Olhovchenkov wrote:

> On Mon, Mar 18, 2019 at 11:00:10PM +0200, Andriy Gapon wrote:
>
>> On 18/03/2019 20:01, Andrey Fesenko wrote:
>>> Not this? ZFS use wired and not clean only reboot?
>>> https://reviews.freebsd.org/D7538?id=25108
>>
>> Wired memory surely has nothing to do with swap.
>
> Wired memory can pressure to swapable memory
>

Yes, it can.  But I am interested in what is in the swap.  Not what caused it to
go to the swap.

--
Andriy Gapon
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: leaked swap?

Tino Engel-5
Why the hell are you running current when you want a stable version? A year
old current.
I'd go to one of the stables at least.

Am 19. März 2019 15:11:16 schrieb Andriy Gapon <[hidden email]>:

> On 19/03/2019 15:02, Slawa Olhovchenkov wrote:
>> On Mon, Mar 18, 2019 at 11:00:10PM +0200, Andriy Gapon wrote:
>>
>>> On 18/03/2019 20:01, Andrey Fesenko wrote:
>>>> Not this? ZFS use wired and not clean only reboot?
>>>> https://reviews.freebsd.org/D7538?id=25108
>>>
>>> Wired memory surely has nothing to do with swap.
>>
>> Wired memory can pressure to swapable memory
>>
>
> Yes, it can.  But I am interested in what is in the swap.  Not what caused
> it to
> go to the swap.
>
> --
> Andriy Gapon
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "[hidden email]"



_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: leaked swap?

Slawa Olhovchenkov
In reply to this post by Andriy Gapon
On Tue, Mar 19, 2019 at 04:07:45PM +0200, Andriy Gapon wrote:

> On 19/03/2019 15:02, Slawa Olhovchenkov wrote:
> > On Mon, Mar 18, 2019 at 11:00:10PM +0200, Andriy Gapon wrote:
> >
> >> On 18/03/2019 20:01, Andrey Fesenko wrote:
> >>> Not this? ZFS use wired and not clean only reboot?
> >>> https://reviews.freebsd.org/D7538?id=25108
> >>
> >> Wired memory surely has nothing to do with swap.
> >
> > Wired memory can pressure to swapable memory
> >
>
> Yes, it can.  But I am interested in what is in the swap.  Not what caused it to
> go to the swap.

procstat -v -a | awk '$10 == "sw"' ?
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[hidden email]"