11.2-STABLE kernel wired memory leak

classic Classic list List threaded Threaded
36 messages Options
12
Reply | Threaded
Open this post in threaded view
|

11.2-STABLE kernel wired memory leak

Eugene Grosbein-10
Hi!

Long story short: 11.2-STABLE/amd64 r335757 leaked over 4600MB kernel wired memory over 81 days uptime
out of 8GB total RAM.

Details follow.

I have a workstation running Xorg, Firefox, Thunderbird, LibreOffice and occasionally VirtualBox for single VM.

It has two identical 320GB HDDs combined with single graid-based array with "Intel"
on-disk format having 3 volumes:
- one "RAID1" volume /dev/raid/r0 occupies first 10GB or each HDD;
- two "SINGLE" volumes /dev/raid/r1 and /dev/raid/r2 that utilize "tails" of HDDs (310GB each).

/dev/raid/r0 (10GB) has MBR partitioning and two slices:
- /dev/raid/r0s1 (8GB) is used for swap;
- /dev/raid/r0s2 (2GB) is used by non-redundant ZFS pool named "os" that contains only
root file system (177M used) and /usr file system (340M used).

There is also second pool (ZMIRROR) named "z" built directly on top of /dev/raid/r[12] volumes,
this pool contains all other file systems including /var, /home, /usr/ports, /usr/local, /usr/{src|obj} etc.

# zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
os    1,98G   520M  1,48G        -         -    55%    25%  1.00x  ONLINE  -
z      288G  79,5G   209G        -         -    34%    27%  1.00x  ONLINE  -

This way I have swap outside of ZFS, boot blocks and partitioning mirrored by means of GEOM_RAID and
can use local console to break to single user mode to unmount all file system other than root and /usr
and can even export bigger ZFS pool "z". And I did that to see that ARC usage
(limited with vfs.zfs.arc_max="3G" in /boot/loader.conf) dropped from over 2500MB
down to 44MB but "Wired" stays high. Now after I imported "z" back and booted to multiuser mode
top(1) shows:

last pid: 51242;  load averages:  0.24,  0.16,  0.13  up 81+02:38:38  22:59:18
104 processes: 1 running, 103 sleeping
CPU:  0.0% user,  0.0% nice,  0.4% system,  0.2% interrupt, 99.4% idle
Mem: 84M Active, 550M Inact, 4K Laundry, 4689M Wired, 2595M Free
ARC: 273M Total, 86M MFU, 172M MRU, 64K Anon, 1817K Header, 12M Other
     117M Compressed, 333M Uncompressed, 2.83:1 Ratio
Swap: 8192M Total, 940K Used, 8191M Free

I have KDB and DDB in my custom kernel also. How do I debug the leak further?

I use nvidia-driver-340-340.107 driver for GK208 [GeForce GT 710B] video card.
Here are outputs of "vmstat -m": http://www.grosbein.net/freebsd/leak/vmstat-m.txt
and "vmstat -z": http://www.grosbein.net/freebsd/leak/vmstat-z.txt
as well as "sysctl hw": http://www.grosbein.net/freebsd/leak/sysctl-hw.txt
and "sysctl vm": http://www.grosbein.net/freebsd/leak/sysctl-vm.txt
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Mark Johnston-2
On Tue, Feb 12, 2019 at 11:14:31PM +0700, Eugene Grosbein wrote:

> Hi!
>
> Long story short: 11.2-STABLE/amd64 r335757 leaked over 4600MB kernel wired memory over 81 days uptime
> out of 8GB total RAM.
>
> Details follow.
>
> I have a workstation running Xorg, Firefox, Thunderbird, LibreOffice and occasionally VirtualBox for single VM.
>
> It has two identical 320GB HDDs combined with single graid-based array with "Intel"
> on-disk format having 3 volumes:
> - one "RAID1" volume /dev/raid/r0 occupies first 10GB or each HDD;
> - two "SINGLE" volumes /dev/raid/r1 and /dev/raid/r2 that utilize "tails" of HDDs (310GB each).
>
> /dev/raid/r0 (10GB) has MBR partitioning and two slices:
> - /dev/raid/r0s1 (8GB) is used for swap;
> - /dev/raid/r0s2 (2GB) is used by non-redundant ZFS pool named "os" that contains only
> root file system (177M used) and /usr file system (340M used).
>
> There is also second pool (ZMIRROR) named "z" built directly on top of /dev/raid/r[12] volumes,
> this pool contains all other file systems including /var, /home, /usr/ports, /usr/local, /usr/{src|obj} etc.
>
> # zpool list
> NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
> os    1,98G   520M  1,48G        -         -    55%    25%  1.00x  ONLINE  -
> z      288G  79,5G   209G        -         -    34%    27%  1.00x  ONLINE  -
>
> This way I have swap outside of ZFS, boot blocks and partitioning mirrored by means of GEOM_RAID and
> can use local console to break to single user mode to unmount all file system other than root and /usr
> and can even export bigger ZFS pool "z". And I did that to see that ARC usage
> (limited with vfs.zfs.arc_max="3G" in /boot/loader.conf) dropped from over 2500MB
> down to 44MB but "Wired" stays high. Now after I imported "z" back and booted to multiuser mode
> top(1) shows:
>
> last pid: 51242;  load averages:  0.24,  0.16,  0.13  up 81+02:38:38  22:59:18
> 104 processes: 1 running, 103 sleeping
> CPU:  0.0% user,  0.0% nice,  0.4% system,  0.2% interrupt, 99.4% idle
> Mem: 84M Active, 550M Inact, 4K Laundry, 4689M Wired, 2595M Free
> ARC: 273M Total, 86M MFU, 172M MRU, 64K Anon, 1817K Header, 12M Other
>      117M Compressed, 333M Uncompressed, 2.83:1 Ratio
> Swap: 8192M Total, 940K Used, 8191M Free
>
> I have KDB and DDB in my custom kernel also. How do I debug the leak further?
>
> I use nvidia-driver-340-340.107 driver for GK208 [GeForce GT 710B] video card.
> Here are outputs of "vmstat -m": http://www.grosbein.net/freebsd/leak/vmstat-m.txt
> and "vmstat -z": http://www.grosbein.net/freebsd/leak/vmstat-z.txt

I suspect that the "leaked" memory is simply being used to cache UMA
items.  Note that the values in the FREE column of vmstat -z output are
quite large.  The cached items are reclaimed only when the page daemon
wakes up to reclaim memory; if there are no memory shortages, large
amounts of memory may accumulate in UMA caches.  In this case, the sum
of the product of columns 2 and 5 gives a total of roughly 4GB cached.

> as well as "sysctl hw": http://www.grosbein.net/freebsd/leak/sysctl-hw.txt
> and "sysctl vm": http://www.grosbein.net/freebsd/leak/sysctl-vm.txt
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Eugene Grosbein-10
12.02.2019 23:34, Mark Johnston wrote:

> I suspect that the "leaked" memory is simply being used to cache UMA
> items.  Note that the values in the FREE column of vmstat -z output are
> quite large.  The cached items are reclaimed only when the page daemon
> wakes up to reclaim memory; if there are no memory shortages, large
> amounts of memory may accumulate in UMA caches.  In this case, the sum
> of the product of columns 2 and 5 gives a total of roughly 4GB cached.

Forgot to note, that before I got system to single user mode, there was heavy swap usage (over 3.5GB)
and heavy page-in/page-out, 10-20 megabytes per second and system was crawling slow due to pageing.

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Eugene Grosbein-10
12.02.2019 23:49, Eugene Grosbein wrote:

> 12.02.2019 23:34, Mark Johnston wrote:
>
>> I suspect that the "leaked" memory is simply being used to cache UMA
>> items.  Note that the values in the FREE column of vmstat -z output are
>> quite large.  The cached items are reclaimed only when the page daemon
>> wakes up to reclaim memory; if there are no memory shortages, large
>> amounts of memory may accumulate in UMA caches.  In this case, the sum
>> of the product of columns 2 and 5 gives a total of roughly 4GB cached.
>
> Forgot to note, that before I got system to single user mode, there was heavy swap usage (over 3.5GB)
> and heavy page-in/page-out, 10-20 megabytes per second and system was crawling slow due to pageing.

There was significant memory shortage due to Firefox having over 5GB RSS
plus other processes like Xorg, parts of xfce4 etc. Still, over 4GB of Wired memory.


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Karl Denninger
In reply to this post by Eugene Grosbein-10

On 2/12/2019 10:49, Eugene Grosbein wrote:

> 12.02.2019 23:34, Mark Johnston wrote:
>
>> I suspect that the "leaked" memory is simply being used to cache UMA
>> items.  Note that the values in the FREE column of vmstat -z output are
>> quite large.  The cached items are reclaimed only when the page daemon
>> wakes up to reclaim memory; if there are no memory shortages, large
>> amounts of memory may accumulate in UMA caches.  In this case, the sum
>> of the product of columns 2 and 5 gives a total of roughly 4GB cached.
> Forgot to note, that before I got system to single user mode, there was heavy swap usage (over 3.5GB)
> and heavy page-in/page-out, 10-20 megabytes per second and system was crawling slow due to pageing.
This is a manifestation of the general issue I've had an ongoing
"discussion" running in a long-running thread on bugzilla and the
interaction between UMA, ARC and the VM system.

The short version is that the VM system does pathological things
including paging out working set when there is a large amount of
allocated-but-unused UMA and the means by which the ARC code is "told"
that it needs to release RAM also interacts with the same mechanisms and
exacerbates the problem.

I've basically given up on getting anything effective to deal with this
merged into the code and have my own private set of patches that I
published for a while in that thread (and which had some collaborative
development and testing) but have given up on seeing anything meaningful
put into the base code.  To the extent I need them in a given workload
and environment I simply apply them on my own and go on my way.

I don't have enough experience with 12 yet to know if the same approach
will be necessary there (that is, what if any changes got into the 12.x
code), and never ran 11.2 much, choosing to stay on 11.1 where said
patches may not have been the most-elegant means of dealing with it but
were successful.  There was also a phabricator thread on this but I
don't know what part of it, if any, got merged (it was more-elegant, in
my opinion, than what I had coded up.)  Under certain workloads here
without the patches I was experiencing "freezes" due to unnecessary
page-outs onto spinning rust that in some cases reached into
double-digit *seconds.*  With them the issue was entirely resolved.

At the core of the issue is that "something" has to be taught that
*before* the pager starts evicting working set to swap if you have large
amounts of UMA allocated to ARC but not in use that RAM should be
released, and beyond that if you have ARC allocated and in use but are
approaching where VM is going to page working set out you need to come
up with some meaningful way of deciding whether to release some of the
ARC rather than take the page hit -- and in virtually every case the
answer to that question is to release the RAM consumed by ARC.  Part of
the issue is that UMA can be allocated for other things besides ARC yet
you really only want to release the ARC-related UMA that is
allocated-but-unused in this instance.

The logic is IMHO pretty simple on this -- a page-out of a process that
will run again always requires TWO disk operations -- one to page it out
right now and a second at a later time to page it back in.  A released
ARC cache *MAY* (if there would have been a cache hit in the future)
require ONE disk operation (to retrieve it from disk.)

Two is always greater than one and one is never worse than "maybe one
later" therefore prioritizing taking two *definite* disk I/Os or one
definite I/O now and one possible one later instead of one *possible*
disk I/O later is always a net lose -- and thus IMHO substantial effort
should be made to avoid doing that.

--
Karl Denninger
[hidden email] <mailto:[hidden email]>
/The Market Ticker/
/[S/MIME encrypted email preferred]/

smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Garrett Wollman-4
In reply to this post by Eugene Grosbein-10
In article <[hidden email]>
[hidden email] writes:

>Long story short: 11.2-STABLE/amd64 r335757 leaked over 4600MB kernel
>wired memory over 81 days uptime
>out of 8GB total RAM.

Not a whole lot of evidence yet, but anecdotally I'm seeing the same
thing on some huge-memory NFS servers running releng/11.2.  They seem
to run fine for a few weeks, then mysteriously start swapping
continuously, a few hundred pages a second.  The continues for hours
at a time, and then stops just as mysteriously.  Over time the total
memory dedicated to ZFS ARC goes down but there's no decrease in wired
memory.  I've tried disabling swap, but this seems to make the server
unstable.  I have yet to find any obvious commonality (aside from the
fact that these are all large-memory NFS servers which don't do much
of anything else -- the only software running on them is related to
managing and monitoring the NFS service).

-GAWollman

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Eugene Grosbein-10
In reply to this post by Mark Johnston-2
12.02.2019 23:34, Mark Johnston wrote:

> I suspect that the "leaked" memory is simply being used to cache UMA
> items.  Note that the values in the FREE column of vmstat -z output are
> quite large.  The cached items are reclaimed only when the page daemon
> wakes up to reclaim memory; if there are no memory shortages, large
> amounts of memory may accumulate in UMA caches.  In this case, the sum
> of the product of columns 2 and 5 gives a total of roughly 4GB cached.
>
>> as well as "sysctl hw": http://www.grosbein.net/freebsd/leak/sysctl-hw.txt
>> and "sysctl vm": http://www.grosbein.net/freebsd/leak/sysctl-vm.txt

It seems page daemon is broken somehow as it did not reclaim several gigs of wired memory
despite of long period of vm thrashing:

$ sed 's/:/,/' vmstat-z.txt | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, $1}' | sort -k1,1 -rn | head
      1892 abd_chunk
   454.629 dnode_t
    351.35 zio_buf_512
   228.391 zio_buf_16384
   173.968 dmu_buf_impl_t
    130.25 zio_data_buf_131072
   93.6887 VNODE
   81.6978 arc_buf_hdr_t_full
   74.9368 256
   57.4102 4096

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Eugene Grosbein-10
In reply to this post by Garrett Wollman-4
13.02.2019 0:57, Garrett Wollman wrote:

> In article <[hidden email]>
> [hidden email] writes:
>
>> Long story short: 11.2-STABLE/amd64 r335757 leaked over 4600MB kernel
>> wired memory over 81 days uptime
>> out of 8GB total RAM.
>
> Not a whole lot of evidence yet, but anecdotally I'm seeing the same
> thing on some huge-memory NFS servers running releng/11.2.  They seem
> to run fine for a few weeks, then mysteriously start swapping
> continuously, a few hundred pages a second.  The continues for hours
> at a time, and then stops just as mysteriously.  Over time the total
> memory dedicated to ZFS ARC goes down but there's no decrease in wired
> memory.  I've tried disabling swap, but this seems to make the server
> unstable.  I have yet to find any obvious commonality (aside from the
> fact that these are all large-memory NFS servers which don't do much
> of anything else -- the only software running on them is related to
> managing and monitoring the NFS service).

I started to understand the issue. FreeBSD 11 has uma(9) zone allocator
for kernel subsystems and vmstat -z shows some stats for UMA zones.

When some subsystem using UMA frees its memory (including networking mbufs or ZFS ARC),
some kernel memory blocks are moved from USED to FREE category
inside corresponding UMA zone (see vmstat -z again) but this memory stays
unavailable to other consumers including userland applications
until pagedaemon reclaims this "FREE" memory back to global free pool.
This part seems to be broken in 11.2-STABLE.

Use following command to see how much memory is wasted in your case:

vmstat -z | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, $1}' | sort -k1,1 -rn | head

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Eugene Grosbein-10
13.02.2019 1:14, Eugene Grosbein wrote:

> Use following command to see how much memory is wasted in your case:
>
> vmstat -z | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, $1}' | sort -k1,1 -rn | head

Oops, small correction:

vmstat -z | sed 's/:/,/' | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, $1}' | sort -k1,1 -rn | head

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Mark Johnston-2
In reply to this post by Eugene Grosbein-10
On Wed, Feb 13, 2019 at 01:03:37AM +0700, Eugene Grosbein wrote:

> 12.02.2019 23:34, Mark Johnston wrote:
>
> > I suspect that the "leaked" memory is simply being used to cache UMA
> > items.  Note that the values in the FREE column of vmstat -z output are
> > quite large.  The cached items are reclaimed only when the page daemon
> > wakes up to reclaim memory; if there are no memory shortages, large
> > amounts of memory may accumulate in UMA caches.  In this case, the sum
> > of the product of columns 2 and 5 gives a total of roughly 4GB cached.
> >
> >> as well as "sysctl hw": http://www.grosbein.net/freebsd/leak/sysctl-hw.txt
> >> and "sysctl vm": http://www.grosbein.net/freebsd/leak/sysctl-vm.txt
>
> It seems page daemon is broken somehow as it did not reclaim several gigs of wired memory
> despite of long period of vm thrashing:

Depending on the system's workload, it is possible for the caches to
grow quite quickly after a reclaim.  If you are able to run kgdb on the
kernel, you can find the time of the last reclaim by comparing the
values of lowmem_uptime and time_uptime.

> $ sed 's/:/,/' vmstat-z.txt | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, $1}' | sort -k1,1 -rn | head
>       1892 abd_chunk
>    454.629 dnode_t
>     351.35 zio_buf_512
>    228.391 zio_buf_16384
>    173.968 dmu_buf_impl_t
>     130.25 zio_data_buf_131072
>    93.6887 VNODE
>    81.6978 arc_buf_hdr_t_full
>    74.9368 256
>    57.4102 4096
>
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Kurt Jaeger-8
In reply to this post by Eugene Grosbein-10
Hi!

> > Use following command to see how much memory is wasted in your case:
> >
> > vmstat -z | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, $1}' | sort -k1,1 -rn | head
>
> Oops, small correction:
>
> vmstat -z | sed 's/:/,/' | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, $1}' | sort -k1,1 -rn | head

On a 8 GB 11.2p8 box doing mostly routing:

  42.3047 abd_chunk
        40 zio_buf_131072
     31.75 zio_data_buf_131072
   19.8901 swblk
   12.9224 RADIX NODE
   11.7344 zio_buf_16384
   10.0664 zio_data_buf_12288
   9.84375 zio_data_buf_40960
     9.375 zio_data_buf_81920
   7.96875 zio_data_buf_98304

So, how do I understand this ? Please note that I use:

vfs.zfs.arc_max=1216348160

--
[hidden email]            +49 171 3101372                    One year to go !
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Eugene Grosbein-10
13.02.2019 1:29, Kurt Jaeger wrote:

> On a 8 GB 11.2p8 box doing mostly routing:
>
>   42.3047 abd_chunk
>         40 zio_buf_131072
>      31.75 zio_data_buf_131072
>    19.8901 swblk
>    12.9224 RADIX NODE
>    11.7344 zio_buf_16384
>    10.0664 zio_data_buf_12288
>    9.84375 zio_data_buf_40960
>      9.375 zio_data_buf_81920
>    7.96875 zio_data_buf_98304
>
> So, how do I understand this ? Please note that I use:
>
> vfs.zfs.arc_max=1216348160

Each line shows how many megabytes is allocated but currently unused by corresponding
UMA zone (and unavailable for other consumers). Your numbers are pretty low,
you have nothing to worry about IMHO. I have hundreds of megabytes and gigabytes there.




_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Eugene Grosbein-10
In reply to this post by Mark Johnston-2
13.02.2019 1:18, Mark Johnston wrote:

> On Wed, Feb 13, 2019 at 01:03:37AM +0700, Eugene Grosbein wrote:
>> 12.02.2019 23:34, Mark Johnston wrote:
>>
>>> I suspect that the "leaked" memory is simply being used to cache UMA
>>> items.  Note that the values in the FREE column of vmstat -z output are
>>> quite large.  The cached items are reclaimed only when the page daemon
>>> wakes up to reclaim memory; if there are no memory shortages, large
>>> amounts of memory may accumulate in UMA caches.  In this case, the sum
>>> of the product of columns 2 and 5 gives a total of roughly 4GB cached.
>>>
>>>> as well as "sysctl hw": http://www.grosbein.net/freebsd/leak/sysctl-hw.txt
>>>> and "sysctl vm": http://www.grosbein.net/freebsd/leak/sysctl-vm.txt
>>
>> It seems page daemon is broken somehow as it did not reclaim several gigs of wired memory
>> despite of long period of vm thrashing:
>
> Depending on the system's workload, it is possible for the caches to
> grow quite quickly after a reclaim.  If you are able to run kgdb on the
> kernel, you can find the time of the last reclaim by comparing the
> values of lowmem_uptime and time_uptime.

Yes, I have debugger compiled into running kernel and have console access.
What commands should I use?


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Mark Johnston-2
On Wed, Feb 13, 2019 at 01:40:06AM +0700, Eugene Grosbein wrote:

> 13.02.2019 1:18, Mark Johnston wrote:
>
> > On Wed, Feb 13, 2019 at 01:03:37AM +0700, Eugene Grosbein wrote:
> >> 12.02.2019 23:34, Mark Johnston wrote:
> >>
> >>> I suspect that the "leaked" memory is simply being used to cache UMA
> >>> items.  Note that the values in the FREE column of vmstat -z output are
> >>> quite large.  The cached items are reclaimed only when the page daemon
> >>> wakes up to reclaim memory; if there are no memory shortages, large
> >>> amounts of memory may accumulate in UMA caches.  In this case, the sum
> >>> of the product of columns 2 and 5 gives a total of roughly 4GB cached.
> >>>
> >>>> as well as "sysctl hw": http://www.grosbein.net/freebsd/leak/sysctl-hw.txt
> >>>> and "sysctl vm": http://www.grosbein.net/freebsd/leak/sysctl-vm.txt
> >>
> >> It seems page daemon is broken somehow as it did not reclaim several gigs of wired memory
> >> despite of long period of vm thrashing:
> >
> > Depending on the system's workload, it is possible for the caches to
> > grow quite quickly after a reclaim.  If you are able to run kgdb on the
> > kernel, you can find the time of the last reclaim by comparing the
> > values of lowmem_uptime and time_uptime.
>
> Yes, I have debugger compiled into running kernel and have console access.
> What commands should I use?

I meant kgdb(1).  If you can run that, try:

(kgdb) p time_uptime
(kgdb) p lowmem_uptime

If you are willing to drop the system into DDB, do so and run

db> x time_uptime
db> x lowmem_uptime
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Eugene Grosbein-10
13.02.2019 1:42, Mark Johnston wrote:

>> Yes, I have debugger compiled into running kernel and have console access.
>> What commands should I use?
>
> I meant kgdb(1).  If you can run that, try:
>
> (kgdb) p time_uptime
> (kgdb) p lowmem_uptime
>
> If you are willing to drop the system into DDB, do so and run
>
> db> x time_uptime
> db> x lowmem_uptime

I will reach the console next day only. Is it wise to use kgdb over ssh for running remote system? :-)



_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Lev Serebryakov
In reply to this post by Eugene Grosbein-10
On 12.02.2019 21:37, Eugene Grosbein wrote:

>> vfs.zfs.arc_max=1216348160
>
> Each line shows how many megabytes is allocated but currently unused by corresponding
> UMA zone (and unavailable for other consumers). Your numbers are pretty low,
> you have nothing to worry about IMHO. I have hundreds of megabytes and gigabytes there.
 I'm have same problem.

 According to top(1) I have 29G Wired, but only 17G Total ARC (12G
difference! System has 32G of RAM), and this statistic shows:

    5487.5 zio_data_buf_524288
   920.125 zio_data_buf_131072
       626 zio_buf_131072
       468 zio_data_buf_1048576
   398.391 zio_buf_16384
   305.464 dnode_t
   227.989 zio_buf_512
     171.5 zio_data_buf_458752
    141.75 zio_data_buf_393216
   116.456 dmu_buf_impl_t

 So, more than 6G (!) is not used in ARC, but hold by ZFS anyway.

--
// Lev Serebryakov


signature.asc (981 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Lev Serebryakov
In reply to this post by Eugene Grosbein-10
On 12.02.2019 21:48, Eugene Grosbein wrote:

> I will reach the console next day only. Is it wise to use kgdb over ssh for running remote system? :-)
 It works for me :-)

BTW, my is:

(kgdb) p time_uptime
$1 = 81369
(kgdb) p lowmem_uptime
$2 = 0

 (yes, this system have been rebooted less than a day ago).

--
// Lev Serebryakov


signature.asc (981 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Mark Johnston-2
In reply to this post by Eugene Grosbein-10
On Wed, Feb 13, 2019 at 01:48:21AM +0700, Eugene Grosbein wrote:

> 13.02.2019 1:42, Mark Johnston wrote:
>
> >> Yes, I have debugger compiled into running kernel and have console access.
> >> What commands should I use?
> >
> > I meant kgdb(1).  If you can run that, try:
> >
> > (kgdb) p time_uptime
> > (kgdb) p lowmem_uptime
> >
> > If you are willing to drop the system into DDB, do so and run
> >
> > db> x time_uptime
> > db> x lowmem_uptime
>
> I will reach the console next day only. Is it wise to use kgdb over ssh for running remote system? :-)

It should be fine.  kgdb opens /dev/(k)mem read-only.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Eugene Grosbein-10
In reply to this post by Mark Johnston-2
13.02.2019 1:18, Mark Johnston wrote:

> Depending on the system's workload, it is possible for the caches to
> grow quite quickly after a reclaim.  If you are able to run kgdb on the
> kernel, you can find the time of the last reclaim by comparing the
> values of lowmem_uptime and time_uptime.

(kgdb) p time_uptime
$1 = 7019265
(kgdb) p lowmem_uptime
$2 = 7000568
(kgdb) quit
# uptime
 2:08  up 81 days,  5:48, 5 users, load averages: 0,19 0,13 0,09

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Eugene Grosbein-10
In reply to this post by Lev Serebryakov
13.02.2019 1:50, Lev Serebryakov wrote:

>  I'm have same problem.
>
>  According to top(1) I have 29G Wired, but only 17G Total ARC (12G
> difference! System has 32G of RAM), and this statistic shows:
>
>     5487.5 zio_data_buf_524288
>    920.125 zio_data_buf_131072
>        626 zio_buf_131072
>        468 zio_data_buf_1048576
>    398.391 zio_buf_16384
>    305.464 dnode_t
>    227.989 zio_buf_512
>      171.5 zio_data_buf_458752
>     141.75 zio_data_buf_393216
>    116.456 dmu_buf_impl_t
>
>  So, more than 6G (!) is not used in ARC, but hold by ZFS anyway.

dnode_t and dmu_buf_impl_t are parts of ZFS too,
so these numbers represent about 9G, not 6G.

Do you have/had some memory pressure here? Growth of swap usage?
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
12