11.2-STABLE kernel wired memory leak

classic Classic list List threaded Threaded
36 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

mdtancsa
On 2/12/2019 1:03 PM, Eugene Grosbein wrote:

> It seems page daemon is broken somehow as it did not reclaim several
> gigs of wired memory
> despite of long period of vm thrashing:
>
> $ sed 's/:/,/' vmstat-z.txt | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, $1}' | sort -k1,1 -rn | head
>       1892 abd_chunk
>    454.629 dnode_t
>     351.35 zio_buf_512
>    228.391 zio_buf_16384
>    173.968 dmu_buf_impl_t
>     130.25 zio_data_buf_131072
>    93.6887 VNODE
>    81.6978 arc_buf_hdr_t_full
>    74.9368 256
>    57.4102 4096

On an nfs server, serving a few large files, my 32G box is showing

vmstat -z | sed 's/:/,/' | awk -F, '{printf "%10s %s\n",
$2*$5/1024/1024, $1}' | sort -k1,1 -rn | head
   11014.3 abd_chunk
    2090.5 zio_data_buf_131072
   1142.67 mbuf_jumbo_page
   1134.25 zio_buf_131072
    355.28 mbuf_jumbo_9k
    233.42 zio_cache
   163.099 arc_buf_hdr_t_full
   130.738 128
   97.2812 zio_buf_16384
   96.5099 UMA Slabs


CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 1348K Active, 98M Inact, 3316K Laundry, 30G Wired, 1022M Free
ARC: 11G Total, 7025M MFU, 3580M MRU, 11M Anon, 78M Header, 681M Other
     9328M Compressed, 28G Uncompressed, 3.05:1 Ratio
Swap: 64G Total, 13M Used, 64G Free


Right now its OK, but prior to limiting ARC, I had an issue with memory
and the disk thrashing due to swapping

pid 643 (devd), uid 0, was killed: out of swap space


    ---Mike



> -
> -------------------
> Mike Tancsa, tel +1 519 651 3400 x203
> Sentex Communications, [hidden email]
> Providing Internet services since 1994 www.sentex.net
> Cambridge, Ontario Canada  
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Eugene Grosbein-10
13.02.2019 2:18, Mike Tancsa wrote:

> On an nfs server, serving a few large files, my 32G box is showing
>
> vmstat -z | sed 's/:/,/' | awk -F, '{printf "%10s %s\n",
> $2*$5/1024/1024, $1}' | sort -k1,1 -rn | head
>    11014.3 abd_chunk
>     2090.5 zio_data_buf_131072
>    1142.67 mbuf_jumbo_page
>    1134.25 zio_buf_131072
>     355.28 mbuf_jumbo_9k
>     233.42 zio_cache
>    163.099 arc_buf_hdr_t_full
>    130.738 128
>    97.2812 zio_buf_16384
>    96.5099 UMA Slabs
>
>
> CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
> Mem: 1348K Active, 98M Inact, 3316K Laundry, 30G Wired, 1022M Free
> ARC: 11G Total, 7025M MFU, 3580M MRU, 11M Anon, 78M Header, 681M Other
>      9328M Compressed, 28G Uncompressed, 3.05:1 Ratio
> Swap: 64G Total, 13M Used, 64G Free
>
>
> Right now its OK, but prior to limiting ARC, I had an issue with memory
> and the disk thrashing due to swapping
>
> pid 643 (devd), uid 0, was killed: out of swap space

mbuf* numbers represent memory being wasted IMHO.

In contrast to ZFS memory that can contain useful cached data,
contents of freed mbufs cannot be re-used, right?

Some amount of "free" mbufs in the zone's just fine to serve bursts of traffic
without need to grow the zone but your numbers are way too big IMHO.

Do you have memory pressure here? Growth of swap usage?

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

mdtancsa
On 2/12/2019 2:38 PM, Eugene Grosbein wrote:

>
> mbuf* numbers represent memory being wasted IMHO.
>
> In contrast to ZFS memory that can contain useful cached data,
> contents of freed mbufs cannot be re-used, right?
>
> Some amount of "free" mbufs in the zone's just fine to serve bursts of traffic
> without need to grow the zone but your numbers are way too big IMHO.
>
> Do you have memory pressure here? Growth of swap usage?
>
I had to radically cut ARC down, otherwise the box would start swapping
all sorts of processes out


% sysctl -a vfs.zfs.arc_max
vfs.zfs.arc_max: 12307744768


    ---Mike



--
-------------------
Mike Tancsa, tel +1 519 651 3400 x203
Sentex Communications, [hidden email]
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada  

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Lev Serebryakov
In reply to this post by Eugene Grosbein-10
Hello Eugene,

Tuesday, February 12, 2019, 10:18:09 PM, you wrote:

>>  I'm have same problem.
>>
>>  According to top(1) I have 29G Wired, but only 17G Total ARC (12G
>> difference! System has 32G of RAM), and this statistic shows:
>>
>>     5487.5 zio_data_buf_524288
>>    920.125 zio_data_buf_131072
>>        626 zio_buf_131072
>>        468 zio_data_buf_1048576
>>    398.391 zio_buf_16384
>>    305.464 dnode_t
>>    227.989 zio_buf_512
>>      171.5 zio_data_buf_458752
>>     141.75 zio_data_buf_393216
>>    116.456 dmu_buf_impl_t
>>
>>  So, more than 6G (!) is not used in ARC, but hold by ZFS anyway.

> dnode_t and dmu_buf_impl_t are parts of ZFS too,
> so these numbers represent about 9G, not 6G.

> Do you have/had some memory pressure here? Growth of swap usage?
 I don't have memory pressure right now, but according to my previous
experience, ARC will not grow anymore even under heavy disk load (I don't
have vfs.zfs.arc_max set).

 Before new ARC (vfs.zfs.abd_scatter_enabled) I had typically ALL
memory occuped by ARC, Wired memory was almost exactly equal to ARC, and
ARC hitrate was higher (but I have not exact numbers, unfortunately).

 Now I have "vfs.zfs.abd_scatter_enabled=0", but still Wired is much larger
that ARC under any disk load (it is mostly torrent box).

--
Best regards,
 Lev                            mailto:[hidden email]

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Andriy Gapon
In reply to this post by Eugene Grosbein-10
On 12/02/2019 20:17, Eugene Grosbein wrote:
> 13.02.2019 1:14, Eugene Grosbein wrote:
>
>> Use following command to see how much memory is wasted in your case:
>>
>> vmstat -z | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, $1}' | sort -k1,1 -rn | head
>
> Oops, small correction:
>
> vmstat -z | sed 's/:/,/' | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, $1}' | sort -k1,1 -rn | head

I have a much uglier but somewhat more informative "one-liner" for
post-processing vmstat -z output:

vmstat -z | tail +3 | awk -F '[:,] *' 'BEGIN { total=0; cache=0; used=0 } {u =
$2 * $4; c = $2 * $5; t = u + c; cache += c; used += u; total += t; name=$1;
gsub(" ", "_", name); print t, name, u, c} END { print total, "TOTAL", used,
cache } ' | sort -n | perl -a -p -e 'while (($j, $_) = each(@F)) { 1 while
s/^(-?\d+)(\d{3})/$1,$2/; print $_, " "} print "\n"' | column -t

This would be much nicer as a small python script.
Or, even, we could add a sort option for vmstat -z / -m.

--
Andriy Gapon
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Eugene Grosbein-10
In reply to this post by Mark Johnston-2
12.02.2019 23:34, Mark Johnston wrote:

> On Tue, Feb 12, 2019 at 11:14:31PM +0700, Eugene Grosbein wrote:
>> Hi!
>>
>> Long story short: 11.2-STABLE/amd64 r335757 leaked over 4600MB kernel wired memory over 81 days uptime
>> out of 8GB total RAM.
>>
>> Details follow.
>>
>> I have a workstation running Xorg, Firefox, Thunderbird, LibreOffice and occasionally VirtualBox for single VM.
>>
>> It has two identical 320GB HDDs combined with single graid-based array with "Intel"
>> on-disk format having 3 volumes:
>> - one "RAID1" volume /dev/raid/r0 occupies first 10GB or each HDD;
>> - two "SINGLE" volumes /dev/raid/r1 and /dev/raid/r2 that utilize "tails" of HDDs (310GB each).
>>
>> /dev/raid/r0 (10GB) has MBR partitioning and two slices:
>> - /dev/raid/r0s1 (8GB) is used for swap;
>> - /dev/raid/r0s2 (2GB) is used by non-redundant ZFS pool named "os" that contains only
>> root file system (177M used) and /usr file system (340M used).
>>
>> There is also second pool (ZMIRROR) named "z" built directly on top of /dev/raid/r[12] volumes,
>> this pool contains all other file systems including /var, /home, /usr/ports, /usr/local, /usr/{src|obj} etc.
>>
>> # zpool list
>> NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
>> os    1,98G   520M  1,48G        -         -    55%    25%  1.00x  ONLINE  -
>> z      288G  79,5G   209G        -         -    34%    27%  1.00x  ONLINE  -
>>
>> This way I have swap outside of ZFS, boot blocks and partitioning mirrored by means of GEOM_RAID and
>> can use local console to break to single user mode to unmount all file system other than root and /usr
>> and can even export bigger ZFS pool "z". And I did that to see that ARC usage
>> (limited with vfs.zfs.arc_max="3G" in /boot/loader.conf) dropped from over 2500MB
>> down to 44MB but "Wired" stays high. Now after I imported "z" back and booted to multiuser mode
>> top(1) shows:
>>
>> last pid: 51242;  load averages:  0.24,  0.16,  0.13  up 81+02:38:38  22:59:18
>> 104 processes: 1 running, 103 sleeping
>> CPU:  0.0% user,  0.0% nice,  0.4% system,  0.2% interrupt, 99.4% idle
>> Mem: 84M Active, 550M Inact, 4K Laundry, 4689M Wired, 2595M Free
>> ARC: 273M Total, 86M MFU, 172M MRU, 64K Anon, 1817K Header, 12M Other
>>      117M Compressed, 333M Uncompressed, 2.83:1 Ratio
>> Swap: 8192M Total, 940K Used, 8191M Free
>>
>> I have KDB and DDB in my custom kernel also. How do I debug the leak further?
>>
>> I use nvidia-driver-340-340.107 driver for GK208 [GeForce GT 710B] video card.
>> Here are outputs of "vmstat -m": http://www.grosbein.net/freebsd/leak/vmstat-m.txt
>> and "vmstat -z": http://www.grosbein.net/freebsd/leak/vmstat-z.txt
>
> I suspect that the "leaked" memory is simply being used to cache UMA
> items.  Note that the values in the FREE column of vmstat -z output are
> quite large.  The cached items are reclaimed only when the page daemon
> wakes up to reclaim memory; if there are no memory shortages, large
> amounts of memory may accumulate in UMA caches.  In this case, the sum
> of the product of columns 2 and 5 gives a total of roughly 4GB cached.

After another day with mostly idle system, "Wired" increased to more than 6GB out of 8GB total.
I've tried to increase sysctl vm.v_free_min from default 12838 (50MB)
upto 1048576 (4GB) and "Wired" dropped a bit but it is still huge, 5060M:

last pid: 61619;  load averages:  1.05,  0.78,  0.40  up 81+22:33:09  18:53:49
119 processes: 1 running, 118 sleeping
CPU:  0.0% user,  0.0% nice, 50.0% system,  0.0% interrupt, 50.0% idle
Mem: 47M Active, 731M Inact, 4K Laundry, 5060M Wired, 2080M Free
ARC: 3049M Total, 216M MFU, 2428M MRU, 64K Anon, 80M Header, 325M Other
     2341M Compressed, 5874M Uncompressed, 2.51:1 Ratio
Swap: 8192M Total, 940K Used, 8191M Free

# sysctl vm.v_free_min
vm.v_free_min: 1048576
# sysctl vm.stats.vm.v_free_count
vm.stats.vm.v_free_count: 533232

ARC probably cached results of nightly periodic jobs traversing file system trees
and hit its limit (3G).

Still cannot understand where have another 2G (5G-3G) of wired memory leaked to?

USED:
#  vmstat -z | sed 's/:/,/' | awk -F, '{printf "%10s %s\n", $2*$4/1024/1024, $1}' | sort -k1,1 -rn | head
    2763,2 abd_chunk
   196,547 zio_buf_16384
   183,711 dnode_t
   128,304 zio_buf_512
   96,3062 VNODE
   79,0076 arc_buf_hdr_t_full
      66,5 zio_data_buf_131072
   63,0772 UMA Slabs
   61,6484 256
   61,2564 dmu_buf_impl_t

FREE:
#  vmstat -z | sed 's/:/,/' | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, $1}' | sort -k1,1 -rn | head
   245,301 dnode_t
   209,086 zio_buf_512
   110,163 dmu_buf_impl_t
   31,2598 64
    21,656 256
   10,9262 swblk
   10,6295 128
    9,0379 RADIX NODE
   8,54521 L VFS Cache
    7,4917 512

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Karl Denninger
In reply to this post by Andriy Gapon

On 2/13/2019 03:59, Andriy Gapon wrote:

> On 12/02/2019 20:17, Eugene Grosbein wrote:
>> 13.02.2019 1:14, Eugene Grosbein wrote:
>>
>>> Use following command to see how much memory is wasted in your case:
>>>
>>> vmstat -z | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, $1}' | sort -k1,1 -rn | head
>> Oops, small correction:
>>
>> vmstat -z | sed 's/:/,/' | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, $1}' | sort -k1,1 -rn | head
> I have a much uglier but somewhat more informative "one-liner" for
> post-processing vmstat -z output:
>
> vmstat -z | tail +3 | awk -F '[:,] *' 'BEGIN { total=0; cache=0; used=0 } {u =
> $2 * $4; c = $2 * $5; t = u + c; cache += c; used += u; total += t; name=$1;
> gsub(" ", "_", name); print t, name, u, c} END { print total, "TOTAL", used,
> cache } ' | sort -n | perl -a -p -e 'while (($j, $_) = each(@F)) { 1 while
> s/^(-?\d+)(\d{3})/$1,$2/; print $_, " "} print "\n"' | column -t
>
> This would be much nicer as a small python script.
> Or, even, we could add a sort option for vmstat -z / -m.
On 12-STABLE, r343809, /without /my patch set (and I've yet to see the
sort of stall problems I used to have regularly with 11.1)

root@NewFS:/disk/karl # sh vmstat.script
0              0                           0
0              AIO                         0              0
0              AIOCB                       0              0
0              AIOLIO                      0              0
0              AIOP                        0              0
0              DMAR_MAP_ENTRY              0              0
0              FFS1_dinode                 0              0
0              FPU_save_area               0              0
0              IPFW_IPv6_dynamic_states    0              0
0              IPFW_parent_dynamic_states  0              0
0              IPsec_SA_lft_c              0              0
0              LTS_VFS_Cache               0              0
0              MAC_labels                  0              0
0              NCLNODE                     0              0
0              STS_VFS_Cache               0              0
0              arc_buf_hdr_t_l2only        0              0
0              audit_record                0              0
0              cryptodesc                  0              0
0              cryptop                     0              0
0              domainset                   0              0
0              itimer                      0              0
0              mbuf_jumbo_16k              0              0
0              mbuf_jumbo_9k               0              0
0              nvme_request                0              0
0              procdesc                    0              0
0              rentr                       0              0
0              sctp_asconf                 0              0
0              sctp_asconf_ack             0              0
0              sctp_asoc                   0              0
0              sctp_chunk                  0              0
0              sctp_ep                     0              0
0              sctp_raddr                  0              0
0              sctp_readq                  0              0
0              sctp_stream_msg_out         0              0
0              sio_cache                   0              0
0              tcp_log                     0              0
0              tcp_log_bucket              0              0
0              tcp_log_node                0              0
0              tfo                         0              0
0              tfo_ccache_entries          0              0
0              udplite_inpcb               0              0
0              umtx_pi                     0              0
0              vtnet_tx_hdr                0              0
0              zio_buf_10485760            0              0
0              zio_buf_12582912            0              0
0              zio_buf_14680064            0              0
0              zio_buf_1572864             0              0
0              zio_buf_16777216            0              0
0              zio_buf_1835008             0              0
0              zio_buf_2097152             0              0
0              zio_buf_2621440             0              0
0              zio_buf_3145728             0              0
0              zio_buf_3670016             0              0
0              zio_buf_4194304             0              0
0              zio_buf_5242880             0              0
0              zio_buf_6291456             0              0
0              zio_buf_7340032             0              0
0              zio_buf_8388608             0              0
0              zio_data_buf_1048576        0              0
0              zio_data_buf_10485760       0              0
0              zio_data_buf_12582912       0              0
0              zio_data_buf_1310720        0              0
0              zio_data_buf_14680064       0              0
0              zio_data_buf_1572864        0              0
0              zio_data_buf_163840         0              0
0              zio_data_buf_16777216       0              0
0              zio_data_buf_1835008        0              0
0              zio_data_buf_196608         0              0
0              zio_data_buf_2097152        0              0
0              zio_data_buf_229376         0              0
0              zio_data_buf_262144         0              0
0              zio_data_buf_2621440        0              0
0              zio_data_buf_3145728        0              0
0              zio_data_buf_327680         0              0
0              zio_data_buf_3670016        0              0
0              zio_data_buf_393216         0              0
0              zio_data_buf_4194304        0              0
0              zio_data_buf_458752         0              0
0              zio_data_buf_524288         0              0
0              zio_data_buf_5242880        0              0
0              zio_data_buf_6291456        0              0
0              zio_data_buf_655360         0              0
0              zio_data_buf_7340032        0              0
0              zio_data_buf_786432         0              0
0              zio_data_buf_8388608        0              0
0              zio_data_buf_917504         0              0
7,168          vmem                        5,376          1,792
15,776         MAP                         696            15,080
16,384         epoch_record_pcpu           1,024          15,360
20,000         sackhole                    0              20,000
20,040         mt_zone                     10,512         9,528
26,880         UMA_Hash                    13,312         13,568
27,888         tcpreass                    0              27,888
31,808         ipq                         0              31,808
40,960         64_pcpu                     26,040         14,920
49,152         IPFW_counters               992            48,160
49,152         mt_stats_zone               28,032         21,120
51,792         sctp_laddr                  0              51,792
52,104         crypto_session              1,344          50,760
60,928         UMA_Kegs                    57,568         3,360
60,928         zio_buf_3584                0              60,928
74,784         hostcache                   1,056          73,728
75,240         KMAP_ENTRY                  1,440          73,800
81,920         zio_buf_5120                10,240         71,680
94,848         fakepg                      0              94,848
108,864        swpctrie                    32,112         76,752
110,592        zio_buf_6144                12,288         98,304
111,360        zil_lwb_cache               1,920          109,440
119,808        zio_buf_3072                15,360         104,448
121,856        zio_buf_7168                0              121,856
123,648        syncache                    0              123,648
124,000        rl_entry                    28,000         96,000
130,416        rtentry                     3,536          126,880
132,000        IPFW_IPv4_dynamic_states    0              132,000
132,000        udpcb                       800            131,200
132,000        zil_zcw_cache               0              132,000
138,600        metaslab_alloc_trace_cache  0              138,600
154,440        IPFW_dynamic_states_data    0              154,440
156,000        ttyinq                      30,080         125,920
157,976        cpuset                      2,808          155,168
171,776        ripcb                       976            170,800
183,264        taskq_zone                  48             183,216
189,440        zio_data_buf_5120           112,640        76,800
194,560        zio_buf_10240               51,200         143,360
198,000        tcptw                       0              198,000
200,704        zio_data_buf_3584           71,680         129,024
204,800        NAMEI                       0              204,800
215,040        zio_buf_14336               0              215,040
215,040        zio_buf_2560                112,640        102,400
225,720        VNODEPOLL                   2,640          223,080
227,840        zio_data_buf_2560           40,960         186,880
230,336        udp_inpcb                   12,200         218,136
230,640        SLEEPQUEUE                  171,440        59,200
234,240        ttyoutq                     26,368         207,872
242,048        tcp_inpcb                   27,328         214,720
242,048        tcpcb                       54,656         187,392
248,000        KNOTE                       69,600         178,400
251,904        12_Bucket                   35,328         216,576
254,800        ksiginfo                    40,432         214,368
258,048        zio_data_buf_3072           52,224         205,824
258,400        pipe                        128,440        129,960
265,216        zio_data_buf_7168           100,352        164,864
272,640        unpcb                       45,824         226,816
277,760        selfd                       51,008         226,752
278,528        DIRHASH                     0              278,528
280,000        Files                       81,840         198,160
282,624        zio_buf_1536                53,760         228,864
288,768        zio_data_buf_6144           153,600        135,168
342,720        TURNSTILE                   291,448        51,272
357,696        filedesc0                   209,760        147,936
366,792        swblk                       303,144        63,648
368,640        zio_data_buf_1536           44,544         324,096
370,440        Mountpoints                 293,608        76,832
387,072        zio_data_buf_14336          200,704        186,368
413,696        zio_buf_1024                48,128         365,568
448,384        16_Bucket                   33,280         415,104
491,520        zio_data_buf_10240          266,240        225,280
492,032        8_Bucket                    19,456         472,576
494,016        6_Bucket                    36,576         457,440
513,936        PROC                        250,992        262,944
516,096        zio_data_buf_2048           61,440         454,656
523,200        socket                      231,952        291,248
537,680        g_bio                       6,768          530,912
540,960        UMA_Zones                   539,000        1,960
557,056        mbuf_jumbo_page             0              557,056
565,248        zio_buf_12288               12,288         552,960
581,632        zio_buf_2048                276,480        305,152
598,016        zio_data_buf_1024           33,792         564,224
605,392        VMSPACE                     251,200        354,192
659,456        zio_buf_28672               28,672         630,784
761,856        64_Bucket                   283,136        478,720
819,200        zio_buf_20480               20,480         798,720
844,000        4_Bucket                    466,496        377,504
906,752        mbuf_packet                 512            906,240
917,504        lz4_ctx                     0              917,504
937,984        128_Bucket                  719,872        218,112
972,096        zio_link_cache              14,592         957,504
1,128,960      32_Bucket                   610,304        518,656
1,130,496      8,192                       737,280        393,216
1,376,256      zio_buf_57344               0              1,376,256
1,376,256      zio_data_buf_512            21,504         1,354,752
1,400,832      zio_buf_24576               49,152         1,351,680
1,409,024      zio_buf_32768               0              1,409,024
1,474,560      zio_buf_81920               0              1,474,560
1,580,000      FFS_inode                   0              1,580,000
1,604,608      mbuf                        535,808        1,068,800
1,622,016      16,384                      1,048,576      573,440
1,802,240      zio_data_buf_16384          327,680        1,474,560
1,870,080      FFS2_dinode                 0              1,870,080
2,064,384      zio_data_buf_28672          516,096        1,548,288
2,097,152      zio_buf_8192                8,192          2,088,960
2,129,920      zio_buf_40960               122,880        2,007,040
2,260,992      zio_buf_49152               0              2,260,992
2,727,936      2,048                       1,646,592      1,081,344
2,894,760      umtx_shm                    2,758,800      135,960
2,949,120      zio_buf_163840              0              2,949,120
2,998,800      THREAD                      2,795,800      203,000
3,145,728      zio_buf_196608              0              3,145,728
3,244,032      zio_data_buf_24576          688,128        2,555,904
3,702,784      zio_data_buf_32768          491,520        3,211,264
3,899,392      zio_buf_229376              0              3,899,392
3,932,160      zio_buf_98304               0              3,932,160
4,128,768      65,536                      2,686,976      1,441,792
4,472,832      zio_buf_114688              0              4,472,832
4,747,264      zio_buf_4096                2,469,888      2,277,376
4,891,488      16                          4,252,368      639,120
5,144,576      32,768                      3,506,176      1,638,400
5,148,672      zio_data_buf_12288          270,336        4,878,336
5,242,880      zio_buf_262144              0              5,242,880
5,242,880      zio_buf_327680              0              5,242,880
5,304,320      zio_data_buf_20480          368,640        4,935,680
5,406,720      zio_data_buf_40960          1,024,000      4,382,720
5,817,088      arc_buf_t                   2,932,160      2,884,928
5,853,184      512                         1,529,344      4,323,840
5,935,104      256_Bucket                  5,836,800      98,304
6,039,544      vmem_btag                   4,980,528      1,059,016
6,502,272      L_VFS_Cache                 2,106,088      4,396,184
7,471,104      zio_buf_393216              0              7,471,104
8,519,680      zio_buf_65536               0              8,519,680
8,988,000      32                          8,279,456      708,544
9,175,040      zio_buf_458752              0              9,175,040
9,535,488      1,024                       9,174,016      361,472
11,376,288     BUF_TRIE                    0              11,376,288
11,640,832     zio_data_buf_57344          860,160        10,780,672
11,657,216     mbuf_cluster                11,446,272     210,944
11,796,480     zio_buf_655360              0              11,796,480
13,271,040     zio_data_buf_81920          737,280        12,533,760
14,024,704     zio_data_buf_65536          917,504        13,107,200
17,039,360     zio_buf_1310720             0              17,039,360
17,301,504     zio_buf_524288              0              17,301,504
18,087,936     zio_data_buf_98304          1,277,952      16,809,984
18,153,456     zio_cache                   388,808        17,764,648
24,144,120     MAP_ENTRY                   16,120,200     8,023,920
26,214,400     zio_buf_1048576             2,097,152      24,117,248
29,379,240     range_seg_cache             21,302,856     8,076,384
29,782,080     RADIX_NODE                  17,761,104     12,020,976
34,511,400     S_VFS_Cache                 31,512,672     2,998,728
38,535,168     zio_buf_786432              0              38,535,168
40,680,144     sa_cache                    40,548,816     131,328
41,517,056     zio_data_buf_114688         1,490,944      40,026,112
42,205,184     zio_buf_917504              0              42,205,184
50,147,328     zio_data_buf_4096           98,304         50,049,024
50,675,712     zio_data_buf_49152          983,040        49,692,672
53,972,736     64                          29,877,888     24,094,848
61,341,696     zio_buf_131072              42,205,184     19,136,512
72,019,200     VM_OBJECT                   71,597,056     422,144
76,731,200     zfs_znode_cache             76,592,208     138,992
88,972,800     256                         82,925,568     6,047,232
90,390,528     4,096                       89,911,296     479,232
94,036,000     UMA_Slabs                   94,033,280     2,720
135,456,000    VNODE                       135,273,600    182,400
171,928,320    arc_buf_hdr_t_full          119,737,344    52,190,976
221,970,432    zio_data_buf_8192           166,076,416    55,894,016
277,923,840    dmu_buf_impl_t              130,788,240    147,135,600
376,586,240    zio_buf_16384               372,719,616    3,866,624
397,680,896    128                         195,944,448    201,736,448
443,023,360    zio_data_buf_131072         158,072,832    284,950,528
535,584,768    zio_buf_512                 255,641,088    279,943,680
713,552,840    dnode_t                     373,756,656    339,796,184
3,849,744,384  abd_chunk                   3,848,310,784  1,433,600
8,418,769,848  TOTAL                       6,459,507,936  1,959,261,912

So far running 12-STABLE "neat" is behaving well for me....

--
Karl Denninger
[hidden email] <mailto:[hidden email]>
/The Market Ticker/
/[S/MIME encrypted email preferred]/

smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Eugene Grosbein-10
In reply to this post by Eugene Grosbein-10
On 13.02.2019 19:04, Eugene Grosbein wrote:

> 12.02.2019 23:34, Mark Johnston wrote:
>
>> On Tue, Feb 12, 2019 at 11:14:31PM +0700, Eugene Grosbein wrote:
>>> Hi!
>>>
>>> Long story short: 11.2-STABLE/amd64 r335757 leaked over 4600MB kernel wired memory over 81 days uptime
>>> out of 8GB total RAM.
>>>
>>> Details follow.
>>>
>>> I have a workstation running Xorg, Firefox, Thunderbird, LibreOffice and occasionally VirtualBox for single VM.
>>>
>>> It has two identical 320GB HDDs combined with single graid-based array with "Intel"
>>> on-disk format having 3 volumes:
>>> - one "RAID1" volume /dev/raid/r0 occupies first 10GB or each HDD;
>>> - two "SINGLE" volumes /dev/raid/r1 and /dev/raid/r2 that utilize "tails" of HDDs (310GB each).
>>>
>>> /dev/raid/r0 (10GB) has MBR partitioning and two slices:
>>> - /dev/raid/r0s1 (8GB) is used for swap;
>>> - /dev/raid/r0s2 (2GB) is used by non-redundant ZFS pool named "os" that contains only
>>> root file system (177M used) and /usr file system (340M used).
>>>
>>> There is also second pool (ZMIRROR) named "z" built directly on top of /dev/raid/r[12] volumes,
>>> this pool contains all other file systems including /var, /home, /usr/ports, /usr/local, /usr/{src|obj} etc.
>>>
>>> # zpool list
>>> NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
>>> os    1,98G   520M  1,48G        -         -    55%    25%  1.00x  ONLINE  -
>>> z      288G  79,5G   209G        -         -    34%    27%  1.00x  ONLINE  -
>>>
>>> This way I have swap outside of ZFS, boot blocks and partitioning mirrored by means of GEOM_RAID and
>>> can use local console to break to single user mode to unmount all file system other than root and /usr
>>> and can even export bigger ZFS pool "z". And I did that to see that ARC usage
>>> (limited with vfs.zfs.arc_max="3G" in /boot/loader.conf) dropped from over 2500MB
>>> down to 44MB but "Wired" stays high. Now after I imported "z" back and booted to multiuser mode
>>> top(1) shows:
>>>
>>> last pid: 51242;  load averages:  0.24,  0.16,  0.13  up 81+02:38:38  22:59:18
>>> 104 processes: 1 running, 103 sleeping
>>> CPU:  0.0% user,  0.0% nice,  0.4% system,  0.2% interrupt, 99.4% idle
>>> Mem: 84M Active, 550M Inact, 4K Laundry, 4689M Wired, 2595M Free
>>> ARC: 273M Total, 86M MFU, 172M MRU, 64K Anon, 1817K Header, 12M Other
>>>      117M Compressed, 333M Uncompressed, 2.83:1 Ratio
>>> Swap: 8192M Total, 940K Used, 8191M Free
>>>
>>> I have KDB and DDB in my custom kernel also. How do I debug the leak further?
>>>
>>> I use nvidia-driver-340-340.107 driver for GK208 [GeForce GT 710B] video card.
>>> Here are outputs of "vmstat -m": http://www.grosbein.net/freebsd/leak/vmstat-m.txt
>>> and "vmstat -z": http://www.grosbein.net/freebsd/leak/vmstat-z.txt
>>
>> I suspect that the "leaked" memory is simply being used to cache UMA
>> items.  Note that the values in the FREE column of vmstat -z output are
>> quite large.  The cached items are reclaimed only when the page daemon
>> wakes up to reclaim memory; if there are no memory shortages, large
>> amounts of memory may accumulate in UMA caches.  In this case, the sum
>> of the product of columns 2 and 5 gives a total of roughly 4GB cached.
>
> After another day with mostly idle system, "Wired" increased to more than 6GB out of 8GB total.
> I've tried to increase sysctl vm.v_free_min from default 12838 (50MB)
> upto 1048576 (4GB) and "Wired" dropped a bit but it is still huge, 5060M:
>
> last pid: 61619;  load averages:  1.05,  0.78,  0.40  up 81+22:33:09  18:53:49
> 119 processes: 1 running, 118 sleeping
> CPU:  0.0% user,  0.0% nice, 50.0% system,  0.0% interrupt, 50.0% idle
> Mem: 47M Active, 731M Inact, 4K Laundry, 5060M Wired, 2080M Free
> ARC: 3049M Total, 216M MFU, 2428M MRU, 64K Anon, 80M Header, 325M Other
>      2341M Compressed, 5874M Uncompressed, 2.51:1 Ratio
> Swap: 8192M Total, 940K Used, 8191M Free
>
> # sysctl vm.v_free_min
> vm.v_free_min: 1048576
> # sysctl vm.stats.vm.v_free_count
> vm.stats.vm.v_free_count: 533232
>
> ARC probably cached results of nightly periodic jobs traversing file system trees
> and hit its limit (3G).
>
> Still cannot understand where have another 2G (5G-3G) of wired memory leaked to?

After several more hours of pretty idle time with quoted settings memory finally
got moved from "Wired" " to "Free" category. I got the system to single user mode again,
exported bigger ZFS pool again and ARC dropped down to 18M and Wired to less than 500M.
After re-import and boot to multiuser with Xorg running, Wired is less than 800M.

So, there seems to be no leak but insanely long time for Wired memory to be freed,
even with extreme value for vm.v_free_min. Even heavy memory pressure does not speed things up.

It seems, page daemon is somewhat broken and fails to free wired memory timely.

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Stefan Esser-3
In reply to this post by Andriy Gapon
Am 13.02.19 um 10:59 schrieb Andriy Gapon:

> On 12/02/2019 20:17, Eugene Grosbein wrote:
>> 13.02.2019 1:14, Eugene Grosbein wrote:
>>
>>> Use following command to see how much memory is wasted in your case:
>>>
>>> vmstat -z | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, $1}' | sort -k1,1 -rn | head
>>
>> Oops, small correction:
>>
>> vmstat -z | sed 's/:/,/' | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, $1}' | sort -k1,1 -rn | head
>
> I have a much uglier but somewhat more informative "one-liner" for
> post-processing vmstat -z output:
>
> vmstat -z | tail +3 | awk -F '[:,] *' 'BEGIN { total=0; cache=0; used=0 } {u =
> $2 * $4; c = $2 * $5; t = u + c; cache += c; used += u; total += t; name=$1;
> gsub(" ", "_", name); print t, name, u, c} END { print total, "TOTAL", used,
> cache } ' | sort -n | perl -a -p -e 'while (($j, $_) = each(@F)) { 1 while
> s/^(-?\d+)(\d{3})/$1,$2/; print $_, " "} print "\n"' | column -t
>
> This would be much nicer as a small python script.

Or, since you are already using perl:

----------------------------------------------------------------
#!/usr/local/bin/perl5





open STDIN, "vmstat -z |" or die "Failed to run vmstat program";
open STDOUT, "| sort -n @ARGV -k 2" or die "Failed to pipe through sort";

$fmt="%-30s %8.3f %8.3f %8.3f %6.1f%%\n";
while (<STDIN>) {
    ($n, $s, undef, $u, $c) = split /[:,] */;
    next unless $s > 0;
    $n =~ s/ /_/g;
    $s /= 1024 * 1024;
    $u *= $s;
    $c *= $s;
    $t =  $u + $c;
    next unless $t > 0;
    printf $fmt, $n, $t, $u, $c, 100 * $c / $t;
    $cache += $c;
    $used  += $u;
    $total += $t;
}
printf $fmt, TOTAL, $total, $used, $cache, 100 * $cached / $total;
close STDOUT;
----------------------------------------------------------------

This script takes additional parameters, e.g. "-r" to reverse the
sort or "-r -k5" to print those entries with the highest percentage
of unused memory at the top.

(I chose to suppress lines with a current "total" of 0 - you may
want to remove the "next" command above the printf in the loop to
see them, but they did not seem useful to me.)

> Or, even, we could add a sort option for vmstat -z / -m.

Yes, and the hardest part might be to select option characters
for the various useful report formats. ;-)

Regards, STefan
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Lev Serebryakov
In reply to this post by Eugene Grosbein-10
Hello Eugene,

Tuesday, February 12, 2019, 10:18:09 PM, you wrote:

> Do you have/had some memory pressure here? Growth of swap usage?
 After several days, ARC is even smaller, but Wired is the same:

Mem: 88M Active, 904M Inact, 29G Wired, 1121M Free
ARC: 13G Total, 5288M MFU, 6937M MRU, 2880K Anon, 44M Header, 1036M Other
     11G Compressed, 15G Uncompressed, 1.32:1 Ratio

  :-\

 Strill "lowmem_uptime" is zero, so no "low memory" situation, but ARC
becomes smaller and smaller (and hit rate goes down too) and it is not clear
where are all this "Wired" RAM :-(

 I had 16G on this system with same load and about half-year ago 14G was
typical ARC with 97% hit rate and now it is 32G (other things are unchanged)
and it is 13G ARC and 94% hit rate. And all memory is still Wired!

--
Best regards,
 Lev                            mailto:[hidden email]

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

mdtancsa
In reply to this post by Eugene Grosbein-10
On 2/12/2019 11:49 AM, Eugene Grosbein wrote:

> 12.02.2019 23:34, Mark Johnston wrote:
>
>> I suspect that the "leaked" memory is simply being used to cache UMA
>> items.  Note that the values in the FREE column of vmstat -z output are
>> quite large.  The cached items are reclaimed only when the page daemon
>> wakes up to reclaim memory; if there are no memory shortages, large
>> amounts of memory may accumulate in UMA caches.  In this case, the sum
>> of the product of columns 2 and 5 gives a total of roughly 4GB cached.
> Forgot to note, that before I got system to single user mode, there was heavy swap usage (over 3.5GB)
> and heavy page-in/page-out, 10-20 megabytes per second and system was crawling slow due to pageing.

I just ran into this issue on a RELENG12 box I was getting ready for the
FreeBSD netperf cluster. It seems pretty easy to trigger. I created a 12
disk raidz pool and did a simple dd test and the box started to run out
of memory

pid 776 (rpc.statd), jid 0, uid 0, was killed: out of swap space
pid 784 (rpc.lockd), jid 0, uid 0, was killed: out of swap space

CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 1120K Active, 628K Inact, 264K Laundry, 30G Wired, 26M Buf, 1133M Free
ARC: 28G Total, 73K MFU, 28G MRU, 32K Anon, 56M Header, 1713K Other
     27G Compressed, 27G Uncompressed, 1.00:1 Ratio

zpool create tanker raidz1 da0p1 da1p1 da2p1 da3p1 da4p1 da5p1 da6p1
da7p1 da8p1 da9p1 da10p1 da11p1

dd if=/dev/zero of=/tanker/test bs=1m count=100000

zpool destroy tanker

last pid:  1078;  load averages:  0.37,  1.32, 
0.84                                                                up
0+00:11:44  19:22:03
32 processes:  1 running, 31 sleeping
CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 564K Active, 792K Inact, 1000K Laundry, 30G Wired, 26M Buf, 1046M Free
Swap:

# vmstat -z | awk -F, '{printf "%10s %s\n", $2*$5/1024/1024, $1}' | sort
-k1,1 -rn | head
   73899.1 mbuf_cluster:          2048
     23693 mbuf_packet:            256
   4912.17 socket:                 872
   4719.04 unpcb:                  256
   147.354 udpcb:                   32
   147.345 udp_inpcb:              488
   28.8717 tcpcb:                  976
   28.8717 tcp_inpcb:              488
   11.6294 mbuf_jumbo_page:       4096
   2.98672 ripcb:                  488
#





> --
> -------------------
> Mike Tancsa, tel +1 519 651 3400 x203
> Sentex Communications, [hidden email]
> Providing Internet services since 1994 www.sentex.net
> Cambridge, Ontario Canada  
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

mdtancsa
On 2/19/2019 2:35 PM, Mike Tancsa wrote:
> dd if=/dev/zero of=/tanker/test bs=1m count=100000

The box has 32G of RAM. If I do a

# sysctl -w vfs.zfs.arc_max=12224866304
vfs.zfs.arc_max: 32224866304 -> 12224866304
#

after WIRED memory is at 29G, it doesnt immediately reclaim it and there
is memory pressure.  Booting the box with
vfs.zfs.arc_max=12224866304
keeps WIRED at 15G


last pid:  1049;  load averages:  2.02,  2.30, 
1.07                                                                up
0+00:04:21  20:17:43
38 processes:  2 running, 36 sleeping
CPU:  3.6% user,  0.0% nice,  5.9% system,  0.0% interrupt, 90.5% idle
Mem: 39M Active, 6212K Inact, 15G Wired, 30M Buf, 16G Free
ARC: 11G Total, 363K MFU, 11G MRU, 117M Anon, 23M Header, 850K Other
     11G Compressed, 11G Uncompressed, 1.01:1 Ratio
Swap:


Re-running the test but this time with swap,

# zpool create tanker raidz1 da0p1 da1p1 da2p1 da3p1 da4p1 da5p1 da6p1
da7p1 da8p1 da9p1 da10p1 da11p1 da12p1 da13p1 da14p1
# pstat -T
120/1043936 files
0M/65536M swap space
#

There is still 30G of Wired memory ?

last pid:  1096;  load averages:  0.78,  2.13, 
1.25                                                                up
0+00:06:42  20:33:21
35 processes:  1 running, 34 sleeping
CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 976K Active, 274M Inact, 88M Laundry, 30G Wired, 1341M Buf, 722M Free
ARC: 1227K Total, 775K MFU, 339K MRU, 32K Anon, 9216 Header, 69K Other
     176K Compressed, 1179K Uncompressed, 6.70:1 Ratio
Swap: 64G Total, 64G Free

I start a new dd on a freshly created pool,

CPU:  0.0% user,  0.0% nice, 31.4% system,  0.8% interrupt, 67.8% idle
Mem: 1396K Active, 460K Inact, 18M Laundry, 30G Wired, 1341M Buf, 720M Free
ARC: 27G Total, 54K MFU, 26G MRU, 1380M Anon, 56M Header, 3459K Other
     26G Compressed, 26G Uncompressed, 1.00:1 Ratio
Swap: 64G Total, 73M Used, 64G Free

and the box starts to swap

last pid:  1104;  load averages: 11.61,  6.10, 
3.04                                                                up
0+00:09:32  20:36:11
36 processes:  2 running, 34 sleeping
CPU:  0.1% user,  0.0% nice, 95.3% system,  2.1% interrupt,  2.4% idle
Mem: 1212K Active, 124K Inact, 688K Laundry, 30G Wired, 1341M Buf, 819M Free
ARC: 27G Total, 54K MFU, 24G MRU, 1620M Anon, 58M Header, 5572K Other
     24G Compressed, 24G Uncompressed, 1.01:1 Ratio
Swap: 64G Total, 90M Used, 64G Free



(avg's oneline script shows)
61,724,160      arc_buf_hdr_t_full          55,263,744      6,460,416
111,783,936     512                         109,534,208     2,249,728
555,884,000     UMA_Slabs                   553,374,640     2,509,360
1,280,966,656   zio_data_buf_131072         905,576,448     375,390,208
28,142,604,288  abd_chunk                   27,965,505,536  177,098,752
30,431,435,848  TOTAL                       29,663,389,468  768,046,380

# vmstat -z | sed 's/:/,/' | awk -F, '{printf "%10s %s\n",
$2*$5/1024/1024, $1}' | sort -k1,1 -rn | head
       358 zio_data_buf_131072
   168.895 abd_chunk
   18.5738 zio_cache
     14.25 zio_buf_786432
        14 zio_buf_1048576
    13.987 BUF TRIE
    13.125 zio_buf_655360
     9.625 zio_buf_917504
      8.75 zio_buf_458752
         8 zio_buf_524288
#

As the box is netbooted over nfs, when it swaps out some of the nfs
helper apps, the box gets a bit unresponsive.  e.g. if I do a find
/usr/src -type f |xargs md5 the box will stall out for periods of time. 
If I limit ARC to say 12G, all is well.


    ---Mike


--
-------------------
Mike Tancsa, tel +1 519 651 3400 x203
Sentex Communications, [hidden email]
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada  

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Eugene Grosbein-10
20.02.2019 3:49, Mike Tancsa wrote:

> On 2/19/2019 2:35 PM, Mike Tancsa wrote:
>> dd if=/dev/zero of=/tanker/test bs=1m count=100000
>
> The box has 32G of RAM. If I do a
>
> # sysctl -w vfs.zfs.arc_max=12224866304
> vfs.zfs.arc_max: 32224866304 -> 12224866304
> #
>
> after WIRED memory is at 29G, it doesnt immediately reclaim it and there
> is memory pressure.  Booting the box with
> vfs.zfs.arc_max=12224866304
> keeps WIRED at 15G

Can you repeat your test with additional vm.v_free_min=131072 (512M) or 262144 (1G)?
For me, it makes kernel reclaim unused UMA memory quickly:
first it goes from WIREE to FREE for a moment, then re-used as needed.


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

mdtancsa
On 2/20/2019 3:14 AM, Eugene Grosbein wrote:

> 20.02.2019 3:49, Mike Tancsa wrote:
>
>> On 2/19/2019 2:35 PM, Mike Tancsa wrote:
>>> dd if=/dev/zero of=/tanker/test bs=1m count=100000
>> The box has 32G of RAM. If I do a
>>
>> # sysctl -w vfs.zfs.arc_max=12224866304
>> vfs.zfs.arc_max: 32224866304 -> 12224866304
>> #
>>
>> after WIRED memory is at 29G, it doesnt immediately reclaim it and there
>> is memory pressure.  Booting the box with
>> vfs.zfs.arc_max=12224866304
>> keeps WIRED at 15G
> Can you repeat your test with additional vm.v_free_min=131072 (512M) or 262144 (1G)?
> For me, it makes kernel reclaim unused UMA memory quickly:
> first it goes from WIREE to FREE for a moment, then re-used as needed.

# sysctl -w vm.v_free_min=131072
vm.v_free_min: 51301 -> 131072
#

Keeping it at 1G seems to make things more smooth and stable and
programs are not being swapped out!




>
>

--
-------------------
Mike Tancsa, tel +1 519 651 3400 x203
Sentex Communications, [hidden email]
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada  


_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Robert Schulze-2
In reply to this post by Eugene Grosbein-10
Hi,

I just want to report a similar issue here with 11.2-RELEASE-p8.

The affected machine has 64 GB ram and does daily backups from several
machines in the night, at daytime there a parallel runs of clamav on a
specific dataset.

One symtom is basic I/O-performance: After upgrading from 11.1 to 11.2
backup times have increased, and are even still increasing. After one
week of operation, backup times have doubled - without having changed
anything else.

Then there is this wired memory and way too lazy reclaim of memory for
user processes: The clamav scans start at 10:30 and get swapped out
immediatly. Although vfs.zfs.arc_max=48G, wired is at 62 GB before the
scans and it takes about 10 minutes for the scan processes to actually
run on system ram, not swap.

There is obviously something broken, as there are several threads with
similar observations.

with kind regards,
Robert Schulze
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: 11.2-STABLE kernel wired memory leak

Paul Mather
On Mar 29, 2019, at 5:52 AM, Robert Schulze <[hidden email]> wrote:

> Hi,
>
> I just want to report a similar issue here with 11.2-RELEASE-p8.
>
> The affected machine has 64 GB ram and does daily backups from several
> machines in the night, at daytime there a parallel runs of clamav on a
> specific dataset.
>
> One symtom is basic I/O-performance: After upgrading from 11.1 to 11.2
> backup times have increased, and are even still increasing. After one
> week of operation, backup times have doubled - without having changed
> anything else.
>
> Then there is this wired memory and way too lazy reclaim of memory for
> user processes: The clamav scans start at 10:30 and get swapped out
> immediatly. Although vfs.zfs.arc_max=48G, wired is at 62 GB before the
> scans and it takes about 10 minutes for the scan processes to actually
> run on system ram, not swap.
>
> There is obviously something broken, as there are several threads with
> similar observations.


I am using FreeBSD 12 (both -RELEASE and -STABLE) and your comment about "way too lazy reclaim of memory" struck a chord with me.  On one system I regularly have hundreds of MB identified as being in the "Laundry" queue but FreeBSD hardly ever seems to do the laundry.  I see the same total for days.

When does FreeBSD decide to do its laundry?  Right now "top" is showing 835M in "Laundry" and the system is >99% idle.

How can I get the system to be more proactive about doing its housekeeping when it has idle time?

It would be much nicer to have it do laundry during a calm time rather than get all flustered when it's down to its last pair of socks (metaphorically speaking) and page even more stuff out to swap. :-)

Cheers,

Paul.

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[hidden email]"
12