Linux基础
1 前言
一个问题，一篇文章，一出故事。
今天发现阿里云服务器服务停止响应事件，一查发现内存比较低的服务发生“kernel: Out of memory: Killed process”事件，详细日志如下，
May 29 15:17:47 www systemd[1]: Starting dnf makecache... May 29 15:17:49 www dnf[1254394]: Extra Packages for Enterprise Linux 9 - x86_64 76 kB/s | 13 kB 00:00 May 29 15:17:54 www dnf[1254394]: Extra Packages for Enterprise Linux 9 - x86_64 4.3 MB/s | 20 MB 00:04 #... May 29 15:35:54 www kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB May 29 15:35:54 www kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB May 29 15:35:54 www kernel: 9835 total pagecache pages May 29 15:35:54 www kernel: 0 pages in swap cache May 29 15:35:54 www kernel: Free swap = 0kB May 29 15:35:54 www kernel: Total swap = 0kB May 29 15:35:54 www kernel: 1048070 pages RAM May 29 15:35:54 www kernel: 0 pages HighMem/MovableOnly May 29 15:35:54 www kernel: 111915 pages reserved May 29 15:35:54 www kernel: 0 pages cma reserved May 29 15:35:54 www kernel: 0 pages hwpoisoned May 29 15:35:54 www kernel: Tasks state (memory values in pages): May 29 15:35:54 www kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name May 29 15:35:54 www kernel: [ 488] 0 488 27533 410 245760 0 -250 systemd-journal May 29 15:35:54 www kernel: [ 502] 0 502 8658 448 81920 0 -1000 systemd-udevd May 29 15:35:54 www kernel: [ 582] 0 582 23216 273 61440 0 -1000 auditd May 29 15:35:54 www kernel: [ 584] 0 584 2018 160 53248 0 0 sedispatch May 29 15:35:54 www kernel: [ 610] 81 610 2968 319 65536 0 -900 dbus-broker-lau May 29 15:35:54 www kernel: [ 611] 81 611 1450 282 53248 0 -900 dbus-broker May 29 15:35:54 www kernel: [ 615] 992 615 675 32 40960 0 0 lsmd May 29 15:35:54 www kernel: [ 616] 0 616 703 64 40960 0 0 mcelog May 29 15:35:54 www kernel: [ 618] 0 618 5066 384 81920 0 0 systemd-logind May 29 15:35:54 www kernel: [ 620] 988 620 21213 140 73728 0 0 chronyd May 29 15:35:54 www kernel: [ 672] 0 672 64757 1242 139264 0 0 NetworkManager May 29 15:35:54 www kernel: [ 686] 0 686 64413 3810 131072 0 0 tuned May 29 15:35:54 www kernel: [ 1002] 997 1002 644591 2756 212992 0 0 polkitd May 29 15:35:54 www kernel: [ 1012] 0 1012 171566 1159 81920 0 0 aliyun-service May 29 15:35:54 www kernel: [ 1015] 0 1015 117197 1014 425984 0 0 rsyslogd May 29 15:35:54 www kernel: [ 1016] 0 1016 4183 320 73728 0 -1000 sshd May 29 15:35:54 www kernel: [ 1018] 0 1018 1172 64 45056 0 0 atd May 29 15:35:54 www kernel: [ 1023] 0 1023 2149 224 53248 0 0 crond May 29 15:35:54 www kernel: [ 1027] 0 1027 760 0 49152 0 0 agetty May 29 15:35:54 www kernel: [ 1029] 0 1029 1402 64 57344 0 0 agetty May 29 15:35:54 www kernel: [ 1045] 0 1045 4870 265 61440 0 0 assist_daemon May 29 15:35:54 www kernel: [ 515926] 0 515926 14449 1793 159744 0 0 php-fpm May 29 15:35:54 www kernel: [ 515955] 0 515955 46790 926 102400 0 0 nginx May 29 15:35:54 www kernel: [ 629865] 987 629865 55195 8627 159744 0 0 nginx May 29 15:35:54 www kernel: [ 629866] 987 629866 55277 8767 159744 0 0 nginx May 29 15:35:54 www kernel: [ 629867] 987 629867 47610 917 98304 0 0 nginx May 29 15:35:54 www kernel: [ 639209] 48 639209 62311 19904 413696 0 0 php-fpm May 29 15:35:54 www kernel: [ 639210] 48 639210 61944 19574 409600 0 0 php-fpm May 29 15:35:54 www kernel: [ 639211] 48 639211 62285 19408 401408 0 0 php-fpm May 29 15:35:54 www kernel: [ 639212] 48 639212 63630 21262 425984 0 0 php-fpm May 29 15:35:54 www kernel: [ 639213] 48 639213 61922 19559 409600 0 0 php-fpm May 29 15:35:54 www kernel: [ 639214] 48 639214 62075 19648 413696 0 0 php-fpm May 29 15:35:54 www kernel: [ 639215] 48 639215 61918 19028 405504 0 0 php-fpm May 29 15:35:54 www kernel: [ 639216] 48 639216 61901 19533 401408 0 0 php-fpm May 29 15:35:54 www kernel: [ 639217] 48 639217 60749 17872 393216 0 0 php-fpm May 29 15:35:54 www kernel: [ 639218] 48 639218 60891 18492 393216 0 0 php-fpm May 29 15:35:54 www kernel: [ 639219] 48 639219 61888 19054 417792 0 0 php-fpm May 29 15:35:54 www kernel: [ 639220] 48 639220 60384 17475 397312 0 0 php-fpm May 29 15:35:54 www kernel: [ 639221] 48 639221 61362 18493 401408 0 0 php-fpm May 29 15:35:54 www kernel: [ 639222] 48 639222 61464 19114 413696 0 0 php-fpm May 29 15:35:54 www kernel: [ 639223] 48 639223 61974 18614 413696 0 0 php-fpm May 29 15:35:54 www kernel: [ 639224] 48 639224 62345 19941 405504 0 0 php-fpm May 29 15:35:54 www kernel: [ 639225] 48 639225 61909 19525 409600 0 0 php-fpm May 29 15:35:54 www kernel: [ 639226] 48 639226 60985 18098 397312 0 0 php-fpm May 29 15:35:54 www kernel: [ 639227] 48 639227 62349 19452 413696 0 0 php-fpm May 29 15:35:54 www kernel: [ 639229] 48 639229 61920 19558 409600 0 0 php-fpm May 29 15:35:54 www kernel: [ 713575] 48 713575 61834 18939 405504 0 0 php-fpm May 29 15:35:54 www kernel: [ 713576] 48 713576 62017 19172 405504 0 0 php-fpm May 29 15:35:54 www kernel: [ 713577] 48 713577 60754 17879 409600 0 0 php-fpm May 29 15:35:54 www kernel: [ 713578] 48 713578 62292 19932 413696 0 0 php-fpm May 29 15:35:54 www kernel: [ 713579] 48 713579 59844 17463 393216 0 0 php-fpm May 29 15:35:54 www kernel: [ 713580] 48 713580 61868 18468 405504 0 0 php-fpm May 29 15:35:54 www kernel: [ 713581] 48 713581 61890 19044 409600 0 0 php-fpm May 29 15:35:54 www kernel: [ 713604] 48 713604 69154 26807 458752 0 0 php-fpm May 29 15:35:54 www kernel: [ 713605] 48 713605 61928 19112 393216 0 0 php-fpm May 29 15:35:54 www kernel: [ 713606] 48 713606 60897 17961 393216 0 0 php-fpm May 29 15:35:54 www kernel: [ 713607] 48 713607 61800 19435 405504 0 0 php-fpm May 29 15:35:54 www kernel: [ 713608] 48 713608 62098 19261 409600 0 0 php-fpm May 29 15:35:54 www kernel: [ 713609] 48 713609 61263 18877 401408 0 0 php-fpm May 29 15:35:54 www kernel: [ 713610] 48 713610 61508 18624 409600 0 0 php-fpm May 29 15:35:54 www kernel: [ 713611] 48 713611 61986 19612 405504 0 0 php-fpm May 29 15:35:54 www kernel: [ 855367] 0 855367 8224 472 81920 0 0 AliYunDunUpdate May 29 15:35:54 www kernel: [ 855428] 0 855428 24368 1843 274432 0 0 AliYunDun May 29 15:35:54 www kernel: [ 855449] 0 855449 38498 6783 364544 0 0 AliYunDunMonito May 29 15:35:54 www kernel: [1254394] 0 1254394 253194 181418 1646592 0 0 dnf May 29 15:35:54 www kernel: [1254401] 0 1254401 4393 446 69632 0 0 crond May 29 15:35:54 www kernel: [1254402] 0 1254402 4393 478 69632 0 0 crond May 29 15:35:54 www kernel: [1254406] 0 1254406 5463 448 77824 0 100 systemd May 29 15:35:54 www kernel: [1254409] 0 1254409 43434 1293 102400 0 100 (sd-pam) May 29 15:35:54 www kernel: [1254417] 0 1254417 1780 96 53248 0 0 sh May 29 15:35:54 www kernel: [1254419] 0 1254419 1780 96 53248 0 0 sh May 29 15:35:54 www kernel: [1254424] 0 1254424 1098 416 49152 0 0 pidstat May 29 15:35:54 www kernel: [1254425] 0 1254425 1601 64 53248 0 0 grep May 29 15:35:54 www kernel: [1254426] 0 1254426 1601 64 53248 0 0 grep May 29 15:35:54 www kernel: [1254427] 0 1254427 1098 384 45056 0 0 pidstat May 29 15:35:54 www kernel: [1254428] 0 1254428 1601 64 57344 0 0 grep May 29 15:35:54 www kernel: [1254429] 0 1254429 1601 64 49152 0 0 grep May 29 15:35:54 www kernel: [1254430] 0 1254430 4826 160 73728 0 0 sort May 29 15:35:54 www kernel: [1254431] 0 1254431 1398 32 49152 0 0 head May 29 15:35:54 www kernel: [1254432] 0 1254432 4826 160 65536 0 0 sort May 29 15:35:54 www kernel: [1254433] 0 1254433 1398 32 53248 0 0 head May 29 15:35:54 www kernel: [1254436] 0 1254436 2149 222 49152 0 0 crond May 29 15:35:54 www kernel: [1254437] 0 1254437 2149 222 49152 0 0 crond May 29 15:35:54 www kernel: [1254438] 0 1254438 4183 326 65536 0 0 sshd May 29 15:35:54 www kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/dnf-makecache.service,task=dnf,pid=1254394,uid=0 May 29 15:35:54 www kernel: Out of memory: Killed process 1254394 (dnf) total-vm:1012776kB, anon-rss:725672kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:1608kB oom_score_adj:0
基于以上日志，可见由于dnf占用大量内存而触发了“kernel: Out of memory: Killed process”事件，因此自发的dnf makecache行为必须被禁止。
2 最佳实践
systemctl stop dnf-makecache.timer systemctl disable dnf-makecache.timer
没有评论