比思論壇

標題: WEB经常宕机，分析原因 [打印本頁]

作者: lcmj44444444 時間: 2013-1-16 17:55
標題: WEB经常宕机，分析原因
从top相关信息来看，每次宕机现象都是系统内核态占用非常高，相关nginx,php-cgi进程虽然不僵死但无法kill

每次宕机都会有kswaped0进程出现，但是从free上来看，内存还是有空闲，swap用得也并不多

top - 22:26:02 up 3 days,  8:06,  1 user,  load average: 15.05, 15.12, 15.09
Tasks: 518 total,  16 running, 502 sleeping, 0 stopped, 0 zombie
Cpu(s):  0.0%us, 87.6%sy,  0.0%ni, 12.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16426616k total, 15985652k used, 440964k free, 102232k buffers
Swap: 16386292k total, 22552k used, 16363740k free,  8320464k cached

  PID USER    PR  NI  VIRT  RES  SHR S %CPU %MEM TIME+  COMMAND
  553 root    20  -5    0 0 0 R 100.9  0.0 115:59.83 kswapd0
  554 root    20  -5    0 0 0 R 100.9  0.0 115:55.12 kswapd1
3417 daemon 25 0 90164  48m 1052 R 100.6  0.3 142:19.56 nginx
3418 daemon 25 0 90840  48m 1048 R 100.6  0.3 143:54.26 nginx
3422 daemon 25 0 94196  52m 1052 R 100.6  0.3 144:34.68 nginx
3423 daemon 25 0 85680  43m 1048 R 100.4  0.3 141:31.40 nginx
3421 daemon 25 0 87720  45m 1056 R 100.1  0.3 143:13.23 nginx
3419 daemon 25 0 90148  48m 1044 R 99.6  0.3 144:06.64 nginx
16942 daemon 25 0  440m  55m  33m R 99.6  0.3 106:08.76 php-cgi
14476 daemon 25 0  432m  55m  33m R 97.3  0.3 102:58.61 php-cgi
8706 daemon 25 0  432m  60m  37m R 93.7  0.4 103:46.72 php-cgi
3416 daemon 25 0 88832  47m 1048 R 93.2  0.3 144:04.28 nginx
7672 daemon 25 0  439m  67m  37m R 82.4  0.4 104:21.30 php-cgi
8102 daemon 25 0  441m  62m  36m R 59.8  0.4 107:12.30 php-cgi
3420 daemon 25 0 89760  47m 1056 R 50.1  0.3 143:38.59 nginx
15963 root    16 0 13008 1428  820 R 22.6  0.0 0:04.34 top
3428 root    15 0  411m 4300 1172 S  2.1  0.0 2:33.80 php-cgi
15965 root    18 0 99.6m 1584  908 S  0.8  0.0 0:00.03 crond
3743 root    15 0 74812 1164  584 S  0.3  0.0 0:00.83 crond
15964 root    18 0 99.6m 1468  800 D  0.3  0.0 0:00.01 crond

top - 22:26:02 up 3 days,  8:06,  1 user,  load average: 15.05, 15.12, 15.09
Tasks: 518 total,  16 running, 502 sleeping, 0 stopped, 0 zombie
Cpu(s):  0.0%us, 87.6%sy,  0.0%ni, 12.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16426616k total, 15985652k used, 440964k free, 102232k buffers
Swap: 16386292k total, 22552k used, 16363740k free,  8320464k cached

  PID USER    PR  NI  VIRT  RES  SHR S %CPU %MEM TIME+  COMMAND
  553 root    20  -5    0 0 0 R 100.9  0.0 115:59.83 kswapd0
  554 root    20  -5    0 0 0 R 100.9  0.0 115:55.12 kswapd1
3417 daemon 25 0 90164  48m 1052 R 100.6  0.3 142:19.56 nginx
3418 daemon 25 0 90840  48m 1048 R 100.6  0.3 143:54.26 nginx
3422 daemon 25 0 94196  52m 1052 R 100.6  0.3 144:34.68 nginx
3423 daemon 25 0 85680  43m 1048 R 100.4  0.3 141:31.40 nginx
3421 daemon 25 0 87720  45m 1056 R 100.1  0.3 143:13.23 nginx
3419 daemon 25 0 90148  48m 1044 R 99.6  0.3 144:06.64 nginx
16942 daemon 25 0  440m  55m  33m R 99.6  0.3 106:08.76 php-cgi
14476 daemon 25 0  432m  55m  33m R 97.3  0.3 102:58.61 php-cgi
8706 daemon 25 0  432m  60m  37m R 93.7  0.4 103:46.72 php-cgi
3416 daemon 25 0 88832  47m 1048 R 93.2  0.3 144:04.28 nginx
7672 daemon 25 0  439m  67m  37m R 82.4  0.4 104:21.30 php-cgi
8102 daemon 25 0  441m  62m  36m R 59.8  0.4 107:12.30 php-cgi
3420 daemon 25 0 89760  47m 1056 R 50.1  0.3 143:38.59 nginx
15963 root    16 0 13008 1428  820 R 22.6  0.0 0:04.34 top
3428 root    15 0  411m 4300 1172 S  2.1  0.0 2:33.80 php-cgi
15965 root    18 0 99.6m 1584  908 S  0.8  0.0 0:00.03 crond
3743 root    15 0 74812 1164  584 S  0.3  0.0 0:00.83 crond
15964 root    18 0 99.6m 1468  800 D  0.3  0.0 0:00.01 crond

free信息:

         total    used    free    shared buffers    cached
Mem:    16426616 16245028    181588       0    159248 12046536
-/+ buffers/cache: 4039244 12387372
Swap:    16386292    28980 16357312

sar -r

03:20:01 PM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
09:30:02 PM 268628  16157988    98.36 149268  11969424  16357312    28980    0.18    2644
09:40:02 PM 256492  16170124    98.44 150628  11980476  16357312    28980    0.18    2644
09:50:02 PM 244752  16181864    98.51 152092  11991476  16357312    28980    0.18    2644
10:00:02 PM 233296  16193320    98.58 153516  12002516  16357312    28980    0.18    2644
10:10:02 PM 219404  16207212    98.66 155032  12013568  16357312    28980    0.18    2644
10:20:02 PM 204912  16221704    98.75 156452  12024544  16357312    28980    0.18    2644
10:30:02 PM 193128  16233488    98.82 158008  12037200  16357312    28980    0.18    2644
Average:    588844  15837772    96.42 216781  11964159  16358029    28263    0.17    1420

sar -u
03:20:01 PM    CPU    %user    %nice %system %iowait %steal    %idle
03:30:01 PM    all    8.37    0.00    0.86    0.23    0.00    90.54
03:40:01 PM    all    8.11    0.00    0.79    0.17    0.00    90.93
03:50:01 PM    all    7.22    0.00    0.81    0.23    0.00    91.74
04:00:01 PM    all    7.60    0.00    0.84    0.16    0.00    91.41
04:10:01 PM    all    7.38    0.00    0.82    0.26    0.00    91.53
04:20:01 PM    all    7.12    0.00    0.81    0.27    0.00    91.80
04:30:01 PM    all    7.10    0.00    0.84    0.42    0.00    91.64
04:40:01 PM    all    7.38    0.00    0.83    0.23    0.00    91.55
04:50:01 PM    all    7.20    0.00    0.82    0.17    0.00    91.80
05:00:01 PM    all    7.54    0.00    0.83    0.19    0.00    91.44
05:10:01 PM    all    6.72    0.00    0.76    0.24    0.00    92.28
05:20:01 PM    all    6.98    0.00    0.79    0.13    0.00    92.09
05:30:02 PM    all    7.18    0.00    0.80    0.16    0.00    91.87
05:40:01 PM    all    7.07    0.00    0.77    0.15    0.00    92.01
05:50:01 PM    all    7.75    0.00    0.88    0.17    0.00    91.20
06:00:01 PM    all    7.46    0.00    0.85    0.25    0.00    91.43
06:10:01 PM    all    7.58    0.00    0.88    0.19    0.00    91.34
06:20:01 PM    all    7.81    0.00    0.88    0.28    0.00    91.03
06:30:01 PM    all    7.90    0.00    0.87    0.22    0.00    91.00
06:40:01 PM    all    7.20    0.00    0.86    0.22    0.00    91.72
06:50:01 PM    all    7.34    0.00    0.90    0.25    0.00    91.51
07:00:01 PM    all    7.94    0.00    0.96    0.23    0.00    90.87
07:10:01 PM    all    8.14    0.00    0.97    0.29    0.00    90.59
07:20:01 PM    all    8.23    0.00    0.97    0.41    0.00    90.39
07:30:01 PM    all    8.12    0.00    0.97    0.30    0.00    90.61
07:40:01 PM    all    7.77    0.00    0.90    0.30    0.00    91.03
07:50:01 PM    all    7.97    0.00    0.92    0.17    0.00    90.94
08:00:01 PM    all    8.44    0.00    1.03    0.43    0.00    90.10
08:10:01 PM    all    7.90    0.00    0.95    0.51    0.00    90.64
08:20:01 PM    all    8.31    0.00    1.03    0.18    0.00    90.48
08:30:01 PM    all    8.19    0.00    0.98    0.29    0.00    90.54
08:40:01 PM    all    8.88    0.00    1.06    0.27    0.00    89.79
08:50:01 PM    all    11.45    0.00    1.39    0.35    0.00    86.81
09:00:01 PM    all    11.67    0.00    1.44    0.48    0.00    86.41
09:10:02 PM    all    0.92    0.00    91.23    0.49    0.00    7.36
09:20:02 PM    all    0.01    0.00    99.88    0.00    0.00    0.11
09:30:02 PM    all    0.01    0.00    99.42    0.00    0.00    0.57
09:40:02 PM    all    0.01    0.00    98.05    0.00    0.00    1.94
09:50:02 PM    all    0.01    0.00    99.31    0.00    0.00    0.68
10:00:02 PM    all    0.01    0.00    99.83    0.00    0.00    0.15
10:10:02 PM    all    0.01    0.00    99.05    0.00    0.00    0.94
10:20:02 PM    all    0.01    0.00    99.69    0.00    0.00    0.30
10:30:02 PM    all    0.01    0.00    99.75    0.00    0.00    0.24
Average:       all    4.86    0.00    7.11    0.16    0.00    87.87

系统、nginx、php-fpm相关信息
系统:
cat /etc/issue
CentOS release 5.5 (Final)
Kernel \r on an \m

nginx:
/usr/local/nginx/sbin/nginx -V
nginx version: nginx/0.8.53
built by gcc 4.1.2 20080704 (Red Hat 4.1.2-50)
TLS SNI support disabled
configure arguments: --user= --group= --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module --with-http_sub_module --with-md5=/usr/lib --with-sha1=/usr/lib --with-http_gzip_static_module

php -v
PHP 5.2.17p1 with Suhosin-Patch 0.9.7 (cli) (built: Jan  7 2012 15:14:25)
Copyright (c) 1997-2009 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2010 Zend Technologies
with eAccelerator v0.9.6.1, Copyright (c) 2004-2010 eAccelerator, by eAccelerator

作者: sun_sdsd 時間: 2014-3-5 14:42
WEB经常宕机，分析原

作者: yct1978 時間: 2014-3-12 12:30
学习学习

歡迎光臨比思論壇 (http://108.170.5.102/)