网易视频云HBase RegionServer宕机案件侦查

如下所示的

Full GC

日志片段：

2016-04-14 21:27:13,174 WARN [JvmPauseMonitor] seMonitor:

Detected pause in JVM or host machine (eg GC): pause of approximately 20542ms

GC pool 'ParNew' had collection(s): count=1 time=0ms

GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=20898ms

2016-04-14 21:27:13,174 WARN [icFlusher]

r: We slept 20936ms instead of 100ms, this is likely due to a long

garbage collecting pause and it's usually bad, see

/#red

可以看出，

HBase

执行了一次

CMS GC

，导致整个进程所有线程被挂起了

20s

。通过

对

MemStore

的监控也可以看出这段时间

GC

力度之大，如下图所示：

GC

时间长最明显的危害是会造成上 层业务的阻塞，通过日志也可以看出些许端倪：

ption: Connection reset by peer

at 0(Native Method)

(:39)

toNativeBuffer(:223)

(:197)

(:384)

at lRead(:2246)

at

ver$$dProcess(

va:1496)

....

2016-04-14 21:32:40,173 WARN

[r=125,queue=5,port=60020] ver:

dercallId: 7540 service: ClientServicemethodName: Multi size:

100.2 K connection: 10.160.247.139:56031: output error

2016-04-14 21:32:40,173 WARN

[r=125,queue=5,port=60020] ver:

r=125,queue=5,port=60020: caught a

ClosedChannelException, this means that the server was processing a request but

the client went away. The error message was: null

上述日志表 示

HBase

服务端因为

Full G C

导致一直无法响应用户请求，用户客户端程

序在一定时间过后 就会

SocketTimeout

并断掉此

Connection

。连接断掉之后，服务器端

就会打印 如上日志。

然而，

这些和我们的终极目标好像并没有太大关系，

别忘了我们的目标

是找到

RS