java.lang.OutOfMemoryError: GC overhead limit exceeded

it2024-10-22 37

背景描述：运行Flink任务报错

TaskManager报错信息

java.lang.OutOfMemoryError: GC overhead limit exceeded at sun.reflect.GeneratedConstructorAccessor10.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at scala.reflect.runtime.JavaMirrors$JavaMirror$JavaVanillaMethodMirror2.jinvokeraw(JavaMirrors.scala:387) at scala.reflect.runtime.JavaMirrors$JavaMirror$JavaMethodMirror.jinvoke(JavaMirrors.scala:342) at scala.reflect.runtime.JavaMirrors$JavaMirror$JavaVanillaMethodMirror.apply(JavaMirrors.scala:358) at org.apache.flink.api.scala.typeutils.ScalaCaseClassSerializer$.$anonfun$lookupConstructor$2(ScalaCaseClassSerializer.scala:114) at org.apache.flink.api.scala.typeutils.ScalaCaseClassSerializer$$$Lambda$677/1363361196.apply(Unknown Source) at org.apache.flink.api.scala.typeutils.ScalaCaseClassSerializer.createInstance(ScalaCaseClassSerializer.scala:49) at org.apache.flink.api.scala.typeutils.ScalaCaseClassSerializer.createInstance(ScalaCaseClassSerializer.scala:39) at org.apache.flink.api.scala.typeutils.CaseClassSerializer.copy(CaseClassSerializer.scala:95) at org.apache.flink.api.scala.typeutils.CaseClassSerializer.copy(CaseClassSerializer.scala:32) at org.apache.flink.api.common.typeutils.base.ListSerializer.copy(ListSerializer.java:99) at org.apache.flink.api.common.typeutils.base.ListSerializer.copy(ListSerializer.java:42) at org.apache.flink.runtime.state.heap.CopyOnWriteStateTable.get(CopyOnWriteStateTable.java:297) at org.apache.flink.runtime.state.heap.CopyOnWriteStateTable.get(CopyOnWriteStateTable.java:321) at org.apache.flink.runtime.state.heap.HeapListState.add(HeapListState.java:95) at org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processElement(WindowOperator.java:391) at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:202) at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:105) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:302) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711) at java.lang.Thread.run(Thread.java:748)

结合JobManager的报错信息可以得到：首先是Akka超时，其次再是TaskManager Failed transfer File From Container 再是TaskManager包GC错误，然后丢弃checkpoint ，最后将状态置为Failing

java.util.concurrent.CompletionException: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/resourcemanager#-1745195078]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage". Caused by: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/resourcemanager#-1745195078]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage". akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://flink/user/resourcemanager#-1745195078]] after [10000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.LocalFencedMessage". 2020-10-21 11:43:25,953 ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler - Failed to transfer file from TaskExecutor container_1603249018017_0018_01_000007. Caused by: org.apache.flink.util.FlinkException: Could not retrieve file from transient blob store. java.lang.OutOfMemoryError: GC overhead limit exceeded 2020-10-21 11:43:26,993 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Discarding checkpoint 5 of job 7c183902007313f276a62d30ed32dadc. 2020-10-21 11:43:26,993 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job SuperID_Stream (7c183902007313f276a62d30ed32dadc) switched from state RUNNING to FAILING. java.lang.OutOfMemoryError: GC overhead limit exceeded

错误描述： GC overhead limit exceeded 超过GC开销限制。通过更改JVM启动配置来增加堆大小

错误原因：JVM花费了98%的时间进行垃圾回收，而只得到2%可用的内存，频繁的进行内存回收(最起码已经进行了5次连续的垃圾回收)，JVM就会曝出ava.lang.OutOfMemoryError: GC overhead limit exceeded错误。被占用的内存，经过多次长时间的GC操作都无法回收，导致可用内存越来越少，俗称内存泄露，JVM就会报java.lang.OutOfMemoryError: GC overhead limit exceeded错误。

解决方式：

1.Disabling the error check altogether, via “-XX:-UseGCOverheadLimit”. 此种方式最终会改报错为java.lang.OutOfMemoryError: Java heap space 治标不治本

2.增大堆内存。既然堆内存少了，那就增加堆内存即可，Increasing the heap size, via “-Xmx1024m” (or more), or

3.优化内存泄漏的代码