flink日志问题

it2024-02-22  71

flink实时任务,使用了debug,但taskManager上运行的任务一直没有debug级别的日志,只有info日志。

最终回到flink配置,弄清flink/conf下面的日志配置文件作用

log4j.properties:在yarn模式下,jobManager和taskManager上用的log配置都依赖于它!

log4j-cli.properties:flink run启动时的日志依赖于该配置

log4j-yarn-session.properties:用yarn-session.sh启动时的日志依赖于该配置

logback.xml、logback-console.xml、logback-yarn.xml分别对应上面三个日志文件

所以在具体的flink任务配置debug并不能生效,如果需要看到taskManager,需要修改flink/conf/log4j.properties,比如设置DEBUG等级日志,如下

################################################################################ #  Licensed to the Apache Software Foundation (ASF) under one #  or more contributor license agreements.  See the NOTICE file #  distributed with this work for additional information #  regarding copyright ownership.  The ASF licenses this file #  to you under the Apache License, Version 2.0 (the #  "License"); you may not use this file except in compliance #  with the License.  You may obtain a copy of the License at # #      http://www.apache.org/licenses/LICENSE-2.0 # #  Unless required by applicable law or agreed to in writing, software #  distributed under the License is distributed on an "AS IS" BASIS, #  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. #  See the License for the specific language governing permissions and # limitations under the License. ################################################################################

# This affects logging for both user code and Flink

# log4j.rootLogger=INFO, file

log4j.rootLogger=DEBUG, file

# Uncomment this if you want to _only_ change Flink's logging #log4j.logger.org.apache.flink=INFO

# The following lines keep the log level of common libraries/connectors on # log level INFO. The root logger does not override this. You have to manually # change the log levels here. log4j.logger.akka=INFO log4j.logger.org.apache.kafka=INFO log4j.logger.org.apache.hadoop=INFO log4j.logger.org.apache.zookeeper=INFO

# Log all infos in the given file log4j.appender.file=org.apache.log4j.FileAppender log4j.appender.file.file=${log.file} log4j.appender.file.append=false log4j.appender.file.layout=org.apache.log4j.PatternLayout log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

# Suppress the irrelevant (wrong) warnings from the Netty channel handler log4j.logger.org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline=ERROR, file

 

 

最新回复(0)