OS环境：CentOS_7 软件环境：Grafana_v7.2.1、 loki_v1.6.0(包括promtail_v1.6.0) 操作用户：root
loki为日志存储服务端，promtail为日志采集客户端。Grafana展现交互用 部署遵循原则：loki和Grafana亲近部署；promtail v1.6.0只支持本地文件采集，所以promtail需要和被采集的日志文件部署在同一服务节点上。 以下有两个注意事项：
loki_v1.6.1(包括promtail_v1.6.1)在配置Grafana_v7.2.1会出现数据源连接正常但找不到label的异常，curl “http://IP:3100/api/prom/label” 或 curl localhost:3100/loki/api/v1/labels 访问laber有值，应该是版本bug，在loki github issue里可以找到类似bug。本记录以loki_v1.6.0（包括promtail_v1.6.0）+Grafana_v7.2.1为例部署promail采集的日志文件大于4M会报错，这是 grpc的数据传输限制。loki master主线已经释出配置接口，估计在1.6.1之后的版本该问题会修复。当前解决方案是应用文件在生成日志后单个文件大小限定在4M之下。下载loki、promtail安装包和配置文件
# curl -OL https://github.com/grafana/loki/releases/download/v1.6.0/loki-linux-amd64.zip # curl -OL https://github.com/grafana/loki/releases/download/v1.6.0/promtail-linux-amd64.zip # wget https://github.com/grafana/loki/blob/v1.6.0/cmd/loki/loki-local-config.yaml # wget https://github.com/grafana/loki/blob/v1.6.0/cmd/promtail/promtail-local-config.yaml部署
# mkdir -p /opt/loki/tmp # mkdir /opt/logs/ # unzip loki-linux-amd64.zip # unzip promtail-linux-amd64.zip # mv loki-linux-amd64 loki-local-config.yaml promtail-linux-amd64 promtail-local-config.yaml /opt/loki # cd /opt/loki/ 此处需要注意文件权限问题 # chown -R root:root * # chmod 777 *amd64修改loki数据存储路径： # vim loki-local-config.yaml :%s+/tmp/+/opt/loki/tmp/+
以下有2种启动方式 1）直接启动loki # nohup ./loki-linux-amd64 -config.file=loki-local-config.yaml > /opt/logs/loki-3100.log 2>&1 & 访问：# curl http://localhost:3100/metrics 如需debug日志，使用以下启动命令： # nohup ./loki-linux-amd64 -config.file=loki-local-config.yaml --log.level=debug > /opt/logs/loki-3100.log 2>&1 &
2）配置服务启动
# cat > /etc/systemd/system/loki.service <<EOF [Unit] Description=loki After=network.target [Service] ExecStart=/opt/loki/loki-linux-amd64 \ -config.file=/opt/loki/loki-local-config.yaml &>> /opt/logs/loki-3100.log Restart=on-failure [Install] WantedBy=multi-user.target EOF启动服务，设置开机自启，并检查服务开启状态
# systemctl daemon-reload # systemctl start loki # systemctl status loki # systemctl enable loki验证 curl http://localhost:3100/metrics
# vim promtail-local-config.yaml :%s+/tmp/+/opt/loki/tmp/+
promtail-local-config.yaml配置文件如下：
server: http_listen_port: 9080 grpc_listen_port: 0 positions: filename: /opt/loki/tmp/positions.yaml clients: - url: http://localhost:3100/loki/api/v1/push scrape_configs: - job_name: system static_configs: - targets: - localhost labels: job: opt-logs __path__: /opt/log/*.log以下有2种启动方式 1）直接启动promtail # nohup ./promtail-linux-amd64 -config.file=promtail-local-config.yaml > /opt/logs/promtail-9080.log 2>&1 & 2）系统启动
# cat > /etc/systemd/system/promtail.service <<EOF [Unit] Description=promtail After=network.target [Service] ExecStart=/opt/loki/promtail-linux-amd64 \ -config.file=/opt/loki/promtail-local-config.yaml &>> /opt/logs/promtail-9080.log Restart=on-failure [Install] WantedBy=multi-user.target EOF启动服务，设置开机自启，并检查服务开启状态
# systemctl daemon-reload # systemctl start promtail # systemctl status promtail # systemctl enable promtail验证： curl "http://127.0.0.1:3100/api/prom/label" curl localhost:3100/loki/api/v1/labels
1、下载安装 # wget https://dl.grafana.com/oss/release/grafana-7.2.1-1.x86_64.rpm # yum install grafana-7.2.1-1.x86_64.rpm 2、启动服务，设置开机自启，并检查服务开启状态
# systemctl daemon-reload # systemctl start grafana-server # systemctl status grafana-server # systemctl enable grafana-server如需要修改默认3000端口，可进行如下配置： 1）# vim /etc/grafana/grafana.ini 修改http_port = 3000 2）重启服务：# systemctl restart grafana-server 3）查看启动及端口修改情况:# systemctl status grafana-server
3、访问：http://[服务器IP]:3000 admin/admin登录
4、下载安装grafana的 loki dashboard json 文件 https://grafana.com/grafana/dashboards/12611
5、loki dashboard需要安装grafana插件： # grafana-cli plugins install grafana-piechart-panel # service grafana-server restart
略
打开http://IP:3000 admin/admin 登陆2次进入（原因不明） 打开Configuration->Data Sources->add data sources->搜索“loki”关键字->select->URL输入loki服务访问地址：http://localhost:3100 ->单击Save & Test 出现绿色提示：Data source connected and labels found.说明数据源配置成功 打开Explore, 选择loki数据源，查询promtail-local-config.yaml配置好的抽取到loki中的日志
可以导入开源的grafana日志监控dashboard https://grafana.com/grafana/dashboards/12611
参考 https://github.com/grafana/loki https://grafana.com/docs/loki/latest/getting-started/troubleshooting/ https://github.com/grafana/loki/issues/271 https://stackoverflow.com/questions/55362342/grpc-received-message-larger-than-max-8653851-vs-4194304 https://grpc.github.io/grpc/cpp/group__grpc__arg__keys.html https://grafana.com/docs/loki/latest/configuration/#grpc_client_config
