2. it技术
  3. ELK日志分析平台：logstash安装，filebeat配置

ELK日志分析平台：logstash安装，filebeat配置

it2023-12-27  63

ELK日志分析平台

ELK架构图例

#mermaid-svg-sv1kSKiaLyKS7oCk .label{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);fill:#333;color:#333}#mermaid-svg-sv1kSKiaLyKS7oCk .label text{fill:#333}#mermaid-svg-sv1kSKiaLyKS7oCk .node rect,#mermaid-svg-sv1kSKiaLyKS7oCk .node circle,#mermaid-svg-sv1kSKiaLyKS7oCk .node ellipse,#mermaid-svg-sv1kSKiaLyKS7oCk .node polygon,#mermaid-svg-sv1kSKiaLyKS7oCk .node path{fill:#ECECFF;stroke:#9370db;stroke-width:1px}#mermaid-svg-sv1kSKiaLyKS7oCk .node .label{text-align:center;fill:#333}#mermaid-svg-sv1kSKiaLyKS7oCk .node.clickable{cursor:pointer}#mermaid-svg-sv1kSKiaLyKS7oCk .arrowheadPath{fill:#333}#mermaid-svg-sv1kSKiaLyKS7oCk .edgePath .path{stroke:#333;stroke-width:1.5px}#mermaid-svg-sv1kSKiaLyKS7oCk .flowchart-link{stroke:#333;fill:none}#mermaid-svg-sv1kSKiaLyKS7oCk .edgeLabel{background-color:#e8e8e8;text-align:center}#mermaid-svg-sv1kSKiaLyKS7oCk .edgeLabel rect{opacity:0.9}#mermaid-svg-sv1kSKiaLyKS7oCk .edgeLabel span{color:#333}#mermaid-svg-sv1kSKiaLyKS7oCk .cluster rect{fill:#ffffde;stroke:#aa3;stroke-width:1px}#mermaid-svg-sv1kSKiaLyKS7oCk .cluster text{fill:#333}#mermaid-svg-sv1kSKiaLyKS7oCk div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);font-size:12px;background:#ffffde;border:1px solid #aa3;border-radius:2px;pointer-events:none;z-index:100}#mermaid-svg-sv1kSKiaLyKS7oCk .actor{stroke:#ccf;fill:#ECECFF}#mermaid-svg-sv1kSKiaLyKS7oCk text.actor>tspan{fill:#000;stroke:none}#mermaid-svg-sv1kSKiaLyKS7oCk .actor-line{stroke:grey}#mermaid-svg-sv1kSKiaLyKS7oCk .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333}#mermaid-svg-sv1kSKiaLyKS7oCk .messageLine1{stroke-width:1.5;stroke-dasharray:2, 2;stroke:#333}#mermaid-svg-sv1kSKiaLyKS7oCk #arrowhead path{fill:#333;stroke:#333}#mermaid-svg-sv1kSKiaLyKS7oCk .sequenceNumber{fill:#fff}#mermaid-svg-sv1kSKiaLyKS7oCk #sequencenumber{fill:#333}#mermaid-svg-sv1kSKiaLyKS7oCk #crosshead path{fill:#333;stroke:#333}#mermaid-svg-sv1kSKiaLyKS7oCk .messageText{fill:#333;stroke:#333}#mermaid-svg-sv1kSKiaLyKS7oCk .labelBox{stroke:#ccf;fill:#ECECFF}#mermaid-svg-sv1kSKiaLyKS7oCk .labelText,#mermaid-svg-sv1kSKiaLyKS7oCk .labelText>tspan{fill:#000;stroke:none}#mermaid-svg-sv1kSKiaLyKS7oCk .loopText,#mermaid-svg-sv1kSKiaLyKS7oCk .loopText>tspan{fill:#000;stroke:none}#mermaid-svg-sv1kSKiaLyKS7oCk .loopLine{stroke-width:2px;stroke-dasharray:2, 2;stroke:#ccf;fill:#ccf}#mermaid-svg-sv1kSKiaLyKS7oCk .note{stroke:#aa3;fill:#fff5ad}#mermaid-svg-sv1kSKiaLyKS7oCk .noteText,#mermaid-svg-sv1kSKiaLyKS7oCk .noteText>tspan{fill:#000;stroke:none}#mermaid-svg-sv1kSKiaLyKS7oCk .activation0{fill:#f4f4f4;stroke:#666}#mermaid-svg-sv1kSKiaLyKS7oCk .activation1{fill:#f4f4f4;stroke:#666}#mermaid-svg-sv1kSKiaLyKS7oCk .activation2{fill:#f4f4f4;stroke:#666}#mermaid-svg-sv1kSKiaLyKS7oCk .mermaid-main-font{font-family:"trebuchet ms", verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-sv1kSKiaLyKS7oCk .section{stroke:none;opacity:0.2}#mermaid-svg-sv1kSKiaLyKS7oCk .section0{fill:rgba(102,102,255,0.49)}#mermaid-svg-sv1kSKiaLyKS7oCk .section2{fill:#fff400}#mermaid-svg-sv1kSKiaLyKS7oCk .section1,#mermaid-svg-sv1kSKiaLyKS7oCk .section3{fill:#fff;opacity:0.2}#mermaid-svg-sv1kSKiaLyKS7oCk .sectionTitle0{fill:#333}#mermaid-svg-sv1kSKiaLyKS7oCk .sectionTitle1{fill:#333}#mermaid-svg-sv1kSKiaLyKS7oCk .sectionTitle2{fill:#333}#mermaid-svg-sv1kSKiaLyKS7oCk .sectionTitle3{fill:#333}#mermaid-svg-sv1kSKiaLyKS7oCk .sectionTitle{text-anchor:start;font-size:11px;text-height:14px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-sv1kSKiaLyKS7oCk .grid .tick{stroke:#d3d3d3;opacity:0.8;shape-rendering:crispEdges}#mermaid-svg-sv1kSKiaLyKS7oCk .grid .tick text{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-sv1kSKiaLyKS7oCk .grid path{stroke-width:0}#mermaid-svg-sv1kSKiaLyKS7oCk .today{fill:none;stroke:red;stroke-width:2px}#mermaid-svg-sv1kSKiaLyKS7oCk .task{stroke-width:2}#mermaid-svg-sv1kSKiaLyKS7oCk .taskText{text-anchor:middle;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-sv1kSKiaLyKS7oCk .taskText:not([font-size]){font-size:11px}#mermaid-svg-sv1kSKiaLyKS7oCk .taskTextOutsideRight{fill:#000;text-anchor:start;font-size:11px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-sv1kSKiaLyKS7oCk .taskTextOutsideLeft{fill:#000;text-anchor:end;font-size:11px}#mermaid-svg-sv1kSKiaLyKS7oCk .task.clickable{cursor:pointer}#mermaid-svg-sv1kSKiaLyKS7oCk .taskText.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-sv1kSKiaLyKS7oCk .taskTextOutsideLeft.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-sv1kSKiaLyKS7oCk .taskTextOutsideRight.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-sv1kSKiaLyKS7oCk .taskText0,#mermaid-svg-sv1kSKiaLyKS7oCk .taskText1,#mermaid-svg-sv1kSKiaLyKS7oCk .taskText2,#mermaid-svg-sv1kSKiaLyKS7oCk .taskText3{fill:#fff}#mermaid-svg-sv1kSKiaLyKS7oCk .task0,#mermaid-svg-sv1kSKiaLyKS7oCk .task1,#mermaid-svg-sv1kSKiaLyKS7oCk .task2,#mermaid-svg-sv1kSKiaLyKS7oCk .task3{fill:#8a90dd;stroke:#534fbc}#mermaid-svg-sv1kSKiaLyKS7oCk .taskTextOutside0,#mermaid-svg-sv1kSKiaLyKS7oCk .taskTextOutside2{fill:#000}#mermaid-svg-sv1kSKiaLyKS7oCk .taskTextOutside1,#mermaid-svg-sv1kSKiaLyKS7oCk .taskTextOutside3{fill:#000}#mermaid-svg-sv1kSKiaLyKS7oCk .active0,#mermaid-svg-sv1kSKiaLyKS7oCk .active1,#mermaid-svg-sv1kSKiaLyKS7oCk .active2,#mermaid-svg-sv1kSKiaLyKS7oCk .active3{fill:#bfc7ff;stroke:#534fbc}#mermaid-svg-sv1kSKiaLyKS7oCk .activeText0,#mermaid-svg-sv1kSKiaLyKS7oCk .activeText1,#mermaid-svg-sv1kSKiaLyKS7oCk .activeText2,#mermaid-svg-sv1kSKiaLyKS7oCk .activeText3{fill:#000 !important}#mermaid-svg-sv1kSKiaLyKS7oCk .done0,#mermaid-svg-sv1kSKiaLyKS7oCk .done1,#mermaid-svg-sv1kSKiaLyKS7oCk .done2,#mermaid-svg-sv1kSKiaLyKS7oCk .done3{stroke:grey;fill:#d3d3d3;stroke-width:2}#mermaid-svg-sv1kSKiaLyKS7oCk .doneText0,#mermaid-svg-sv1kSKiaLyKS7oCk .doneText1,#mermaid-svg-sv1kSKiaLyKS7oCk .doneText2,#mermaid-svg-sv1kSKiaLyKS7oCk .doneText3{fill:#000 !important}#mermaid-svg-sv1kSKiaLyKS7oCk .crit0,#mermaid-svg-sv1kSKiaLyKS7oCk .crit1,#mermaid-svg-sv1kSKiaLyKS7oCk .crit2,#mermaid-svg-sv1kSKiaLyKS7oCk .crit3{stroke:#f88;fill:red;stroke-width:2}#mermaid-svg-sv1kSKiaLyKS7oCk .activeCrit0,#mermaid-svg-sv1kSKiaLyKS7oCk .activeCrit1,#mermaid-svg-sv1kSKiaLyKS7oCk .activeCrit2,#mermaid-svg-sv1kSKiaLyKS7oCk .activeCrit3{stroke:#f88;fill:#bfc7ff;stroke-width:2}#mermaid-svg-sv1kSKiaLyKS7oCk .doneCrit0,#mermaid-svg-sv1kSKiaLyKS7oCk .doneCrit1,#mermaid-svg-sv1kSKiaLyKS7oCk .doneCrit2,#mermaid-svg-sv1kSKiaLyKS7oCk .doneCrit3{stroke:#f88;fill:#d3d3d3;stroke-width:2;cursor:pointer;shape-rendering:crispEdges}#mermaid-svg-sv1kSKiaLyKS7oCk .milestone{transform:rotate(45deg) scale(0.8, 0.8)}#mermaid-svg-sv1kSKiaLyKS7oCk .milestoneText{font-style:italic}#mermaid-svg-sv1kSKiaLyKS7oCk .doneCritText0,#mermaid-svg-sv1kSKiaLyKS7oCk .doneCritText1,#mermaid-svg-sv1kSKiaLyKS7oCk .doneCritText2,#mermaid-svg-sv1kSKiaLyKS7oCk .doneCritText3{fill:#000 !important}#mermaid-svg-sv1kSKiaLyKS7oCk .activeCritText0,#mermaid-svg-sv1kSKiaLyKS7oCk .activeCritText1,#mermaid-svg-sv1kSKiaLyKS7oCk .activeCritText2,#mermaid-svg-sv1kSKiaLyKS7oCk .activeCritText3{fill:#000 !important}#mermaid-svg-sv1kSKiaLyKS7oCk .titleText{text-anchor:middle;font-size:18px;fill:#000;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-sv1kSKiaLyKS7oCk g.classGroup text{fill:#9370db;stroke:none;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);font-size:10px}#mermaid-svg-sv1kSKiaLyKS7oCk g.classGroup text .title{font-weight:bolder}#mermaid-svg-sv1kSKiaLyKS7oCk g.clickable{cursor:pointer}#mermaid-svg-sv1kSKiaLyKS7oCk g.classGroup rect{fill:#ECECFF;stroke:#9370db}#mermaid-svg-sv1kSKiaLyKS7oCk g.classGroup line{stroke:#9370db;stroke-width:1}#mermaid-svg-sv1kSKiaLyKS7oCk .classLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.5}#mermaid-svg-sv1kSKiaLyKS7oCk .classLabel .label{fill:#9370db;font-size:10px}#mermaid-svg-sv1kSKiaLyKS7oCk .relation{stroke:#9370db;stroke-width:1;fill:none}#mermaid-svg-sv1kSKiaLyKS7oCk .dashed-line{stroke-dasharray:3}#mermaid-svg-sv1kSKiaLyKS7oCk #compositionStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-sv1kSKiaLyKS7oCk #compositionEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-sv1kSKiaLyKS7oCk #aggregationStart{fill:#ECECFF;stroke:#9370db;stroke-width:1}#mermaid-svg-sv1kSKiaLyKS7oCk #aggregationEnd{fill:#ECECFF;stroke:#9370db;stroke-width:1}#mermaid-svg-sv1kSKiaLyKS7oCk #dependencyStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-sv1kSKiaLyKS7oCk #dependencyEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-sv1kSKiaLyKS7oCk #extensionStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-sv1kSKiaLyKS7oCk #extensionEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-sv1kSKiaLyKS7oCk .commit-id,#mermaid-svg-sv1kSKiaLyKS7oCk .commit-msg,#mermaid-svg-sv1kSKiaLyKS7oCk .branch-label{fill:lightgrey;color:lightgrey;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-sv1kSKiaLyKS7oCk .pieTitleText{text-anchor:middle;font-size:25px;fill:#000;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-sv1kSKiaLyKS7oCk .slice{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-sv1kSKiaLyKS7oCk g.stateGroup text{fill:#9370db;stroke:none;font-size:10px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-sv1kSKiaLyKS7oCk g.stateGroup text{fill:#9370db;fill:#333;stroke:none;font-size:10px}#mermaid-svg-sv1kSKiaLyKS7oCk g.statediagram-cluster .cluster-label text{fill:#333}#mermaid-svg-sv1kSKiaLyKS7oCk g.stateGroup .state-title{font-weight:bolder;fill:#000}#mermaid-svg-sv1kSKiaLyKS7oCk g.stateGroup rect{fill:#ECECFF;stroke:#9370db}#mermaid-svg-sv1kSKiaLyKS7oCk g.stateGroup line{stroke:#9370db;stroke-width:1}#mermaid-svg-sv1kSKiaLyKS7oCk .transition{stroke:#9370db;stroke-width:1;fill:none}#mermaid-svg-sv1kSKiaLyKS7oCk .stateGroup .composit{fill:white;border-bottom:1px}#mermaid-svg-sv1kSKiaLyKS7oCk .stateGroup .alt-composit{fill:#e0e0e0;border-bottom:1px}#mermaid-svg-sv1kSKiaLyKS7oCk .state-note{stroke:#aa3;fill:#fff5ad}#mermaid-svg-sv1kSKiaLyKS7oCk .state-note text{fill:black;stroke:none;font-size:10px}#mermaid-svg-sv1kSKiaLyKS7oCk .stateLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.7}#mermaid-svg-sv1kSKiaLyKS7oCk .edgeLabel text{fill:#333}#mermaid-svg-sv1kSKiaLyKS7oCk .stateLabel text{fill:#000;font-size:10px;font-weight:bold;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-sv1kSKiaLyKS7oCk .node circle.state-start{fill:black;stroke:black}#mermaid-svg-sv1kSKiaLyKS7oCk .node circle.state-end{fill:black;stroke:white;stroke-width:1.5}#mermaid-svg-sv1kSKiaLyKS7oCk #statediagram-barbEnd{fill:#9370db}#mermaid-svg-sv1kSKiaLyKS7oCk .statediagram-cluster rect{fill:#ECECFF;stroke:#9370db;stroke-width:1px}#mermaid-svg-sv1kSKiaLyKS7oCk .statediagram-cluster rect.outer{rx:5px;ry:5px}#mermaid-svg-sv1kSKiaLyKS7oCk .statediagram-state .divider{stroke:#9370db}#mermaid-svg-sv1kSKiaLyKS7oCk .statediagram-state .title-state{rx:5px;ry:5px}#mermaid-svg-sv1kSKiaLyKS7oCk .statediagram-cluster.statediagram-cluster .inner{fill:white}#mermaid-svg-sv1kSKiaLyKS7oCk .statediagram-cluster.statediagram-cluster-alt .inner{fill:#e0e0e0}#mermaid-svg-sv1kSKiaLyKS7oCk .statediagram-cluster .inner{rx:0;ry:0}#mermaid-svg-sv1kSKiaLyKS7oCk .statediagram-state rect.basic{rx:5px;ry:5px}#mermaid-svg-sv1kSKiaLyKS7oCk .statediagram-state rect.divider{stroke-dasharray:10,10;fill:#efefef}#mermaid-svg-sv1kSKiaLyKS7oCk .note-edge{stroke-dasharray:5}#mermaid-svg-sv1kSKiaLyKS7oCk .statediagram-note rect{fill:#fff5ad;stroke:#aa3;stroke-width:1px;rx:0;ry:0}:root{--mermaid-font-family: '"trebuchet ms", verdana, arial';--mermaid-font-family: "Comic Sans MS", "Comic Sans", cursive}#mermaid-svg-sv1kSKiaLyKS7oCk .error-icon{fill:#522}#mermaid-svg-sv1kSKiaLyKS7oCk .error-text{fill:#522;stroke:#522}#mermaid-svg-sv1kSKiaLyKS7oCk .edge-thickness-normal{stroke-width:2px}#mermaid-svg-sv1kSKiaLyKS7oCk .edge-thickness-thick{stroke-width:3.5px}#mermaid-svg-sv1kSKiaLyKS7oCk .edge-pattern-solid{stroke-dasharray:0}#mermaid-svg-sv1kSKiaLyKS7oCk .edge-pattern-dashed{stroke-dasharray:3}#mermaid-svg-sv1kSKiaLyKS7oCk .edge-pattern-dotted{stroke-dasharray:2}#mermaid-svg-sv1kSKiaLyKS7oCk .marker{fill:#333}#mermaid-svg-sv1kSKiaLyKS7oCk .marker.cross{stroke:#333} :root { --mermaid-font-family: "trebuchet ms", verdana, arial;} #mermaid-svg-sv1kSKiaLyKS7oCk { color: rgba(0, 0, 0, 0.75); font: ; } ES Cluster Logstash web cluster web1 web2 web3 Elasticsearch Elasticsearch Elasticsearch Elasticsearch Elasticsearch output filter input filebeat apache filebeat apache filebeat apache kibana

logstash安装

购买云主机
主机IP地址配置logstash192.168.1.47最低配置2核2Gweb192.168.1.48最低配置1核1G
web云主机安装
[root@web ~]# yum install -y httpd [root@web ~]# systemctl enable --now httpd [root@web ~]# echo "hello world" >/var/www/html/info.html [root@web ~]# curl http://192.168.1.48/info.html
logstash云主机安装
[root@logstash ~]# vim /etc/hosts 192.168.1.41 es-0001 192.168.1.42 es-0002 192.168.1.43 es-0003 192.168.1.44 es-0004 192.168.1.45 es-0005 192.168.1.47 logstash [root@logstash ~]# yum install -y java-1.8.0-openjdk logstash [root@logstash ~]# touch /etc/logstash/logstash.conf
基础配置样例
[root@logstash ~]# vim /etc/logstash/logstash.conf input { stdin {} } filter{ } output{ stdout{} } [root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
插件与调试格式

使用json格式字符串测试 {“a”:“1”, “b”:“2”,“c”:“3”}

[root@logstash ~]# vim /etc/logstash/logstash.conf input { stdin { codec => "json" } } filter{ } output{ stdout{ codec => "rubydebug" } } [root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf

官方手册地址

https://www.elastic.co/guide/en/logstash/current/index.html

input file插件
[root@logstash ~]# vim /etc/logstash/logstash.conf input { file { path => ["/tmp/c.log"] type => "test" start_position => "beginning" sincedb_path => "/var/lib/logstash/sincedb" } } filter{ } output{ stdout{ codec => "rubydebug" } } [root@logstash ~]# rm -rf /root/.sincedb_* [root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
filter grok插件

正则表达式分组匹配格式: (?<名字>正则表达式)

正则表达式宏调用格式: %{宏名称:名字}

宏文件路径

/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-2.0.5/patterns/grok-patterns

[root@logstash ~]# echo '192.168.1.252 - - [29/Jul/2020:14:06:57 +0800] "GET /info.html HTTP/1.1" 200 119 "-" "curl/7.29.0"' >/tmp/c.log [root@logstash ~]# vim /etc/logstash/logstash.conf input { file { path => ["/tmp/c.log"] type => "test" start_position => "beginning" sincedb_path => "/dev/null" } } filter{ grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } } output{ stdout{ codec => "rubydebug" } } [root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
output elasticsearch插件
[root@logstash ~]# vim /etc/logstash/logstash.conf input { file { path => ["/tmp/c.log"] type => "test" start_position => "beginning" sincedb_path => "/dev/null" } } filter{ grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } } output{ stdout{ codec => "rubydebug" } elasticsearch { hosts => ["es-0001:9200", "es-0002:9200", "es-0003:9200"] index => "weblog" } } [root@logstash ~]# curl -XDELETE http://es-0001:9200/* [root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf

浏览器打开 head 插件，通过 web 页面浏览验证 http://公网IP:9200/_plugin/head/

filebeat配置

web服务安装filebeat
[root@web ~]# yum install -y filebeat [root@web ~]# vim /etc/filebeat/filebeat.yml 15: - /var/log/httpd/access_log 72: document_type: apache_log 183: #注释掉该行 188: #注释掉该行 278： logstash: 280： hosts: ["192.168.1.47:5044"] [root@web ~]# grep -Pv "^\s*(#|$)" /etc/filebeat/filebeat.yml [root@web ~]# systemctl enable --now filebeat
logstash beats插件
[root@logstash ~]# vim /etc/logstash/logstash.conf input { file { path => ["/tmp/c.log"] type => "test" start_position => "beginning" sincedb_path => "/var/lib/logstash/sincedb" } beats { port => 5044 } } filter{ grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } } output{ stdout{ codec => "rubydebug" } elasticsearch { hosts => ["es-0001:9200", "es-0002:9200", "es-0003:9200"] index => "weblog" } } [root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf

访问 web 页面，浏览器打开 head 插件，通过 web 页面浏览验证

网站日志分析实战

1、停止 kibana 服务

[root@kibana ~]# systemctl stop kibana

2、清空 elasticsearch 中所有数据

[root@kibana ~]# curl -XDELETE http://es-0001:9200/*

3、配置 web 日志，获取用户真实IP 通过 ELB 把 web 服务发布公网 https://support.huaweicloud.com/elb_faq/elb_faq_0090.html

[root@web ~]# vim /etc/httpd/conf/httpd.conf #57 行新添加 RemoteIPHeader X-Forwarded-For RemoteIPInternalProxy 100.125.0.0/16 # 修改 198 行 LogFormat "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined [root@web ~]# systemctl restart httpd

4、配置 filebeat 详见配置文件 filebeat.yml 重启服务

[root@web ~]# systemctl restart filebeat

5、配置 logstash 详见配置文件 logstash.conf 启动服务

[root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf

6、配置 kibana 启动服务，通过web页面配置 kibana

[root@kibana ~]# systemctl start kibana
常见错误

使用通配符删除报错

[root@es-0001 ~]# curl -XDELETE http://localhost:9200/* {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Wildcard expressions or all indices are not allowed"}],"type":"illegal_argument_exception","reason":"Wildcard expressions or all indices are not allowed"},"status":400} # 由于设置了destructive_requires_name 参数，不允许使用通配符 # 查看及解决方式 [root@es-0001 ~]# curl -XGET http://es-0001:9200/_cluster/settings?pretty { "persistent" : { "action" : { "destructive_requires_name" : "true" } }, "transient" : { } } [root@es-0001 ~]# curl -XPUT http://localhost:9200/_cluster/settings -d ' { "persistent": { "action": { "destructive_requires_name": "true" } } }' [root@es-0001 ~]# curl -XDELETE http://localhost:9200/* {"acknowledged":true}
最新回复(0)