环境： 三台Centos7(2G内存、2个处理器、2个内核) es1:redis kibana elasticsearch jdk es2: httpd filebeat elasticsearch jdk es3: httpd filebeat elasticsearch jdk
实验环境： 环境： 两台Centos7(2G内存、2个处理器、2个内核)
es1: jdk elasticsearch logstash kibana redis(192.168.194.130) es2: jdk elasticsearch httpd filebeat(192.168.194.131)
elasticsearch包：
链接: https://pan.baidu.com/s/15_W_Kqfu0Gvk14bFunXNGg 提取码: 4a99 复制这段内容后打开百度网盘手机App，操作更方便哦
jdk包：
链接: https://pan.baidu.com/s/1YXeyNrqLHAQuj7LVQwxepQ 提取码: njmf 复制这段内容后打开百度网盘手机App，操作更方便哦
logstash包：
链接: https://pan.baidu.com/s/1hcRpNOEG5_5RfGgAulD1dA 提取码: jrcg 复制这段内容后打开百度网盘手机App，操作更方便哦
kibana包：
链接: https://pan.baidu.com/s/185sbP5Ey4WkNuh7pNA5nIg 提取码: 5fae 复制这段内容后打开百度网盘手机App，操作更方便哦
filebeat包：
链接: https://pan.baidu.com/s/1QM61Te_Jj7hg5z8KTTfhjg 提取码: 5zqf 复制这段内容后打开百度网盘手机App，操作更方便哦
redis包：
链接: https://pan.baidu.com/s/1F0CIi909rvjE4OTcs5WxEA 提取码: 9icb 复制这段内容后打开百度网盘手机App，操作更方便哦
因为elasticsearch服务运行需要java环境，因此两台elasticsearch服务器需要安装java环境。
[root@localhost ~]# yum -y install jdk-8u131-linux-x64_.rpm配置elasticsearch，elk2配置一个相同的节点，通过组播进行通信，如果无法通过组播查询，修改成单播即可。
[root@localhost ~]# vim /etc/elasticsearch/elasticsearch.yml [root@localhost ~]# cat /etc/elasticsearch/elasticsearch.yml|egrep -v "^#|^$" cluster.name: my-application ##ELK的集群名称，名称相同即属于是同一个集群 node.name: node-1 #本机在集群内的节点名称 path.data: /var/lib/elasticsearch #数据存放目录 path.logs: /var/log/elasticsearch #日志保存目录 network.host: 192.168.194.130 #监听的IP地址 http.port: 9200 #服务监听的端口 discovery.zen.ping.unicast.hosts: ["192.168.194.130", "192.168.194.131"] #单播配置一台即可 [root@localhost elasticsearch]# systemctl start elasticsearch #启动服务2个集群显示为2即为同步
[root@localhost ~]# curl http://192.168.194.130:9200/_cluster/health?pretty=true { "cluster_name" : "my-application", "status" : "green", "timed_out" : false, "number_of_nodes" : 2, "number_of_data_nodes" : 2, "active_primary_shards" : 6, "active_shards" : 12, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 }root@192.168.194.131:/etc/systemd/system
[root@localhost ~]# yum -y install logstash-6.6.0.rpm [root@localhost ~]# cd /etc/logstash/conf.d [root@localhost conf.d]# vim httpd.conf input { redis { data_type => "list" host => "192.168.194.130" password => "123321" port => "6379" db => "1" key => "filebeat-httpd" } } output { elasticsearch { hosts => ["192.168.194.130:9200"] index => "redis-httpdlog-%{+YYYY.MM.dd}" } }没有redis包执行此命令
[root@localhost ~]# wget http://download.redis.io/releases/redis-5.0.0.tar.gz部署redis
[root@localhost ~]# tar xf redis-5.0.0.tar.gz [root@localhost ~]# cp -r redis-5.0.0 /usr/local/redis [root@localhost ~]# cd /usr/local/redis [root@localhost redis-5.0.0]# make distclean #要清除所有生成的文件 [root@localhost redis-5.0.0]# make报错：
解决方法：
[root@localhost redis]# yum -y install gcc-c++ 说关于分配器allocator， 如果有MALLOC 这个 环境变量， 会有用这个环境变量的 去建立Redis。而且libc 并不是默认的 分配器， 默认的是 jemalloc, 因为 jemalloc 被证明 有更少的 fragmentation problems 比libc。但是如果你又没有jemalloc 而只有 libc 当然 make 出错。 所以加这么一个参数 [root@localhost redis]# make报错信息：
报错解决：
注意：报错zmalloc.h:50:31: fatal error: jemalloc/jemalloc.h: No such file or directory改成：make MALLOC=libc参考网页： https://www.cnblogs.com/richerdyoung/p/8066373.html
[root@localhost redis]# vim README.md [root@localhost redis]# make MALLOC=libc [root@localhost redis]# ln -sv /usr/local/redis/src/redis-server /usr/bin/redis-server "/usr/bin/redis-server" -> "/usr/local/redis/src/redis-server" [root@localhost redis]# ln -sv /usr/local/redis/src/redis-cli /usr/bin/redis-cli "/usr/bin/redis-cli" -> "/usr/local/redis/src/redis-cli" [root@localhost conf.d]# cd /usr/local/redis [root@localhost redis]# cp redis.conf redis.conf.bak #备份文件 [root@localhost redis]# vim redis.conf [root@localhost redis]# cat redis.conf|egrep "^bind|^requirepass|^daemonize" bind 192.168.194.130 daemonize yes requirepass 123321 [root@localhost redis]# redis-server ./redis.conf 16810:M 22 Oct 2020 15:37:48.980 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. 解决方法： [root@localhost redis]# echo 511 > /proc/sys/net/core/somaxconn 16810:M 22 Oct 2020 15:37:48.981 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect 解决方法： [root@localhost redis]# vim /etc/sysctl.conf [root@localhost redis]# cat /etc/sysctl.conf|grep vm vm.overcommit_memory = 1 16810:M 22 Oct 2020 15:37:48.981 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. 16810:M 22 Oct 2020 15:37:48.981 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root 解决方法： [root@localhost redis]# sysctl vm.overcommit_memory=1 vm.overcommit_memory = 1 [root@localhost redis]# echo never > /sys/kernel/mm/transparent_hugepage/enabled and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled. 解决方法： [root@localhost redis]# vim /etc/rc.local [root@localhost redis]# cat /etc/rc.local|grep echo echo never > /sys/kernel/mm/transparent_hugepage/enabled因为elasticsearch服务运行需要java环境，因此两台elasticsearch服务器需要安装java环境。
[root@localhost ~]# yum -y install jdk-8u131-linux-x64_.rpm配置elasticsearch，elk2配置一个相同的节点，通过组播进行通信，如果无法通过组播查询，修改成单播即可。
[root@localhost ~]# vim /etc/elasticsearch/elasticsearch.yml [root@localhost ~]# cat /etc/elasticsearch/elasticsearch.yml|egrep -v "^#|^$" cluster.name: my-application ##ELK的集群名称，名称相同即属于是同一个集群 node.name: node-2 #本机在集群内的节点名称 path.data: /var/lib/elasticsearch #数据存放目录 path.logs: /var/log/elasticsearch #日志保存目录 network.host: 192.168.194.131 #监听的IP地址 http.port: 9200 #服务监听的端口 [root@localhost elasticsearch]# systemctl start elasticsearch #启动服务2个集群显示为2即为同步
[root@localhost ~]# curl http://192.168.194.131:9200/_cluster/health?pretty=true { "cluster_name" : "my-application", "status" : "green", "timed_out" : false, "number_of_nodes" : 2, "number_of_data_nodes" : 2, "active_primary_shards" : 6, "active_shards" : 12, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 }filebeat替代logstash收集日志
filebeat简介Filebeat是轻量级单用途的日志收集工具，用于在没有安装java的服务器上专门收集日志，可以将日志转发到logstash、elasticsearch或redis等场景中进行下一步处理。 官网下载地址：https://www.elastic.co/cn/downloads/past-releases#filebeat 官方文档：https://www.elastic.co/guide/en/beats/filebeat/current/configuring-howto-filebeat.html
[root@localhost ~]# yum -y install filebeat-6.8.1-x86_64.rpm [root@localhost ~]# cd /etc/filebeat [root@localhost filebeat]# cp filebeat.yml filebeat.yml.`date +%F` #备份文件登录apache网页(192.168.194.131) 查看是否有日志产生
[root@localhost filebeat]# tailf /tmp/filebeat.txt {"@timestamp":"2020-10-22T08:16:06.740Z","@metadata":{"beat":"filebeat","type":"doc","version":"6.8.1"},"message":"192.168.194.1 - - [22/Oct/2020:16:15:56 +0800] \"GET / HTTP/1.1\" 403 4897 \"-\" \"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36\"","prospector":{"type":"log"},"input":{"type":"log"},"beat":{"name":"localhost.localdomain","hostname":"localhost.localdomain","version":"6.8.1"},"host":{"name":"localhost.localdomain","containerized":false,"architecture":"x86_64","os":{"platform":"centos","version":"7 (Core)","family":"redhat","name":"CentOS Linux","codename":"Core"},"id":"1c6ed991c7e64fdb99475e39fd5092b7"},"source":"/var/log/httpd/access_log","offset":0,"log":{"file":{"path":"/var/log/httpd/access_log"}}}
