安装ElasticSearch

拉取镜像

docker pull elasticsearch:7.8.0

配置sysctl.conf

vim /etc/sysctl.conf
# 增加以下参数
vm.max_map_count = 262144
sysctl -p

增加es配置文件

mkdir /data/elk/
cd /data/elk/
vim elasticsearch.yml

es配置文件内容

cluster.name: "docker-cluster"
network.host: 0.0.0.0
# 访问ID限定，0.0.0.0为不限制，生产环境请设置为固定IP
transport.host: 0.0.0.0
# elasticsearch节点名称
node.name: node-1
# elasticsearch节点信息
cluster.initial_master_nodes: ["node-1"]
# 下面的配置是关闭跨域验证
http.cors.enabled: true
http.cors.allow-origin: "*"

启动容器

docker run -dit --restart=always -p 9200:9200 -p 9300:9300 -v /data/elk/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml --name es elasticsearch:7.8.0
docker cp -L /usr/share/zoneinfo/Asia/Shanghai es:/etc/localtime

安装IK分词器

前往github下载对应zip插件包：
https://github.com/medcl/elasticsearch-analysis-ik

docker cp elasticsearch-analysis-ik-7.8.0.zip es:/
docker exec -it es bash
mkdir -p /usr/share/elasticsearch/plugins/ik
cd /usr/share/elasticsearch/plugins/ik
mv /elasticsearch-analysis-ik-7.8.0.zip ./
unzip elasticsearch-analysis-ik-7.8.0.zip
rm -rf elasticsearch-analysis-ik-7.8.0.zip

安装Logstash

拉取镜像

docker pull logstash:7.8.0

增加logstash配置文件

docker run -itd --name logstash  -p 5044:5044 logstash:7.8.0
docker cp logstash:/usr/share/logstash/pipeline/logstash.conf logstash.conf
docker cp logstash:/usr/share/logstash/config/logstash.yml logstash.yml
docker stop logstash
docker rm logstash

logstash.yml配置文件内容

http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: [ "http://es_ip:9200" ]

logstash.conf配置文件内容

input {
  beats {
    port => 5044
  }
}

filter {
  if "nginx" in [fields][appid] {
    mutate {
      split => { "message" => " - - " }
      add_field => { "remote_addr" => "%{[message][0]}" }
    }
  }
  if "service" in [fields][apptype] {
    grok {
      match => {
        "message" => "%{TIMESTAMP_ISO8601:createTime} \[%{DATA:thread}\] \[%{DATA:traceId}\] %{LOGLEVEL:level} %{DATA:classPath} - %{DATA:method} - %{DATA:msg}$"
      }
      overwrite => [ "message" ]
    }
    mutate {
      split => { "[log][file][path]" => "/" }
      add_field => { "[fields][appid]" => "%{[log][file][path][4]}" }
      add_field => { "[fields][serviceid]" => "%{[log][file][path][6]}" }
    }
    multiline {
      pattern => "^\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}"
      negate => true
      what => "previous"
    }
    date {
      match => ["createTime", "yyyy-MM-dd HH:mm:ss.SSS"]
      target => "@timestamp"
    }
    ruby {
        code => "event.set('timestamp', event.get('@timestamp').time.localtime - 8*60*60)"
    }
    ruby {
        code => "event.set('@timestamp',event.get('timestamp'))"
    }
    mutate {
        remove_field => ["timestamp"]
    }
  }
}

output {
  stdout {
    codec => rubydebug
  }
  elasticsearch {
    hosts => ["esip:9200"]
    index => ["%{[fields][appid]}-%{+YYYY-MM-dd}"]
  }
}

注意：需要安装插件：

docker exec -it logstash bash
/usr/share/logstash/bin/logstash-plugin install logstash-filter-multiline

启动容器

docker run -dit --restart=always -p 5044:5044 -e "ELASTICSEARCH_HOSTS=http://esip:9200" -v /data/elk/logstash.conf:/usr/share/logstash/pipeline/logstash.conf -v /data/elk/logstash.yml:/usr/share/logstash/config/logstash.yml --name logstash logstash:7.8.0
docker cp -L /usr/share/zoneinfo/Asia/Shanghai logstash:/etc/localtime

安装Kibana

拉取镜像

docker pull kibana:7.8.0

增加kibana配置文件

vim kibana.yml

kibana配置文件内容

server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://esip:9200"]
# 操作界面语言设置
i18n.locale: "zh-CN"

启动容器

docker run -dit --restart=always -p 5601:5601 -v /data/elk/kibana.yml:/usr/share/kibana/config/kibana.yml --name kibana kibana:7.8.0
docker cp -L /usr/share/zoneinfo/Asia/Shanghai kibana:/etc/localtime

安装Filebeat

拉取镜像

docker pull docker.elastic.co/beats/filebeat:7.8.0
docker tag docker.elastic.co/beats/filebeat:7.8.0 filebeat:7.8.0

增加filebeat配置文件

vim filebeat.yml

filebeat配置文件内容

filebeat.inputs:
- type: log
  enable: true
  paths:
    - /var/log/nginx/*.log
  fields:
    appid: nginx-dev


output.logstash:
  hosts: ["192.168.4.20:5044"]

启动容器

docker run --user=root -dit -v /var/log/nginx/:/var/log/nginx/ -v /data/elk/filebeat.yml:/usr/share/filebeat/filebeat.yml -v /var/lib/docker/containers:/var/lib/docker/containers -v /var/run/docker.sock:/var/run/docker.sock --name filebeat filebeat:7.8.0
docker cp -L /usr/share/zoneinfo/Asia/Shanghai filebeat:/etc/localtime

日志换行问题

docker exec -it logstash bash
./logstash-plugin install  logstash-filter-multiline

logstash.conf添加以下配置：

multiline {
  pattern => "^\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}"
  negate => true
  what => "previous"
}

服务filebeat安装及配置

docker pull docker.elastic.co/beats/filebeat:7.8.0
docker tag docker.elastic.co/beats/filebeat:7.8.0 filebeat:7.8.0
docker run --user=root -dit -v /data/deploy/jar:/data/deploy/jar -v /data/filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml -v /var/lib/docker/containers:/var/lib/docker/containers -v /var/run/docker.sock:/var/run/docker.sock --name filebeat filebeat:7.8.0
docker logs -f --tail=20 filebeat

filebeat.yml配置文件：

filebeat.inputs:
- type: log
  enable: true
  paths:
    - /data/deploy/jar/*/logs/*/*.log
  fields:
    apptype: service


output.logstash:
  hosts: ["192.168.4.20:5044"]