使用Logstash简单搜集Nginx Access日志并储存到Elasticsearch

  1. 首先安装Java环境:apt-get install openjdk-8-jdk
  2. 在官网下载Logstash:https://www.elastic.co/cn/downloads/logstash
  3. 在conf目录新建一个配置文件:
input {
    #这里可以同时监控多个文件
    file {
        path => ["/usr/local/nginx/logs/error.log"]
        start_position => "beginning"
        type => "error"
    }
    file {
        path => ["/usr/local/nginx/logs/www.xxx.com.access.log"]
        start_position => "beginning"
        type => "access"
    }
}
filter {
    #每种文件需要配置自己的grok插件语法来搜集需要的数据
    if [type] == "access"{
        grok {
            match => {
                #这里的须发需要自定义配置
                "message" => "^%{IPV4:remote_addr} \[%{HTTPDATE:timestamp}\] \"%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}\" %{INT:status} %{INT:body_bytes_sent} \"%{NOTSPACE:http_referer}\" %{NUMBER:request_time} \"%{IPV4:upstream_addr}:%{POSINT:upstream_port}\" %{NUMBER:upstream_response_time} \"%{DATA:http_user_agent}\" \"%{NOTSPACE:http_x_forwarded_for}\""
            }
        }
        #配置GeoIP的数据库解析ip
        geoip {
            source => "remote_addr"
        }   
    }
}
output {
    # 不满足筛选条件的就不写入
    if "_grokparsefailure" not in [tags] {
        #数据输出到elasticsearch
        elasticsearch {
            hosts => ["127.0.0.1:9200"]
            index => "logstash-nginx-%{type}-%{+YYYY-MM}"
        }
    }
    #调试
    stdout{codec => rubydebug}
}

NginxAccess日志格式:

log_format  main  '$remote_addr [$time_local] "$request" $status $body_bytes_sent "$http_referer" $request_time "$upstream_addr" $upstream_response_time "$http_user_agent" "$http_x_forwarded_for"';

日志示例:

69.126.145.85 [25/Jun/2018:07:31:27 +0000] "POST /api/userInfoRongCloud HTTP/1.1" 200 197 "-" 0.191 "18.191.5.101:9000" 0.191 "dating/1.0.5 (iPhone; iOS 12.0; Scale/3.00)" "-"

这一步的grok语法需要自己调试,在线调试地址:

https://grokdebug.herokuapp.com/

这里可以使用很多已经写好的模式变量,参见:

https://github.com/elastic/logstash/blob/1.4/patterns/grok-patterns
  1. 使用-f参数指定配置文件测试解析结果是否正确
  2. 添加到守护进程中执行:
[program:logstash-worker]
process_name=%(program_name)s_%(process_num)02d
command=/usr/local/logstash-6.3.0/bin/logstash -f /usr/local/logstash-6.3.0/config/nginx-access.conf
autostart=true
autorestart=true
user=root
numprocs=1
redirect_stderr=true
stdout_logfile=/var/log/logstash-worker.log
supervisorctl reread
supervisorctl update
文章来源: 使用Logstash简单搜集Nginx Access日志并储存到Elasticsearch

人吐槽 人点赞

猜你喜欢

发表评论

用户名: 密码:
验证码: 匿名发表

你可以使用这些语言

查看评论:使用Logstash简单搜集Nginx Access日志并储存到Elasticsearch