Nginx access 日志通过 Filebeat 8.15.5 写入 Elasticsearch 8 实战流程

一、环境准备

1. 安装 Filebeat

# Ubuntu/Debian
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.15.5-amd64.deb
sudo dpkg -i filebeat-8.15.5-amd64.deb

# RHEL/CentOS
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.15.5-x86_64.rpm
sudo rpm -ivh filebeat-8.15.5-x86_64.rpm

2. 安装 Elasticsearch（如果未安装）

# Ubuntu/Debian
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.15.5-amd64.deb
sudo dpkg -i elasticsearch-8.15.5-amd64.deb

# 启动 Elasticsearch
sudo systemctl start elasticsearch
sudo systemctl enable elasticsearch

二、配置 Filebeat

1. 备份原始配置

sudo cp /etc/filebeat/filebeat.yml /etc/filebeat/filebeat.yml.backup

2. 配置 Filebeat（/etc/filebeat/filebeat.yml）

# =========================== Filebeat inputs ============================
filebeat.inputs:
- type: filestream
  id: nginx-access
  paths:
    - /var/log/nginx/access.log
    - /var/log/nginx/access.log.*
  fields:
    log_type: nginx_access
  fields_under_root: true
  tags: ["nginx", "access"]

  # 处理多行日志（如果有错误堆栈）
  multiline:
    pattern: '^\s+(at|\.{3})\b|^Caused by:'
    match: after
    negate: false

# =========================== Processors ============================
processors:
  # 添加主机信息
  - add_host_metadata:
      when.not.contains.tags: forwarded

  # 添加云元数据
  - add_cloud_metadata: ~

  # 删除无用字段
  - drop_fields:
      fields: ["log.offset", "input.type"]

  # 解析 Nginx 访问日志
  - dissect:
      tokenizer: '%{client_ip} - %{remote_user} [%{timestamp}] "%{http_method} %{request} HTTP/%{http_version}" %{status_code} %{body_sent_bytes} "%{http_referer}" "%{user_agent}"'
      field: "message"
      target_prefix: "nginx"

  # 转换数据类型
  - convert:
      fields:
        - {from: "nginx.status_code", type: "integer"}
        - {from: "nginx.body_sent_bytes", type: "integer"}

  # 提取 User-Agent 信息
  - user_agent:
      field: "nginx.user_agent"
      target_field: "user_agent"

  # 提取地理信息（需要 geoip 数据库）
  - geoip:
      database: "/usr/share/filebeat/geoip/GeoLite2-City.mmdb"
      field: "nginx.client_ip"
      target_field: "geo"

  # 添加时间戳
  - timestamp:
      field: "nginx.timestamp"
      layouts:
        - "02/Jan/2006:15:04:05 -0700"
      test:
        - "10/Oct/2024:14:30:00 +0800"

# =========================== Elasticsearch Output ============================
output.elasticsearch:
  hosts: ["localhost:9200"]

  # Elasticsearch 8.x 安全配置
  username: "elastic"
  password: "your_password"
  ssl:
    enabled: true
    certificate_authorities: ["/etc/elasticsearch/certs/http_ca.crt"]

  # 索引配置
  indices:
    - index: "nginx-access-%{+yyyy.MM.dd}"
      when.equals:
        log_type: "nginx_access"

  # 启用 ILM（索引生命周期管理）
  ilm.enabled: true
  ilm.rollover_alias: "nginx-access"
  ilm.pattern: "{now/d}-000001"
  ilm.policy_name: "nginx-access-policy"

# =========================== Setup ============================
setup.ilm.enabled: true
setup.ilm.overwrite: true
setup.template.enabled: true
setup.template.name: "nginx-access"
setup.template.pattern: "nginx-access-*"
setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 1
  index.lifecycle.name: "nginx-access-policy"

三、配置 Elasticsearch

1. 创建 ILM 策略

curl -X PUT "https://localhost:9200/_ilm/policy/nginx-access-policy" \
  -H "Content-Type: application/json" \
  -u "elastic:your_password" \
  --cacert /etc/elasticsearch/certs/http_ca.crt \
  -d '{
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_size": "50gb",
              "max_age": "30d"
            }
          }
        },
        "warm": {
          "min_age": "7d",
          "actions": {
            "forcemerge": {
              "max_num_segments": 1
            }
          }
        },
        "cold": {
          "min_age": "30d",
          "actions": {
            "freeze": {}
          }
        },
        "delete": {
          "min_age": "90d",
          "actions": {
            "delete": {}
          }
        }
      }
    }
  }'

2. 创建索引模板

curl -X PUT "https://localhost:9200/_index_template/nginx-access-template" \
  -H "Content-Type: application/json" \
  -u "elastic:your_password" \
  --cacert /etc/elasticsearch/certs/http_ca.crt \
  -d '{
    "index_patterns": ["nginx-access-*"],
    "template": {
      "settings": {
        "number_of_shards": 1,
        "number_of_replicas": 1,
        "index.lifecycle.name": "nginx-access-policy"
      },
      "mappings": {
        "properties": {
          "@timestamp": { "type": "date" },
          "nginx.client_ip": { "type": "ip" },
          "nginx.remote_user": { "type": "keyword" },
          "nginx.timestamp": { "type": "date", "format": "dd/MMM/yyyy:HH:mm:ss Z" },
          "nginx.http_method": { "type": "keyword" },
          "nginx.request": { "type": "text" },
          "nginx.http_version": { "type": "keyword" },
          "nginx.status_code": { "type": "integer" },
          "nginx.body_sent_bytes": { "type": "long" },
          "nginx.http_referer": { "type": "keyword" },
          "nginx.user_agent": { "type": "text" },
          "user_agent": { "type": "object" },
          "geo": { "type": "geo_point" }
        }
      }
    }
  }'

四、安装和配置 Kibana（可选）

# 安装 Kibana
wget https://artifacts.elastic.co/downloads/kibana/kibana-8.15.5-amd64.deb
sudo dpkg -i kibana-8.15.5-amd64.deb

# 启动 Kibana
sudo systemctl start kibana
sudo systemctl enable kibana

五、启动和测试

1. 测试配置

# 测试配置文件
sudo filebeat test config

# 测试输出到 Elasticsearch
sudo filebeat test output

# 测试解析的字段
sudo filebeat -e -c /etc/filebeat/filebeat.yml \
  -E output.elasticsearch.enabled=false \
  -E output.file.path=/tmp/filebeat-debug.log

2. 加载 Kibana 仪表板

# 加载预定义的仪表板
sudo filebeat setup --dashboards

3. 启动 Filebeat

sudo systemctl start filebeat
sudo systemctl enable filebeat

# 查看状态
sudo systemctl status filebeat

4. 验证数据写入

# 查看 Elasticsearch 索引
curl -X GET "https://localhost:9200/_cat/indices/nginx-access*?v" \
  -u "elastic:your_password" \
  --cacert /etc/elasticsearch/certs/http_ca.crt

# 查看文档数量
curl -X GET "https://localhost:9200/nginx-access-*/_count" \
  -H "Content-Type: application/json" \
  -u "elastic:your_password" \
  --cacert /etc/elasticsearch/certs/http_ca.crt \
  -d '{"query": {"match_all": {}}}'

六、高级配置选项

1. 日志轮转处理

filebeat.inputs:
- type: filestream
  id: nginx-access
  paths:
    - /var/log/nginx/access.log
  # 处理轮转日志
  close.on_state_change.older_than: 1h
  # 忽略旧文件
  ignore_older: 72h
  # 从头开始读取
  harvester_limit: 10

2. 多个 Nginx 实例

filebeat.inputs:
- type: filestream
  id: nginx-web1
  paths:
    - /var/log/nginx/web1/access.log
  fields:
    server: web1
    log_type: nginx_access

- type: filestream
  id: nginx-web2
  paths:
    - /var/log/nginx/web2/access.log
  fields:
    server: web2
    log_type: nginx_access

3. 使用 Logstash 处理（如果需要复杂处理）

output.logstash:
  hosts: ["localhost:5044"]

  # SSL 配置
  ssl.enabled: true
  ssl.certificate_authorities: ["/etc/logstash/certs/ca.crt"]

七、故障排查

1. 查看 Filebeat 日志

sudo journalctl -u filebeat -f
sudo tail -f /var/log/filebeat/filebeat

2. 调试模式

# 前台运行并输出详细日志
sudo filebeat -e -c /etc/filebeat/filebeat.yml -d "*"

3. 常见问题解决

# 1. 权限问题
sudo chown root:root /etc/filebeat/filebeat.yml
sudo chmod 600 /etc/filebeat/filebeat.yml

# 2. 重新加载配置
sudo systemctl reload filebeat

# 3. 重置 Filebeat 状态
sudo rm -rf /var/lib/filebeat/registry/filebeat
sudo systemctl restart filebeat

八、监控和维护

1. 创建监控看板

# 在 Kibana 中创建监控仪表板
# Dev Tools → Console
GET _cat/indices/nginx-access*?v&s=index
GET _cluster/health
GET _nodes/stats/ingest

2. 设置报警规则（Kibana）

{
  "rule_type_id": "logs.alert.document.count",
  "params": {
    "index": ["nginx-access-*"],
    "timeField": "@timestamp",
    "timeWindowSize": 5,
    "timeWindowUnit": "m",
    "threshold": [1000],
    "thresholdComparator": ">"
  }
}

这个配置提供了完整的 Nginx 日志收集解决方案，包括：

• 日志解析和字段提取
• 地理位置信息
• 用户代理分析
• 索引生命周期管理
• 安全连接配置
• 监控和报警设置