27 grudnia 2021

Portainer #7 - Grafana Loki

Centrum zbierani logów będzie kontener z aplikacją Loki. Będzie ona źródłem logów dla Grafana.

Aplikacja Loki

version: '3.8'

networks:
  grafana-net:
    name: grafana-net
    driver: bridge

services:
  grafana:
    container_name: grafana
    image: grafana/grafana:latest
    networks: [grafana-net]
    ports: ['3000:3000']
    depends_on: [prometheus, loki]
    environment: {}
    volumes:
      - grafana_storage:/var/lib/grafana
    
  loki:
    container_name: loki
    image: grafana/loki:2.4.1
    networks: [grafana-net]
    ports: [3100:3100]
    command: -config.file=/etc/loki/loki-config.yaml
    volumes:
      - all_config:/etc/loki


  prometheus:
    container_name: prometheus
    image: prom/prometheus
    networks: [grafana-net]
    ports: [9090:9090]
    command:
      - '--config.file=/config/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      #- '--storage.tsdb.retention=200h'
      - '--web.enable-lifecycle'
    volumes:
      - all_config:/config
      - prometheus_storage:/prometheus
      

volumes:
  grafana_storage:
    driver: local
    name: grafana_storage
  prometheus_storage:
    driver: local
    name: prometheus_storage
  all_config:
    driver: local
    name: all_config

Konfiguracja Loki loki-config.yaml

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

Przekierowanie logów z docker

Konfiguracja Docker PC

sudo docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions
docker plugin ls
sudo nano /etc/docker/daemon.json
sudo systemctl restart docker

Zawartość pliku /etc/docker/daemon.json.

{
    "log-driver": "loki",
    "log-opts": {
        "loki-url": "http://portainer-pc.lan:3100/loki/api/v1/push",
        "loki-batch-size": "400"
    }
}

Promtail

Jeśli chcemy monitorować systemowe pliki logów lub logi dokera to możemy uruchomić agenta, który monitoruje wybrany katalog i przesyła logi do Loki.

  promtail:
    image: grafana/promtail:2.4.0
    container_name: promtail
    ports: [1514:1514, 9080:9080]
    command: -config.file=/etc/promtail/promtail-config.yml
    volumes:
      - /var/log:/var/log
      - /var/lib/docker:/var/lib/docker
      - all_config:/etc/promtail

Konfiguracja Promtail promtail-config.yml

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://portainer-pc.lan:3100/loki/api/v1/push

scrape_configs:

  - job_name: local
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/*log

  - job_name: docker
    pipeline_stages:
      - docker: {}
    static_configs:
      - labels:
          job: docker
          __path__: /var/lib/docker/containers/*/*-json.log

  # syslog target

  - job_name: syslog
    syslog:
      listen_address: 0.0.0.0:1514 # make sure you also expose this port on the container
      idle_timeout: 60s
      label_structured_data: yes
      labels:
        job: "syslog"
    relabel_configs:
      - source_labels: ['__syslog_message_hostname']
        target_label: 'host'

Sprawdzenie

Sprawdzenie, czy Loki i Promtail działają poprawnie:

http://portainer-pc.lan:3100/metrics
http://portainer-pc.lan:3100/ready
http://portainer-pc.lan:9080/targets

Linki

============================================================