Add Loki logging stack

README.md

@@ -53,6 +53,7 @@ If you have a spare domain name you can configure applications to be accessible
 * [Komga](https://komga.org/) - a media server for your comics, mangas, BDs and magazines
 * [Krusader](https://krusader.org/) - Twin panel file management for your desktop
 * [Lidarr](https://github.com/lidarr/Lidarr) - Music collection manager for Usenet and BitTorrent users
+* [Loki](https://grafana.com/oss/loki/) - Loki is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus.
 * [Mealie](https://hay-kot.github.io/mealie/) - A self-hosted recipe manager and meal planner
 * [Minecraft Server](https://www.minecraft.net/) - Server edition of the popular building and exploring game
 * [MiniDLNA](https://sourceforge.net/projects/minidlna/) - simple media server which is fully compliant with DLNA/UPnP-AV clients
@@ -79,6 +80,7 @@ If you have a spare domain name you can configure applications to be accessible
 * [Plex](https://www.plex.tv/) - Plex Media Server
 * [Portainer](https://portainer.io/) - for managing Docker and running custom images
 * [Prometheus](https://prometheus.io/) - Time series database and monitoring system (via stats role).
+* [Promtail](https://grafana.com/docs/loki/latest/clients/promtail/) - Promtail is an agent which ships the contents of local logs to a private Grafana Loki instance
 * [Prowlarr](https://github.com/Prowlarr/Prowlarr) - Indexer aggregator for Sonarr, Radarr, Lidarr, etc.
 * [pyLoad](https://pyload.net/) - A download manager with a friendly web-interface
 * [PyTivo](http://pytivo.org) - An HMO and GoBack server for TiVos.
@@ -107,6 +109,12 @@ If you have a spare domain name you can configure applications to be accessible
 * [YouTubeDL-Material](https://github.com/Tzahi12345/YoutubeDL-Material) - Self-hosted YouTube downloader built on Material Design
 * [ZNC](https://wiki.znc.in/ZNC) - IRC bouncer to stay connected to favourite IRC networks and channels
 
+## Preconfigured Application Stacks
+
+Ansible-NAS application [stacks](https://ansible-nas.io/docs/category/stacks/) are a number of applications deployed together and preconfigured to perform a common goal.
+
+* [Logging](https://ansible-nas.io/docs/applications/stacks/logging/) - application logging capture and search service based on Grafana Loki.
+
 ## What This Could Do
 
 Ansible-NAS can run anything that's in a Docker image, which is why Portainer is included. A NAS configuration is a pretty personal thing based on what you download, what media you view, how many photos you take...so it's difficult to please everyone.
@@ -115,7 +123,7 @@ That said, if specific functionality you want isn't included and you think other
 
 ## What This Doesn't Do
 
-Ansible NAS doesn't set up your disk partitions, primarily because getting it wrong can be incredibly destructive. That aside, configuring partitions is usually a one-time (or very infrequent) event, so there's not much to be gained by automating it. Check out the [docs](https://davestephens.github.io/ansible-nas) for recommended setups.
+Ansible NAS doesn't set up your disk partitions, primarily because getting it wrong can be incredibly destructive. That aside, configuring partitions is usually a one-time (or very infrequent) event, so there's not much to be gained by automating it. Check out the [docs](https://ansible-nas.io/docs/) for recommended setups.
 
 ## Installation
 
@@ -138,7 +146,7 @@ Read the [migrating from FreeNAS](https://ansible-nas.io/docs/further-configurat
 
 Getting help is easy! You can:
 
-* Read the [docs](https://davestephens.github.io/ansible-nas)
+* Read the [docs](https://ansible-nas.io/docs/)
 * Start a [discussion](https://github.com/davestephens/ansible-nas/discussions)
 * Raise an [issue](https://github.com/davestephens/ansible-nas/issues) if you think you've found a bug
 * Chat on [Gitter](https://gitter.im/Ansible-NAS/Chat)

nas.yml

@@ -46,6 +46,15 @@
         - ansible-nas-docker
         - ansible-nas
 
+    ###
+    ### Stacks
+    ###
+
+    - role: logging
+      tags:
+        - logging
+
+
     ###
     ### Applications
     ###
@@ -193,6 +202,10 @@
       tags:
         - lidarr
 
+    - role: loki
+      tags:
+        - loki
+
     - role: mealie
       tags:
         - mealie
@@ -297,6 +310,10 @@
       tags:
         - prowlarr
 
+    - role: promtail
+      tags:
+        - promtail
+
     - role: pyload
       tags:
         - pyload

roles/logging/defaults/main.yml

@@ -0,0 +1,2 @@
+---
+logging_stack_enabled: false

roles/logging/tasks/main.yml

@@ -0,0 +1,29 @@
+---
+- name: Start logging stack
+  block:
+    - name: Enable logging roles
+      ansible.builtin.set_fact:
+        grafana_enabled: true
+        minio_enabled: true
+        loki_enabled: true
+        promtail_enabled: true
+  when: logging_stack_enabled is true
+  # vars:
+  #   minio_enabled: true
+  #   loki_enabled: true
+  #   promtail_enabled: true
+
+
+- name: Stop logging stack
+  block:
+    - name: Disable logging roles
+      ansible.builtin.set_fact:
+        grafana_enabled: false
+        minio_enabled: false
+        loki_enabled: false
+        promtail_enabled: false
+  when: logging_stack_enabled is false
+  # vars:
+  #   minio_enabled: false
+  #   loki_enabled: false
+  #   promtail_enabled: false

roles/loki/defaults/main.yml

@@ -0,0 +1,24 @@
+---
+# enable or disable the application
+loki_enabled: false
+loki_available_externally: false
+
+# directories
+loki_data_directory: "{{ docker_home }}/loki"
+
+# network
+loki_hostname: "loki"
+loki_network_name: "loki"
+loki_http_port: "3100"
+
+# docker
+loki_container_name: "loki"
+loki_image_name: "grafana/loki"
+loki_image_version: "latest"
+
+# specs
+loki_memory: "1g"
+
+# config
+loki_log_retention: 672h
+loki_log_level: warn

roles/loki/handlers/main.yml

@@ -0,0 +1,8 @@
+---
+- name: Restart Grafana
+  community.docker.docker_container:
+    name: grafana
+    image: grafana/grafana:latest
+    state: started
+    restart: true
+  listen: "restart grafana"

roles/loki/tasks/main.yml

@@ -0,0 +1,78 @@
+---
+- name: Start Loki
+  block:
+    - name: Check for Minio installation
+      ansible.builtin.fail:
+        msg: "Loki requires Minio enabled and running for storage, please set that up first."
+      when: minio_enabled is false
+
+    - name: Check for Grafana installation
+      ansible.builtin.fail:
+        msg: "Loki requires Grafana enabled and running for visualisation, please set that up first."
+      when: grafana_enabled is false
+
+    - name: Include Stats variables
+      ansible.builtin.include_vars: ../../stats/defaults/main.yml
+
+    - name: Create Loki Directories
+      ansible.builtin.file:
+        path: "{{ item }}"
+        state: directory
+      with_items:
+        - "{{ loki_data_directory }}"
+
+    - name: Create Minio buckets for Loki
+      ansible.builtin.file:
+        path: "{{ item }}"
+        state: directory
+      with_items:
+        - "{{ minio_data_directory }}/data/loki-data"
+        - "{{ minio_data_directory }}/data/loki-ruler"
+
+    - name: Template Loki config
+      ansible.builtin.template:
+        src: config.yml
+        dest: "{{ loki_data_directory }}/config.yml"
+      register: loki_config
+
+    - name: Create loki Docker Container
+      community.docker.docker_container:
+        name: "{{ loki_container_name }}"
+        image: "{{ loki_image_name }}:{{ loki_image_version }}"
+        pull: true
+        command: "-config.file=/etc/loki/config.yml"
+        ports:
+          - "{{ loki_http_port }}:3100"
+          - 7946
+          - 9095
+        volumes:
+          - "{{ loki_data_directory }}/config.yml:/etc/loki/config.yml"
+        restart_policy: unless-stopped
+        memory: "{{ loki_memory }}"
+        restart: "{{ loki_config is changed }}"
+        labels:
+          traefik.enable: "{{ loki_available_externally | string }}"
+          traefik.http.routers.loki.rule: "Host(`{{ loki_hostname }}.{{ ansible_nas_domain }}`)"
+          traefik.http.routers.loki.tls.certresolver: "letsencrypt"
+          traefik.http.routers.loki.tls.domains[0].main: "{{ ansible_nas_domain }}"
+          traefik.http.routers.loki.tls.domains[0].sans: "*.{{ ansible_nas_domain }}"
+          traefik.http.services.loki.loadbalancer.server.port: "3100"
+          prometheus.io/scrape: "true"
+          prometheus.io/port: "3100"
+          prometheus.io/path: "/metrics"
+
+    - name: Template Grafana Loki data source
+      ansible.builtin.template:
+        src: grafana-datasource.yml
+        dest: "{{ stats_grafana_config_directory }}/provisioning/datasources/loki.yml"
+        owner: "472"
+      notify: restart grafana
+  when: loki_enabled is true
+
+- name: Stop loki
+  block:
+    - name: Stop loki
+      community.docker.docker_container:
+        name: "{{ loki_container_name }}"
+        state: absent
+  when: loki_enabled is false

roles/loki/templates/config.yml

@@ -0,0 +1,43 @@
+---
+server:
+  http_listen_port: {{ loki_http_port }}
+  log_level: {{ loki_log_level }}
+
+memberlist:
+
+schema_config:
+  configs:
+    - from: 2021-08-01
+      store: boltdb-shipper
+      object_store: s3
+      schema: v11
+      index:
+        prefix: index_
+        period: 24h
+
+common:
+  path_prefix: /loki
+  replication_factor: 1
+  storage:
+    s3:
+      endpoint: "{{ hostvars[inventory_hostname]['ansible_default_ipv4']['address'] }}:{{ minio_api_port }}"
+      insecure: true
+      bucketnames: loki-data
+      access_key_id: "{{ minio_admin_username }}"
+      secret_access_key: "{{ minio_admin_password }}"
+      s3forcepathstyle: true
+  ring:
+    kvstore:
+      store: memberlist
+
+ruler:
+  storage:
+    s3:
+      bucketnames: loki-ruler
+
+chunk_store_config:
+  max_look_back_period: {{ loki_log_retention }}
+
+table_manager:
+  retention_deletes_enabled: true
+  retention_period: {{ loki_log_retention }}

roles/loki/templates/grafana-datasource.yml

@@ -0,0 +1,12 @@
+---
+apiVersion: 1
+
+datasources:
+  - name: Loki
+    type: loki
+    access: proxy
+    url: http://{{ hostvars[inventory_hostname]['ansible_default_ipv4']['address'] }}:{{ loki_http_port }}
+    jsonData:
+      httpHeaderName1: "X-Scope-OrgID"
+    secureJsonData:
+      httpHeaderValue1: "1"

roles/promtail/defaults/main.yml

@@ -0,0 +1,23 @@
+---
+# enable or disable the application
+promtail_enabled: false
+promtail_available_externally: false
+
+# directories
+promtail_data_directory: "{{ docker_home }}/promtail"
+
+# network
+promtail_hostname: "promtail"
+promtail_network_name: "promtail"
+promtail_port: "9080"
+
+# docker
+promtail_container_name: "promtail"
+promtail_image_name: "grafana/promtail"
+promtail_image_version: "latest"
+
+# specs
+promtail_memory: "1g"
+
+# config
+promtail_loki_url: http://{{ hostvars[inventory_hostname]['ansible_default_ipv4']['address'] }}:{{ loki_http_port }}/loki/api/v1/push

roles/promtail/tasks/main.yml

@@ -0,0 +1,54 @@
+---
+- name: Start Promtail
+  block:
+    - name: Check for Loki installation
+      ansible.builtin.fail:
+        msg: "Promtail requires Loki enabled and running as a write target, please set that up first."
+      when: loki_enabled is false
+
+    - name: Include Loki variables
+      ansible.builtin.include_vars: ../../loki/defaults/main.yml
+
+    - name: Create Promtail Directories
+      ansible.builtin.file:
+        path: "{{ item }}"
+        state: directory
+      with_items:
+        - "{{ promtail_data_directory }}/config"
+        - "{{ promtail_data_directory }}/data"
+
+    - name: Template Promtail config
+      ansible.builtin.template:
+        src: config.yml
+        dest: "{{ promtail_data_directory }}/config/config.yml"
+
+    - name: Create Promtail Docker Container
+      community.docker.docker_container:
+        name: "{{ promtail_container_name }}"
+        image: "{{ promtail_image_name }}:{{ promtail_image_version }}"
+        pull: true
+        command: "-config.file=/etc/promtail/config.yml"
+        ports:
+          - "{{ promtail_port }}:9080"
+        volumes:
+          - "{{ promtail_data_directory }}/config/config.yml:/etc/promtail/config.yml"
+          - "{{ promtail_data_directory }}/data/:/data"
+          - "/var/run/docker.sock:/var/run/docker.sock"
+        restart_policy: unless-stopped
+        memory: "{{ promtail_memory }}"
+        labels:
+          traefik.enable: "{{ promtail_available_externally | string }}"
+          traefik.http.routers.promtail.rule: "Host(`{{ promtail_hostname }}.{{ ansible_nas_domain }}`)"
+          traefik.http.routers.promtail.tls.certresolver: "letsencrypt"
+          traefik.http.routers.promtail.tls.domains[0].main: "{{ ansible_nas_domain }}"
+          traefik.http.routers.promtail.tls.domains[0].sans: "*.{{ ansible_nas_domain }}"
+          traefik.http.services.promtail.loadbalancer.server.port: "9080"
+  when: promtail_enabled is true
+
+- name: Stop promtail
+  block:
+    - name: Stop promtail
+      community.docker.docker_container:
+        name: "{{ promtail_container_name }}"
+        state: absent
+  when: promtail_enabled is false

roles/promtail/templates/config.yml

@@ -0,0 +1,22 @@
+---
+server:
+  http_listen_port: 9080
+  grpc_listen_port: 0
+
+positions:
+  filename: /data/positions.yml
+
+clients:
+  - url: {{ promtail_loki_url }}
+    tenant_id: 1
+
+scrape_configs:
+  - job_name: docker_scrape
+    docker_sd_configs:
+      - host: unix:///var/run/docker.sock
+        refresh_interval: 5s
+    relabel_configs:
+      - source_labels: ['__meta_docker_container_name']
+        regex: '/(.*)'
+        target_label: 'container'
+

website/docs/applications/observability/_category_.json

@@ -0,0 +1,7 @@
+{
+  "label": "Observability",
+  "link": {
+    "type": "generated-index",
+    "description": "All of the observability tooling available for installation with Ansible-NAS."
+  }
+}

website/docs/applications/observability/grafana.md

@@ -0,0 +1,17 @@
+---
+title: "Grafana"
+---
+
+Homepage: <https://grafana.com/>
+
+Docker image: [Grafana](https://hub.docker.com/r/grafana/grafana)
+
+Query, visualize, alert on, and understand your data no matter where it’s stored. With Grafana you can create, explore, and share all of your data through beautiful, flexible dashboards.
+
+## Usage
+
+Set `stats_enabled: true` in your `inventories/<your_inventory>/group_vars/nas.yml` file.
+
+Grafana's web interface can be found at <http://ansible_nas_host_or_ip:3000>.
+
+

website/docs/applications/stacks/_category_.json

@@ -0,0 +1,7 @@
+{
+  "label": "Stacks",
+  "link": {
+    "type": "generated-index",
+    "description": "Application stacks install a number of different apps together to perform a common goal. There are all of the application stacks available for installation with Ansible-NAS."
+  }
+}

website/docs/applications/stacks/logging.md

@@ -0,0 +1,29 @@
+---
+title: "Logging"
+---
+
+The logging stack sets up a fully functional application logging capture and search service based on [Loki](https://grafana.com/oss/loki/), viewable via Grafana.
+
+To enable it, add the following to your `inventories/<your_inventory>/group_vars/nas.yml`:
+
+```
+logging_stack_enabled: true
+```
+
+Which is equivalent to:
+
+```
+minio_enabled: true
+loki_enabled: true
+promtail_enabled: true
+grafana_enabled: true
+```
+
+Once set up, all container stdout logs will be captured and stored. You'll find the Loki data source available in Grafana.
+
+Read more:
+
+  - [Grafana](../observability/grafana.md)
+  - [Loki](../system-tools/loki.md)
+  - [Minio](../system-tools/minio.md)
+  - [Promtail](../system-tools/promtail.md)

website/docs/applications/system-tools/loki.md

@@ -0,0 +1,19 @@
+---
+title: "Loki"
+---
+
+Homepage: <https://grafana.com/oss/loki/>
+
+Docker Container: [Loki](https://hub.docker.com/r/grafana/loki)
+
+Loki is a log aggregation system designed to store and query logs from all your applications and infrastructure.
+
+## Usage
+
+Set `loki_enabled: true` in your `inventories/<your_inventory>/nas.yml` file.
+
+Loki doesn't have a web interface. To see what it's doing look at the container logs from your Ansible-NAS shell:
+
+```
+docker logs loki -f
+```

website/docs/applications/system-tools/minio.md

@@ -1,5 +1,5 @@
 ---
-title: "Minioo"
+title: "Minio"
 ---
 
 Homepage: <https://min.io/>

website/docs/applications/system-tools/promtail.md

@@ -0,0 +1,21 @@
+---
+title: "Promtail"
+---
+
+Homepage: <https://grafana.com/docs/loki/latest/clients/promtail/>
+
+Docker image: [Promtail](https://hub.docker.com/r/grafana/promtail)
+
+Promtail is an agent which ships the contents of local logs to a private Grafana Loki instance or Grafana Cloud. It is usually deployed to every machine that has applications needed to be monitored.
+
+It primarily:
+
+ - Discovers targets
+ - Attaches labels to log streams
+ - Pushes them to the Loki instance.
+
+## Usage
+
+Set `promtail_enabled: true` in your `inventories/<your_inventory>/nas.yml` file.
+
+To see what Promtail is doing (and what containers it has discovered for tailing), visit the web interface at at <http://ansible_nas_host_or_ip:9080>.