The surprise

On one nice spring evening, I was away from home and wanted to check some of my old photos. I have my photo library service hosted on my lan, which i access through WirGuard VPN when I’m away. As per usual I launched the WireGuard app on my phone, turned the VPN on, but to my surprise the photo service wasn’t reachable. Weird. I played with it for a while restarted my phone, nothing helped. Ultimately i didn’t think much of it and decided to investigate once I’m back home.

Everything is failing

After a few days, I decided to take a look at what might be causing the issue on my raspberry pi. I tried accessing multiple services that were hosted on the pi, but sadly without any luck. I tried SSHing in the pi - nothing. I rebooted the pi with an HDMI cable plugged in, and noticed a bunch of errors related to storage drive failure. Turns out my SD card was toast, i was able to mount it on my laptop and back everything up, however it would no longer boot up even after formatting and rewriting the image.

The SD card failure didn’t really surprise me as it was running 24/7 for more than a year, however i didn’t want another card dying on me so i bought an SSD in hopes that it will be more reliable than SD card.

Embracing automation

I didn’t have that much going on my pi prior to the failure, however i knew that setting everything up manually is not the way i want to proceed. I decided that this is a perfect opportunity to learn some ansible and although I had to fight the temptation to re-build everything manually, I’m happy i chose the automation way.

Here’s the list of things i wanted to bootstrap using ansible:

  • Install some base packages;
  • Add new user and enabling sudo permissions;
  • Add SSH key for user to connect;
  • Disable SSH password authentication;
  • Update my dotfiles;
  • Install docker;
  • Run the following services in docker:

I decided to separate the tasks into different roles, although i could put everything into a single playbook, i thought using roles would be more beneficial:

  1. I can easily make changes to a single component without impacting the others;
  2. I can re-use the roles in different playbooks;

Another design decision i made was to have templated docker-compose.yml file for each service running in docker, this way i can continue to interact with each docker service using docker-compose command without the need to use ansible. Here’s an example template that i have for pi-hole service:

# {{ ansible_managed }}
---
version: "3"

# More info at https://github.com/pi-hole/docker-pi-hole/ and https://docs.pi-hole.net/
services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    hostname: 'pihole'
    ports:
      - 53:53/tcp
      - 53:53/udp
      - 67:67/udp
      - {{ pihole_web_ui_port }}:80/tcp
      - {{ pihole_https_port }}:443/tcp
    environment:
      TZ: "{{ timezone }}"
      WEBPASSWORD: "{{ pihole_webpass }}"
      ServerIP: "{{ pihole_server_ip }}"
    dns:
      - 127.0.0.1
      - 1.1.1.1
    volumes:
      - {{ docker_home_dir }}/pihole/etc-pihole/:/etc/pihole/
      - {{ docker_home_dir }}/pihole/etc-dnsmasq.d/:/etc/dnsmasq.d/
    cap_add:
      - NET_ADMIN
    restart: unless-stopped

Each template like this must be placed in:

roles/{ROLE NAME}/templates/{FILE NAME}.j2

So in this example the template file is being sourced in:

roles/pihole/templates/docker-compose.yml.j2

Also as some may have noticed the template has multiple variables like:

{{ pihole_web_ui_port }}

I’m sourcing those variables from config.yml file which i have in my projects root dir. This way it’s really easy to make changes, for example if i need to change port for web ui, i can simply modify the pihole_web_ui_port in my config.yml:

---

...
# pihole config
pihole_web_ui_port: 83
pihole_https_port: 9443
pihole_server_ip: 192.168.122.119
pihole_webpass: "password"
...

I am really happy with this design as it allows me to have modular setup where i can easily add more roles or configure the existing ones by interacting mostly with one config file.

Here’s what my final structure of ansible project looks like:

.
└── ansible
    ├── ansible.cfg
    ├── config.default.yml
    ├── config.yml
    ├── inventory.ini
    ├── pi_setup.yml
    └── roles
        ├── init
        │   ├── defaults
        │   │   └── main.yml
        │   ├── files
        │   │   └── keys
        │   │       └── gedas
        │   ├── handlers
        │   │   └── main.yml
        │   └── tasks
        │       └── main.yml
        ├── pihole
        │   ├── handlers
        │   │   └── main.yml
        │   ├── meta
        │   │   └── main.yml
        │   ├── tasks
        │   │   └── main.yml
        │   └── templates
        │       └── docker-compose.yml.j2
        
        ...
        other roles with repeating structure
        ...

I have everything in a github repository, so you can see everything in more details here

Takeaways

I’m glad that my SD card died as i definitely wouldn’t have done all of this otherwise. After i was done writing all of the playbooks it almost felt like magic when i had everything up in the required state by running a single command. It actually felt so good that i even wiped the SSD just to re-run the playbook and have everything back again.

I’m definitely gonna continue and try to have the rest of my infrastructure managed by ansible.

All questions and feedback are welcome