# Setting up LXD Lab with Ansible In the world of local infrastructure automation and playbook testing, **Vagrant/VirtualBox** has long been the go-to combo. But what if you could replace all that overhead with something faster, lighter, and closer to real-world Linux systems? That’s exactly what I set out to do with [**lxd-lab**](https://github.com/tylalin/lxd-lab) - that uses **Ansible** and **LXD** to spin up containerised Linux environments for testing and development. No bloated VMs. No GUI. Just containers, automation, and security done right. In a previous post, I shared how I streamlined [**SSH key management in LXD**](https://en.itmatic101.com/virtualisation/ssh-key-management-in-lxd) after migrating from Vagrant. This was a simple solution leveraging `cloud-init`. Later, I delved into the enhanced security of [**SSH certificate authentication**](https://en.itmatic101.com/linux/ssh-certificate-authentication). Despite its robust security, that process involved numerous manual, repetitive steps – particularly when adding new servers or clients. This repetition sparked an idea: automate the entire lab setup and its intricate workflow. My tool of choice for this challenge? **Ansible**. After a couple of weeks of dedicated effort in my downtime, I'm excited to share the results of that automation journey in this article. ## What is lxd-lab? It provides a complete, **Ansible-orchestrated lab environment** built upon **LXC containers**. It utilises a dedicated `lxd_manage` Ansible role to automate the provisioning and configuration of these containers, delivering a highly efficient, secure, and modern workflow. This solution empowers developers and DevOps teams to: * Validate **Ansible playbooks** within a truly representative environment. * Rapidly deploy **ephemeral development and testing systems** without the resource demands of traditional hypervisors like VirtualBox. * Implement **SSH certificate-based authentication**, enhancing security and streamlining connectivity by eliminating the need for manual host key management. ## Upgrading My Workflow: Why LXD + Ansible Outperforms Vagrant For many developers and DevOps professionals, Vagrant paired with VirtualBox has been the go-to for local development environments. However, the overhead of full virtualisation often translates to sluggish performance, high resource consumption, and a less-than-ideal user experience on contemporary Linux desktops. Enter LXD and Ansible – a powerful duo that revolutionises local lab management. This pairing offers significant improvements in speed, efficiency, and security: | Feature | Vagrant/VirtualBox | LXD + Ansible | | -------------------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | **Boot Time** | Slow (Spinning up full virtual machines) | Fast (Nearly instantaneous container starts) | | **Disk/CPU Usage** | High (Each VM consumes dedicated resources) | Low (Containers share the host kernel efficiently) | | **SSH Setup** | Often manual host key acceptance (TOFU) or basic key-based auth | Seamlessly integrated CA-signed certificates (Enhanced security, no manual key acceptance) | | **Native Integration** | (VirtualBox is a hypervisor, not native to Linux's core container tech) | (LXD is built directly on Linux container technology, offering native performance) | | **Automation Flexibility** | Limited to VM-specific commands and guest OS config | Full Ansible power, orchestrating everything from host to container configuration | ## Understanding the lxd-lab Workflow The lxd-lab environment is orchestrated through a combination of key components designed for efficiency and security: ### lxd\_manage Ansible Role: Container Orchestration The custom **lxd\_manage Ansible role** is the engine behind container provisioning. It systematically performs the following actions: * **Container Creation:** Instantiates LXC containers using specified remote images, such as ubuntu:24.04. * **Initial Configuration:** Applies predefined profiles and configures hostnames and network settings. * **Readiness Check:** Waits for cloud-init to fully initialise within the container, ensuring it's ready for subsequent operations. ### Secure Connectivity: SSH Certificate Authentication A cornerstone of this lab's security is its reliance on **SSH Certificate Authentication**. Instead of traditional host key verification, a dedicated **SSH Certificate Authority (CA)** is used to sign all host keys. This method delivers several key benefits: * **Eliminates Manual Fingerprint Confirmation:** You'll never encounter "Trust On First Use" (TOFU) prompts, streamlining initial connections. * **Centralised Trust Management:** Provides a single point of control for issuing, trusting, and revoking SSH host certificates. * **Enhanced Scalability:** Ideal for environments with frequently changing or numerous instances, offering a secure and efficient authentication mechanism for labs, server fleets, and CI/CD pipelines. ## Getting Started We can get up and running in a few steps: * Clone the repository ```bash $ git clone https://github.com/tylalin/lxd-lab.git ``` * View its directory structure ```bash $ cd lxd-lab $ tree . ├── ansible.cfg ├── inventory │   └── hosts.yml ├── LICENSE ├── lxd-lab.yml ├── README.md ├── roles │   └── lxd_manage │   ├── defaults │   │   └── main.yml │   ├── meta │   │   └── main.yml │   ├── tasks │   │   └── main.yml │   └── templates │   ├── hosts.j2 │   └── user-data.yml.j2 └── templates └── hosts.j2 9 directories, 12 files ``` * Here is how the Ansible inventory (`inventory/hosts.yml`) looks like. Of course, all of those variables and IP addresses can be changed to your liking as desired. ```yaml --- all: vars: ssh_ca_dir: ~/ca ssh_ca_key_name: homelab_ssh_ca ssh_ca_key_comment: Homelab SSH CA domain: home.lab user: tyla children: cts: children: servers: vars: host_key_name: ssh_host_rsa_key host_crtkey: "{{ host_key_name }}-cert.pub" hosts: server1: ansible_host: 10.18.34.10 server2: ansible_host: 10.18.34.11 server3: ansible_host: 10.18.34.12 server4: ansible_host: 10.18.34.13 clients: vars: user_key_name: id_rsa user_crtkey: "{{ user_key_name }}-cert.pub" hosts: client1: ansible_host: 10.18.34.20 client2: ansible_host: 10.18.34.21 ``` * The main Ansible playbook (`lxd-lab.yml`) is shown as below. ```yaml --- - name: Manage LXD lab hosts: all connection: local gather_facts: false roles: - lxd_manage - name: Configure /etc/hosts for DNS name resolution hosts: all gather_facts: false tasks: - name: Wait for SSH wait_for_connection: timeout: 180 - name: Render /etc/hosts from template to remote target(s) template: src: hosts.j2 dest: /etc/hosts mode: 0644 - name: Configure SSH CA hosts: localhost gather_facts: false become: true become_user: "{{ user }}" tasks: - name: Ensure CA key directory exists file: path: "{{ ssh_ca_dir }}" state: directory mode: 0700 - name: Generate SSH RSA key pair for CA if not present openssh_keypair: path: "{{ ssh_ca_dir }}/{{ ssh_ca_key_name }}" comment: "{{ ssh_ca_key_comment }}" - name: Create directory structure for all hosts file: path: "{{ ssh_ca_dir }}/{{ item }}" state: directory mode: 0755 loop: "{{ groups['all'] }}" - name: Cleanup SSH CA directory upon destroy file: path: "{{ ssh_ca_dir }}" state: absent tags: [destroy, never] - name: Configure server-side for SSH certificate authentication hosts: servers gather_facts: false tasks: - name: Get ssh_host_rsa_key.pub from server fetch: src: "/etc/ssh/{{ host_key_name }}.pub" dest: "{{ ssh_ca_dir }}/{{ inventory_hostname }}/" flat: yes - name: Copy CA public key to server copy: src: "{{ ssh_ca_dir }}/{{ ssh_ca_key_name }}.pub" dest: "/etc/ssh/" mode: 0644 - name: Sign SSH host key with CA private key command: | ssh-keygen -s {{ ssh_ca_key_name }} -I {{ inventory_hostname }} -V +52w -h -n {{ inventory_hostname }}.{{ domain }} {{ inventory_hostname }}/{{ host_key_name }}.pub args: chdir: "{{ ssh_ca_dir }}" creates: "{{ ssh_ca_dir }}/{{ inventory_hostname }}/{{ host_crtkey }}" delegate_to: localhost become: true become_user: "{{ user }}" - name: Copy signed host public key to server copy: src: "{{ ssh_ca_dir }}/{{ inventory_hostname }}/{{ host_crtkey }}" dest: "/etc/ssh/" mode: 0644 - name: Ensure SSH CA settings are in sshd_config blockinfile: path: /etc/ssh/sshd_config block: | HostKey /etc/ssh/{{ host_key_name }} HostCertificate /etc/ssh/{{ host_crtkey }} TrustedUserCAKeys /etc/ssh/{{ ssh_ca_key_name }}.pub marker: "# {mark} ANSIBLE MANAGED SSH CA CONFIG" notify: Restart SSH handlers: - name: Restart SSH systemd_service: name: ssh state: restarted daemon_reload: true - name: Configure client-side for SSH certificate authentication hosts: clients gather_facts: false tasks: - name: Ensure .ssh directory exists for {{ user }} file: path: "/home/{{ user }}/.ssh" state: directory owner: "{{ user }}" group: "{{ user }}" mode: 0700 - name: Generate SSH RSA key pair for {{ user }} if not present openssh_keypair: path: "/home/{{ user }}/.ssh/{{ user_key_name }}" owner: "{{ user }}" group: "{{ user }}" comment: "{{ user }}@{{ inventory_hostname }}" - name: Get {{ user_key_name }}.pub from client fetch: src: "/home/{{ user }}/.ssh/{{ user_key_name }}.pub" dest: "{{ ssh_ca_dir }}/{{ inventory_hostname }}/" flat: yes - name: Sign SSH user key with CA private key command: | ssh-keygen -s {{ ssh_ca_key_name }} -I {{ inventory_hostname }} -V +52w -n {{ user }} {{ inventory_hostname }}/{{ user_key_name }}.pub args: chdir: "{{ ssh_ca_dir }}" creates: "{{ ssh_ca_dir }}/{{ inventory_hostname }}/{{ user_crtkey }}" delegate_to: localhost become: true become_user: "{{ user }}" - name: Copy signed user public key to client copy: src: "{{ ssh_ca_dir }}/{{ inventory_hostname }}/{{ user_crtkey }}" dest: "/home/{{ user }}/.ssh/" owner: "{{ user }}" group: "{{ user }}" mode: 0644 - name: Ensure CA public key is present in ssh_known_hosts lineinfile: path: /etc/ssh/ssh_known_hosts create: yes line: "@cert-authority *.{{ domain }} {{ lookup('file', lookup('env', 'HOME') + '/ca/homelab_ssh_ca.pub') }}" state: present ``` Given the self-documenting nature of Ansible playbooks, I'll skip a line-by-line explanation of each play and task. ## Standing Up ### Prerequisites * Ubuntu 24.04 LTS * LXD installed (with snapd) and properly setup * Ansible (version - 2.16.3) * Git to clone my GitHub repository - [**lxd-lab**](https://github.com/tylalin/lxd-lab) ### Ansible Magic Run the following command to run Ansible playbook `lxd-lab.yml`. ```bash $ ansible-playbook lxd-lab.yml PLAY [Manage LXD lab] ************************************************************************************************************************************************************************ TASK [lxd_manage : Create LXD instance(s)] *************************************************************************************************************************************************** changed: [server1] changed: [server2] changed: [client2] changed: [server3] changed: [server4] changed: [client1] TASK [lxd_manage : Update localhost /etc/hosts file for DNS name resolution] ***************************************************************************************************************** ok: [server1] PLAY [Configure /etc/hosts for DNS name resolution] ****************************************************************************************************************************************** TASK [Wait for SSH] ************************************************************************************************************************************************************************** ok: [server4] ok: [server1] ok: [server3] ok: [server2] ok: [client2] ok: [client1] TASK [Render /etc/hosts from template to remote target(s)] *********************************************************************************************************************************** changed: [client1] changed: [client2] changed: [server4] changed: [server2] changed: [server1] changed: [server3] PLAY [Configure SSH CA] ********************************************************************************************************************************************************************** TASK [Ensure CA key directory exists] ******************************************************************************************************************************************************** changed: [localhost] TASK [Generate SSH RSA key pair for CA if not present] *************************************************************************************************************************************** changed: [localhost] TASK [Create directory structure for all hosts] ********************************************************************************************************************************************** changed: [localhost] => (item=server1) changed: [localhost] => (item=server2) changed: [localhost] => (item=server3) changed: [localhost] => (item=server4) changed: [localhost] => (item=client1) changed: [localhost] => (item=client2) PLAY [Configure server-side for SSH certificate authentication] ****************************************************************************************************************************** TASK [Get ssh_host_rsa_key.pub from server] ************************************************************************************************************************************************** changed: [server3] changed: [server4] changed: [server1] changed: [server2] TASK [Copy CA public key to server] ********************************************************************************************************************************************************** changed: [server2] changed: [server1] changed: [server4] changed: [server3] TASK [Sign SSH host key with CA private key] ************************************************************************************************************************************************* changed: [server2 -> localhost] changed: [server1 -> localhost] changed: [server4 -> localhost] changed: [server3 -> localhost] TASK [Copy signed host public key to server] ************************************************************************************************************************************************* changed: [server1] changed: [server3] changed: [server2] changed: [server4] TASK [Ensure SSH CA settings are in sshd_config] ********************************************************************************************************************************************* changed: [server3] changed: [server2] changed: [server4] changed: [server1] RUNNING HANDLER [Restart SSH] **************************************************************************************************************************************************************** changed: [server2] changed: [server3] changed: [server1] changed: [server4] PLAY [Configure client-side for SSH certificate authentication] ****************************************************************************************************************************** TASK [Ensure .ssh directory exists for tyla] ************************************************************************************************************************************************* ok: [client1] ok: [client2] TASK [Generate SSH RSA key pair for tyla if not present] ************************************************************************************************************************************* changed: [client1] changed: [client2] TASK [Get id_rsa.pub from client] ************************************************************************************************************************************************************ changed: [client1] changed: [client2] TASK [Sign SSH user key with CA private key] ************************************************************************************************************************************************* changed: [client2 -> localhost] changed: [client1 -> localhost] TASK [Copy signed user public key to client] ************************************************************************************************************************************************* changed: [client1] changed: [client2] TASK [Ensure CA public key is present in ssh_known_hosts] ************************************************************************************************************************************ changed: [client1] changed: [client2] PLAY RECAP *********************************************************************************************************************************************************************************** client1 : ok=9 changed=7 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 client2 : ok=9 changed=7 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 localhost : ok=3 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 server1 : ok=10 changed=8 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 server2 : ok=9 changed=8 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 server3 : ok=9 changed=8 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 server4 : ok=9 changed=8 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 ``` ### Verification Process Let's verify if all those LXC containers are actually up and running with `lxc list` command first. ```bash $ lxc list +---------+---------+--------------------+-----------------------------------------------+-----------+-----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +---------+---------+--------------------+-----------------------------------------------+-----------+-----------+ | client1 | RUNNING | 10.18.34.20 (eth0) | fd42:2751:df65:31e1:216:3eff:fee3:7ec8 (eth0) | CONTAINER | 0 | +---------+---------+--------------------+-----------------------------------------------+-----------+-----------+ | client2 | RUNNING | 10.18.34.21 (eth0) | fd42:2751:df65:31e1:216:3eff:fe36:ec14 (eth0) | CONTAINER | 0 | +---------+---------+--------------------+-----------------------------------------------+-----------+-----------+ | server1 | RUNNING | 10.18.34.10 (eth0) | fd42:2751:df65:31e1:216:3eff:fe6d:f4ac (eth0) | CONTAINER | 0 | +---------+---------+--------------------+-----------------------------------------------+-----------+-----------+ | server2 | RUNNING | 10.18.34.11 (eth0) | fd42:2751:df65:31e1:216:3eff:fec6:ccd3 (eth0) | CONTAINER | 0 | +---------+---------+--------------------+-----------------------------------------------+-----------+-----------+ | server3 | RUNNING | 10.18.34.12 (eth0) | fd42:2751:df65:31e1:216:3eff:feb2:e242 (eth0) | CONTAINER | 0 | +---------+---------+--------------------+-----------------------------------------------+-----------+-----------+ | server4 | RUNNING | 10.18.34.13 (eth0) | fd42:2751:df65:31e1:216:3eff:fe86:de3c (eth0) | CONTAINER | 0 | +---------+---------+--------------------+-----------------------------------------------+-----------+-----------+ ``` To verify the SSH certificate authentication workflow, go through the following process. ```bash # SSH into client1 $ ssh client1 Warning: Permanently added 'client1' (ED25519) to the list of known hosts. Welcome to Ubuntu 24.04.2 LTS (GNU/Linux 6.8.0-60-generic x86_64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/pro System information as of Thu Jun 5 11:22:40 UTC 2025 System load: 0.89 Usage of /: 2.6% of 18.96GB Memory usage: 0% Swap usage: 0% Temperature: 34.0 C Processes: 22 Users logged in: 0 IPv4 address for eth0: 10.18.34.20 IPv6 address for eth0: fd42:2751:df65:31e1:216:3eff:fee3:7ec8 Expanded Security Maintenance for Applications is not enabled. 0 updates can be applied immediately. Enable ESM Apps to receive additional future security updates. See https://ubuntu.com/esm or run: sudo pro status The list of available updates is more than a week old. To check for new updates run: sudo apt update tyla@client1:~$ # From client1 or client2, SSH into server1, server2, server3 or server4 as below tyla@client1:~$ ssh server1.home.lab Welcome to Ubuntu 24.04.2 LTS (GNU/Linux 6.8.0-60-generic x86_64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/pro System information as of Thu Jun 5 11:24:31 UTC 2025 System load: 0.57 Usage of /: 2.6% of 18.96GB Memory usage: 0% Swap usage: 0% Temperature: 35.0 C Processes: 22 Users logged in: 0 IPv4 address for eth0: 10.18.34.10 IPv6 address for eth0: fd42:2751:df65:31e1:216:3eff:fe6d:f4ac Expanded Security Maintenance for Applications is not enabled. 0 updates can be applied immediately. Enable ESM Apps to receive additional future security updates. See https://ubuntu.com/esm or run: sudo pro status The list of available updates is more than a week old. To check for new updates run: sudo apt update tyla@server1:~$ ``` As you can see, there is no TOFU prompt upon the first login attempt and SSH authentication is successful with SSH certificates. ## Teardown After you have done with homelab testing and development works, it's also quite easy to clean up the environment with the following command. ```bash $ ansible-playbook lxd-lab.yml -t destroy PLAY [Manage LXD lab] ************************************************************************************************************************************************************************ TASK [lxd_manage : Remove LXD instance(s)] *************************************************************************************************************************************************** changed: [server1] changed: [client2] changed: [client1] changed: [server4] changed: [server3] changed: [server2] PLAY [Configure /etc/hosts for DNS name resolution] ****************************************************************************************************************************************** PLAY [Configure SSH CA] ********************************************************************************************************************************************************************** TASK [Cleanup SSH CA directory upon destroy] ************************************************************************************************************************************************* changed: [localhost] PLAY [Configure server-side for SSH certificate authentication] ****************************************************************************************************************************** PLAY [Configure client-side for SSH certificate authentication] ****************************************************************************************************************************** PLAY RECAP *********************************************************************************************************************************************************************************** client1 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 client2 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 localhost : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 server1 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 server2 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 server3 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 server4 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 $ lxc list +------+-------+------+------+------+-----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +------+-------+------+------+------+-----------+ ``` ## Elevated Security: Leveraging an SSH Certificate Authority This is a **game-changer** for lab security, often overlooked in many home setups. Instead of letting Ansible blindly trust new hosts or forcing you to manually manage host keys, this configuration implements a robust SSH Certificate Authority (CA): * **Your CA signs each container's SSH host key.** * The **CA's public key is distributed to all clients.** * **`/etc/ssh/ssh_known_hosts` is configured with `@cert-authority`**, establishing a trusted chain. The outcome? **No more annoying fingerprint prompts**, **no risk of Man-in-the-Middle (MITM) surprises**, and an overall **cleaner, more secure SSH experience.** ## Real-World Use Cases `lxd-lab` isn't just for tinkering; it's a powerful tool for practical applications: * **Develop and test Ansible roles** efficiently, without the overhead of full virtual machines. * **Validate playbooks** in a controlled, repeatable lab environment, ensuring consistency. * Leverage it as a **lightweight Continuous Integration (CI) testing environment** for rapid feedback. * **Replace heavy Vagrant/VirtualBox + VM setups** for basic infrastructure simulation and proof-of-concept work. ## Final Thoughts If you're ready to modernise your local development workflow and move beyond the overhead of traditional virtual machines, **`lxd-lab`** is your answer. It's a minimal, automated, and secure environment, built with the same robust tools you likely already use in production. With `lxd-lab`, you'll discover how effortlessly containers can become your new, more efficient VMs.