Hardening Debian & Proxmox VE: From Bare Metal to Automated LXC Stacks

Homelab infrastructure commonly evolves organically: an extra mini-PC here, a repurposed workstation there, default passwords left unrotated, and Docker daemons running indiscriminately as root on bare metal. In enterprise deployments, an unhardened hypervisor represents a catastrophic single point of compromise. If an adversary escapes a container or compromises a management daemon, they gain direct DMA access to hypervisor RAM, raw ZFS storage pools, and out-of-band management interfaces.

This guide details the complete, battle-tested hardening playbook employed across DenverNerd production bare-metal nodes running Proxmox Virtual Environment (PVE) 8.x on top of Debian 12 (Bookworm).

graph TD
    Internet[Untrusted External / WAN] -->|UFW Ingress Filter| HostFW[Host Linux Firewall / UFW]
    HostFW -->|Drop Non-Whitelisted| Blackhole[Drop & Log]
    HostFW -->|Port 22 SSH Key-Only| SSHD[Hardened OpenSSH Daemon]
    HostFW -->|Port 8006 TLS 1.3 Only| PVEAPI[PVE Web GUI & REST API]

    subgraph "Bare-Metal Hypervisor (Debian 12 Bookworm Core)"
        ZFS[ZFS rpool & datapool / Native Encryption]
        PBS[Proxmox Backup Server Snapshot Agent]
        Fail2ban[Fail2ban Jail / PVE & SSH log monitors]
    end

    subgraph "Isolated Virtual Networks (Linux Bridges)"
        vmbr0[vmbr0: Public Management Bridge / 10.10.10.0/24]
        vmbr1[vmbr1: Isolated Container Transit / 172.16.20.0/24]
    end

    HostFW --> vmbr0
    vmbr0 --> vmbr1

    subgraph "Workload Isolation Layer"
        LXC1[Unprivileged LXC 101: Traefik / WireGuard]
        LXC2[Unprivileged LXC 102: Rootless Docker Daemon]
        LXC3[Privileged Quarantined VM: TrueNAS Core]
    end

    vmbr1 --> LXC1
    vmbr1 --> LXC2
    vmbr1 --> LXC3

1. Post-Installation Kernel & Package Stream Sanity

Proxmox ships with the enterprise subscription repository enabled by default. On non-subscription homelab clusters, update /etc/apt/sources.list.d/pve-enterprise.list and attach the pve-no-subscription channel immediately to avoid apt update stalls:

# 1. Disable enterprise repository
sed -i 's/^deb/#deb/' /etc/apt/sources.list.d/pve-enterprise.list

# 2. Add community no-subscription channel
cat <<'EOF' > /etc/apt/sources.list.d/pve-no-subscription.list
deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription
EOF

# 3. Synchronize package metadata and elevate packages
apt-get update && apt-get dist-upgrade -y

Removing the Subscription Nag Dialogue

To eliminate the recurring subscription pop-up dialogue across web browser sessions without modifying core PVE binaries, execute this automated sed injection targeting /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js:

sed -Ezi.bak "s/(Ext.Msg.show\(\{\s+title: gettext\('No valid sub)/void\(\{ \/\/\1/g" \
  /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js

systemctl restart pveproxy.service

2. Hardening Host SSH & Disabling Password Authentication

Proxmox clusters require secure root administrative access. Passwords, even complex ones, are vulnerable to credential stuffing, dictionary scans, and memory scraping. Enforce asymmetric Ed25519 authentication with explicit port remapping:

# Generate dedicated operator key on your administrative workstation:
# ssh-keygen -t ed25519 -a 100 -C "operator@denvernerd-homelab"

Configure /etc/ssh/sshd_config.d/99-hardened.conf on the Proxmox host:

# /etc/ssh/sshd_config.d/99-hardened.conf
Port 2222
Protocol 2
PermitRootLogin prohibit-password
PasswordAuthentication no
ChallengeResponseAuthentication no
KerberosAuthentication no
GSSAPIAuthentication no
X11Forwarding no
MaxAuthTries 3
MaxSessions 2
ClientAliveInterval 300
ClientAliveCountMax 2
AllowAgentForwarding no
AllowTcpForwarding no
HostKey /etc/ssh/ssh_host_ed25519_key
KexAlgorithms curve25519-sha256@libssh.org
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com
MACs hmac-sha2-512-etm@openssh.com

Reload the OpenSSH daemon:

sshd -t && systemctl restart sshd

3. Host Firewall & Proxmox Web GUI Ingress Restriction

By default, the Proxmox administration daemon (pveproxy) listens on 0.0.0.0:8006. This exposes the administration interface to every interface, including untrusted networks. Deploy ufw to enforce strict stateful inspection:

apt-get install -y ufw

# Set baseline policies
ufw default deny incoming
ufw default allow outgoing

# Allow hardened SSH
ufw allow 2222/tcp comment "Hardened SSH Daemon"

# Allow local management subnet to Proxmox Web UI
ufw allow from 10.10.10.0/24 to any port 8006 proto tcp comment "PVE Web Console Trusted LAN"

# Allow Corosync cluster heartbeat (if multi-node cluster)
ufw allow in proto udp from 10.10.10.0/24 to any port 5405:5412 comment "Corosync Heartbeat"

# Enable and inspect
ufw --force enable
ufw status verbose

4. Fail2ban Jails for Proxmox GUI & SSH

Even behind restricted subnets, internal hosts can be compromised. Install fail2ban to monitor authentication logs across both the PVE ticket auth endpoint and the SSH daemon.

apt-get install -y fail2ban

Create the custom filter /etc/fail2ban/filter.d/proxmox.conf:

[Definition]
failregex = pvedaemon\[.*authentication failure; rhost=<HOST> user=.* msg=.*
ignoreregex =

Create /etc/fail2ban/jail.d/proxmox-homelab.local:

[DEFAULT]
bantime  = 1h
findtime = 10m
maxretry = 3
banaction = ufw

[sshd]
enabled = true
port    = 2222
logpath = %(sshd_log)s
backend = systemd

[proxmox]
enabled = true
port    = 8006
filter  = proxmox
logpath = /var/log/daemon.log
maxretry = 3
bantime  = 24h

Start and verify fail2ban:

systemctl enable --now fail2ban
fail2ban-client status proxmox

5. Secure Unprivileged LXC ID Remapping

Privileged containers are a relic of early virtualization where user UID 0 inside the container maps directly to UID 0 on the host kernel. If an attacker exploits a containerized process, they possess immediate root execution rights across all storage pools.

Unprivileged LXC containers remap UID 0 in the container to a high unprivileged user ID on the host (typically UID 100000).

Container (UID 0: root)   ──▶   Host Kernel (UID 100000: unprivileged)
Container (UID 33: www)    ──▶   Host Kernel (UID 100033: unprivileged)
Container (UID 1000: user) ──▶   Host Kernel (UID 101000: unprivileged)

Create an unprivileged Debian 12 container via pct:

pct create 101 /var/lib/vz/template/cache/debian-12-standard_12.2-1_amd64.tar.zst \
  --ostype debian \
  --hostname lxc-edge-worker \
  --memory 2048 \
  --swap 512 \
  --cores 2 \
  --net0 name=eth0,bridge=vmbr1,ip=172.16.20.101/24,gw=172.16.20.1 \
  --storage local-zfs \
  --rootfs local-zfs:16 \
  --unprivileged 1 \
  --features nesting=1 \
  --onboot 1

Passing Host ZFS Storage Mounts into Unprivileged Containers

When sharing an external dataset (/datapool/app-data) with container 101, adjust the filesystem permissions on the host so container user UID 1000 (which is host UID 101000) can write to it:

# On Proxmox Host:
chown -R 101000:101000 /datapool/app-data
chmod 770 /datapool/app-data

# Append mount point to /etc/pve/lxc/101.conf:
echo "mp0: /datapool/app-data,mp=/mnt/data" >> /etc/pve/lxc/101.conf

# Start container
pct start 101

6. Running Rootless Docker Inside an Unprivileged LXC

Many homelabbers compromise security by enabling Docker in a privileged container or running Docker as root inside an unprivileged container with unsafe capabilities. The true production standard is running Rootless Docker via fuse-overlayfs:

# Enter container
pct enter 101

# Inside Container:
apt-get update && apt-get install -y \
  ca-certificates \
  curl \
  gnupg \
  fuse-overlayfs \
  uidmap \
  dbus-user-session

# Create non-root application user
useradd -m -s /bin/bash -u 1000 devops
loginctl enable-linger devops

# Switch to devops user
su - devops

# Install Rootless Docker
curl -fsSL https://get.docker.com/rootless | sh

# Configure environment exports in ~/.bashrc
cat <<'EOF' >> ~/.bashrc
export XDG_RUNTIME_DIR="/run/user/$(id -u)"
export DOCKER_HOST="unix:///run/user/$(id -u)/docker.sock"
export PATH="/home/devops/bin:$PATH"
EOF

source ~/.bashrc

# Start rootless daemon as user systemd service
systemctl --user start docker
systemctl --user enable docker

# Verify isolation
docker info | grep "Rootless"

The output confirms:

 Security Options:
  rootless

Now, even if a container running on this Docker engine is compromised via an RCE vulnerability, the breakout process cannot compromise the LXC namespace, let alone the Proxmox bare-metal host.

7. Automated ZFS Snapshots & Pruning

ZFS is the crown jewel of homelab resilience. Snapshots cost almost zero I/O and capture instantaneous copy-on-write states. Implement automated snapshot retention with this lightweight POSIX shell script /usr/local/bin/zfs-snapshot-lifecycle.sh:

cat <<'EOF' > /usr/local/bin/zfs-snapshot-lifecycle.sh
#!/bin/bash
set -euo pipefail

POOL="local-zfs"
PREFIX="auto-snap"
TIMESTAMP=$(date -u +"%Y%m%d_%H%M%S")

echo "[$(date -u)] Initializing ZFS Snapshot Rotation for ${POOL}..."

# 1. Take fresh snapshot recursively
zfs snapshot -r "${POOL}@${PREFIX}-${TIMESTAMP}"

# 2. Prune snapshots older than 7 days
CUTOFF=$(date -u -d "7 days ago" +"%Y%m%d_%H%M%S")

zfs list -H -o name -t snapshot | grep "${POOL}@${PREFIX}-" | while read -r SNAP; do
    SNAP_TIME="${SNAP##*${PREFIX}-}"
    if [[ "${SNAP_TIME}" < "${CUTOFF}" ]]; then
        echo "Pruning expired snapshot: ${SNAP}"
        zfs destroy "${SNAP}"
    fi
done

echo "[$(date -u)] Snapshot lifecycle rotation completed successfully."
EOF

chmod +x /usr/local/bin/zfs-snapshot-lifecycle.sh

Register this script with cron:

cat <<'EOF' > /etc/cron.d/zfs-snapshots
# Run snapshot script every hour at minute 0
0 * * * * root /usr/local/bin/zfs-snapshot-lifecycle.sh >> /var/log/zfs-snapshot.log 2>&1
EOF

8. Offsite Backup Integration: Proxmox Backup Server (PBS)

Local snapshots protect against file deletion, failed container migrations, and bad package upgrades. They do not protect against house fires, power surges that destroy SAS backplanes, or catastrophic water leaks.

Install the proxmox-backup-client on the host to dispatch deduplicated, client-side encrypted backup chunks to an offsite PBS node over an encrypted WireGuard Homelab Tunnel:

# 1. Install PBS client
apt-get install -y proxmox-backup-client

# 2. Generate client-side AES-GCM 256-bit encryption key
proxmox-backup-client key create /etc/pve/backup-encryption.key

# 3. Create backup script for hypervisor configurations
cat <<'EOF' > /usr/local/bin/backup-pve-configs.sh
#!/bin/bash
set -euo pipefail

export PBS_REPOSITORY="backup-user@pbs.homelab.internal:datastore-offsite"
export PBS_PASSWORD="super-secret-pbs-auth-token"
export PBS_ENCRYPTION_KEY="/etc/pve/backup-encryption.key"
export PBS_FINGERPRINT="SHA256:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx"

proxmox-backup-client backup \
  pve-etc.pxar:/etc/pve \
  network.pxar:/etc/network \
  storage.pxar:/etc/pve/storage.cfg \
  --backup-id pve-node-01 \
  --backup-time $(date +%s)

echo "Offsite PBS backup completed successfully."
EOF

chmod 700 /usr/local/bin/backup-pve-configs.sh

Verification & Hardening Audit Checklist

Before declaring your hypervisor production-ready, execute these validation probes:

• SSH Root Password Challenge: ssh -p 2222 root@host prompts strictly for public-key passphrase; password prompts are dropped.
• Fail2ban Trigger Test: Send 3 bogus requests to port 8006 and verify IP is added to ufw status ban list.
• Rootless Docker Namespace Test: Run docker run --rm alpine id inside LXC 101. Verify output returns uid=0(root) gid=0(root), but checking host ps aux reveals the process running under UID 101000.
• ZFS Snapshot Rollback Test: Verify that zfs list -t snapshot shows active hourly snapshots with automated 7-day retention.

This architecture forms the bedrock of the DenverNerd edge engine. Zero-trust principles implemented at the hypervisor foundation guarantee that every container, edge proxy, and background worker operates within an unyielding cryptographic sandbox.