92 is half of 99

Proxmox Best Practice Part 4 – Security: Protecting systems from various threats

System updates and patch management: The Foundation of Security

Why updates are so critical

Outdated software is one of the most common gateways for attackers. Every day, new vulnerabilities (CVEs) are discovered and published – an unpatched system is like a house with open doors and windows. In addition, Proxmox is a complex virtualization platform that has direct access to the hardware resources. A successfully compromised hypervisor means that attackers can gain full access to all virtual machines and containers running on it. This worst case should be avoided at all costs.

The biggest risks of unpatched systems are remote code execution (RCE) vulnerabilities, privilege escalation attacks or even known exploit frameworks such as Metasploit, which automatically search for vulnerable systems. It becomes particularly critical when zero-day exploits become public – administrators often only have hours or days before automated attacks start.

Repository configuration: Stability vs. timeliness

Choosing the right repository is a classic conflict between security and stability. Proxmox offers different repository options, each with different pros and cons.

Enterprise Repository – The gold standard for production environments:

The Enterprise Repository goes through a multi-stage quality assurance process. Updates are first tested internally, then validated in controlled environments and only released after extensive testing. This minimizes the risk of updates themselves leading to system failures – a scenario that can be devastating in critical production environments.

The main disadvantage, of course, is not the cost factor but the delayed availability of security updates. In extreme cases, critical patches may take weeks to become available. However, for organizations with strict compliance requirements or critical production systems, this delay may also be an acceptable trade-off for additional stability.

#v8 # Enable Enterprise Repository cat > /etc/apt/sources.list.d/pve-enterprise.list << 'EOF' deb https://enterprise.proxmox.com/debian/pve bookworm pve-enterprise EOF

#v9 #File #/etc/apt/sources.list.d/pve-enterprise.sources Types: deb URIs: https://enterprise.proxmox.com/debian/pve Suites: trixie Components: pve-enterprise Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg

No-Subscription Repository – The balancing act between cost and risk:

The free repository gets updates faster, but these go through less rigorous testing. This is where so-called ‘regression bugs’ can occur – updates that damage existing functionalities or introduce new security vulnerabilities. For homelab environments or test systems, this is usually acceptable, as the failure is not business-critical.

A particular risk lies in the fact that the No-Subscription Repository sometimes includes experimental features. These can lead to unexpected security flaws or instabilities. Administrators must pay particular attention here and first validate updates in test environments. A tradeoff that pays off, however, especially with ZeroDay gaps. For this reason, the no-sub repo is definitely available faster when it is on fire.

v8 # Enable No-Subscription Repository cat > /etc/apt/sources.list.d/pve-no-subscription.list << 'EOF' deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription EOF # Default Debian repositories for security updates cat > /etc/apt/sources.list << 'EOF' deb http://deb.debian.org/debian bookworm main contrib non-free non-free-firmware deb http://security.debian.org/debian-security bookworm-security main contrib non-free non-free-firmware deb http://deb.debian.org/debian bookworm-updates main contrib non-free non-free-firmware EOF

v9 # Enable No-Subscription Repository Types: deb deb-src URIs: http://deb.debian.org/debian/ Suites: trixie trixie-updates Components: main non-free-firmware Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg Types: deb deb-src URIs: http://security.debian.org/debian-security/ Suites: trixie-security Components: main non-free-firmware Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg

Automatic security updates: Comfort vs. control

Automatic updates are a double-edged sword. On the one hand, they significantly reduce the window for attacks – most successful system compromises are based on vulnerabilities for which patches were already available. On the other hand, automatic updates can lead to unplanned outages if an update is incompatible with existing configurations.

The biggest risk of automatic updates lies in the lack of control over the time. An update could crash an important application during critical business hours. This is particularly problematic for kernel updates that require a reboot or for updates of critical services such as the Proxmox Cluster Stack.

The configuration presented here therefore also takes a rather conservative approach: Only security updates are installed automatically, and automatic restarts are disabled. This provides a good compromise between security and control.

# Install and configure unattended-upgrades apt update && apt install -y unattended-upgrades apt-listchanges # Create detailed configuration cat > /etc/apt/apt.conf.d/50unattended-upgrades << 'EOF' // Automatic updates only for security patches Unattended-Upgrade::Origins-Pattern { "origin=Debian,codename=${distro_codename},label=Debian-Security"; "origin=Proxmox,codename=${distro_codename}"; }; // Restart important system services after updates Unattended upgrade::Automatic reboot "false"; Unattended-Upgrade::Automatic-Reboot-WithUsers "false"; Unattended upgrade::Remove Unused kernel packages "true"; Unattended-Upgrade::Remove-Unused-Dependencies "true"; // Notifications and Logging Unattended upgrade::Mail "admin@example.com"; Unattended-Upgrade::MailReport "on-change"; Unattended upgrade::AutoFixInterruptedDpkg "true"; Unattended upgrade::MinimalSteps "true"; // Debugging in case of problems Unattended upgrade::Debug "false"; EOF # enable automatic updates cat > /etc/apt/apt.conf.d/20auto-upgrades << 'EOF' APT::Periodic::Update-Package-Lists "1"; APT::Periodic::Unattended upgrade "1"; APT::Periodic::AutocleanInterval "7"; EOF # Start and activate service systemctl enable --now unattended-upgrades

Important considerations for automatic updates:

Email notifications are essential – administrators need to be notified of installed updates to be able to respond quickly to issues. The option MailReport "on-change" ensures that emails are sent only in the event of actual changes, which reduces spam.

Automatic removal of unused packages (Remove-Unused-Dependencies "true") reduces the attack surface, but in rare cases can lead to unexpected problems if dependencies are detected incorrectly. Be sure to keep an eye on it!

Firewall configuration: Defense in Depth

The concept of multi-layered defence

A single firewall layer is no longer sufficient in modern IT environments. The defense-in-depth principle requires multiple layers of security that complement and secure each other. At Proxmox, we have three natural firewall layers: Datacenter (cluster-wide), node (host-specific) and VM/container (guest-specific).

The advantage of this approach lies in redundancy: Even if one layer is compromised or misconfigured, the other layers still provide protection. At the same time, it allows granular control – different VMs can have different security policies without compromising cluster-wide security.

Datacenter-level firewall: The outer protective wall

The data center firewall acts as the first line of defense and defines basic security policies for the entire cluster. Here we implement the ‘default deny’ principle: Everything is blocked by default, only explicitly allowed traffic is allowed through.

Attacks that are prevented:

  • Port scanning by external attackers
  • Unauthorized access to management interfaces
  • Lateral movement between clusters
  • Data Exfiltration via Unusual Ports

Potential disadvantages: The standard restrictive policy can cause connectivity issues when adding new services. Administrators must deliberately create new rules, which can be time-consuming at first. In case of misconfiguration, administrators can lock themselves out, so firewall changes should always be made using a second access route (e.g. IPMI/iLO). 

# Enable datacenter firewall via CLI pvesh set /cluster/firewall/options --enable 1 # Set default policy (Drop by default, explicitly allow) pvesh set /cluster/firewall/options --policy_in DROP pvesh set /cluster/firewall/options --policy_out ACCEPT # pvesh create /cluster/firewall/rules --pos 0 --action ACCEPT --type in --proto tcp --dport 22 --comment "SSH Management" pvesh create /cluster/firewall/rules --pos 1 --action ACCEPT --type in --proto tcp --dport 8006 --comment "Proxmox Web Interface" # VNC/SPICE for VM consoles (optional, only if required) pvesh create /cluster/firewall/rules --pos 2 --action ACCEPT --type in --proto tcp --dport 5900:5999 --comment "VNC Console Access" # Cluster communication (only between known nodes) pvesh create /cluster/firewall/rules --pos 3 --action ACCEPT --source 192.168.1.0/24 --type in --proto tcp --dport 22000:22050 --comment "Cluster Communication" # ICMP for Network Troubleshooting pvesh create /cluster/firewall/rules --pos 4 --action ACCEPT --type in --proto icmp --comment "ICMP Ping"

Special considerations on the VNC rule: VNC access (ports 5900-5999) allows console access to VMs, but is also a potential security risk. VNC traffic is unencrypted by default and can be tapped. In production environments, VNC should only be used via VPN or with additional TLS encryption.

Node-level firewall: Specialized security

Each Proxmox node can have specific roles – storage node, compute node, backup node, etc. The node-level firewall allows these roles to be reflected in security policies.

Safety benefits:

  • Isolation of node-specific services
  • Granular control over storage access
  • Better containment when compromising individual nodes

Complexity risks: As the number of nodes increases, firewall management becomes complex. Inconsistent rules between nodes can lead to network problems that are difficult to diagnose. Central documentation and automated deployment processes become essential.

# Enable node-specific firewall pvesh set /nodes/$(hostname)/firewall/options --enable 1 # Node-specific rules (example for backup servers) pvesh create /nodes/$(hostname)/firewall/rules --action ACCEPT --type in --proto tcp --dport 8007 --comment "Proxmox Backup Server" # Storage access (NFS, iSCSI, etc.) pvesh create /nodes/$(hostname)/firewall/rules --action ACCEPT --source 192.168.1.0/24 --type in --proto tcp --dport 2049 --comment "NFS Storage"

VM-level firewall: Micro-segmentation

The VM-level firewall implements microsegmentation – each VM is treated as a separate security zone. This is especially important in multi-tenant environments or in the isolation of critical applications.

Protection against lateral movement: When an attacker compromises a VM, the VM-level firewall prevents the automatic spread to other systems. This is one of the most effective protection mechanisms against modern APT (Advanced Persistent Threats) attacks.

Performance considerations: Each firewall rule causes additional CPU load. In environments with many VMs and complex control sets, this can affect performance. Proxmox's firewall engine is optimized, but with hundreds of VMs each with dozens of rules, this can be felt.

# Enable VM firewall (for VM ID 100) qm set 100 --firewall 1 # Web server rules pvesh create /nodes/$(hostname)/qemu/100/firewall/rules --action ACCEPT --type in --proto tcp --dport 80 --comment "HTTP" pvesh create /nodes/$(hostname)/qemu/100/firewall/rules --action ACCEPT --type in --proto tcp --dport 443 --comment "HTTPS" pvesh create /nodes/$(hostname)/qemu/100/firewall/rules --action DROP --type in --comment "Deny all other traffic"

Advanced authentication: More than just passwords

The Problem with Password-Based Authentication

Passwords alone are completely inadequate in today's threat landscape. Most successful breaches start with compromised credentials, whether through phishing, credential stuffing, or brute force attacks. With a system like Proxmox that allows full administrative access to critical infrastructure, the impact of a compromised account is devastating.

Modern attackers use sophisticated techniques: They use lists of billions of previously leaked passwords (credential stuffing), deploy AI-powered attacks, and have access to specialized hardware for brute force attacks. A single password – even a complex one – is unfortunately powerless against these threats, at least in the long term.

Multi-factor authentication: The game changer

Multi-factor authentication (MFA) is one of the most effective security measures available. Even if a password is compromised, an attacker cannot access without the second factor. Proxmox VE supports TOTP (Time-based One-Time Passwords) that work with apps like Google Authenticator, Authy or andUOTP.

Safety benefits:

  • Protection from 99.9% All automated attacks
  • Warning of compromise (failed MFA attempts)
  • Compliance compliance (many standards require MFA)

Operational challenges: MFA can reduce usability and complicate initial setup processes. If the MFA device is lost, users can lock themselves out. Emergency codes or alternative authentication methods are essential, but once again increase complexity.

# Enable 2FA for users (via web interface or CLI) # Datacenter → Authentication → Two Factor # Example: Configure TOTP for users pvesh create /access/tfa --userid admin@pam --type totp --description "Admin TOTP Token"

Central user management: LDAP and Active Directory

In larger organizations, decentralized user management is a security risk. Employees change roles, leave the company, or need temporary access - all these changes must be reflected in all systems in a timely manner. Central user management solves this problem with single sign-on (SSO) and central rights control.

Safety benefits:

  • Immediate blocking of personnel changes
  • Uniform password policies
  • Central auditing and compliance
  • Reduced number of accounts to manage

Implementation risks: Central user management creates a single point of failure. If the LDAP/AD server fails, users will no longer be able to log in. Network partitioning between Proxmox and the authentication server causes the same problems. A local emergency account with strong authentication is therefore essential. Some of you may know it as an "Oh-Shit Account" whose token/password/smartcard is in a safe. ⁇

# LDAP integration via CLI pvesh create /access/domains --realm company.local --type ldap \ --server ldap.company.local --port 636 --secure 1 \ --base-dn "dc=company,dc=local" \ --user-attr sAMAccountName \ --bind-dn "cn=proxmox,ou=service-accounts,dc=company,dc=local" # Alternatively: Active Directory Integration pvesh create /access/domains --realm company.local --type ad \ --server ad.company.local --port 636 --secure 1 \ --domain company.local

Role-based access control: The Principle of Minimum Permission

The principle of least privilege is fundamental to secure systems. Users should only be given the rights they need for their tasks – no more, no less. In Proxmox, the role-based system allows very granular control over permissions.

Typical role definitions:

  • VM Operator: Can start/stop/manage VMs but not create new ones
  • Storage admin: Full access to storage configuration, but no VM rights
  • Monitor user: Read-only access for monitoring and reporting
  • Backup operator: Can create and restore backups

Complexity management: As the size of the organization increases, role management becomes complex. Role-to-role dependencies, temporary permissions, and changing job roles require regular reviews. Clear documentation and automated provisioning processes are essential.

# Create custom role for VM operators pvesh create /access/roles --roleid VMOperator \ --privs "VM.Allocate,VM.Config.Disk,VM.Config.Memory,VM.Console,VM.PowerMgmt,VM.Monitor" # Restricted role for monitoring pvesh create /access/roles --roleid Monitor \ --privs "Sys.Audit,VM.Audit,Datastore.Audit" # User role assignment with path restriction pvesh create /access/acl --path /vms/100 --users operator@company.local --roles VMOperator pvesh create /access/acl --path /nodes/proxmox1 --users monitoring@company.local --roles Monitor

SSH hardening: The Achilles heel of many systems

Why SSH is the main target

SSH (Secure Shell) is enabled by default on most Linux systems and provides direct shell access with root permission. For attackers, it is therefore the most attractive target: Successfully compromised SSH access often means complete system control.

Most SSH attacks occur via:

  • Brute force attacks against weak passwords
  • Credential stuffing with leaked password databases
  • Exploitation of known SSH vulnerabilities
  • Man-in-the-middle attacks on unencrypted connections
  • Social Engineering to Obtain SSH Keys

Statistics show the reality: Unprotected SSH services are attacked within minutes of being online. Automated botnets continuously scan the Internet for open SSH ports and immediately launch attacks with the most common usernames and passwords.

Comprehensive SSH security configuration

The default SSH configuration is optimized for compatibility, not security. A hardened configuration eliminates most attack vectors, but requires careful planning to avoid locking out administrators.

Critical security measures explain:

Root login ban: Direct root login is a massive security risk. Attackers know the root username and only need to crack the password. With PermitRootLogin no Attackers must know both a valid username and password and then perform privilege escalation.

Public key authentication: SSH keys are cryptographically much stronger than passwords. A 2048-bit RSA key is roughly equivalent to a 256-character password. Keys cannot be cracked by brute force, but are vulnerable to theft when stored insecurely.

Algorithm limitations: Older SSH algorithms have known vulnerabilities. The configuration is limited to modern, secure algorithms such as Curve25519 and ChaCha20-Poly1305.

# Backup of the original configuration cp /etc/ssh/sshd_config /etc/ssh/sshd_config.backup # Create secure SSH configuration cat > /etc/ssh/sshd_config << 'EOF' # Protocol and Encryption Protocol 2 Port 22 #Port 2222 # Alternative: Use non-standard port # HostKeys (secure algorithms only) HostKey /etc/ssh/ssh_host_ed25519_key HostKey /etc/ssh/ssh_host_rsa_key # Cryptographic settings KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group16-sha512 Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr MACs hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,ac-sha2-256,hmac-sha2122-5126-etm@openssh.com # Authentication PermitRootLogin no PubkeyAuthentication yes PasswordAuthentication no PermitEmptyPasswords no ChallengeResponseAuthentication no UsePAM yes # Authorized Keys Configuration PubkeyAcceptedKeyTypes ssh-ed25519,rsa-sha2-256,rsa-sha2-512 AuthorizedKeysFile .ssh/authorized_keys # Session Management MaxAuthTries 3 MaxStartups 3:30:10 LoginGraceTime 60 ClientAliveInterval 300 ClientAliveCountMax 2 MaxSessions 2 # AllowUsers admin@192.168.1.0/24 operator@192.168.1.0/24 # AllowGroups ssh-users # Logging and Monitoring SyslogFacility AUTH LogLevel VERBOSE # Security features StrictModes yes IgnoreRhosts yes HostbasedAuthentication no PermitUserEnvironment no AllowAgentForwarding no AllowTcpForwarding no X11Forwarding no PrintMotd no TCPKeepAlive no Compression no EOF # Test SSH configuration sshd -t && systemctl reload sshd

Important considerations for SSH curing:

Port change: Changing the SSH port from 22 to a non-standard port significantly reduces automated attacks. However, it is security-by-obscurity and does not provide protection against targeted attacks. Port scanners find alternative ports quickly.

Timeout settings: ClientAliveInterval 300 and ClientAliveCountMax 2 Close inactive connections after 10 minutes. This prevents attackers from hijacking long-term connections, but can be disruptive for administrators with long maintenance tasks.

Forwarding restrictions: AllowTcpForwarding no and AllowAgentForwarding no prevent SSH from being misused as a tunnel for other services. However, this can hinder desired administrative tasks, such as secure database connections via SSH tunnels.

Token-based SSH authentication: The gold standard

The most secure method for SSH access is the exclusive use of SSH keys (tokens) combined with the complete deactivation of password authentication. This method eliminates virtually all brute force attacks and provides significantly higher security than even the strongest passwords.

Why token authentication is superior:

SSH keys are based on asymmetric cryptography with key lengths of 2048-4096 bits (RSA) or modern elliptic curve algorithms such as Ed25519. A 2048-bit RSA key is roughly equivalent to a 617-digit password – a complexity that is virtually impossible to crack with brute force. Ed25519 keys are even safer and much more powerful.

Attacks that are prevented:

  • Credential stuffing: Leaked password databases are worthless
  • Phishing: SSH Keys Cannot Be Captured by Social Engineering
  • Keylogger: Malware on client systems cannot record keys
  • Brute force: Mathematically impossible with correct key length
  • Rainbow table attacks: Pre-computed attacks don't work against keys

Operational security considerations:

The biggest disadvantage of token authentication is key management. If an administrator loses his private key, he is permanently locked out. At the same time, the private key must be kept absolutely secure – a compromised key provides immediate full access.

Best practice is the use of hardware security modules (HSMs) or at least password-protected private keys. YubiKeys or similar hardware tokens provide additional security as the private key never leaves the device.

# Secure SSH key generation (on the client system) # Ed25519 - modern, secure curve ssh-keygen -t ed25519 -C "admin@company.com" -f ~/.ssh/proxmox_ed25519 # Alternatively: RSA with 4096 bit (for older systems) ssh-keygen -t rsa -b 4096 -C "admin@company.com" -f ~/.ssh/proxmox_rsa # Protect key with strong passphrase (recommended) ssh-keygen -t ed25519 -C "admin@company.com" -f ~/.ssh/proxmox_ed25519 -N "$(openssl rand -base64 32)" # Transfer public key to Proxmox server ssh-copy-id -i ~/.ssh/proxmox_ed25519.pub admin@proxmox-server.local # Alternatively: Manually upload the public key cat ~/.ssh/proxmox_ed25519.pub  ⁇  ssh admin@proxmox-server.local "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys && chmod 700 ~/.ssh"

Advanced key management strategies:

For production environments, SSH keys should be managed centrally. This allows for fast locking of compromised keys and simplifies rotation processes.

# Central key management with authorized_keys options cat > /home/admin/.ssh/authorized_keys << 'EOF' # Emergency Admin Key - only from management network from="192.168.100.0/24",no-port-forwarding,no-X11-forwarding ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILx... emergency-admin@company.com # Standard Admin Key - temporary from="192.168.1.0/24",no-port-forwarding,command="/usr/local/bin/admin-shell" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAbc... admin@company.com # Monitoring Key - only specific commands from="192.168.1.100",no-pty,no-port-forwarding,command="/usr/bin/monitoring-script" ssh-rsa AAAAB3NzaC1yc2EAAAA... monitoring@company.com EOF chmod 600 /home/admin/.ssh/authorized_keys

authorized_keys options explained:

  • from="IP/Network": Limits key usage to specific source IPs
  • command="command": Executes only the specified command, ignores client requests
  • no-port-forwarding: Prevents SSH tunneling
  • no-pty: Prevents interactive terminal sessions
  • no-X11-forwarding: Disables X11 forwarding

Network-based SSH access control: Defense in Depth

Even with perfect SSH configuration, it makes sense to restrict access at the network level. This provides an additional layer of security and significantly reduces the attack surface.

Geographical restrictions:

If administrators only access from specific countries or regions, geographic IP blocks can be implemented. This stops most automated attacks, as they often come from certain countries.

Advantages of network segmentation:

  • Reduced attack surface: SSH is only available for authorized networks
  • Early detection: Accesses from unauthorized networks are immediately suspicious
  • Compliance: Many frameworks require network segmentation
  • Performance: Fewer connection attempts reduce CPU load

Disadvantages and challenges:

  • Mobility: Remote work is made more difficult
  • Emergencies: Access from unexpected networks can be blocked
  • Dynamic IPs: ISP changes can lock out administrators
  • Maintenance effort: IP lists need to be maintained
# 1. Firewall-based SSH restriction (iptables/nftables) # Only management network allow iptables -A INPUT -p tcp --dport 22 -s 192.168.100.0/24 -j ACCEPT iptables -A INPUT -p tcp --dport 22 -s 10.0.0.0/8 -j ACCEPT # Specific Administrator IPs iptables -A INPUT -p tcp --dport 22 -s 203.0.113.10 -j ACCEPT # Admin Home Office iptables -A INPUT -p tcp --dport 22 -s 198.51.100.50 -j ACCEPT # Backup admin # Reject all other SSH connections iptables -A INPUT -p tcp --dport 22 -j DROP # 2. Proxmox Firewall integration # About web interface: Node → Firewall → Add Rule # Or via CLI: pvesh create /nodes/$(hostname)/firewall/rules --type in --action ACCEPT \ --proto tcp --dport 22 --source 192.168.100.0/24 --comment "Management SSH" pvesh create /nodes/$(hostname)/firewall/rules --type in --action ACCEPT \ --proto tcp --dport 22 --source 203.0.113.10 --comment "Admin Home Office" pvesh create /nodes/$(hostname)/firewall/rules --type in --action DROP \ --proto tcp --dport 22 --comment "Block all other SSH"

VPN-based SSH access: The ultimate security solution

For maximum security, SSH should only be accessible via VPN connections. This combines the benefits of network segmentation with strong encryption and authentication.

VPN SSH Architecture Benefits:

  • Double encryption: VPN + SSH offer redundant encryption
  • Central authentication: VPN server can force MFA
  • Audit trail: All accesses are via controlled VPN gateway
  • Emergency disconnect: VPN connection can be disconnected centrally

Implementation options:

# 1. OpenVPN-based solution # Allow SSH only via VPN interface iptables -A INPUT -p tcp --dport 22 -i tun0 -j ACCEPT # VPN interface iptables -A INPUT -p tcp --dport 22 -i lo -j ACCEPT # Loopback iptables -A INPUT -p tcp --dport 22 -j DROP # All others # 2. WireGuard-based solution (modern alternative) # Only WireGuard peers allow SSH iptables -A INPUT -p tcp --dport 22 -s 10.0.0.0/24 -j ACCEPT # WireGuard net iptables -A INPUT -p tcp --dport 22 -j DROP # 3. IPSec-based solution (Enterprise) # SSH only via IPSec tunnel iptables -A INPUT -p tcp --dport 22 -m policy --dir in --pol ipsec -j ACCEPT iptables -A INPUT -p tcp --dport 22 -j DROP

Geographic IP Restrictions: GeoIP integration

For organizations with clearly defined geographic locations, GeoIP-based restrictions can be implemented. This automatically blocks access from unexpected countries.

Implementation with GeoIP:

# Install GeoIP databases apt install -y geoip-database-contrib xtables-addons-common # GeoIP modules load modprobe xt_geoip # Allow only certain countries (ISO codes) iptables -A INPUT -p tcp --dport 22 -m geoip --src-cc DE,AT,CH -j ACCEPT # DACH region iptables -A INPUT -p tcp --dport 22 -m geoip ! --src-cc DE,AT,CH -j DROP # Block the rest # Logging for blocked countries iptables -A INPUT -p tcp --dport 22 -m geoip ! --src-cc DE,AT,CH -j LOG --log-prefix "SSH-GeoBlock: "

Dynamic SSH access control: Port knocking

Port knocking can be implemented for particularly safety-critical environments. SSH is blocked by default and only opens after a specific sequence of network packets.

Port knocking advantages:

  • Stealth mode: SSH port is invisible for port scans
  • Additional authentication: Knock sequence acts as pre-authentication
  • Automatic timeouts: Access is closed again after a defined time

Disadvantages:

  • Complexity: Significantly increases administrative complexity
  • Single Point of Failure: Incorrectly configured knocking locks everyone out
  • Debugging: Network problems are harder to diagnose
# knockd Installation and configuration apt install -y knockd # knockd Configuration cat > /etc/knockd.conf << 'EOF' [options] UseSyslog [openSSH] sequence = 7000,8000,9000 seq_timeout = 5 command = /sbin/iptables -A INPUT -s %IP% -p tcp --dport 22 -j ACCEPT tcpflags = syn [closeSSH] sequence = 9000,8000,7000 seq_timeout = 5 command = /sbin/iptables -D INPUT -s %IP% -p tcp --dport 22 -j ACCEPT tcpflags = syn EOF # knockd Activate service systemctl enable --now knockd # Client-side usage knock proxmox-server.local 7000 8000 9000 # Open SSH ssh admin@proxmox-server.local # SSH connection knock proxmox-server.local 9000 8000 7000 # Close SSH

SSH Session Monitoring and Alerting

In addition to access restrictions, all SSH sessions should be monitored and suspicious activity reported.

# Extend SSH session logging cat >> /etc/ssh/sshd_config << 'EOF' # Detailed Logging LogLevel VERBOSE SyslogFacility AUTH # Session recording (if legally allowed) ForceCommand /usr/local/bin/ssh-session-logger EOF # Session-Logger Script (example) cat > /usr/local/bin/ssh-session-logger << 'EOF' #!/bin/bash SESSION_ID=$(date +%Y%m%d_%H%M%S)_$ LOG_DIR="/var/log/ssh-sessions" mkdir -p "$LOG_DIR" # Session information log echo "$(date): SSH Session Start - User: $USER, From: $SSH_CLIENT, TTY: $SSH_TTY" >> "$LOG_DIR/session_$SESSION_ID.log" # Start original shell (with optional recording) if [ -n "$SSH_ORIGINAL_COMMAND" ]; then exec $SSH_ORIGINAL_COMMAND otherwise exec $SHELL fi EOF chmod +x /usr/local/bin/ssh-session-logger

These advanced SSH security measures provide multi-layered defense against various attack vectors. The combination of token authentication, network segmentation and monitoring creates a robust security architecture that is remarkably resilient to even sophisticated attacks.

Intrusion detection: Recognizing the Enemy

Fail2Ban – The automatic bouncer

Fail2Ban is an intrusion prevention system that monitors log files in real time and automatically blocks IP addresses in case of suspicious activity. It is particularly effective against brute force attacks as it locks out attackers after a few failed attempts.

Functionality and benefits: Fail2Ban analyzes log files with regular expressions (Regex) and detects attack patterns. For SSH, these are failed login attempts, for Proxmox Web-Interface Authentication-Failures. Detected attackers are blocked for a configurable time, which stops most automated attacks.

Limitations and disadvantages: Fail2Ban is reactive – the first attack attempt is always successful. For example, attackers also use distributed attacks from many IP addresses, keeping them below detection thresholds. And let's not forget: Fail2Ban can also lock out legitimate users if they enter incorrect passwords multiple times.

# Fail2Ban install apt update && apt install -y fail2ban # Create Proxmox-specific configuration cat > /etc/fail2ban/jail.local << 'EOF' [DEFAULT] # Basic settings bantime = 3600 findtime = 600 maxretry = 3 backend = systemd ignoreip = 127.0.0.1/8 192.168.1.0/24 # Email Notifications destemail = admin@example.com sendername = Fail2Ban mta = sendmail action = %(action_mwl)s # SSH Protection [sshd] enabled = true mode = aggressive port = ssh logpath = %(sshd_log)s banaction = iptables-multiport maxretry = 2 findtime = 3600 bantime = 86400 # Proxmox Web Interface Protection [proxmox] enabled = true port = https,http,8006 filter = proxmox backend = systemd maxretry = 3 findtime = 3600 bantime = 3600 EOF # Create Proxmox filter cat > /etc/fail2ban/filter.d/proxmox.conf << 'EOF' [Definition] failregex = pvedaemon\[.*authentication failure; rhost=<HOST> user=.* msg=.* ignoreregex = EOF # Fail2Ban start and activate systemctl enable --now fail2ban # Check status fail2ban-client status

Configuration optimizations explains:

Bantime strategy: A one-hour lockdown (bantime = 3600) is sufficient for most automated attacks. Longer locks can become problematic if legitimate users are accidentally locked. SSH uses a 24-hour lock, as SSH attacks are usually more targeted.

Escalation policy: The configuration uses progressive locks – SSH attacks lead to longer locks than web interface attacks, as SSH allows direct system access and is therefore more critical.

Whitelist management: The ignoreip-Setting prevents internal IP addresses from being blocked. This is important for administrative networks, but can also be exploited by attackers who are already on the internal network.

System monitoring with Auditd: The digital forensic scientist

Auditd (Linux Audit Daemon) is the official Linux subsystem for security auditing. It logs system calls, file accesses, process executions, and other kernel-level security-related events. These logs are essential for forensic analysis, compliance detection, and advanced persistent threat (APT) detection.

Why Auditd is especially important at Proxmox: Proxmox manages critical infrastructure - VMs, storage, network configurations. A compromised hypervisor can affect dozens or hundreds of guest systems. Auditd helps detect suspicious activity before it can spread.

Compliance aspects: Many compliance frameworks (PCI-DSS, HIPAA, SOX) require detailed audit logs. Auditd can meet these requirements, but also generates significant amounts of data. Attention: A typical system can produce gigabyte-by-gigabyte audit logs daily.

Performance impact: Auditd works at the kernel level and can affect system performance. Especially I/O-intensive workloads can become noticeably slower. The configuration presented here focuses on safety-critical events in order to minimize the performance impact.

# Install Audited apt install -y auditd audispd-plugins # Create audit rules for Proxmox cat > /etc/audit/rules.d/proxmox.rules << 'EOF' # Monitoring of Proxmox configuration files -w /etc/pve/ -p wa -k proxmox-config -w /etc/systemd/system/pve*.service -p wa -k proxmox-services # Monitor VM/container operations -w /usr/bin/qm -p x -k vm-management -w /usr/bin/pct -p x -k container-management # Monitor privileged commands -a always,exit -F arch=b64 -S execve -F euid=0 -k root-commands -a always,exit -F arch=b32 -S execve -F euid=0 -k root-commands # Monitor network configuration -w /etc/network/interfaces -p wa -k network-config -a always,exit -F arch=b64 -S socket -F success=1 -k network-socket EOF # Systemctl enable --now auditd augenrules --load

Audit rules in detail:

File monitoring (-w): These rules monitor changes to critical configuration files. /etc/pve/ contains the entire cluster configuration, /etc/network/interfaces the network settings. Every change is logged, including the triggering process and user.

System call monitoring (-a always,exit): These rules monitor specific system calls. execve is called at each program execution – monitoring of root executions (euid=0) Helps detect privilege escalation attacks.

Log management challenges: Audit logs grow rapidly and can deplete storage space. Automatic rotation and archiving is essential. At the same time, the logs must be protected from manipulation – a compromised administrator account could try to blur its tracks.

TLS/SSL certificates: Trust in an untrustworthy world

The Problem with Self-Signed Certificates

Proxmox automatically creates self-signed TLS certificates for the web interface and cluster communication during installation. While they provide encryption, they do not provide authentication – users cannot verify that they are actually communicating with the real Proxmox server.

Security risks of self-signed certificates:

  • Man-in-the-middle attacks: Attackers can create fake certificates and pose as Proxmox servers
  • Certificate Pinning Impossible: Clients can't safely distinguish between real and fake certificates
  • User behavior: The constant browser warnings cause users to ignore security warnings

Compliance issues: Many security policies and compliance frameworks prohibit self-signed certificates in production environments. PCI-DSS, for example, explicitly requires trusted certificates for all publicly accessible services.

Implementing Trusted Certificates

There are several options for internal services: own Certificate Authority (CA), Let’s Encrypt for publicly accessible services, or commercial certificates for maximum compatibility.

Let’s Encrypt Integration: Let’s Encrypt offers free, automatically renewed certificates. Proxmox has built-in ACME support that automates the entire process. The disadvantage: Let’s Encrypt requires public DNS resolution or HTTP accessibility, which is not always possible or desired.

Own CA for internal services: A separate Certificate Authority provides complete control and also works in isolated networks. However, the CA root certificates must be installed on all client systems, which becomes complex in large environments.

# Let's Encrypt certificate for Proxmox web interface # First configure ACME plugin (via web interface: Datacenter → ACME) # Alternatively: Include your own CA certificate # 1. Copy certificate and key to /etc/ssl/certs/ # 2. About web interface: Node → Certificates → Upload Custom Certificate # TLS configuration for pveproxy harden cat >> /etc/default/pveproxy << 'EOF' # TLS security settings CIPHER_SUITE="ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256" PROTOCOL="TLSv1.2,TLSv1.3" EOF systemctl restart pveproxy

TLS configuration hardening explains:

Cipher Suite limitation: The configuration is limited to modern, secure encryption algorithms. Older ciphers such as RC4, 3DES or MD5-based algorithms are known to be weak and excluded.

Protocol restriction: TLS 1.0 and 1.1 have known vulnerabilities and should no longer be used. Limiting to TLS 1.2 and 1.3 eliminates these risks, but can cause compatibility issues with very old clients.

Backup security: Protection from the worst

Why Backup Encryption is Essential

Backups contain all sensitive data of the original system – passwords, configurations, user data, trade secrets. Unencrypted backups are a massive security risk, especially when stored on external media or cloud storage.

Attack vectors on backups:

  • Physical Theft: Stolen Backup Media Enables Offline Attacks
  • Cloud breaches: Compromised cloud storage accounts expose all backups
  • Insider threats: Administrators with backup access can extract sensitive data
  • Ransomware: Modern ransomware variants specifically destroy backups before encryption

Compliance requirements: GDPR explicitly requires the protection of personal data also in backups. Unencrypted backups can result in significant fines. Similar requirements can be found in HIPAA, PCI-DSS and other frameworks.

Encrypted backup strategies

Proxmox offers integrated backup encryption with AES-256. The implementation takes place at the backup job level and is transparent for the guest systems.

Key management challenges: The backup encryption key is the single point of failure for all backups. If you lose the key, all backups are useless. At the same time, the key must be stored so securely that attackers cannot obtain it. A Hardware Security Module (HSM) or Key Management Service (KMS) are recommended for critical environments.

# Create backup encryption key pvesm set <storage-id> --encryption-key /etc/pve/backup-encryption.key # Configure automatic encrypted backups cat > /etc/cron.d/proxmox-backup << 'EOF' # Daily encrypted VM backups at 2:00 am 0 2 * * * root /usr/bin/vzdump --all --compress zstd --encrypt 1 --storage backup-storage EOF

Immutable backup storage: Protection from ransomware

Immutable backups can no longer be modified or deleted after they have been created. This is the most effective protection against ransomware attacks, as even a fully compromised administrator cannot destroy the backups.

Implementation options:

  • Proxmox Backup Server: Provides native immutability features
  • Object storage: S3-compatible storage with Object Lock
  • WORM-Media: Write-Once-Read-Many hardware solutions
  • Air-Gapped Systems: Physically separated backup systems

Retention policy considerations: Immutable backups require careful planning of retention times. Too short retention times offer little protection, too long cause high storage costs. The 3-2-1 backup rule (3 copies, 2 different media, 1 offsite) should always be followed.

# Configuring Proxmox Backup Server with retention policies # (About PBS web interface: Datastore → Prune & GC) # Example of retention policy cat > /etc/proxmox-backup/prune-jobs.json < 'EOF' { "keep-daily": 7, "keep-weekly": 4, "keep-monthly": 6, "keep-yearly": 1 } EOF

Compliance and security audits: Measure what matters

Automated security audits with Tools like Lynis

Manual security checks are time-consuming and error-prone. Automated tools Like Lynis, they can systematically review hundreds of security aspects and identify vulnerabilities that people would overlook.

What Lynis does: Lynis performs over 300 different security tests, from kernel parameters to file permissions to network configurations. It not only detects known vulnerabilities, but also configuration errors and best practice violations.

Limitations of automated audits: Automated tools can only detect known problems. They do not understand the context of the application and can therefore generate false positives. A firewall rule criticised by Lynis as ‘too permissive’ could be correct for the specific use case.

# Lynis install apt install -y lynis # Complete security scan lynis audit system --cronjob --logfile /var/log/lynis.log # Analyze results grep "Suggestion" /var/log/lynis.log

Integration into CI/CD pipelines: Lynis can be integrated into automated deployment pipelines. New system deployments are automatically scanned for security issues before they go into production. This allows for ‘security by design’ instead of subsequent hardening.

CIS Benchmark Implementation: Follow industry standards

The Center for Internet Security (CIS) publishes detailed security benchmarks for all common operating systems and applications. These benchmarks are based on the consensus of cybersecurity experts and reflect best practices.

Why CIS benchmarks are important:

  • Legal protection: Many cyber insurance companies demand CIS compliance
  • Compliance frameworks: NIST, ISO 27001 and others refer to CIS benchmarks
  • Systematic approach: More than 100 specific control points for Linux systems
  • Risk-based: Each control is classified according to risk and impact

Implementation challenges: Full CIS compliance can make systems unusable. Many controls are too restrictive for specific use cases. A pragmatic approach selects the most important controls and documents deliberate exceptions. This can look like this:

# CIS compliant kernel parameters cat >> /etc/sysctl.d/99-cis-hardening.conf << 'EOF' # Network Security net.ipv4.ip_forward = 1 net.ipv4.conf.all.send_redirects = 0 net.ipv4.confault.default.send_redirects = 0 net.ipv4.conf.all.accept_source_route = 0 net.ipv4.confault.accept_redirects = 0 net.ipv4.conf.all.accept_source_route = 0 net.ipv4.confault.default.accept_source_route = 0 net.ipv4.conf.all.log_martians = 1 net.ipv4.icmp_echo_ignore_broadcasts = 1 net.ipv4.icmp_ignore_bogus_error_responses = 1 net.ipv4.confall.rpfilter = 1 net.ipv4.conf.conf.all_broadcasts = 1 net. = 1 net.ipv4.tcp_syncookies = 1 # IPv6 Security (if enabled) net.ipv6.conf.all.accept_redirects = 0 net.ipv6.conf.default.accept_redirects = 0 net.ipv6.conf.all.accept_source_route = 0 net.ipv6.conf.default.accept_source_route = 0 EOF sysctl -p /etc/sysctl.d/99-cis-hardening.conf

Kernel parameters in detail:

IP Forwarding: Proxmox must net.ipv4.ip_forward = 1 Stay active because the bridge functionality requires it. Default CIS configurations would disable this.

ICMP Redirects: Disabling ICMP redirects prevents certain routing attacks, but can cause connectivity issues in complex network topologies.

Reverse path filtering: rp_filter = 1 Enables strict reverse path testing and prevents IP spoofing attacks. However, this can cause problems with asymmetric routing.

Monitoring and Incident Response: Know what's happening

Central logging strategies

Millions of log entries are created every day in modern IT environments. Without central aggregation and intelligent analysis, it is impossible to detect security-relevant events. A well-thought-out logging strategy is therefore fundamental to effective security.

Log aggregation with Syslog: Syslog is the default for Unix/Linux systems, but the default configuration is not optimized for security monitoring. Important events can go down in the flood of debug messages. A structured categorization according to Severity and Facility is essential.

SIEM integration: Security Information and Event Management (SIEM) systems can correlate log data and detect anomalies. However, they are complex to configure and require continuous tuning processes to reduce false positives.

# Configure Rsyslog for central log forwarding cat >> /etc/rsyslog.conf << 'EOF' # Remote logging for security events *.info;mail.none;authpriv.none;cron.none @@syslog-server.company.local:514 authpriv.* @@syslog-server.company.local:514 EOF systemctl restart rsyslog

Data protection and log retention: Logs may contain personal data and are therefore subject to GDPR/GDPR regulations. A clear retention policy and anonymization strategies are legally required. At the same time, logs must remain available for forensic analysis – a difficult balancing act. We could probably create several articles of our own.

Metric-based monitoring using the example of Prometheus

While logs document events, metrics provide continuous insights into system health and performance. Anomalies in metrics can be early indicators of security issues. As an example, I like to take unusual CPU load here, this could indicate crypto mining. An abnormal network traffic, on the other hand, would indicate data exfiltration.

Prometheus architecture advantages:

  • Pull-based: Prometheus actively queries metrics, which is more robust than push models
  • Time series: Historical data enables trend analysis and baseline creation
  • Flexible alerting: Complex alert rules based on multiple metrics possible

Scaling challenges: Prometheus stores all data locally, causing storage problems in large environments. Federated setups or remote storage solutions become complex. Querying large periods of time can become very resource-intensive.

# Install node exporter for system metrics apt install -y prometheus-node-exporter # Export Proxmox-specific metrics cat > /etc/systemd/system/pve-exporter.service << 'EOF' [Unit] Description=Proxmox VE Exporter After=network.target [Service] Type=simple ExecStart=/usr/bin/pve-exporter Restart=always User=prometheus [Install] WantedBy=multi-user.target EOF

Maintenance Plan and Operational Safety

The Rhythm of Safety

You know, I'll say it again: Security is not a one-off project, but a continuous process. Without regular maintenance and updates, even the best security concept degenerates. Your structured maintenance plan ensures that all aspects are checked regularly!

Daily security tasks: These tasks should be automated or carried out as part of the daily routine. They serve the early detection of problems before they become critical.

  • Fail2Ban-Logs Review: Unusual attack patterns can indicate targeted attacks
  • Backup status: Failed backups are a critical security risk
  • Resource monitoring: Anomalies may indicate compromises

Weekly deepening: Weekly tasks require more time and attention, but provide deeper insights into security status.

  • Audit log analysis: Identify trends and anomalies in user behavior
  • Security update review: Evaluate available updates and plan deployment
  • Backup integrity tests: Perform random recovery test

Monthly strategic reviews: These tasks often require collaboration between teams and can result in major changes.

  • Penetration tests: External or internal security tests
  • Firewall rule reviews: Identify outdated or too permissive rules
  • Certificate monitoring: Renew expired certificates on time
  • User access reviews: Verification of user permissions

Quarterly depth analyses: These comprehensive reviews ensure that the security strategy keeps pace with changing threats and business needs.

  • Disaster recovery tests: Play through full DR scenarios
  • Incident response updates: Integrate Lessons Learned from Incidents
  • Security awareness training: Inform employees about new threats
  • Role-based access control reviews: Adapt role definitions to organizational changes

Change Management and Documentation

Any security change must be documented and comprehensible. Undocumented changes are a significant risk – they can be reversed accidentally or lead to confusion when staff changes occur.

Documentation best practices:

  • Rationale: Why was the change made?
  • Impact assessment: Which systems are affected?
  • Rollback plan: How can the change be reversed?
  • Test results: How was the change validated?

Four Eyes principle: Security-critical changes should always be validated by a second administrator. This reduces errors and also provides you with additional security, for example against insider threats.

Summary: Secure Proxmox environments in practice

The security measures presented here form a comprehensive defense-in-depth system. No single action is perfect, but when combined, they provide robust protection against most threats.

Implementation strategy: Implement these measures gradually, starting with the most basic ones (updates, firewall, SSH hardening) and then work your way up to more complex systems (monitoring, compliance). Don't forget: Test each change thoroughly first in a staging/laboratory environment.

Balancing security and usability: Excessive security measures can render systems unusable and cause users to bypass security controls. Find the right balance for your environment and the corresponding threat landscape.

Continuous improvement: The threat landscape is constantly evolving. What is safe today can be vulnerable tomorrow. Stay informed about new threats and security technologies and adjust your strategy accordingly.

Important note on implementation: All configurations presented here should be adapted to your specific environment. Be sure to thoroughly test each change before using it in production. Always keep up-to-date backups of your configuration and carefully document any changes.

And yes, I'm probably repeating myself for the thirteenth time: The security of the Proxmox infrastructure is also a marathon, not a sprint. With the practices presented here and a disciplined approach, you can build and operate a robust, secure virtualization platform.

The remaining parts of this article series I have also linked here again for you: Part 1: network | Part 2: storage | Part 3: backup | Part 4: security | Part 5: performance

Stephan
|
| | | | | |
to the top | homepage | to search