Procedures

Recovery and validation procedure for the HP ProDesk 600 G4 Proxmox host after reboot, outage, certificate problems, VM startup failures, or other host-level issues.

Homelab Overview | HP ProDesk 600 G4 | Proxmox Host | VM100 - Jellyfin | VM101 - OpenLabFrame Dev | VM102 - Management | Uptime Kuma | Beszel

Purpose

This procedure is used when the primary Proxmox host:

HP ProDesk 600 G4

needs to be recovered or validated after:

• Planned reboot
• Unexpected restart
• Proxmox web UI outage
• VM startup failure
• Certificate problem
• Network problem
• Quick Sync / GPU passthrough issue
• Monitoring alert
• Service recovery problem

The goal is to recover from the host outward rather than randomly troubleshooting individual applications first.

Host Information

Physical host:

HP ProDesk 600 G4 SFF

Proxmox hostname:

Proxmox host

LAN IP:

<private Proxmox host>

Primary trusted URL:

<private Proxmox URL>

Direct recovery URL:

<private service endpoint>

Current VM Layout

VM100 - Jellyfin
<private Jellyfin host>
Always on

VM101 - OpenLabFrame Dev
<private development host>
On-demand

VM102 - Management
<private management host>
Always on

Expected behavior after a normal reboot:

VM100
→ Starts automatically

VM102
→ Starts automatically

VM101
→ May remain OFF intentionally

VM101 being off is not automatically a recovery failure.

Recovery Order

Always work downward through the dependency chain:

Physical HP ProDesk
      ↓
Proxmox host
      ↓
Host networking
      ↓
Proxmox services
      ↓
VM100 / VM102
      ↓
Guest Linux
      ↓
Docker
      ↓
Applications
      ↓
Monitoring

Do not start troubleshooting Jellyfin, NPM, or another application until the underlying host and VM are confirmed healthy.

Phase 1 - Confirm Physical Host Is Running

First confirm that the HP ProDesk is actually powered on.

Check:

Power LED
Fans
Network link LEDs
Display / console if attached

If the machine is completely off, application troubleshooting is premature.

Phase 2 - Test Network Reachability

From the Windows PC:

ping <private Proxmox host>

Expected:

Reply from <private Proxmox host>

If ping succeeds:

Physical host
+
Basic LAN networking

are probably working.

If ping fails, continue checking:

• Ethernet connection
• Switch
• BE550 router
• Host boot state
• Proxmox network configuration
• Physical NIC state

Phase 3 - Test Proxmox Web Interface

Preferred:

<private Proxmox URL>

Fallback:

<private service endpoint>

The direct IP is useful when troubleshooting:

DNS
TLS certificate
Hostname resolution

separately from Proxmox itself.

Phase 4 - Access the Proxmox Shell

Use either:

Proxmox web shell

or direct administrative access to the host.

Expected shell identity:

<admin-user>@<proxmox-host>

Confirm hostname:

hostname

Expected:

Proxmox host

Phase 5 - Check Host Uptime

uptime

This helps determine whether the server:

• Recently rebooted
• Has remained running
• Unexpectedly restarted

Also check:

who -b

This displays the most recent system boot time.

Phase 6 - Check Proxmox Services

Check the main Proxmox services:

systemctl status pveproxy pvedaemon pvestatd --no-pager

Expected:

active (running)

for the important services.

A failed:

pveproxy

can explain why the web UI is unavailable even though the host itself is reachable.

Quick Service Summary

systemctl --failed

Healthy target:

0 loaded units listed

or otherwise no unexpected failed services.

Beszel has previously observed approximately:

72 services
0 failed

on the healthy Proxmox host.

Phase 7 - Check Proxmox Port

The Proxmox web interface normally listens on:

<Proxmox management port>

Check:

ss -ltnp | grep <Proxmox management port>

A healthy system should show a listener on port:

<Proxmox management port>

If the host responds to ping but nothing listens on <Proxmox management port>, investigate pveproxy.

Phase 8 - Check VM State

List all VMs:

qm list

Expected layout includes:

100
101
102

Check individually:

qm status 100
qm status 101
qm status 102

Expected normal always-on state:

VM100
status: running

VM102
status: running

VM101 may show:

status: stopped

and still be completely normal.

Phase 9 - Start Missing Always-On VMs

If VM100 is unexpectedly stopped:

qm start 100

If VM102 is unexpectedly stopped:

qm start 102

Then verify:

qm status 100
qm status 102

Expected:

status: running

Do not automatically start VM101 unless website development is needed.

Phase 10 - Verify Autostart Configuration

If VM100 or VM102 repeatedly fails to start automatically after host reboot, inspect its configuration.

qm config 100

and:

qm config 102

Look for the expected startup behavior.

The current design requires:

VM100
→ Always on

VM102
→ Always on

VM101
→ On-demand

Do not change VM101 into an always-on workload simply to make every VM appear green.

Phase 11 - Check Guest Network Reachability

After the VMs show running, test them from another LAN device.

VM100

ping <private Jellyfin host>

VM102

ping <private management host>

Expected:

Reply

Remember:

VM running
≠
Guest OS completely ready

Linux and Docker may still be initializing.

Phase 12 - Validate VM100

SSH from Windows:

ssh <jellyfin-alias>

Then check:

uptime

and Docker:

docker ps

Important containers should begin returning.

Examples include:

Jellyfin
Gluetun
ARR Stack
Beszel Agent
Portainer Agent

Phase 13 - Validate Jellyfin

Direct backend:

<private service endpoint>

From a trusted machine:

curl -I <private service endpoint>

or open Jellyfin normally.

The important question is:

Does Jellyfin actually respond?

rather than simply:

Is VM100 running?

Phase 14 - Validate Gluetun

On VM100:

docker ps

Confirm Gluetun returns to a healthy state.

The custom Uptime Kuma checker can also be tested:

sudo /usr/local/sbin/check-gluetun-health
echo "Exit code: $?"

Healthy:

Exit code: 0

Phase 15 - Validate Gluetun Monitoring Timer

Check:

systemctl status gluetun-kuma-check.timer --no-pager

Expected:

active (waiting)

Also:

systemctl list-timers gluetun-kuma-check.timer --no-pager

The timer should schedule another check approximately every minute.

Phase 16 - Validate VM102

SSH:

ssh <management-alias>

Check:

uptime
docker ps

Important containers include:

Nginx Proxy Manager
Portainer
Beszel Agent

Phase 17 - Validate Nginx Proxy Manager

Direct NPM backend:

<private service endpoint>

Test:

curl -sS -o /dev/null -w "HTTP %{http_code}\n" \
  <private service endpoint>

Expected:

HTTP 200

Phase 18 - Validate Portainer

Direct backend:

<private service endpoint>

Example:

curl -k -sS -o /dev/null -w "HTTP %{http_code}\n" \
  <private service endpoint>

Expected:

HTTP 200

-k is acceptable here for the direct backend health test because the local Portainer certificate may not match the raw IP address.

Phase 19 - Validate Internal NPM Services

After VM102 and NPM recover, test friendly internal names.

Examples:

<private Beszel URL>
<private monitoring URL>
<private proxy-admin URL>
<private container-admin URL>

If direct backends work but friendly URLs fail, investigate:

Pi-hole
DNS
NPM
TLS

rather than the backend applications themselves.

Phase 20 - Check Intel Quick Sync on the Proxmox Host

The HP ProDesk contains:

Intel Core i7-8700
Intel UHD 630

Jellyfin uses Intel Quick Sync hardware acceleration.

On the Proxmox host, check that the graphics device still exists:

lspci | grep -Ei 'vga|display'

Expected to include the Intel integrated graphics device.

Also check:

ls -l /dev/dri

Expected graphics device nodes generally include entries such as:

card*
renderD*

The exact device assignment should match the known-good VM100 passthrough configuration.

Phase 21 - Check Quick Sync Inside VM100

Inside VM100:

ls -l /dev/dri

The expected GPU/render device should be visible.

If /dev/dri is missing after recovery:

Do not immediately change Jellyfin

First verify:

Proxmox GPU visibility
→ VM100 device assignment
→ Guest device visibility
→ Jellyfin hardware acceleration

Phase 22 - Verify Jellyfin Hardware Acceleration

After /dev/dri is confirmed inside VM100:

1. Start a file that requires transcoding.
2. Open the Jellyfin Dashboard.
3. Inspect the active playback session.

Confirm that hardware acceleration behaves normally.

The server has previously successfully used the Intel UHD 630 / Quick Sync path after Proxmox recovery.

Certificate Recovery / Validation

The normal trusted Proxmox hostname is:

<private Proxmox hostname>

If the UI works through:

<private service endpoint>

but the trusted hostname produces certificate errors, treat the problem as:

TLS / certificate

rather than a full Proxmox outage.

Inspect the Active Certificate

From the Proxmox host:

openssl s_client \
  -connect <loopback>:<SERVICE_PORT> \
  -servername <private Proxmox hostname> \
  </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates

Check:

Subject
Issuer
Not Before
Not After

The certificate should correspond to the trusted Proxmox hostname.

Certificate Key-Match Verification

If recovering a certificate and private key manually, verify that they belong together before installing them.

Using placeholders:

openssl x509 \
  -in <CERT_FILE> \
  -pubkey -noout \
  | openssl pkey -pubin -outform DER \
  | sha256sum

Then:

openssl pkey \
  -in <KEY_FILE> \
  -pubout -outform DER \
  | sha256sum

Expected:

Both SHA-256 hashes match

If they do not match:

STOP

Do not install that certificate/key pair.

Certificate Recovery Rule

Never overwrite a known-good working certificate configuration without first having:

Backup
+
Verified certificate/key match
+
Recovery path

The trusted Proxmox certificate has already been restored successfully in the current environment.

ACME is also configured for the trusted hostname.

Restarting pveproxy

If the correct certificate is present but the web proxy needs to reload:

systemctl restart pveproxy

Then verify:

systemctl status pveproxy --no-pager

Expected:

active (running)

Then test:

<private Proxmox URL>

Beszel Recovery Validation

The Proxmox host runs a native Beszel Agent.

Service:

beszel-agent.service

Check:

systemctl status beszel-agent --no-pager

Expected:

active (running)

Beszel Agent Port

Expected listener:

<Beszel agent port>

Check:

ss -ltnp | grep <Beszel agent port>

The Proxmox system should return to the Beszel dashboard as:

HP ProDesk - Proxmox

Uptime Kuma Recovery Validation

After the host and services recover, open Uptime Kuma.

Check monitors for:

Proxmox
VM100
VM102
Jellyfin
Gluetun
NPM
Portainer
Other critical VM100 services

Expected:

UP

Expected Recovery Timing

Services recover in layers.

Physical host starts
      ↓
Proxmox networking
      ↓
pveproxy / host services
      ↓
VM100 + VM102 start
      ↓
Guest Linux starts
      ↓
Docker starts
      ↓
Containers initialize
      ↓
Application checks pass
      ↓
Kuma returns monitors to UP

Do not expect every service to recover at exactly the same second.

Controlled Reboot Already Tested

A controlled reboot has already been performed successfully:

sudo reboot

During the test:

• Proxmox restarted
• VM100 became temporarily unavailable
• VM102 became temporarily unavailable
• Jellyfin went down temporarily
• Gluetun Push heartbeat stopped
• NPM went down
• Portainer went down
• Uptime Kuma detected application outages
• VMs started automatically
• Docker started automatically
• Applications returned automatically
• Beszel agents reconnected
• Uptime Kuma reported recovery

No manual application startup was required.

Important Test Limitation

The completed test was a:

Graceful controlled reboot

It was not an abrupt power-loss test.

Therefore:

Graceful reboot recovery
→ VERIFIED

Unexpected hard power loss
→ Not yet intentionally tested

Do not document abrupt-power recovery as proven until it has actually been tested.

Proxmox Monitor Timing

During the previous reboot test, application monitors went down while the Proxmox host monitor remained Up.

This occurred because the original Proxmox monitor retry window was long enough for the physical host to reboot before Kuma officially declared it Down.

The Proxmox monitor was then tightened to approximately:

Heartbeat Interval:       30 seconds
Retries:                  1
Heartbeat Retry Interval: 15 seconds
Global Timeout:           10 seconds

This should detect a genuine physical-host outage more quickly.

Recovery Interpretation

Proxmox Reachable, VM100 Down

Check:

qm status 100

Then:

VM startup
Guest Linux
Docker

Proxmox Reachable, VM102 Down

Check:

qm status 102

Then:

VM startup
Docker
NPM
Portainer

VM100 Up, Jellyfin Down

Focus on:

Docker
Jellyfin container
Application logs

not Proxmox itself.

VM102 Up, NPM Down

Focus on:

NPM container
Docker
Proxy admin port
NPM logs

Friendly URLs Down, Direct Backends Up

Focus on:

Pi-hole DNS
NPM
TLS

Proxmox UI Down, Ping Works

Focus on:

pveproxy
Port <Proxmox management port>
TLS certificate

Proxmox Host Missing From Beszel Only

If Proxmox otherwise works:

Do not reboot the host immediately

Check:

systemctl status beszel-agent --no-pager

and:

ss -ltnp | grep <Beszel agent port>

This may only be a monitoring-agent problem.

Minimal Emergency Checklist

If fast recovery is required:

1. Is HP ProDesk powered on?
2. Ping <private Proxmox host>
3. Open <private service endpoint>
4. Check Proxmox services
5. Run qm list
6. Confirm VM100 running
7. Confirm VM102 running
8. Test <private Jellyfin host>
9. Test <private management host>
10. Check Docker in both VMs
11. Test Jellyfin
12. Test NPM
13. Test Portainer
14. Check /dev/dri / Quick Sync
15. Check Beszel
16. Check Uptime Kuma

Useful Commands

Proxmox Host

hostname
uptime
who -b
systemctl --failed
systemctl status pveproxy pvedaemon pvestatd --no-pager
ss -ltnp | grep <Proxmox management port>
qm list

VM State

qm status 100
qm status 101
qm status 102

Start Always-On VMs

qm start 100
qm start 102

Only if unexpectedly stopped.

Intel GPU

lspci | grep -Ei 'vga|display'
ls -l /dev/dri

Beszel Agent

systemctl status beszel-agent --no-pager
ss -ltnp | grep <Beszel agent port>

VM100

ssh <jellyfin-alias>

Then:

uptime
docker ps
ls -l /dev/dri
systemctl status gluetun-kuma-check.timer --no-pager

VM102

ssh <management-alias>

Then:

uptime
docker ps

Things Not to Do Immediately

Avoid reacting to an outage by immediately:

• Reinstalling Proxmox
• Resetting certificates
• Resetting Tailscale
• Recreating VMs
• Deleting Docker volumes
• Resetting Uptime Kuma Push tokens
• Changing working network settings
• Rebuilding Jellyfin
• Disabling Quick Sync
• Restoring backups before understanding the failure

First determine:

Which layer actually failed?

Recovery Principle

The most important troubleshooting rule is:

Start at the lowest failed layer
and work upward.

Example:

Physical host
   ↓
Hypervisor
   ↓
VM
   ↓
Docker
   ↓
Application
   ↓
Reverse proxy
   ↓
DNS
   ↓
Client

Do not troubleshoot higher layers until lower layers are proven healthy.

Recovery Complete Criteria

Proxmox recovery can be considered complete when:

HP ProDesk reachable
Proxmox web UI reachable
Trusted hostname works
VM100 running
VM102 running
VM101 state intentional
VM100 network reachable
VM102 network reachable
Docker healthy on both VMs
Jellyfin responding
Gluetun healthy
NPM responding
Portainer responding
Quick Sync device visible
Beszel reporting
Uptime Kuma critical monitors green

Related Notes

Homelab Overview
HP ProDesk 600 G4
Proxmox Host
VM100 - Jellyfin
VM101 - OpenLabFrame Dev
VM102 - Management
Jellyfin
Nginx Proxy Manager
Beszel
Uptime Kuma
VM Restore
Certificate Recovery

Back to Procedures

Procedure for restoring a Proxmox VM from backup while preserving the correct VM identity, network role, and application behavior.

Homelab Overview | Proxmox Host | Proxmox Recovery | VM100 - Jellyfin | VM101 - OpenLabFrame Dev | VM102 - Management

Purpose

Use this procedure when a Proxmox VM needs to be restored after:

• VM corruption
• Failed update
• Broken Docker configuration
• Accidental configuration damage
• Failed migration
• Storage problem
• VM deletion
• Recovery testing
• Returning to a known-good state

The goal is to restore the VM without creating:

Duplicate VM IDs
Duplicate IP addresses
Duplicate hostnames
Duplicate Tailscale identities
Conflicting Docker services

Current VM Inventory

Current Proxmox VM layout:

VM100 - Jellyfin
Network: Private LAN
Role: Always-on media server

VM101 - OpenLabFrame Dev
Network: Private LAN
Role: On-demand development VM

VM102 - Management
Network: Private LAN
Role: Always-on NPM / Portainer / management VM

Always verify which VM is being restored before starting.

Important Principle

A restore is not complete when Proxmox says:

Restore finished

A complete restore means:

VM exists
↓
VM boots
↓
Network identity is correct
↓
Guest OS works
↓
Docker/services work
↓
Monitoring reconnects
↓
Dependent services recover

Before Restoring

Before making any changes, identify:

Which VM?
Which backup?
Which VMID?
Which storage?
Which network identity?
Which services depend on it?

Do not restore first and investigate later.

Step 1 - Identify the VM

Determine the correct VM:

100
101
or
102

From the Proxmox host:

qm list

Expected entries:

100
101
102

Record whether the target VM currently exists.

Step 2 - Check Current VM State

For VM100:

qm status 100

For VM101:

qm status 101

For VM102:

qm status 102

If the VM still exists and is running:

Do not restore over it immediately.

First determine whether:

• It should be stopped
• It should be backed up
• It should be renamed/replaced
• A restore test should use another VMID

Step 3 - Identify the Backup

In the Proxmox web interface:

Datacenter
→ Storage
→ Backups

Locate the correct backup for the VM.

Verify:

VM ID
Backup timestamp
Backup size
Storage location
Backup type

Do not rely only on the newest timestamp if the newest backup may contain the problem being recovered from.

Backup Selection Rule

Choose:

Newest known-good backup

not automatically:

Newest backup

If a recent update broke the VM, the immediately preceding backup may be the correct recovery point.

Step 4 - Decide Restore Strategy

There are two common restore strategies.

Replace Original VM

Use this when:

Original VM is broken
Original identity should remain
Original VMID should be preserved

Example:

VM100 backup
→ Restore as VM100

Only do this after ensuring the old VM100 will not conflict.

Restore to Temporary VMID

Use this when:

Testing backup integrity
Comparing configurations
Recovering files
Validating a restore before replacement

Example:

VM100 backup
→ Restore temporarily as VM900

This can be safer for validation.

However, a temporary clone/restore must not be allowed to create network conflicts.

Duplicate Identity Warning

A restored VM may contain the same:

Static IP
Hostname
SSH host keys
Tailscale state
Docker configuration
Application data

as the original.

If both are powered on simultaneously:

Network conflicts may occur.

Never boot both copies blindly.

Static IP Conflict Example

Suppose the original VM100 is still online:

VM100
<private Jellyfin host>

and a restored test copy also boots with:

<private Jellyfin host>

then:

Two machines
→ Same IP
→ Unpredictable connectivity

Therefore test restores should remain isolated until their networking is changed.

Step 5 - Stop Original VM if Replacing It

Example for VM100:

qm shutdown 100

Check:

qm status 100

Expected:

status: stopped

If the guest does not shut down cleanly, investigate before forcing it.

Avoid:

qm stop 100

unless a hard stop is actually necessary.

Step 6 - Preserve Current VM if Needed

If the existing VM still contains useful data or configuration, consider keeping it temporarily before replacement.

Options may include:

Backup current VM
Clone current VM
Rename documentation
Leave powered off temporarily

The exact method depends on the failure situation.

Step 7 - Restore Through Proxmox GUI

Typical GUI path:

Datacenter
→ Storage
→ Backups
→ Select backup
→ Restore

Confirm:

VM ID
Target storage
Restore options

Use the intended VMID.

CLI Restore

Proxmox VM backups can also be restored through the shell.

The exact command depends on:

Backup path
VMID
Storage
Backup format

Conceptually:

qmrestore <backup-file> <vmid>

Example structure only:

qmrestore /path/to/vzdump-qemu-100-YYYY_MM_DD.vma.zst 100

Do not copy this example blindly.

Use the actual backup path shown by Proxmox.

Step 8 - Confirm Restore Completion

After restore:

qm list

Confirm that the restored VM appears.

Then:

qm status <VMID>

Do not start it yet if there may still be a duplicate network identity.

Step 9 - Inspect VM Configuration

Before startup:

qm config <VMID>

Example:

qm config 100

Review important settings such as:

Memory
CPU cores
Boot order
Disks
Network adapter
Startup behavior
Machine type
BIOS/UEFI
PCI/device passthrough

VM100 Special Check - Intel GPU

VM100 depends on Intel UHD 630 / Quick Sync access.

If restoring VM100, verify the required hardware/device configuration has also returned.

Check:

qm config 100

Look for the known-good graphics/device assignment.

After boot, verify inside the guest:

ls -l /dev/dri

Do not assume hardware acceleration survived the restore until verified.

Step 10 - Confirm Startup Behavior

Expected VM roles:

VM100
→ Always on

VM101
→ On-demand

VM102
→ Always on

A restore should not unintentionally change those roles.

VM101 should not be converted to automatic always-on operation unless deliberately desired.

Step 11 - Start the Restored VM

Example:

qm start 100

Then:

qm status 100

Expected:

status: running

Give the guest OS time to boot.

Step 12 - Verify Network Reachability

VM100

ping <private Jellyfin host>

VM101

ping <private development host>

VM102

ping <private management host>

Expected:

Reply

If ping fails:

Do not immediately rebuild the VM.

Check:

• Guest boot
• Network adapter
• Bridge assignment
• Static IP
• Interface naming
• Router/LAN
• Duplicate IP conflict

Step 13 - SSH Test

VM100

ssh <jellyfin-alias>

VM101

ssh <development-alias>

VM102

ssh <management-alias>

If SSH reports a host-key mismatch after a legitimate restore, first confirm that the restored VM is truly the expected system before changing known-host entries.

Do not ignore SSH identity warnings automatically.

Step 14 - Check Guest Identity

Inside the restored VM:

hostname

Verify it matches the expected role.

Examples:

VM100
→ Jellyfin/media host identity

VM101
→ Open Lab Frame development host

VM102
→ <private-management-hostname>

Also verify:

ip addr

or:

hostname -I

Confirm the expected private network identity.

Step 15 - Check System Health

Inside Linux:

uptime

Then:

systemctl --failed

Unexpected failures should be investigated before declaring recovery complete.

Step 16 - Check Docker

For VMs that run Docker:

docker ps

Also:

docker ps -a

The second command reveals containers that exist but failed to start.

VM100 Restore Validation

VM100 is the most service-heavy VM.

Expected major containers include:

Jellyfin
Gluetun
ARR Stack
Beszel Agent
Portainer Agent

A restore is not complete until the important services return.

VM100 - Jellyfin Validation

Direct endpoint:

<private service endpoint>

Test from a client or browser.

Confirm:

Login page loads
Libraries appear
Media metadata exists
Playback works

VM100 - Quick Sync Validation

Inside VM100:

ls -l /dev/dri

Then test a transcoding session in Jellyfin.

The goal is to confirm:

Intel Quick Sync hardware acceleration

rather than falling back unexpectedly to CPU transcoding.

VM100 - Gluetun Validation

Check:

docker ps

Confirm Gluetun is:

Up
Healthy

Then:

sudo /usr/local/sbin/check-gluetun-health
echo "Exit code: $?"

Expected:

Exit code: 0

VM100 - Kuma Timer Validation

Check:

systemctl status gluetun-kuma-check.timer --no-pager

Expected:

active (waiting)

The restored VM should continue sending its Uptime Kuma Push heartbeat.

VM100 - Tailscale Validation

VM100 currently uses Tailscale for private and VPS connectivity.

Check:

tailscale status

and:

tailscale ip -4

Expected known VM100 Tailscale address:

<private Jellyfin Tailscale endpoint>

If the restored VM receives a different identity unexpectedly, investigate before assuming the public Jellyfin path is healthy.

Why Tailscale Identity Matters

The VPS Caddy backend currently points to:

<private Jellyfin Tailscale endpoint>:<SERVICE_PORT>

If VM100's restored Tailscale identity changes:

VPS
→ stale Tailscale endpoint
→ no backend

Public Jellyfin may fail.

Therefore verify the Tailscale identity after restore.

VM100 - Public Jellyfin Validation

After Tailscale is healthy, test:

https://jellyfin.openlabframe.com

Expected route:

Internet
   ↓
VPS Caddy
   ↓
Tailscale
   ↓
VM100

This confirms more than just local Jellyfin health.

VM101 Restore Validation

VM101 - OpenLabFrame Dev is different because it is not a critical always-on service.

Expected private network identity:

<private development host>

Expected SSH:

ssh <development-alias>

VM101 Git Repository

Verify:

cd <SOURCE_REPOSITORY>
git status

Expected branch:

main

Also check:

git remote -v

The repository should still reference:

<private Open Lab Frame source repository>

VM101 Toolchain

Verify:

node --version
npm --version

Current known environment has used:

Node.js 24.21.0
npm 11.19.0
nvm 0.40.7

If the backup predates that environment, versions may differ.

Do not silently change them until confirming whether the backup itself is simply older.

VM101 Build Test

From:

cd <SOURCE_REPOSITORY>

run:

npm ci

then:

npm run build

Expected:

dist/

A successful build is the most important application-level validation for VM101.

VM101 Development Preview

Expected backend:

<private development host>:<SERVICE_PORT>

Friendly URL:

<private development URL>

Path:

Pi-hole
→ VM102 / NPM
→ <private development endpoint>

Remember:

VM101 being off when not developing is normal.

VM102 Restore Validation

VM102 - Management is an always-on management VM.

Expected private network identity:

<private management host>

SSH:

ssh <management-alias>

VM102 Docker Validation

Check:

docker ps

Important containers include:

Nginx Proxy Manager
Portainer
Beszel Agent

VM102 NPM Validation

Direct:

<private service endpoint>

Test:

curl -sS -o /dev/null -w "HTTP %{http_code}\n" \
  <private service endpoint>

Expected:

HTTP 200

VM102 Portainer Validation

Direct:

<private service endpoint>

Test:

curl -k -sS -o /dev/null -w "HTTP %{http_code}\n" \
  <private service endpoint>

Expected:

HTTP 200

VM102 Internal Proxy Validation

Test private friendly URLs such as:

<private Beszel URL>
<private monitoring URL>
<private proxy-admin URL>
<private container-admin URL>

If direct NPM works but friendly services do not, investigate:

Pi-hole DNS
NPM Proxy Hosts
Wildcard certificate
Backend connectivity

Beszel Validation

Restored VMs using Beszel should return to:

Beszel

Current monitored systems:

HP ProDesk - Proxmox
Pi 5
VM100 - Jellyfin
VM102 - Management

VM101 is not currently a primary always-on Beszel target.

Uptime Kuma Validation

After restore, check Uptime Kuma.

Relevant monitors should recover to:

UP

Examples:

VM100
Jellyfin
Gluetun
VM102
NPM
Portainer

Remember that monitor recovery can lag behind actual service startup by one or more heartbeat intervals.

Restore Test Strategy

A backup should ideally be proven through a restore test.

A safe test process is:

Backup
   ↓
Restore to temporary VMID
   ↓
Keep networking isolated
   ↓
Boot
   ↓
Verify filesystem / services
   ↓
Power off
   ↓
Delete test VM

This validates backup integrity without replacing production.

Temporary Restore Warning

Do not boot a restored production copy onto the normal LAN if it still has the production:

IP
Tailscale identity
Hostname
Application identity

unless the original is offline.

Tailscale Duplicate Warning

Tailscale state may be included inside the restored VM's filesystem.

A test restore of VM100 could therefore appear as the same Tailscale node.

For isolated restore testing:

Avoid unrestricted network connectivity

until identity behavior is understood.

Backup Age Considerations

An older VM backup may restore:

Older Docker images
Older application databases
Older configs
Older SSH state
Older packages
Older certificates
Older Tailscale state

After restoration, do not immediately mass-update everything.

First verify the restored state works.

Then update deliberately.

Application Data May Be External

A VM backup may not necessarily contain every piece of application data if some storage is mounted externally.

Always identify whether the application uses:

VM-local disk
Host mount
NAS mount
Network share
Passthrough disk
External volume

before assuming the Proxmox backup contains everything.

VM100 Media Data

Jellyfin's application and automation environment is inside VM100, but large media data may eventually live separately on the future NAS Compute Build.

In that future architecture:

Restoring VM100
≠
Restoring the media library storage

These should be treated as separate recovery layers.

Restore Does Not Mean Rollback Everything

If only one application is broken, a full VM restore may be excessive.

Recovery should move from least destructive to most destructive:

Restart service
↓
Repair container/config
↓
Restore application data
↓
Restore VM

Use full VM restore when the entire guest state needs to be rolled back.

Do Not Restore Because of a Monitoring Failure Alone

Example:

Beszel says VM100 offline

does not automatically mean the VM needs restoration.

First verify:

Ping
SSH
Docker
Application
Agent

The problem may only be the monitoring agent.

Do Not Restore Because NPM Is Down

If:

This example was not included in the public notes.

fails but:

<private service endpoint>

The problem may only be:

NPM
DNS
TLS

A VM restore would be unnecessary.

Restore Decision Tree

Service broken?
   ↓
Is VM reachable?
   ├── Yes
   │    ↓
   │  Check application / Docker
   │
   └── No
        ↓
   Is VM running in Proxmox?
        ├── Yes
        │    ↓
        │  Check guest/network
        │
        └── No
             ↓
         Try normal startup
             ↓
       Still unusable?
             ↓
       Consider restore

Post-Restore Security Checks

After restore, verify:

• SSH still uses expected key authentication
• No unexpected passwords were re-enabled
• Tailscale identity is correct
• Application users/permissions are correct
• Reverse proxy configuration is correct
• Certificates are valid
• No old secrets/configurations unintentionally returned

A backup may reintroduce older security settings.

Post-Restore Update Strategy

After confirming the restored VM is stable:

1. Verify service health
2. Verify data
3. Verify networking
4. Verify monitoring
5. Take a fresh backup if appropriate
6. Then consider updates

Avoid updating everything before confirming that the restore itself succeeded.

Recovery Complete - VM100

VM100 recovery is complete when:

VM100 running
<private Jellyfin host> reachable
SSH works
Docker works
Jellyfin responds
Media libraries present
Quick Sync works
Gluetun healthy
ARR Stack services healthy
Tailscale healthy
<private Jellyfin Tailscale endpoint> correct
Public Jellyfin works
Beszel reports
Uptime Kuma green

Recovery Complete - VM101

VM101 recovery is complete when:

VM boots when requested
<private development host> reachable
SSH works
Git repository present
main branch correct
Node/npm usable
npm ci works
npm run build works
dist/ generated
Development preview works
<private development hostname> works while VM is on

Recovery Complete - VM102

VM102 recovery is complete when:

VM102 running
<private management host> reachable
SSH works
Docker works
NPM responds
Portainer responds
Private proxy hosts work
Wildcard HTTPS works
Beszel agent reports
Uptime Kuma monitors green

Minimal Restore Checklist

1. Identify VM and known-good backup
2. Check whether original VM still exists
3. Prevent duplicate IP/identity conflicts
4. Stop original VM if replacing it
5. Restore backup
6. Check qm config
7. Verify network/device settings
8. Start VM
9. Ping expected IP
10. SSH into guest
11. Check systemctl --failed
12. Check Docker
13. Validate application
14. Validate Tailscale where applicable
15. Validate Quick Sync for VM100
16. Validate NPM/Portainer for VM102
17. Validate build environment for VM101
18. Check Beszel
19. Check Uptime Kuma
20. Confirm dependent services work

Things Not to Do

Avoid:

• Restoring over a running production VM blindly
• Booting two copies with the same static IP
• Booting two copies with the same Tailscale identity
• Deleting the original VM before confirming backup validity
• Assuming a completed restore means applications are healthy
• Resetting Tailscale unnecessarily
• Resetting Uptime Kuma Push tokens unnecessarily
• Recreating all Docker data immediately
• Mass-updating the restored VM before validation
• Ignoring hardware passthrough on VM100

Related Notes

Homelab Overview
Proxmox Host
Proxmox Recovery
HP ProDesk 600 G4
VM100 - Jellyfin
VM101 - OpenLabFrame Dev
VM102 - Management
Jellyfin
Nginx Proxy Manager
Tailscale
Beszel
Uptime Kuma
Certificate Recovery

Back to Procedures

Safe troubleshooting procedure for Docker-based homelab services without accidentally deleting persistent data or turning a small container problem into a larger outage.

Homelab Overview | Raspberry Pi 5 | VM100 - Jellyfin | VM102 - Management | Uptime Kuma | Beszel

Purpose

Use this procedure when a Docker-based service is:

• Not starting
• Restarting repeatedly
• Unreachable
• Reporting unhealthy
• Missing from docker ps
• Working directly but not through NPM
• Working on one interface but not another
• Failing after a Compose change
• Failing after a host reboot
• Reporting unexpected networking behavior
• Using the wrong port
• Unable to reach another service
• Showing an error in Uptime Kuma
• Showing unusual status in Beszel

The main rule is:

Diagnose first.
Preserve data.
Change one layer at a time.

Docker Hosts in the Current Homelab

Docker is currently used heavily on several systems.

Raspberry Pi 5

Raspberry Pi 5

LAN IP:

<private Pi 5 host>

Important Docker services include:

Open Lab Frame / Nginx
cloudflared
Grafana
Prometheus
Node Exporter
Pi-hole Exporter
Uptime Kuma
Beszel Hub
Beszel Agent
Loki
Alloy
Portainer Agent

VM100 - Jellyfin

VM100 - Jellyfin

LAN IP:

<private Jellyfin host>

Important Docker services include:

Jellyfin
Gluetun
ARR Stack
Beszel Agent
Portainer Agent

VM102 - Management

VM102 - Management

LAN IP:

<private management host>

Important Docker services include:

Nginx Proxy Manager
Portainer
Beszel Agent

Troubleshooting Order

Always work through Docker problems in this order:

Host
↓
Docker Engine
↓
Compose Project
↓
Container
↓
Container Logs
↓
Ports / Bindings
↓
Docker Networking
↓
Persistent Volumes
↓
Backend Service
↓
Reverse Proxy
↓
DNS
↓
Client

Do not start at DNS or NPM if the container itself is not running.

Phase 1 - Confirm the Host Is Reachable

Before troubleshooting Docker, verify the host itself.

Examples:

Raspberry Pi 5

ping <private Pi 5 host>

VM100

ping <private Jellyfin host>

VM102

ping <private management host>

If the host is unreachable, Docker is not the first problem.

Phase 2 - SSH Into the Correct Host

Raspberry Pi 5

ssh <pi5-alias>

VM100

ssh <jellyfin-alias>

VM102

ssh <management-alias>

Always confirm you are on the correct machine before making changes.

Phase 3 - Confirm Docker Is Running

Check:

systemctl status docker --no-pager

Expected:

active (running)

If Docker is not running:

systemctl --failed

Then:

journalctl -u docker -n 100 --no-pager

Do not start deleting containers because Docker itself failed.

Phase 4 - Check Running Containers

docker ps

This shows currently running containers.

Important fields include:

CONTAINER ID
IMAGE
COMMAND
STATUS
PORTS
NAMES

Look for the expected container.

Phase 5 - Check All Containers

If the service is missing from docker ps, run:

docker ps -a

This also shows:

• Exited containers
• Failed containers
• Created but never started containers

Example:

Exited (1)

usually means the container started and then failed.

Do Not Immediately Remove an Exited Container

An exited container contains useful troubleshooting information.

Before recreating it, inspect:

docker logs <container>

and:

docker inspect <container>

Once the container is removed, some useful runtime information may disappear.

Phase 6 - Check Container Logs

Example:

This example was not included in the public notes.

Recent output only:

This example was not included in the public notes.

Follow live logs:

This example was not included in the public notes.

Stop live output with:

Ctrl+C

Add Timestamps

For clearer troubleshooting:

docker logs --timestamps --tail 100 <container>

This helps correlate:

Container failure
+
Host reboot
+
Uptime Kuma alert
+
Network event

Common Log Categories

Look for:

Permission denied
Connection refused
Address already in use
File not found
Database locked
Authentication failure
DNS resolution failure
Network unreachable
Invalid configuration
Read-only filesystem
Out of space

Do not assume every warning is fatal.

Look for the first error around the time the container stopped.

Phase 7 - Check Container Status

Inspect status:

docker inspect \
  --format='{{.State.Status}}' \
  <container>

Examples:

running
exited
restarting
created

Check Exit Code

docker inspect \
  --format='{{.State.ExitCode}}' \
  <container>

Common example:

0

usually means a clean exit.

Non-zero often indicates an error.

Check Restart Count

docker inspect \
  --format='{{.RestartCount}}' \
  <container>

A rapidly increasing restart count indicates a restart loop.

Phase 8 - Check Docker Health

If the container defines a Docker HEALTHCHECK:

docker inspect \
  --format='{{json .State.Health}}' \
  <container>

Possible states:

healthy
unhealthy
starting

Health None Is Not Automatically a Problem

Some containers do not define a Docker health check.

Beszel may show:

Health: None

This means:

No Docker HEALTHCHECK is configured

not:

Service is broken

Use Uptime Kuma to verify actual service availability.

Phase 9 - Identify the Compose Project

Before editing configuration, locate the correct project directory.

Examples:

This example was not included in the public notes.

Beszel

<BESZEL_DIRECTORY>

Nginx / Open Lab Frame

<WEB_STACK_DIRECTORY>

VM100 and VM102 projects should similarly be managed from their actual Compose directories rather than from random working directories.

Phase 10 - Inspect Compose Files

Move into the project:

cd /path/to/project

List files:

ls -lah

Look for:

compose.yaml
docker-compose.yml
.env
config/
data/

Do not edit a Compose file until confirming it is the one actually used by the running service.

Validate Compose Before Applying Changes

Run:

docker compose config

For a quiet validation:

docker compose config >/dev/null && echo "Compose config: OK"

Expected:

Compose config: OK

If validation fails:

STOP

Fix the YAML/configuration before recreating containers.

Why Compose Validation Matters

A typo such as:

wrong indentation
invalid key
missing variable
bad volume syntax
bad port mapping

can turn a working service into an outage.

Always validate first.

Phase 11 - Check Current Compose Status

From the project directory:

docker compose ps

This is often more useful than global docker ps because it shows only the current project.

Phase 12 - Check Port Bindings

Global view:

docker ps --format \
  'table {{.Names}}\t{{.Ports}}'

This shows which ports are exposed.

Inspect a Specific Container's Ports

docker port <container>

Example:

This example was not included in the public notes.

80/tcp
→ <loopback>:<SERVICE_PORT>
→ <private Pi 5 host>:<SERVICE_PORT>

Host Binding Matters

These are not equivalent:

<loopback>:<SERVICE_PORT>

and:

<private Pi 5 host>:<SERVICE_PORT>

<loopback>

Reachable only from the same host.

<private Pi 5 host>

Reachable from other LAN systems such as VM102.

A localhost-only binding:

<local service endpoint>

Adding:

<private Pi 5 host>:<SERVICE_PORT>

allowed:

VM102
→ <private Pi 5 host>:<SERVICE_PORT>

while retaining localhost access.

Phase 13 - Test the Backend Directly

Before troubleshooting NPM or DNS, test the service itself.

curl -sS -o /dev/null -w "HTTP %{http_code}\n" \
  <private service endpoint>/

Expected:

HTTP 200

Example Beszel

curl -sS -o /dev/null -w "HTTP %{http_code}\n" \
  <private service endpoint>/

Expected:

HTTP 200

Example NPM

curl -sS -o /dev/null -w "HTTP %{http_code}\n" \
  <private service endpoint>/

Expected:

HTTP 200

Backend First Rule

If:

Direct backend
→ FAILS

then troubleshoot:

Container
Docker
Port binding
Application
Host firewall

Do not start with NPM.

If:

Direct backend
→ WORKS

Friendly URL
→ FAILS

then troubleshoot:

Pi-hole
NPM
TLS
Routing

Phase 14 - Test From the Correct Source Host

A service may work locally but not from another machine.

From Pi 5

curl <private service endpoint>

may work.

But the important NPM test is from:

VM102

using:

curl <private service endpoint>

This verifies the actual network path NPM needs.

Phase 15 - Check Listening Ports on the Host

Use:

ss -ltnp

Filter a port:

ss -ltnp | grep 8000

or:

ss -ltnp | grep 8090

This helps determine whether Docker is actually listening on the expected address/port.

Address Already in Use

If logs show:

address already in use

identify the current listener:

ss -ltnp | grep <PORT>

Then:

docker ps --format \
  'table {{.Names}}\t{{.Ports}}'

Do not simply choose a random new port until understanding the conflict.

Phase 16 - Check Docker Networks

List networks:

docker network ls

Inspect one:

docker network inspect <network>

This is useful when containers communicate using names such as:

nginx-proxy-manager
portainer

instead of direct private network addresses.

Container Name Resolution

Containers on the same Docker network may reach each other by service/container name.

Example:

NPM
→ <internal container-admin endpoint>

This depends on both containers sharing a suitable Docker network.

If container-name routing fails, inspect networking before changing application configuration.

Host Networking

Some services use:

network_mode: host

Beszel Agents are an example where host networking can be appropriate.

With host mode:

Container networking
≈ Host networking

Port behavior differs from normal Docker bridge networking.

Do not troubleshoot a host-network container as though it has standard ports: mappings.

Phase 17 - Inspect Mounts

Check:

docker inspect <container>

Or display mounts only:

docker inspect \
  --format='{{json .Mounts}}' \
  <container>

Persistent data commonly comes from:

• Bind mounts
• Named Docker volumes
• External storage
• Host directories

Critical Rule - Containers Are Disposable, Data Is Not

A container can usually be recreated.

Persistent data must be preserved.

Conceptually:

Container
→ Replaceable

Persistent volume
→ Important

Never Delete Persistent Data During Basic Troubleshooting

Avoid commands such as:

rm -rf /path/to/data

or:

docker volume rm ...

unless the explicit goal is to permanently remove that data and a verified backup exists.

Docker Compose Down Warning

Command:

docker compose down

normally removes:

• Containers
• Compose networks

but preserves named volumes unless additional flags are used.

However, do not become careless with destructive variants.

Dangerous Volume Removal

Be especially careful with:

docker compose down -v

The -v option can remove named volumes.

Do not use it as a generic troubleshooting step.

Dangerous Docker Cleanup

Commands like:

docker system prune

or:

docker system prune -a

should not be used casually.

They can remove:

• Unused images
• Networks
• Build cache
• Containers

and, depending on options, more.

A cleanup command is not a repair strategy.

Phase 18 - Check Disk Space

Docker failures can result from a full filesystem.

Check:

df -h

Also:

df -i

A filesystem can run out of:

Space

or:

Inodes

Both can break Docker.

Docker Disk Usage

docker system df

This displays Docker storage consumption.

Do not immediately prune based only on this output.

First identify what is actually consuming space.

Large Logs

Docker JSON logs can occasionally grow significantly.

Check container log configuration and host disk usage before deleting anything.

If logs are suspected:

docker inspect <container> \
  --format='{{.LogPath}}'

This reveals the log-file location.

Do not manually delete active Docker log files without understanding the consequences.

Phase 19 - Check File Permissions

Many container problems are actually host filesystem permission problems.

Inspect:

ls -lah /path/to/data

and:

stat /path/to/data

Errors may include:

Permission denied
Read-only filesystem
Unable to create file
Unable to access database

Do not recursively chmod 777 as a generic solution.

Permission Rule

Use the minimum required permissions.

Before changing ownership or permissions:

Identify:
- Which UID/GID runs the container?
- Which directory needs access?
- What permissions currently exist?

Phase 20 - Check Environment Variables

Compose services may use:

environment:

or:

env_file:

Check the rendered Compose config:

docker compose config

Be careful because this may display secrets.

Do not paste secret-bearing output into Obsidian or screenshots.

Secret Handling

Never document actual:

• Passwords
• API keys
• Push tokens
• Tailscale auth keys
• Discord webhook URLs
• ntfy credentials

Use placeholders such as:

<TOKEN>
<PASSWORD>
<API_KEY>

Phase 21 - Restart Only the Affected Container

If configuration looks correct and a restart is appropriate:

docker restart <container>

Example:

This example was not included in the public notes.

Then:

docker ps

Prefer restarting one affected service instead of all Docker workloads.

Phase 22 - Compose Restart

From the project directory:

docker compose restart

This restarts the project's services without recreating them.

Use when the project itself is healthy but applications need a clean restart.

Phase 23 - Recreate a Container Safely

If Compose configuration changed:

docker compose config >/dev/null && echo "Compose config: OK"

Then:

docker compose up -d

If a forced recreation is actually required:

docker compose up -d --force-recreate

When Force-Recreate Makes Sense

Use it when:

• Port bindings changed
• Environment variables changed
• Mounts changed
• Container configuration changed
• The existing container needs replacement

Do not use it automatically for every outage.

Recreate Does Not Mean Delete Data

If persistent storage is correctly mounted:

Container recreation
→ New container
→ Same persistent data

This is one of Docker's main operational advantages.

Still verify mounts before recreating security-sensitive services.

Phase 24 - Pull a New Image Only When Appropriate

To update images:

docker compose pull

Then:

docker compose up -d

Do not mix:

Troubleshooting
+
Application upgrade

unless the upgrade is deliberately part of the fix.

Changing versions during diagnosis introduces another variable.

Troubleshoot Before Updating

Better order:

Identify failure
↓
Restore current version to working state
↓
Validate
↓
Then update separately

Phase 25 - Check Restart Policy

Inspect:

docker inspect \
  --format='{{.HostConfig.RestartPolicy.Name}}' \
  <container>

Common values include:

no
always
unless-stopped
on-failure

Restart Policy Matters After Reboot

An always-on infrastructure service generally should return automatically after its host reboots.

However, not every container necessarily uses the same policy.

Do not claim a restart policy has been changed until it has actually been verified in Compose or docker inspect.

Host Reboot Recovery

The homelab has already proven that major Docker services on:

VM100
VM102

recover automatically after a controlled Proxmox reboot.

The Pi 5 also hosts always-on Docker infrastructure.

After any host reboot:

docker ps

should be part of the validation process.

Phase 26 - Check Application Dependency Order

Some containers depend on others.

Example VM100:

Gluetun
↓
VPN-dependent applications

A dependent application can appear broken even though its own container is running.

Check dependencies rather than restarting random containers.

Gluetun Example

If a VPN-dependent container loses connectivity:

Application
→ Running

but

Gluetun
→ Unhealthy

the real problem may be Gluetun.

Check:

docker ps

and:

sudo /usr/local/sbin/check-gluetun-health
echo "Exit code: $?"

Healthy:

Exit code: 0

Phase 27 - Check DNS Inside a Container

If a container cannot resolve external or internal names:

docker exec <container> getent hosts example.com

If the image contains the appropriate utilities, other tests may include:

docker exec <container> ping ...

or:

docker exec <container> curl ...

Do not assume host DNS and container DNS behave identically.

Phase 28 - Check Connectivity From Inside a Container

Example:

docker exec <container> curl -I http://target:port

This helps determine whether:

Host can reach target

but:

Container cannot

which points toward Docker networking.

Phase 29 - Reverse Proxy Troubleshooting

For NPM-backed services, always test in this order:

Backend direct
↓
NPM direct
↓
Friendly hostname

1.
<private service endpoint>

2.
NPM on <private management host>

3.

Friendly URL Failure Example

If:

<private service endpoint>
→ HTTP 200

but:

→ fails

Investigate:

Pi-hole
NPM
Certificate
Routing

Phase 30 - Use Uptime Kuma Correctly

Uptime Kuma answers:

Is the service reachable?

A Kuma alert should trigger investigation, not immediate container deletion.

Example:

NPM DOWN

Check:

VM102 reachable?
Docker running?
NPM container running?
Proxy admin port listening?
HTTP response?

Phase 31 - Use Beszel Correctly

Beszel answers:

Is the host/container running?
How much resource is it using?

Beszel can help identify:

• CPU spikes
• RAM pressure
• Disk growth
• Container restart state
• Host temperature
• Network traffic

But:

Beszel Health: None

does not prove a service is broken.

Common Problem - Container Is Up but Service Is Down

Possible causes:

Application still starting
Application failed internally
Port not bound
Wrong configuration
Database problem
Dependency unavailable
Reverse proxy misconfigured

Check logs and direct backend response.

Common Problem - Service Works Locally Only

Example:

<loopback>:<SERVICE_PORT>
→ works

<private Pi 5 host>:<SERVICE_PORT>
→ fails

Likely causes:

Loopback-only binding
Host firewall
Wrong bind address

Common Problem - Container Keeps Restarting

Check:

docker ps -a

Then:

docker logs --tail 100 <container>

Then:

docker inspect \
  --format='{{.RestartCount}}' \
  <container>

Do not keep manually restarting it without reading the error.

Common Problem - Container Exited After Reboot

Check:

docker inspect \
  --format='{{.HostConfig.RestartPolicy.Name}}' \
  <container>

Then inspect Compose.

The issue may simply be:

restart policy

rather than application corruption.

Common Problem - Port Conflict

Error:

bind: address already in use

Check:

ss -ltnp | grep <PORT>

and:

docker ps --format \
  'table {{.Names}}\t{{.Ports}}'

Identify the legitimate owner before changing ports.

Common Problem - Permission Denied

Check:

docker logs <container>

Then:

ls -lah /path/to/data

and:

stat /path/to/data

Avoid:

chmod -R 777

as a generic fix.

Common Problem - Database Locked

For applications with embedded databases, multiple processes or improper shutdown can cause locking.

Before modifying database files:

Stop
Back up data
Identify database type
Read application logs

Do not delete the database to make the container start.

Common Problem - No Space Left on Device

Check:

df -h
df -i
docker system df

Then identify the cause.

Possible causes include:

• Media/download data
• Logs
• Old images
• Build cache
• Database growth
• Backup accumulation

Clean deliberately.

Common Problem - Image Pull Failure

Check:

docker compose pull

Possible causes:

DNS
Internet connectivity
Registry outage
Authentication
Rate limiting
Invalid image tag

Do not delete the running container merely because a newer image failed to download.

Common Problem - NPM Shows 502 Bad Gateway

A 502 Bad Gateway often means:

NPM is reachable
but
backend is not

Check the backend directly.

Example:

curl <private service endpoint>

Then verify the NPM forward host and port.

Common Problem - NPM Shows 504 Gateway Timeout

A timeout can mean:

Backend route unavailable
Firewall
Wrong IP
Backend hanging
Network problem

Test connectivity from VM102 specifically.

Common Problem - Friendly Name Does Not Resolve

If:

backend works
NPM works
hostname does not resolve

check Pi-hole.

Example:

This example was not included in the public notes.

Expected private target:

<private management host>

Common Problem - Remote Tailscale Access Fails

If local access works but remote private access fails, check:

Tailscale
Pi 3 subnet route
Pi-hole DNS
Remote client Tailscale connection

The Docker service may be perfectly healthy.

Example Troubleshooting - Beszel

Problem:

<private Beszel hostname> unavailable

Check direct Hub:

<private service endpoint>

If direct Hub works:

Beszel = healthy

Then investigate:

NPM
Pi-hole
TLS

Example Troubleshooting - Jellyfin

Problem:

Jellyfin unavailable

Check:

VM100 reachable?
Docker running?
Jellyfin container running?
<private Jellyfin host>:<SERVICE_PORT> responding?

Then check the applicable route:

LAN
Tailscale
VPS/Caddy

Do not assume the VPS is broken if local Jellyfin itself is down.

Example Troubleshooting - NPM

Problem:

All private friendly URLs fail

Check:

VM102 reachable?
Docker running?
NPM container running?
<private management host>:<SERVICE_PORT> responding?

If NPM itself is down:

multiple friendly URLs can fail simultaneously

even while all backend services remain healthy.

Back Up Before Risky Changes

Before changing:

• Volumes
• Application databases
• Compose storage paths
• Large configuration blocks
• Critical persistent data

make a backup first.

The more destructive the proposed change, the more important the backup.

Configuration Backup Pattern

Before editing an important Compose file:

cp compose.yaml \
  compose.yaml.backup-$(date +%Y%m%d-%H%M%S)

Then edit the real file.

This makes rollback much easier.

Compare Changes

After editing:

diff -u \
  compose.yaml.backup-YYYYMMDD-HHMMSS \
  compose.yaml

Review exactly what changed.

Safe Apply Workflow

Preferred sequence:

Backup configuration
↓
Edit
↓
docker compose config
↓
Review
↓
docker compose up -d
↓
docker compose ps
↓
docker logs
↓
Direct backend test
↓
Friendly URL test
↓
Uptime Kuma
↓
Beszel

Rollback Workflow

If a Compose change breaks the service:

Stop making additional changes
↓
Restore known-good Compose file
↓
Validate with docker compose config
↓
docker compose up -d
↓
Test backend
↓
Test proxy

Avoid layering multiple speculative fixes on top of one another.

What Not to Delete

Never casually delete:

/data
/config
Database files
Application volumes
Docker named volumes
Media directories
NPM database/config
Jellyfin configuration
Download state
ARR Stack databases

These are often the parts that actually matter.

What Can Usually Be Recreated

Usually safer to recreate:

Container object
Docker image
Compose-created bridge network
Temporary cache

provided persistent data is properly mounted and backed up.

Things Not to Do

Avoid:

• docker compose down -v as a generic fix
• Deleting volumes without backups
• chmod -R 777
• docker system prune -a without understanding the impact
• Recreating every container because one service failed
• Editing multiple Compose projects simultaneously
• Updating images during unrelated troubleshooting
• Resetting Tailscale for a Docker-only problem
• Changing Pi-hole for an application-only problem
• Rebooting Proxmox because one container failed
• Deleting databases because of startup errors
• Exposing services publicly just to make access easier

Minimal Docker Troubleshooting Checklist

1. Confirm host is reachable
2. SSH to correct host
3. Check Docker service
4. Run docker ps
5. Run docker ps -a
6. Read container logs
7. Check exit code / restart count
8. Check Compose project
9. Validate docker compose config
10. Check ports / bindings
11. Test backend directly
12. Test from the actual source host
13. Check Docker networking
14. Check mounts
15. Check disk space
16. Check permissions
17. Restart only affected service if appropriate
18. Recreate only if configuration requires it
19. Check Uptime Kuma
20. Check Beszel

Useful Commands

Docker Service

systemctl status docker --no-pager

Running Containers

docker ps

All Containers

docker ps -a

Container Logs

docker logs --tail 100 <container>

Follow Logs

docker logs -f <container>

Container Exit Code

docker inspect \
  --format='{{.State.ExitCode}}' \
  <container>

Restart Count

docker inspect \
  --format='{{.RestartCount}}' \
  <container>

Restart Policy

docker inspect \
  --format='{{.HostConfig.RestartPolicy.Name}}' \
  <container>

Compose Validation

docker compose config >/dev/null && echo "Compose config: OK"

Compose Status

docker compose ps

Apply Compose

docker compose up -d

Force Recreate

Only when needed:

docker compose up -d --force-recreate

Port Bindings

docker ps --format \
  'table {{.Names}}\t{{.Ports}}'

Listening Ports

ss -ltnp

Disk Space

df -h

Inodes

df -i

Docker Disk Usage

docker system df

Networks

docker network ls

Container Inspection

docker inspect <container>

Troubleshooting Principle

The most useful rule is:

Test the shortest path first.

For a web application:

Container
↓
Backend endpoint
↓
Reverse Proxy
↓
DNS
↓
Remote Routing
↓
Client

Find the first layer that fails.

That is usually where the real problem is.

Recovery Complete Criteria

A Docker issue can be considered resolved when:

Host healthy
Docker engine healthy
Expected container running
Logs show no critical repeating failure
Persistent data present
Correct ports bound
Direct backend works
Dependencies work
Proxy works if applicable
DNS works if applicable
Remote route works if applicable
Uptime Kuma reports UP
Beszel reports expected status

Related Notes

Homelab Overview
Raspberry Pi 5
VM100 - Jellyfin
VM102 - Management
Jellyfin
Beszel
Uptime Kuma
Nginx Proxy Manager
Pi-hole
Tailscale
Proxmox Recovery

Back to Procedures

Safe recovery and validation procedure for the trusted Proxmox HTTPS certificate used by <private Proxmox hostname>.

Homelab Overview | Proxmox Host | HP ProDesk 600 G4 | Proxmox Recovery | VM Restore

Purpose

Use this procedure when the Proxmox web interface has a certificate-related problem such as:

• Browser certificate warning
• Expired certificate
• Wrong certificate being served
• Certificate hostname mismatch
• Certificate/private-key mismatch
• ACME renewal problem
• Certificate replacement after recovery
• pveproxy serving an unexpected certificate

The trusted Proxmox hostname is:

<private Proxmox hostname>

Normal trusted interface:

<private Proxmox URL>

Direct recovery interface:

<private service endpoint>

Current Proxmox Host

Physical system:

HP ProDesk 600 G4 SFF

Proxmox hostname:

Proxmox host

LAN IP:

<private Proxmox host>

Expected shell:

<admin-user>@<proxmox-host>

Current Certificate State

The trusted certificate for:

<private Proxmox hostname>

has already been restored and validated successfully.

ACME is also configured for the Proxmox hostname.

This procedure exists so that future certificate work can be performed safely without repeating risky trial-and-error changes.

Important Principle

A certificate problem does not automatically mean Proxmox itself is broken.

For example:

<private service endpoint>

may still work while:

<private Proxmox URL>

shows a certificate problem.

That means the problem may only exist at the:

TLS / certificate layer

rather than:

Proxmox host
VMs
Networking
Applications

Recovery Order

Use this order:

Confirm host is reachable
        ↓
Confirm port <Proxmox management port> is listening
        ↓
Inspect currently served certificate
        ↓
Check hostname / expiration
        ↓
Verify replacement certificate
        ↓
Verify replacement private key
        ↓
Verify certificate and key match
        ↓
Back up current working files
        ↓
Install only if required
        ↓
Restart / reload pveproxy
        ↓
Verify served certificate
        ↓
Test trusted hostname

Do not begin by overwriting certificate files.

Phase 1 - Confirm Proxmox Is Actually Running

From Windows:

ping <private Proxmox host>

Expected:

Reply from <private Proxmox host>

Then test the direct interface:

<private service endpoint>

If the direct interface loads, the host and web service are likely operational.

Phase 2 - Check Port <Proxmox management port>

From the Proxmox shell:

ss -ltnp | grep <Proxmox management port>

Expected:

A listener on <Proxmox management port>

If nothing is listening on <Proxmox management port>, investigate:

pveproxy

before working on certificate replacement.

Phase 3 - Check pveproxy

systemctl status pveproxy --no-pager

Expected:

active (running)

Also:

systemctl status pvedaemon pvestatd --no-pager

These help confirm that the Proxmox management stack itself is healthy.

Phase 4 - Inspect the Certificate Currently Being Served

Run:

openssl s_client \
  -connect <loopback>:<SERVICE_PORT> \
  -servername <private Proxmox hostname> \
  </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates

Review:

subject
issuer
notBefore
notAfter

The certificate should correspond to:

<private Proxmox hostname>

and should not be expired.

Inspect Subject Alternative Names

To confirm the hostname is actually included in the certificate:

openssl s_client \
  -connect <loopback>:<SERVICE_PORT> \
  -servername <private Proxmox hostname> \
  </dev/null 2>/dev/null \
  | openssl x509 -noout -ext subjectAltName

Look for:

DNS:<private Proxmox hostname>

If the hostname is absent, browsers may reject the certificate even if the certificate itself is otherwise valid.

Inspect Full Certificate Details

For deeper inspection:

openssl s_client \
  -connect <loopback>:<SERVICE_PORT> \
  -servername <private Proxmox hostname> \
  </dev/null 2>/dev/null \
  | openssl x509 -noout -text

This can show:

• Subject
• Issuer
• Validity dates
• Subject Alternative Names
• Public key type
• Signature algorithm
• Key usage

External Certificate Check

You can also check what the hostname presents from another trusted Linux system:

openssl s_client \
  -connect <private Proxmox hostname>:<service-port> \
  -servername <private Proxmox hostname> \
  </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates

This verifies what clients actually see.

Certificate File Validation

Before installing a replacement certificate, inspect it directly.

Use the real certificate file path:

openssl x509 \
  -in <CERT_FILE> \
  -noout \
  -subject \
  -issuer \
  -dates

Example structure:

<CERT_FILE>

should be replaced with the actual certificate file being evaluated.

Do not guess the file path.

Check Certificate Hostnames

openssl x509 \
  -in <CERT_FILE> \
  -noout \
  -ext subjectAltName

Expected to include:

DNS:<private Proxmox hostname>

Private Key Validation

Before pairing a private key with a certificate, make sure OpenSSL can read it.

openssl pkey \
  -in <KEY_FILE> \
  -check \
  -noout

A valid key should complete without an error.

If OpenSSL reports that the key cannot be read:

STOP

Do not attempt installation.

Certificate / Private-Key Match Verification

This is one of the most important safety checks.

The certificate and private key must have the same public key.

Hash the Certificate Public Key

openssl x509 \
  -in <CERT_FILE> \
  -pubkey -noout \
  | openssl pkey \
      -pubin \
      -outform DER \
  | sha256sum

This produces a SHA-256 hash.

Example:

abc123...  -

Hash the Private Key Public Key

openssl pkey \
  -in <KEY_FILE> \
  -pubout \
  -outform DER \
  | sha256sum

This also produces a SHA-256 hash.

Expected Result

The two hashes must be identical.

Conceptually:

Certificate public key:
ABCDEF123456...

Private key public key:
ABCDEF123456...

Result:

MATCH

If the Hashes Do Not Match

If:

Certificate hash
≠
Private key hash

then:

STOP

Do not install the pair.

A mismatched certificate/private-key pair can prevent TLS from functioning correctly and may break the Proxmox web interface.

Why the Public-Key Hash Method Is Used

This method works regardless of whether the certificate uses:

RSA
EC / ECDSA

because the comparison is based on the actual public key rather than an RSA-specific modulus command.

Optional Fingerprint

You can identify a certificate using:

openssl x509 \
  -in <CERT_FILE> \
  -noout \
  -fingerprint \
  -sha256

This is useful for documenting which certificate is being evaluated without exposing the private key.

Never Display the Private Key

Do not use commands that print the private key contents into:

• Screenshots
• Chat
• Obsidian
• GitHub
• Logs
• Public notes

Do not run something like:

cat <private-key-file>

for documentation purposes.

Back Up Before Replacement

Before changing a working certificate configuration, make copies of the current certificate-related files.

The exact file locations should first be identified on the active host.

Create backups with a timestamp.

Conceptually:

current certificate
→ backup

current private key
→ backup

Do not overwrite the only known-good copy.

Safe Backup Pattern

Once the correct active paths are known, a backup pattern can be:

sudo cp <ACTIVE_CERT_FILE> \
  <ACTIVE_CERT_FILE>.backup-$(date +%Y%m%d-%H%M%S)

and:

sudo cp <ACTIVE_KEY_FILE> \
  <ACTIVE_KEY_FILE>.backup-$(date +%Y%m%d-%H%M%S)

Keep restrictive permissions on anything containing a private key.

Private-Key Permissions

A private key should never become broadly readable.

After copying/replacing a key, confirm permissions with:

ls -l <KEY_FILE>

Do not make a private key world-readable just to solve a permissions problem.

Install Only After Validation

Before any replacement, confirm all of the following:

Certificate readable
Private key readable
Hostname correct
Certificate not expired
Certificate/key hashes match
Current files backed up
Recovery path available

Only then should certificate installation proceed.

Restarting pveproxy

After a legitimate certificate update:

systemctl restart pveproxy

Then:

systemctl status pveproxy --no-pager

Expected:

active (running)

If pveproxy Does Not Start

Immediately check:

journalctl -u pveproxy -n 100 --no-pager

Possible causes include:

• Invalid certificate
• Invalid private key
• Certificate/key mismatch
• Wrong file permissions
• Incorrect path
• Malformed PEM content

Do not continue changing additional files until the error is understood.

Verify Port <Proxmox management port> After Restart

ss -ltnp | grep <Proxmox management port>

Expected:

<Proxmox management port> listening

Verify the New Active Certificate

After restart:

openssl s_client \
  -connect <loopback>:<SERVICE_PORT> \
  -servername <private Proxmox hostname> \
  </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates

Then:

openssl s_client \
  -connect <loopback>:<SERVICE_PORT> \
  -servername <private Proxmox hostname> \
  </dev/null 2>/dev/null \
  | openssl x509 -noout -ext subjectAltName

Confirm:

Correct hostname
Correct issuer
Correct expiration

Browser Validation

Open:

<private Proxmox URL>

Expected:

Trusted HTTPS
No unexpected certificate warning
Proxmox login screen loads

Direct-IP Certificate Warning

The direct address:

<private service endpoint>

may not match the trusted certificate hostname.

A hostname mismatch on the raw IP is not necessarily a certificate failure.

The trusted target is:

<private Proxmox hostname>

Use the IP primarily for recovery/troubleshooting.

DNS Validation

If the certificate is correct but the hostname cannot be reached, check DNS separately.

From Windows:

nslookup <private Proxmox hostname>

or:

Resolve-DnsName <private Proxmox hostname>

The problem may be:

DNS

rather than:

Certificate

Certificate vs DNS Problem

Case 1

<private service endpoint>
works

<private Proxmox hostname>
does not resolve

Likely:

DNS problem

Case 2

Hostname resolves
Web UI loads
Browser warns about certificate

Likely:

TLS certificate problem

Case 3

Host resolves
Port <Proxmox management port> closed

Likely:

pveproxy / Proxmox service problem

Case 4

Ping fails
Direct IP fails

Likely:

Host / network problem

Start with Proxmox Recovery instead.

ACME

ACME is configured for the trusted Proxmox hostname.

The purpose of ACME is to automate certificate issuance and renewal rather than manually replacing certificates every time they expire.

The preferred long-term state is:

ACME working
        ↓
Certificate renews automatically
        ↓
pveproxy serves renewed certificate

Manual certificate recovery should therefore be treated primarily as:

Emergency / repair procedure

rather than the normal renewal process.

If ACME Renewal Fails

Do not immediately replace everything manually.

First determine:

Why did renewal fail?

Possible categories include:

• DNS challenge problem
• ACME account problem
• Domain configuration
• API credential issue
• Certificate order failure
• Connectivity
• Permissions

The exact ACME failure should be inspected before resetting the configuration.

ACME Credential Security

Do not store:

Cloudflare API token
ACME secret
DNS API credentials

inside this Obsidian note.

Document where credentials are stored, not their contents.

Certificate Expiration Check

You can quickly check the certificate being served with:

echo | openssl s_client \
  -connect <private Proxmox hostname>:<service-port> \
  -servername <private Proxmox hostname> \
  2>/dev/null \
  | openssl x509 -noout -dates

Example output structure:

notBefore=...
notAfter=...

Check Days Until Expiration

A simple OpenSSL check can determine whether the certificate remains valid for a period.

Example: check whether it remains valid for at least 30 days:

echo | openssl s_client \
  -connect <private Proxmox hostname>:<service-port> \
  -servername <private Proxmox hostname> \
  2>/dev/null \
  | openssl x509 \
      -checkend $((30*24*60*60)) \
      -noout

Possible result:

Certificate will not expire

or:

Certificate will expire

Browser Cache Consideration

After replacing a certificate, a browser may temporarily display stale connection information.

If command-line validation proves the new certificate is being served but the browser still looks wrong:

• Close/reopen the browser tab
• Open a private/incognito window
• Reconnect
• Inspect the browser certificate details

Do not replace the certificate repeatedly solely because of a stale browser session.

Rollback

If a replacement causes problems and the old configuration was known-good:

Stop making additional changes
        ↓
Restore known-good certificate backup
        ↓
Restore known-good key backup
        ↓
Check permissions
        ↓
Restart pveproxy
        ↓
Validate <Proxmox management port>
        ↓
Inspect served certificate

This is why backups are mandatory before replacement.

pveproxy Logs

If TLS or startup problems occur:

journalctl -u pveproxy -n 100 --no-pager

For live monitoring:

journalctl -u pveproxy -f

Stop with:

Ctrl+C

Proxmox Service Check

After certificate recovery:

systemctl status \
  pveproxy \
  pvedaemon \
  pvestatd \
  --no-pager

Expected:

active (running)

Full Recovery Validation

Certificate recovery is not finished until all of these are true:

<private Proxmox host> reachable
<Proxmox management port> listening
pveproxy active
Trusted hostname resolves
Certificate hostname correct
Certificate not expired
Certificate and key match
<private Proxmox hostname> loads
Browser trusts HTTPS
Proxmox login page works
VMs remain unaffected

VM Safety

Certificate recovery should normally not require:

• Stopping VM100
• Stopping VM102
• Restarting all VMs
• Rebooting Proxmox
• Changing VM storage
• Modifying Docker
• Resetting Tailscale

A pveproxy restart affects the management interface, not the running VM workloads themselves.

Avoid turning a certificate problem into a larger outage.

Do Not Reset Unrelated Systems

During certificate troubleshooting, do not unnecessarily reset:

Tailscale
Pi-hole
NPM
Jellyfin
Uptime Kuma
Beszel
VM networking
Router settings

unless evidence shows one of those systems is actually involved.

Minimal Certificate Recovery Checklist

1. Ping <private Proxmox host>
2. Open <private service endpoint>
3. Check pveproxy
4. Check port <Proxmox management port>
5. Inspect currently served certificate
6. Check SAN for <private Proxmox hostname>
7. Check expiration
8. Inspect candidate replacement certificate
9. Validate candidate private key
10. Compare certificate/key SHA-256 public-key hashes
11. Back up active certificate/key
12. Install only if necessary
13. Restart pveproxy
14. Confirm pveproxy active
15. Confirm <Proxmox management port> listening
16. Inspect newly served certificate
17. Open <private Proxmox URL>
18. Confirm browser trust

Commands Summary

Host Reachability

ping <private Proxmox host>

pveproxy

systemctl status pveproxy --no-pager

Port <Proxmox management port>

ss -ltnp | grep <Proxmox management port>

Active Certificate

openssl s_client \
  -connect <loopback>:<SERVICE_PORT> \
  -servername <private Proxmox hostname> \
  </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates

Active SAN

openssl s_client \
  -connect <loopback>:<SERVICE_PORT> \
  -servername <private Proxmox hostname> \
  </dev/null 2>/dev/null \
  | openssl x509 -noout -ext subjectAltName

Candidate Certificate

openssl x509 \
  -in <CERT_FILE> \
  -noout \
  -subject \
  -issuer \
  -dates \
  -ext subjectAltName

Candidate Private Key

openssl pkey \
  -in <KEY_FILE> \
  -check \
  -noout

Certificate Public-Key Hash

openssl x509 \
  -in <CERT_FILE> \
  -pubkey -noout \
  | openssl pkey -pubin -outform DER \
  | sha256sum

Private-Key Public-Key Hash

openssl pkey \
  -in <KEY_FILE> \
  -pubout -outform DER \
  | sha256sum

Restart Proxy

systemctl restart pveproxy

Proxy Logs

journalctl -u pveproxy -n 100 --no-pager

Things Not to Do

Do not:

• Install a certificate before checking its hostname
• Install a certificate before checking expiration
• Install a key that OpenSSL cannot parse
• Install a certificate/key pair whose hashes differ
• Delete the existing certificate before making a backup
• Print the private key into screenshots
• Store the private key in Obsidian
• Store ACME/API secrets in Obsidian
• Reinstall Proxmox because of a certificate warning
• Reboot every VM because pveproxy has a TLS problem
• Reset working DNS or Tailscale configuration without evidence
• Keep replacing certificates without checking which one is actually being served

Recovery Principle

The key rule is:

Verify first.
Back up second.
Replace last.

Certificate recovery should be a controlled change, not trial and error.

Related Notes

Homelab Overview
HP ProDesk 600 G4
Proxmox Host
Proxmox Recovery
VM Restore
Network Topology

Back to Procedures