How to Rollback Embedded Linux Firmware

Why Rollback Matters

A bad firmware update on a fielded device is expensive — sometimes a truck roll, sometimes a brick. Pantavisor handles rollback automatically in two failure modes and gives you a manual workflow for the third (a logically-bad update that boots and runs but does the wrong thing).

Rollback Mode 1 — Boot Failure (Automatic)

If a new revision panics, fails to mount the rootfs, or never gets Pantavisor to PID 1 successfully, the bootloader / Pantavisor combo falls back to the last booted-good revision on next boot.

You don’t configure this. You don’t script it. The device recovers itself.

Rollback Mode 2 — status_goal Not Met (Automatic)

Each container declares a status_goal (typically STARTED or MOUNTED) in its run.json:

{
  "group": "platform",
  "name": "wificonnect",
  "restart_policy": "system",
  "root-volume": "root.squashfs",
  "type": "lxc"
}

device.json declares the goal per group:

{
  "groups": [
    {"name": "data",     "restart_policy": "system",    "status_goal": "MOUNTED"},
    {"name": "root",     "restart_policy": "system",    "status_goal": "STARTED"},
    {"name": "platform", "restart_policy": "system",    "status_goal": "STARTED"},
    {"name": "app",      "restart_policy": "container", "status_goal": "STARTED"}
  ]
}

Pantavisor watches each container reach its goal within the configured window. If any container in a system restart-policy group fails, Pantavisor reverts the entire state to the previous revision.

This catches updates that boot fine but break a service — exactly the failure mode that bricks devices in classic OTA setups.

Rollback Mode 3 — Manual Revert to a Specific Revision

When the new revision boots, runs, and meets every status_goal but is logically wrong (regression in app behavior, bad config, wrong dataset), no automatic rollback fires. Revert manually:

Step 1 — Find the Revision You Want

Every pvr commit + pvr post produces a numbered trail step in Pantahub. Browse them:

# View device history in the Pantahub web UI
# Or list via API
curl https://pvr.pantahub.com/<USER>/<DEVICE_NICK>/steps/

Each step has a numeric <REV> (0, 1, 2, …) and a content-addressed state JSON.

Step 2 — Clone the Target Revision

pvr clone https://pvr.pantahub.com/<USER>/<DEVICE_NICK>/steps/<REV> rollback-ws
cd rollback-ws

This pulls the exact state JSON and objects from that historical step into a fresh workspace.

Step 3 — Re-Post It as a New Revision

# Optionally re-sign before pushing
pvr sig add --parts=<list>

pvr post https://pvr.pantahub.com/<USER>/<DEVICE_NICK>

The device pulls the (historical) state, switches atomically, and runs the older composition. The trail step is appended — full audit history is preserved.

Why “Pin a Historical State” Beats “Rewind”

Rolling back by re-posting a historical state as a new revision keeps the audit trail intact: every change to the device is monotonic and recoverable. There’s no “lost” revision in the timeline. Compliance, forensics, and reproducibility all benefit.

Reproducibility Guarantee

Each state JSON is content-addressed by SHA256 over every object it references. Cloning step <REV> on Tuesday and again next year produces byte-identical artifacts — assuming Pantahub still hosts the objects (it does, by default, unless you garbage-collect).

This is how Pantavisor delivers bit-exact reproducible rollback without relying on container-image mutability or registry tag stability.

Watching the Rollback in Real Time

pvr device logs <DEVICE_NICK>/pantavisor.log@INFO

Pantavisor logs every revision switch, including the reason (“status_goal not met for group ‘platform’”, “boot failure detected”, etc.).

Common Pitfalls

  • Assuming pvr checkout rolls back — pvr checkout (alias pvr reset) discards uncommitted changes in the workspace. It does not affect the device.
  • Deleting failed revisions — leave them in Pantahub. They are forensic evidence and let you re-test the regression in CI.
  • Skipping status_goal in custom containers — without it Pantavisor cannot decide success/failure, so automatic rollback won’t trigger. Always declare it.

Next Steps