Root cause found for arr-bootstrap "couldn't verify root folder" — stale API keys, not a verification bug
Posted by Bram
Hi SparkBox team, (details written by AI) Wanted to share a full root-cause writeup for an issue I hit after a fresh reboot. The bootstrap logs were misleading, so I'm flagging both the actual cause and a couple of related gaps in case they're useful. Symptom sudo sparkbox arr-bootstrap reported: self-heal: radarr: couldn't verify root folder /data/Movies yet (radarr may still be starting) self-heal: sonarr: couldn't verify root folder /data/TV yet (sonarr may still be starting) ...on every run, indefinitely — not just once after a reboot. Root folders and download clients were not actually created/linked initially Before any of the fixes below, Sonarr/Radarr's root folders and qBittorrent download clients were not set up by bootstrap at all — I ended up creating/linking these manually through each app's web UI to get a working state. This makes sense in hindsight as a downstream symptom of the same root cause described below: any bootstrap-driven attempt to POST a root folder or register a download client would have hit the same 401 Unauthorized as every other API call, so those registration steps failed silently right alongside the verification step. Once the API keys were reconciled (see below), bootstrap's own self-heal was able to register the root folders correctly on its own (confirmed by seeing self-heal: radarr: root folder /data/Movies registered. in the logs post-fix) — so the underlying automation logic for this seems sound, it just never got a chance to run successfully. Actual root cause: stale API keys, not a timing/verification issue state/arr-api-keys.json contained API keys that no longer matched the live <ApiKey values in Sonarr's and Radarr's config.xml (and, we later found, Prowlarr's too). Every self-heal API call was sent with the wrong key and got a 401 Unauthorized — confirmed directly in each app's debug logs (radarr.debug.txt, sonarr.debug.txt). The arrrootfolderpresent() check doesn't inspect the HTTP status code — it tries to parse the response body as a rootfolder JSON array. A 401 body isn't parseable as that array, so it falls into the "unknown/still starting" branch and logs the (in this case) misleading "may still be starting" message. That message is accurate for its intended case (fresh container, not up yet) but is indistinguishable from "wrong credentials" in the logs, which cost a fair bit of debugging time. Why the keys went stale arr-bootstrap.sh's Step 1 only pre-writes config.xml from arr-api-keys.json on a fresh install (if [[ ! -f "${KEYSFILE}" ]]). Once state/arr-bootstrapped exists, later re-runs skip that step entirely and the self-heal block just trusts whatever is in KEYSFILE — it never re-reads the live key from config.xml. If Sonarr/Radarr/Prowlarr ever regenerate their config.xml (container recreate, config corruption + recovery, etc.) after the initial bootstrap, KEYSFILE silently drifts out of sync and every subsequent self-heal call 401s forever, with no automatic recovery path — including, apparently, the very first bootstrap run that's supposed to create the root folders and download clients in the first place. Worth noting: wirelidarr() already implements the correct pattern for this — it adopts the live key from config.xml on every invocation rather than trusting a persisted file. The sonarr/radarr/prowlarr self-heal path doesn't follow that same pattern, which is presumably how this drifted. Fix applied locally Added a reconcilekeyfromconfig() function to arr-bootstrap.sh, called for sonarr, radarr, and prowlarr at the top of the "Already bootstrapped" self-heal block. It compares the live <ApiKey in each app's config.xml against what's stored in KEYSFILE and rewrites KEYSFILE if they differ, before any of the existing rootfolder/download-client checks run. Stress-tested by deliberately reverting the keys file to stale values and re-running arr-bootstrap — it detected the mismatch, self-corrected, and root folder registration succeeded immediately after. Secondary finding: Seerr seeding can silently persist stale keys seerr-seed.sh ran and exited 0, logged "Seerr: seeding Sonarr + Radarr entries... Seed complete," and wrote the seerr-seeded marker — but it had pulled the same stale keys from arr-api-keys.json at the time, so Seerr's settings.json ended up with credentials that don't work. There's no post-write verification that Seerr can actually reach Sonarr/Radarr with what was just written, so the failure was completely silent — the log reads as a clean success. I ended up re-pairing Sonarr/Radarr manually via the Seerr UI, which worked immediately once given the correct (post-reconcile) keys. Two possible improvements here, if useful: 1. Have seerr-seed.sh verify against live config.xml the same way the reconcile fix does, rather than trusting arr-api-keys.json directly. 2. Add a lightweight post-check after seeding — e.g. hit Seerr's own settings/test-connection endpoint for each service — before writing the seerr-seeded marker, so a stale-key failure surfaces instead of reading as success. Summary of what was actually broken vs. what looked broken | Looked like | Actually was | |---|---| | "Root folder verification is buggy / has a timing issue" | Root folders were fine (once created); the API calls used wrong keys and got 401'd | | Root folders/download clients just weren't set up | Bootstrap's own registration attempts also hit 401s and silently failed, so nothing got created automatically at all | | "Seerr didn't get wired to Sonarr/Radarr" | Seerr was seeded, but with stale keys that silently didn't work | | Isolated to sonarr/radarr | Same class of bug also hit Prowlarr independently | Happy to share the exact patch if it's useful — it's a fairly small, self-contained addition that doesn't touch existing rootfolder/download-client/application logic, just adds a key-freshness check ahead of it. Thanks for a great project — the self-heal design is solid, this was just one gap in it.
1 replies
tomspark wrote:
This is a fantastic writeup — thank you. I checked it against the source and you nailed it: the self-heal block only ever reads arr-api-keys.json and never re-checks the live config.xml, and the rootfolder check genuinely can't tell a 401 from 'still starting', so a key drift looks like a startup delay forever. The lidarr path already adopts the live key on every run, so the fix is extending that same pattern to sonarr/radarr/prowlarr plus making the check call out auth failures explicitly — your writeup was thorough enough that we can fix it from here, no patch needed. One note: on the current build, seerr-seed already pulls keys straight from each container's config.xml rather than the JSON, so your box may have been a version or two behind there — but the 'verify before writing the seeded marker' idea is a good one and it's noted too. Fix is on the list.