NPM cert requests + renewals fail every time — 256MB Docker memory limit OOM-kills sb-npm mid-issuance

Posted by ukonvasara

Summary: Every Let's Encrypt cert request through NPM on my box gets killed mid-write, leaving orphaned DB rows with no backing files on disk. Reproduced across multiple separate cert attempts — all with the identical failure signature. What happens: 1. Request/renew a cert in NPM UI. 2. Logs show certbot start the request, then the container process gets Killed before it finishes writing the renewal config and cert files. 3. NPM's DB still has a row for the cert, but nothing on disk backs it. 4. The next renewal check fails, reporting no certificate found at the expected renewal config path. 5. Trying to delete/revoke the broken cert in the UI also fails the same way (file-not-found on the cert files), so the orphaned row can't be cleaned up through the UI at all. 6. The frontend shows JSON.parse: unexpected character at line 1 column 1 — because the backend died mid-request and the browser got a truncated/non-JSON response instead of an error. Root cause (confirmed): sb-npm's Docker container has a hard memory limit of exactly 256MB: docker inspect sb-npm --format '{{.HostConfig.Memory}}' 268435456 NPM's node backend already runs with --maxoldspacesize=250 (250MB heap cap) — almost no headroom under the 256MB container ceiling. When certbot spins up alongside node during issuance (Python interpreter + DNS challenge/API calls), the combined footprint gets OOM-killed by the kernel before the write finishes. Ruled out: host RAM. My NAS has plenty of free memory — this is a container-level cgroup limit, not a host resource problem. Impact: this isn't just manual requests — renewals run through the exact same process/container/memory cap, so scheduled auto-renewals are equally exposed. A cert could silently fail to renew and nobody notices until it's expired. Ask: raise the default memory limit on sb-npm in the core module compose (or the node heap cap, or both) so there's real headroom for certbot to run alongside node during issuance/renewal. Hand-editing the compose file myself isn't a real fix since it gets overwritten on sparkbox up/update.

3 replies

Chris wrote:

Confirmed -- you nailed it. sb-npm ships with a 256MB memory cap on the container that's been there since day one and was never tuned per-app, so node plus certbot running together during issuance is exactly tight enough to get killed mid-write. That's a real bug on our end, not your box -- the orphaned DB row with no file behind it, and the broken JSON on delete, both follow from the same root cause. Flagged this with Tom so it gets a proper fix. Really appreciate the clean writeup -- the docker inspect output made this a five-minute confirm instead of a guessing game.

ukonvasara wrote:

Follow-up: dmesg confirms kernel-level OOM kills, but exact culprit still unconfirmed Ran sudo dmesg | grep -i "killed process\|out of memory" after reproducing this a few more times. Kernel is genuinely OOM-killing processes inside sb-npm's memory cgroup during cert operations — this isn't a guess, it's showing up directly in the kernel log: [22769.388358] Memory cgroup out of memory: Killed process 107326 (node) total-vm:11809816kB, anon-rss:66976kB, file-rss:47016kB, shmem-rss:0kB, UID:0 pgtables:1900kB oomscoreadj:0 [44544.529918] Memory cgroup out of memory: Killed process 1149075 (node) total-vm:11808396kB, anon-rss:69596kB, file-rss:46576kB, shmem-rss:0kB, UID:0 pgtables:1844kB oomscoreadj:0 [44550.128697] Memory cgroup out of memory: Killed process 1149042 (nginx) total-vm:141184kB, anon-rss:37868kB, file-rss:9852kB, shmem-rss:512kB, UID:0 pgtables:156kB oomscoreadj:0 [67704.259822] Memory cgroup out of memory: Killed process 2082827 (node) total-vm:11810904kB, anon-rss:69084kB, file-rss:36016kB, shmem-rss:0kB, UID:0 pgtables:1940kB oomscoreadj:0 [70635.323303] Memory cgroup out of memory: Killed process 3073646 (node) total-vm:11807400kB, anon-rss:65768kB, file-rss:35468kB, shmem-rss:0kB, UID:0 pgtables:1896kB oomscoreadj:0 [70708.522138] Memory cgroup out of memory: Killed process 3218551 (node) total-vm:11804584kB, anon-rss:65688kB, file-rss:35424kB, shmem-rss:0kB, UID:0 pgtables:1808kB oomscoreadj:0 [71286.317247] Memory cgroup out of memory: Killed process 3223787 (node) total-vm:11810216kB, anon-rss:73232kB, file-rss:34004kB, shmem-rss:0kB, UID:0 pgtables:2012kB oomscoreadj:0 [71397.822323] Memory cgroup out of memory: Killed process 3253904 (node) total-vm:11807152kB, anon-rss:72800kB, file-rss:34628kB, shmem-rss:0kB, UID:0 pgtables:1924kB oomscoreadj:0 Important correction to my root-cause theory above: note that none of the killed processes were individually near the 256MB limit — node's anon-rss is only ~65-73MB each time, nginx's is ~37MB. So this isn't simply "node's 250MB heap flag doesn't fit in a 256MB container." Something else in the same cgroup is pushing the combined total over 256MB at that moment, and the kernel picks off whichever process it picks — not necessarily the actual hog. My guess is certbot spiking alongside node/nginx during issuance (timing lines up), but I haven't captured certbot's own memory footprint mid-run, so that part's still a theory, not confirmed. Also worth noting: several kills are tightly clustered in time (e.g. 44544→44550, then four in a row between 70635-71397), which looks like retry-storm behavior — the backend dies, something retries almost immediately, dies again. Net takeaway: the cgroup-level 256MB cap being overrun during cert ops is confirmed. The exact process breakdown at the moment of overrun isn't — but regardless of which process tips it over first, a container that needs to run node + nginx + certbot simultaneously during issuance/renewal probably needs more headroom than 256MB gives it.

tomspark wrote:

Really appreciate you digging deeper on this -- that dmesg output is exactly the kind of confirmation that makes the fix easier to get right. You're correct that no single process is anywhere near the 256MB ceiling on its own; it's the combined total during cert issuance that tips it over, which lines up with certbot running alongside node and nginx. Doesn't change the fix direction -- the container just needs more memory headroom, same as a couple other apps we've bumped before. Passed this update along with Tom so it's factored in when the fix goes out. Thanks again for the thorough writeup.