no such service: metrics
Posted by cbj11200
So, Prometheus is not starting. I will paste some troubleshooting that TomAI had me do, but TomAI referred me here. This is the conversation between me and TomAI: got it, all working now. prometheus is still starting Yeah those are fine to ignore. Looking at the stopped containers from earlier: - sb-prometheus — stopped - sb-grafana-init — stopped - sb-prometheus-init — stopped The -init ones are supposed to stop — they're one-shot setup containers that run once and exit. That's normal. sb-prometheus being stopped is slightly more interesting. Try: docker logs sb-prometheus --tail 30 Paste that and I'll tell you if it's actually broken or just slow to start. ↗ Didn’t fully fix it? Share to demox so Tom can ship an update colin11200@armament11200:~$ docker logs sb-prometheus --tail 30 colin11200@armament11200:~$ No logs at all — it never really started. Try forcing it up: sparkbox up metrics Then check if it's running: docker ps | grep prometheus If it starts and stays up, you're good. If it exits again immediately, run: docker logs sb-prometheus --tail 30 right after and paste what comes back. ↗ Didn’t fully fix it? Share to demox so Tom can ship an update colin11200@armament11200:~$ sparkbox up metrics / | | | | ) \ \| ' \ / | '| |/ / \ / \ \/ / ) | |) | (| | | | <| |) | () < |/| ./ \,|| ||\\/ \//\\ || SparkBox v1.6.245 - Self-Hosted Privacy Stack Created by Tom Spark | youtube.com/@TomSparkReviews Network profile: private [INFO] Starting SparkBox... [INFO] Targeted up: metrics [INFO] Prepared 1 module config dir(s) for PUID=1000 PGID=1000 [INFO] .env changed since last up — recreating containers to pick up new values [+] up 5/6 ✔ Container sb-node-exporter Started 1.3s ✔ Container sb-cadvisor Started 1.3s ✔ Container sb-prometheus-init Exited 1.8s ✔ Container sb-grafana-init Started 1.3s ⠼ Container sb-prometheus Starting 1.6s ✔ Container sb-grafana Recreated 0.4s Error response from daemon: failed to set up container networking: Address already in use colin11200@armament11200:~$ docker ps | grep prometheus 3fe18a2fc9c1 quay.io/prometheus/node-exporter:v1.11.1 "/bin/nodeexporter …" 39 seconds ago Up 38 seconds 9100/tcp sb-node-exporter "Address already in use" — something else on the host has already claimed the port Prometheus wants. Check what's sitting on it: sudo ss -tlnp | grep 9090 If something's there, paste the output and we'll see what process is squatting on it. If that comes back empty, try: sudo ss -tlnp | grep 909 Prometheus defaults to 9090 — if another process grabbed it before the container started, Docker can't bind it. ↗ Didn’t fully fix it? Share to demox so Tom can ship an update they both come back empty Weird — port's free but Docker still says it's in use. This sometimes happens when Docker's network state gets stale after a failed start. Try: sparkbox down metrics Wait a few seconds, then: sparkbox up metrics If it fails again with the same error, run: docker logs sb-prometheus --tail 50 immediately after and paste that — want to catch whatever it logs before it dies. ↗ Didn’t fully fix it? Share to demox so Tom can ship an update it came back empty Ok so Prometheus starts, produces zero logs, and Docker reports "Address already in use" — that's a weird combo. Could be a Docker network conflict rather than a port conflict. Check what port Prometheus is actually trying to publish: grep -i prometheus /opt/sparkbox/modules/metrics/docker-compose.yml And also: docker inspect sb-prometheus | grep -A5 PortBindings Paste both — want to see exactly what port Docker is trying to bind and whether there's a stale network entry confusing things. ↗ Didn’t fully fix it? Share to demox so Tom can ship an update colin11200@armament11200:~$ grep -i prometheus /opt/sparkbox/modules/metrics/docker-compose.yml Prometheus + Grafana description: "Collect time-series metrics with Prometheus and visualize them in Grafana dashboards. See CPU, RAM, disk, network, and per-service metrics across your entire SparkBox in real time." beforeinstall: "Prometheus scrapes metrics every 15s and keeps 30 days of history. Grafana comes pre-wired to Prometheus and ships with two ready-made SparkBox dashboards - just log in." text: "Grafana's community library at grafana.com/dashboards has thousands of ready-made dashboards. In Grafana go to Dashboards, then New, then Import, and paste a dashboard ID. Dashboards built for 'node-exporter' (try ID 1860) or 'cadvisor' work out of the box because SparkBox already runs both collectors. When the import asks for a datasource, pick Prometheus." - source: "templates/prometheus-config.yml.tmpl" dest: "modules/metrics/config/prometheus.yml" prometheus: friendlyname: "Prometheus" description: "Dashboards and visualizations for all your metrics. Pre-configured to read from Prometheus, with two SparkBox dashboards built in." description: "Collects host-level metrics (CPU, RAM, disks, network) for Prometheus" description: "Collects per-container metrics for Prometheus" --- Prometheus init --- One-shot chown step so Prometheus's nobody(65534) user can write to the bind-mounted data directory. Without this Prometheus boots once, fails on /prometheus/queries.active "permission denied", enters a prometheus-init: containername: sb-prometheus-init command: ["sh", "-c", "mkdir -p /prometheus && chown -R 65534:65534 /prometheus"] - ${SBROOT:-/opt/sparkbox}/modules/metrics/data/prometheus:/prometheus --- Prometheus --- prometheus: image: prom/prometheus:v3.11.3 containername: sb-prometheus prometheus-init: - '--config.file=/etc/prometheus/prometheus.yml' - '--storage.tsdb.path=/prometheus' - ${SBROOT:-/opt/sparkbox}/modules/metrics/config/prometheus.yml:/etc/prometheus/prometheus.yml:ro - ${SBROOT:-/opt/sparkbox}/modules/metrics/data/prometheus:/prometheus as prometheus-init — one-shot chown before the real container starts. prometheus: Pre-wired Prometheus datasource + the two stock SparkBox dashboards Host-level metrics (CPU/RAM/disk/network). The shipped Prometheus image: quay.io/prometheus/node-exporter:v1.11.1 with). Prometheus's /targets page is the health signal instead. scraped by the shipped Prometheus config, never actually deployed... second command: colin11200@armament11200:~$ docker inspect sb-prometheus | grep -A5 PortBindings "PortBindings": { "9090/tcp": [ { "HostIp": "", "HostPort": "9092" } So Prometheus wants to bind host port 9092, and ss says it's free. But the live ports list in your dashboard shows sb-prometheus: 9092 — which means Docker might have a stale network allocation from the previous failed start. Try nuking the stopped container entirely and letting SparkBox recreate it fresh: docker rm sb-prometheus sparkbox up metrics That clears the stale container state before Docker tries to bind the port again. ↗ Didn’t fully fix it? Share to demox so Tom can ship an update [INFO] Starting SparkBox... [INFO] Targeted up: metrics [INFO] Prepared 1 module config dir(s) for PUID=1000 PGID=1000 [INFO] .env changed since last up — recreating containers to pick up new values [+] up 5/6 ✔ Container sb-prometheus-init Exited 1.8s ✔ Container sb-node-exporter Started 1.3s ✔ Container sb-cadvisor Started 1.3s ✔ Container sb-grafana-init Started 1.3s ⠴ Container sb-prometheus Starting 1.7s ✔ Container sb-grafana Recreated 0.4s Error response from daemon: failed to set up container networking: Address already in use Same error even with a clean container. This points at a Docker network issue, not a port or container state issue. Check if there's a conflicting network bridge: docker network ls | grep metrics And: ip addr | grep 172 Paste both — I want to see if there's a stale bridge interface sitting on an IP range that conflicts with what Docker is trying to allocate for the metrics network. That "Address already in use" on networking setup (not port binding) is almost always a bridge IP collision. ↗ Didn’t fully fix it? Share to demox so Tom can ship an update colin11200@armament11200:~$ docker network ls | grep metrics b48baa91b015 sparkboxsbmetrics bridge local colin11200@armament11200:~$ ip addr | grep 172 inet 172.20.0.1/24 brd 172.20.0.255 scope global br-65be62216f71 inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 inet 172.20.22.1/24 brd 172.20.22.255 scope global br-7ce2e4bc4126 inet 172.20.2.1/24 brd 172.20.2.255 scope global br-cb0347600df5 inet 172.20.33.1/24 brd 172.20.33.255 scope global br-7d5b9c8e8c37 inet 172.20.5.1/24 brd 172.20.5.255 scope global br-12413957b886 inet 172.20.4.1/24 brd 172.20.4.255 scope global br-49a6f922def9 inet 172.18.0.1/16 brd 172.18.255.255 scope global br-a828f2c16c5e inet 172.20.28.1/24 brd 172.20.28.255 scope global br-b48baa91b015 There's your problem. br-b48baa91b015 matches the sparkboxsbmetrics network ID (b48baa91b015) — that bridge already exists on the host but the container can't attach to it cleanly. Stale bridge from the previous failed starts. Safe fix — let SparkBox tear it down properly: sparkbox down metrics Then check the bridge is gone: ip addr | grep b48baa91b015 If it's still there after sparkbox down, then: docker network rm sparkboxsbmetrics Then bring it back up: sparkbox up metrics sparkbox down is the safe way to clear this — it lets Docker clean up the network properly rather than yanking the interface by hand. ↗ Didn’t fully fix it? Share to demox so Tom can ship an update colin11200@armament11200:~$ ip addr | grep b48baa91b015 38: br-b48baa91b015: <BROADCAST,MULTICAST,UP,LOWERUP mtu 1500 qdisc noqueue state UP group default inet 172.20.28.1/24 brd 172.20.28.255 scope global br-b48baa91b015 216: veth5ae20a2@if2: <BROADCAST,MULTICAST,UP,LOWERUP mtu 1500 qdisc noqueue master br-b48baa91b015 state UP group default 218: vethb1ba4bb@if2: <BROADCAST,MULTICAST,UP,LOWERUP mtu 1500 qdisc noqueue master br-b48baa91b015 state UP group default 223: veth16b8897@if2: <BROADCAST,MULTICAST,UP,LOWERUP mtu 1500 qdisc noqueue master br-b48baa91b015 state UP group default Bridge is still up and has 3 veth interfaces attached to it — those are the other metrics containers (node-exporter, cadvisor, grafana) that started fine. Prometheus is the only one that failed to attach. That's actually a different problem than a stale bridge. Something else is going on with Prometheus specifically trying to attach to that bridge. Try removing just the Prometheus container and the network together, then letting SparkBox rebuild: sparkbox down metrics docker network rm sparkboxsbmetrics sparkbox up metrics sparkbox down should remove the network, but given it's not doing so cleanly, the explicit docker network rm in the middle should force it. The other containers will all recreate fine on sparkbox up. If that still fails with the same error, this is getting weird enough that I'd want Tom to look at it — hit the Share to demox button below and post the full output there. ↗ Didn’t fully fix it? Share to demox so Tom can ship an update colin11200@armament11200:~$ docker network rm sparkboxsbmetrics Error response from daemon: error while removing network: network sparkboxsbmetrics has active endpoints (name:"sb-grafana" id:"18f87b82f08c", name:"sb-node-exporter" id:"49b67e0589b0", name:"sb-cadvisor" id:"13956c4ecfc0") exit status 1 Right, the other containers are still attached. Do sparkbox down metrics first to stop them all, then remove the network: sparkbox down metrics && docker network rm sparkboxsbmetrics && sparkbox up metrics That should stop everything, clear the stale network, and bring it all back up clean in one shot. ↗ Didn’t fully fix it? Share to demox so Tom can ship an update colin11200@armament11200:~$ sparkbox down metrics [INFO] Stopping SparkBox... no such service: metrics no such service: metrics no such service: metrics no such service: metrics no such service: metrics no such service: metrics no such service: metrics colin11200@armament11200:~$ ip addr | grep b48baa91b015 38: br-b48baa91b015: <BROADCAST,MULTICAST,UP,LOWERUP mtu 1500 qdisc noqueue state UP group default inet 172.20.28.1/24 brd 172.20.28.255 scope global br-b48baa91b015 216: veth5ae20a2@if2: <BROADCAST,MULTICAST,UP,LOWERUP mtu 1500 qdisc noqueue master br-b48baa91b015 state UP group default 218: vethb1ba4bb@if2: <BROADCAST,MULTICAST,UP,LOWERUP mtu 1500 qdisc noqueue master br-b48baa91b015 state UP group default 223: veth16b8897@if2: <BROADCAST,MULTICAST,UP,LOWERUP mtu 1500 qdisc noqueue master br-b48baa91b015 state UP group default colin11200@armament11200:~$ sparkbox down metrics [INFO] Stopping SparkBox... no such service: metrics no such service: metrics no such service: metrics no such service: metrics no such service: metrics no such service: metrics no such service: metrics colin11200@armament11200:~$ docker network rm sparkboxsbmetrics Error response from daemon: error while removing network: network sparkboxsbmetrics has active endpoints (name:"sb-grafana" id:"18f87b82f08c", name:"sb-node-exporter" id:"49b67e0589b0", name:"sb-cadvisor" id:"13956c4ecfc0") exit status 1 sparkbox down metrics is saying "no such service: metrics" — that's the real problem here. Something's off with how the metrics module compose is being read. The containers are running but SparkBox can't find them to stop them. Stop the metrics containers directly with Docker: docker stop sb-grafana sb-node-exporter sb-cadvisor sb-grafana-init Then remove the network: docker network rm sparkboxsbmetrics Then bring it back up: sparkbox up metrics If sparkbox down metrics is consistently saying "no such service" that's worth flagging to Tom — hit Share to demox after this and include that output, even if Prometheus comes up fine. That's not normal behavior.
7 replies
tomspark wrote:
Thanks for the thorough writeup — and you did spot a real bug (more on that below). First, let's get Prometheus unstuck. That 'Address already in use' is Docker's network state gone stale after the failed starts, not an actual port clash. Clear the metrics containers and their network fully, then let SparkBox rebuild: docker rm -f sb-prometheus sb-grafana sb-node-exporter sb-cadvisor sb-grafana-init sb-prometheus-init docker network rm sparkboxsbmetrics sudo sparkbox up metrics The -f removes them whether running or stopped, which frees the network so it'll actually delete this time (that's what kept failing on you). Your Grafana dashboards and Prometheus data live in folders on disk, so nothing is lost. If it STILL says 'Address already in use' after that, the sure fix is: sudo systemctl restart docker && sudo sparkbox up — that rebuilds all of Docker's networking from scratch (it briefly bounces everything). On 'no such service: metrics' from sparkbox down — you're right, that's not normal, and I've flagged it to get fixed. One more thing worth knowing: you're on 1.6.245 and current is 1.6.285, so once metrics is healthy a sudo sparkbox update will pull in a lot of fixes. Let me know how the rebuild goes!
cbj11200 wrote:
colin11200@armament11200:~$ ip addr | grep b48baa91b015 38: br-b48baa91b015: <BROADCAST,MULTICAST,UP,LOWERUP mtu 1500 qdisc noqueue state UP group default inet 172.20.28.1/24 brd 172.20.28.255 scope global br-b48baa91b015 216: veth5ae20a2@if2: <BROADCAST,MULTICAST,UP,LOWERUP mtu 1500 qdisc noqueue master br-b48baa91b015 state UP group default 218: vethb1ba4bb@if2: <BROADCAST,MULTICAST,UP,LOWERUP mtu 1500 qdisc noqueue master br-b48baa91b015 state UP group default 223: veth16b8897@if2: <BROADCAST,MULTICAST,UP,LOWERUP mtu 1500 qdisc noqueue master br-b48baa91b015 state UP group default colin11200@armament11200:~$ sparkbox down metrics [INFO] Stopping SparkBox... no such service: metrics no such service: metrics no such service: metrics no such service: metrics no such service: metrics no such service: metrics no such service: metrics colin11200@armament11200:~$ docker network rm sparkboxsbmetrics Error response from daemon: error while removing network: network sparkboxsbmetrics has active endpoints (name:"sb-grafana" id:"18f87b82f08c", name:"sb-node-exporter" id:"49b67e0589b0", name:"sb-cadvisor" id:"13956c4ecfc0") exit status 1 colin11200@armament11200:~$ docker stop sb-grafana sb-node-exporter sb-cadvisor sb-grafana-init sb-grafana sb-node-exporter sb-cadvisor sb-grafana-init colin11200@armament11200:~$ docker stop sb-grafana sb-node-exporter sb-cadvisor sb-grafana-init sb-grafana sb-node-exporter sb-cadvisor sb-grafana-init colin11200@armament11200:~$ docker network rm sparkboxsbmetrics sparkboxsbmetrics colin11200@armament11200:~$ sparkbox up metrics [INFO] Starting SparkBox... [INFO] Targeted up: metrics [INFO] Prepared 1 module config dir(s) for PUID=1000 PGID=1000 [INFO] .env changed since last up — recreating containers to pick up new values [+] up 6/7 ✔ Network sparkboxsbmetrics Created 0. ✔ Container sb-grafana-init Started 1. ✔ Container sb-prometheus-init Exited 1. ✔ Container sb-node-exporter Started 1. ✔ Container sb-cadvisor Started 1. ⠸ Container sb-prometheus Starting 1. ✔ Container sb-grafana Recreated 0. Error response from daemon: failed to set up container networking: Address already in use colin11200@armament11200:~$ sparkbox down metrics [INFO] Stopping SparkBox... no such service: metrics no such service: metrics no such service: metrics no such service: metrics no such service: metrics no such service: metrics no such service: metrics colin11200@armament11200:~$ docker rm -f sb-prometheus sb-grafana sb-node-exporter sb-cadvisor sb-grafana-init sb-prometheus-init sb-prometheus sb-grafana sb-node-exporter sb-cadvisor sb-grafana-init sb-prometheus-init colin11200@armament11200:~$ docker network rm sparkboxsbmetrics sparkboxsbmetrics colin11200@armament11200:~$ sudo sparkbox up metrics [INFO] Starting SparkBox... [INFO] Targeted up: metrics [INFO] Prepared 4 module config dir(s) for PUID=1000 PGID=1000 [INFO] .env changed since last up — recreating containers to pick up new values [+] up 6/7 ✔ Network sparkboxsbmetrics Created 0. ✔ Container sb-prometheus-init Exited 1. ✔ Container sb-grafana-init Started 0. ✔ Container sb-node-exporter Started 0. ✔ Container sb-cadvisor Started 0. ⠼ Container sb-prometheus Starting 1. ✔ Container sb-grafana Created 0. Error response from daemon: failed to set up container networking: Address already in use colin11200@armament11200:~$ sudo systemctl restart docker && sudo sparkbox up [INFO] Starting SparkBox... [INFO] Active modules: adguard cloud core dashboard jellystat media metrics [INFO] Prepared 4 module config dir(s) for PUID=1000 PGID=1000 [INFO] .env changed since last up — recreating containers to pick up new values [+] up 1/1 ✔ Container sb-adguard Started 1. [+] up 2/3 ⠧ Container sb-nextcloud-redis Waiting 16. ✘ Container sb-nextcloud-db Error dependency nextcloud-db failed to start 16. ✔ Container sb-nextcloud Recreated 3. dependency failed to start: container sb-nextcloud-db is unhealthy [+] up 3/3 ✔ Container sb-portainer Started 3. ✔ Container sb-homepage Started 3. ✔ Container sb-npm Started 4. [+] up 2/2 ✔ Container sb-jellystat-db Healthy 11.
cbj11200 wrote:
✔ Container sb-jellystat Started 11. [+] up 12/12 ✔ Container sb-gluetun Healthy 17. ✔ Container sb-jellyfin-media Started 12. ✔ Container sb-seerr Started 12. ✔ Container sb-bazarr Started 12. ✔ Container sb-deunhealth Started 11. ✔ Container sb-prowlarr Started 16. ✔ Container sb-qbittorrent Started 16. ✔ Container sb-sonarr Started 16. ✔ Container sb-lidarr Started 16. ✔ Container sb-radarr Started 16. ✔ Container sb-byparr Started 16. ✔ Container sb-sabnzbd Started 16. [+] up 5/6 ✔ Container sb-cadvisor Started 1. ✔ Container sb-grafana-init Started 1. ✔ Container sb-prometheus-init Exited 1. ✔ Container sb-node-exporter Started 1. ⠦ Container sb-prometheus Starting 1. ✔ Container sb-grafana Recreated 0. Error response from daemon: failed to set up container networking: Address already in use
tomspark wrote:
Found it — and this one's our bug, not your box. Your metrics config pins Prometheus to a fixed network address (172.20.28.2), but Docker hands that same address to one of the other metrics apps first, so Prometheus can't get it — that's the 'Address already in use', and it's exactly why nuking containers and networks never helped (the conflict is baked into the config). Nothing actually needs that fixed address, so removing it lets Prometheus grab any free one. Quick fix on your box: sudo sed -i '/ipv4address: 172.20.28.2/d' /opt/sparkbox/modules/metrics/docker-compose.yml sudo sparkbox up metrics That deletes the one line causing it, then brings metrics up — Prometheus should finally stay running. I'm also shipping a proper fix so this never bites anyone again. (And yes, that 'no such service: metrics' on sparkbox down is a separate real bug — already flagged to fix.) Let me know if Prometheus comes up green!
cbj11200 wrote:
colin11200@armament11200:~$ sudo sed -i'/ipv4address:172.20.28.2/d'/opt/sparkbox/modules/metrics/docker-compose.yml [sudo: authenticate] Password: Usage: sed [OPTION]... {script-only-if-no-other-script} [input-file]... -n, --quiet, --silent suppress automatic printing of pattern space --debug annotate program execution -e script, --expression=script add the script to the commands to be executed -f script-file, --file=script-file add the contents of script-file to the commands to be executed --follow-symlinks follow symlinks when processing in place -i[SUFFIX], --in-place[=SUFFIX] edit files in place (makes backup if SUFFIX supplied) -l N, --line-length=N specify the desired line-wrap length for the l' command --posix disable all GNU extensions. -E, -r, --regexp-extended use extended regular expressions in the script (for portability use POSIX -E). -s, --separate consider files as separate rather than as a single, continuous long stream. --sandbox operate in sandbox mode (disable e/r/w commands). -u, --unbuffered load minimal amounts of data from the input files and flush the output buffers more often -z, --null-data separate lines by NUL characters --help display this help and exit --version output version information and exit If no -e, --expression, -f, or --file option is given, then the first non-option argument is taken as the sed script to interpret. All remaining arguments are names of input files; if no input files are specified, then the standard input is read. GNU sed home page: <https://www.gnu.org/software/sed/. General help using GNU software: <https://www.gnu.org/gethelp/.
tomspark wrote:
Ah, that's on me — the spaces in that command got mangled on the way to you, and sed choked on it. Let's skip sed and edit by hand, it's clearer anyway: sudo nano /opt/sparkbox/modules/metrics/docker-compose.yml Once it's open: press Ctrl+W, type 172.20.28.2, and hit Enter — it jumps straight to the line. Press the Home key to go to the start of that line, then Ctrl+K to delete the whole line. Save with Ctrl+O then Enter, and exit with Ctrl+X. Then: sudo sparkbox up metrics That clears the line pinning Prometheus to an address that's already taken, and it should come up and stay. Let me know how it looks!
cbj11200 wrote:
That worked, thank you very much!!