# Monitoring

> Hear about a site that goes down, and about the server itself dying, with a heartbeat and a chat webhook.

`sitesolide setup` installs a monitor that checks the server every minute. Until you connect a heartbeat and a webhook, it writes to the server's journal only.

## What it checks

| Check | Alerts when |
|---|---|
| Caddy | Not running, or restarted by systemd on its own. |
| Every served site | No valid HTTPS answer, or a status of 500 or above. |
| Project services | A unit failed, stopped or crashing in a loop, or a scheduled run failed. |
| Certificates | One expires within 14 days. |
| Disk and memory | A disk 90% full, or less than 10% of memory available. |
| Backups | The last run failed, or is over 26 hours old. |

A failure is reported after two failed passes in a row, or three in five: within two to three minutes. You get one message when it goes down, one when it recovers, and the dashboard lists it under **Issues**.

## Create a heartbeat check

The heartbeat is a dead man's switch: the monitor pings an outside service every minute, and that service alerts you when the pings stop. It is what notices the server itself dying.

1. Sign up at [healthchecks.io](https://healthchecks.io). The free tier is enough, and your email is already an alert channel.
2. Add a check with a period of 1 minute and a grace of 5 minutes.
3. Copy its ping URL, `https://hc-ping.com/<uuid>`. Keep it secret: whoever holds it can hide an outage.

## Set the addresses

Create the root-owned alerting file on the server, once:

```bash
ssh <user@host> 'sudo install -m 600 -o root -g root /dev/null /etc/sitesolide/dashboard-monitor.env'
```

Then fill it from the dashboard:

1. Open the **dashboard** site, then **Secrets**, and unlock.
2. In `dashboard-monitor.env`, set `HEARTBEAT_URL` to the ping URL. The monitor reads it at its next pass, with nothing to restart.
3. For chat messages too, set `ALERT_WEBHOOK_URL`: a Slack, Discord or Google Chat webhook, or `https://ntfy.sh/<topic>` with a topic nobody would guess.

Both addresses must be `https`. The message format follows the address; `ALERT_WEBHOOK_FORMAT` (`slack`, `discord`, `googlechat`, `text` or `json`) sets it for any other.

> [!NOTE]
> Only a platform failure, such as Caddy, a full disk or the dashboard, sends the heartbeat to `<url>/fail`. One project down still pings success, and lists the failure in the ping's body.

## Check it worked

```bash
ssh <user@host> 'sudo journalctl -u sitesolide-monitor -n 20 -o cat | grep "checks," | tail -n 1'
```

```text
41 checks, 0 down, 0 failing; heartbeat ok, webhook idle
```

Look for `heartbeat ok`, and for the check to turn green on healthchecks.io within two minutes. Without a webhook, the line ends with `webhook unconfigured`.

## What an alert looks like

```text
sitesolide monitor, example.com: 1 down
DOWN https://shop.example.com/ answered 502 (since 2026-10-04 14:02 UTC)
```

When the site answers again, a second message says `RECOVERED`.

## Next steps

- [Backups](backups.md): the snapshots the backup check watches.
- [Your server](server.md): reboot and look after the server the monitor watches.

Full reference: https://github.com/cthiriet/sitesolide/blob/main/monitor/README.md