Upgrading a running installation

From 0.3 on, bringing a machine to a new release is two commands, run from the workstation by the person who owns the machine. Read the notes of every release in between first: a release that asks for more than this says so below, under its own heading.

1. The CLI. Install the new binary, or git pull a checkout and run bin/test.sh there. Both read the same ~/.config/sitesolide/config.json.

   curl -fsSL https://github.com/cthiriet/sitesolide/releases/latest/download/install.sh | sh

2. What would change. It reads the machine and changes nothing there:

   sitesolide upgrade --dry-run

   Every component is listed up to date, out of date with what differs and what would run, or missing. A component is out of date when what this release would install differs from what is installed, measured on both sides, never assumed from version numbers.

3. The upgrade. It redeploys the components found out of date, and only those, one after the other:

   sitesolide upgrade

   Run it again afterwards: every component reads up to date, and nothing changes on the machine. Another installation is upgraded with SITESOLIDE_CONFIG_DIR=<dir> before the command, as for every command.

Upgrade the CLI when traffic is lowest: a dashboard, a portal or a Caddyfile redeployed reloads Caddy, which resets the connections being opened at that instant, for about a tenth of a second; a new drop-in restarts it.

Each component is checked first, reading only, then redeployed only if the check found a difference, through the very script or sitesolide deploy that sitesolide setup runs for it. The check is what setup checks for the component (active, enabled, its files present), plus:

A component built by its script, the steward or the monitor for instance, is measured by the script itself: --fingerprint builds it exactly as it would to install it and prints the SHA-256 of every file it would lay, which the check compares with sha256sum on the machine, the measure the script verifies after installing. The same sources built by the same Bun give the same bytes; a binary and a checkout built with another Bun may differ, and then upgrade redeploys what they build, once.

The order is the one this page has always given: the backup install before the steward, whose unit opens the backups' folder only if it exists when the steward starts; the egress proxy before the steward too, which starts again after it so that it may write the connectors' folder; the root components that only listen before the dashboard, which they accept old and new; the collector, the gatekeeper and the installer after the dashboard; the Caddyfile, the shared service, the portal and the monitor last.

• Install a component. One the machine does not carry is reported missing, and left alone: sitesolide setup, run again for that machine, installs it, the backups, the installer and the egress proxy of a machine set up with --minimal for instance.
• Touch a secret. No file of /etc/sitesolide is read, no password is drawn nor rotated: those are setup's first install and the dashboard's.
• Force. A unit or a Caddy block someone edited on the machine stays as sitesolide deploy leaves it: a unit is reported and kept, a block stops the component with the differing lines. sitesolide deploy --force in that component's folder of a checkout replaces it, on your decision.
• Stop or start Caddy but through bin/deploy-caddy.sh and systemctl; never caddy stop nor caddy start.

A failure stops at its component and says which, the end of what its script printed, and the command that shows more, such as ssh deploy@203.0.113.10 'sudo journalctl -u sitesolide-steward -n 50'. The components before it are upgraded, the ones after it untouched. Fix the cause, then run sitesolide upgrade again: the components already up to date are skipped, and it resumes at the one that failed.

• Each site. A site is redeployed by sitesolide deploy in its folder, when you next deploy it; upgrade only brings the platform's own components. The units a release generates reach the apps already deployed without their code through sitesolide refresh --all, see commands.md.
• The loopback rule. Setup lays it; a release that changes it says so in its notes, and bin/deploy-loopback.sh close lays it again, from a checkout or from the kit the binary unpacked into ~/.cache/sitesolide/<version>-<hash>/. It changes what every service may reach, and keeps its own safety net.

A machine upgrades one release at a time: from 0.1, follow From 0.1 to 0.2 first, then From 0.2 to 0.3, then sitesolide upgrade for every release after, reading the notes below for each, Access: one registry first, then Anyone with the code, from the dashboard.

An app could make any system call its account was allowed. The platform's own services ran under systemd's @system-service filter; the units generated for projects did not, so a compromised app reached every call of the kernel, the ones that trace a process, mount or load a module included, and any flaw of the kernel behind them. Every generated service unit now carries SystemCallArchitectures=native, SystemCallFilter=@system-service and SystemCallErrorNumber=EPERM: a call outside the set fails with EPERM, the process goes on.

What sitesolide upgrade brings: the installer, with the new generator, so that a deployment by token writes the same unit as one by SSH; and the dashboard and the portal, deployed as projects, whose units take the filter then. Every other app takes it at its next deployment. A unit 0.6 or an earlier release wrote is the generator's own and is replaced without --force; one edited by hand stays as it is, and says so.

Every app at once, without sending its code: sitesolide refresh --all, after the upgrade, gives every app on the machine the filter, and the port rules too to an app last deployed before 0.6.0. It writes each project's units again from the manifest its last deployment left on the machine, restarts its services and checks them as deploy does, one project at a time, and stops at the first that does not hold. sitesolide refresh --all --dry-run first lists the units it would write; see commands.md.

What changes for you. Nothing for a server in Bun, Node, Python or Go. Each was deployed under the filter on a Debian 13 machine on x86_64, Bun 1.3.14 and 1.4.3, Node 20, Python 3.13 and Go 1.24, serving, writing its data, starting a child process, a thread or a worker. One call was refused: Node asks for memory protection keys as it starts (pkey_alloc), which systemd's default action would have answered by killing it, and which EPERM turns into a Node that goes on without them. A program that needs a call outside the set, a profiler that traces, gets EPERM and says so in its own error; give it that call in a drop-in, see Isolation.

Going back. A drop-in holding [Service] and an empty SystemCallFilter= lifts the filter from one unit. The previous release's deploy reads a unit with the filter as edited by hand and leaves it, unless run with --force.

One project could take the egress proxy down for every other. The connectors' port counted no connections, read an answer as fast as its upstream sent it whether the app read it or not, and read a body as fast as the app sent it whatever the upstream took: thousands of idle connections from one project, one large answer read slowly, or 32 bodies to a slow upstream, ran the proxy out of descriptors or of memory, and every project's egress and connectors with it. The connectors' port now runs on the same raw listener as the CONNECT port, counts its connections with it, and streams both ways under the same byte budgets. A review of the rewrite closed what it brought or kept: a body in one-byte chunks cost the heap twenty times what the budgets counted, an upstream connection given up on stayed open, a TRACE went up with the credential the proxy adds and came back echoed, a GET or a HEAD carried a body an upstream could read as a second request, and a header byte past ASCII was sent twice encoded, on both ports.

What sitesolide upgrade brings: the egress proxy alone, the steward restarted after it as always. Nothing else moves, and nothing on the machine changes form: the connectors, the grants and the audit are read as they were.

What changes for you. Nothing, unless an app calls a connector:

• A connector call is one connection, answered Connection: close, with a TLS connection of its own upstream, whose certificate is checked against the connector's host name. A project's connections count against its 128 on both ports together. A call is refused 503 past its project's 32 under way, counted until the last byte of its answer and no longer its headers, past the proxy's 256, or while its project or the proxy holds too many bytes nobody has read; an answer whose head does not come within 30 s is a 504, no longer a 502.
• An answer reaches the app as the upstream sent it, compressed only if the app's own client asked with Accept-Encoding, as Bun's fetch does and decodes.
• Refused: HTTP/1.0, 505; a request with both Content-Length and Transfer-Encoding, 400; TRACE, TRACK and CONNECT, 405. A GET or a HEAD goes up without a body.

Going back. sitesolide upgrade with the previous binary puts the previous proxy back; nothing it reads changed.

A review of the whole platform found ways past its boundaries, and this release closes them. None took more than what the platform hands out on purpose: an app deployed on the machine, a token, a repository someone cloned and deployed over SSH, a page on a sibling subdomain.

Every app could read every preview code. The gatekeeper wrote the preview locks' fragment, /etc/caddy/locks/verrous.caddy, readable by every account, with every code in force in the clear, and a project's service sees /etc/caddy. The fragment is now root's and Caddy's group's, 0640. Any code drawn before this release may have been read by any project: draw each again, below.

An app could listen where others are sent. The loopback rule judges who connects to a port, never who listens on it. An app could wait on the portal's port while the portal restarted, during an upgrade, and answer every sign-in check as it liked; on the egress proxy's ports, it read every project's outgoing calls. Every generated unit now lets a server listen on its own port alone among Caddy's admin API, 2019, the range 3000 to 3099 and the proxy's 3128 and 3129, and a worker or a scheduled service on none of them (SocketBindAllow=, SocketBindDeny=); a service's backup command, the manifest's install and the extraction of a token's archive too. The slug public is refused as well: a request for .<zone>, an empty first label, landed in the folder a project of that name would have, its manifest and its code served past its portal.

A cloned repository deployed over SSH could reach the machine's files. deploy now refuses, before anything is pushed:

• a secret file that is not the project's. The unit reads every declared file as root: dashboard-backup.env handed the code the backups' bucket keys and passphrase, cms.env the secrets of cms. A project declares <slug>.env and <slug>-<name> files alone, never the machine's own nor one a more specific site deployed there holds, and a file already on the machine must belong to site-<slug>;
• a publicDir that is a link out of the repository, which sent ~/.ssh and served it to anyone;
• a link among the public files that leads out of them, which Caddy follows on the machine: robots.txt -> /etc/caddy/cloudflare.env served the zone's DNS token.

sitesolide remove deleted every secret file the folder's manifest named, dashboard.env included; it now deletes the project's own alone, and says which it kept.

A token could do more than its scope said. Its deployment could switch the project's domain on, which sitesolide domain --activate does only after checking the DNS: the machine's switch now stands. It could create a project under the name of one removed and inherit what the machine still keeps under that name, the snapshots its creator then restores as Admin and the connector grants: no token creates such a name now, reserved. Its refused changes of access went to the journal unbounded, and pushed out every change of secrets made in the hour; its accepted ones could fill the access log, which then lets nobody more in for 180 days but the owner over SSH. Both are bounded now. An expired token still counted toward the 200 live ones, so that tokens a second long stopped everyone minting. And the CLI sent its token to whatever address SITESOLIDE_API named: it now sends the token it keeps to the dashboard login checked it against, and nowhere else.

Smaller ones. A WebSocket opened from a sibling subdomain reached a restricted site with its visitor's cookie: the portal now wants the site's Origin on a handshake, a GET though it is. Signing in with password access cleared the failures counted against the owner's password; a sign-out minted a ticket for anyone who posted, which signed out of the portal whoever opened it. The dashboard could be framed by a page on another subdomain, read a sign-in body of any size, and the collector, root, followed a link at the place of the dashboard's previous snapshot. The settings file of the backups, written from the Secrets page, could name the program root runs as restic. A connector's request body sent in chunks went past its 10 MiB, and a call whose app hung up kept its answer streaming into the proxy. A value with a Unicode noncharacter made systemd drop every variable of its file; a secret file created under the name of an earlier one kept that one's previous version; a project deployed over SSH as dashboard-backup listed the backups' keys in its own Secrets; a token's label could carry a terminal control. Each is closed, and each component's README says how.

What sitesolide upgrade brings, in its usual order: the backups, whose settings file no longer moves root's programs or folders, the zone read after it, and whose service backup commands listen on none of the machine's ports; the egress proxy, which counts a connector's body as it streams and lets an upstream go when its caller hangs up; the steward, which keeps from every token the name of a removed project the machine still keeps snapshots or grants of, bounds a token's changes of access, counts live tokens alone, refuses the labels and the values above, creates a secret file with no previous version, keeps the machine's files to the dashboard, and reads the portal's answer within a bound; the dashboard, with its sign-in body bounded, its pages refusing to be framed, the collector's new reading, and its own unit with the bind lines; the gatekeeper, which writes the fragment 0640 to Caddy's group, and whose script closes the one in place; the installer, which keeps the machine's domain switch and writes the bind lines, its runs of install and of the extraction listening on none of those ports; the shared service, whose only change is a comment of src/locks.ts; the portal, with the WebSocket's Origin, the sign-out ticket, and the failures it keeps until the owner signs in or a quiet day passes; the monitor, whose unit reads the zone after dashboard-monitor.env, so that a SITESOLIDE_ZONE written from Secrets never points its probes elsewhere. Caddy's drop-in, the collector's units, the Caddyfile and the loopback rule do not move. No file on the machine changes form: team.json, the access registry and its log, the secret files and the backups' index are read as they stand.

Then by hand, once, after sitesolide upgrade has run to its end:

1. Check the preview locks' fragment. The gatekeeper's script, which upgrade runs, gives it to Caddy's group and closes it; a machine that never opened a site with a code has none.

   ssh <server> 'sudo stat -c "%U:%G %a" /etc/caddy/locks/verrous.caddy'
   # -> root:caddy 640

   Anything else, 644 from an upgrade stopped before the gatekeeper, is closed by hand, the group first, so that Caddy never meets the file closed to it. Nothing reloads, nothing served changes:

   ssh <server> 'sudo chgrp caddy /etc/caddy/locks/verrous.caddy && sudo chmod 0640 /etc/caddy/locks/verrous.caddy'

2. Draw a new code for every site opened with a code. Every project could read the old ones. The sites whose general access is Anyone with the code:

   ssh <server> "grep -lE '\"lock\": *true' /srv/sites/*/sitesolide.json"

   For each, New code in its Access section, under the unlock, or sitesolide lock --new-code in its folder; then send the new link to whoever should still have it. The old link stops opening the site at once.

3. Deploy every app again, so that its units gain the bind lines: sitesolide deploy in each project's folder, over your SSH, or a deployment with its token. The dashboard and the portal are done, upgrade having just deployed them. A static site has no unit. A unit the previous release generated is recognised as the generator's own and replaced without --force. Check one:

   ssh <server> 'systemctl cat <slug> | grep -c SocketBindDeny'
   # -> 3, and as many in each <slug>.<name> of the project

   Deploy them after upgrade, not before: the installer from before reads a unit with the new lines as edited by hand, and stops that project's token deployments on edited-by-hand until the installer is upgraded.

4. Check that systemd enforces those lines. It does through BPF, and a systemd built without it ignores them without a word, the units starting as before:

   ssh <server> 'systemctl --version | grep -o +BPF_FRAMEWORK'
   # -> +BPF_FRAMEWORK; nothing printed, the lines hold nothing on that machine

5. Check that no project is named public, a name now refused:

   ssh <server> 'test -e /srv/sites/public && echo "a project named public" || echo none'

   One that is there serves its manifest and its code at .<zone> until it is removed. This release's CLI refuses its manifest, the remove command included: run sitesolide remove --confirm public in its folder with the previous binary, then deploy it under another slug.

Worth a look, reading only: the links among the deployed sites' public files, ssh <server> 'sudo find /srv/sites/*/public -type l -exec ls -l {} +'. One that leads out of its folder, which the next deploy of that site refuses, had Caddy serve its target until now: change what it pointed at, then deploy the site with the link replaced. And the names in Sites > dashboard > Secrets > dashboard-backup.env: the bucket's BACKUP_S3_*, BACKUP_ENCRYPTION_PASSPHRASE, the retention, the reserve and the child's time, nothing else. A name you did not write there was written by whoever held the dashboard's unlock.

What changes for you.

• A changed manifest no longer needs --force for its block. A block in service that is exactly what the manifest deposited on the machine generates is the generator's own, and deploy replaces it, as the installer already did for a token: upgrade, which never forces, would otherwise stop at the dashboard, whose manifest gains the headers that refuse framing. A block edited by hand on the machine still stops deploy until --force.
• Over your SSH, deploy may stop on a manifest that deployed before, saying which file or link, with nothing pushed: secrets: a file that is not this project's, publicDir leads outside the repository, or carries links that lead out of publicDir. Rename a secret file after the project, <slug>.env or <slug>-<name>, and move its variables there from Secrets; replace a link with the file it points to. A manifest in your sites repository keeps a publicDir that leads anywhere, as its source already did. deploy also reads, before anything is pushed, who owns each declared file on the machine, through sudo, never its content.
• remove keeps what is not the project's, and prints each with its reason, kept: dashboard.env belongs to the machine, root's, and no project reads it. A file it kept that really was the removed project's, made by hand under another name, you remove yourself.
• A token no longer creates a name a removed project left things under. The refusal, reserved, counts the snapshots and the connector grants left, never what they hold. Deploy it yourself over SSH, which is how a removed project comes back with its data; or withdraw the grants on the Connectors page and forget the snapshots by hand (dashboard/src/backup/README.md), and a token may create it again.
• Each app's units change at its next deployment, without --force. A bind refused fails with EPERM: a server that opened a second port of the machine's, 2019, 3000 to 3099 or 3128 and 3129, or UDP on its own port, and a worker, a scheduled service or a backup command that listened inside that range. Declare a service of its own, with its own port, for what listens there. A port outside those stays free. A unit edited by hand on the machine stays as it is, without the lines, until --force.
• A token's deployment keeps the machine's domain switch. A domain it declares, a new name or alias, stays inactive until sitesolide domain --activate in the project's folder over your SSH.
• The slug public is refused, to the owner as to a token.
• A token fills half of the access log at most. Past half, what it would let in, Can open given or a project created, is refused not-available until older rows age out; its removals still work, and people and you are not held back. Its refused changes of access count among its 60 refusals an hour, then too-many-attempts.
• Live means neither revoked nor expired, for the 200 tokens of the machine as for the ten of a person.
• A label with a control character is refused, C1 included. A label minted before that carries one is printed by the CLI with ? in its place, when its person leaves.
• A value with a Unicode noncharacter is refused, U+FDD0 to U+FDEF and the last two code points of every plane, from the dashboard, a token or the CLI: systemd would read none of the file's variables.
• A secret file created from the dashboard or by a token has no previous version: one kept under its name, from an earlier file, is dropped first.
• A WebSocket on a restricted site needs the site's Origin. Every browser sends it; a client of your own outside a browser sends it too, or is refused 403.
• A site's sign-in failures are no longer cleared by password access signing in, only by the owner's own success, or forgotten after 24 hours with none.
• A sign-out with a provider ends the portal's session of the person who signed out alone, and only the owner's ends whatever session that browser holds; password access is signed out of the site and sent home. A sign-out link opened in another browser ends nobody's session there.
• The backups' settings file sets the bucket, the retention and the reserve. A BACKUP_RESTIC, SYSTEMCTL, SITES_DIR or any other variable that moved a program or a folder is ignored under systemd, and so is a SITESOLIDE_ZONE there.
• The dashboard refuses to be framed, Content-Security-Policy: frame-ancestors 'none' and X-Frame-Options: DENY, and a sign-in body over 8 KiB, 413.
• A connector's request body over 10 MiB is refused 413 as it streams, body too large in the proxy's audit, whether it announced its length or came in chunks.

Whoever works with a token installs the new CLI too, from the same install.sh as step 1 above: the token's binding to its dashboard and the prompt that no longer echoes it are the CLI's. sitesolide login now writes the address it checked the token against beside it in the vault, team-token-api. A vault filled by an earlier login holds none, and its token goes to the api of the config.json beside the vault's folder, as before. A SITESOLIDE_API or a config.json naming another dashboard is refused unauthenticated with the vault's token: sitesolide login --url again, or SITESOLIDE_TOKEN beside it, as in a sandbox. A vault kept elsewhere, by vault or SITESOLIDE_VAULT, needs sitesolide login once more.

Going back. sitesolide upgrade with the previous binary. The previous gatekeeper writes the fragment 0644 again at its next change of a site's general access or code, every code readable by every project once more. A unit with the bind lines reads as edited by hand to the previous CLI, which keeps it, and to the previous installer, which stops that project's token deployments on edited-by-hand: sitesolide deploy --force in its folder, with the previous CLI, writes it without them. team-token-api stays in the vault, unread. The registry, its log and team.json stay as they are.

A token may now manage secrets. Minted with Manage secrets, it lists its projects' secret files and variable names, and sets and removes the variables of <slug>.env, never reading one back; any token that deploys a project also restarts it. From a workstation with a token, sitesolide secrets, secrets set, secrets remove and sitesolide restart do it, see access.md.

What sitesolide upgrade brings, in its usual order: the steward, which judges the new routes and keeps each token's hourly bounds; the dashboard, with the routes, Manage secrets in the New token dialog, and Token activity reading the steward's journal; the installer, whose refusal of a missing secret now names sitesolide secrets set. The backups, the egress proxy, the collector, the gatekeeper, the shared service, the portal, the monitor, the Caddyfile and Caddy's drop-in do not move. No file on the machine changes form: team.json keeps its version, and a token minted before reads as without the permission.

A dashboard ahead of its steward, deployed alone or with the steward's step failed, answers the new routes not-available, saying to run sitesolide upgrade, and refuses to mint a token with Manage secrets, the dialog saying the server's steward predates the permission. Running upgrade to the end mends both.

What changes for you. One thing right away: every token already minted restarts the services of the projects it deploys, a person's own while they are Developer or Admin there, 30 times an hour at most, each restart in the journal under the token. No permission is ticked for it: a token that deploys code already restarts the service at each deployment. Revoke a token whose holder should not restart. Secrets wait for someone to mint a token with Manage secrets: every token already minted, yours and everyone's, lists and manages none. Then:

• whoever holds that token sets variables without an unlock: ticking the box is that decision. Mint it for the projects that need it, with an expiry;
• an owner's token that does not deploy public sites changes the secrets of the Restricted projects it could deploy as they stand alone;
• the Tokens page's Token activity lists, beside deployments, the secrets a token set or removed and the restarts it made, and so does each project's Activity;
• over your SSH, sitesolide secrets and sitesolide restart point to the dashboard, as secrets did.

Whoever works with a token installs the new CLI too, from the same install.sh as step 1 above: the commands are the CLI's, and the binary they have predates them. A previous CLI refuses them before anything is sent: sitesolide secrets as a command that needs the owner's SSH, and sitesolide restart as a command it does not know, a usage error. An agent without the CLI calls the routes over HTTPS (access.md).

Going back. sitesolide upgrade with the previous binary. The previous steward reads team.json as it stands, the new key ignored, and knows no secrets route, which the dashboard answers not-available. The values tokens set stay in the files, and their lines in the journal.

The installer now refuses, before anything is written, a deployment by token whose project would read or take a secret file that is not its own: secret-taken, naming the file, never what it holds. That is a file kept for root (dashboard-backup.env, dashboard-monitor.env), a declared file already in /etc/sitesolide that is not a plain file of site-<slug>, and a file another site holds that a new project would take by its name, cms-smtp taking cms's cms-smtp.env. The steward refuses a token creating a project named after a platform project and a dash, dashboard-*, portal-*, api-*, analytics-*, landing-*, www-*, as reserved; a project of such a name already on the machine stays deployable.

What sitesolide upgrade brings: the steward, which refuses the new names; the dashboard, whose folder carries the sources of both, with nothing new for it; the installer, which judges the files. The backups, the egress proxy, the collector, the gatekeeper, the shared service, the portal, the monitor, the Caddyfile and Caddy's drop-in do not move. Nothing changes form on the machine.

What changes for you. A token's deployment that read a file of another project, or of the machine, stops at its next run, and says which file: move the variable into the project's own <slug>.env from Secrets, or, for a file created by hand, give it to site-<slug> (sudo chown site-<slug>:site-<slug>) if it is the project's. Over your SSH, nothing changes.

Going back. sitesolide upgrade with the previous binary: the previous installer reads the declared files whoever owns them, as before.

A project can run work that answers no request. Under services, an internal service without a port is a worker, which runs for good and listens on nothing; a service with a schedule, a systemd calendar expression, is started by a timer at the times it names and runs to its end. cpu caps a service's processor. A project of such services alone, with no publicDir, serves no page and its address says so. See docs/manifest.md.

What sitesolide upgrade brings: the components that read a manifest, rebuilt: the backups, the egress proxy, the steward, the dashboard, the gatekeeper, the installer and the monitor. The Caddyfile, Caddy's drop-in, the collector, the shared service, the portal and the loopback rule do not move. The units and blocks of the projects already deployed are generated as before, line for line: none reads as edited by hand, none needs --force.

What changes for you.

• Upgrade before deploying the first project with a schedule. The previous monitor and dashboard read a scheduled service between two runs, inactive by design, as a service down. The new monitor reports a failed run as a warning and a timer that stopped as an outage.
• deploy watches the services it restarted for ten seconds, a server or a worker, and fails when one is no longer active or was restarted by systemd meanwhile. A project that fell at once after its restart, and was restarted every two seconds behind a successful deployment, now fails its deployment: sitesolide logs says why. The wait runs beside the Caddy step and the check over HTTPS.
• deploy asks systemd on the machine whether it reads each schedule, before anything is written.
• sitesolide logs reads the journal through sudo -n. On a machine sitesolide setup installed, it printed "No entries" for every project: the deployment account is in no group that reads the system journal. It needs the account's sudo without a password, which every deployment already does.
• A token deployment no longer chooses a port for an internal service that names none: it is a worker. One the machine already gave a port to keeps it.
• A project whose services are all internal, with no publicDir, was refused, and now deploys as a project that serves no page.
• Removing a project clears the failed state of its units, which systemd otherwise keeps in systemctl --failed once their files are gone.

Going back. Remove first, with this release's CLI, every project that declares a worker, a schedule or cpu: the previous release refuses those keys, and would read a scheduled service between two runs as a service down. Then sitesolide upgrade with the previous binary.

Whose token, in the New token dialog, is a menu drawn by the page rather than the browser's: each person of People over a line saying where they deploy and whether they may create projects. A Viewer everywhere is no longer offered, since the steward mints them no token; with nobody to offer, the menu is left out.

What sitesolide upgrade brings: the dashboard alone.

What changes for you. Nothing beyond the dialog.

Going back. sitesolide upgrade with the previous binary.

The New token dialog lists the projects on the server to tick, where the owner used to type slugs from memory; creating projects is one more row of that list, New projects. A permission a chosen project cannot be deployed without is ticked for you and says which project needs it: public sites for a project that is not Restricted, or that exempts paths from the portal, and a domain for one that declares it. A token with nothing to deploy, no project and no New projects, is refused by the form.

What sitesolide upgrade brings: the dashboard alone. The tokens already minted are untouched, and the steward judges a new one as before.

What changes for you. Nothing beyond the dialog.

Going back. Deploy the previous release's dashboard: sitesolide upgrade with the previous binary. The tokens minted with this dialog stay as they are.

Every run now takes, first, a snapshot named machine: /etc/sitesolide, every project's secret files; /var/lib/sitesolide-steward, the access registry, the previous versions of the secrets and the steward's log; and /etc/caddy, the preview codes, the zone's settings, the DNS token and the domains table. It is read back, kept, copied to the bucket and checked like a project's snapshot. Until now a lost machine took all of it along: the data came back from the bucket, and every key had to be minted again.

What sitesolide upgrade brings: the backup component, which takes it; the steward and the dashboard, which never take machine for a site. Check after the next run, or at once with ssh you@your-machine 'sudo systemctl start sitesolide-backup.service': sudo cat /var/lib/sitesolide-backup/last-run.json names machine first, with a snapshot. Its files are put back by hand, never from a page: dashboard/src/backup/README.md.

What changes for you.

• deploy refuses the slug machine. A project deployed under that slug before this release is no longer a site to the steward nor to the backups: deploy it under another slug before upgrading, and remove the old one.
• The monitor's backup line counts machine among the projects, and names it when its snapshot fails.
• The secrets now sit in the bucket too, encrypted with BACKUP_ENCRYPTION_PASSPHRASE: whoever holds that passphrase and read access to the bucket holds every secret of the machine, as they already held every project's data.

Going back. Forget the snapshots of machine first, in the server's repository and in the bucket's: the previous steward takes any deployed folder for a site, machine included, and a project deployed under that slug with the previous CLI would list them and could restore every secret into its own data.

ssh you@your-machine
sudo -i
export RESTIC_REPOSITORY=/var/backups/sitesolide-restic RESTIC_PASSWORD_FILE=/var/backups/sitesolide-restic.key
restic forget --prune $(restic snapshots --path /machine.tar --json | python3 -c 'import json, sys; print(" ".join(s["id"] for s in json.load(sys.stdin)))')

Then the same with the bucket's repository, as Restoring by hand opens it. The previous component takes no new snapshot of machine.

Restricted and Anyone with the code now close a site on its own domain too, not only on its preview: the dashboard and sitesolide lock used to refuse a site that served its own domain, since the domain would have stayed open. The domain's other names, its aliases and the www the table adds to a second level domain, now send to the domain with a permanent redirect. They used to fall into Caddy's nameless block, which serves public/ and never wakes an app: an app on its own domain answered its www without its service, every /api/* there a 404.

What sitesolide upgrade brings, in its usual order: the steward and the dashboard, which offer all three accesses to a site on its own domain and name both addresses before a change; the gatekeeper, which checks the domain beside the preview after every change, and writes an app's block anew when the one in service is an earlier release's; the Caddyfile, whose nameless block lets a static site's code close its domain; and the portal, which judges a domain's visitors on its site's people. Two changes reach sites at once, and nothing else does until a site is deployed or its access changed:

• the nameless block no longer serves an app's folder: a name the table carries for an app and its block does not claim answers 404 instead of the app's public/, which skipped its portal. The www of an app deployed before this release is one such name, until the app is deployed again;
• the preview locks in service keep matching their previews alone; the gatekeeper writes all of them in the new form at its next change of any site's code.

Then each app site that serves its own domain, when traffic allows: sitesolide deploy in its folder. It says the block "was written by an earlier release, the current one replaces it", without --force. Check: curl -sI https://www.<domain>/ answers 301 to https://<domain>/, and https://<domain>/ answers as before. Until then its www keeps falling into the nameless block; restricting it or giving it a code from the dashboard writes the new block in the same change.

What changes for you.

• A site that opens with a code and serves its own domain stays open there until its stanza is written again: sitesolide lock in its folder, or New code in its Access section, after the deployment above. Either rewrites every site's stanza at once. The dashboard no longer calls such a code in place.
• sitesolide domain --activate on a restricted site, or one that opens with a code, no longer refuses the restricted one, and refuses either while what Caddy serves would not close the domain: an app's block other than the one its manifest generates (deploy it first), a Caddyfile from before (upgrade first), or a code's stanza from before (sitesolide lock first).
• A deployment refuses a domain name, an alias or their www that another project declares, active or not, and an alias in the served zone.
• The link that carries a site's code leads to its own domain once it serves one.
• bin/lock.sh state, which sitesolide lock --status runs, measures the domain with the code too.
• An app reading X-Sitesolide-* behind the portal receives them on its domain as on its preview.

Going back. A portal from before refuses the announcement of a domain's block, so a restricted site on its own domain is closed to everyone there, its preview unaffected: make it public before going back, or upgrade again. The previous gatekeeper refuses to change the access of a site whose block is this release's, as one it did not write: deploy such a site with the previous release first. The redirect of a domain's other names stays until the site is deployed with the previous release.

Who may do what on a project now lives in one place, the steward's access registry, and the portal reads a projection of it. sitesolide upgrade is all it takes; read this first, for what changes for the sites and for you.

Before you run it.

• An app that reads X-Sitesolide-Role must be updated, and deployed, before the portal is: the header now carries the role, admin for the owner's password and the admin emails, visitor, viewer, developer or admin for someone signed in with a company account, visitor for password access, which carries no X-Sitesolide-User. Its old values, member and guest, are gone. An app that let guest read only now tests role === "visitor" && user === null, or better, the role it means.
• Everyone with a role above Can open on a project opens its site when it is restricted: a Developer of a restricted site no longer needs Can open besides. Read each restricted site's people with access afterwards, sitesolide share in its folder.
• sitesolide members is gone: sitesolide share <email> --role <role> from a project's folder gives a role, sitesolide people lists everyone and sitesolide people <email> --may-create gives the right to create projects. sitesolide share --domain and --only-admins are gone too: a domain is written @acme.com, and general access, public or restricted, is the dashboard's.

What changes in the dashboard.

• One place per question. In a site, one Access section replaces its Access, Sharing, Guests and Members: its general access first, Public, Restricted or Anyone with the code, then its people with access, each with a role, and Add people. At the machine level, People, the owner's, replaces Members: everyone across projects, their roles, who may create projects, the domains; Tokens replaces Team, each token saying who made it. The old addresses, bookmarks included, lead to the new pages.
• Password access waits for the unlock when it is given from the dashboard, as a role above Can open does: the owner's password, or an Admin's forced sign-in, since it lets in someone from outside the company. So does a whole domain while OIDC_ALLOWED_DOMAINS is empty. Can open for a company account or one of the listed domains, lowering and removing still never wait, nor does restricting a site, and the owner's sitesolide share over SSH asks for nothing.
• Someone who can only open sites is told so when they try the dashboard, can-open-only, apart from someone on no list, no-role.
• Activity lists a change of general access as access.general, where it said door.update: a filter or an export that looked for the old name looks for the new one, or for access. to find every change of access. The rows written before the registry still read, in today's words.
• Changes of access are kept 180 days, and bounded. The steward writes every accepted change of access to a file of its own, access-log.jsonl beside its journal, seeded from the journal and topped up at each start with what it lacks; refusals stay in the journal. Its rows are never pushed out before 180 days: once it holds 20,000 of them, or 12 MB, a change that lets more people in is refused, log-full, until older ones age out, while removing and lowering still work, and you still change access from your workstation with sitesolide share and sitesolide people. Such changes are also counted, 120 an hour for each person, each token and you in the dashboard, 2,000 for you over SSH, too-many-changes past it.
• Every token from before becomes someone's, once, at the steward's first start. A token whose email, what you typed when creating it and shown as its label, is the address of a person who signs in to the dashboard, a role above Can open or the create right, becomes that person's, made by you: it is narrowed to their roles from then on, and revoked when they leave, removed or lowered so that they keep no role above Can open and no create right. Every other token becomes your own, its scope unchanged. To keep a shared CI token yours when its email is a person's: after upgrading, on the Tokens page, make a token with Whose token set to Mine and the same projects, put it in the CI, then revoke the one that became the person's.

What sitesolide upgrade runs, in this order.

1. The steward. At its first start it makes the registry, once, from members.json and the portal's database, then writes the portal's projection: migration.md says what it carries over, what it sets aside and why. The running portal, the old one, keeps deciding from its own tables, which still say the same. bin/deploy-steward.sh ends on access registry 600 root:root, its projection for the portal 640 root:site-portal. A store that does not read leaves no registry, and is tried again later; when it is the portal's database, a table past 16 MB (oversized-table) among others, sitesolide people --migrate-without-portal carries the rest over without it, and the tokens are made someone's at once.
2. The dashboard. Its access API speaks to the steward, and its pages are the ones above: a site's Access, People and Tokens.
3. The installer, which reads the registry when it starts.
4. The portal, last: from its first request it decides from the projection, and leaves the mark that keeps it from ever reading its old tables again.

Run them together, as sitesolide upgrade does: a change of access made in the old dashboard between the steward's step and the dashboard's would go to the portal's old tables, which nothing reads any more.

Check.

sitesolide people
cd <a restricted project> && sitesolide share
ssh deploy@203.0.113.10 'sudo curl -s http://127.0.0.1:3026/admin/access'

people lists everyone with their roles; share lists the site's people with access and says nothing of the portal; the portal answers "reading": "steward". A password the portal handed out before the upgrade still opens its site, as password access, and a cookie set before it is still valid. In the dashboard, a site's Access lists the same people as share, and People the same as people.

Going back is the previous commit of the steward, the dashboard, the installer and the portal, which read the old stores as they stood at the migration: every change of access made since is lost for them. See migration.md.

Deleting access.json is no way to start over. The steward makes it again at the next request from the old stores, as they stood at the migration, so every change of access made since is gone; within the next 30 seconds its sweep revokes the live tokens of everyone the rebuilt registry gives no rights. A deletion made by mistake is undone by putting back a copy of the file, root 0600: ssh deploy@203.0.113.10 'sudo install -m 600 -o root -g root <copy> /var/lib/sitesolide-steward/access.json'. Everyone's roles and access come back with it; the changes made to the rebuilt registry in between are lost, and the tokens the sweep revoked stay revoked, to be minted again. The machine's backups hold the steward's state from 0.4.2 on, an hour old at most; make a copy yourself all the same, sudo cp -p of the file, before you edit it by hand.

A site's general access is now chosen among its three from its Access section, Anyone with the code included: the owner and the site's Admins open a site with a code there, read the code, draw a new one, and go back to Public or Restricted, a switch between Restricted and the code in one transaction. sitesolide lock and unlock no longer change the machine themselves: they ask the steward on its owner socket, which launches the same gatekeeper. One path changes a site's general access, whoever asks.

What sitesolide upgrade brings, in its usual order: the steward and its POST /general routes, the dashboard, then the gatekeeper and its four templates, -on@, -off@, -code@ and -renew@. bin/deploy-gatekeeper.sh lays what they open for writing and cannot create, when missing: /etc/caddy/locks/, an empty /etc/caddy/locks-codes.json for the deployment account in 0600, and /srv/garde/. A machine that has locked a preview before carries them already, and they are left as they are, codes in force included. The Caddyfile changes by a comment only, which upgrade redeploys through bin/deploy-caddy.sh all the same.

Until the gatekeeper step has run, sitesolide lock on that machine says to run sitesolide upgrade, and the dashboard's Use a code is refused by a gatekeeper that does not know it: nothing changes, and running upgrade to the end mends both.

What changes for you.

• bin/lock.sh enable, code and disable are gone; bin/lock.sh state, which sitesolide lock --status runs, stays, and reads what the machine's manifest asks for rather than the repository's.
• The next sitesolide deploy writes into sitesolide.json the general access the machine carries, lock included, as it already did for portal; sitesolide lock writes it at once.
• sitesolide lock on a site that serves its own domain is refused, as the dashboard refuses it: the code would close the preview and leave the domain open. Switch the site back to its preview first, sitesolide domain --deactivate. LOCK_DESPITE_DOMAIN is gone with the script that read it. The release after this one closes the domain too, see A site closes on every address.
• A Viewer or a Developer of a site that opens with a code reads that it opens with one, no longer the code itself: its Admins and the owner hold it.

Going back. The previous release's bin/lock.sh writes the same files the gatekeeper does, sitesolide.json, the codes file and verrous.caddy: a workstation still on it keeps working, without the dashboard knowing what it did until the next snapshot. Going back to the previous gatekeeper leaves the two new templates installed, which nothing launches once the previous steward is back.

The backup component stores its snapshots with restic: one repository on the server, /var/backups/sitesolide-restic, and, with a bucket, a second one in it, <prefix>-restic, filled by restic copy. An hour now stores what changed since the hour before, compressed, rather than a whole archive; the bucket is read back with stock restic on any machine, the passphrase alone. What runs around it does not change: the copy as the project, the backup commands, the bounds, the status file, the restore and its rollback. See dashboard/src/backup/README.md.

What sitesolide upgrade brings, in its usual order: the backup install first, which now installs Debian's restic package if it is missing (and refuses one older than 0.18.0), draws the repository's key and initialises the repository; then the steward, which reads the snapshots from the index the run writes instead of the archives' folder, and the dashboard, whose Backups section shows what each snapshot added and the last daily check; the monitor last, which learns to warn on a failed check. The run's and the restore's units go from 256M to 512M of memory, restic running beside Bun.

Then a run by hand, which upgrade does not start:

ssh you@your-machine 'sudo systemctl start sitesolide-backup.service; sudo cat /var/lib/sitesolide-backup/last-run.json'

It imports the archives of the version before into the repository, each proved byte for byte, and writes the index the new steward reads; until it runs, within the hour otherwise, the Backups section lists no snapshot, the archives still on the disk. With a bucket, it also creates the bucket's repository with the server's chunker parameters, copies every snapshot there, and imports the objects the server no longer has. The archives and the objects are deleted seven days after their copy was verified.

What changes for you.

• Snapshots are named <folder>-<time>.tar, .tar.gz being the archives of the version before; sitesolide backups gains an ADDED column.
• The bucket's passphrase, BACKUP_ENCRYPTION_PASSPHRASE, is now the bucket repository's password: changing it in the dashboard alone cuts the bucket off. Change it with restic key add first, as the README says.
• On Backblaze B2, a lifecycle rule keeping prior versions 30 days, and the multipart one, as the README's The offsite copy says.
• A recovery drill from your workstation with stock restic, once: a passphrase never tried is a passphrase you do not have.

Going back. Within seven days of the import, the previous release's bin/deploy-backup.sh install, steward and dashboard find their archives and objects where they left them, and never see the repository. The snapshots taken since exist only in restic, restored by hand. Past seven days, the previous release starts from an empty history. The next release removes the import, and its install refuses a machine that has not imported everything: upgrade through this one.

0.3 changes how you install and drive sitesolide, not what runs on the machine. Nothing there needs to be redeployed, and no site reloads.

1. The CLI. Install the binary, or keep a checkout and git pull: both read the same ~/.config/sitesolide/config.json.

   curl -fsSL https://github.com/cthiriet/sitesolide/releases/latest/download/install.sh | sh

   From a checkout, bin/test.sh first, as always. The binary deploys dashboard/ and portal/ from the copies it carries, built at release time; a checkout builds them as before.

2. Terraform is gone. Your machine, its firewall, its DNS records and its Hetzner backups stay exactly as they are. ~/.config/sitesolide/terraform/ is no longer read: copy its two tokens into your password manager, then keep the folder as a record or delete it. Never run terraform apply or terraform destroy against that state from a 0.3 checkout: with the .tf files gone, Terraform plans the destruction of everything it created, the server first. Whoever wants Terraform to keep managing the machine keeps a 0.2 checkout for it.

3. sitesolide setup on this machine. It recognises a machine it did not install (no setup.json beside your configuration) and only reads it: it reports when every step is done and refuses otherwise, changing nothing. --dry-run shows what it finds:

   sitesolide setup deploy@203.0.113.10 --zone example.com --email you@example.com --dry-run

4. sitesolide upgrade --dry-run, then sitesolide upgrade, as above: a binary builds the components with its own Bun, and may find some of them to redeploy.

New machines are made with sitesolide machine create and sitesolide setup: see install.md.

How to bring a machine that runs 0.1 to 0.2 without any site going down, from a 0.2 checkout: sitesolide upgrade came later, and this jump also asks for gestures it never makes, a portal deployed with --force and components installed for the first time. Every command runs from the workstation, by the person who owns the machine, in this order. Each step works without the next, so the upgrade can stop anywhere and resume another day; each one says what to check and how to go back.

This order was rehearsed on a test machine installed from 0.1 and serving the landing, a static site, an app, a restricted site and analytics, with a probe opening a fresh connection to every site every second throughout. No request failed because of the upgrade itself. Every Caddy reload, though, resets the connections that are being opened at that instant, for about a tenth of a second: that is how Caddy behaves on every deploy, in 0.1 as in 0.2. Steps 3, 4 and 6 each reload Caddy, as does switching a site between public and restricted; in the rehearsal one reload out of ten fell on the probe and reset that second's requests. Upgrade when traffic is lowest.

The components each have a longer section of their own, linked at each step, with every check and every rollback in detail. This page is the order.

The CLI is stricter, and some deployments that passed in 0.1 are refused or behave differently. bin/test.sh validates every manifest of this repository and of your sites repository against the new rules: run it first, and fix any manifest it refuses before deploying anything.

The whole list, and why each changed: commands.md and dashboard/README.md.

git pull
bin/test.sh

Every test passes, the manifests of your sites repository included. Check that ~/.config/sitesolide/config.json names your sites repository under sites. If it names a folder that holds no project with a package.json, three tests fail for want of a project to read (no folder of the sites repository is forgotten, at least one application site is covered, and the one that reads the sites repository's manifests): that is the folder, not your manifests.

The drop-in gains Restart=always: a Caddy stopped through its admin API comes back in two seconds instead of staying down. Applying it needs a daemon-reload and no restart.

ssh you@machine 'sudo install -m 644 /dev/stdin /etc/systemd/system/caddy.service.d/override.conf && sudo systemctl daemon-reload' < infra/caddy/caddy.service.d/override.conf
ssh you@machine 'systemctl show caddy -p Restart -p StartLimitIntervalUSec -p MainPID'

Check: Restart=always, StartLimitIntervalUSec=0, and the same MainPID as before the command. Back: install the previous file, daemon-reload. Details: infra/README.md.

They accept the old dashboard and the new one, and change nothing served.

bin/deploy-backup.sh install   # the backup component's folders and units; starts nothing
bin/deploy-steward.sh          # the new routes: tokens, backups, connectors, the new secret files

The backup install comes first because the steward's unit makes its state folder writable only if it exists when the steward starts. Check: the steward's script verifies itself; the dashboard still shows every site's secrets. Back: the same scripts from 0.1.

cd dashboard && sitesolide deploy && cd ..
bin/deploy-gatekeeper.sh

The dashboard's block was written by 0.1's generator: deploy says "written by an earlier release, the current one replaces it" and replaces it without --force. Its install now runs as site-dashboard. This deploy also lays 0.2's Caddyfile: sitesolide deploy installs a block through bin/deploy-caddy.sh, which deposits the repository's Caddyfile with it. Its file servers start hiding .git and .env* here, and the landing starts taking the identity headers off what visitors send. The gatekeeper embeds the block generator: it comes right after, so that switching a site between public and restricted from the dashboard writes the current blocks. Check: the new pages, for tokens and connectors, and sections, Backups among them, appear and say what is not installed yet; with the dashboard unlocked, making a test site public, then restricted again, still works. Pick a site whose / answers 200 without the portal: the gatekeeper restores the previous block when the open site answers 404, as analytics does, which is a correct refusal and no sign of trouble. Back: deploy the previous dashboard/, run the previous gatekeeper script; the new tables of dashboard.db stay, unread.

bin/deploy-caddy.sh
cd portal && sitesolide deploy --force && cd ..

bin/deploy-caddy.sh answers nothing to do when step 3 already laid the Caddyfile; it is here for a machine where the dashboard was deployed some other way. The portal needs --force: its unit gains outbound network, to reach a sign-in provider, and its block gains the sign-in routes. Run cd portal && sitesolide deploy without --force first: it stops before pushing anything and prints the divergence, cut short after five lines. portal/README.md lists what it must be. Then run it with --force. Check: curl -s -o /dev/null -w '%{http_code}\n' 'https://portal.<zone>/admin/sharing/..%2f..%2fsante' answers 400; a protected site still opens with the cookie you already had; a password access still opens its site.

Each of these is independent and opt-in. None changes a site until you use it.

Nothing forces a redeploy. When you next run sitesolide deploy in an app's folder, its block is replaced without --force (a static site has no block, and its redeploy changes nothing in Caddy): a file server gains the hiding of .git and .env*, a protected site the identity headers, an open site the removal of forged ones. Until a site is redeployed it keeps the behaviour of 0.1, which includes an open app receiving whatever X-Sitesolide-* header a visitor sends: an app that starts reading those headers is redeployed before it trusts them.

Check out 0.1 and run steps 3 and 4 from it, then step 2. Sites already redeployed with 0.2 keep working; their next deploy from 0.1 needs --force once, since 0.1 does not know the newer blocks. Data written by the new components (tokens, backups, audit rows) stays on the machine, unread.