Incident response

Last updated 7 October 2026. Source: sheptra.core/docs/runbooks.md · sheptra.vantage/src/pages/docs/backup-and-restore.md

How Sheptra finds out something is wrong, what happens next, and how and when you hear about it. Written from the runbooks our operators actually follow.

What counts as an incident#

• Availability — an environment not serving, jobs not being picked up, timers firing late, a deploy that failed to roll.
• Data — loss or corruption of customer data, or a restore that rewinds it.
• Security — unauthorized access to customer data, a credential exposed, a vulnerability in the service or a dependency.

Detection#

Every component pushes traces, logs and metrics to one telemetry stack; nothing is scraped, so a host that goes quiet is itself a signal. Alert rules are written in the engine’s repository beside the code they watch and synced to the stack on every merge, so the rules and the runbooks cannot drift apart. Each rule has a runbook entry that answers the same three questions in the same order: what it means in terms of the engine, the one query that confirms or refutes it, and what to do — including what not to do, because several conditions here get worse if you scale out. The rules cover the oldest unclaimed job, queue growth, incident-rate spikes, each host and scheduler going dark, the telemetry pipeline itself, database connection saturation, server-error rate, late timers, stalled sweeps, dead-letter arrival and backlog, and ingress pause failures.

Alerts are grouped per environment and per host so one cause is one notification, and a pipeline outage inhibits the host alerts it would otherwise fake. The platform’s own deploy and domain failures alert separately.

Where this stands today. The rules, routing and runbooks exist and are synced from CI. The telemetry stack for the staging environment is being provisioned, so until it is, the hosts export nothing and the alerts exist on paper and in the pipeline rather than in anyone’s inbox. Alerts route to email first; a chat receiver follows. Sheptra is a small team and there is no 24-hour rota in the beta; response targets are agreed per contract on the Enterprise tier. There is no public status page yet.

Response#

1. Confirm or refute first. Every runbook names the one dashboard or query that says whether the alert is real before anyone acts on it.
2. Act by the runbook. Including its prohibitions: no bulk-resolving incidents to clear a queue, no adding replicas when the database’s connection budget is the bottleneck.
3. Roll back by digest. A bad release is reverted by pinning the previous image digests, applied by the same pipeline that deployed them. A fleet rollout verifies each cell after moving it and reverts in the same run if the checks go red; the schema is designed to stay compatible one release back so a rollback needs no migration.
4. Restore from backup is a fork of the database at a named second and a re-point of the cell at the fork — the sequence, its two repairs and its hazards are a written runbook. Recovery time is unmeasured until the first drill row exists; see backups and recovery.
5. Write it down. Each incident is recorded with its timeline, cause, the action taken and what changes as a result; a customer affected by one can ask for that summary.

Telling you#

Planned maintenance. A customer on a dedicated cell receives a notice before a minor release, with their maintenance window filled in: what changes, that database changes only add to the schema, that timers falling due during the restart fire late rather than being lost, that running instances are not interrupted, and that a release failing our checks is reverted automatically within the window. Patches ship with their release notes and need no window.

A security incident involving your data. We tell you without undue delay once we have confirmed it, with what happened, when, what data and which organizations were affected, what we have done, and what you should do. The binding notification period is in the data processing agreement.

Availability incidents. Reported on request in the beta and per the contract on paid plans. We publish availability figures when we can stand behind them with data, and not before.

Reporting a vulnerability#

If you believe you have found a security issue in Sheptra, write to contact@sheptracloud.com with the subject “Security vulnerability”. We acknowledge every report, keep you informed as we work on it, and credit you if you want. Please do not test against another customer’s data or run denial-of-service tests; there is no bug bounty programme at this stage.