Events belong here, gauges do not

Functional Requirements — the events plane

The events plane is what one did: the runs it wrapped, the phases inside them, and the lines they printed, stored in this box's own ClickHouse and written through this box's own hcapi. Structural map: docs/architecture_platform.md (§The event path, §ClickHouse — the events). This file is the behavior contract, written from outside with no access to the implementation; every requirement is a check a change can pass or fail.

The plane's shape follows the estate's measurement of what a time-series store is for (~/greyandfast/garage/notes/research/prometheus-for-snc): ClickHouse is right for events and runs — heterogeneous rows with their own identity, queried by group and by time — and wrong for gauges, which are a fixed series set sampled on a clock. So the plane stores facts and lifecycle rows, and the fleet questions are GROUP BY over time, never a metric path. A new metric does not belong here; a new kind of event does.

The tables

Three tables, three questions.

• events is what happened. One row per event one emitted, with the project, the environment, the host, the run (and its parent, when a run was launched by another), the microsecond timestamp, the kind, the name, a monotonic per-run seq, a string-to-string attribute map, and free-form tags. Queryable for 180 days from ts.
• runs is what is happening now. One row per run, carrying the host, pid, tag, argv, started_at, ended_at (null while open), exit_code, signal, outcome, attempts, and uptime_ms. The table is a ReplacingMergeTree versioned on ended_at, so a run posted open and later posted closed collapses to one row and the closed version wins. No TTL: the lifecycle row outlives the log lines it produced.
• logs is what it said. One row per captured line: the run, the microsecond timestamp, the stream (stdout or stderr), and the line. Queryable for 30 days from ts.

Retention is expressed as a table TTL and enforced by ClickHouse dropping whole parts; a query at the boundary therefore sees only rows inside the window and must not assume a row exists at now - 180d (or now - 30d for logs). Every read that spans the boundary filters on ts explicitly — the TTL is the guarantee of eventual eviction, never a promise that the edge row survives.

The event JSON

one writes exactly this object per journal line, one line per event, and the flusher posts batches of them unchanged. v is the schema version, always 1 for this revision.

{
  "v": 1,
  "project_id": "uuid",
  "env": "string",
  "host": "string",
  "run_id": "uuid",
  "parent_run_id": "uuid or null",
  "ts": "RFC3339 with microseconds",
  "kind": "string",
  "name": "string",
  "seq": 0,
  "attrs": {"key": "value"},
  "tags": ["string"]
}

kind is the coarse class and name is the specific event. The closed set of kind values is lifecycle, phase, heartbeat, log, meta; name is a free lowercase snake string within that class. A consumer that needs the event's position in one run's story orders by seq, which is per-run monotonic from 0.

A name outside its kind's class is a client bug, not a new class: kind is what the store partitions and groups on, so it is a closed set and a change to it is a schema change. Adding a new phase is not — that is a new name under phase.

attrs carries only strings — a number, a duration, or a boolean is written as its string form so the map needs no schema negotiation. v is not a column: it is a version gate, consumed at the door and not stored.

runs and logs are not carried on this wire. runs is projected from the lifecycle/start and lifecycle/exit events, and logs from the log events' line attribute — the one binary emits one JSON stream and hcapi fans it into three tables.

The ingest endpoint

• POST /api/one/events accepts a JSON array of event objects. A batch is scored as a batch: the body is one array, and the array is the unit of accept or reject.

• On success it returns 202 Accepted with a body carrying the count accepted, e.g. {"accepted": 42}. 202, not 200: the write to ClickHouse may follow the response, and the flusher only needs to know the batch was taken.

• A batch whose any element's v is not 1 is rejected with 400 and the count of offending elements. A batch is never partially accepted on a version mismatch — a mixed batch is a client bug and fails whole.

• The endpoint is idempotent by (run_id, seq). Re-posting a batch that was already applied does not duplicate a row. The mechanism is two-part, and both parts are required:

  1. the writer derives an insert_deduplication_token per batch — a digest of the batch's ordered (run_id, seq) pairs — and every table in the cell sets non_replicated_deduplication_window = 1000, so ClickHouse discards a second insert carrying a token it has already applied;
  2. runs is additionally a ReplacingMergeTree keyed on (project_id, run_id), so even a batch that slipped past the window overwrites the existing run row instead of adding a second one.

  The token is the idempotency statement; the replacing engine is the backstop for the run table. A change that removes either half fails this requirement.

Failure behaviour

The collector is never the run's problem (ONEF-102: one performs no network call on the child's hot path).

• When ClickHouse is down, hcapi buffers to disk at $XDG_STATE_HOME/one/hcapi-spool/ in the same JSON lines it would have posted, and drains the spool oldest-first when the store returns.
• The spool is capped at 256 MB. On overflow the drop order is fixed: log lines are dropped first, oldest first; lifecycle rows are kept — every lifecycle event and the whole runs projection survive even when the log tail that explains them is gone. A full spool degrades the record's detail and never its skeleton.
• A rejected batch is retried, not dropped, with the same token — which is what makes the retry safe.
• one itself never blocks on any of this. A slow or dead collector is invisible to the child command; the queue drops oldest non-lifecycle batches and the exit code is the child's.

The cap and the drop order live here as the contract; the spool path and the implementation are hcapi's own.

Checks

A change passes the plane when all of these hold:

1. SHOW TABLES FROM onefrontier lists events, runs, logs, runs_latest, host_activity, failed_runs.
2. SHOW CREATE TABLE onefrontier.events shows PARTITION BY toYYYYMM(ts), ORDER BY (project_id, ts, run_id), and a 180-day TTL on ts.
3. SHOW CREATE TABLE onefrontier.logs shows a 30-day TTL on ts.
4. SHOW CREATE TABLE onefrontier.runs shows a ReplacingMergeTree whose version derives from ended_at and no TTL.
5. bin/demo-load.sh twice in a row leaves the same row counts (720 events, 60 runs, 360 logs), and runs_latest reads 60.
6. SELECT * FROM onefrontier.host_activity LIMIT 1 returns a host, an hour, a kind, an event count and a run count.
7. POST /api/one/events with v: 2 returns 400; with v: 1 returns 202 and the accepted count.
8. Re-posting a batch returns 202 and leaves the row counts unchanged.