In-house telemetry pipelineMetrics
KBVE's in-house observability platform — a scoped Rust/axum service that ingests client telemetry into a self-hosted ClickHouse database, replacing Sentry, Datadog, and PostHog with one pipeline across three lenses — errors, performance and product.
Three lenses, one session
One pipeline exposes errors, performance, and product lenses, a table each, correlated by session_id — no SaaS error/metrics tooling.
- Sinks — telemetry.{errors,perf,events}_distributed.
- Grouping — sha256(project + type + top-5 frames).
What it gives you
Features
Three lenses, one session
Errors, performance, and product views, a table each, correlated by session_id, replacing Sentry / Datadog / PostHog.
Off-thread batched inserts
A background tokio mpsc flusher batches rows by size and interval through the shared jedi ClickHouse client — no new database plumbing.
Hardened ingest
Requests are CORS-allowlisted, per-IP rate-limited, body-capped, and run through a noise/PII sanitizer before insert.
Client-agnostic
A platform field (web / unreal / unity / …) lets every client feed the same pipe, so the errors lens extends to game clients with no schema changes.
What it is
In-house observability
services/metrics is a scoped Rust/Axum service that ingests client telemetry into a
self-hosted ClickHouse telemetry database. It replaces SaaS error/metrics tooling
(Sentry / Datadog / PostHog) with one pipeline exposing three lenses — errors,
performance, and product — correlated by session_id.
Each lens is its own table rather than one wide table: a Web Vitals sample settles when the browser says so (INP can land minutes after LCP), and a wide row would have to be rewritten after the fact, which MergeTree does not do cheaply. Rolling the samples up into quantiles is a read-side concern instead.
Ingest path
Pipeline
@kbve/observ (browser SDK) → POST metrics.kbve.com/api/v1/ingest/{errors,perf,events} → services/metrics (axum, port 5500) → one queue + one flusher per lens → jedi ClickHouseConfig::execute_insert (JSONEachRow, batched) → telemetry.{errors,perf,events}_distributedOne queue per destination table, not one shared queue: a rejected insert takes its whole batch with it, so a malformed perf sample would otherwise drop the errors queued beside it.
| Route | Lens | Read as |
|---|---|---|
POST /api/v1/ingest/errors | window errors, promise rejections | GET /api/v1/groups, GET /api/v1/events |
POST /api/v1/ingest/perf | Web Vitals (LCP/INP/CLS/FCP/TTFB) | GET /api/v1/perf |
POST /api/v1/ingest/events | named product events | GET /api/v1/product |
The read routes are staff-gated; the ingest routes take a shared token.
A project named in METRICS_AUTH_PROJECTS takes nothing anonymous: its events
come from a signed-in Supabase user, whose sub becomes the row’s user_id
whatever the body says, or from a service presenting METRICS_SERVICE_TOKEN as
x-kbve-service. The anon key is a signed token too, so only
role: authenticated counts as a user. Every other project is taken as before. Note that
/api/v1/events is the read side of the errors lens — it lists the events behind
a fingerprint — while product events are written to /api/v1/ingest/events and read
at /api/v1/product.
The ingest path reuses the shared jedi ClickHouse client — no new database
plumbing. A background tokio mpsc flusher batches rows by size and interval.
Requests are CORS-allowlisted, per-IP rate-limited, body-capped, and run through a
noise/PII sanitizer. Error grouping uses a sha256(project + error_type + top-5 stack frames) fingerprint.
Sanitizing is per lens and deliberately strict about what reaches a
LowCardinality column. An unknown perf metric is dropped rather than defaulted —
the read path groups by it, so coercing junk to lcp would poison a quantile
instead of losing one row — and a non-finite value is rejected outright, because
NaN serializes to JSON null, which ClickHouse refuses for a Float64 and which
fails the entire insert batch rather than the offending row.
/readiness checks that every table ingest writes to exists, not just that
ClickHouse answers: SELECT 1 passes against a database with no schema at all, and
a half-applied schema is exactly the case where the service accepts one kind of
event and discards another. The answer is re-verified on a window
(METRICS_SCHEMA_RECHECK_MS, default 30s) rather than latched, so a cluster that
loses its schema after startup is noticed instead of being reported ready forever.
Client-agnostic
The platform field (web / unreal / unity / …) lets every client feed the
same pipe, so the errors lens shipped for the frontend extends to game clients
without schema changes.
Runtime
Deployment
Runs in the kbve namespace behind the Cilium Gateway at metrics.kbve.com,
reusing the existing clickhouse-credentials ExternalSecret.
The production DDL is the script in kube/metrics/manifests/telemetry-ch-setup-job.yaml,
an ArgoCD hook job; schemas/clickhouse/schema/telemetry.sql records the desired state
of the same tables. The e2e harness runs a real two-shard cluster and applies that same
setup script, so there is one artifact rather than a production one and a single-node
stand-in.
The same image also runs as an hourly triage CronJob: it reads the error groups
over a window, and for each group past an event or session threshold opens a GitLab
issue labelled by project area, or notes on the existing one and reopens it when
the error comes back after it was closed, so a new crash becomes a ticket
without anyone watching the dashboard.
Serving more domains is a gateway concern, not a deployment one: additional
hostnames (metrics.rareicon.com, …) go on the existing HTTPRoute and reach the same
pods. Tenants are separated inside the one process by METRICS_PROJECT_RATE_LIMIT_PER_MIN
and the CORS allowlist.
Recovery covers the schema, not the data. There are no ClickHouse backups and
every table carries a 30-day TTL, so a wipe loses the telemetry in it permanently.
The e2e suite drills the schema half against a real cluster: it drops the database
across both shards, asserts readiness goes degraded, re-runs the production setup job
and asserts ingest resumes. It also pins the privilege that job needs — revoking
REMOTE from the ingest user must make the script fail, because that is the grant
whose absence wedged this app for 23 days.
Questions
Frequently asked
What is the Metrics service?
Metrics (services/metrics) is KBVE's in-house observability platform, a scoped Rust/axum service that ingests client telemetry into a self-hosted ClickHouse telemetry database. It replaces SaaS tooling like Sentry, Datadog, and PostHog with one pipeline exposing errors, performance, and product lenses, one table per lens, correlated by session_id.
How does the Metrics ingest pipeline work?
The browser SDK @kbve/observ POSTs to metrics.kbve.com/api/v1/ingest/errors, /perf or /events, the axum service on port 5500 reuses the shared jedi ClickHouse client to batch JSONEachRow inserts, and each lens has its own queue and flusher writing to its own table. Requests are CORS-allowlisted, rate-limited, body-capped, and PII-sanitized.
How is Metrics client-agnostic across web and game clients?
A platform field (web, unreal, unity, and more) lets every client feed the same pipe, so the errors lens built for the frontend extends to Unreal and Unity game clients with no schema changes.
Discuss on the forum
Questions, fixes and notes on this page live in the KBVE forum.