Skip to content
rust · axum · observability · clickhouse · telemetry · docker

In-house telemetry pipelineMetrics

KBVE's in-house observability platform — a scoped Rust/axum service that ingests client telemetry into a self-hosted ClickHouse database, replacing Sentry, Datadog, and PostHog with one pipeline across three lenses — errors, performance and product.

Three lenses, one session

One pipeline exposes errors, performance, and product lenses, a table each, correlated by session_id — no SaaS error/metrics tooling.

  • Sinks — telemetry.{errors,perf,events}_distributed.
  • Grouping — sha256(project + type + top-5 frames).
metricsApp
Rust + axumStack
ClickHouseSink
port 5500Listen

What it gives you

Features

Three lenses, one session

Errors, performance, and product views, a table each, correlated by session_id, replacing Sentry / Datadog / PostHog.

Off-thread batched inserts

A background tokio mpsc flusher batches rows by size and interval through the shared jedi ClickHouse client — no new database plumbing.

Hardened ingest

Requests are CORS-allowlisted, per-IP rate-limited, body-capped, and run through a noise/PII sanitizer before insert.

Client-agnostic

A platform field (web / unreal / unity / …) lets every client feed the same pipe, so the errors lens extends to game clients with no schema changes.

What it is

In-house observability

services/metrics is a scoped Rust/Axum service that ingests client telemetry into a self-hosted ClickHouse telemetry database. It replaces SaaS error/metrics tooling (Sentry / Datadog / PostHog) with one pipeline exposing three lenses — errors, performance, and product — correlated by session_id.

Each lens is its own table rather than one wide table: a Web Vitals sample settles when the browser says so (INP can land minutes after LCP), and a wide row would have to be rewritten after the fact, which MergeTree does not do cheaply. Rolling the samples up into quantiles is a read-side concern instead.

Ingest path

Pipeline

@kbve/observ (browser SDK)
→ POST metrics.kbve.com/api/v1/ingest/{errors,perf,events}
→ services/metrics (axum, port 5500)
→ one queue + one flusher per lens
→ jedi ClickHouseConfig::execute_insert (JSONEachRow, batched)
→ telemetry.{errors,perf,events}_distributed

One queue per destination table, not one shared queue: a rejected insert takes its whole batch with it, so a malformed perf sample would otherwise drop the errors queued beside it.

RouteLensRead as
POST /api/v1/ingest/errorswindow errors, promise rejectionsGET /api/v1/groups, GET /api/v1/events
POST /api/v1/ingest/perfWeb Vitals (LCP/INP/CLS/FCP/TTFB)GET /api/v1/perf
POST /api/v1/ingest/eventsnamed product eventsGET /api/v1/product

The read routes are staff-gated; the ingest routes take a shared token. A project named in METRICS_AUTH_PROJECTS takes nothing anonymous: its events come from a signed-in Supabase user, whose sub becomes the row’s user_id whatever the body says, or from a service presenting METRICS_SERVICE_TOKEN as x-kbve-service. The anon key is a signed token too, so only role: authenticated counts as a user. Every other project is taken as before. Note that /api/v1/events is the read side of the errors lens — it lists the events behind a fingerprint — while product events are written to /api/v1/ingest/events and read at /api/v1/product.

The ingest path reuses the shared jedi ClickHouse client — no new database plumbing. A background tokio mpsc flusher batches rows by size and interval. Requests are CORS-allowlisted, per-IP rate-limited, body-capped, and run through a noise/PII sanitizer. Error grouping uses a sha256(project + error_type + top-5 stack frames) fingerprint.

Sanitizing is per lens and deliberately strict about what reaches a LowCardinality column. An unknown perf metric is dropped rather than defaulted — the read path groups by it, so coercing junk to lcp would poison a quantile instead of losing one row — and a non-finite value is rejected outright, because NaN serializes to JSON null, which ClickHouse refuses for a Float64 and which fails the entire insert batch rather than the offending row.

/readiness checks that every table ingest writes to exists, not just that ClickHouse answers: SELECT 1 passes against a database with no schema at all, and a half-applied schema is exactly the case where the service accepts one kind of event and discards another. The answer is re-verified on a window (METRICS_SCHEMA_RECHECK_MS, default 30s) rather than latched, so a cluster that loses its schema after startup is noticed instead of being reported ready forever.

Client-agnostic

The platform field (web / unreal / unity / …) lets every client feed the same pipe, so the errors lens shipped for the frontend extends to game clients without schema changes.

Runtime

Deployment

Runs in the kbve namespace behind the Cilium Gateway at metrics.kbve.com, reusing the existing clickhouse-credentials ExternalSecret.

The production DDL is the script in kube/metrics/manifests/telemetry-ch-setup-job.yaml, an ArgoCD hook job; schemas/clickhouse/schema/telemetry.sql records the desired state of the same tables. The e2e harness runs a real two-shard cluster and applies that same setup script, so there is one artifact rather than a production one and a single-node stand-in.

The same image also runs as an hourly triage CronJob: it reads the error groups over a window, and for each group past an event or session threshold opens a GitLab issue labelled by project area, or notes on the existing one and reopens it when the error comes back after it was closed, so a new crash becomes a ticket without anyone watching the dashboard.

Serving more domains is a gateway concern, not a deployment one: additional hostnames (metrics.rareicon.com, …) go on the existing HTTPRoute and reach the same pods. Tenants are separated inside the one process by METRICS_PROJECT_RATE_LIMIT_PER_MIN and the CORS allowlist.

Recovery covers the schema, not the data. There are no ClickHouse backups and every table carries a 30-day TTL, so a wipe loses the telemetry in it permanently. The e2e suite drills the schema half against a real cluster: it drops the database across both shards, asserts readiness goes degraded, re-runs the production setup job and asserts ingest resumes. It also pins the privilege that job needs — revoking REMOTE from the ingest user must make the script fail, because that is the grant whose absence wedged this app for 23 days.

Questions

Frequently asked

What is the Metrics service?

Metrics (services/metrics) is KBVE's in-house observability platform, a scoped Rust/axum service that ingests client telemetry into a self-hosted ClickHouse telemetry database. It replaces SaaS tooling like Sentry, Datadog, and PostHog with one pipeline exposing errors, performance, and product lenses, one table per lens, correlated by session_id.

How does the Metrics ingest pipeline work?

The browser SDK @kbve/observ POSTs to metrics.kbve.com/api/v1/ingest/errors, /perf or /events, the axum service on port 5500 reuses the shared jedi ClickHouse client to batch JSONEachRow inserts, and each lens has its own queue and flusher writing to its own table. Requests are CORS-allowlisted, rate-limited, body-capped, and PII-sanitized.

How is Metrics client-agnostic across web and game clients?

A platform field (web, unreal, unity, and more) lets every client feed the same pipe, so the errors lens built for the frontend extends to Unreal and Unity game clients with no schema changes.

Discuss on the forum

Questions, fixes and notes on this page live in the KBVE forum.