docker · web

The web service behindmeme.sh

The KBVE web service behind meme.sh — an Astro/Starlight front end and the axum that serves it, released by bumping the version in this doc and served through a Cilium gateway on Kubernetes.

View source Routing Gateway gotcha

One listener per multi-SAN cert

Cilium derives the Envoy filter chain SNI list from the cert SAN, not the listener hostname. Splitting apex and www across two listeners on one cert emits duplicate filter chain matches that Envoy rejects — a silent fall-through 404.

Do — one listener, HTTPRoute enumerates hostnames.
Or — one cert per hostname so SNI lists don't collide.

memesh-webApp

meme.shDomain

Astro + axumStack

memes-service:4321Service

One project, one version

apps/memesh holds web/, e2e/ and the server that will serve them; the version in this doc is the tag, the image and the number in the site footer.

Astro static frontend

apps/memesh/web builds static to web/dist, which the server packages into the memesh-web image.

Cilium gateway routing

Cloudflare → gateway.kbve.com → Cilium LB → kbve-gateway https-memes listener → memes-route → memes-service:4321.

Cross-namespace TLS

memes-cert (RSA 2048, letsencrypt-http HTTP-01) lives in memes/memes-tls and is consumed by kbve-gateway via a ReferenceGrant.

What it is

Overview

Application service serving meme.sh and www.meme.sh. The site is the Astro/Starlight build in apps/memesh/web, packaged with the axum that serves it into the memesh-web image. Deployment lives at kube/memes/manifests/.

Release

Bumping the version

A release is one edit:

moon run version-sync:sync -- bump memesh-web 0.2.1

That writes version: here, apps/memesh/version.toml and the version in apps/memesh/server/Cargo.toml — the site footer reads the first, the binary reports the second on /api/health, and nothing else moves. Pushing it to main fires memesh-web in apps/memesh/ci.yml, which stamps the same number back over those files in its own working tree, refuses the build if the tag already exists, and publishes memesh-web:0.2.1. memesh-web-deploy then opens a merge request repointing the manifests, which merges itself once memesh-deploy-check agrees the diff is only that.

The site reads version.toml at build time — web/src/lib/version.ts, inlined by vite, nothing read at runtime — and prints it in the footer, so what a visitor sees is the tag that was published.

The image carries both halves: the Astro build, and the axum in apps/memesh/server that serves it and answers /api/v1. That API is a gateway in front of the meme edge function, which refuses any caller that is not service_role and takes the acting user from user_id in the body — so the service key stays in this process and the caller’s own token is verified against GoTrue before a user id is put in front of it.

Two things still to do before deploy can rewrite the manifest:

memes-deployment.yaml still runs ghcr.io/kbve/memes, built in the KBVE tree. version-sync matches on the image basename and owner, so it will not recognise that line as memesh-web — the manifest has to be repointed once by hand.
That pod spec carries no imagePullSecrets. ghcr.io/kbve/memes is public and this workspace’s registry is not, so the credential has to be added with the repoint or the pod will ImagePullBackOff.

Traffic path

Routing & TLS

Public traffic enters via Cloudflare → CNAME to gateway.kbve.com → 142.132.206.71 (Cilium gateway LB) → kbve-gateway https-memes listener (hostname meme.sh, cert memes-tls covers both meme.sh and www.meme.sh via SAN) → memes-route (spec.hostnames = [meme.sh, www.meme.sh]) → memes-service:4321.

The cert lives in memes/memes-tls and is consumed cross-namespace by kbve-gateway via memes-tls-from-kbve-gateway ReferenceGrant.

TLS / certificate

memes-cert is RSA 2048, issued by the letsencrypt-http ClusterIssuer (HTTP-01). The algorithm matches the cert already in memes-tls; do not change to ECDSA without first verifying that the ACME HTTP-01 challenge path on Cilium gateway is healthy (see runbook below) — otherwise cert-manager flags IncorrectCertificate and queues a reissue that can never complete.

Cilium gateway

Never split apex + www across listeners with one cert

The original setup had two listeners (https-memes for meme.sh, https-memes-www for www.meme.sh) both pointing at the same memes-tls Secret. Cilium derives the Envoy filter chain SNI list from the cert SAN (not the listener hostname), so each listener emitted a filter chain with the same SNI array [meme.sh, www.meme.sh]. Envoy rejects duplicate filter chain match → falls through to the default 404 handler → 73 days of meme.sh returning nginx-style 404s while kubectl get gateway reported Programmed=True / Accepted=True / ResolvedRefs=True and the memes-service cluster sat healthy with zero cx_total.

A second-order victim was ACME HTTP-01 — cm-acme-http-solver-* HTTPRoutes share the same listener, so the cert reissue loop also couldn’t complete (Ready=False, Reason=IncorrectCertificate).

Rule for this gateway: if one cert covers multiple SANs, use one listener with the apex hostname and let the HTTPRoute spec.hostnames enumerate every hostname the route serves. If you need per-host listeners, issue one cert per hostname so the SNI lists don’t collide.

Diagnose

Runbook — meme.sh returns 404

Confirm origin is the actual culprit (not Cloudflare): curl -s https://meme.sh/ | head — an nginx-flavored 404 body means the request reached origin and the gateway returned it.
Compare against a sibling host on the same gateway, e.g. curl -sk --resolve kbve.com:443:142.132.206.71 -H 'Host: kbve.com' https://kbve.com/ — if that’s 200, the gateway is healthy and the breakage is memes-specific.
Dump the Envoy filter chains and look for two with overlapping SNI arrays: kubectl exec -n kube-system <cilium-pod> -- cilium-dbg envoy admin config | jq '.. | objects | select(.filter_chains) | .filter_chains[] | select(.filter_chain_match.server_names[]? | contains("meme")) | .filter_chain_match.server_names' — if you see the SAN list duplicated across two chains, the listener split is the bug.
Collapse to one listener (this PR’s fix), reapply via ArgoCD, and kubectl rollout restart -n kube-system ds/cilium-envoy to flush the stale filter chains.
The stuck cm-acme-http-solver-* HTTPRoutes drain automatically once the cert can validate; if they don’t, kubectl delete httproute -l acme.cert-manager.io/http01-solver -n memes to force cert-manager to recreate fresh ones.
Re-curl meme.sh — should return 200.

What is the Memesh Web application?

Memesh Web is the KBVE web service serving meme.sh and www.meme.sh. The site is the Astro/Starlight build in apps/memesh, served by the axum in apps/memesh/server and packaged into the memesh-web image.

How does traffic reach the Memes service?

Public traffic enters via Cloudflare, CNAMEs to gateway.kbve.com, hits the Cilium gateway LB, then the kbve-gateway https-memes listener (cert memes-tls covering meme.sh and www.meme.sh), routes through memes-route, and lands on memes-service:4321.

Why should apex and www not be split across gateway listeners?

Cilium derives the Envoy filter chain SNI list from the cert SAN, not the listener hostname. Two listeners sharing one multi-SAN cert emit duplicate filter chain matches, which Envoy rejects, causing a fall-through 404. Use one listener with the apex hostname and let the HTTPRoute enumerate every hostname.