docker · web

The web service behindmeme.sh

The KBVE web service behind meme.sh — an axum + Askama Rust backend and an Astro static frontend, packaged as kbve/memes and served through a Cilium gateway on Kubernetes.

View source Routing Gateway gotcha

One listener per multi-SAN cert

Cilium derives the Envoy filter chain SNI list from the cert SAN, not the listener hostname. Splitting apex and www across two listeners on one cert emits duplicate filter chain matches that Envoy rejects — a silent fall-through 404.

Do — one listener, HTTPRoute enumerates hostnames.
Or — one cert per hostname so SNI lists don't collide.

memesApp

meme.shDomain

axum + AstroStack

memes-service:4321Service

Askama-templated backend

apps/memes/axum-memes is a Rust backend rendering Askama templates behind the meme.sh origin.

Astro static frontend

apps/memes/astro-memes builds the static frontend packaged alongside the backend into the kbve/memes image.

Cilium gateway routing

Cloudflare → gateway.kbve.com → Cilium LB → kbve-gateway https-memes listener → memes-route → memes-service:4321.

Cross-namespace TLS

memes-cert (RSA 2048, letsencrypt-http HTTP-01) lives in memes/memes-tls and is consumed by kbve-gateway via a ReferenceGrant.

What it is

Overview

Memes application service serving meme.sh and www.meme.sh. Built from apps/memes/axum-memes (Rust backend + Askama templates) plus the apps/memes/astro-memes static frontend, packaged into the kbve/memes image. Deployment lives at apps/kube/memes/manifest/.

Traffic path

Routing & TLS

Public traffic enters via Cloudflare → CNAME to gateway.kbve.com → 142.132.206.71 (Cilium gateway LB) → kbve-gateway https-memes listener (hostname meme.sh, cert memes-tls covers both meme.sh and www.meme.sh via SAN) → memes-route (spec.hostnames = [meme.sh, www.meme.sh]) → memes-service:4321.

The cert lives in memes/memes-tls and is consumed cross-namespace by kbve-gateway via memes-tls-from-kbve-gateway ReferenceGrant.

TLS / certificate

memes-cert is RSA 2048, issued by the letsencrypt-http ClusterIssuer (HTTP-01). The algorithm matches the cert already in memes-tls; do not change to ECDSA without first verifying that the ACME HTTP-01 challenge path on Cilium gateway is healthy (see runbook below) — otherwise cert-manager flags IncorrectCertificate and queues a reissue that can never complete.

Cilium gateway

Never split apex + www across listeners with one cert

The original setup had two listeners (https-memes for meme.sh, https-memes-www for www.meme.sh) both pointing at the same memes-tls Secret. Cilium derives the Envoy filter chain SNI list from the cert SAN (not the listener hostname), so each listener emitted a filter chain with the same SNI array [meme.sh, www.meme.sh]. Envoy rejects duplicate filter chain match → falls through to the default 404 handler → 73 days of meme.sh returning nginx-style 404s while kubectl get gateway reported Programmed=True / Accepted=True / ResolvedRefs=True and the memes-service cluster sat healthy with zero cx_total.

A second-order victim was ACME HTTP-01 — cm-acme-http-solver-* HTTPRoutes share the same listener, so the cert reissue loop also couldn’t complete (Ready=False, Reason=IncorrectCertificate).

Rule for this gateway: if one cert covers multiple SANs, use one listener with the apex hostname and let the HTTPRoute spec.hostnames enumerate every hostname the route serves. If you need per-host listeners, issue one cert per hostname so the SNI lists don’t collide.

Diagnose

Runbook — meme.sh returns 404

Confirm origin is the actual culprit (not Cloudflare): curl -s https://meme.sh/ | head — an nginx-flavored 404 body means the request reached origin and the gateway returned it.
Compare against a sibling host on the same gateway, e.g. curl -sk --resolve kbve.com:443:142.132.206.71 -H 'Host: kbve.com' https://kbve.com/ — if that’s 200, the gateway is healthy and the breakage is memes-specific.
Dump the Envoy filter chains and look for two with overlapping SNI arrays: kubectl exec -n kube-system <cilium-pod> -- cilium-dbg envoy admin config | jq '.. | objects | select(.filter_chains) | .filter_chains[] | select(.filter_chain_match.server_names[]? | contains("meme")) | .filter_chain_match.server_names' — if you see the SAN list duplicated across two chains, the listener split is the bug.
Collapse to one listener (this PR’s fix), reapply via ArgoCD, and kubectl rollout restart -n kube-system ds/cilium-envoy to flush the stale filter chains.
The stuck cm-acme-http-solver-* HTTPRoutes drain automatically once the cert can validate; if they don’t, kubectl delete httproute -l acme.cert-manager.io/http01-solver -n memes to force cert-manager to recreate fresh ones.
Re-curl meme.sh — should return 200.

What is the Memes application?

Memes is the KBVE web service serving meme.sh and www.meme.sh. It is built from apps/memes/axum-memes (a Rust backend with Askama templates) plus the apps/memes/astro-memes static frontend, packaged into the kbve/memes image.

How does traffic reach the Memes service?

Public traffic enters via Cloudflare, CNAMEs to gateway.kbve.com, hits the Cilium gateway LB, then the kbve-gateway https-memes listener (cert memes-tls covering meme.sh and www.meme.sh), routes through memes-route, and lands on memes-service:4321.

Why should apex and www not be split across gateway listeners?

Cilium derives the Envoy filter chain SNI list from the cert SAN, not the listener hostname. Two listeners sharing one multi-SAN cert emit duplicate filter chain matches, which Envoy rejects, causing a fall-through 404. Use one listener with the apex hostname and let the HTTPRoute enumerate every hostname.