From e616638c36b2c1b3e6ac8ef00a779a163794cf11 Mon Sep 17 00:00:00 2001 From: Tyler Hallada Date: Fri, 4 Sep 2026 21:29:10 +0000 Subject: [PATCH] Document the CDN rollout and refresh the reverse-proxy section MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The README's nginx snippet had drifted from the box: it was missing http2, the security-headers include, the gzip block, the keepalive bump and the trusted certificate. Replace it with what actually runs, then add the two CDN changes. `set_real_ip_from` for Cloudflare's ranges plus `real_ip_header CF-Connecting-IP` has to come first, or every request looks like it came from Cloudflare and the login throttle becomes global. Then `X-Forwarded-For` is *set* from `$remote_addr` rather than appended to: the throttle keys on the first entry, so a client-supplied header must never survive into the app. The ranges are listed for reference but the reader is told to regenerate them from cloudflare.com/ips-v4 and /ips-v6, with a one-liner that does it. The gzip comment claimed the stylesheet carries 445 KB of base64 fonts; it has not since the faces moved back out to their own URLs. The new runbook covers the order that matters: the DNS move (Free plan means a full nameserver change, so the rest of the zone has to survive it), the zone settings that are quietly wrong by default — Browser Cache TTL *raises* the origin's max-age unless set to "Respect Existing Headers", and Rocket Loader and Email Obfuscation both inject scripts the site's `script-src 'self'` CSP blocks — the two cache rules, locking the origin to Cloudflare, wiring the purge token, and the `cf-cache-status` checks that prove each row of the header matrix. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01Va5eMEmWEnjMXBsBob5FDW --- README.md | 126 ++++++++++++++-- docs/runbooks/cdn-rollout.md | 281 +++++++++++++++++++++++++++++++++++ 2 files changed, 396 insertions(+), 11 deletions(-) create mode 100644 docs/runbooks/cdn-rollout.md diff --git a/README.md b/README.md index 43e5441..80c415d 100644 --- a/README.md +++ b/README.md @@ -135,6 +135,7 @@ daily-epub features prune # stale embeddings, old telemetry and assessment daily-epub backfill-social [--days 7] # re-poll social scores for recent articles daily-epub db migrate # run migrations (also automatic on every start) daily-epub config check # validate the config, print the resolved roles, keys and paths +daily-epub cdn purge # purge the CDN edge cache for the configured zone daily-epub users add USER [--admin] [--password-stdin] daily-epub users passwd USER [--password-stdin] daily-epub users role USER user|admin @@ -369,6 +370,10 @@ prints what resolved. | `server.login_window_minutes` | `15` | Length of the login throttle window. | | `server.jobs_enabled` | `true` | Allow the dashboard to start the fixed systemd job catalogue. | | `server.journal_lines` | `300` | Journal lines shown on a dashboard job page (10–5000). | +| `cdn.provider` | unset | `cloudflare`, or unset for no CDN integration. Setting it requires the zone id and the token. | +| `cdn.cloudflare_zone_id` | unset | Zone id from the Cloudflare dashboard overview. | +| `cdn.api_token` | — | **`DAILY_EPUB_CDN__API_TOKEN`**. Needs exactly one permission: `Zone → Cache Purge`, scoped to that zone. | +| `cdn.purge_after_publish` | `true` | Purge the whole edge cache after `generate` publishes. A failure is logged, never fatal; dry runs never purge. | `[curation.ranking]` holds the ranker's tunables. The learned signals are gated: `knn` (rated-neighbour preference) ramps from `knn_floor` (8) to @@ -464,27 +469,63 @@ Node for the XTC converter, whose JIT needs W+X pages. The rating links baked into every article chapter point at `server.public_url`, so `daily.hallada.net` must resolve and serve TLS from the internet (e-readers tap these links). The OPDS catalog rides on the same host. -The login throttle's `SmartIpKeyExtractor` trusts `X-Forwarded-For`; this is -safe only because the configured bind address is loopback and nginx is the -only process that can reach it. -With an existing certificate, a minimal nginx site is: + +The login throttle's `SmartIpKeyExtractor` keys on the first entry of +`X-Forwarded-For`, so nginx must *set* that header from the connection's peer +address rather than appending to whatever the client sent — otherwise anyone +can mint a fresh throttle bucket per attempt by sending their own header. With +Cloudflare in front, the connection's peer is Cloudflare, so `real_ip` has to +rewrite `$remote_addr` from `CF-Connecting-IP` first (see the snippet below). +The whole arrangement is only safe because the configured bind address is +loopback and nginx is the only process that can reach it. + +The site is served through Cloudflare; `docs/runbooks/cdn-rollout.md` covers the +zone setup and cache rules. The nginx side of it is the `set_real_ip_from` +snippet and the `X-Forwarded-For` line below. ```nginx server { - listen 443 ssl; - listen [::]:443 ssl; + listen 443 ssl http2; + listen [::]:443 ssl http2; server_name daily.hallada.net; ssl_certificate /etc/letsencrypt/live/daily.hallada.net/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/daily.hallada.net/privkey.pem; + ssl_trusted_certificate /etc/letsencrypt/live/daily.hallada.net/fullchain.pem; + + include /etc/nginx/snippets/security-headers.conf; + + # Trust Cloudflare's edge for the client address: $remote_addr becomes the + # visitor, not the proxy. Must come before the X-Forwarded-For line below. + include /etc/nginx/snippets/cloudflare-real-ip.conf; + + # The global `gzip on` only covers text/html. The stylesheet is ~12 KB of + # plain CSS again — the Newsreader faces are separate .woff2 URLs, not + # base64 inside it (see the doc comment on `web::APP_CSS`) — so this list is + # about the JSON, XML and SVG responses. woff2 is already compressed and is + # deliberately absent. + gzip_types text/css application/javascript application/json + application/speculationrules+json image/svg+xml + application/atom+xml application/xml text/plain; + gzip_min_length 1024; + gzip_vary on; + gzip_proxied any; + + # Keep a browser's connection open longer than the 75 s default so the + # first click after a pause reuses it instead of paying a new TLS handshake. + keepalive_timeout 300s; # XTCH files can be tens of MB; don't buffer them through nginx memory. proxy_max_temp_file_size 0; location / { proxy_pass http://127.0.0.1:3499; + proxy_http_version 1.1; proxy_set_header Host $host; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + # SET, not append. The login throttle keys on the first X-Forwarded-For + # entry, so a client-supplied header must never survive into the app. + # $remote_addr is the real visitor because of the real_ip snippet above. + proxy_set_header X-Forwarded-For $remote_addr; proxy_set_header X-Forwarded-Proto $scheme; } } @@ -497,12 +538,75 @@ server { } ``` -Enable it (`ln -s` into `sites-enabled`, `nginx -t`, `systemctl reload nginx`), -then `curl https://daily.hallada.net/healthz` should return `ok`. No auth is -needed at the proxy layer: rating links are self-authenticating (HMAC tokens) -and the OPDS/file routes use the app-level Basic auth from +`/etc/nginx/snippets/cloudflare-real-ip.conf` is one `set_real_ip_from` line per +published Cloudflare range plus the header to read. Cloudflare adds ranges from +time to time, so **regenerate it from the source rather than copying this list** +(, ): + +```sh +{ curl -s https://www.cloudflare.com/ips-v4; echo; \ + curl -s https://www.cloudflare.com/ips-v6; echo; } \ +| awk 'NF {print "set_real_ip_from " $0 ";"} END {print "real_ip_header CF-Connecting-IP;"}' \ +| sudo tee /etc/nginx/snippets/cloudflare-real-ip.conf +``` + +As of this writing that produces: + +```nginx +set_real_ip_from 173.245.48.0/20; +set_real_ip_from 103.21.244.0/22; +set_real_ip_from 103.22.200.0/22; +set_real_ip_from 103.31.4.0/22; +set_real_ip_from 141.101.64.0/18; +set_real_ip_from 108.162.192.0/18; +set_real_ip_from 190.93.240.0/20; +set_real_ip_from 188.114.96.0/20; +set_real_ip_from 197.234.240.0/22; +set_real_ip_from 198.41.128.0/17; +set_real_ip_from 162.158.0.0/15; +set_real_ip_from 104.16.0.0/13; +set_real_ip_from 104.24.0.0/14; +set_real_ip_from 172.64.0.0/13; +set_real_ip_from 131.0.72.0/22; +set_real_ip_from 2400:cb00::/32; +set_real_ip_from 2606:4700::/32; +set_real_ip_from 2803:f800::/32; +set_real_ip_from 2405:b500::/32; +set_real_ip_from 2405:8100::/32; +set_real_ip_from 2a06:98c0::/29; +set_real_ip_from 2c0f:f248::/32; +real_ip_header CF-Connecting-IP; +``` + +Enable the site (`ln -s` into `sites-enabled`, `nginx -t`, `systemctl reload +nginx`), then `curl https://daily.hallada.net/healthz` should return `ok`. No +auth is needed at the proxy layer: rating links are self-authenticating (HMAC +tokens) and the OPDS/file routes use the app-level Basic auth from `server.basic_auth_user`/`_pass` if you set them. +#### HTTP caching + +The origin says what may be cached and for how long; the CDN is configured to +respect it (runbook §2). The whole matrix: + +| route | `Cache-Control` | +|---|---| +| `/`, `/issues`, `/issues/{date}`, `/feed.xml`, `/issues.json` (anonymous) | `public, max-age=300, s-maxage=86400` | +| the same pages with a `daily_session=` cookie | `private, no-store` | +| `/robots.txt` | `public, max-age=86400` | +| `/static/*?v=` | `public, max-age=31536000, immutable` | +| `/static/*` with no `?v=` | `public, max-age=3600` | +| `/files/epub/*`, `/files/xtc/*`, `/opds*` (every status) | `private, no-store` | +| `/dashboard*`, `/login`, `/account`, errors, anything else | `no-store` | + +The five-minute browser age and the one-day edge age are the same header doing +two jobs: a reader's tab revalidates soon after a new issue lands, while the +edge is allowed to answer for a whole day because `generate` purges it the +moment it publishes (`[cdn]`, `daily-epub cdn purge`). The `no-store` on the +last row is a default applied by the `security_headers` middleware to any +response that set no policy of its own, so a route added later cannot silently +inherit the CDN's default TTL. + ### Installing the XTC converter ```sh diff --git a/docs/runbooks/cdn-rollout.md b/docs/runbooks/cdn-rollout.md new file mode 100644 index 0000000..fc5a8ad --- /dev/null +++ b/docs/runbooks/cdn-rollout.md @@ -0,0 +1,281 @@ +# Runbook — putting daily.hallada.net behind Cloudflare + +**Written:** 2026-09-04 for the production host. Steps 1–2 happen in the Cloudflare +and registrar dashboards; steps 3–6 are on the server as an operator with `sudo`. +The origin-side changes (the `Cache-Control` matrix and `daily-epub cdn purge`) +ship in the same release, so deploy the new binary before step 5. + +Read the whole thing before starting step 1: a nameserver change is the one step +that cannot be undone in seconds. + +## 0. What the origin already does + +The app decides what is cacheable; Cloudflare is configured only to obey it. + +| route | `Cache-Control` | +|---|---| +| `/`, `/issues`, `/issues/{date}`, `/feed.xml`, `/issues.json` (anonymous) | `public, max-age=300, s-maxage=86400` | +| the same pages with a `daily_session=` cookie | `private, no-store` | +| `/robots.txt` | `public, max-age=86400` | +| `/static/*?v=` | `public, max-age=31536000, immutable` | +| `/static/*` with no `?v=` | `public, max-age=3600` | +| `/files/epub/*`, `/files/xtc/*`, `/opds*` (every status, 401 and 404 included) | `private, no-store` | +| `/dashboard*`, `/login`, `/account`, error pages, anything else | `no-store` | + +Two facts follow from that table and matter for every choice below: + +- The edge is allowed to hold a public page for a **day** (`s-maxage=86400`). + That is only correct because `generate` purges the whole zone right after it + publishes. If the purge is broken, the site serves yesterday's paper until the + day elapses. Step 5 is not optional. +- Anything gated by a cookie or Basic auth already says `private, no-store`, so + even a misconfigured cache-everything rule cannot make a download public. The + cookie bypass rule in step 3 is defence in depth, not the only defence. + +## 1. Move DNS to Cloudflare + +The Free plan has no partial (CNAME) setup, so this is a full nameserver move: +Cloudflare becomes authoritative for the entire `hallada.net` zone, not just this +host. Everything else in the zone must survive the move unchanged. + +1. Export the current zone from Route 53 as BIND: + + ```sh + ZONE=$(aws route53 list-hosted-zones-by-name --dns-name hallada.net. \ + --query 'HostedZones[0].Id' --output text | cut -d/ -f3) + aws route53 list-resource-record-sets --hosted-zone-id "$ZONE" > /tmp/hallada-net.json + ``` + + The AWS CLI has no BIND exporter; the console's **Export zone file** button on + the hosted zone does produce one. Keep both files — the JSON is the rollback + reference. + +2. In Cloudflare, **Add a site** → `hallada.net` → Free plan → **Import DNS + records** and upload the BIND file. Then read the imported list against the + export line by line. Pay attention to `MX`, `TXT` (SPF/DKIM/DMARC), and any + `CNAME` used for verification: those must all stay **DNS only** (grey cloud). + +3. Set **only** the `daily` record to **Proxied** (orange cloud). Nothing else in + the zone should be proxied — proxying an MX target or a mail-related host + breaks it. + +4. Change the nameservers at the registrar to the two Cloudflare assigns. + Propagation is usually minutes and can be hours. Cloudflare emails when the + zone goes active. + +5. Before the switch, drop the TTL on the `daily` record (60 s) so a rollback is + quick. Verify afterwards: + + ```sh + dig +short NS hallada.net + dig +short daily.hallada.net # Cloudflare anycast addresses once proxied + dig +short MX hallada.net # unchanged + ``` + +## 2. Zone settings + +Under the zone's **SSL/TLS**, **Speed** and **Scrape Shield** sections: + +- **SSL/TLS → Overview → Full (strict)**. The origin has a real Let's Encrypt + certificate, so there is no reason to accept anything weaker. *Flexible* would + make Cloudflare talk plain HTTP to the origin and would break the redirect in + the nginx `:80` server block into a loop. +- **Caching → Configuration → Browser Cache TTL → Respect Existing Headers.** + This one is easy to miss and quietly wrong: the default (**4 hours**) *raises* + the `max-age` Cloudflare sends to browsers, so the deliberate 5-minute browser + age on the public pages would become 4 hours and a reader's tab would show a + stale paper long after a purge. +- **Speed → Optimization → Rocket Loader: OFF.** +- **Scrape Shield → Email Address Obfuscation: OFF.** + Both inject a Cloudflare-hosted script into the HTML. The site's CSP is + `script-src 'self'`, so the browser blocks the injected script and the page + either breaks or logs CSP violations on every load. There is nothing to gain: + the site has no email addresses in its markup and its own JS is two small + files. +- **Do not enable Web Analytics with automatic injection** — same reason, it is + an injected third-party script. If analytics are wanted later, they have to be + first-party and listed in the CSP. +- **Always Use HTTPS: ON** is fine and lets the origin's `:80` block stay as a + backstop. + +## 3. Cache Rules + +**Caching → Cache Rules.** Order matters: rule 1 must sit above rule 2. + +**Rule 1 — "Bypass cache for signed-in readers"** + +- When incoming requests match: `http.cookie contains "daily_session="` +- Then: **Bypass cache** + +The origin already sends `private, no-store` to a cookie-bearing request, but +this makes the bypass a property of the request rather than of the response, so +nothing is ever *looked up* in a shared cache for a signed-in reader. + +**Rule 2 — "Cache by origin headers"** + +- When incoming requests match: `http.host eq "daily.hallada.net"` +- Then: **Eligible for cache** +- **Edge TTL:** *Use cache-control header if present, bypass cache if not* +- **Browser TTL:** *Respect origin* + +That Edge TTL mode is the whole point of the origin work: a response with +`s-maxage` is cached for exactly that long, and a response with `no-store` (or +one that somehow arrives with no policy at all) is not cached. It is the reason +the `security_headers` middleware defaults unknown routes to `no-store` — with +this mode, "no header" means "do not cache" rather than "cache for 2 hours". + +Leave **Cache Key** at its default. Do not add "Ignore query string": the +`?v=` on `/static/*` is what makes the immutable one-year `max-age` safe. + +## 4. Lock the origin to Cloudflare + +Once traffic arrives through the edge, direct hits on the origin's port 443 +should stop. Either is enough; the first is simpler. + +- **Firewall:** allow 443 only from the published Cloudflare ranges. + + ```sh + { curl -s https://www.cloudflare.com/ips-v4; echo; \ + curl -s https://www.cloudflare.com/ips-v6; echo; } \ + | awk 'NF {print $0}' \ + | xargs -I{} sudo ufw allow proto tcp from {} to any port 443 comment 'cloudflare' + sudo ufw delete allow 443/tcp # remove the open rule last + ``` + + Keep a way in that does not depend on this (SSH from your own address) before + removing the open rule. + +- **Authenticated Origin Pulls** (SSL/TLS → Origin Server) instead: install + Cloudflare's client CA on the origin and add `ssl_client_certificate` + + `ssl_verify_client on` to the nginx server block. Stronger, but one more + certificate to keep track of. + +Also apply the nginx changes from the README's *Reverse proxy* section now: +the `include /etc/nginx/snippets/cloudflare-real-ip.conf;` (generated from +Cloudflare's published ranges, `real_ip_header CF-Connecting-IP;` at the end) and +`proxy_set_header X-Forwarded-For $remote_addr;` in place of +`$proxy_add_x_forwarded_for`. Without the first, every request looks like it came +from Cloudflare and the login throttle becomes global; without the second, a +client could forge the throttle key. Then: + +```sh +sudo nginx -t && sudo systemctl reload nginx +``` + +Check a login attempt from a phone on cellular and one from the LAN land in +different throttle buckets, and that `journalctl -u daily-epub` shows real client +addresses rather than Cloudflare's. + +## 5. Configure the purge + +Create the API token in the Cloudflare dashboard: **My Profile → API Tokens → +Create Token → Custom token**, with the single permission **Zone → Cache Purge → +Purge**, **Zone Resources → Include → Specific zone → hallada.net**. Nothing +else — the app makes exactly one API call. Copy the token once; it is not shown +again. The zone id is on the zone's **Overview** page. + +In `/etc/daily-epub/config.toml`: + +```toml +[cdn] +provider = "cloudflare" +cloudflare_zone_id = "…" +purge_after_publish = true +``` + +and in the systemd environment file (the one the units already load, mode `0600`, +owned by `daily-epub`): + +``` +DAILY_EPUB_CDN__API_TOKEN=… +``` + +Setting `provider` without both the zone id and the token is a **config error**: +the app refuses to start rather than publishing into a stale edge. Confirm and +then purge by hand: + +```sh +sudo systemctl restart daily-epub +sudo -u daily-epub daily-epub --config /etc/daily-epub/config.toml config check | grep '^cdn' +# cdn: cloudflare · zone … · token present · purge_after_publish true +sudo -u daily-epub daily-epub --config /etc/daily-epub/config.toml cdn purge +# purged the whole cloudflare cache (id …) +``` + +`cdn purge` takes no run lock and touches neither the database nor the publish +directories, so it is safe to run at any time, including during a `generate`. + +From then on `generate` purges the whole zone after every successful publish. The +purge is *purge everything* on purpose: a new issue also changes the previous +issue's page (its "latest" nav marker moves), `/issues`, `/feed.xml` and +`/issues.json`, and a per-URL list of that set would rot. A purge failure is +logged at `warn` and never fails the run — so check for it after the first live +run: + +```sh +journalctl -u daily-epub-generate.service --since today | grep -i 'purge' +``` + +## 6. Verify + +Anonymous public page — expect `HIT` on the second request (the first fills the +edge) and the origin's own two-part `Cache-Control`: + +```sh +curl -sI https://daily.hallada.net/ | grep -iE 'cf-cache-status|cache-control|age' +curl -sI https://daily.hallada.net/ | grep -i cf-cache-status # HIT +``` + +Signed in — the cookie rule must take it out of the cache entirely: + +```sh +curl -sI -H 'Cookie: daily_session=whatever' https://daily.hallada.net/ \ + | grep -iE 'cf-cache-status|cache-control' +# cf-cache-status: BYPASS (DYNAMIC is also acceptable) +# cache-control: private, no-store +``` + +Static assets — versioned URLs should settle on `HIT`: + +```sh +curl -s https://daily.hallada.net/ | grep -o '/static/app.css?v=[0-9a-f]*' +curl -sI 'https://daily.hallada.net/static/app.css?v=' \ + | grep -iE 'cf-cache-status|cache-control' +# cf-cache-status: HIT +# cache-control: public, max-age=31536000, immutable +``` + +Downloads and OPDS — never cached, at any status: + +```sh +curl -sI https://daily.hallada.net/opds | grep -iE 'cf-cache-status|cache-control' +curl -sI https://daily.hallada.net/files/epub/does-not-exist.epub \ + | grep -iE 'cf-cache-status|cache-control' +# cache-control: private, no-store (cf-cache-status: BYPASS or DYNAMIC) +``` + +The dashboard and the login page: + +```sh +curl -sI https://daily.hallada.net/login | grep -iE 'cf-cache-status|cache-control' +# cache-control: no-store +``` + +Finally, the end-to-end check the whole exercise is about: run a `generate`, then +immediately `curl -sI https://daily.hallada.net/ | grep -i cf-cache-status` and +confirm it reads `MISS` (the purge emptied the edge) and that the page shows the +new issue. + +## Rollback + +- **Cache misbehaving:** turn on **Development Mode** (Caching → Configuration) + for a three-hour edge bypass while you look, or **Purge Everything**. Neither + needs a deploy. +- **Something worse:** set the `daily` DNS record back to **DNS only** (grey + cloud). Traffic goes straight to the origin again within the record's TTL, and + nothing about the origin's behaviour depends on Cloudflare being there — the + `Cache-Control` headers are correct without it, and `cdn.provider` can be + removed from the config at leisure. +- **Full retreat:** point the registrar's nameservers back at Route 53. The + hosted zone still exists unless it was deleted; do not delete it until the + Cloudflare setup has run for a few weeks.