Providers & BYOK
Every external number in lumen comes through the provider interface — five boundaries (keywords, serp, pagespeed, crux, authority), seven built-in sources, zero vendor calls anywhere else. Providers are selected per boundary in lumen.config.json; an unknown name is a config error listing the valid options.
The model
- Kind labels are part of the output. Every value carries its source: official, community, heuristic, lab or field.
- No key, no call. A BYOK provider without its key is skipped with an explicit not configured result — never a crash, never a keyless call to an endpoint whose terms require one.
- Keys are names, not values. The config stores environment variable names; values are read at call time and never persisted or logged.
Built-in providers
| Provider | Kind | Boundary | Key | What leaves your machine |
|---|---|---|---|---|
google-suggest | heuristic | keywords | — | Your seed term, to Google’s public autocomplete endpoint. |
wikipedia-demand | official | keywords | — | Article titles derived from your seed, to the Wikimedia Pageviews API. |
pagespeed | official | pagespeed | LUMEN_PSI_KEY | The target page URL, to the PageSpeed Insights API. |
crux | official | crux | LUMEN_CRUX_KEY | The target page URL and form factor, to the CrUX API. |
openpagerank | official | authority | LUMEN_OPR_KEY | The domains you ask about, to the Open PageRank API (Bearer auth). |
tranco | official | authority | — | Nothing per query — the daily Tranco list is fetched and cached locally; lookups are local. |
ddg-serp | gray | serp | — | Your keyword, to DuckDuckGo’s HTML endpoint. |
Notes per provider
google-suggest- Undocumented endpoint (gray): cached aggressively, degrades without drama on 429s. Ideas are labeled autocomplete-derived, never volumes.
wikipedia-demand- No key. lumen sends an identifying User-Agent with a contact URL per Wikimedia policy; pageview counts are a demand proxy, labeled as such.
pagespeed- Key optional for occasional trial use, required for automated queries. Without it, lab/field page data is simply reported not-configured.
crux- Requires a free Google Cloud API key (BYOK). Returned field data carries CC BY 4.0 attribution — see attributions.
openpagerank- Free tier with monthly quota. Returns page-rank scores as authority signals, labeled with the provider.
tranco- No key. Use requires attribution — see attributions.
ddg-serp- Default SERP provider, best-effort by nature: bot protection can degrade it at any time. Rank results are labeled gray and best-effort.
BYOK semantics
Three providers take keys, via environment variables only:
LUMEN_PSI_KEYLUMEN_CRUX_KEYLUMEN_OPR_KEY
Export the ones you have; leave the rest unset. Nothing about the unconfigured providers appears as an error — their sections of any report are marked unavailable, and everything else works. See configuration for the byok block if you prefer non-default variable names.
Rate-limit etiquette
lumen treats shared free endpoints the way you’d want a stranger to treat yours: identifying User-Agent with a contact URL, per-host rate limiting on crawls, bounded retries with exponential backoff and jitter, Retry-After honored on 429s, and aggressive caching on gray sources. Provider failures surface as typed errors with the provider name attached — never as silent zeros.