← All posts

Feature flags in production: shipping without a binary release

How to design feature flags: evaluation model, segments, stickiness, client vs server, kill switches, flag debt, OpenFeature, and common gradual-rollout failures.

Feature flags in production: shipping without a binary release
Contents

An all-or-nothing release is almost always more expensive than it looks: one bug in a new code path hits the whole audience at once. Feature flags promise a different contract — the code is already in production, and behavior turns on for a segment, a percentage, or a single customer. In 2026 this is baseline product engineering, not a luxury for hyperscalers. This site still lacked a dedicated pillar: we have the economics of failure cost, we have LLM gateways and FinOps, but no map of “how to live with flags so they do not become a second configuration hell.” Below is an engineering walkthrough of evaluation, server vs client delivery, stickiness, lifecycle, and the usual holes.

Key takeaways

A flag is a runtime decision, not a git branch. Both sides of the code are already deployed; only the evaluation response changes for a given request context.

Without a server source of truth, a client flag is a hint, not a security boundary. Anything that touches money, permissions, or data must be checked on the backend with the same flag key.

Stickiness matters more than “random 10%.” A user must not bounce between old and new UI on every refresh. Hash a stable identifier with a per-flag salt — that is the minimum contract.

A flag without a removal date taxes the codebase. Every toggle that survives a release doubles test paths. Lifecycle must include an owner, a removal criterion, and automated debt.

OpenFeature and a vendor SDK are a portability layer, not magic. A standard evaluation API lowers the cost of switching providers; you still design segments and audit.

Flags vs config, branches, and experiments

Environment config answers “which database URL and which timeout.” It changes rarely, often needs a restart, and is not meant to personalize per user. A VCS branch answers “which code we built”; turning that branch off after deploy means rollback or hotfix. An A/B experiment answers “which variant wins on a metric” — it needs a hypothesis, sample size, and a stop date.

A feature flag answers “which behavior is allowed for this context right now.” Context usually includes user or device id, tenant, plan, geo, client version, and segment attributes. The same flag key can serve several patterns: dark launch (code runs in shadow, UI unchanged), canary percentage, internal allowlist, kill switch for payments.

Incidents make the difference obvious. If a new signature check breaks login, rolling back a container takes minutes and may undo unrelated fixes. Turning off auth_v2_verify takes seconds and leaves the rest of the release in place. If your “flag” is only an environment variable with no segments, you are back to a binary switch for the whole environment.

Tool What it changes Speed Personalization Typical risk
Env config Process parameters Minutes–hours Low Wrong secret, full restart
Release rollback Whole artifact Minutes None Losing neighboring fixes
A/B experiment Variant for a metric Hours–days of planning High Eternal test with no decision
Feature flag Behavior by context Seconds High Path debt and client/server drift

Evaluation model: key, context, rules, default

Any mature flag system collapses to one function: (key, context, environment) → value. Values may be boolean, string, number, or structured JSON — but the team must agree what “off” means on every layer.

The flag key is a stable identifier in code (checkout_express_v1), not the marketing label on a button. Context is the attribute set rules may read: userId, accountId, plan, country, appVersion, employee. Rules are an ordered priority list: internal accounts first, then customer allowlists, then percentage, then default. Environment (dev / staging / prod) separates experiments from production: one key, different rule tables.

Critical invariant: the default must be safe when the flag service is unreachable. If the SDK cannot fetch remote config, the app must not “accidentally” enable a new payment scheme. Safe fallback is usually the old behavior. Exception: a kill switch — when payment integrity is doubtful, refusing a risky operation beats continuing on a half-broken path. Name these modes explicitly: failOpen vs failClosed.

type FlagValue = boolean | string | number | Record<string, unknown>;

type EvaluationContext = {
  targetingKey: string; // stable id for stickiness
  attributes: Record<string, string | number | boolean>;
};

function evaluate(
  key: string,
  ctx: EvaluationContext,
  fallback: FlagValue,
): FlagValue {
  // 1) local rules snapshot
  // 2) match by priority
  // 3) percentage from hash(targetingKey + key + salt)
  // 4) else fallback
  return fallback;
}

Stickiness and percentage rollouts

“Enable for 10% of users” without stickiness turns the product into a lottery: each request may land in a different branch. That breaks cache, analytics, support, and trust. Stickiness means: for a given targetingKey and flag key, the percentage rule stays stable until salt or threshold changes.

Classic scheme — hash targetingKey + flagKey + salt, map to 0…99 or 0…9999, compare to threshold. Salt prevents one cohort from always being the “first 10%” across every flag. When you raise 10% to 25%, people already inside stay inside; newcomers fill the gap.

Decide what targetingKey is. For B2B, often accountId: the whole tenant sees one behavior, or support cannot explain “my colleague has the button.” For consumer UI — userId. For anonymous pre-login — a stable device cookie, with an explicit migration to userId after sign-in (otherwise the shopper “loses” the new cart on auth).

Where to evaluate: server, edge, client

Server evaluation is the source of truth for authz, billing, writes, and anything the browser must not own. The client receives an already-made decision or a short signed snapshot of booleans for UI. Edge / BFF helps when you must change HTML or headers before hydration without exposing internal rules.

Client-side evaluation is tempting for speed: SDK pulls a JSON rule set and decides locally. Cost — leaking segment logic, drifting from the server, and forging answers in DevTools. Fine for cosmetics (“show a banner”); unacceptable as the only check for “may I call an expensive API.” Pattern that survives production: server evaluates and stores the result in session or a short signed payload; client only renders.

Offline needs discipline. If the app queues writes in an outbox, as in offline-first React, the flag at enqueue time may differ from the flag at delivery. Either stamp the behavior version on the event, or keep the server idempotent across both schemes during the transition.

For LLM stacks, flags shine as routing: which model, which token budget, whether semantic cache is on. That pairs with LLM gateway FinOps — cost and quality change without deploy, with an audit trail of who enabled the expensive model for a segment.

Kill switch, dark launch, and canary

Three patterns get mashed under one word “flag.”

A kill switch is a boolean lever to disable a dangerous subsystem now. It should be a separate key with failClosed for risky operations, reachable by on-call without a full change board at 3 a.m. Document exactly what turns off: payment intake, report generation, writes to a new table.

A dark launch runs new code in shadow: compute, log, compare to the old path, without changing the user response. The flag enables load and diff collection. Cheaper than a full A/B when the goal is correctness confidence, not conversion measurement.

A canary release shows new behavior for real to a small percentage or segment. You need metrics and stop conditions: 5xx rise, p95 latency, support ticket spike. Without a pre-written “when we turn the flag off,” canary is hope.

Flags do not replace tests. They shrink blast radius while failure economics still demand a pyramid on both paths. Minimum — unit tests for both branches and a contract that server and client share one key.

Flag lifecycle and complexity tax

The expensive part of flags is not the SDK — it is branches nobody deleted. Six months later if (flag) spreads across services, mobile clients, and reports. The team fears touching “temporary” code because ownership of the off state is unclear.

Working ritual:

  1. On create: owner, goal, environments, safe fallback, and a review date.
  2. After full rollout (100% + stability): task to delete the dead branch and the key itself.
  3. Each sprint: report flags older than N days stuck at 100% or 0%.
  4. Ban new dependencies on flags marked retired.

Naming helps automation: exp_ for experiments, ops_ for kill switches, perm_ for long-lived entitlements (pro plan sees a module). Mixing them in one registry without prefixes confuses access control with marketing tests.

Long-lived flags are fine when they are product entitlement config, not “temporary rollout.” Keep them next to billing and ACL, with change audit, and without letting on-call accidentally disable payments for all of Europe.

OpenFeature, homegrown rules, and provider choice

OpenFeature standardizes the evaluation API in your app: you code against an abstraction; the provider (Unleash, Flagsmith, LaunchDarkly, homegrown Redis+JSON) plugs in beside it. Upside — vendor swaps without rewriting hundreds of ifs. Downside — you still design context, segments, and observability.

A thin custom engine is fine when rules are few, the team is small, and audit needs are modest. Once you need complex segments, percentages, multi-region, change permissions, and “who flipped this at 03:00,” self-built admin UI usually costs more than a license or self-hosted Unleash.

Selection criteria that actually hurt when wrong:

  • rule propagation latency (seconds vs minutes);
  • server and client SDKs under one key contract;
  • audit and RBAC on flag changes;
  • export of evaluation events into your analytics (without it A/B lies);
  • partition behavior and size of the local rules snapshot.

Do not ship a full segment map with PII into the client bundle. Send computed values or minimized rules without internal notes like “VIP from deal #4821.”

Security, multitenancy, and leaks

A flag that opens an admin API on a client-only check is a hole. A flag that exposes another tenant’s data on a bad segment is a GDPR-class incident. Segment rules must use attributes the server already trusts (session after JWT hardening, IdP claims), not a ?vip=1 query param.

Multitenancy: never evaluate “globally on” where a tenant perimeter is required. Either context always carries tenantId, or use separate keys per contour. Evaluation logs must not write secrets or full PII — a hash of the targeting key and the flag name is enough.

Supply chain applies to flags too: a weak admin login is a way to disable defenses or enable debug for everyone. Treat the flag system as critical control-plane: SSO, short session TTL, change journal, no anonymous write API.

Observability and rollout criteria

Without metrics, a flag is religion. Minimum set: evaluation counters by key and value, SDK errors, snapshot sync latency, business metrics for on vs off segments. For canaries, fix rollback thresholds and decision owner in advance.

Log flagKey, value, reason (targeting_match, percentage, default, error) in structured form — it speeds “why does this customer still see the old UI.” Do not log every hot-path call without sampling: flag evaluation often runs tens of times per page.

If an AI agent changes behavior via flags, apply the same audit as for humans — agentic engineering is exactly about governed change without quality loss. Auto “flip to 100% after green tests” is allowed only with hard stop markers from production signals.

Common mistakes

One flag for five unrelated changes

Hard to roll back “half.” Split by product meaning: cart UI separate from tax recalculation.

Client on, server off

User sees a button and gets 403. Always one key and one server check on mutations.

Percentage from Math.random()

No stickiness — no analytics and no support.

Flag in code with no registry entry

Ghosts cannot be cleaned. The registry (even a table in Unleash) must own names.

Eternal experiment

A/B without a stop date and without a “which variant stays” decision permanently doubles paths.

Secrets and URLs gated only by a client flag

Anything in the bundle can be read. Secrets stay on the server; the flag only decides whether to call an API.

Minimal product skeleton

  1. Pick a provider or a thin Redis layer with a versioned rules snapshot.
  2. Put OpenFeature (or your facade) in the BFF and workers; ban feature modules from talking to the vendor SDK directly.
  3. Standardize context: targetingKey, tenantId, plan, appVersion.
  4. For every new flag: owner, fallback, type (ops / exp / perm), review date.
  5. On canary: dashboard with rollback threshold.
  6. After 100%: delete the dead branch in the same epic as “close the rollout.”

That is enough for flags to reduce risk instead of becoming a warehouse of conditionals. Frontend stack can stay any of the React 2026 map — delivery discipline changes, not the UI library choice.

FAQ

How is a feature flag different from remote config?

Remote config is often parameters (timeouts, copy, numbers). A feature flag branches behavior and rolls out code that is already deployed. Systems overlap in practice; do not store secrets in client remote config or substitute it for server authorization.

Do I need a separate flag per platform (iOS, Web, API)?

Prefer one key; encode platform differences as context attributes or percentage rules on platform. Otherwise “enabled on iOS, forgotten on API” becomes normal.

How do I test flagged code?

Unit-test both branches via a mocked provider. Contract-test that mutations without the server flag fail. In e2e use a fixture rules snapshot, not a live admin dependency.

Can flags replace database schema migrations?

Not as a substitute. Flags help dual-write / dual-read transitions, but data migration and column compatibility remain a separate plan. Otherwise “flag off” leaves half the rows in the new format.

What about SSR and hydration?

Evaluate on the render server and serialize the same values to the client. Otherwise content flash: server painted old, client after hydration enabled new.

Is OpenFeature mandatory?

No. A team-controlled facade is mandatory. OpenFeature is a strong standard when provider SDKs already cover your languages.

How do I stop marketing from killing payments?

Admin RBAC, split ops_ and exp_, dual confirm for dangerous keys, audit and alerts on kill-switch changes in prod.

Conclusion

Feature flags move the “enable behavior” decision from deploy to runtime — and only pay off when critical paths evaluate on the server, percentages stick, every key has an owner and removal date, and admin is hardened like control-plane. Without that you buy release speed with a second legacy inside every if.

Comments

Loading comments…