Glenn Lum Software Engineering

2.6.2 Configuration Hierarchy and Override Models

Most engineers think of configuration as a data problem: you have keys, you have values, you store them somewhere external to the code. The Level 1 principle — separate config from code — is well understood. What is not well understood is the resolution problem. When the same key exists at multiple levels of your system — a global default, an environment-specific override, a service-level setting, an instance-level escape hatch — something has to decide which value wins. That decision logic, the override model, is where nearly all real configuration incidents originate. The value in the database wasn’t wrong. Someone set it intentionally, at a level that silently took precedence over the value someone else was relying on. Understanding configuration hierarchy means understanding the resolution rules that determine, for any given key at any given moment, which layer’s value is the one your application actually sees.

The Layer Model

A configuration hierarchy is a stack of layers, ordered from least specific to most specific. A minimal version looks like this:

global defaults
  → environment (dev, staging, production)
    → service (payments, auth, frontend)
      → instance (payments-us-east-1a-03)

Each layer can define values for any configuration key. The resolution rule, in almost every system, is most-specific-wins: the value defined at the most specific layer that has an opinion about a key is the value the application receives. If an instance-level override exists, it wins. If not, the service-level value applies. If the service doesn’t define it, the environment value applies. If no environment override exists, the global default is used.

This is conceptually simple. A four-level stack with a clear precedence order. But the simplicity is deceptive, because the real behavior depends on details that vary between implementations: what counts as a “level,” how values at adjacent levels are combined, and whether the system makes the resolution path visible to operators.

In practice, hierarchies are rarely this clean. Real systems accumulate additional layers: region, availability zone, deployment ring, tenant, canary group. A configuration system for a large platform might resolve a single key through seven or eight layers. Each additional layer multiplies the number of places a value could be hiding.

How Resolution Actually Works

When an application requests a configuration value — say, http.client.timeout_ms — the configuration system walks the hierarchy from the most specific applicable layer to the least specific, returning the first defined value it finds.

The word “applicable” is doing real work here. For a given request, the system must know the full identity of the requester: which service, which environment, which instance, which region. This identity is the resolution context. The resolution context determines which layers are in scope. A request from payments-us-east-1a-03 running in production would check:

instance: payments-us-east-1a-03   → defined? → use it
service:  payments                  → defined? → use it
region:   us-east-1                 → defined? → use it
environment: production             → defined? → use it
global defaults                     → defined? → use it
(none found)                        → return system default or error

This is a linear scan through ordered scopes. The resolution context is what makes the hierarchy dynamic — the same key resolves to different values depending on who is asking.

This design has a critical property: any layer can silently intercept a value that a less-specific layer intended to provide. If the platform team sets http.client.timeout_ms = 5000 at the global level, and an SRE debugging a latency issue sets it to 30000 at the instance level for one host, that override is invisible to anyone looking only at the global or service layer. The value is correct at the layer it was set. It is unexpected at the layer someone else is reading from.

Merge Semantics: Replace vs. Combine

For scalar values — a number, a string, a boolean — resolution is straightforward. The most specific value replaces less specific ones entirely. But configuration values are often structured: a list of allowed origins for CORS, a map of retry policies per downstream dependency, a nested object describing a logging configuration.

When a more specific layer defines a structured value, the system must decide: does this replace the less specific value entirely, or does it merge with it?

Consider a global logging configuration:

# global defaults
logging:
  level: INFO
  format: json
  outputs:
    - stdout
    - syslog

A service override wants to add debug logging:

# service: payments
logging:
  level: DEBUG

Under shallow replace semantics, the service-level logging key replaces the entire global logging object. The payments service gets level: DEBUG and loses format and outputs entirely. Under deep merge semantics, the service-level keys are merged into the global object, producing a result where level is overridden to DEBUG but format and outputs are inherited.

Neither behavior is universally correct. Shallow replace is predictable — you always know exactly what a layer is providing because it provides the complete value. But it forces every override to restate everything, which defeats the purpose of having defaults. Deep merge preserves inheritance but introduces ambiguity: if the service-level config omits outputs, does that mean “inherit the global value” or “I intentionally want no outputs defined”? There is no way to distinguish absence-as-inheritance from absence-as-intent in a deep merge model without introducing an explicit deletion marker — a sentinel value like null or ~delete~ that means “remove this key even if a less-specific layer defines it.”

The merge strategy is the single most important design decision in a configuration hierarchy, and it is the one most often made implicitly rather than explicitly. Many systems default to deep merge because it feels convenient, and then teams discover the ambiguity problem six months later during an incident where a service inherited a value nobody expected it to still have.

Lists Are Especially Dangerous

Lists compound the merge problem. If the global layer defines allowed_origins: [a.com, b.com] and the service layer defines allowed_origins: [c.com], does the result contain three entries or one? If the merge is append-style, you get all three — but then there’s no way for a more-specific layer to remove an entry added by a less-specific one. If the merge is replace-style, the service layer must duplicate every global entry it wants to keep.

Most mature configuration systems handle this by treating lists as atomic values that are always replaced, not merged. This is a pragmatic choice: the combinatorics of list merging (append, prepend, insert-at-position, remove-by-value) are complex enough that the behavior becomes unpredictable. Replacing the entire list at a more-specific layer is easier to reason about, even if it requires some duplication.

Provenance: The Debuggability Problem

The hardest operational question in a layered configuration system is not “what is the value?” but “where did this value come from?” This is the provenance problem.

When an operator inspects a running service and sees http.client.timeout_ms = 30000, they need to know: is this the global default? Was it set at the environment level? Did someone apply an instance override three weeks ago during an incident and forget to remove it? Without provenance, debugging configuration is archaeology — you check each layer manually, if you even know which layers exist and where to look.

Systems that solve this well expose not just the resolved value but the resolution chain: which layers were consulted, which had a value defined, and which one won. Hashicorp’s Consul, for instance, supports this through its key-value hierarchy and API. Kubernetes achieves a version of this through the well-defined precedence of ConfigMaps, environment variables, and command-line arguments, though tracing which layer contributed a given value in a running pod still requires deliberate tooling.

Systems that solve this poorly — which includes most homegrown configuration systems and many file-based hierarchies — give you the resolved value and nothing else. You get the output of the function but no visibility into its evaluation. This is manageable with three layers and twenty keys. It is unmanageable with seven layers, five hundred keys, and configuration that is read by dozens of services.

Where Hierarchy Breaks

The Orphaned Override

An operator sets an instance-level override during an incident. The incident is resolved. The override stays. Three months later, the platform team changes the global default for the same key. Every instance in the fleet picks up the new default — except the one with the orphaned override, which silently continues running with the old value. The fleet is no longer homogeneous, and nobody knows until the inconsistency causes a failure.

This is the most common failure mode in configuration hierarchies. It is not a technology failure; it is a lifecycle failure. Overrides at specific layers are easy to create and easy to forget. Without a discipline of expiry — either automated (time-to-live on overrides, mandatory review of instance-level overrides) or procedural (runbook steps that include “remove temporary overrides”) — specific-level overrides accumulate as invisible technical debt.

Precedence Confusion Across Teams

In organizations where different teams own different layers — the platform team owns global defaults, service teams own service-level config, SRE owns environment-level config — the override model creates implicit authority conflicts. If the platform team sets a connection pool size as a global default, can a service team override it? Technically, yes — the hierarchy allows it. Organizationally, should they? The platform team might have set that value to protect a shared database from being overwhelmed. The service team, unaware of that constraint, overrides it to improve their own throughput. The hierarchy faithfully applies the more-specific value and the shared database falls over.

Configuration hierarchy is a governance model, not just a technical model. The precedence rules encode who is allowed to override whom. If those rules do not match the actual authority structure of your organization, the hierarchy becomes a vector for well-intentioned changes that violate system-wide invariants.

The Exploding Test Matrix

Every layer in the hierarchy that can override a value multiplies the number of configurations your system can run in. With four layers and a hundred keys, the theoretical space of possible configurations is enormous. In practice, most combinations never occur, but the ones that do occur are hard to test exhaustively. Integration tests typically run against a single resolved configuration — usually the one that matches the test environment. The production configuration, with its environment overrides and instance-specific exceptions, is a different resolved set entirely. “It works in staging” is often “it works with staging’s resolved configuration,” which is a different statement from “it works with production’s resolved configuration.”

This is an inherent cost of hierarchical configuration. The more layers you have, the more possible resolved states exist, and the less confidence any single test run provides about other environments.

The Model to Carry Forward

Configuration hierarchy is a function, not a dictionary. It takes a resolution context — the identity of the requester — and a key, walks a stack of layers in precedence order, applies merge semantics at each layer, and returns a resolved value. Every operational property of the system — debuggability, safety, predictability — depends on three design choices: how many layers exist, what the merge semantics are, and whether the resolution path is visible to operators.

The key conceptual shift is that most configuration bugs are not about wrong values at a single layer. They are about unexpected interactions between layers. A value that is correct in isolation at the layer where it was set becomes incorrect in context because of what another layer does or does not define. If you cannot answer the question “where did this value come from and why did it win?” for every key in your running system, your hierarchy is a liability.

Key Takeaways

  • Most-specific-wins is the universal default, but “most specific” is only meaningful if the hierarchy’s layers and their ordering are explicitly defined and understood by every team that touches configuration.

  • Merge semantics — replace vs. deep merge — determine whether your overrides are predictable. Deep merge is convenient until you need to distinguish “I didn’t set this key” from “I want this key removed,” which it cannot express without explicit deletion markers.

  • Lists should almost always be treated as atomic, replace-only values in a hierarchy, because the semantics of list merging (append, remove, reorder) are ambiguous and produce surprising results under composition.

  • The provenance of a resolved value — which layer it came from and why — is as operationally important as the value itself. Systems that expose only the resolved value and not the resolution path make debugging configuration incidents needlessly difficult.

  • Orphaned overrides at specific layers are the most common source of configuration drift. Any system that allows instance-level or service-level overrides needs a corresponding lifecycle mechanism — TTLs, audits, or mandatory review — to prevent forgotten overrides from silently diverging from fleet-wide intent.

  • Configuration hierarchy encodes organizational authority. If the precedence rules do not match who is actually responsible for which constraints, the hierarchy enables well-intentioned overrides that violate system-wide invariants.

  • Every additional layer in the hierarchy multiplies the space of possible resolved configurations, reducing the confidence that any single test environment provides about the behavior of other environments. Add layers only when they represent a genuinely distinct axis of variation.

← Back to Home