Configuration and secrets

What you'll learn

• Separate configuration from code: environment variables for deployment, TOML files for structure
• Load settings into one typed object at startup and fail fast when something required is missing
• Keep secrets out of the repository with .env files that are ignored and injected in production
• Layer defaults, files, environment, and command-line flags in a predictable order

Concept

Configuration is everything about a program that changes between where it runs: the database URL, the port, the log level, the API key. None of it belongs in the code, because the same code must run on your laptop, in tests, and in production with different values, and because some of those values are secrets. The twelve-factor rule is the standard: configuration lives in the environment, the program reads it at startup, and the code never contains a value that differs between deployments.

Environment variables are the universal mechanism: every operating system, container, and CI system can set them, and they never touch disk in the repository. os.environ["DATABASE_URL"] reads one, and raises KeyError if it is missing, which is the right behavior for something required. For structured, non-secret settings that have many keys, a TOML file is clearer than twenty variables, and tomllib in the standard library parses it. Command-line flags override everything for a single run, which is what argparse gave you.

Read configuration once, at startup, into a single typed object, and validate it there. A frozen dataclass Settings with a from_env() classmethod converts strings to the right types, applies defaults, and raises a clear error listing every missing required value, before the program does any work. Failing fast at startup with "DATABASE_URL is not set" beats failing an hour later with a KeyError in a request handler. The rest of the program takes Settings as a parameter, which also makes it trivial to test with a hand-built one.

Secrets need one more rule: they must never be committed, logged, or printed. Locally, a .env file holds them and .gitignore excludes it; a small loader reads it into the environment at startup for development only. In production the platform injects them: CI secrets, container environment, or a secrets manager. Log that a secret was loaded, never its value; make repr of the settings object mask it. A committed secret is compromised even after deletion, because Git history keeps it: rotate it immediately.

Code

A Settings object loaded from the environment with defaults, conversion, and fail-fast validation. The repr masks the secret.

# settings.py
import os
from dataclasses import dataclass, field


class ConfigError(Exception):
    pass


@dataclass(frozen=True)
class Settings:
    database_url: str
    api_key: str = field(repr=False)
    port: int = 8000
    debug: bool = False
    log_level: str = "INFO"

    @classmethod
    def from_env(cls, env=None):
        env = os.environ if env is None else env
        missing = [name for name in ("DATABASE_URL", "API_KEY") if not env.get(name)]
        if missing:
            raise ConfigError(f"missing required settings: {', '.join(missing)}")
        return cls(
            database_url=env["DATABASE_URL"],
            api_key=env["API_KEY"],
            port=int(env.get("PORT", 8000)),
            debug=env.get("DEBUG", "false").lower() in ("1", "true", "yes"),
            log_level=env.get("LOG_LEVEL", "INFO").upper(),
        )

    def __repr__(self):
        return (f"Settings(database_url={self.database_url!r}, api_key='***', "
                f"port={self.port}, debug={self.debug}, log_level={self.log_level!r})")


if __name__ == "__main__":
    try:
        Settings.from_env({})
    except ConfigError as err:
        print("startup refused:", err)

    settings = Settings.from_env({
        "DATABASE_URL": "sqlite:///app.db", "API_KEY": "sk-secret", "DEBUG": "true", "PORT": "9000",
    })
    print(settings)
    print(settings.port + 1, settings.debug, settings.api_key == "sk-secret")

startup refused: missing required settings: DATABASE_URL, API_KEY
Settings(database_url='sqlite:///app.db', api_key='***', port=9000, debug=True, log_level='INFO')
9001 True True

Structured settings from TOML, with environment variables overriding individual values. tomllib parses; a small merge applies the layers in order.

# layered.py
import os
import tomllib
from pathlib import Path

Path("config.toml").write_text('''
[server]
port = 8000
workers = 4

[logging]
level = "INFO"
format = "json"
''', encoding="utf-8")

DEFAULTS = {"server": {"port": 8000, "workers": 1, "timeout": 30}, "logging": {"level": "WARNING", "format": "text"}}


def load(path, env):
    config = {section: dict(values) for section, values in DEFAULTS.items()}
    with open(path, "rb") as f:
        for section, values in tomllib.load(f).items():
            config.setdefault(section, {}).update(values)
    for key, value in env.items():                       # APP_SERVER_PORT=9000 -> config["server"]["port"]
        if key.startswith("APP_"):
            _, section, name = key.lower().split("_", 2)
            current = config.get(section, {}).get(name)
            config.setdefault(section, {})[name] = type(current)(value) if current is not None else value
    return config


print(load("config.toml", {})["server"])
print(load("config.toml", {"APP_SERVER_PORT": "9000", "APP_LOGGING_LEVEL": "DEBUG"}))

{'port': 8000, 'workers': 4, 'timeout': 30}
{'server': {'port': 9000, 'workers': 4, 'timeout': 30}, 'logging': {'level': 'DEBUG', 'format': 'json'}}

A .env loader for local development, the .gitignore line that keeps it out, and the check that the secret never reaches the log.

# dotenv.py
import logging
import os
from pathlib import Path

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
log = logging.getLogger("startup")


def load_dotenv(path=".env"):
    """Set variables from a KEY=value file, without overriding ones already set."""
    loaded = []
    for line in Path(path).read_text(encoding="utf-8").splitlines():
        line = line.strip()
        if not line or line.startswith("#") or "=" not in line:
            continue
        key, value = line.split("=", 1)
        if key not in os.environ:
            os.environ[key] = value.strip().strip('"')
            loaded.append(key)
    return loaded


Path(".env").write_text('# local only\nDATABASE_URL="sqlite:///dev.db"\nAPI_KEY=sk-local-secret\n', encoding="utf-8")
Path(".gitignore").write_text(".env\n.venv/\n__pycache__/\n", encoding="utf-8")
os.environ.pop("API_KEY", None)
os.environ["DATABASE_URL"] = "postgres://prod"          # already set: .env must not override it

loaded = load_dotenv()
log.info("loaded %d settings from .env: %s", len(loaded), ", ".join(loaded))
print(os.environ["DATABASE_URL"], "<- kept the existing value")
print("secret present:", bool(os.environ["API_KEY"]), "| printed anywhere:", "sk-local" in open(".gitignore").read())
print(".env ignored by git:", ".env" in Path(".gitignore").read_text().split())

INFO: loaded 1 settings from .env: API_KEY
postgres://prod <- kept the existing value
secret present: True | printed anywhere: False
.env ignored by git: True

Command-line flags as the last layer, and the whole thing wired into a main that fails fast. The program body receives a finished Settings and never touches the environment.

# main.py
import argparse
import sys

from settings import ConfigError, Settings


def run(settings):
    return f"serving on port {settings.port} (debug={settings.debug}, log={settings.log_level})"


def main(argv=None, env=None):
    parser = argparse.ArgumentParser(prog="app")
    parser.add_argument("--port", type=int)
    parser.add_argument("--debug", action="store_true")
    args = parser.parse_args(argv)
    try:
        settings = Settings.from_env(env)
    except ConfigError as err:
        print(f"error: {err}", file=sys.stderr)
        return 1
    if args.port:
        settings = Settings(**{**settings.__dict__, "port": args.port})
    if args.debug:
        settings = Settings(**{**settings.__dict__, "debug": True})
    print(run(settings))
    return 0


base = {"DATABASE_URL": "sqlite:///app.db", "API_KEY": "k", "PORT": "8000"}
print("exit", main([], base))
print("exit", main(["--port", "9100", "--debug"], base))
print("exit", main([], {}))

serving on port 8000 (debug=False, log=INFO)
exit 0
serving on port 9100 (debug=True, log=INFO)
exit 0
error: missing required settings: DATABASE_URL, API_KEY
exit 1

Try it yourself

Give the expense CLI a Settings object: the input folder, a currency, a report format, and an optional API key for a future exchange-rate service. Load defaults, then config.toml, then environment variables, then flags, in that order, and fail at startup with every missing required name listed. Add a .env for local values, confirm it is gitignored, and write a test that builds Settings directly and a test that checks the missing-variable error message.

Common mistakes

• Reading os.environ throughout the code. Scattered reads make configuration undiscoverable and untestable; read once into Settings and pass it.
• Committing .env. It holds secrets; add it to .gitignore before creating it, and rotate anything that slips through.
• Printing or logging the settings object with the secret inside. Mask secrets in repr, and never log their values.

What's next

• Configuration lives outside the code: defaults, then a TOML file, then environment variables, then flags; each layer overrides the last.
• Load once into a typed, validated Settings and fail fast with every missing name; pass it to the code that needs it.
• Secrets come from .env locally and the platform in production, are never committed, and are masked in logs.

Your program is configured safely. Next you will make quality automatic: ruff, mypy, and tests enforced by pre-commit locally and by CI on every push.