tlcconfig

Configuration system internals.

Infrastructure package for 3LC’s configuration system. Intended for plugin authors and internal bootstrapping; not part of the public user-facing API.

Two layers:

  1. Options (tlcconfig.options): pure metadata describing each configuration option (key, type, default, env-var, CLI mapping).

  2. Store (tlcconfig.store): per-source ledger of raw values, tracking where each value came from. No validation, no transformation, no business logic.

Layering contract: tlcconfig is independent of higher packages. No module in this package may eagerly import from upstream layers. Lazy imports inside function bodies are tolerated only where unavoidable — each such import is a known layering violation and should be scheduled for removal.

Package Contents

Classes

Class

Description

ConfigGroup

Namespace that delegates storage and notifications to a parent Configuration.

ConfigLogic

Centralized configuration transformation and validation logic.

ConfigProperty

Descriptor that couples option docs with transformation and validation.

ConfigSource

The source of an option’s value.

ConfigStore

Lightweight configuration store.

Option

Immutable definition of a configuration option.

OptionCategory

Categories for organizing configuration options.

OptionRegistry

Central registry of all configuration options.

ProvenancedItem

A single sub-element of a compound option, with its own provenance.

API

class ConfigGroup(
parent: Any,
)

Namespace that delegates storage and notifications to a parent Configuration.

ConfigProperty descriptors on a ConfigGroup call _get_value/_set_value, which this class forwards to the parent. The parent owns the cache, the notification system, and the dirty/timestamp tracking.

classmethod add_property(
name: str,
option: Option[Any],
*,
transform: Callable[[Any, ConfigSource, str | None], Any] | None = None,
validate: Callable[[Any], bool] | None = None,
) → ConfigProperty[Any]

Attach a ConfigProperty to an existing ConfigGroup subclass.

Use this when a subsystem needs to extend a group after class creation — e.g. a plugin module registers options against a group owned by another package. Python’s descriptor protocol does not invoke __set_name__ on plain setattr, so a directly-assigned descriptor would leave ConfigProperty._name empty (breaking object_property_name and Object-schema serialization). This helper wires it up correctly.

For class-body declarations (the usual case in a subsystem’s own ConfigGroup subclass), keep using the normal foo: T = ConfigProperty(OPT) form — __set_name__ fires automatically at class creation.

class ConfigLogic

Centralized configuration transformation and validation logic.

All business rules for configuration values live here. This keeps the schema pure metadata and the loader dumb.

static expand_path(
raw: str | None,
source: ConfigSource,
source_info: str | None,
absolutize: bool = True,
) → str

Expand environment variables, ~, and anchored paths.

Parameters:

  • raw – The raw path string.

  • source – Where the value came from (for anchored path resolution).

  • source_info – Source context (config file path for anchoring).

  • absolutize – Whether to convert relative paths to absolute.

Returns:

The expanded path string.

static is_valid_display_progress(
value: bool,
) → bool

Validate display progress value.

Parameters:

value – The display progress value.

Returns:

True if value is a boolean.

static is_valid_log_level(
level: str,
) → bool

Validate that a log level is valid.

Parameters:

level – The log level string.

Returns:

True if level is a valid Python log level.

static is_valid_port(
port: int,
) → bool

Validate that a port number is valid.

Parameters:

port – The port number.

Returns:

True if port is in valid range (1-65535).

static is_writable_directory(
path: str,
) → bool

Validate that a path is a writable directory.

Creates the directory if it doesn’t exist.

Parameters:

path – The path to validate.

Returns:

True if the path is writable.

static is_writable_file(
path: str,
) → bool

Validate that a file path is writable.

Creates parent directory if it doesn’t exist.

Parameters:

path – The file path to validate.

Returns:

True if the file path is writable.

static transform_aliases(
raw: dict[str, Any] | None,
source: ConfigSource,
source_info: str | None,
) → dict[str, str]

Expand paths in alias values.

Parameters:

  • raw – The raw aliases dict.

  • source – Where the value came from.

  • source_info – Source context for path expansion.

Returns:

Dict with expanded paths as values.

static transform_root_url(
raw: Any,
source: ConfigSource,
source_info: str | None,
) → Any

Transform root URL to a Url object.

Parameters:

  • raw – The raw root URL (string, Url, or None).

  • source – Where the value came from.

  • source_info – Source context for path expansion.

Returns:

A Url object wrapping the expanded path.

Raises:

TypeError – If raw is not a string, Url, or None.

static transform_scan_urls(
raw: str | list | None,
source: ConfigSource,
source_info: str | None,
) → list[dict]

Normalize scan URLs to a list of dicts.

Every entry is returned in dict form so downstream consumers see a single shape. Plain-string entries are wrapped as {"url": <expanded>, "layout": "project"}; dict entries pass through with the url field path-expanded and layout defaulted to "project" if unset.

Handles input formats:

  • String: “url1,url2” -> [{“url”: “url1”, …}, {“url”: “url2”, …}]

  • List of strings: [“url1”, …] -> [{“url”: “url1”, …}, …]

  • List of dicts: [{“url”: “…”, “layout”: “flat”}] -> passed through (path-expanded)

Parameters:

  • raw – The raw scan URLs value.

  • source – Where the value came from.

  • source_info – Source context for path expansion.

Returns:

List of dicts. Each dict has at least a url key and a layout key.

class ConfigProperty(
option: Option[tlcconfig.property.T],
transform: Callable[[Any, ConfigSource, str | None], tlcconfig.property.T] | None = None,
validate: Callable[[tlcconfig.property.T], bool] | None = None,
)

Bases: typing.Generic[tlcconfig.property.T]

Descriptor that couples option docs with transformation and validation.

This descriptor provides:

  • __doc__ from the option (single source of truth)

  • Transformation on get (via transform callable)

  • Validation on set (via validate callable)

The descriptor delegates to the owner’s _get_value and _set_value methods, which handle caching, source tracking, and notifications. Typical owners are Configuration or a ConfigGroup attached to one.

Variables:

  • option – The option definition (provides docs, default, key).

  • transform – Optional transformation function for get.

  • validate – Optional validation function for set.

Initialize the property descriptor.

Parameters:

  • option – The option defining this property.

  • transform – Optional function to transform raw values. Signature: (value, source, source_info) -> T. The input is intentionally untyped (Any) because transforms normalize heterogeneous raw inputs (YAML/env strings, pre-parsed objects, None) into the logical type T.

  • validate – Optional function to validate values on set, invoked after transform (or the data_type isinstance check) has produced a T. Signature: (value: T) -> bool (True if valid).

property object_property_name: str

The property name to use in Object schema.

Uses option.object_property_name if set, otherwise falls back to the attribute name (set via set_name).

class ConfigSource

Bases: enum.Enum

The source of an option’s value.

This enum is used during option resolution to track and store from which source the option was set.

API = 8

The option was set using the API.

COMMAND_LINE = 7

The option was set on the command line.

DATA_CONFIG_FILE = 3

The option was loaded from a config file discovered alongside data — e.g. shipped on a read-only data mount, or scanned via the indexing table. The system never assumes write access here.

DEFAULT = 0

The option was set with the default value.

ENVIRONMENT = 6

The option was loaded from an environment variable.

INDEXER = 2

The option was set as runtime bookkeeping by an indexer or other internal subsystem.

Lower precedence than any user-authored source so user env / CLI / config-file writes always win. Not persisted to disk by file writers — entries are derived from runtime state (e.g. an indexer discovering a project-level scan-url) and would otherwise pollute saved configs with state that re-derives itself next session.

MIXED = 1

The option was set from multiple sources. Indicates a compound value.

PROJECT_ROOT_CONFIG_FILE = 4

The option was loaded from a project-root config.3lc.yaml. Beats DATA (data-bundled) so project-level settings override what travels with data; loses to USER so the same project can be run by different users without each user’s setup leaking into the project.

USER_CONFIG_FILE = 5

The option was loaded from the user’s writable, persistent config (~/.config/3lc/config.yaml, or whatever --config-file / TLC_CONFIG_FILE points at). One file per user, shared across all projects that user runs.

class ConfigStore(
config_file: str | None = None,
load_defaults: bool = True,
load_env: bool = True,
)

Lightweight configuration store.

This store is intentionally “dumb” — it reads values and tracks sources, but performs no validation or transformation. This keeps it fast to import and free of side effects.

The store follows a precedence order (highest to lowest):

  1. API (programmatic setting via set())

  2. Command line arguments

  3. Environment variables

  4. Config file

  5. Secondary config files (project-level configs)

  6. Defaults (from registered options)

Variables:

config_files – List of config files that were loaded.

Initialize the configuration store.

Parameters:

  • config_file – Path to config file. If None, auto-discovers. If empty string “”, skips config file loading.

  • load_defaults – Whether to load default values from registered options.

  • load_env – Whether to load values from environment variables.

add_item(
key: str,
sub_key: Any,
value: Any,
source: ConfigSource,
source_info: str | None = None,
) → None

Add a single sub-element to a compound option without replacing the rest of the slice.

Unlike :meth:set_value (which replaces the entire (source, source_info) slice with the new value), add_item appends one ProvenancedItem leaving any other items already present at the same source intact. Used by the facade verbs append / update and by callers that want surgical, single-element writes.

Parameters:

  • key – Option key.

  • sub_key – Dict sub-key to set, or None for list-compound append.

  • value – Value to set / append.

  • source – ConfigSource for this write.

  • source_info – Additional source context.

Raises:

  • ValueError – If the option is not registered or is not a compound.

  • ValueError – For dict-compound options when sub_key is None.

bootstrap_from_root_config() → None

Load <ROOT_URL>/config.3lc.yaml if it exists.

Resolves ROOT_URL from this store, joins it with the default config-file name, and loads the file at PROJECT_ROOT_CONFIG_FILE precedence if present. No-op when ROOT_URL is unset or no file exists at the resolved path.

Lives on the store (not on Configuration) so callers that only need tlcconfig — notably tlccli.main’s root callback — can invoke it without triggering import tlc / init_global_objects.

claim_pending(
key: str,
) → None

Move any pending entries for key into the live store.

Idempotent. Called by OptionRegistry.register when an option is registered after a config source has already been loaded. Routes each pending entry through _set_raw so compound options get their per-item splitting applied retroactively (the option wasn’t registered when the value was first stashed, so the splitter couldn’t see it then — now it can).

claim_pending_env(
option: Option[Any],
) → None

Drain stashed TLC_* env vars matching option’s envvar bindings.

Mirrors the eager-pass logic in _load_environment: canonical wins over deprecated aliases, regex envvars match every stashed name once. Deprecation warnings fire here (claim time) rather than at load time because the option wasn’t registered when the env was scanned.

Idempotent. Called by OptionRegistry.register.

property config_files: Mapping[str, ConfigSource]

Read-only mapping of loaded config file paths to the tier they were loaded at, in load order.

in membership, iteration (yields paths), and .get(path) all work as expected. For the first user-tier file (the writable target for 3lc config project-root) use :attr:user_config_path. For tier-filtered lookups use :meth:files_at.

files_at(
source: ConfigSource,
) → list[str]

Return the loaded config files at the given tier, in load order.

get(
key: str,
) → Any | None

Get the highest-precedence raw value for a key.

Parameters:

key – Option key (e.g., “service.port”).

Returns:

The raw value, or None if not set.

get_all_raw(
key: str,
) → list[RawValue]

Get all raw values for a key (useful for compound types).

For non-compound keys: returns the list of raw entries verbatim. For compound keys: synthesizes one RawValue per (source, source_info) group by reassembling the per-item ProvenancedItem entries — back-compat shim for callers that used to receive whole-tier writes here. Use

Meth:

iter_provenanced if you actually want per-item provenance.

Parameters:

key – Option key.

Returns:

List of all RawValues for the key.

get_merged(
key: str,
) → RawValue | None

Get the merged value for compound types, or highest-precedence for simple types.

For compound types (dict/list), merges all per-item entries in precedence order (lowest first so higher precedence overwrites on dict-key collisions; latest-wins on list-item structural equality). The wrapper’s source is set to

Class:

ConfigSource.MIXED when items span tiers, otherwise the single tier shared by all items. For simple types, returns the highest-precedence value (same as get_raw).

Parameters:

key – Option key.

Returns:

RawValue with merged value and provenance summary, or None if not set.

get_provenance(
key: str,
) → ProvenancedItem | list[ProvenancedItem] | dict[Any, ProvenancedItem] | None

Return per-element provenance, polymorphic by the option’s compound type.

  • Scalar option (or unregistered key): single

    class:

    ProvenancedItem with the highest-precedence value, or None if unset.

  • List-compound option: list[ProvenancedItem] in merged order, deduped by structural identity. The first tier (lowest precedence) that contributed each item is the surviving provenance — matches the get_merged dedup rule. Empty list if unset.

  • Dict-compound option: dict[K, ProvenancedItem] keyed by sub-key. Per-key, the highest-tier writer wins. Empty dict if unset.

Pure query — does not mutate the store, never warns. Use the returned container with native Python idioms; there are no index= or sub_key= kwargs (you index/iterate the container yourself).

Parameters:

key – Option key.

Returns:

Polymorphic provenance view; see above.

get_raw(
key: str,
) → RawValue | None

Get raw value with source info.

For compound options, delegates to :meth:get_merged so callers that haven’t been ported to per-item provenance still see a single RawValue wrapper. The wrapper’s source will be

Class:

ConfigSource.MIXED if items span tiers.

Parameters:

key – Option key.

Returns:

RawValue with value and source info, or None if not set.

get_source(
key: str,
) → ConfigSource | None

Get the source of a value.

Parameters:

key – Option key.

Returns:

The ConfigSource, or None if not set.

classmethod instance() → ConfigStore

Get the global ConfigStore instance.

Creates a new instance if none exists.

Returns:

The global ConfigStore instance.

load_cli(
args: dict[str, Any],
) → None

Load values from parsed CLI arguments.

Parameters:

args – Dictionary of CLI argument values (from argparse or click).

load_config(
path: str,
source: ConfigSource = ConfigSource.USER_CONFIG_FILE,
) → None

Load a YAML config file at the given source tier.

Source-explicit public loader. Reloading the same (path, source) replaces any previously loaded entries from that path at that tier so callers can re-invoke this method when the file changes.

Parameters:

  • path – Path to the YAML file.

  • source – The :class:ConfigSource tier to load at. Defaults to USER_CONFIG_FILE.

load_data_config(
path: str,
) → None

Load a data-bundled (secondary) config file. Wrapper for

Meth:

load_config at DATA_CONFIG_FILE tier.

Lower precedence than project-root, user-global, environment, and command line. Used for config files discovered alongside project data — e.g. shipped on a read-only mount or scanned via the indexing table.

load_project_root_config(
path: str,
) → None

Load a project-root config.3lc.yaml. Wrapper for

Meth:

load_config at PROJECT_ROOT_CONFIG_FILE tier.

Beats data-bundled, loses to user-global so the user’s writable global config still wins.

pending_count() → int

Return the number of unclaimed pending entries.

Sum of YAML keys parked in _pending_unknown and TLC_* env vars parked in _pending_env — i.e. values the user supplied that no Option has claimed yet. Pure query; does not warn. Callers can build optional strict modes, status panes, or post-bootstrap diagnostics on top of this.

pending_env_vars() → list[str]

Return the list of TLC_* env vars stashed for an unregistered option.

pending_unknown_keys() → list[str]

Return the list of keys still waiting for an Option to claim them.

remove_item(
key: str,
sub_key_or_value: Any,
source: ConfigSource,
source_info: str | None = None,
) → bool

Remove a single sub-element from a compound option at the given tier.

For dict-compound options, sub_key_or_value is the sub-key to remove. For list-compound options, it’s the value to match by structural identity (the same json.dumps(sort_keys=True) rule get_merged uses for dedup). Only items at the specified (source, source_info) tier are affected — lower-tier items resurface in the merged view.

Parameters:

  • key – Option key.

  • sub_key_or_value – For dict, the sub-key to remove; for list, the value to match.

  • source – ConfigSource of the entry to remove.

  • source_info – Source-info of the entry to remove.

Returns:

True if at least one item was removed, False otherwise.

Raises:

ValueError – If the option is not registered or is not a compound.

classmethod reset_instance() → None

Reset the global instance. Primarily for testing.

set(
key: str,
value: Any,
) → None

Programmatically set a value (highest precedence).

Parameters:

  • key – Option key.

  • value – Value to set.

classmethod set_instance(
store: ConfigStore,
) → None

Set the global ConfigStore instance.

Used by CLI to set up the store before subcommands run.

Parameters:

store – The store instance to use globally.

set_value(
key: str,
value: Any,
source: ConfigSource,
source_info: str | None = None,
) → None

Set a value with an explicit source.

Any existing raw entry with the same (source, source_info) tuple is replaced so repeated calls stay idempotent and the internal raw value list does not grow unboundedly. For compound options the sweep operates on the per-item store (_compound).

Parameters:

  • key – Option key.

  • value – Value to set.

  • source – The ConfigSource indicating where this value came from.

  • source_info – Additional context about the source (e.g., file path).

stash_unknown(
key: str,
value: Any,
source: ConfigSource,
source_info: str | None = None,
) → None

Record a value for a key that no Option has claimed yet.

Used by loaders (and by callers contributing unknown values from higher-precedence tiers) so that a later OptionRegistry.register can replay the value at its original source. See claim_pending.

to_default_yaml(
include_docs: bool = True,
) → str

Generate YAML with only default values.

Parameters:

include_docs – Whether to include documentation comments.

Returns:

YAML-formatted configuration with defaults.

to_yaml(
include_docs: bool = True,
detail: bool = False,
) → str

Serialize current configuration to YAML format.

Parameters:

  • include_docs – Whether to include documentation comments.

  • detail – Whether to include source info for non-default values.

Returns:

YAML-formatted configuration string.

property user_config_path: str | None

First file loaded at USER_CONFIG_FILE tier, or None.

Authoritative answer to “where do I write user-level config?” — used by tlccli config project-root. Returning the first file in

Attr:

config_files is wrong: it can be a DATA-tier file the indexer picked up before the user-tier load.

validate() → None

Validate that required options are set.

Raises:

ValueError – If any required options are missing or empty.

warn_unclaimed_pending() → None

Emit a WARNING for each newly-unclaimed pending key/env var.

Intended to be called at the end of bootstrap (e.g. in init_global_objects) so plugin keys that never got claimed — typos, removed-but-still-configured options, plugins not yet imported — surface to the user instead of silently rotting.

The pending stash is not cleared: a plugin imported later in the same process can still claim its values via the late-bind path. Already-warned entries are tracked so repeated checkpoint calls (e.g. periodic re-init in long-running services) stay quiet unless new unclaimed entries appear. Use pending_count to observe the queue size programmatically.

write_to_yaml_file(
path: str,
force: bool = False,
include_docs: bool = True,
detail: bool = False,
) → bool

Write configuration to a YAML file.

Parameters:

  • path – Path to write to.

  • force – Whether to overwrite existing file.

  • include_docs – Whether to include documentation comments.

  • detail – Whether to include source info.

Returns:

True if file was written, False if file exists and force=False.

class Option

Bases: typing.Generic[tlcconfig.options.T]

Immutable definition of a configuration option.

This class contains ONLY metadata — no logic, no transformation, no validation. This is the single source of truth for option documentation.

Variables:

  • key – Fully qualified key path in config file (e.g., “service.port”).

  • category – Category for grouping and organization.

  • data_type – Python type of the option value.

  • default – Default value when not configured. None means no default.

  • default_factory – Callable that returns a computed default value. Used when default is None but a runtime-computed default is needed.

  • required – Whether the option must be set.

  • envvar – Environment variable name, or regex pattern for compound options.

  • envvar_aliases – Deprecated env var names still honored as input (with a warning). Use to keep an old env var working after a rename. String aliases only — patterns are not supported.

  • cli_argument – CLI argument flag (e.g., “–port”).

  • cli_metavar – Placeholder shown in CLI help (e.g., “PORT”).

  • cli_command – CLI command this option belongs to (e.g., “service”). If set, the CLI argument is only exposed under that command.

  • compound_type – For compound options, the container type (list or dict).

  • description – Full documentation string. First line used as brief help.

  • cli_visible – When True (the default), the option is surfaced through CLI tooling — it appears in 3lc --help (the dynamic-options generator picks it up) and in the 3lc config show default output. Set False for runtime / dev flags set by the framework itself rather than the operator.

  • serializable – When True (the default), the option’s value is included in serialized representations of Configuration: written to YAML by ConfigStore.write_to_yaml_file / to_default_yaml, published on the /configuration Object Service endpoint, and rendered on the API reference page. Set False for credentials and internal runtime state that should never round-trip through a config file or be exposed to clients. Independent of cli_visible — credentials are typically cli_visible=True (so --license / --api-key work) but serializable=False. Convention: the matching attribute on Configuration / ConfigGroup carries a leading underscore for non-serializable options, as a redundant in-source readability cue.

  • hide_default_in_config – If True, default is not written to config files.

  • choices – Valid values for enum-like options.

  • object_property_name – Name to use in Object schema. Defaults to key if not set.

property brief: str

First line of description, used for CLI help.

category: OptionCategory = None
choices: list[Any] | None = None
cli_argument: str = <Multiline-String>
cli_command: str | None = None
cli_metavar: str | None = None
cli_visible: bool = True
compound_type: type | None = None
data_type: type[tlcconfig.options.T] = None
default: Any = None
default_factory: Callable[[], tlcconfig.options.T] | None = field(...)
description: str = <Multiline-String>
envvar: str | Pattern[str] = <Multiline-String>
envvar_aliases: tuple[str, ...] = ()
hide_default_in_config: bool = False
key: str = None
object_property_name: str | None = None
required: bool = False
serializable: bool = True
class OptionCategory

Bases: enum.Enum

Categories for organizing configuration options.

Categories are used for:

  • Grouping options in CLI help output

  • Organizing options in config file output

  • Filtering options by domain (e.g., service-only options)

External subsystems (e.g., URL adapters) can register their own categories and options to own their config namespace. Use OptionRegistry.register() with a custom category to add options from outside the core config package.

EXTENSIONS = extensions

Options for loading external extension classes (URL adapters, sample types, exporters).

INDEXING = indexing

Options related to project and data indexing.

LOGGING = logging

Options for logging configuration.

SERVICE = service

Options for the Object Service server.

TLC = tlc

General 3LC options.

class OptionRegistry

Central registry of all configuration options.

This registry enables CLI introspection and iteration over options without coupling to business logic.

.. rubric:: Example

# Register a new option
MY_OPTION = OptionRegistry.register(Option(
    key="my.option",
    category=OptionCategory.TLC,
    data_type=str,
    default="value",
    description="My custom option.",
))

# Get option by key
option = OptionRegistry.get("my.option")

# Iterate all options
for key, option in OptionRegistry.all().items():
    print(key)
classmethod all() → dict[str, Option[Any]]

Get all registered options.

Returns:

Dictionary mapping keys to options.

classmethod by_category(
category: OptionCategory,
) → dict[str, Option[Any]]

Get options filtered by category.

Parameters:

category – The category to filter by.

Returns:

Dictionary of options in the given category.

classmethod by_command(
command: str,
) → dict[str, Option[Any]]

Get options that belong to a specific CLI command.

Parameters:

command – The CLI command name (e.g., “service”).

Returns:

Dictionary of options for that command.

classmethod clear() → None

Clear all registered options. Primarily for testing.

classmethod get(
key: str,
) → Option[Any]

Get option by key.

Parameters:

key – The option key (e.g., “service.port”).

Returns:

The option.

Raises:

KeyError – If no option exists for the key.

classmethod register(
option: Option[tlcconfig.options.T],
) → Option[tlcconfig.options.T]

Register an option.

Parameters:

option – The option to register.

Returns:

The registered option (for assignment to module-level variable).

Raises:

ValueError – If an option with the same key is already registered.

classmethod with_cli_argument(
include_cli_invisible: bool = False,
) → dict[str, Option[Any]]

Get options that have CLI arguments.

Parameters:

include_cli_invisible – When True, include options marked cli_visible=False (otherwise they are filtered out).

Returns:

Dictionary of options that can be set via CLI.

class ProvenancedItem

A single sub-element of a compound option, with its own provenance.

Compound options (lists, dicts) can mix items from different sources — e.g. some scan-urls from YAML, others from env vars. Each item is stored independently so callers can ask “where did this URL come from?” via :meth:ConfigStore.get_provenance (which returns list[ProvenancedItem] for list-compounds, dict[K, ProvenancedItem] for dict-compounds, or a single instance for scalars). Frozen so instances are hashable and safe to dedupe.

Variables:

  • value – The single sub-element (a list element, or a one-key dict slice {k: v} for dict-compounds).

  • source – Where this specific element came from.

  • source_info – Additional context (filename, env var name).

source: ConfigSource = None
source_info: str | None = None
value: Any = None