`tlc.configuration`¶

The 3LC configuration types.

3LC reads its settings — indexing roots, logging verbosity, extension toggles, and more — from a single precedence-resolved store that merges environment variables, configuration files, and runtime overrides. At runtime that store is reached through tlc.config, a lazily-resolved alias for the live Configuration singleton:

import tlc

print(tlc.config.project_root_url)
tlc.config.indexing.scan_urls = ["./data"]
tlc.config.logging.level = "DEBUG"

This module is where the types behind that instance live. Configuration is the store itself, and its logging, indexing, and extensions attributes are instances of the ConfigGroup subclasses LoggingConfig, IndexingConfig, and ExtensionsConfig. The classes are exposed here so the attributes on tlc.config have a documented type to point at; most code reaches them through the live instance rather than constructing them directly.

Package Contents¶

Classes¶

Class	Description
`Configuration`	3LC runtime configuration.
`ExtensionsConfig`	Extensions configuration group (`config.extensions.*`).
`IndexingConfig`	Indexing configuration group (`config.indexing.*`).
`LoggingConfig`	Logging configuration group (`config.logging.*`).

API¶

Bases: tlc._core.objects.mutable_object.MutableObject

3LC runtime configuration.

This singleton object contains all runtime configuration settings for this instance of 3LC, including root and scan URLs, aliases, loglevel and more.

Parameters:

url – The URL of the object.
created – The creation time of the object.
init_parameters – A dictionary containing the initial values of the object’s properties.

aliases: dict = {}¶

URL alias definitions for path shortcuts.

Dictionary mapping alias names to target paths/URLs. Aliases are used with angle brackets in URLs: <ALIAS_NAME>/path/to/file.

Example: aliases: DATASETS: /data/datasets MODELS: s3://bucket/models

Config key: aliases
Environment variable: TLC_ALIAS_([A-Z][A-Z0-9_]*)
Python attribute: config.aliases

append( option: Option[list], value: Any, *, source: ConfigSource | None = None, source_info: str = 'programmatic', notify: bool = True, ) → None¶

Append a single item to a list-compound option at the given tier.

Unlike whole-list assignment (config.scan_urls = [...]), this leaves all other items at the same (source, source_info) tier in place. Per-item provenance is preserved — each sub-element carries its own source.

Parameters:

option – A list-compound option (Option[list]).
value – The single item to append.
source – Tier to write at. Defaults to ConfigSource.API.
source_info – Source context. Defaults to "programmatic".
notify – Fire the internal _on_changed hook (bumping last_modified) if the post-transform effective value changed. Default True.

Raises:

ValueError – If option is not a list-compound.

static bootstrap_from_root_config( store: Any = None, ) → None¶

Load a project-level config file from the directory pointed to by ROOT_URL.

Thin wrapper around :meth:tlcconfig.store.ConfigStore.bootstrap_from_root_config, retained for back-compat — callers that only need tlcconfig should prefer the store method directly to avoid pulling tlc.* into their import graph.

Parameters:: store – The ConfigStore to read/write. Defaults to the singleton instance.

property config_files: Mapping[str, ConfigSource]¶: Mapping of loaded config file paths to their load tier.

configuration_instance: Configuration | None = None¶

delete_item( option: Option[dict], sub_key: Any, *, source: ConfigSource | None = None, source_info: str = 'programmatic', notify: bool = True, ) → None¶

Remove one sub-key from a dict-compound option at the given tier.

Only the entry at (source, source_info) is removed; lower-tier entries for the same sub-key resurface in the merged view.

Named delete_item (not delete) to avoid colliding with AddressableObject.delete(), which deletes the Configuration object itself from storage.

Parameters:

option – A dict-compound option.
sub_key – The sub-key to remove.
source – Tier to remove from. Defaults to ConfigSource.API.
source_info – Source context. Defaults to "programmatic".
notify – Fire the internal _on_changed hook (bumping last_modified) if the post-transform effective value changed. Default True.

Raises:

ValueError – If option is not a dict-compound.
KeyError – If no matching entry exists at the requested tier.

display_progress: bool = True¶

Whether to display progress bars.

Config key: display-progress
Environment variable: TLC_DISPLAY_PROGRESS
CLI argument: --display-progress
Python attribute: config.display_progress

property effective_scan_urls: list[dict[str, Any]]¶

The composed list of URLs the indexer will scan.

The list is derived from project_root_url, scan_urls, and contributions from registered UrlAdapter instances that override the default_*_scan_urls extension hooks — it is not a configurable option. Mutate it by setting either of the underlying properties or by registering an adapter that contributes defaults.

Composition order:

Project root, stamped with layout=project.
User-supplied entries from scan_urls (plain strings wrapped to dicts with layout=project; existing dicts pass through with layout defaulted).
Adapter-contributed defaults — project / table / run entries collected from each registered adapter. Read on every property access so adapters registered after Configuration materializes still contribute on the next read.

User-facing layout values (project, flat) are preserved here. See effective_scan_urls_for() for the typed, indexer-ready projection that does the flat → single_dir translation, per-type filtering, and parsing through _parse_scan_url.

effective_scan_urls_for( constrain_to_type: str, ) → list[Any]¶

The composed, indexer-ready list of scan URLs for one object type.

Builds on effective_scan_urls and applies the indexer-side projection in one place so each indexer subclass doesn’t repeat it:

Filters out entries whose object_type doesn’t match constrain_to_type. Entries without an object_type are kept and stamped with constrain_to_type.
Translates the user-facing flat layout to the internal single_dir name.
Validates and parses each entry through _parse_scan_url, producing typed _ScanUrl dicts (with url as a Url object, etc.) ready for an indexer to consume.

The return type is annotated as list[Any] to keep tlc._core.objects.tables.system_tables.indexing out of this module’s import graph; the actual element type is _ScanUrl.

Parameters:: constrain_to_type – One of "table", "run", "config".
Returns:: Typed scan-URL entries. Project root is always first.

ensure_dependent_properties() → None¶

Framework override of :meth:SerializableObject.ensure_dependent_properties.

Not a user-facing API — the framework’s polymorphic dispatch invokes this on every SerializableObject. The Configuration override wires up the lazy ConfigIndexingTable link the first time the singleton is materialized, and then delegates to the indexer’s own ensure_dependent_properties so its dependent state stays current.

extensions: ExtensionsConfig = None¶

get( schema_name: str, default: Any = None, ) → Any¶

Flat lookup used by the Object serialization machinery.

JsonHelper.to_minimal_dict reaches for attributes with getattr(obj, key) and falls back to obj.get(key). Top-level ConfigProperty descriptors are already resolvable via getattr; this fallback resolves the group-prefixed names produced by _iter_config_properties (service_port → self.service.port).

Parameters:

schema_name – Flat schema name as registered in _add_object_properties_to_schema.
default – Value returned when the name cannot be resolved.

get_provenance( option: Option[Any], ) → ProvenancedItem | list[ProvenancedItem] | dict[Any, ProvenancedItem] | None¶

Return per-element provenance for option, polymorphic by compound type.

Thin wrapper over :meth:tlcconfig.store.ConfigStore.get_provenance that takes an Option handle (the natural caller-side identifier) instead of a string key. Returns:

scalar option → single :class:ProvenancedItem with the highest-precedence value, or None if unset.
list-compound option → list[ProvenancedItem] in merged order, deduped by structural identity. The first tier (lowest precedence) to contribute each item is the surviving provenance. Empty list if unset.
dict-compound option → dict[Any, ProvenancedItem] keyed by sub-key, with the highest-tier writer winning per sub-key. Empty dict if unset.

Re-queries the store on every call.

Parameters:: option – The option to inspect. Use the option constants from tlcconfig.options (e.g. SCAN_URLS, ALIASES, SERVICE_PORT).
Returns:: Polymorphic provenance view; see above. @overload selects the precise return shape based on option’s compound type.

.. rubric:: Example

import tlc
from tlcconfig.options import ALIASES, SERVICE_PORT, SCAN_URLS

config = tlc.config

# Scalar — single ProvenancedItem
port_prov = config.get_provenance(SERVICE_PORT)

# Dict — index by sub-key
datasets_prov = config.get_provenance(ALIASES)["DATASETS"]

# List — iterate / index / filter
env_urls = [
    p for p in config.get_provenance(SCAN_URLS)
    if p.source.name == "ENVIRONMENT"
]

get_source( key: str, ) → ConfigSource | None¶: Return the source (file, env, CLI, API) that set the value for key.

indexing: IndexingConfig = None¶

static instance() → Configuration¶: Returns the singleton Configuration object.

load_data_config( path: str, ) → None¶: Load a project-level (secondary) config file into the runtime configuration.

logging: LoggingConfig = None¶

project_root_url: Url = <computed at runtime>¶

Location for reading and writing 3LC project data.

This option is mandatory and must point to a location with write access. Supports local paths and remote URLs (s3://, gs://, az://). Environment variables and ~ are expanded. The location will be created if needed.

Config key: project-root-url
Environment variable: TLC_PROJECT_ROOT_URL
CLI argument: --project-root-url
Python attribute: config.project_root_url
Required: yes

classmethod register_group( name: str, group_class: type[ConfigGroup], ) → None¶

Subsystems call this at import time to make their group available on every Configuration — the existing singleton (if any) and any future instance created by Configuration(). This decouples group ownership from the core layer: the owning subsystem registers itself when imported.

Idempotent on repeat calls with the same group_class; raises if a different class is registered under the same name.

Parameters:

name – Attribute name (e.g., “service”).
group_class – ConfigGroup subclass to instantiate for each Configuration.

Raises:

ValueError – If name is already registered with a different class.

remove( option: Option[list], value: Any, *, source: ConfigSource | None = None, source_info: str = 'programmatic', notify: bool = True, ) → None¶

Remove a single item from a list-compound option at the given tier.

Matches by structural identity (the same dedup rule the merge pipeline uses). Only the entry at (source, source_info) is removed; lower-tier copies of the same value resurface in the merged view via the normal precedence merge.

Parameters:

option – A list-compound option.
value – The item to remove.
source – Tier to remove from. Defaults to ConfigSource.API.
source_info – Source context. Defaults to "programmatic".
notify – Fire the internal _on_changed hook (bumping last_modified) if the post-transform effective value changed. Default True.

Raises:

ValueError – If option is not a list-compound.
KeyError – If no matching entry exists at the requested tier.

scan_urls: list = []¶

Locations to scan for 3LC objects.

Each entry can be a plain URL string (defaults to layout: project) or a dict with explicit attributes:

scan-urls: # Plain string — scanned as a project tree - s3://bucket/projects - /local/projects

# Dict with explicit layout
- url: /local/loose-objects
  layout: flat

# Full control
- url: /data/tables-only
  layout: flat
  object_type: table

Supported layout values: “project” (recursive project scan), “flat” (single directory). Supported object_type values: “table”, “run” (omit to scan for all types).

The TLC_SCAN_URLS environment variable accepts a comma-separated list of URL strings.

Live propagation. Mutations to scan-urls (via Configuration.append / Configuration.remove / file reload / env reload) write through to ConfigStore under its RLock. IndexingTable instances constructed with share_configuration=True (the default for the three subclass instance() factories) run reconcile_with_configuration() at the top of every scan cycle; when nothing has changed the reconcile is a cheap pair of dict comprehensions with two no-op diff loops. Callers wanting synchronous propagation can call indexer.reconcile_with_configuration() directly. Mutations are safe from any thread under ConfigStore’s RLock.

Config key: scan-urls
Environment variable: TLC_SCAN_URLS
Python attribute: config.scan_urls

service: tlc._service.config.ServiceConfig = None¶

set_value( key: str, value: Any, source: ConfigSource, source_info: str | None = None, *, notify: bool = False, ) → None¶

Set a config value and record its provenance.

Writes to the underlying ConfigStore. Descriptor reads (e.g. via the ConfigProperty machinery) go through _get_value which reads the merged store value on every call and re-applies the option’s transform — no local cache to invalidate.

Parameters:

key – The option key.
value – The new value.
source – The provenance to record (USER_CONFIG_FILE, ENVIRONMENT, API, …).
source_info – Additional source context (file path, env var name, …).
notify – If True, fire the internal _on_changed hook when the effective value actually changes (bumps last_modified). Defaults to False so bulk loaders (load_data_config, CLI argument processing, file/env initial loads) stay quiet — those paths record provenance for values that aren’t really changing. User-driven programmatic mutations (register_url_alias and friends) should pass notify=True.

update_items( option: Option[dict], mapping: dict[Any, Any], *, source: ConfigSource | None = None, source_info: str = 'programmatic', notify: bool = True, ) → None¶

Add or replace one or more sub-keys on a dict-compound option.

Like Python’s dict.update: each key in mapping becomes one

Class:: ProvenancedItem at the given tier. Existing entries at (source, source_info) for the same sub-keys are replaced; sub-keys not in mapping are untouched, regardless of tier.

Named update_items (not update) to avoid colliding with MutableObject.update(init_parameters), which has different semantics inherited from the Object framework.

Parameters:

option – A dict-compound option.
mapping – {sub_key: value} pairs to write.
source – Tier to write at. Defaults to ConfigSource.API.
source_info – Source context. Defaults to "programmatic".
notify – Fire the internal _on_changed hook (bumping last_modified) if the post-transform effective value changed. Default True.

Raises:

ValueError – If option is not a dict-compound.

class ExtensionsConfig( parent: Any, )¶

Bases: tlcconfig.group.ConfigGroup

Extensions configuration group (config.extensions.*).

Lists of external classes (URL adapters, sample types, exporters) loaded at service startup.

exporters: list = []¶

Custom exporters to load on startup.

A list of dictionaries specifying custom exporter classes to load. Each dictionary should contain:

module: The fully qualified module name
class: The exporter class name
kwargs (optional): Constructor arguments
force (optional): If true, override an existing format
Config key: extensions.exporters
Python attribute: config.extensions.exporters

sample_types: list = []¶

Custom sample types to load on startup.

A list of dictionaries specifying custom sample type classes to load. Each dictionary should contain:

module: The fully qualified module name
class: The sample type class name
name (optional): Registration name (defaults to the class name)
force (optional): If true, override an existing sample type with the same name
Config key: extensions.sample-types
Python attribute: config.extensions.sample_types

url_adapters: list = []¶

Custom URL adapters to load on startup.

A list of dictionaries specifying custom URL adapter classes to load. Each dictionary should contain:

module: The fully qualified module name
class: The adapter class name
kwargs (optional): Constructor arguments

Example:

extensions: url-adapters: - module: my.custom.adapters class: MyAdapter kwargs: some_arg: value

Config key: extensions.url-adapters
Python attribute: config.extensions.url_adapters

class IndexingConfig( parent: Any, )¶

Bases: tlcconfig.group.ConfigGroup

Indexing configuration group (config.indexing.*).

debounce_backoff_max_level: int = 0¶

Maximum backoff level for timestamp debouncing.

Controls the upper limit of the exponential backoff for timestamp writes. Zero means no backoff for debouncing.

Config key: indexing.debounce-backoff-max-level
Environment variable: TLC_INDEXING_DEBOUNCE_BACKOFF_MAX_LEVEL
Python attribute: config.indexing.debounce_backoff_max_level

debounce_backoff_multiplier: float = 1.2¶

Multiplier for timestamp debounce backoff.

Controls how aggressively the debounce interval increases when multiple writes occur within the current interval.

Config key: indexing.debounce-backoff-multiplier
Environment variable: TLC_INDEXING_DEBOUNCE_BACKOFF_MULTIPLIER
Python attribute: config.indexing.debounce_backoff_multiplier

debounce_backoff_threshold: int = 2¶

Threshold for timestamp debouncing.

Controls the threshold for the exponential backoff for timestamp writes. The threshold is the number of writes that must occur within the debounce interval before the backoff is applied.

Config key: indexing.debounce-backoff-threshold
Environment variable: TLC_INDEXING_DEBOUNCE_BACKOFF_THRESHOLD
Python attribute: config.indexing.debounce_backoff_threshold

debounce_interval: float = 2.0¶

Initial debounce interval for timestamp-file writing.

Controls the initial delay (in seconds) before writing timestamp files. This is part of a dynamic debounce system that adapts to write frequency.

Config key: indexing.debounce-interval
Environment variable: TLC_INDEXING_DEBOUNCE_INTERVAL
Python attribute: config.indexing.debounce_interval

scan_interval: float = 10.0¶

The interval between indexer scans for new objects.

This option governs the overall responsiveness of the indexing system. The setting specifies how frequently (in seconds) the indexer will check for new or modified objects in the configured scan locations.

Config key: indexing.scan-interval
Environment variable: TLC_INDEXING_SCAN_INTERVAL
Python attribute: config.indexing.scan_interval

class LoggingConfig( parent: Any, )¶

Bases: tlcconfig.group.ConfigGroup

Logging configuration group (config.logging.*).

file: str = <computed at runtime>¶

Log file path for the 3LC logger.

The directory will be created if it does not exist. Environment variables and ~ are expanded. Must be a local filesystem path (remote URLs not allowed).

Config key: logging.file
Environment variable: TLC_LOG_FILE
CLI argument: --log-file
Python attribute: config.logging.file

level: str = WARNING¶

Log level for the 3LC logger.

Standard Python logging levels:

DEBUG: Detailed diagnostic information
INFO: Confirmation that things are working
WARNING: Unexpected events (default)
ERROR: Serious problems
CRITICAL: Program may not continue
Config key: logging.level
Environment variable: TLC_LOG_LEVEL
CLI argument: --log-level
Python attribute: config.logging.level
Choices: 'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'

tlc.configuration¶

Package Contents¶

Classes¶

API¶

`tlc.configuration`¶