`tlcconfig.options`¶

Configuration option definitions.

This module defines pure metadata for configuration options. Options contain no logic, no transformation, and no validation — only metadata that describes the option.

An Option is the single source of truth for:

Option key names
Data types and defaults
Environment variable mappings
CLI argument mappings
Documentation

.. rubric:: Example

from tlcconfig.options import OptionRegistry, SERVICE_PORT

# Get option by key
option = OptionRegistry.get("service.port")
print(option.description)

# Iterate all options
for key, option in OptionRegistry.all().items():
    print(f"{key}: {option.default}")

Module Contents¶

Classes¶

Class	Description
`Option`	Immutable definition of a configuration option.
`OptionCategory`	Categories for organizing configuration options.
`OptionRegistry`	Central registry of all configuration options.

Data¶

Data	Description
`ALIASES`	URL alias definitions for path shortcuts.
`DISPLAY_PROGRESS`	Whether to display progress bars.
`EXPORTERS`	Custom exporters to load on startup.
`INDEXER_BACKOFF_MULTIPLIER`	Scales transient-class retry cadences in the indexer’s failed-URL skip store.
`INDEXING_CLOUD_IO_CONCURRENCY`	Advanced: max concurrent cloud (S3/GCS/ABFS) I/O operations during indexing scans.
`INDEXING_DEBOUNCE_BACKOFF_MAX_LEVEL`	Maximum backoff level for timestamp debouncing.
`INDEXING_DEBOUNCE_BACKOFF_MULTIPLIER`	Multiplier for timestamp debounce backoff.
`INDEXING_DEBOUNCE_BACKOFF_THRESHOLD`	Threshold for timestamp debouncing.
`INDEXING_DEBOUNCE_INTERVAL`	Initial debounce interval for timestamp-file writing.
`INDEXING_LOCAL_IO_CONCURRENCY`	Advanced: max concurrent local-filesystem I/O operations during indexing scans.
`INDEXING_SCAN_INTERVAL`	The interval between indexer scans for new objects.
`LOG_FILE`	Log file path for the 3LC logger.
`LOG_LEVEL`	Log level for the 3LC logger.
`OBJECT_SERVICE_DISABLE_EXTERNAL_MEDIA_URLS`	Disable external self-authenticated media URLs.
`ROOT_URL`	Location for reading and writing 3LC project data.
`SAMPLE_TYPES`	Custom sample types to load on startup.
`SCAN_URLS`	Locations to scan for 3LC objects.
`SERVICE_CACHE_SIZE`	Maximum size in bytes for the in-memory object cache.
`SERVICE_CACHE_TIMEOUT`	Cache item time-to-live in seconds.
`SERVICE_HOST`	Host address for the Object Service server.
`SERVICE_PORT`	Port for the Object Service server.
`SERVICE_WATCH_FOLDERS`	Enable the watch service to monitor folders for external changes.
`T`
`URL_ADAPTERS`	Custom URL adapters to load on startup.

API¶

ALIASES: Option[dict] = {}¶

URL alias definitions for path shortcuts.

Dictionary mapping alias names to target paths/URLs. Aliases are used with angle brackets in URLs: <ALIAS_NAME>/path/to/file.

Individual aliases can also be set through environment variables of the form TLC_ALIAS_=, e.g. TLC_ALIAS_DATASETS=/data/datasets.

Example: aliases: DATASETS: /data/datasets MODELS: s3://bucket/models

Config key: aliases
Environment variable: TLC_ALIAS_([A-Z][A-Z0-9_]*)
Python attribute: config.aliases

DISPLAY_PROGRESS: Option[bool] = True¶

Whether to display progress bars.

Config key: display-progress
Environment variable: TLC_DISPLAY_PROGRESS
CLI argument: --display-progress
Python attribute: config.display_progress

EXPORTERS: Option[list] = []¶

Custom exporters to load on startup.

A list of dictionaries specifying custom exporter classes to load. Each dictionary should contain:

module: The fully qualified module name
class: The exporter class name
kwargs (optional): Constructor arguments
force (optional): If true, override an existing format
Config key: extensions.exporters
Python attribute: config.extensions.exporters

INDEXER_BACKOFF_MULTIPLIER: Option[float] = 1.0¶

Scales transient-class retry cadences in the indexer’s failed-URL skip store.

The Rust indexer skips URLs that fail in characteristic ways (network blips, timeouts, list-time NotFound races) and retries them after a per-reason backoff. This multiplier scales the transient cadences (NetworkError, Timeout, RaceNotFound) — set above 1.0 on slow networks where 10s/30s/60s is too tight; set below 1.0 for tight dev loops where you want faster retries.

Terminal cadences (PermissionDenied 1h, MalformedContent 6h, UnregisteredType never) are NOT affected by this multiplier: they’re operator-action-bound, so tuning the retry rate doesn’t change recovery time.

Config key: indexing.backoff-multiplier
Environment variable: TLC_INDEXER_BACKOFF_MULTIPLIER
Python attribute: config.indexing.backoff_multiplier

INDEXING_CLOUD_IO_CONCURRENCY: Option[int] = 16¶

Advanced: max concurrent cloud (S3/GCS/ABFS) I/O operations during indexing scans.

Dev-facing tuning knob. Caps total in-flight requests across all cloud scan sources (the thundering-herd / rate-limit ceiling). The authoritative value lives natively (tlc-storage); this option seeds it once at engine init. Higher hides more cloud latency but risks saturating client connections; the sweet spot for metadata-heavy scans is ~32. Leave unset unless benchmarking a specific store.

Config key: indexing.cloud-io-concurrency
Environment variable: TLC_CLOUD_IO_CONCURRENCY
Python attribute: config.indexing.cloud_io_concurrency

INDEXING_DEBOUNCE_BACKOFF_MAX_LEVEL: Option[int] = 0¶

Maximum backoff level for timestamp debouncing.

This option is accepted for configuration compatibility but is not consulted by the indexer.

Config key: indexing.debounce-backoff-max-level
Environment variable: TLC_INDEXING_DEBOUNCE_BACKOFF_MAX_LEVEL
Python attribute: config.indexing.debounce_backoff_max_level

INDEXING_DEBOUNCE_BACKOFF_MULTIPLIER: Option[float] = 1.2¶

Multiplier for timestamp debounce backoff.

This option is accepted for configuration compatibility but is not consulted by the indexer.

Config key: indexing.debounce-backoff-multiplier
Environment variable: TLC_INDEXING_DEBOUNCE_BACKOFF_MULTIPLIER
Python attribute: config.indexing.debounce_backoff_multiplier

INDEXING_DEBOUNCE_BACKOFF_THRESHOLD: Option[int] = 2¶

Threshold for timestamp debouncing.

This option is accepted for configuration compatibility but is not consulted by the indexer.

Config key: indexing.debounce-backoff-threshold
Environment variable: TLC_INDEXING_DEBOUNCE_BACKOFF_THRESHOLD
Python attribute: config.indexing.debounce_backoff_threshold

INDEXING_DEBOUNCE_INTERVAL: Option[float] = 2.0¶

Initial debounce interval for timestamp-file writing.

This option is accepted for configuration compatibility but is not consulted by the indexer.

Config key: indexing.debounce-interval
Environment variable: TLC_INDEXING_DEBOUNCE_INTERVAL
Python attribute: config.indexing.debounce_interval

INDEXING_LOCAL_IO_CONCURRENCY: Option[int] = 4¶

Advanced: max concurrent local-filesystem I/O operations during indexing scans.

Dev-facing tuning knob (see indexing.cloud-io-concurrency). Local reads are throughput/syscall-bound rather than latency-bound, so keep this low — high concurrency helps SSDs a little but thrashes spinning disks / network mounts.

Config key: indexing.local-io-concurrency
Environment variable: TLC_LOCAL_IO_CONCURRENCY
Python attribute: config.indexing.local_io_concurrency

INDEXING_SCAN_INTERVAL: Option[float] = 10.0¶

The interval between indexer scans for new objects.

This option governs the overall responsiveness of the indexing system. The setting specifies how frequently (in seconds) the indexer will check for new or modified objects in the configured scan locations.

Config key: indexing.scan-interval
Environment variable: TLC_INDEXING_SCAN_INTERVAL
Python attribute: config.indexing.scan_interval

LOG_FILE: Option[str] = <computed at runtime>¶

Log file path for the 3LC logger.

The directory will be created if it does not exist. Environment variables and ~ are expanded. Must be a local filesystem path (remote URLs not allowed).

Config key: logging.file
Environment variable: TLC_LOG_FILE
CLI argument: --log-file
Python attribute: config.logging.file

LOG_LEVEL: Option[str] = WARNING¶

Log level for the 3LC logger.

Standard Python logging levels:

DEBUG: Detailed diagnostic information
INFO: Confirmation that things are working
WARNING: Unexpected events (default)
ERROR: Serious problems
CRITICAL: Program may not continue
Config key: logging.level
Environment variable: TLC_LOG_LEVEL
CLI argument: --log-level
Python attribute: config.logging.level
Choices: 'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'

OBJECT_SERVICE_DISABLE_EXTERNAL_MEDIA_URLS: Option[bool] = False¶

Disable external self-authenticated media URLs.

By default the service may resolve media URLs (images, videos, audio) used by the dashboard into external self-authenticated URLs (e.g. S3 presigned, GCS signed, Azure SAS) so the browser can fetch directly from cloud storage. When this option is set, all such media URLs are instead served through this service via a token-authenticated proxy endpoint.

This is useful when the machine running the service can access the cloud storage but the machine(s) used to run the dashboard webapp cannot or should not do so.

Config key: service.disable-external-media-urls
Environment variable: TLC_SERVICE_DISABLE_EXTERNAL_MEDIA_URLS
CLI argument: --disable-external-media-urls
Python attribute: config.service.disable_external_media_urls

class Option¶

Bases: typing.Generic[tlcconfig.options.T]

Immutable definition of a configuration option.

This class contains ONLY metadata — no logic, no transformation, no validation. This is the single source of truth for option documentation.

Variables:

key – Fully qualified key path in config file (e.g., “service.port”).
category – Category for grouping and organization.
data_type – Python type of the option value.
default – Default value when not configured. None means no default.
default_factory – Callable that returns a computed default value. Used when default is None but a runtime-computed default is needed.
required – Whether the option must be set.
envvar – Environment variable name, or regex pattern for compound options.
envvar_aliases – Deprecated env var names still honored as input (with a warning). Use to keep an old env var working after a rename. String aliases only — patterns are not supported.
cli_argument – CLI argument flag (e.g., “–port”).
cli_metavar – Placeholder shown in CLI help (e.g., “PORT”).
cli_command – CLI command this option belongs to (e.g., “service”). If set, the CLI argument is only exposed under that command.
compound_type – For compound options, the container type (list or dict).
projection – Storage-projection tag naming the backend that holds this option’s values (e.g. "memory" or "native-aliases"). Defaults to the in-memory projection (MEMORY_PROJECTION).
description – Full documentation string. First line used as brief help.
cli_visible – When True (the default), the option is surfaced through CLI tooling — it appears in 3lc --help (the dynamic-options generator picks it up) and in the 3lc config show default output. Set False for runtime / dev flags set by the framework itself rather than the operator.
serializable – When True (the default), the option’s value is included in serialized representations of Configuration: written to YAML by ConfigStore.write_to_yaml_file / to_default_yaml, published on the /configuration Object Service endpoint, and rendered on the API reference page. Set False for credentials and internal runtime state that should never round-trip through a config file or be exposed to clients. Independent of cli_visible — credentials are typically cli_visible=True (so --license / --api-key work) but serializable=False. Convention: the matching attribute on Configuration / ConfigGroup carries a leading underscore for non-serializable options, as a redundant in-source readability cue.
hide_default_in_config – If True, default is not written to config files.
choices – Valid values for enum-like options.
consumer_only – When True, this option is ignored (with a one-shot warning) if it appears in a data-bundled producer config — one loaded at the DATA_CONFIG_FILE tier. Defaults to False: a data-bundled config may set the option like any other tier. Set True for consumer-only options — notably scan-urls — so a producer shipping data can describe how its own data is indexed but can never expand what the consumer scans (the non-transitive rule).
object_property_name – Name to use in Object schema. Defaults to key if not set.

property brief: str¶: First line of description, used for CLI help.

category: OptionCategory = None¶

choices: list[Any] | None = None¶

cli_argument: str = <Multiline-String>¶

cli_command: str | None = None¶

cli_metavar: str | None = None¶

cli_visible: bool = True¶

compound_type: type | None = None¶

consumer_only: bool = False¶

data_type: type[tlcconfig.options.T] = None¶

default: Any = None¶

default_factory: Callable[[], tlcconfig.options.T] | None = field(...)¶

description: str = <Multiline-String>¶

envvar: str | Pattern[str] = <Multiline-String>¶

envvar_aliases: tuple[str, ...] = ()¶

hide_default_in_config: bool = False¶

key: str = None¶

object_property_name: str | None = None¶

projection: str = None¶

required: bool = False¶

serializable: bool = True¶

class OptionCategory¶

Bases: enum.Enum

Categories for organizing configuration options.

Categories are used for:

Grouping options in CLI help output
Organizing options in config file output
Filtering options by domain (e.g., service-only options)

External subsystems (e.g., URL adapters) can register their own categories and options to own their config namespace. Use OptionRegistry.register() with a custom category to add options from outside the core config package.

EXTENSIONS = extensions¶: Options for loading external extension classes (URL adapters, sample types, exporters).

INDEXING = indexing¶: Options related to project and data indexing.

LOGGING = logging¶: Options for logging configuration.

SERVICE = service¶: Options for the Object Service server.

TLC = tlc¶: General 3LC options.

class OptionRegistry¶

Central registry of all configuration options.

This registry enables CLI introspection and iteration over options without coupling to business logic.

.. rubric:: Example

# Register a new option
MY_OPTION = OptionRegistry.register(Option(
    key="my.option",
    category=OptionCategory.TLC,
    data_type=str,
    default="value",
    description="My custom option.",
))

# Get option by key
option = OptionRegistry.get("my.option")

# Iterate all options
for key, option in OptionRegistry.all().items():
    print(key)

classmethod all() → dict[str, Option[Any]]¶

Get all registered options.

Returns:: Dictionary mapping keys to options.

classmethod by_category( category: OptionCategory, ) → dict[str, Option[Any]]¶

Get options filtered by category.

Parameters:: category – The category to filter by.
Returns:: Dictionary of options in the given category.

classmethod by_command( command: str, ) → dict[str, Option[Any]]¶

Get options that belong to a specific CLI command.

Parameters:: command – The CLI command name (e.g., “service”).
Returns:: Dictionary of options for that command.

classmethod clear() → None¶: Clear all registered options. Primarily for testing.

classmethod get( key: str, ) → Option[Any]¶

Get option by key.

Parameters:: key – The option key (e.g., “service.port”).
Returns:: The option.
Raises:: KeyError – If no option exists for the key.

classmethod register( option: Option[tlcconfig.options.T], ) → Option[tlcconfig.options.T]¶

Parameters:: option – The option to register.
Returns:: The registered option (for assignment to module-level variable).
Raises:: ValueError – If an option with the same key is already registered.

classmethod with_cli_argument( include_cli_invisible: bool = False, ) → dict[str, Option[Any]]¶

Get options that have CLI arguments.

Parameters:: include_cli_invisible – When True, include options marked cli_visible=False (otherwise they are filtered out).
Returns:: Dictionary of options that can be set via CLI.

ROOT_URL: Option[str] = <computed at runtime>¶

Location for reading and writing 3LC project data.

This option is mandatory and must point to a location with write access. Supports local paths and remote URLs (s3://, gs://, az://). Environment variables and ~ are expanded. The location will be created if needed.

Config key: project-root-url
Environment variable: TLC_PROJECT_ROOT_URL
CLI argument: --project-root-url
Python attribute: config.project_root_url
Required: yes

SAMPLE_TYPES: Option[list] = []¶

Custom sample types to load on startup.

A list of dictionaries specifying custom sample type classes to load. Each dictionary should contain:

module: The fully qualified module name
class: The sample type class name
name (optional): Registration name (defaults to the class name)
force (optional): If true, override an existing sample type with the same name
Config key: extensions.sample-types
Python attribute: config.extensions.sample_types

SCAN_URLS: Option[list] = []¶

Locations to scan for 3LC objects.

Each entry can be a plain URL string (defaults to layout: project) or a dict with explicit attributes:

scan-urls: # Plain string — scanned as a project tree - s3://bucket/projects - /local/projects

# Dict with explicit layout
- url: /local/loose-objects
  layout: flat

# Full control
- url: /data/tables-only
  layout: flat
  object_type: table

Supported layout values: “project” (recursive project scan), “flat” (single directory). Supported object_type values: “table”, “run” (omit to scan for all types).

The TLC_SCAN_URLS environment variable accepts a comma-separated list of URL strings.

Mutations to scan-urls are safe from any thread and take effect on the indexer’s next synchronous URL query, or immediately when an explicit reconcile is requested.

Config key: scan-urls
Environment variable: TLC_SCAN_URLS
Python attribute: config.scan_urls

SERVICE_CACHE_SIZE: Option[int] = 1073741824¶

Maximum size in bytes for the in-memory object cache.

Setting the value to 0 will disable in-memory caching. Default: 1073741824 (1 GB)

Config key: service.cache.size
CLI argument: --cache-size
Python attribute: config.service.cache.size

SERVICE_CACHE_TIMEOUT: Option[int] = 3600¶

Cache item time-to-live in seconds.

Setting the value to 0 will disable cache eviction based on time. Default: 3600 (1 hour)

Config key: service.cache.timeout
CLI argument: --cache-timeout
Python attribute: config.service.cache.timeout

SERVICE_HOST: Option[str] = 127.0.0.1¶

Host address for the Object Service server.

Config key: service.host
Environment variable: TLC_SERVICE_HOST
CLI argument: --host
Python attribute: config.service.host

SERVICE_PORT: Option[int] = 5015¶

Port for the Object Service server.

Config key: service.port
Environment variable: TLC_SERVICE_PORT
CLI argument: --port
Python attribute: config.service.port

SERVICE_WATCH_FOLDERS: Option[bool] = False¶

Enable the watch service to monitor folders for external changes.

When enabled, the watch service monitors configured local folders and automatically updates timestamp files when changes are detected. Useful for monitoring changes made by processes outside of 3LC.

WARNING: Avoid monitoring mounted folders (FUSE mounts, cloud storage mounts) as they may cause performance issues or unexpected behavior.

Config key: service.watch-folders
CLI argument: --watch-local-folders
Python attribute: config.service.watch_folders

T = TypeVar(...)¶

URL_ADAPTERS: Option[list] = []¶

Custom URL adapters to load on startup.

A list of dictionaries specifying custom URL adapter classes to load. Each dictionary should contain:

module: The fully qualified module name
class: The adapter class name
kwargs (optional): Constructor arguments

Example:

extensions: url-adapters: - module: my.custom.adapters class: MyAdapter kwargs: some_arg: value

Config key: extensions.url-adapters
Python attribute: config.extensions.url_adapters

tlcconfig.options¶

Module Contents¶

Classes¶

Data¶

API¶

`tlcconfig.options`¶