`tlcconfig.options`#

Configuration options for 3LC

This module contains the configuration options for 3LC. These options can be set in a config file, as environment variables, or as command line arguments when launching the object service.

Module Contents#

Classes#

Class	Description
`ConfigSource`	The source of an option’s value.
`OPTION`	Base class for all 3LC options
`OBJECT_SERVICE_PORT`	Port for the server.
`OBJECT_SERVICE_HOST`	Host for the server.
`OBJECT_SERVICE_LICENSE`	Specify license or license file
`OBJECT_SERVICE_CACHE_IN_MEMORY_SIZE`	Specify the amount of memory to use for caching, in bytes.
`OBJECT_SERVICE_CACHE_TIME_OUT`	Specify the cache item time out, in seconds.
`PROJECT_ROOT_URL_OPTION`	Location for reading and writing 3LC data.
`PROJECT_SCAN_URLS_OPTION`	Locations to scan for 3LC objects (runs and tables) following a standard 3LC project layout.
`EXTRA_TABLE_SCAN_URLS_OPTION`	Extra (non-recursive) locations to scan for 3LC Table objects.
`EXTRA_RUN_SCAN_URLS_OPTION`	Extra (non-recursive) locations to scan for 3LC Run objects.
`LOGFILE`	Log file for the 3LC logger.
`LOGLEVEL`	Log level for the 3LC logger.
`DISPLAY_PROGRESS`	Whether to display progress bars or not.
`TELEMETRY`	Whether to send Telemetry or not. This option is deprecated and not in use.
`ALIASES`	List of registered aliases

Data#

Data	Description
`get_default_root_dir`	Alias for :func:`tlcconfig.default_locations.get_default_root_dir`
`get_default_log_dir`	Alias for :func:`tlcconfig.default_locations.get_default_log_dir`
`get_default_config_dir`	Alias for :func:`tlcconfig.default_locations.get_default_config_dir`

API#

class tlcconfig.options.ConfigSource#

Bases: enum.Enum

The source of an option’s value.

This enum is used during option resolution to track and store from which source the option was set.

DEFAULT = 0#: The option was set with the default value.

MIXED = 1#: The option was set from multiple sources. Indicates a compound value.

SECONDARY_CONFIG_FILE = 2#: The option was loaded from a configuration file with lower precedence. Typically used for config discovered with data.

DATA_CONFIG_FILE = 2#: Alias for SECONDARY_CONFIG_FILE. Deprecated.

CONFIG_FILE = 3#: The option was loaded from a normal configuration file.

ENVIRONMENT = 4#: The option was loaded from an environment variable.

COMMAND_LINE = 5#: The option was set on the command line.

API = 6#: The option was set using the API.

tlcconfig.options.get_default_root_dir = None#: Alias for :func:tlcconfig.default_locations.get_default_root_dir

tlcconfig.options.get_default_log_dir = None#: Alias for :func:tlcconfig.default_locations.get_default_log_dir

tlcconfig.options.get_default_config_dir = None#: Alias for :func:tlcconfig.default_locations.get_default_config_dir

class tlcconfig.options.OPTION#

Base class for all 3LC options

default: ClassVar[Any] = None#: The default value for this option if it is not specified anywhere.

envvar: str | re.Pattern = <Multiline-String>#: The environment variable to use for this option. If the value is a regex pattern, it is implied that the option is compound.

key: str = <Multiline-String>#

The key used for this option in the config file.

This is the fully qualified key, e.g. “object-server.port” corresponds to the following yaml:

object-server:
    port: 5015

argument: str = <Multiline-String>#: The command line argument for this option

data_type: Any = None#: The type of the option’s value

required: bool = False#: Whether this option is required or not

validate_func: Callable[[Any], bool] | None = None#: A function to validate the value of this option

transform_func: Callable[[Any, tlcconfig.options.ConfigSource, str | None], Any] | None = None#

A function to transform the value of this option before reading it.

The use case for this is e.g. to expand environment variables or do shell expansion in the value. If the option is compound, the function will be applied to compound input and should return a compound value.

format_func: Callable[[Any], object] = None#

A function to format the value for this option in serialization.

This is useful for options that have complex values, such as lists or dictionaries. Default is identity.

compound_value_type: type[list] | type[dict] | None = None#: Specifies the data structure type (dict or list) of a compound OPTION

hide_default_in_config = False#: Whether to hide the default value in the config file or not.

classmethod class_from_key(key: str) → type[tlcconfig.options.OPTION]#

Get the option class from a key.

Parameters:: key – The key to look up, which should corresponds to the OPTION.key attribute of an option class.
Returns:: The option class corresponding to the key.
Raises:: RuntimeError – If the key does not corresponds to a valid option class.

classmethod is_hidden() → bool#

Whether this option should be hidden or not.

Returns:: True if this option should be hidden, False otherwise.

class tlcconfig.options.OBJECT_SERVICE_PORT#

Bases: tlcconfig.options.OPTION

Port for the server.

default = 5015#

key = service.port#

data_type = None#

transform_func = None#

envvar = TLC_SERVICE_PORT#

argument = --port#

class tlcconfig.options.OBJECT_SERVICE_HOST#

Bases: tlcconfig.options.OPTION

Host for the server.

key = service.host#

envvar = TLC_SERVICE_HOST#

argument = --host#

default = 127.0.0.1#

data_type = None#

class tlcconfig.options.OBJECT_SERVICE_LICENSE#

Bases: tlcconfig.options.OPTION

Specify license or license file

The option can either be the license key or point to a local file containing the license key.

default = <Multiline-String>#

key = service.license#

envvar = TLC_LICENSE#

argument = --license#

data_type = None#

transform_func = None#

class tlcconfig.options.OBJECT_SERVICE_CACHE_IN_MEMORY_SIZE#

Bases: tlcconfig.options.OPTION

Specify the amount of memory to use for caching, in bytes.

Setting the value to 0 will disable in-memory caching.

Default: 1073741824 (1 GB)

default = 1073741824#

key = service.cache_in_memory_size#

argument = --cache-size#

data_type = None#

class tlcconfig.options.OBJECT_SERVICE_CACHE_TIME_OUT#

Bases: tlcconfig.options.OPTION

Specify the cache item time out, in seconds.

Setting the value to 0 will disable cache eviction based on time.

Default: 3600 (1 hour)

default = 3600#

key = service.cache_time_out#

argument = --cache-time-out#

data_type = None#

class tlcconfig.options.PROJECT_ROOT_URL_OPTION#

Bases: tlcconfig.options.OPTION

Location for reading and writing 3LC data.

This option is mandatory and must point to a location (e.g. directory on disk or object store bucket) with write access. The location will be created if it does not exist.

If the option value contains an environment variable, it will be expanded.

default = _consider_expandvars(...)#: As returned by :func:tlcconfig.default_locations.get_default_root_dir

key = indexing.project-root-url#

envvar = TLC_CONFIG_PROJECT_ROOT_URL#

argument = --project-root-url#

required = True#

data_type = None#

transform_func = None#

local_ensure_file_is_writable() → bool#

validate_func = None#

class tlcconfig.options.PROJECT_SCAN_URLS_OPTION#

Bases: tlcconfig.options.OPTION

Locations to scan for 3LC objects (runs and tables) following a standard 3LC project layout.

The option value should be a list of URLs to scan for 3LC objects. Items in the list can be represented either by a string value (e.g., “s3://bucket”) or as a dictionary with the required field ‘url’ and optional extra fields (e.g., {“url”: “s3://bucket”, “static”: True}).

Example values:

“C:\Users\user\Documents\3LC\projects”
“s3://my-bucket/3LC/projects”
{“url”: “s3://my-bucket/3LC/read-only-projects”, “static”: True}

The only public optional field is ‘static’, which, when set to ‘True’, prevents the indexer from re-scanning the location.

Each URL in this list is assumed to contain sub-projects. Default (sub-)directories will be created if they do not exist.

default: ClassVar = []#

key = indexing.project-scan-urls#

envvar = TLC_CONFIG_PROJECT_SCAN_URLS#

data_type = None#

transform_func = None#

compound_value_type = None#

format_func = None#

class tlcconfig.options.EXTRA_TABLE_SCAN_URLS_OPTION#

Bases: tlcconfig.options.OPTION

Extra (non-recursive) locations to scan for 3LC Table objects.

This option allows searching individual folders/locations for 3LC Table objects outside the standard hierarchy of a 3LC project structure. For example:

“C:\Users\user\Documents\3LC\tables”
“s3://my-bucket/3LC/tables”.

Note: these locations will not be scanned recursively. See PROJECT_SCAN_URLS_OPTION for details on URL options.

Indexed Tables from these extra locations can be used interchangeably with other Tables discovered by the system.

default: ClassVar = []#

key = indexing.extra-table-scan-urls#

envvar = TLC_CONFIG_EXTRA_TABLE_SCAN_URLS#

data_type = None#

transform_func = None#

compound_value_type = None#

format_func = None#

hide_default_in_config = True#

class tlcconfig.options.EXTRA_RUN_SCAN_URLS_OPTION#

Bases: tlcconfig.options.OPTION

Extra (non-recursive) locations to scan for 3LC Run objects.

This option allows searching individual folders/locations for 3LC Run objects outside the standard hierarchy of a 3LC project structure. For example:

“C:\Users\user\Documents\3LC\runs”
“s3://my-bucket/3LC/runs”.

Note: these locations will not be scanned recursively. See PROJECT_SCAN_URLS_OPTION for details on URL options.

Indexed Runs from these extra locations can be used interchangeably with other Runs discovered by the system.

default: ClassVar = []#

key = indexing.extra-run-scan-urls#

envvar = TLC_CONFIG_EXTRA_RUN_SCAN_URLS#

data_type = None#

transform_func = None#

compound_value_type = None#

format_func = None#

hide_default_in_config = True#

class tlcconfig.options.LOGFILE#

Bases: tlcconfig.options.OPTION

Log file for the 3LC logger.

The directory will be created if it does not exist.

If the option value contains an environment variable, it will be expanded.

default = join(...)#: As returned by :func:tlcconfig.default_locations.get_default_log_dir / 3LC.log

key = logging.logfile#

envvar = TLC_LOGFILE#

argument = --logfile#

data_type = None#

transform_func = None#

local_ensure_file_is_writable() → bool#

validate_func = None#

class tlcconfig.options.LOGLEVEL#

Bases: tlcconfig.options.OPTION

Log level for the 3LC logger.

The tlc Python package adheres to the standard Python logging levels:

DEBUG: Detailed information, typically of interest only when diagnosing problems.
INFO: Confirmation that things are working as expected.
WARNING: An indication that something unexpected happened, or indicative of some problem in the near future (e.g. “disk space low”). The software is still working as expected.
ERROR: Due to a more serious problem, the software has not been able to perform some function.
CRITICAL: A serious error, indicating that the program itself may be unable to continue running.

default = WARNING#

key = logging.loglevel#

envvar = TLC_LOGLEVEL#

argument = --loglevel#

data_type = None#

transform_func = None#

validate_func = None#

class tlcconfig.options.DISPLAY_PROGRESS#

Bases: tlcconfig.options.OPTION

Whether to display progress bars or not.

The option can be either 0 or 1, where 0 means no progress bars and 1 means progress bars.

default = 1#

data_type = None#

transform_func = None#

key = tlc.display-progress#

envvar = TLC_DISPLAY_PROGRESS#

argument = --display-progress#

validate_func = None#

class tlcconfig.options.TELEMETRY#

Bases: tlcconfig.options.OPTION

Whether to send Telemetry or not. This option is deprecated and not in use.

default = _get_default_telemetry(...)#

data_type = None#

key = tlc.telemetry#

envvar = TLC_TELEMETRY#

argument = --telemetry#

transform_func = None#

hide_default_in_config = True#

class tlcconfig.options.ALIASES#

Bases: tlcconfig.options.OPTION

List of registered aliases

The option value should be a dictionary where each key is an alias name and its value is the alias target (i.e. a path).

Anchored alias syntax

When aliases are defined in configuration files, their values can reference paths relative to the file’s location. Use the $. prefix (and subsequent dots) to move up directories relative to the configuration file:

$./data => the data directory next to the configuration file.
$../data => the data directory one level above the configuration file.
$.../data => the data directory two levels above the configuration file.

Default aliases

Default aliases come from special configuration files named default_aliases.3lc.yaml in indexed locations, providing baseline values without local configuration. They have lower precedence than locally defined aliases.

The system searches for default aliases inside each indexed project, hence the name project aliases is often used:

<PROJECT_SCAN_URL>/<project_xyz>/default_aliases.3lc.yaml # project aliases

For completeness, the system also searches for default aliases in the following scan locations:

<PROJECT_SCAN_URL>/default_aliases.3lc.yaml
<EXTRA_TABLE_SCAN_URL>/default_aliases.3lc.yaml
<EXTRA_RUN_SCAN_URL>/default_aliases.3lc.yaml

Examples

aliases:
  PROJECT_XYZ_REMOTE_ALIAS: s3://my-bucket/3LC/projects/project-xyz
  PROJECT_XYZ_LOCAL_ALIAS: /home/user/projects/project-xyz

Special anchored syntax examples:

aliases:
  PROJECT_XYZ_DATA_ALIAS: $./data # expands to the data directory relative to the configuration file itself
  PROJECT_XYZ_PARENT_DATA_ALIAS: $../data # relative to parent dir
  PROJECT_XYZ_PARENT_PARENT_DATA_ALIAS: $.../data # relative to parent's parent dir

default: ClassVar = None#

data_type = None#

key = aliases#

envvar = compile(...)#

transform_func = None#

compound_value_type = None#

format_func = None#

tlcconfig.options#

Module Contents#

Classes#

Data#

API#

`tlcconfig.options`#