`tlcconfig.options`#

Configuration options for 3LC

This module contains the configuration options for 3LC. These options can be set in a config file, as environment variables, or as command line arguments when launching the object service.

Module Contents#

Classes#

Class	Description
`ConfigSource`	The source of an option’s value.
`OPTION`	Base class for all 3LC options
`OBJECT_SERVICE_PORT`	Port for the server.
`OBJECT_SERVICE_HOST`	Host for the server.
`OBJECT_SERVICE_LICENSE`	Specify license or license file
`OBJECT_SERVICE_CACHE_IN_MEMORY_SIZE`	Specify the amount of memory to use for caching, in bytes.
`OBJECT_SERVICE_CACHE_TIME_OUT`	Specify the cache item time out, in seconds.
`PROJECT_ROOT_URL_OPTION`	Location for reading and writing 3LC data.
`PROJECT_SCAN_URLS_OPTION`	Locations to scan for 3LC objects (runs and tables) following a standard 3LC project layout.
`EXTRA_TABLE_SCAN_URLS_OPTION`	Extra (non-recursive) locations to scan for 3LC Table objects.
`EXTRA_RUN_SCAN_URLS_OPTION`	Extra (non-recursive) locations to scan for 3LC Run objects.
`LOGFILE`	Log file for the 3LC logger.
`LOGLEVEL`	Log level for the 3LC logger.
`DISPLAY_PROGRESS`	Whether to display progress bars or not.
`TELEMETRY`	Whether to send Telemetry or not. This option is deprecated and not in use.
`ALIASES`	List of registered aliases

Data#

Data	Description
`get_default_root_dir`	Alias for :func:`tlcconfig.default_locations.get_default_root_dir`
`get_default_log_dir`	Alias for :func:`tlcconfig.default_locations.get_default_log_dir`
`get_default_config_dir`	Alias for :func:`tlcconfig.default_locations.get_default_config_dir`

API#

class tlcconfig.options.ConfigSource#

Bases: enum.Enum

The source of an option’s value.

This enum is used during option resolution to track and store from which source the option was set.

DEFAULT = 0#: The option was set with the default value.

CONFIG_FILE = 1#: The option was loaded from a configuration file.

ENVIRONMENT = 2#: The option was loaded from an environment variable.

COMMAND_LINE = 3#: The option was set on the command line.

API = 4#: The option was set using the API.

tlcconfig.options.get_default_root_dir = None#: Alias for :func:tlcconfig.default_locations.get_default_root_dir

tlcconfig.options.get_default_log_dir = None#: Alias for :func:tlcconfig.default_locations.get_default_log_dir

tlcconfig.options.get_default_config_dir = None#: Alias for :func:tlcconfig.default_locations.get_default_config_dir

class tlcconfig.options.OPTION#

Base class for all 3LC options

default: Any = None#: The default value for this option if it is not specified anywhere.

envvar: str = <Multiline-String>#: The environment variable to use for this option

key: str = <Multiline-String>#

The key used for this option in the config file.

This is the fully qualified key, e.g. “object-server.port” corresponds to the following yaml:

object-server:
    port: 5015

argument: str = <Multiline-String>#: The command line argument for this option

data_type: Any = None#: The type of the option’s value

required: bool = False#: Whether this option is required or not

validate_func: Callable[[Any], bool] | None = None#: A function to validate the value of this option

transform_func: Callable[[Any, tlcconfig.options.ConfigSource], Any] | None = None#

A function to transform the value of this option before reading it.

The use case for this is e.g. to expand environment variables or do shell expansion in the value.

apply_transform_func_per_element: bool = True#

Whether to apply the transform_func individually for each element of composite values like lists or dicts.

If set to False, the transform_func will be applied to the composite value as a whole. For dicts, the function will be applied to the values only.

format_func: Callable[[Any], str | list[str]] = None#

A function to format the value for this option in serialization.

This is useful for options that have complex values, such as lists or dictionaries. Default is identity.

multi_entry_format_func: Callable[[Any], list[tuple[str, str]]] | None = None#

classmethod class_from_key(key: str) → type[tlcconfig.options.OPTION]#

Get the option class from a key.

Parameters:: key – The key to look up, which should corresponds to the OPTION.key attribute of an option class.
Returns:: The option class corresponding to the key.
Raises:: RuntimeError – If the key does not corresponds to a valid option class.

classmethod is_hidden() → bool#

Whether this option should be hidden or not.

Returns:: True if this option should be hidden, False otherwise.

class tlcconfig.options.OBJECT_SERVICE_PORT#

Bases: tlcconfig.options.OPTION

Port for the server.

default = 5015#

key = service.port#

data_type = None#

transform_func = None#

envvar = TLC_SERVICE_PORT#

argument = --port#

class tlcconfig.options.OBJECT_SERVICE_HOST#

Bases: tlcconfig.options.OPTION

Host for the server.

key = service.host#

envvar = TLC_SERVICE_HOST#

argument = --host#

default = 127.0.0.1#

data_type = None#

class tlcconfig.options.OBJECT_SERVICE_LICENSE#

Bases: tlcconfig.options.OPTION

Specify license or license file

The option can either be the license key or point to a local file containing the license key.

default = <Multiline-String>#

key = service.license#

envvar = TLC_LICENSE#

argument = --license#

data_type = None#

transform_func = None#

class tlcconfig.options.OBJECT_SERVICE_CACHE_IN_MEMORY_SIZE#

Bases: tlcconfig.options.OPTION

Specify the amount of memory to use for caching, in bytes.

Setting the value to 0 will disable in-memory caching.

Default: 1073741824 (1 GB)

default = 1073741824#

key = service.cache_in_memory_size#

argument = --cache-size#

data_type = None#

class tlcconfig.options.OBJECT_SERVICE_CACHE_TIME_OUT#

Bases: tlcconfig.options.OPTION

Specify the cache item time out, in seconds.

Setting the value to 0 will disable cache eviction based on time.

Default: 3600 (1 hour)

default = 3600#

key = service.cache_time_out#

argument = --cache-time-out#

data_type = None#

class tlcconfig.options.PROJECT_ROOT_URL_OPTION#

Bases: tlcconfig.options.OPTION

Location for reading and writing 3LC data.

This option is mandatory and must point to a location (e.g. directory on disk or object store bucket) with write access. The location will be created if it does not exist.

If the option value contains an environment variable, it will be expanded.

default = join(...)#: As returned by :func:tlcconfig.default_locations.get_default_root_dir

key = indexing.project-root-url#

envvar = TLC_CONFIG_PROJECT_ROOT_URL#

argument = --project-root-url#

required = True#

data_type = None#

transform_func = None#

local_ensure_file_is_writable() → bool#

validate_func = None#

class tlcconfig.options.PROJECT_SCAN_URLS_OPTION#

Bases: tlcconfig.options.OPTION

Locations to scan for 3LC objects (runs and tables) following a standard 3LC project layout.

The option value should be a list of URLs to scan for 3LC objects. Items in the list can be represented either by a string value (e.g., “s3://bucket”) or as a dictionary with the required field ‘url’ and optional extra fields (e.g., {“url”: “s3://bucket”, “static”: True}).

Example values:

“C:\Users\user\Documents\3LC\projects”
“s3://my-bucket/3LC/projects”
{“url”: “s3://my-bucket/3LC/read-only-projects”, “static”: True}

The only public optional field is ‘static’, which, when set to ‘True’, prevents the indexer from re-scanning the location.

Each URL in this list is assumed to contain sub-projects. Default (sub-)directories will be created if they do not exist.

default = []#

key = indexing.project-scan-urls#

envvar = TLC_CONFIG_PROJECT_SCAN_URLS#

data_type = None#

transform_func = None#

apply_transform_func_per_element = False#

format_func = None#

multi_entry_format_func = None#

class tlcconfig.options.EXTRA_TABLE_SCAN_URLS_OPTION#

Bases: tlcconfig.options.OPTION

Extra (non-recursive) locations to scan for 3LC Table objects.

This option allows searching individual folders/locations for 3LC Table objects outside the standard hierarchy of a 3LC project structure. For example:

“C:\Users\user\Documents\3LC\tables”
“s3://my-bucket/3LC/tables”.

Note: these locations will not be scanned recursively. See PROJECT_SCAN_URLS_OPTION for details on URL options.

Indexed Tables from these extra locations can be used interchangeably with other Tables discovered by the system.

default = []#

key = indexing.extra-table-scan-urls#

envvar = TLC_CONFIG_EXTRA_TABLE_SCAN_URLS#

data_type = None#

transform_func = None#

apply_transform_func_per_element = False#

format_func = None#

multi_entry_format_func = None#

class tlcconfig.options.EXTRA_RUN_SCAN_URLS_OPTION#

Bases: tlcconfig.options.OPTION

Extra (non-recursive) locations to scan for 3LC Run objects.

This option allows searching individual folders/locations for 3LC Run objects outside the standard hierarchy of a 3LC project structure. For example:

“C:\Users\user\Documents\3LC\runs”
“s3://my-bucket/3LC/runs”.

Note: these locations will not be scanned recursively. See PROJECT_SCAN_URLS_OPTION for details on URL options.

Indexed Runs from these extra locations can be used interchangeably with other Runs discovered by the system.

default = []#

key = indexing.extra-run-scan-urls#

envvar = TLC_CONFIG_EXTRA_RUN_SCAN_URLS#

data_type = None#

transform_func = None#

apply_transform_func_per_element = False#

format_func = None#

multi_entry_format_func = None#

class tlcconfig.options.LOGFILE#

Bases: tlcconfig.options.OPTION

Log file for the 3LC logger.

The directory will be created if it does not exist.

If the option value contains an environment variable, it will be expanded.

default = join(...)#: As returned by :func:tlcconfig.default_locations.get_default_log_dir / 3LC.log

key = logging.logfile#

envvar = TLC_LOGFILE#

argument = --logfile#

data_type = None#

transform_func = None#

local_ensure_file_is_writable() → bool#

validate_func = None#

class tlcconfig.options.LOGLEVEL#

Bases: tlcconfig.options.OPTION

Log level for the 3LC logger.

The tlc Python package adheres to the standard Python logging levels:

DEBUG: Detailed information, typically of interest only when diagnosing problems.
INFO: Confirmation that things are working as expected.
WARNING: An indication that something unexpected happened, or indicative of some problem in the near future (e.g. “disk space low”). The software is still working as expected.
ERROR: Due to a more serious problem, the software has not been able to perform some function.
CRITICAL: A serious error, indicating that the program itself may be unable to continue running.

default = WARNING#

key = logging.loglevel#

envvar = TLC_LOGLEVEL#

argument = --loglevel#

data_type = None#

transform_func = None#

validate_func = None#

class tlcconfig.options.DISPLAY_PROGRESS#

Bases: tlcconfig.options.OPTION

Whether to display progress bars or not.

The option can be either 0 or 1, where 0 means no progress bars and 1 means progress bars.

default = 1#

data_type = None#

transform_func = None#

key = tlc.display-progress#

envvar = TLC_DISPLAY_PROGRESS#

argument = --display-progress#

validate_func = None#

class tlcconfig.options.TELEMETRY#

Bases: tlcconfig.options.OPTION

Whether to send Telemetry or not. This option is deprecated and not in use.

default = _get_default_telemetry(...)#

data_type = None#

key = tlc.telemetry#

envvar = TLC_TELEMETRY#

argument = --telemetry#

transform_func = None#

class tlcconfig.options.ALIASES#

Bases: tlcconfig.options.OPTION

List of registered aliases

The option value should be a dictionary with the alias name as the key and the alias value as the value. In addition to the aliases in this list, the system will also consider ‘data aliases’ stored in special data configuration files and merge them with the aliases defined here. A data configuration file is a partial 3LC configuration that contains only one key: aliases. The system will look for such files, upon startup only, in the following locations:

<PROJECT_SCAN_URL>/config.3lc.yaml # top level
<PROJECT_SCAN_URL>/<project_xyz>/config.3lc.yaml # per project
<EXTRA_TABLE_SCAN_URL>/config.3lc.yaml
<EXTRA_RUN_SCAN_URL>/config.3lc.yaml

Special alias syntax

For aliases defined inside data configuration files it is possible to define an alias-value that is relative to the configuration file in which the alias is defined. This is done by prefixing the alias-value with $., $.. and so forth. The system will then expand the alias-value relative to the configuration file in which the alias is defined.

Examples

Example values (for all configuration files):

  PROJECT_XYZ_REMOTE_ALIAS: s3://my-bucket/3LC/projects/project-xyz
  PROJECT_XYZ_LOCAL_ALIAS: /home/user/projects/project-xyz

Special syntax examples (only for data configuration files):

  PROJECT_XYZ_DATA_ALIAS: $./data # expands to the data directory relative to the configuration file itself
  PROJECT_XYZ_PARENT_DATA_ALIAS: $../data # relative to parent dir
  PROJECT_XYZ_PARENT_PARENT_DATA_ALIAS: $.../data # relative to parent's parent dir

default = None#

data_type = None#

key = aliases#

transform_func = None#

apply_transform_func_per_element = False#

multi_entry_format_func = None#

format_func = None#

tlcconfig.options#

Module Contents#

Classes#

Data#

API#

Special alias syntax

Examples

`tlcconfig.options`#