Background Indexing System¶

3LC maintains an index over all known projects and objects. The index is kept up to date by a background indexing engine and normally has little direct impact on 3LC usage.

The index is used for two purposes:

To serve the 3LC Dashboard with up-to-date available Tables and Runs, c.f. the Object Service
To make it possible to deduce dataset lineage and keep track of revisions, as in Table.latest()

Background indexing is not started until requested, either explicitly during Object Service startup, or lazily by calls that depend on the index, such as Table.latest(). Indexing can be started and stopped without impact; it will be restarted by any call that depends on the latest index status.

What gets indexed¶

To create the index, 3LC recursively scans all configured project locations to discover Tables, Runs, and configuration files. The following configuration settings control which locations are scanned:

project-root-url: a single URL — the primary location for 3LC project data
scan-urls: a list of additional URLs to scan

The project root is always scanned, so it does not need to be repeated in scan-urls. Each scan-urls entry can be a plain URL string or a dict with explicit attributes (url, layout, object_type, static). Locations may be local file paths or cloud storage URLs (s3://, gs://, abfs://).

See the configuration documentation for more information.

Configuration files (config.3lc.yaml and default_aliases.3lc.yaml) discovered inside scanned locations are also ingested, so data-bundled settings such as URL aliases take effect automatically as part of indexing.

When indexing runs¶

Each scan location is polled on a fixed interval, controlled by the indexing.scan-interval configuration option (default 10 seconds, environment variable TLC_INDEXING_SCAN_INTERVAL). Scanning project locations — especially remote cloud storage — can take time and incur expense, so the indexer uses a change signal to avoid unnecessary scans.

Change-signal markers¶

3LC automatically creates and manages small index.3lc.json marker files in indexed locations. Before each scan, the indexer checks the marker for the location; if it is unchanged since the previous scan, the scan is skipped entirely. This means:

Unchanged locations cost a single metadata lookup per interval instead of a full scan
For cloud storage (S3, GCS, etc.), this dramatically reduces API calls and costs
Changes are quickly propagated to the 3LC Dashboard (from the Object Service)

Creation and modification of Tables, Runs, and revisions through the Python package update the markers automatically, as do edits made from the Dashboard or other 3LC processes. A location that has no marker yet is scanned, after which the indexer writes the initial marker itself.

The markers are safe to ignore in version control. The marker body is a small JSON timestamp payload readable by legacy 3LC services, so installations on different versions can index the same locations.

External file changes¶

If you copy, move, or modify 3LC project files outside of the Python package (e.g., using a file explorer, the command line, or other tools), the markers are not updated and the changes are not detected by the skip check. The following options are available:

Announce the change from Python with tlc.discovery.notify_write() or tlc.discovery.notify_delete():
```
import tlc

tlc.discovery.notify_write("s3://mybucket/projects/my-project/datasets/mnist/tables/train")
tlc.discovery.notify_delete("s3://mybucket/projects/my-project/runs/train-2026-06-01")
```
A notification writes change markers at the affected discovery scopes (for tables and runs in the canonical project layout: the enclosing project directory and the scan root above it) and prompts a running indexer in the same process to revisit the URL right away — including URLs the indexer had previously recorded as failed. Indexers in other processes pick the change up through the markers on their next scan interval. This is the right tool for pipelines and external services that produce or remove 3LC objects through means the indexer cannot observe. The affected scopes are deduced from the URL: a canonical project-layout table or run refreshes its project directory and the scan root above it, while any other location refreshes its containing directory.
Delete (or re-save) the index.3lc.json marker file in the affected directory and at the scan root — the indexer detects the marker change and re-scans on its next interval.
Restart the Python process — the first scan after startup always runs in full.

Read-only and static data sources¶

When publishing datasets that will never change after publication (e.g., shared cloud buckets), the scan location can be declared static so consumers scan it exactly once per process instead of polling it:

Consumer-side: mark the entry in scan-urls with static: true:

scan-urls:
  - url: s3://mybucket/published-datasets
    static: true

Publisher-side: ship a config.3lc.yaml file containing scope.static: true alongside the published data:
```
scope:
  static: true
```
The setting is picked up during indexing and applies to the file’s directory subtree, so every consumer gets the one-shot behavior without configuring anything. (Per-location markers live under the scope key; the process-level indexing.* options — such as scan-interval — are not honored from a published config.)

A static location is scanned once when indexing starts and is not re-polled for the rest of the process lifetime.

Failure handling¶

URLs that fail during indexing are recorded in a skip store and retried on a per-reason cadence, so a problematic location cannot stall the rest of the index:

Transient failures (network errors, timeouts, listing races) are retried within seconds, with the delay escalating on repeated failures.
Failures requiring operator action (permission denied, malformed content) are retried on long cadences (hours). Objects of unrecognized types are skipped until the type is registered.

A failure that affects listing a directory skips the whole subtree; a failure loading a single object skips only that URL. Writing a new version of a previously-failing object through the Python package — or announcing an external repair with tlc.discovery.notify_write() — clears its skip entry so it is retried immediately.

The retry cadence for transient failures can be scaled with the indexing.backoff-multiplier configuration option (environment variable TLC_INDEXER_BACKOFF_MULTIPLIER): values above 1.0 retry more slowly (useful on slow networks), values below 1.0 retry faster. All indexing issues are recorded by 3LC logging and can be inspected at the log location.

Indexing and Performance¶

The indexing system runs in the background, but some calls, like Table.latest(), require the index to be up to date and will block until completed (unless a timeout has been given). With the change-signal optimization outlined above, this is normally only an issue for the first call, but may still impose an unwanted penalty, especially if the set of scan URLs has grown unwieldy. The indexer must read the full set of scan URLs, so the following guidelines are useful to consider, especially in training scripts:

Prefer local, per project configuration (instead of global)
Prune unused or inactive scan URLs from the configuration
Declare published, immutable locations as static