Custom URL adapters¶

3LC reads and writes data through URL adapters. There are built-in adapters for file://, s3://, gs://, abfs://, and http(s)://. You can add support for a new scheme, or override a built-in one, by registering your own adapter.

A custom adapter is a subclass of UrlAdapter. The base class ships working defaults for the optional operations and the string-encoding forwarders, so a subclass implements only what its backing store needs — at minimum the three required methods below.

Writing an adapter¶

Subclass UrlAdapter and implement the three required methods; override optional methods to add capabilities. Methods receive Url values, and content I/O is bytes-only — text encoding/decoding lives in the base class.

from tlc import Url
from tlc.url import UrlAdapter, UrlAdapterDirEntry


class MyAdapter(UrlAdapter):
    # --- required ---

    def schemes(self) -> list[str]:
        return ["my"]

    def read_binary_content_from_url(self, url: Url) -> bytes:
        ...

    def exists(self, url: Url) -> bool:
        ...

    # --- optional: override to add capabilities ---

    def write_binary_content_to_url(self, url: Url, content: bytes) -> Url:
        # Return the URL the content landed at — usually `url`, or a different
        # URL for a store that relocates (e.g. a versioning backend).
        ...

    def list_dir(self, url: Url):
        # Yield UrlAdapterDirEntry(name, path, is_dir=..., size=..., mtime=...).
        # Required for a scheme the indexer should discover.
        ...

    def is_dir(self, url: Url) -> bool: ...
    def stat(self, url: Url) -> UrlAdapterDirEntry: ...
    def is_writable(self, url: Url) -> bool: ...
    def delete_url(self, url: Url) -> None: ...

Only schemes, read_binary_content_from_url, and exists are required — every other method has a default (e.g. get_file_size reads the whole object, copy_url is read-then-write, make_dirs is a no-op for flat stores, is_writable returns False). Override the ones your store supports.

Signal failure by raising: FileNotFoundError for missing content, NotImplementedError for an unsupported operation. Exceptions propagate to the calling operation — including calls the indexer makes while scanning the scheme.

The adapter never sees credentials and never expands aliases: the registry resolves aliases and enforces activation/licensing around these calls, so enforcement stays in one place rather than in every adapter.

Capabilities — opt in by overriding¶

The required surface (schemes / read_binary_content_from_url / exists) plus list_dir is what makes a scheme discoverable and readable — the indexer lists a scan root and reads objects under it. A read-only reference scheme that is never scanned can leave list_dir at its default. Everything else is opt-in: override the relevant method and the capability turns on, since the registry routes each operation to the matching method (using the base default when the subclass doesn’t override it).

Capability	Override	Applies when
read (required)	`schemes`, `read_binary_content_from_url`, `exists`	always
listing	`list_dir`, `is_dir`	the scheme is discovered/scanned by the indexer
metadata	`stat`, `get_file_size`	per-URL metadata beyond `list_dir` entries
writing	`write_binary_content_to_url`, `delete_url`, `copy_url`, `touch`, `make_dirs`, `is_writable`	the backend is writable
signed URLs	`generate_self_authenticated_url`	browser-fetchable URLs (feature, not perf)
skip-if-unchanged	`change_token` (and optionally `mark_changed`)	the indexer should skip scopes whose content is unchanged

list_dir and stat return UrlAdapterDirEntry instances — construct them as UrlAdapterDirEntry(name, path, is_dir=..., size=..., mtime=...).

Override change_token(self, url: Url) -> str to opt into change-signalling: return a stable token that only advances when the scope’s content changes, and the indexer skips re-scanning a scope whose token is unchanged. Without it, the scheme is treated as always-changed (re-scanned every poll). An adapter may additionally define mark_changed(self, url: Url) to receive change-marker writes for the scopes it serves; without it those writes are no-ops.

Registering an adapter¶

Three ways to register. Each scheme has exactly one adapter; conflicts are resolved at registration time, not at lookup — a registration for a scheme that already has an adapter is rejected unless it carries force = True, in which case it replaces the existing adapter.

Decorator — apply register_url_adapter() to your subclass. It is instantiated with no arguments and registered for every scheme it returns from schemes():

from tlc.url import register_url_adapter, UrlAdapter


@register_url_adapter
class MyAdapter(UrlAdapter):
    def schemes(self) -> list[str]:
        return ["my"]

    ...

Entry point — declare it in your package’s pyproject.toml so it is discovered when your package is installed:
```
[project.entry-points."tlc.url_adapters"]
my = "mypackage.adapters:MyAdapter"
```

Configuration — list it in the 3LC configuration file under extensions.url-adapters:

extensions:
  url-adapters:
    - module: mypackage.adapters
      class: MyAdapter

Adding scan locations for the indexer¶

Registering an adapter teaches 3LC how to read your scheme — it does not, by itself, tell the indexer where to look. To have your objects discovered, add the scan locations to the configuration’s scan-urls. Each entry is a plain URL string (scanned as a project tree) or a dict with explicit layout (project or flat) and object_type (table / run).

Set it wherever suits — in the 3LC config file:

scan-urls:
  - url: my://root
    layout: flat
    object_type: table

via the TLC_SCAN_URLS environment variable, or at runtime through tlc.config:

import tlc

tlc.config.scan_urls = [*tlc.config.scan_urls, {"url": "my://root", "layout": "flat", "object_type": "table"}]

Adapter-owned scan locations¶

For a virtual scheme whose locations are fixed and known to the adapter — e.g. pxt3lc:// over a Pixeltable catalog, where the scan root pxt3lc://pixeltable is not something a user would (or could) put in their config — have the adapter contribute the scan URL itself, on construction, so it is always present without any user setup. Do it idempotently, since the adapter may be instantiated more than once (entry-point discovery, tests):

import tlc
from tlc.url import UrlAdapter


class PxtAdapter(UrlAdapter):
    _SCAN_URL = {"url": "pxt3lc://pixeltable", "layout": "single_dir", "object_type": "table"}

    def __init__(self) -> None:
        super().__init__()
        if self._SCAN_URL not in tlc.config.scan_urls:
            tlc.config.scan_urls = [*tlc.config.scan_urls, self._SCAN_URL]

    def schemes(self) -> list[str]:
        return ["pxt3lc"]

    ...

scan-urls mutations are thread-safe and take effect on the indexer’s next cycle, so contributing at construction time means registering the adapter and making its objects discoverable happen together.

Important

Keep this __init__ minimal and exception-safe. Discovery instantiates each adapter inside a try/except, so if __init__ raises, the whole adapter is dropped — losing both its I/O and its scan-url. Do the idempotent check-and-append shown above and nothing heavier (no network, no backend handshake).

How a custom adapter wins over a built-in one¶

Built-in adapters are the default, not a monopoly. When you register an adapter for a scheme, it is installed in the dispatch table and takes precedence over the built-in handler for that scheme — including a built-in scheme such as s3. Every operation for that scheme is then routed to your adapter, not the built-in one.

This means you can:

Add a brand-new scheme (e.g. my://) that 3LC has no built-in support for.
Override a built-in scheme to change how 3LC talks to it — for example, replacing the built-in s3:// handler with one that adds custom request signing, routes through a proxy, or talks to an S3-compatible store with non-standard semantics.

Overriding a built-in scheme requires force = True, because it shadows a handler 3LC ships. Set it as a class attribute:

from tlc.url import register_url_adapter, UrlAdapter


@register_url_adapter
class MyS3Adapter(UrlAdapter):
    force = True

    def schemes(self) -> list[str]:
        return ["s3"]

    def read_binary_content_from_url(self, url: Url) -> bytes:
        ...

Once registered, a call such as Url("s3://bucket/key").read_bytes() is served by MyS3Adapter.read_binary_content_from_url — the built-in S3 adapter is not taken. The precedence holds everywhere the registry is consulted, including from the indexing subsystem.