`tlc.url`¶

Public URL API.

tlc.Url is hoisted to the top level; the surrounding URL surface lives here.

See URLs and Custom URL Adapters for narrative usage.

Package Contents¶

Classes¶

Class	Description
`AdapterInfo`	Structured metadata for a registered URL adapter.
`IfExistsOption`	An Enum indicating what to do if content already exists at the target URL
`Scheme`	String constants for URL schemes (convenience only).
`UrlAdapter`	The base class for all URL adapters.
`UrlAdapterDirEntry`	A directory entry returned by URL adapter operations.

Functions¶

Function	Description
`list_url_adapters`	List all registered URL adapters with structured metadata.
`list_url_schemes`	List the URL schemes of all registered adapters.
`register_url_adapter`	Class decorator that registers a `UrlAdapter` subclass.

API¶

class AdapterInfo¶

Bases: typing.TypedDict

Structured metadata for a registered URL adapter.

Initialize self. See help(type(self)) for accurate signature.

adapter_class: str = None¶

module: str = None¶

scheme: str = None¶

source: tlcurl.url_adapters._registry.AdapterSource = None¶

class IfExistsOption¶

Bases: str, enum.Enum

An Enum indicating what to do if content already exists at the target URL

from tlc import Url

url = Url("s3://my-bucket/my-file.txt")
url.write_text("Hello World!", if_exists="overwrite")

Initialize self. See help(type(self)) for accurate signature.

OVERWRITE = overwrite¶: If content already exists at the target URL, overwrite the content (default).

RAISE = raise¶: If content already exists at the target URL, raise an exception.

RENAME = rename¶: If content already exists at the target URL, leave that content, and write the new content to a new URL that is similar to the original target URL.

REUSE = reuse¶: If content already exists at the target URL, reuse the existing content.

class Scheme¶

String constants for URL schemes (convenience only).

This class provides string constants for the schemes that ship with 3LC. It exists purely for convenience (IDE autocomplete, avoiding typos) and documentation purposes.

The actual available schemes are determined entirely by what adapters are registered (see tlc.url.register_url_adapter()). Custom schemes work automatically when an adapter is registered - no need to modify this class.

Schemes that ship with 3LC::

Scheme.FILE = "file"         # file://  - Local filesystem
Scheme.S3 = "s3"             # s3://    - Amazon S3
Scheme.GS = "gs"             # gs://    - Google Cloud Storage
Scheme.ABFS = "abfs"         # abfs://  - Azure Blob Storage
Scheme.ABFSS = "abfss"       # abfss:// - Azure Blob Storage (Secure)
Scheme.API = "api"           # api://   - In-memory API objects
Scheme.HTTP = "http"         # http://  - HTTP URLs
Scheme.HTTPS = "https"       # https:// - HTTPS URLs

Internal/virtual schemes (not backed by standalone adapters)::

Scheme.RELATIVE = "relative" # Relative paths (handled by FileUrlAdapter)
Scheme.ALIAS = "alias"       # URL aliases (expanded against the process-wide alias registry)

Example usage::

from tlcurl.url import Scheme, Url

url = Url("/path/to/file")
if url.scheme == Scheme.FILE:
    print("This is a local file")

# Custom schemes work automatically when an adapter is registered
url = Url("mydb://database/table/key")
print(url.scheme)  # "mydb"

.. note::

These are string constants, not an enum. Comparisons use string equality: url.scheme == Scheme.FILE is equivalent to url.scheme == "file".

ABFS: str = abfs¶: Azure Blob File System scheme (abfs://).

ABFSS: str = abfss¶: Azure Blob File System Secure scheme (abfss://).

ALIAS: str = alias¶: URL alias scheme for expandable path aliases.

API: str = api¶: In-memory API object scheme (api://).

FILE: str = file¶: Local filesystem scheme (file://).

GS: str = gs¶: Google Cloud Storage scheme (gs://).

HTTP: str = http¶: HTTP scheme for web URLs.

HTTPS: str = https¶: HTTPS scheme for secure web URLs.

RELATIVE: str = relative¶: Relative URL scheme (no prefix, path only).

S3: str = s3¶: Amazon S3 scheme (s3://).

class UrlAdapter¶

Bases: abc.ABC

The base class for all URL adapters.

URL adapters provide I/O operations for specific URL schemes. All operations are synchronous; the core operations also have *_async variants on this class that default to running the sync method in a thread pool and may be overridden for natively async backends.

Subclass this ABC to implement a custom adapter — it ships sensible defaults for the optional methods and the string-encoding forwarders, so a subclass only has to provide the required abstract methods.

To implement a custom adapter, subclass this and implement only the required abstract methods:

schemes() - Return the URL schemes this adapter handles
read_binary_content_from_url() - Read binary content from a URL
exists() - Check if a URL exists

Custom adapters can be registered automatically via Python entry points. Declare the adapter in your package’s pyproject.toml::

[project.entry-points."tlc.url_adapters"]
myscheme = "mypackage.adapters:MyAdapter"

The adapter will be discovered and registered at startup when your package is installed.

All other methods have sensible default implementations or raise helpful NotImplementedError messages indicating which method to override for full functionality.

Optional methods to override:

write_binary_content_to_url() - Write binary content (enables writes)
list_dir() - List directory contents
is_dir() - Check if URL is a directory
delete_url() - Delete content
copy_url() - Copy content between URLs (default reads+writes)
make_dirs() - Create directories (default no-op for flat hierarchies)
get_file_size() - Get file size (default reads entire file)
is_writable() - Check if URL is writable (default False)
stat() - Get file metadata
touch() - Update modification time

.. seealso::

Complete examples in the `3lc-examples <https://github.com/3lc-ai/3lc-examples>`_ repository:
``examples/http-image-url-adapter``, ``examples/kitti-virtual-table``,
``examples/nifti-virtual-table``.

copy_url( source: Url, destination: Url, ) → None¶

Copy content from one URL to another.

Default implementation reads and writes. Override for more efficient server-side copies.

Parameters:

source – The source URL.
destination – The destination URL.

create_unique_url( url: Url, ) → Url¶

Create a unique version of the URL by finding the next available name.

This method generates a unique URL by appending a numeric suffix (e.g., _0000, _0001) to the stem of the URL if the original URL already exists.

Override for scheme-specific uniquification behavior.

Parameters:: url – The URL to make unique.
Returns:: A unique URL (may be the same URL if it doesn’t exist).

delete_url( url: Url, ) → None¶

Delete content at a URL.

Override to enable deletion.

Parameters:: url – The URL to delete.
Raises:: NotImplementedError – By default.

async delete_url_async( url: Url, ) → None¶

Delete content at a URL asynchronously.

Default implementation wraps :meth:delete_url in a thread pool. Override for native async I/O.

Parameters:: url – The URL to delete.

abstract exists( url: Url, ) → bool¶

Check if a URL exists.

This method MUST be overridden by subclasses.

Parameters:: url – The URL to check.
Returns:: True if the URL exists.

async exists_async( url: Url, ) → bool¶

Check if a URL exists asynchronously.

Default implementation wraps :meth:exists in a thread pool. Override for native async I/O.

Parameters:: url – The URL to check.
Returns:: True if the URL exists.

force: ClassVar[bool] = False¶: Class-level registration parameter. Set to True in a subclass to replace an already-registered adapter for the same scheme(s), including built-ins. Read by entry point discovery and the @register_url_adapter decorator.

format_url( scheme: str, path: str, query: str = '', ) → str¶

Format a URL for display/serialization.

Default implementation returns scheme://path[?query].

Parameters:

scheme – The URL scheme.
path – The normalized path.
query – Optional query string.

Returns:

The formatted URL string.

generate_self_authenticated_url( url: Url, expiration_seconds: int = DEFAULT_SELF_AUTHENTICATED_URL_EXPIRY_SECONDS, ) → str | None¶

Generate a self-authenticated URL that browsers can fetch without authorization headers.

Self-authenticated URLs carry their own authentication embedded in the URL itself, allowing browsers to load content in <img>, <video>, and <audio> tags without custom Authorization headers. The embedded authentication is time-limited.

The mechanism is adapter-specific (e.g. presigned URL for S3).

Default implementation returns None, indicating this adapter does not support self-authenticated URLs and the caller should fall back to proxying through the object service.

Parameters:

url – The URL to generate a self-authenticated URL for.
expiration_seconds – How long the self-authenticated URL should remain valid (default: :data:DEFAULT_SELF_AUTHENTICATED_URL_EXPIRY_SECONDS).

Returns:

A self-authenticated URL string, or None if not supported.

async generate_self_authenticated_url_async( url: Url, expiration_seconds: int = DEFAULT_SELF_AUTHENTICATED_URL_EXPIRY_SECONDS, ) → str | None¶

Generate a self-authenticated URL that browsers can fetch without authorization headers asynchronously.

The mechanism is adapter-specific (e.g. presigned URL for S3).

Default implementation wraps :meth:generate_self_authenticated_url in a thread pool, which itself defaults to returning None, indicating this adapter does not support self-authenticated URLs and the caller should fall back to proxying through the object service.

Derived adapters can override the sync method to support self-authenticated URLs or this method to support native async I/O (e.g., using aiobotocore, aiofiles).

Parameters:

url – The URL to generate a self-authenticated URL for.
expiration_seconds – How long the self-authenticated URL should remain valid.

Returns:

A self-authenticated URL string, or None if this adapter does not support self-authenticated URLs.

get_file_size( url: Url, ) → int¶

Get the size of the file at the URL.

Default implementation reads the entire file to get its size. Override for efficiency with large files.

Parameters:: url – The URL to check.
Returns:: The file size in bytes.

get_scheme_for_url( url_string: str, ) → str | None¶

Get the scheme for the given URL string if this adapter can handle it.

Default implementation checks if the URL string starts with any of the adapter’s prefixes. Override for complex logic (e.g., FILE scheme handling Windows paths).

Parameters:: url_string – The URL string to check.
Returns:: The scheme string if this adapter can handle it, None otherwise.

get_url_prefixes() → list[str]¶

Get URL prefixes this adapter handles (e.g., [‘s3://’]).

Default implementation derives prefixes from schemes().

Returns:: List of URL prefixes handled by this adapter.

is_absolute_scheme() → bool¶

Whether this adapter’s schemes represent absolute URLs.

Default is True. Override to return False for relative/virtual schemes.

Returns:: True if URLs with this scheme are inherently absolute.

is_dir( url: Url, ) → bool¶

Check if a URL is a directory.

Default implementation returns False (assumes files only).

Parameters:: url – The URL to check.
Returns:: True if the URL is a directory.

is_file_hierarchy_flat() → bool¶

Determine if the file hierarchy is flat for this adapter.

Flat hierarchies (like cloud storage) treat paths as opaque keys, so make_dirs is a no-op. Hierarchical file systems (like local disk) require explicit directory creation.

Default is True (flat), consistent with the no-op default of make_dirs. Override to return False for hierarchical file systems that need make_dirs.

Returns:: True if the file hierarchy is flat.

is_writable( url: Url, ) → bool¶

Check if a URL is writable.

Default implementation returns False. Override and return True if writes are supported.

Parameters:: url – The URL to check.
Returns:: True if the URL is writable.

async is_writable_async( url: Url, ) → bool¶

Check if a URL is writable asynchronously.

Default implementation wraps :meth:is_writable in a thread pool. Override for native async I/O.

Parameters:: url – The URL to check.
Returns:: True if the URL is writable.

list_dir( url: Url, ) → Iterator[UrlAdapterDirEntry]¶

List directory contents.

Override to enable directory listing.

Parameters:: url – The directory URL to list.
Returns:: An iterator of directory entries.
Raises:: NotImplementedError – By default.

make_dirs( url: Url, exist_ok: bool = False, ) → None¶

Create a leaf directory and all intermediate ones.

Default is a no-op (assumes flat hierarchy like cloud storage). Override if your storage requires directory creation.

Parameters:

url – The directory URL to create.
exist_ok – If True, don’t raise if directory exists.

abstract read_binary_content_from_url( url: Url, ) → bytes¶

Read binary content from a URL.

This method MUST be overridden by subclasses.

Parameters:: url – The URL to read from.
Returns:: The binary content at the URL.

async read_binary_content_from_url_async( url: Url, ) → bytes¶

Read binary content from a URL asynchronously.

Default implementation wraps :meth:read_binary_content_from_url in a thread pool. Override for native async I/O (e.g., using aiobotocore, aiofiles).

Parameters:: url – The URL to read from.
Returns:: The binary content at the URL.

read_string_content_from_url( url: Url, encoding: str = 'utf-8', ) → str¶

Read string content from a URL.

Concrete forwarding method — calls read_binary_content_from_url() and decodes with encoding (default UTF-8). Subclasses should not normally override this; override read_binary_content_from_url() instead.

Parameters:

url – The URL to read from.
encoding – The text encoding (default: utf-8).

Returns:

The string content at the URL.

async read_string_content_from_url_async( url: Url, encoding: str = 'utf-8', ) → str¶

Read string content from a URL asynchronously.

Default implementation wraps :meth:read_string_content_from_url in a thread pool. Override for native async I/O.

Parameters:

url – The URL to read from.
encoding – The text encoding (default: utf-8).

Returns:

The string content at the URL.

abstract schemes() → list[str]¶

Get URL schemes this adapter handles.

This method MUST be overridden by subclasses.

Returns:: List of scheme strings this adapter handles (e.g., [“file”], [“s3”]).

stat( url: Url, ) → UrlAdapterDirEntry¶

Get file metadata.

Default implementation creates a simple entry from available info. Override for more accurate metadata.

Parameters:: url – The URL to get metadata for.
Returns:: A directory entry with metadata.
Raises:: FileNotFoundError – If the URL doesn’t exist.

supports_relativization() → bool¶

Whether URLs with this adapter’s schemes can be relativized.

Default is True. Override to return False for schemes like API where relativization doesn’t make sense.

Returns:: True if URLs can be made relative.

touch( url: Url, ) → None¶

Update the modification timestamp or create an empty file.

Default implementation creates an empty file if it doesn’t exist.

Parameters:: url – The URL to touch.

write_binary_content_to_url( url: Url, content: bytes, ) → Url¶

Write binary content to a URL.

Override this method to enable write support.

Parameters:

url – The URL to write to.
content – The binary content to write.

Returns:

The URL the content was written to. Normally the input url; adapters that map writes onto a different location may return the resolved URL.

Raises:

NotImplementedError – By default.

async write_binary_content_to_url_async( url: Url, content: bytes, ) → Url¶

Write binary content to a URL asynchronously.

Default implementation wraps :meth:write_binary_content_to_url in a thread pool. Override for native async I/O.

Parameters:

url – The URL to write to.
content – The binary content to write.

Returns:

See :meth:write_binary_content_to_url.

write_string_content_to_url( url: Url, content: str, encoding: str = 'utf-8', ) → Url¶

Write string content to a URL.

Concrete forwarding method — encodes content with encoding (default UTF-8) and calls :meth:write_binary_content_to_url. Subclasses should not normally override this; override

Meth:

write_binary_content_to_url instead.

Parameters:

url – The URL to write to.
content – The string content to write.
encoding – The text encoding (default: utf-8).

Returns:

The URL the content was written to. Normally the input url; adapters that map writes onto a different location may return the resolved URL.

async write_string_content_to_url_async( url: Url, content: str, encoding: str = 'utf-8', ) → Url¶

Write string content to a URL asynchronously.

Default implementation wraps :meth:write_string_content_to_url in a thread pool. Override for native async I/O.

Parameters:

url – The URL to write to.
content – The string content to write.
encoding – The text encoding (default: utf-8).

Returns:

See :meth:write_string_content_to_url.

class UrlAdapterDirEntry( name: str, path: str, is_dir: bool = False, mtime: datetime | None = None, size: int = 0, )¶

A directory entry returned by URL adapter operations.

This class can be used directly or subclassed for custom implementations that wrap native objects (e.g., os.DirEntry, cloud storage listing info).

Initialize a directory entry.

Parameters:

name – The entry’s base filename.
path – The entry’s full path.
is_dir – Whether this entry is a directory.
mtime – The modification time (defaults to now).
size – The file size in bytes.

is_dir() → bool¶: Return True if this entry is a directory.

is_file() → bool¶: Return True if this entry is a file.

mtime() → Any¶: Return the modification time object for this entry.

mtime_datetime() → datetime¶: Return the modification time for this entry as a datetime object.

property name: str¶: The entry’s base filename.

property path: str¶: The entry’s full path name.

property size: int¶: The entry’s size in bytes, or zero for a directory or a store that does not report one.

list_url_adapters() → list[AdapterInfo]¶

List all registered URL adapters with structured metadata.

Entry-point plugins are discovered at package import time, so the list reflects everything installed via pip in addition to the built-in adapters.

Returns:: One AdapterInfo dict per registered scheme.

list_url_schemes() → list[str]¶

List the URL schemes of all registered adapters.

Use list_url_adapters() for full structured metadata.

Returns:: Sorted list of recognized scheme strings. Includes built-in schemes even when their backend packages are not installed, so URLs using those schemes can always be parsed.

register_url_adapter( cls: type[UrlAdapter], ) → type[UrlAdapter]¶

Class decorator that registers a UrlAdapter subclass.

Instantiates the class with no arguments and registers it for all schemes returned by schemes(). If instantiation or registration fails, the error is logged and the class is returned unregistered.

This means adapters with optional dependencies can raise ImportError in __init__ — the decorator will catch it gracefully.

Built-in adapters (those whose schemes are all built-in schemes) are recorded with source="builtin"; all others get source="runtime".

Parameters:: cls – The UrlAdapter subclass to register.
Returns:: The class, unchanged.

tlc.url¶

Package Contents¶

Classes¶

Functions¶

API¶

`tlc.url`¶