# 3LC (Python package)

> 3LC is a system for inspecting and improving ML training data sample-by-sample. It has two halves: a Python package (`tlc`, this document) and a separate Dashboard. The Python side describes training data as **Tables** and records per-sample model behaviour as **Runs**. The Dashboard reads those, lets a human inspect/curate, and writes edits back as new Table revisions that the next training run picks up.

This file is for LLM coding agents writing `tlc` code. It covers the Python package only — Dashboard usage, installation, and account setup live elsewhere. Read top-to-bottom once, then jump to the linked pages for depth.

**How to use this file.** It is a primer, not a complete API spec. Names and short rationales are listed; signatures and edge cases are not. When you need exact arguments or to confirm a symbol still exists in the user's installed version, check the venv at runtime (`dir(tlc.schemas)`, `help(tlc.Table.from_dict)`, `inspect.signature(...)`) and the [API reference](python-package/apidocs/index.html) — both are more current than this document. Treat anything here that contradicts the installed library as out of date.

## How it fits together

Three objects do almost all the work:

- **Table** — an immutable, schema-aware view of a dataset. Acts as a drop-in `torch.utils.data.Dataset` in training scripts (`table[i]` returns a sample dict), and as the canonical "what data went into this run" record in the Dashboard. Edits in the Dashboard produce new revisions; nothing is mutated in place.
- **Run** — a container for one experiment's metrics (per-sample and aggregate) plus its hyperparameters. Created with `tlc.init()` (or by an integration).
- **Metrics tables** — model output written into a Run. Per-sample tables (loss, predictions, embeddings, IoU, …) join back to the input Table by row id so a user clicking an outlier in the Dashboard lands on the source sample. Aggregate metrics (per-epoch accuracy, mean loss, …) are written alongside and surface as time-series in the Dashboard.

The Table is the *bridge*: same object on disk that the training loop iterates and that the Dashboard renders/edits. Anything that breaks that bridge (copying data into a Table, hand-rolling a metrics CSV) defeats the design.

## Designing your Table

This is the part agents most often get wrong. A Table is not just "the data" — it is **the data plus a schema** that tells the Dashboard how to render each column and tells the training loop how to materialise each sample. Time spent picking the right schemas saves a lot of "why doesn't this render?" later.

Two rules of thumb:

1. **Don't copy data into the Table.** Store paths, URLs, or small structured values — not raw tensors or decoded image arrays. The Table is metadata over the underlying data; let it stay a thin layer. This is why `Table.from_torch_dataset()` is rarely the right call: it works, but it tends to materialise tensors and obscure the source files, which is the opposite of the point.
2. **Pick the right schema per column.** A "label" column with no schema is just an int. With `CategoricalLabelSchema`, the Dashboard knows the class names, renders a dropdown, and the user can correct labels in place.

### The `from_dict` showcase

`Table.from_dict()` is the clearest illustration of the schema-driven design (defined in `src/tlc/_core/objects/table.py`):

```python
import tlc

table = tlc.Table.from_dict(
    {
        "image": ["s3://bucket/a.jpg", "s3://bucket/b.jpg", "/local/c.jpg"],
        "label": [0, 1, 0],
    },
    schema={
        "image": tlc.schemas.ImageSchema(),
        "label": tlc.schemas.CategoricalLabelSchema(classes=["cat", "dog"]),
    },
    project_name="demo",
    dataset_name="train",
)
```

Note what is *not* happening: no image bytes are copied, no tensors are stored. The Table holds URLs and ints; the schema teaches the Dashboard to render thumbnails and class names, and teaches the training loop how to decode `table[i]["image"]` into a PIL image on access.

### Built-in schemas (the user-facing ones)

Live in `tlc.schemas` (e.g. `tlc.schemas.ImageSchema()`). They are *not*
re-exported on the top-level `tlc` namespace — `tlc.ImageSchema` will raise.

- Images / media — `ImageSchema`, `SemanticSegmentationSchema`, `VideoUrlSchema`
- Labels — `CategoricalLabelSchema`, `CategoricalLabelListSchema`
- Embeddings — `EmbeddingSchema`
- Metric-shaped numerics — `ConfidenceSchema`, `FractionSchema`, `IoUSchema`, `ProbabilitySchema`
- Plain numerics / strings — `Int32Schema`, `Float32Schema`, `StringSchema`, etc.
- System columns — `EpochSchema`, `ExampleIdSchema`, `SampleWeightSchema`, `ForeignTableIdSchema`

`ImageSchema` is one class with four behaviours selected by `sample_type`:
`"pil_png"` (default), `"pil_jpeg"`, `"pil_webp"` — accept PIL images or URL
strings on write, return PIL on read; `"url"` — URL passthrough on both sides,
no PIL involvement, the right choice when the images are already on disk and
you don't want `tlc` to touch them.

For object detection, keypoints, polygons, oriented boxes, and similar structured columns, use the **annotation dataclasses in `tlc.data_types`** and their `.schema()` classmethod to build the column schema:

```python
import tlc

schema = {
    "image": tlc.schemas.ImageSchema(sample_type="url"),
    "bbs": tlc.data_types.BoundingBoxes2D.schema(classes=["cat", "dog"]),
    "keypoints": tlc.data_types.Keypoints2D.schema(num_keypoints=2, classes=["dog", "cat"]),
    "masks": tlc.data_types.SegmentationMasks.schema(classes=["bg", "fg"]),
}
```

Available: `BoundingBoxes2D`, `BoundingBoxes3D`, `Keypoints2D`, `OrientedBoundingBoxes2D`, `OrientedBoundingBoxes3D`, `SegmentationMasks`, `SegmentationPolygons`. Each dataclass also describes the Python-side shape of a row's value, so the same type is used to *construct* annotations in code and to *describe* the column schema. Don't reach for the underlying `tlc.sample_types.*SampleType` classes directly — those are an implementation detail; the dataclass `.schema()` factory is the supported entry point.

### When `from_dict` isn't enough: factories and TableWriter

If the user's data is in a known format, use the matching `Table.from_*` factory rather than writing a loader. A few representative examples:

```python
tlc.Table.from_image_folder(root, project_name=..., dataset_name=...)
tlc.Table.from_pandas(df)
tlc.Table.from_coco(annotations_url, image_folder_url)
tlc.Table.from_yolo_url(images_url, task="detect")
tlc.Table.from_hugging_face_dataset(hf_dataset)
```

Other formats covered by built-in factories: CSV, Parquet, NDJSON, HuggingFace Hub. Run `[f for f in dir(tlc.Table) if f.startswith("from_")]` to see what the installed version offers, and read each factory's docstring for the supported arguments.

A few notes:

- `Table.from_yolo()` is deprecated — use `Table.from_yolo_url()` for a single folder/text-file, and the [Ultralytics YOLO integration](integration-yolo) for YAML-configured datasets.
- `Table.from_url(url)` loads an *already-serialized* Table from disk — it is not a generic "fetch any data from a URL" loader. Reach for it to reopen a Table you wrote earlier, not as an import path for raw data.
- `Table.from_torch_dataset()` exists but is generally not the right call — see rule 1 above.

For custom, streaming, or programmatically-built data, use `tlc.TableWriter` (in `src/tlc/_core/writers/table_writer.py`). Declare the schema up front and append rows or batches:

```python
writer = tlc.TableWriter(
    project_name="demo",
    dataset_name="train",
    schema={
        "image": tlc.schemas.ImageSchema(sample_type="url"),
        "label": tlc.schemas.CategoricalLabelSchema(classes=["cat", "dog", "bird"]),
    },
)
for path, label in user_data:
    writer.add_row({"image": path, "label": label})
table = writer.finalize()
```

### Extensibility (advanced)

Two extension points let almost any data be made into a Table without moving it:

- **UrlAdapter** (`src/tlcurl/url_adapters/`) — pluggable I/O per URL scheme. Built-ins cover `file://`, `s3://`, `gcs://`, `azure://`, and HTTP. Register your own with `@register_url_adapter` to teach `tlc` to read from a private store. Existing Table code keeps working unchanged.
- **SampleType** (`src/tlc/sample_types/`) — defines how a row's value converts to/from a Python object on access. Register with `@register_sample_type` to support a custom annotation format, on-disk representation, or external blob type.

Reach for these when the data lives somewhere unusual or has a shape no built-in covers. Most users never need to.

## Using a Table in training, eval, and metrics collection

`table[i]` returns the *raw* sample shape declared by the schema — e.g. a `PIL.Image` plus an `int` label. That shape is right for inspection, but rarely what a model wants in a forward pass, and PIL won't go through the default PyTorch `collate_fn`. Don't bake the model-facing transforms into the Table; attach them on read:

```python
def to_tensor(sample):
    return {"image": TF.to_tensor(sample["image"]), "label": sample["label"]}

train_view = train_table.with_transform(augment_then_tensor)  # heavy aug, train only
eval_view  = train_table.with_transform(to_tensor)            # no aug, for metrics
val_view   = val_table.with_transform(to_tensor)
```

`with_transform(fn)` returns a `TableView` — a `MapDataset` you can hand directly to a `torch.utils.data.DataLoader` or to `tlc.collect_metrics(table=...)`. The transform runs per sample on read, so default collation works once the shapes are tensors/ints. The Table itself is unchanged; you can attach different transforms to the same underlying Table for train, eval, and collection.

Two gotchas:

- The transform must be **picklable** when the view is consumed by a `DataLoader` with `num_workers > 0`. Use a top-level function, not a lambda or local closure.
- Each call to `with_transform` returns a fresh view object — hoist `view = table.with_transform(fn)` if you need a stable reference (e.g. for caching).

## Runs and metrics

### Starting a Run

```python
run = tlc.init(
    project_name="my-project",
    run_name="experiment-42",
    parameters={"lr": 1e-3, "epochs": 10},   # hyperparams, visible in Dashboard
    if_exists="rename",                       # "reuse" / "overwrite" / "raise" also valid
)
```

One Run per experiment. `tlc.init()` sets the Run "active" so later metric writes attach to it. Re-running the script creates a new Run by default.

Hyperparameters can also be added (or updated) after the Run starts: `run.set_parameters({...})`. Useful when some hyperparameters are only known after data prep or a sweep injects them.

### Collecting per-sample metrics: four entry points, from most managed to most manual

**(1) Integration** — if the user is on a supported framework, the integration starts the Run, runs metrics on eval, and manages lifecycle. Maturity varies:

- `tlc.integration.hugging_face.trainer` — `Trainer` wrapper, well-supported.
- `tlc.integration.detectron2` — hooks (`MetricsCollectionHook`, `DetectronMetricsCollectionHook`), `register_coco_instances`, `BoundingBoxMetricsCollector`.
- `tlc.integration.super_gradients` — `MetricsCollectionCallback`, `DetectionMetricsCollectionCallback`, `PoseEstimationMetricsCollectionCallback`.
- `tlc.integration.torch` — not a "framework integration" per se, but sampler helpers (`RepeatByWeightSampler`, `create_weighted_sampler`, …) that consume the per-sample weights edited from the Dashboard.

YOLO/Ultralytics is supported at the *data* layer via `Table.from_yolo_url()`. The Ultralytics trainer integration lives in a separate repo: [github.com/3lc-ai/3lc-ultralytics](https://github.com/3lc-ai/3lc-ultralytics) — install from there, not from `tlc.integration`.

**(2) `tlc.collect_metrics()`** — the canonical SDK pattern. You hand it a Table, a predictor (model), and one or more metrics collectors; it runs the inference loop for you. Use this from a custom PyTorch training loop after each epoch — it's the cleanest demonstration of how the SDK is meant to be used.

Collector classes and the `Predictor` wrapper live in `tlc.metrics` (not on the top-level `tlc.*` namespace). `Predictor` is the bridge between a plain `nn.Module` and any collector that needs more than the forward output — most importantly, hidden-layer activations for `EmbeddingsMetricsCollector`:

```python
predictor = tlc.metrics.Predictor(model, layers=[idx])  # idx into model.named_modules()

tlc.collect_metrics(
    table=val_table,
    metrics_collectors=[
        tlc.metrics.ClassificationMetricsCollector(classes=["cat", "dog", "bird"]),
        tlc.metrics.EmbeddingsMetricsCollector(layers=[idx]),
    ],
    predictor=predictor,
    split="val",
    constants={"epoch": epoch},
)
```

`EmbeddingsMetricsCollector(layers=...)` takes **layer indices** (`Sequence[int]`), not `nn.Module` references — the index is the position of the module in `model.named_modules()`. The same indices must be passed to `Predictor(layers=...)` so the activations actually get captured. Resolve the index up front by walking `named_modules()` for the layer you want.

Note `predictor=` (not `model=`). A plain `model` works too if no collector needs hidden activations.

Built-in collectors (all in `tlc.metrics`, except where noted):

- `FunctionalMetricsCollector(fn)` — wrap any `(batch, predictions) -> dict[str, list]` callable.
- `ClassificationMetricsCollector` — loss, predicted class, accuracy, confidence.
- `EmbeddingsMetricsCollector` — captures hidden-layer activations; pair with the Dashboard's scatter view (typically after dimensionality reduction).
- `SegmentationMetricsCollector` — per-class IoU and friends.
- `BoundingBoxMetricsCollector` (in `tlc.integration.detectron2`) — bounding-box predictions in 3LC's format.

A worked custom-PyTorch-loop example lives at `docs/source/examples/tutorials/3-training-and-metrics/pytorch-fashion-mnist-resnet-training.ipynb`.

**(3) `tlc.MetricsTableWriter`** — when the user already has a validation loop and you just need to push numbers in, batch by batch. No predictor, no Table iteration controlled by `tlc`. Use it as a context manager — on exit it finalizes the table *and* updates the Run automatically:

```python
with tlc.MetricsTableWriter(
    foreign_table_url=val_table.url,  # required to link rows back to the input Table
) as writer:
    for batch in val_loader:
        preds = model(batch["image"])
        writer.add_batch({
            "example_id": batch["id"],   # required when foreign_table_url is set
            "loss": losses,
            "confidence": confs,
        })
```

`run_url` defaults to the active Run, so you don't need to pass it. Each batch must include `example_id` (linear indices into the foreign table); the Dashboard joins on those.

**(4) `run.add_metrics(...)`** — when the metrics dict is already computed in full (e.g. you ran your own loop and collected lists of per-sample numbers). One call, no streaming:

```python
run.add_metrics(
    {"loss": losses, "confidence": confs, "example_id": ids},
    foreign_table_url=val_table.url,
)
```

This is the right entry point for "I have a dict in hand" and the wrong one for "I'm in a loop" — use `MetricsTableWriter` for the latter so memory doesn't grow unboundedly.

**The loop must be an eval loop, even when iterating the training Table.** No data augmentation, no random crops/flips, no shuffling — each row must be produced in its canonical form so the metrics line up with what the Dashboard renders. This applies to (2) as well: `collect_metrics()` uses eval-mode iteration by default, but if you pass `dataloader_args`, don't enable augmentations or shuffling there.

### Aggregate metrics: `tlc.log()`

Per-sample metrics are 3LC's distinctive feature, but you usually also want a few scalars-per-epoch on the Run overview (mean loss, accuracy, learning rate). Use `tlc.log()` for that — anything in the dict with key `epoch` or `iteration` becomes the X-axis of an automatically-created chart:

```python
tlc.log({"epoch": epoch, "train_loss": mean_loss, "val_accuracy": acc})
```

`tlc.log()` writes to the active Run; pass `run=` to target a specific one. Aggregate metrics live alongside the per-sample tables and are independent of them — log both.

## Anti-patterns

The mistakes coding agents most often make:

- **Reaching for `tlc.X` for everything.** The top-level package only re-exports
  the nouns and verbs you build with: `Table`, `Run`, `TableWriter`,
  `MetricsTableWriter`, `init`, `collect_metrics`. Schemas live in `tlc.schemas`,
  collectors and the `Predictor` wrapper live in `tlc.metrics`, geometry
  dataclasses live in `tlc.data_types`. If `tlc.Foo` raises `AttributeError`,
  the answer is almost always to use the submodule path.
- **Copying data into the Table.** `Table.from_torch_dataset()` is the easiest
  way to do this accidentally — it materialises tensors out of an existing
  PyTorch Dataset and obscures the source files. Use `Table.from_image_folder`,
  `from_coco`, `from_yolo_url`, `from_pandas`, or `TableWriter` with URL-shaped
  columns instead.
- **Hand-rolling a metrics CSV / pickle / W&B log alongside the Run.** Metrics
  must be written through `collect_metrics()` or `MetricsTableWriter` to land
  on the Run with `example_id` joins back to the input Table. Anything else
  breaks the Dashboard's metric→sample drill-down — the feature 3LC exists for.
- **Iterating the eval Table with shuffling or augmentation.** Per-sample
  metrics must line up with the canonical row order. No `shuffle=True`, no
  random crops/flips, no augmentation transforms. This holds whether you're
  using `collect_metrics()` (don't pass augmenting transforms in
  `dataloader_args`) or `MetricsTableWriter` (run the same iteration order as
  the Table itself).
- **Passing `nn.Module` references to `EmbeddingsMetricsCollector(layers=...)`.**
  It takes `Sequence[int]` — positions in `model.named_modules()`. Same
  indices go to `tlc.metrics.Predictor(layers=...)`. The module-reference
  shortcut does not exist.
- **Skipping `Predictor` and passing the bare model when you want embeddings.**
  Hidden-layer activations are extracted via forward hooks that `Predictor`
  installs. `predictor=model` alone gives you `EmbeddingsMetricsCollector`
  nothing to read.
- **Forgetting `foreign_table_url=` (and `example_id` in each batch) when
  using `MetricsTableWriter` directly.** Without these the metrics rows can't
  be joined back to the input Table, so the Dashboard treats them as
  free-floating numbers — the link from metric to sample is lost.
- **Mutating a Table in place.** Tables are immutable; the Dashboard produces
  new revisions on edit. Don't try to overwrite rows; build a new Table (or
  let the Dashboard do it) and read the new revision in the next training
  run.
- **Storing raw arrays or tensors in schemas that exist for IDs/URLs.** A
  `label` column with no schema is just an int — usable, but the Dashboard
  can't render a class dropdown. A `bbox` column stored as a flat list won't
  render as boxes. Always reach for the schema or `data_types` dataclass
  that matches the column's meaning.

## When 3LC is the wrong tool

3LC is for inspecting and curating training data and per-sample model behaviour. It covers the basics of experiment tracking — hyperparameters, aggregate metrics over time — but it is **not** a full experiment-tracking platform (no artifact store, no system-metric capture, no model registry) and **not** a training framework. If the user only needs artifact logging and scalar dashboards, pair it with (or use) Weights & Biases or TensorBoard. 3LC's distinctive value is the link from a metric to the *specific samples* behind it, and the ability to edit the dataset from there — that is what to lean on.

## Where to look next

- [Getting Started](getting-started/index.html) — install, first Run, Dashboard setup
- [Tables](user-guide/tables/index.html) — schemas, revisions, sample/row views, bulk data
- [Runs](user-guide/runs/index.html) — lifecycle, hyperparameters, metrics tables
- [Integrations](python-package/integrations/index.html) — Hugging Face, Detectron2, SuperGradients
- [Examples](examples/index.html) — end-to-end notebooks (start here for full working code)
- [API Reference](python-package/apidocs/index.html) — exhaustive `tlc.*` symbol index
- [Project Layout](user-guide/project-layout.html) — on-disk file structure
- [URLs](python-package/urls.html) — `tlc://`, S3, GCS, Azure
- [FAQ](faq/faq.html)