`tlc.metrics.collectors.metrics_collector_base`¶

Base class for all metrics collectors in PyTorch.

Module Contents¶

Classes¶

Class	Description
`MetricsCollector`	Base class for all metrics collectors.

Data¶

Data	Description
`MetricsCollectorCallableType`	Type alias for a metrics collector or a list of metrics collectors.
`MetricsCollectorType`

API¶

class MetricsCollector( *, preprocess_fn: Callable[[Any, PredictorOutput], tuple[Any, Any]] | None = None, compute_aggregates: bool = True, )¶

Base class for all metrics collectors.

A MetricsCollector is a class that computes metrics from a batch of data together with corresponding predictor output, usually the result of passing the batch through a model.

Custom metrics collectors can be created by subclassing this class and implementing the compute_metrics(), and optionally overriding the column_schemas() property.

A simpler way to create a custom metrics collector is to use the FunctionalMetricsCollector class, which takes a function that computes the metrics as input.

Several pre-built metrics collectors are provided in the metrics_collectors module.

Example:

table = ...
model = timm.create_model("resnet18", pretrained=False)

mc1 = tlc.metrics.EmbeddingsMetricsCollector(layers=[99])
mc2 = lambda batch, predictor_output: {"metric": [...]}

predictor = tlc.metrics.Predictor(model, layers=[99])

tlc.collect_metrics(table, [mc1, mc2], predictor)

Create a new metrics collector.

Parameters:

preprocess_fn – A function that pre-processes the batch and predictor output before computing the metrics.
compute_aggregates – Whether to compute aggregates for the metrics.

property aggregate_values: dict[str, float]¶

The aggregate metrics accumulated across all processed batches.

Returns:: A mapping from aggregate metric name to its finalized value. Empty unless the collector was created with compute_aggregates=True.

property column_schemas: dict[str, Schema]¶

Schema overrides for the columns this collector produces.

Override in a subclass to give the collected metrics explicit schemas; any column not listed here has its schema inferred from the data.

Returns:: A mapping from column name to schema. Empty by default (infer everything).

abstract compute_metrics( batch: Any, predictor_output: PredictorOutput, ) → dict[str, Any]¶

Compute metrics from a batch of data and corresponding predictor output.

Subclasses should implement this method to compute the metrics.

Parameters:

batch – The batch of data.
predictor_output – The predictor output.

Returns:

A dictionary of metrics.

preprocess( batch: Any, predictor_output: PredictorOutput, ) → tuple[Any, Any]¶

Pre-process the batch and predictor output before computing the metrics.

Calls the provided pre-process function if one is provided, otherwise returns the batch and predictor output unchanged.

reset() → None¶

Reset any accumulated state between metrics-collection passes.

Clears the running aggregates so the collector can be reused for a fresh pass. Subclasses that maintain additional state should override this and call super().reset().

MetricsCollectorCallableType = None¶: Type alias for a metrics collector or a list of metrics collectors.

MetricsCollectorType = None¶

tlc.metrics.collectors.metrics_collector_base¶

Module Contents¶

Classes¶

Data¶

API¶

`tlc.metrics.collectors.metrics_collector_base`¶