tlc.client.torch.metrics.metrics_collectors.metrics_collector_base#

Base class for all metrics collectors in PyTorch.

Module Contents#

Classes#

Class

Description

MetricsCollector

Base class for all metrics collectors.

Data#

Data

Description

MetricsCollectorCallableType

Type alias for a metrics collector or a list of metrics collectors.

MetricsCollectorType

API#

class tlc.client.torch.metrics.metrics_collectors.metrics_collector_base.MetricsCollector(preprocess_fn: Callable[[tlc.core.builtins.types.SampleData, tlc.client.torch.metrics.predictor.PredictorOutput], tuple[Any, Any]] | None = None, compute_aggregates: bool = True)#

Base class for all metrics collectors.

A MetricsCollector is a class that computes metrics from a batch of data together with corresponding predictor output, usually the result of passing the batch through a model.

Custom metrics collectors can be created by subclassing this class and implementing the compute_metrics(), and optionally overriding the column_schemas() property.

A simpler way to create a custom metrics collector is to use the FunctionalMetricsCollector class, which takes a function that computes the metrics as input.

Several pre-built metrics collectors are provided in the metrics_collectors module.

Example:

table = ...
model = timm.create_model("resnet18", pretrained=False)

mc1 = tlc.ClassificationMetricsCollector()
mc2 = tlc.EmbeddingMetricsCollector()
mc3 = lambda batch, predictor_output: {"metric": [...]}

predictor = tlc.Predictor(model, layers=[99])

tlc.collect_metrics(table, [mc1, mc2, mc3], predictor)

Create a new metrics collector.

Parameters:

  • preprocess_fn – A function that pre-processes the batch and predictor output before computing the metrics.

  • compute_aggregates – Whether to compute aggregates for the metrics.

abstract compute_metrics(batch: tlc.core.builtins.types.SampleData, predictor_output: tlc.client.torch.metrics.predictor.PredictorOutput) → dict[str, tlc.core.builtins.types.MetricData]#

Compute metrics from a batch of data and corresponding predictor output.

Subclasses should implement this method to compute the metrics.

Parameters:

  • batch – The batch of data.

  • predictor_output – The predictor output.

Returns:

A dictionary of metrics.

preprocess(batch: tlc.core.builtins.types.SampleData, predictor_output: tlc.client.torch.metrics.predictor.PredictorOutput) → tuple[Any, Any]#

Pre-process the batch and predictor output before computing the metrics.

Calls the provided pre-process function if one is provided, otherwise returns the batch and predictor output unchanged.

reset() → None#
property aggregate_values: dict[str, float]#
property column_schemas: dict[str, tlc.core.schema.Schema]#
tlc.client.torch.metrics.metrics_collectors.metrics_collector_base.MetricsCollectorCallableType = None#

Type alias for a metrics collector or a list of metrics collectors.

tlc.client.torch.metrics.metrics_collectors.metrics_collector_base.MetricsCollectorType = None#