tlc.client.torch.metrics.metrics_collectors.metrics_collector_base
#
Base class for all metrics collectors in PyTorch.
Module Contents#
Classes#
Class |
Description |
---|---|
Base class for all metrics collectors. |
API#
- class tlc.client.torch.metrics.metrics_collectors.metrics_collector_base.MetricsCollector(preprocess_fn: Callable[[tlc.core.builtins.types.SampleData, tlc.client.torch.metrics.predictor.PredictorOutput], tuple[Any, Any]] | None = None, compute_aggregates: bool = True)#
Base class for all metrics collectors.
A MetricsCollector is a class that computes metrics from a batch of data together with corresponding predictor output, usually the result of passing the batch through a model.
Custom metrics collectors can be created by subclassing this class and implementing the
compute_metrics()
, and optionally overriding thecolumn_schemas()
property.A simpler way to create a custom metrics collector is to use the
FunctionalMetricsCollector
class, which takes a function that computes the metrics as input.Several pre-built metrics collectors are provided in the
metrics_collectors
module.- Example:
table = ... model = timm.create_model("resnet18", pretrained=False) mc1 = tlc.ClassificationMetricsCollector() mc2 = tlc.EmbeddingMetricsCollector() mc3 = lambda batch, predictor_output: {"metric": [...]} predictor = tlc.Predictor(model, layers=[99]) tlc.collect_metrics(table, [mc1, mc2, mc3], predictor)
Create a new metrics collector.
- Parameters:
preprocess_fn – A function that pre-processes the batch and predictor output before computing the metrics.
compute_aggregates – Whether to compute aggregates for the metrics.
- abstract compute_metrics(batch: tlc.core.builtins.types.SampleData, predictor_output: tlc.client.torch.metrics.predictor.PredictorOutput) dict[str, tlc.core.builtins.types.MetricData] #
Compute metrics from a batch of data and corresponding predictor output.
Subclasses should implement this method to compute the metrics.
- Parameters:
batch – The batch of data.
predictor_output – The predictor output.
- Returns:
A dictionary of metrics.
- preprocess(batch: tlc.core.builtins.types.SampleData, predictor_output: tlc.client.torch.metrics.predictor.PredictorOutput) tuple[Any, Any] #
Pre-process the batch and predictor output before computing the metrics.
Calls the provided pre-process function if one is provided, otherwise returns the batch and predictor output unchanged.
- property column_schemas: dict[str, tlc.core.schema.Schema]#