tlc.client.torch.metrics.collect#

Collect per-sample metrics with a tlc Table.

Module Contents#

Functions#

Function

Description

collect_metrics

Collect per-sample metrics with a Table.

API#

tlc.client.torch.metrics.collect.collect_metrics(table: tlc.core.objects.table.Table, metrics_collectors: tlc.client.torch.metrics.metrics_collectors.MetricsCollectorType, predictor: torch.nn.Module | tlc.client.torch.metrics.predictor.Predictor | None = None, constants: dict[str, Any] = {}, constants_schemas: dict[str, tlc.core.schema.Schema] | None = None, run_url: tlc.core.url.Url | str | None = None, collect_aggregates: bool = True, split: str = '', exclude_zero_weights: bool = False, *, dataloader_args: dict[str, Any] | None = None) → None#

Collect per-sample metrics with a Table.

  • Writes a single metrics table which uses the input table as foreign table. The written metrics table will contain any constants contained in the constants argument, as well as any metrics computed by the metrics collectors.

  • Adds the metadata of the metrics table to the metrics property of the Run.

  • Adds the Url of the input table to the Run as an input.

  • Collects aggregate values from the metrics collectors and add them to the Run.

Parameters:

  • table – The Table to collect metrics from.

  • metrics_collectors – A list of metrics collectors to use. Can be a single metrics collector, a list of metrics collectors, or a list of callables with the signature Callable[[SampleData, PredictorOutput], dict[str, Any]].

  • constants – A dictionary of constants to use when collecting metrics.

  • constants_schemas – A dictionary of schemas for the constants. If no schemas are provided, the schemas will be inferred from the constants.

  • run_url – The url of the run to add the metrics to. If not specified, the active run will be used. If no active run is found, a new run will be created.

  • collect_aggregates – Whether to collect aggregate values from the metrics collectors and add them to the Run. This allows an aggregate view to be shown in the Project page of the 3LC Dashboard. Aggregate values are computed for all computable columns in the metrics collectors, and are prefixed with the split name. For example, if a metrics collector defines a computable column called “accuracy”, and the split is “train”, then the aggregate value will be called “train_accuracy_avg”.

  • split – The split of the dataset. This will be prepended to the aggregate metric names.

  • exclude_zero_weights – Whether to exclude samples with zero weights when collecting metrics.

  • dataloader_args – Additional arguments to pass to the dataloader.