tlc.integration.super_gradients.callbacks.base_callback

Callback that logs metrics and hyperparameters from a SuperGradients Trainer in a 3LC Run, and facilitates per-sample metrics and embeddings collection.

Module Contents

Classes

Class

Description

MetricsCollectionCallback

Callback that collects per-sample metrics and logs SuperGradients aggregate metrics to a 3LC run.

API

class MetricsCollectionCallback(
project_name: str | None = None,
run_name: str | None = None,
run_description: str | None = None,
image_column_name: str = 'image',
label_column_name: str | None = None,
metrics_collection_epochs: list[int] | None = None,
collect_metrics_on_train_end: bool = True,
collect_val_only: bool = False,
batch_size: int | None = 32,
pipeline_params: PipelineParams | dict[str, Any] | None = None,
collect_predictions: bool | None = None,
collect_embeddings: bool = False,
embeddings_dim: int = 2,
embeddings_method: Literal[pacmap, umap] = 'pacmap',
inference_chunk_size: int | None = None,
dataloader_args: dict[str, Any] | None = None,
)

Bases: super_gradients.training.utils.callbacks.Callback

Callback that collects per-sample metrics and logs SuperGradients aggregate metrics to a 3LC run.

To collect per-sample metrics, subclasses must implement the methods compute_metrics and metrics_column_schemas, and the property label_column_name.

Create a new metrics collection callback.

Parameters:

  • project_name – The name of the 3LC project to use if no active run exists.

  • run_name – The name of the 3LC run to use if no active run exists.

  • run_description – The description of the 3LC run to use if no active run exists.

  • image_column_name – The name of the column in the table that contains the images.

  • label_column_name – The name of the column in the table that contains the labels. If not provided, a task-specific default will be used.

  • metrics_collection_epochs – The zero-indexed epochs after which to collect metrics.

  • collect_metrics_on_train_end – Whether to collect metrics after training finishes.

  • collect_val_only – Whether to collect metrics only on the validation set.

  • metrics_collection_dataloader_args – Additional arguments to pass to the dataloaders used for metrics collection.

  • batch_size – The batch size to use for metrics collection, passed to the SuperGradients Pipelines when performing inference. Controls the number of images in each batch on the device.

  • pipeline_params – The pipeline parameters to use for metrics collection. Can be provided as a PipelineParams instance, a dictionary, or None (default) which will use a default PipelineParams instance.

  • collect_predictions – Whether to collect predictions. Default is None, which means predictions will be logged for all task-specific subclasses if not explicitly set to False. On using the base callback, no predictions will be logged by default.

  • collect_embeddings – Whether to collect embeddings. Is only applied if the model has a backbone attribute (Yolo-NAS models have this). Default is False because embeddings collection performs additional inference and therefore spends additional time.

  • embeddings_dim – The dimensionality to reduce the embeddings to, default is 2. 2 or 3 are recommended.

  • embeddings_method – The method to use for reducing the embeddings.

  • inference_chunk_size – How many images to load into CPU memory at once for each dataloader worker batch. The number of (full-size) images in CPU memory at a time is batch_size * inference_chunk_size. If not provided, batch_size is used if not set to None, else 32 is used.

  • dataloader_args – Additional arguments to pass to the dataloader used for metrics collection. By default, 8 workers and no pinning of memory is used. batch_size is set to the value of the parameter inference_chunk_size, any value provided here is ignored. If inference_chunk_size is not provided, batch_size is used. Shuffling is disallowed.

collect_metrics_direct(
model: super_gradients.training.models.sg_module.SgModule,
tables: list[Table],
constants: dict[str, Any] | None = None,
) → None

Collect metrics directly on tables.

Parameters:

  • model – The model to collect metrics from.

  • tables – The tables to collect metrics from.

  • constants – Additional constants to pass to the metrics collection.

compute_metrics(
images: list[str],
predictions: super_gradients.training.utils.predict.prediction_results.ImagesPredictions | super_gradients.training.utils.predict.prediction_results.ImagePrediction,
table: Table,
) → dict[str, Any]

Compute metrics for a batch of images and corresponding predictions.

Parameters:

  • images – A list of absolute image URLs as strings.

  • predictions – A list of predictions or a single prediction.

  • table – The table containing the images and predictions.

Returns:

A dictionary of computed metrics.

property label_column_name: str

The name of the label column.

metrics_column_schemas(
table: Table,
) → dict[str, Schema]

Return the column schemas for the metrics of this callback.

on_train_loader_end(
context: super_gradients.training.utils.callbacks.PhaseContext,
) → None

Called when the training loader ends, logs aggregate SuperGradients metrics to the 3LC run.

on_training_end(
context: super_gradients.training.utils.callbacks.PhaseContext,
) → None

Called when training ends, invokes metrics collection if configured to and updates the run status.

Parameters:

context – The PhaseContext received from the SuperGradients Trainer.

on_training_start(
context: super_gradients.training.utils.callbacks.PhaseContext,
) → None

Called when training starts, creates a 3LC run if no active run exists and logs hyperparameters.

Raises:

ValueError – If project_name or run_name is provided and different from those of an existing active run.

on_validation_loader_end(
context: super_gradients.training.utils.callbacks.PhaseContext,
) → None

Called when the validation loader ends, logs aggregate SuperGradients metrics to the 3LC run.

property run: Run