Collect and reduce classifier embeddings¶

In this tutorial, we will use an existing classifier model to generate per-instance embeddings for a COCO-style object detection dataset. We will then reduce these embeddings to 3D using PaCMAP.

To run this notebook, you must also have run:

1-train-crop-model.ipynb

Install dependencies¶

[ ]:

%pip install -q 3lc[pacmap]
%pip install -q git+https://github.com/3lc-ai/3lc-examples.git
%pip install -q timm

Imports¶

[ ]:

import tlc

from tlc_tools.augment_bbs.extend_table_with_metrics import extend_table_with_metrics

Project setup¶

[ ]:

PROJECT_NAME = "3LC Tutorials - COCO128"
TMP_PATH = "../../../transient_data"
MODEL_NAME = "efficientnet_b0"
BATCH_SIZE = 32
NUM_COMPONENTS = 3

[ ]:

MODEL_CHECKPOINT = TMP_PATH + "/instance_classifier.pth"

Get input Table¶

[ ]:

# Open the Table used in the previous notebook
input_table = tlc.Table.from_names(
    table_name="initial-segmentation",
    dataset_name="COCO128",
    project_name=PROJECT_NAME,
)

Collect embeddings and metrics from pre-trained model¶

[ ]:

output_table_url, pacmap_reducer, fit_embeddings = extend_table_with_metrics(
    input_table=input_table,
    output_table_name="extended",
    add_embeddings=True,
    add_image_metrics=True,
    model_name=MODEL_NAME,
    model_checkpoint=MODEL_CHECKPOINT,
    batch_size=BATCH_SIZE,
    num_components=NUM_COMPONENTS,
)

[ ]:

output_table_url