Keypoints and Pose Estimation in 3LC

../../_images/horse-keypoints-dark.png
../../_images/horse-keypoints-light.png

What is Pose Estimation?

Pose estimation is a computer vision task that identifies and tracks the spatial configuration of objects by detecting keypoints.

Working with Keypoints in 3LC

3LC Tables store keypoint data in a hierarchical structure of object instances. A typical keypoint structure includes 2D points, with a skeleton topology, per-point visibility, a bounding box, and a label.

The structure of a keypoints column is defined by its Schema. The Keypoints2DSchema class is used to define the schema of a keypoints column.

Creating Keypoint Tables

To get started with keypoint data, you’ll need to create keypoint Tables:

When working with keypoints Tables - such as during custom data loading or prediction writing - the Keypoints2DInstances helper class simplifies conversion between Table rows and numpy arrays.

This helper class provides:

  • Reading from Tables: Convert a Table row to structured numpy arrays with from_row()

  • Writing to Tables: Convert numpy arrays back to Table row format with to_row()

  • Building from scratch: Create empty instances with create_empty() and add data incrementally with add_instance()

from tlc import Keypoints2DInstances

# Reading: Convert Table row to numpy arrays
kpts = Keypoints2DInstances.from_row(table_row)
kpts.keypoints.shape         # (num_instances, num_keypoints, 2)
kpts.instance_labels.shape   # (num_instances,)

# Writing: Convert numpy arrays back to Table format
updated_row = kpts.to_row()

Reading Keypoint Metadata

3LC provides the KeypointHelper class to extract keypoint metadata and schema information from Tables.

Shapes and index flattening are handled internally; these helpers return standard numpy arrays and Python lists.

from tlc import KeypointHelper

# Get keypoint shape (number of keypoints, channels)
shape = KeypointHelper.get_keypoint_shape_from_table(table)
shape  # (17, 3)

# Get keypoint names and attributes
keypoint_attrs = KeypointHelper.get_keypoint_attributes_from_table(table)
keypoint_attrs  # [{'internal_name': 'nose'}, {'internal_name': 'left_eye'}, ...]

# Get skeleton connections
skeleton = KeypointHelper.get_lines_from_table(table)
skeleton  # [0, 1, 0, 2, 1, 3, ...]

# Get flip indices for data augmentation
flip_indices = KeypointHelper.get_flip_indices_from_table(table)
flip_indices  # [0, 2, 1, 4, 3, ...]

# Get OKS sigmas for evaluation
oks_sigmas = KeypointHelper.get_oks_sigmas_from_table(table)
oks_sigmas  # [0.026, 0.025, 0.025, ...]

# Modify or set the OKS sigmas
edited_table = KeypointHelper.edit_oks_sigmas(table, [0.025, 0.025, 0.025, 0.025])

# Modify or set the default keypoint coordinates
edited_table = KeypointHelper.edit_default_keypoints(table, [0.025, 0.025, 0.025, 0.025])

# Modify or set the default skeleton connections
edited_table = KeypointHelper.edit_default_lines(table, [0, 1, 0, 2, 1, 3])

Keypoint Visibility

When working with keypoint ground truths, 3LC uses a three-state integer channel for keypoint visibility (COCO standard):

Value

Meaning

Description

0

Not labeled

Keypoint is not annotated or its location is unknown. Keypoint coordinates are typically (0, 0)

1

Labeled but not visible

Keypoint exists but is occluded or not visible in the image

2

Labeled and visible

Keypoint is visible and annotated in the image

OKS Sigmas

Object Keypoint Similarity (OKS) sigmas define the expected spatial variance for each keypoint. They are critical for evaluating keypoint detection quality.

OKS sigmas are set at Table creation time and remain immutable throughout the Table’s lifetime. Each keypoint has an associated sigma value reflecting how precisely that keypoint can typically be localized. For example:

  • Highly visible keypoints (like eyes or nose) have smaller sigmas

  • Harder-to-localize keypoints (like hips or elbows) have larger sigmas

New derived Tables with modified sigmas can be created using the KeypointHelper.edit_oks_sigmas() method.

Training vs Evaluation

During runs, OKS sigmas serve two independent purposes:

  1. Loss Computation: Used during training to weight keypoint predictions

  2. Evaluation Metrics: Used to compute mAP and other metrics

Important

While you can use different sigma values for loss computation during training, all evaluation metrics computed in the 3LC Dashboard will always use the Table’s OKS sigmas. If sigmas are not provided, default uniform sigmas of size 1/num_keypoints will be used. This ensures consistency when comparing model performance across different Runs.

Warning

Avoid comparing evaluation metrics between Tables with different OKS sigmas, as the metrics will not be directly comparable. Always use Tables with consistent OKS sigmas when benchmarking multiple models or Runs.

Dashboard Workflows

Accepting Predictions as Ground Truth

When accepting or updating ground truths based on model predictions in the 3LC Dashboard, visibility values are automatically assigned according to the following rules:

  • Filtered-in keypoints receive visibility value 2 (visible)

  • Filtered-out keypoints receive visibility value 1 (not visible)

  • Not labeled keypoints (value 0) must be set explicitly through manual editing

Since predictions typically contain confidence values rather than visibility flags, the keypoint confidence filter is commonly used to determine which keypoints are filtered in or out. For example, setting a confidence threshold filters out low-confidence predictions, which will then receive visibility value 1 when accepted as ground truth.

Tip

Keypoint attributes including visibility can always be manually edited in the Dashboard. The automatic visibility assignment rules above only apply when using model predictions to update the ground truth set.

For more information about working with keypoints in the 3LC Dashboard, see the how-to article.

Framework Integration

3LC keypoint Tables can be used with the following frameworks:

Currently Supported

  • SuperGradients: Full integration with the YOLO-NAS pose estimation models

  • Ultralytics YOLO: Full integration with the YOLO-pose models

  • Custom PyTorch Models: Use direct Table access for custom training loops

Examples and Tutorials

Additional Resources