How to work on pose/keypoints

Pose and keypoints refer to ways of describing the position and orientation of objects, and the terms may be used interchangeably for 3LC. A set of predefined keypoints, lines, and triangles can be displayed and edited in the Dashboard. A bounding box is often a part of a pose set. Individual elements, such as keypoints, may not be added or removed, but adding or removing an entire pose object (including all keypoints, lines, BB) is allowed.

See also the Keypoints tutorial for more details.

Create an image+keypoints chart

Creating an image+keypoints chart is essentially the same as creating an image+BB chart. Select the IMAGE and KEYPOINTS columns and press 2.

Display keypoint properties

When clicking on a keypoint on a chart, all available properties of the keypoint, such as the vertex role and visibility, will be displayed next to it. To select multiple keypoints, you can either hold Ctrl while clicking the keypoints or use one of the polygon selection tools to select the keypoints of interest.

The BBs(instances)/keypoints/lines/triangles have their own properties. The display of these properties can be changed in the context menu as shown below. The ways to set those display properties are the same as those for BBs. Learn more in this page.

Edit pose/keypoints

Each keypoint is usually represented by (x,y,v) format. x and y are the 2D coordinates, and v is the visibility. You can edit (x,y) coordinates by moving the keypoint on a keypoints chart and change the visibility on the context menu in the chart. When the keypoints are moved, the associated lines will be moved automatically.

To edit a pose or a keypoint, here are what you can do:

  • To move a pose, click and drag the BB

  • To resize a pose, click and drag on an edge or a corner of the BB (all keypoints will also be resized proportionally)

  • To move a keypoint, click and drag that keypoint

  • To edit the visibility of a keypoint, click the keypoint and then change the visibility in the context menu (as shown below)

Work on a pose/keypoints Run

There are a lot of similarities between working on a BB Run and a pose/keypoints Run. Some standard metrics, such as predicted pose/keypoints, loss, and confidence, are collected with 3lc-ultralytics during training. Below is an example showing the collected metrics in the table and two charts with ground truth and predicted pose/keypoints, respectively. Similar to BB detection, the ground truth is presented with solid lines, while the predictions use dashed lines. Note that there are two confidence columns in the table. One is for the BB predictions, while the other is for each individual keypoint.

Similar to IOU for BBs, you can derive OKS (Object Keypoint Similarity) virtual columns for the ground truth and the predictions. In addition, you can also derive per-keypoint OKS virtual columns. The former is a standard OKS that calculates the average of all visible keypoints’ OKS for a pose, while the latter presents a value for each keypoint. The context menu shown below can be brought up by selecting the ground truth and predicted keypoint columns and RightClick on one of the column headers.

In a keypoints chart, the OKS metric will be displayed below the BB, and the per-keypoint OKS will be next to the keypoint.

In derived TP/FP/FN virtual columns, the TP/FP/FN counts are based on OKS in a similar fashion as IOU for BBs. You can adjust the OKS threshold (default 0.5) for such counts.

Filter on pose/keypoints

The main difference between BBs and pose/keypoints is that a pose consists of a BB, a set of keypoints, and a set of lines. Therefore, the filters for pose/keypoints have some additions on top of standard BB filters. For pose/keypoints, you can filter on any metrics or properties associated with BBs, keypoints or lines. For example, if you filter on a BB metric such as the predicated BB confidence, the BBs and their associated keypoints/lines will be filtered all together. If you filter on a keypoint metric or property such as the vertex role, the filter will only apply to the keypoints. Other than that, filters for pose/keypoints are pretty much the same as those for BBs.

Convert predicted pose/keypoints

All the operations related to converting predicted pose/keypoints to the ground truth are the same with converting predicted BBs. Learn more on this page.