Computer Vision

Computer Vision applications usually involve working with a column of images and a column of spatial data that describe shapes or regions of interest, with additional data for each such shape or region. These may include axis-aligned or oriented bounding boxes, point clouds, poses, segmentation masks, polygon lists, and more. 3LC provides a flexible format for storing and working with these annotation types, which can be used both in the 3LC Dashboard and in downstream machine learning workflows.

3LC supports representing and working with a wide range of computer vision data types:

• Overview
• Images
• Bounding Boxes
• Oriented Bounding Boxes (OBBs)
• Keypoints and Pose Estimation
• Instance Segmentation
• Semantic Segmentation

Data Layout

You don’t need to understand the internal data structure to effectively use geometric data in 3LC. However, knowing the details can be helpful if you have advanced or custom requirements.

A geometry column is any composite column that has the following top-level attributes:

{
  "instances": [], # list of instances
  "instances_additional_data": {}, # mapping from extra name to array of additional data for each instance
  "x_min": 0, # minimum x coordinate of the scene
  "x_max": 1, # maximum x coordinate of the scene
  "y_min": 0, # minimum y coordinate of the scene
  "y_max": 1, # maximum y coordinate of the scene
  "z_min": 0, # minimum z coordinate of the scene (optional)
  "z_max": 1, # maximum z coordinate of the scene (optional)
}

A single geometry instance may contain the following values:

{
  "vertices_{2d,3d}": [], # list of xy(z) coordinates,
  "lines": [], # list of vertex-pairs,
  "triangles": [], # list of vertex-triples,
  "vertices_{2d,3d}_additional_data": {}, # mapping from extra name to list of additional data for each vertex,
  "lines_additional_data": {}, # mapping from extra name to list of additional data for each line,
  "triangles_additional_data": {}, # mapping from extra name to list of additional data for each triangle,
  "bbs_{2d,3d}": [], # list of bbs [x_min, y_min, x_max, y_max] absolute coordinates
  "oriented_bbs_{2d,3d}": [], # list of oriented bbs [cx, cy, w, h, rotation], or [cx, cy, cz, w, h, d, yaw, pitch, roll] absolute coordinates with angles in radians
}

The vertices_{2d,3d}_additional_data, lines_additional_data, triangles_additional_data, and instances_additional_data attributes are optional and can be used to store additional metadata for each vertex, line, triangle, or instance. For example, you can store the visibility of each vertex, the name of each line, the role of each triangle, or the label of each instance.

The Geometry2DSchema and Geometry3DSchema classes are used to define the schema of the geometry data. This is where you can provide additional metadata such as default skeleton topologies, keypoint names, etc.

The classes Geometry2DInstances and Geometry3DInstances are used to store geometry data in 3LC. They are utilized to convert geometry data to and from the Table format.