Instance Segmentation

Image segmentation is a fundamental computer vision task that divides images into meaningful parts by assigning labels to individual pixels. This enables machines to understand the content and structure of images at a detailed level. 3LC supports two main types of segmentation. Instance segmentation distinguishes between instances of the same class, e.g. each car gets a unique mask, enabling counting and individual object analysis. See Semantic Segmentation for the case where each pixel is classified individually.

Common Data Formats

In real-world applications, segmentation data comes in various formats. 3LC supports most common formats used in the industry, making it easy to work with existing datasets and annotations:

Input Data Type	Task	3LC Supported	Example
Binary mask per instance	Instance	✓	numpy arrays, bitmaps, ..
YOLO text files	Instance	✓	yolo-format
COCO polygon lists	Instance	✓	coco-format
COCO RLE format	Instance	✓	coco-format
Custom RLE formats	Instance	✓	E.g. (run-offset, run-length) pairs

Working with Segmentation in 3LC

The first step to working with segmentation data in 3LC is to create Tables. This can be done manually, using one of the importer Table types, or automatically through one of our integrations.

3LC provides several tutorials covering various aspects of working with segmentation data. Whether you’re starting with raw masks, COCO annotations, or custom formats, there’s a tutorial to help you get started:

Tutorial	Task	Dataset	Description
create-semantic-segmentation-table.ipynb	Semantic	ADE20K subset	Manually creates a semantic segmentation table from a set of masks
huggingface-semantic-segmentation-example.ipynb	Semantic	ADE20K subset	Fine-tunes a SegFormer model using Pytorch
pytorch-lightning-segformer.ipynb	Semantic	Balloons	Fine-tunes a SegFormer model using Pytorch Lightning
create-table-from-coco.ipynb	Instance	COCO128	Creates an instance segmentation table from a COCO annotations file
create-instance-segmentation-polygons-table.ipynb	Instance	COCO128	Manually creates a instance segmentation table from polygon lists
create-instance-segmentation-masks-table.ipynb	Instance	COCO128	Manually creates a instance segmentation table from a set of masks
create-instance-segmentations-from-masks.ipynb	Instance	LIACI Underwater Ship Inspection	Creates a instance segmentation table from multiple bitmap images
create-instance-segmentations-from-custom.ipynb	Instance	Sartorius Cell Segmentation	Creates a instance segmentation table from a custom RLE format
create-instance-segmentations-from-image-masks.ipynb	Instance	ADE20K subset	Converts a semantic segmentation table (PNG grayscale images) into a instance segmentation table
add-segmentation-column.ipynb	Instance	COCO128	Uses SAM to generate segmentation masks from images with bounding box annotations
collect-instance-segmentation-metrics.ipynb	Instance	COCO128	Collects instance segmentation masks from a pretrained Mask2Former model

Once your data is ready in 3LC, you can use it together with your favorite model and framework.

Working with Runs and Tables with segmentation data in the 3LC Dashboard is very similar to working with bounding box data, which is covered in-depth in the document on Bounding Boxes.

Supported Models and Frameworks

3LC is designed to be model and framework agnostic, allowing you to use your preferred tools with minimal effort. We provide examples and integrations with popular frameworks including:

• SAM (Segment Anything Model): State-of-the-art foundation model for segmentation

• YOLO: Popular instance segmentation framework (https://github.com/3lc-ai/3lc-ultralytics)

• Hugging Face Transformers: Access to various pre-trained models, datasets and trainers

• Detectron2: Meta’s computer vision library

Format Conversion Details

At its core, 3LC uses the COCO RLE (Run-Length Encoding) format as the internal format for storage and transfer. When data is requested in the sample-view, it can be converted to masks or polygons. Depending on the original data format, conversion might be required.

When working with segmentation data in 3LC, you should be aware of these key aspects:

• Lossless Conversions: Conversion between masks and RLE format is completely lossless

• Polygon Handling: Multi-polygon instances are automatically merged into single polygon paths

• Edge Cases: Some information loss may occur when merging multi-polygon instances, but this rarely affects training outcomes, as the model will be trained on rasterized masks

• Implementation: We use proven implementations from pycocotools and OpenCV for format conversions

• API: The SegmentationHelper class provides a set of helper functions for working with different segmentation formats

The diagram below illustrates how the data is stored and converted:

Additional Resources

For deeper understanding and reference, we recommend these resources:

• COCO Dataset: The standard benchmark for instance segmentation

• pycocotools: Essential tools for working with COCO format

• YOLO Segmentation: Real-time segmentation guide

• Hugging Face Segmentation Guide: Working with transformer-based models

Visualization

How to create a mask and image overlay chart

To create a mask-image overlay chart, select both image and mask (or segmentations) columns and create a 2D chart by pressing 2.

Editing

Instance segmentation masks are a collection of individual paint layers for an image that can overlap. This differs from semantic segmentation masks, which have a single paint layer. Thus, how you edit instance segmentation masks differs from how you edit semantic segmentation masks.

When you pull up a masks+image chart, there is a segmentation context menu at the lower left of the chart (figure below). It contains all the functionalities for editing a mask. They are, from left to right, paint, erase, erase all unselected layers when painting, delete layer, add layer, and select label. The function is active when the icon is green.

Edit an existing mask

To add paint on an existing mask, you can follow these steps:

• select the mask (click it on the chart) that you want to edit

• click the “paint” icon, if it is not active, in the segmentation context menu

• select one of the paint tools (next to the arrow icon) at the upper right corner of the Charts panel

• paint on the chart

How to select a mask underneath another one

If the mask you want to select is completely covered by another one, you can click the same spot multiple times to loop through all the masks that are under the pixel you clicked on.

To erase some paint from an existing mask, you just need to select the “erase” icon in the segmentation context menu. All other operations are the same as above.

Tips

You can press space to switch between the pan and paint mode while painting a mask, so you can easily go back and forth between painting the mask and panning the image.

Create and paint a new mask

You just need to start with clicking the “add layer” icon in the segmentation context menu and paint the newly added mask the same way as above. Note: you don’t need to select the newly added mask before painting, because it’s automatically selected right after it’s created.

When you paint a mask, the default behavior is that any pixels that belong to other masks will be erased while you are painting the currently selected mask. This ensures that a pixel can only belong to one mask. If you do not want other masks to be modified, you can click the “erase all unselected layers when painting” icon to turn it off.

Delete a mask

To delete a mask, select the mask and click the “delete layer” icon in the segmentation context menu. Alternatively, after selecting the mask, you can RightClick and then click Delete layer in the context menu or simply press Delete to delete the selected mask. You can also delete multiple masks at once by selecting them all together, which can be done by holding the Ctrl key while clicking the masks.