Semantic Segmentation

Image segmentation is a fundamental computer vision task that divides images into meaningful parts by assigning labels to individual pixels. This enables machines to understand the content and structure of images at a detailed level.

3LC supports two main types of segmentation. Semantic segmentation assigns class labels to pixels (e.g., all ‘car’ pixels get same label), focusing on what object types are present in the image and where. See Instance Segmentation for instance-level segmentation.

Schema

Use SemanticSegmentationSchema to define a mask column. The classes parameter provides the class map that the Dashboard uses for visualization and editing:

import tlc
from PIL import Image

# Saving mask PIL Images (stored as bulk data PNG files)
writer = tlc.TableWriter(
    project_name="Segmentation Project",
    dataset_name="train",
    schema={
        "image": tlc.schemas.ImageSchema(),
        "mask": tlc.schemas.SemanticSegmentationSchema(classes=["background", "car", "person"]),
    },
)

for img, mask in my_data:
    writer.add_row({"image": img, "mask": mask})

table = writer.finalize()

# Referencing existing mask files on disk
table = tlc.Table.from_dict(
    data={
        "image": ["img_0.png", "img_1.png"],
        "mask": ["mask_0.png", "mask_1.png"],
    },
    schema={
        "image": tlc.schemas.ImageSchema(sample_type=None),
        "mask": tlc.schemas.SemanticSegmentationSchema(classes=["background", "car", "person"]),
    },
)

Masks are stored as single-channel grayscale PNGs where each pixel value is a class index.

Visualization and Editing

How to edit a semantic segmentation mask

Mask editing utilizes the polygon selection tools in the Chart panel to paint on the mask. In order to edit a mask, a mask chart needs to be created first. Creating a mask and image overlay chart is similar as creating a BB and image overlay chart. Select the Mask and Image columns and press 2 to create the overlay chart. If you just want to create a mask chart without the image, you just select the Mask column and press 2.

When using a polygon selection tool on a mask chart, the polygon selection tool will be served as mask editing.

To start editing a mask, select one of the polygon selection tools such as lasso, select a desired label (balloon in this example), and then start drawing the lasso polygon to trace the balloon on the image.

To select a desired label, there are three options:

• Click the label name next to the MASK context menu on the lower left corner of the chart

• Press K or L to go to the previous or next label until the desired one is reached

• Press J when the mouse cursor hovers over an area of the desired label on the mask

The lasso polygon area will be painted as the color of Class balloon once the lasso drawing is complete. You may want to finetune the edges of the newly painted mask, if they are not perfect, by zooming in and repeating the same painting process.

Accept predictions

You can convert a model’s predicted mask to a ground truth mask. To do this, first create an image + predicted mask chart, then RightClick to bring up the context menu and click on Immediately accept predictions this row or press I to accept the current sample’s predicted mask as the ground truth. The ground truth mask will be updated to be the same as the predicted one.

You can accept the predicted masks for all filtered-in samples at once by dragging the Predicted_mask column header to the Mask column header.