Per Bounding Box Luminosity Calculation

Open in GitHub

This notebook demonstrates how to calculate the luminosity of images and their respective bounding boxes. We will write a new table combining the columns of the input table with the calculated luminosity properties.

Info: This notebook demonstrates a technique for adding columns to an existing Table. While this is a useful technique, there are some drawbacks to this approach:

• The new Table will not be part of the lineage of the input Table.

• The new Table will contain a literal copy of all data in the input Table

In a future release of 3LC, adding columns to an existing Table will be supported natively.

Project Setup

PROJECT_NAME = "Luminosity"
DATASET_NAME = "COCO128"
TEST_DATA_PATH = "./data"
TLC_PUBLIC_EXAMPLES_DEVELOPER_MODE = True
INSTALL_DEPENDENCIES = False

%%capture
if INSTALL_DEPENDENCIES:
    %pip --quiet install torch --index-url https://download.pytorch.org/whl/cu118
    %pip --quiet install torchvision --index-url https://download.pytorch.org/whl/cu118
    %pip --quiet install 3lc

Imports

from __future__ import annotations

from io import BytesIO

import tqdm
import numpy as np
from PIL import Image

import tlc

Set Up Input Table

We will use a TableFromCoco to load the input dataset from a annotations file and a folder of images.

table_url = tlc.Url.create_table_url(
    project_name=PROJECT_NAME,
    dataset_name=DATASET_NAME,
    table_name="table_from_coco",
)

annotations_file = tlc.Url(TEST_DATA_PATH + "/coco128/annotations.json").to_absolute()
images_dir = tlc.Url(TEST_DATA_PATH + "/coco128/images").to_absolute()

input_table = tlc.Table.from_coco(
    table_url=table_url,
    annotations_file=annotations_file,
    image_folder=images_dir,
    description="COCO 128 dataset",
    if_exists="overwrite",
)

input_table.ensure_fully_defined()

Calculate the Luminosity of Images and Bounding Boxes

In this section, we will calculate the luminosity property for each image as well as for each bounding box within the images.

We build the variables per_image_luminosity and per_bb_luminosity to store the luminosity properties for each image and bounding box, respectively.

def calculate_luminosity(image: Image) -> float:
    np_image = np.array(image)
    axes_to_reduce = tuple(range(np_image.ndim - 1))
    avg_luminosity = np.mean(np_image, axis=axes_to_reduce) / 255.0
    return float(np.mean(avg_luminosity))

per_bb_luminosity: list[list[float]] = []
per_image_luminosity: list[float] = []

bb_schema = input_table.row_schema.values["bbs"].values["bb_list"]

for row in tqdm.tqdm(input_table.table_rows, total=len(input_table), desc="Calculating luminosity"):
    image_filename = row["image"]
    image_bbs = row["bbs"]["bb_list"]

    image_bytes = tlc.Url(image_filename).read()
    image = Image.open(BytesIO(image_bytes))

    image_luminosity = calculate_luminosity(image)
    per_image_luminosity.append(image_luminosity)

    bb_luminosity_list: list[float] = []
    h, w = image.size

    for bb in image_bbs:
        bb_crop = tlc.BBCropInterface.crop(image, bb, bb_schema)
        bb_luminosity = calculate_luminosity(bb_crop)
        bb_luminosity_list.append(bb_luminosity)

    per_bb_luminosity.append(bb_luminosity_list)

Calculating luminosity: 100%|██████████| 128/128 [00:02<00:00, 58.06it/s]

Create new Table containing luminosity properties

After calculating the luminosity, we will create a new table using a TableWriter.

Setup the Schema of the output Table

# Each entry in the list is a list of luminosity values for each bounding box in the image
per_bb_luminosity_schema = tlc.Schema(
    value=tlc.Float32Value(
        value_min=0,
        value_max=1,
        number_role=tlc.NUMBER_ROLE_FRACTION,
    ),
    size0=tlc.DimensionNumericValue(value_min=0, value_max=1000),  # Max 1000 bounding boxes
    sample_type="hidden",  # Hide this column when iterating over the "sample view" of the table
    writable=False,
)

per_image_luminosity_schema = tlc.Schema(
    value=tlc.Float32Value(
        value_min=0,
        value_max=1,
        number_role=tlc.NUMBER_ROLE_FRACTION,
    ),
    sample_type="hidden",  # Hide this column when iterating over the "sample view" of the table
    writable=False,
)

schemas = {
    "per_bb_luminosity": per_bb_luminosity_schema,
    "per_image_luminosity": per_image_luminosity_schema,
}
schemas.update(input_table.row_schema.values)  # Copy over the schema from the input table

Write the output Table

We will use a TableWriter to write the output table as a TableFromParquet.

from collections import defaultdict

table_writer = tlc.TableWriter(
    project_name=PROJECT_NAME,
    dataset_name=DATASET_NAME,
    description="Table with added per-bb luminosity metrics",
    table_name="added_luminosity_metrics",
    column_schemas=schemas,
    if_exists="overwrite",
)

# TableWriter accepts data as a dictionary of column names to lists
data = defaultdict(list)

# Copy over all rows from the input table
for row in input_table.table_rows:
    for column_name, column_value in row.items():
        data[column_name].append(column_value)

# Add the luminosity metrics
data["per_image_luminosity"] = per_image_luminosity
data["per_bb_luminosity"] = per_bb_luminosity

table_writer.add_batch(data)
new_table = table_writer.finalize()

Inspect the properties of the output Table

print(len(new_table))
print(new_table.columns)
print(new_table.url.to_relative(input_table.url))

128
['image_id', 'image', 'width', 'height', 'bbs', 'weight', 'per_image_luminosity', 'per_bb_luminosity']
../added_luminosity_metrics

Let’s check which columns are present in the sample view / table view of the input and output tables:

# Sample view of input table
input_table[0].keys()

dict_keys(['image_id', 'image', 'bbs', 'width', 'height'])

# Table view of input table
input_table.table_rows[0].keys()

dict_keys(['image_id', 'image', 'width', 'height', 'bbs', 'weight'])

# Sample view of output table (does not contain the luminosity columns due to the sample_type="hidden" flag)
new_table[0].keys()

dict_keys(['image_id', 'image', 'width', 'height', 'bbs'])

# Table view of output table (contains the luminosity columns)
new_table.table_rows[0].keys()

dict_keys(['image_id', 'image', 'width', 'height', 'bbs', 'weight', 'per_image_luminosity', 'per_bb_luminosity'])