Per Bounding Box Luminosity Calculation#

This notebook demonstrates how to calculate the luminosity of images and their respective bounding boxes. We will write a new table combining the columns of the input table with the calculated luminosity properties.

Info: This notebook demonstrates a technique for adding columns to an existing Table. While this is a useful technique, there are some drawbacks to this approach:

  • The new Table will not be part of the lineage of the input Table.

  • The new Table will contain a literal copy of all data in the input Table

In a future release of 3LC, adding columns to an existing Table will be supported natively.

Project Setup#

[2]:
PROJECT_NAME = "Luminosity"
DATASET_NAME = "COCO128"
TEST_DATA_PATH = "./data"
TLC_PUBLIC_EXAMPLES_DEVELOPER_MODE = True
INSTALL_DEPENDENCIES = False
[4]:
%%capture
if INSTALL_DEPENDENCIES:
    %pip --quiet install torch --index-url https://download.pytorch.org/whl/cu118
    %pip --quiet install torchvision --index-url https://download.pytorch.org/whl/cu118
    %pip --quiet install tlc

Imports#

[8]:
from __future__ import annotations

from io import BytesIO

import tqdm
import numpy as np
from PIL import Image

import tlc

Set Up Input Table#

We will use a TableFromCoco to load the input dataset from a annotations file and a folder of images.

[9]:
table_url = tlc.Url.create_table_url(
    project_name=PROJECT_NAME,
    dataset_name=DATASET_NAME,
    table_name="table_from_coco",
)

annotations_file = tlc.Url(TEST_DATA_PATH + "/coco128/annotations.json").to_absolute()
images_dir = tlc.Url(TEST_DATA_PATH + "/coco128/images").to_absolute()

input_table = tlc.Table.from_coco(
    table_url=table_url,
    annotations_file=annotations_file,
    image_folder=images_dir,
    description="COCO 128 dataset",
    if_exists="overwrite",
)

input_table.ensure_fully_defined()

Calculate the Luminosity of Images and Bounding Boxes#

In this section, we will calculate the luminosity property for each image as well as for each bounding box within the images.

We build the variables per_image_luminosity and per_bb_luminosity to store the luminosity properties for each image and bounding box, respectively.

[10]:
def calculate_luminosity(image: Image) -> float:
    np_image = np.array(image)
    axes_to_reduce = tuple(range(np_image.ndim - 1))
    avg_luminosity = np.mean(np_image, axis=axes_to_reduce) / 255.0
    return float(np.mean(avg_luminosity))
[11]:
per_bb_luminosity: list[list[float]] = []
per_image_luminosity: list[float] = []

bb_schema = input_table.row_schema.values["bbs"].values["bb_list"]

for row in tqdm.tqdm(input_table.table_rows, total=len(input_table), desc="Calculating luminosity"):
    image_filename = row["image"]
    image_bbs = row["bbs"]["bb_list"]

    image_bytes = tlc.Url(image_filename).read()
    image = Image.open(BytesIO(image_bytes))

    image_luminosity = calculate_luminosity(image)
    per_image_luminosity.append(image_luminosity)

    bb_luminosity_list: list[float] = []
    h, w = image.size

    for bb in image_bbs:
        bb_crop = tlc.BBCropInterface.crop(image, bb, bb_schema)
        bb_luminosity = calculate_luminosity(bb_crop)
        bb_luminosity_list.append(bb_luminosity)

    per_bb_luminosity.append(bb_luminosity_list)
Calculating luminosity: 100%|██████████| 128/128 [00:02<00:00, 57.21it/s]

Create new Table containing luminosity properties#

After calculating the luminosity, we will create a new table using a TableWriter.

Setup the Schema of the output Table#

[12]:
# Each entry in the list is a list of luminosity values for each bounding box in the image
per_bb_luminosity_schema = tlc.Schema(
    value=tlc.Float32Value(
        value_min=0,
        value_max=1,
        number_role=tlc.NUMBER_ROLE_FRACTION,
    ),
    size0=tlc.DimensionNumericValue(value_min=0, value_max=1000),  # Max 1000 bounding boxes
    sample_type="hidden",  # Hide this column when iterating over the "sample view" of the table
    writable=False,
)

per_image_luminosity_schema = tlc.Schema(
    value=tlc.Float32Value(
        value_min=0,
        value_max=1,
        number_role=tlc.NUMBER_ROLE_FRACTION,
    ),
    sample_type="hidden",  # Hide this column when iterating over the "sample view" of the table
    writable=False,
)

schemas = {
    "per_bb_luminosity": per_bb_luminosity_schema,
    "per_image_luminosity": per_image_luminosity_schema,
}
schemas.update(input_table.row_schema.values)  # Copy over the schema from the input table

Write the output Table#

We will use a TableWriter to write the output table as a TableFromParquet.

[13]:
from collections import defaultdict

table_writer = tlc.TableWriter(
    project_name=PROJECT_NAME,
    dataset_name=DATASET_NAME,
    description="Table with added per-bb luminosity metrics",
    table_name="added_luminosity_metrics",
    column_schemas=schemas,
    if_exists="overwrite",
)

# TableWriter accepts data as a dictionary of column names to lists
data = defaultdict(list)

# Copy over all rows from the input table
for row in input_table.table_rows:
    for column_name, column_value in row.items():
        data[column_name].append(column_value)

# Add the luminosity metrics
data["per_image_luminosity"] = per_image_luminosity
data["per_bb_luminosity"] = per_bb_luminosity

table_writer.add_batch(data)
new_table = table_writer.finalize()

Inspect the properties of the output Table#

[14]:
print(len(new_table))
print(new_table.columns)
print(new_table.url.to_relative(input_table.url))
128
['image_id', 'image', 'width', 'height', 'bbs', 'weight', 'per_image_luminosity', 'per_bb_luminosity']
../added_luminosity_metrics

Let’s check which columns are present in the sample view / table view of the input and output tables:

[15]:
# Sample view of input table
input_table[0].keys()
[15]:
dict_keys(['image_id', 'image', 'bbs', 'width', 'height'])
[16]:
# Table view of input table
input_table.table_rows[0].keys()
[16]:
dict_keys(['image_id', 'image', 'width', 'height', 'bbs', 'weight'])
[17]:
# Sample view of output table (does not contain the luminosity columns due to the sample_type="hidden" flag)
new_table[0].keys()
[17]:
dict_keys(['image_id', 'image', 'width', 'height', 'bbs'])
[18]:
# Table view of output table (contains the luminosity columns)
new_table.table_rows[0].keys()
[18]:
dict_keys(['image_id', 'image', 'width', 'height', 'bbs', 'weight', 'per_image_luminosity', 'per_bb_luminosity'])