Per Bounding Box Luminosity Calculation#
This notebook demonstrates how to calculate the luminosity of images and their respective bounding boxes. We will write a new table combining the columns of the input table with the calculated luminosity properties.
Project Setup#
[2]:
PROJECT_NAME = "Luminosity"
DATASET_NAME = "COCO128"
TEST_DATA_PATH = "./data"
TLC_PUBLIC_EXAMPLES_DEVELOPER_MODE = True
INSTALL_DEPENDENCIES = False
[4]:
%%capture
if INSTALL_DEPENDENCIES:
%pip --quiet install torch --index-url https://download.pytorch.org/whl/cu118
%pip --quiet install torchvision --index-url https://download.pytorch.org/whl/cu118
%pip --quiet install 3lc
Imports#
Set Up Input Table#
We will use a TableFromCoco
to load the input dataset from a annotations file and a folder of images.
[9]:
table_url = tlc.Url.create_table_url(
project_name=PROJECT_NAME,
dataset_name=DATASET_NAME,
table_name="table_from_coco",
)
annotations_file = tlc.Url(TEST_DATA_PATH + "/coco128/annotations.json").to_absolute()
images_dir = tlc.Url(TEST_DATA_PATH + "/coco128/images").to_absolute()
input_table = tlc.Table.from_coco(
table_url=table_url,
annotations_file=annotations_file,
image_folder=images_dir,
description="COCO 128 dataset",
if_exists="overwrite",
)
input_table.ensure_fully_defined()
Calculate the Luminosity of Images and Bounding Boxes#
In this section, we will calculate the luminosity property for each image as well as for each bounding box within the images.
We build the variables per_image_luminosity
and per_bb_luminosity
to store the luminosity properties for each image and bounding box, respectively.
[10]:
[11]:
per_bb_luminosity: list[list[float]] = []
per_image_luminosity: list[float] = []
bb_schema = input_table.row_schema.values["bbs"].values["bb_list"]
for row in tqdm.tqdm(input_table, total=len(input_table), desc="Calculating luminosity"):
image_filename = row["image"]
image_bbs = row["bbs"]["bb_list"]
image_bytes = tlc.Url(image_filename).read()
image = Image.open(BytesIO(image_bytes))
image_luminosity = calculate_luminosity(image)
per_image_luminosity.append(image_luminosity)
bb_luminosity_list: list[float] = []
h, w = image.size
for bb in image_bbs:
bb_crop = tlc.BBCropInterface.crop(image, bb, bb_schema)
bb_luminosity = calculate_luminosity(bb_crop)
bb_luminosity_list.append(bb_luminosity)
per_bb_luminosity.append(bb_luminosity_list)
Calculating luminosity: 100%|██████████| 128/128 [00:02<00:00, 56.67it/s]
Create new Table containing luminosity properties#
After calculating the luminosity, we will create a new table using a TableWriter
.
Setup the Schema of the output Table#
[12]:
# Each entry in the list is a list of luminosity values for each bounding box in the image
per_bb_luminosity_schema = tlc.Schema(
value=tlc.Float32Value(
value_min=0,
value_max=1,
number_role=tlc.NUMBER_ROLE_FRACTION,
),
size0=tlc.DimensionNumericValue(value_min=0, value_max=1000), # Max 1000 bounding boxes
sample_type="hidden", # Hide this column when iterating over the "sample view" of the table
writable=False,
)
per_image_luminosity_schema = tlc.Schema(
value=tlc.Float32Value(
value_min=0,
value_max=1,
number_role=tlc.NUMBER_ROLE_FRACTION,
),
sample_type="hidden", # Hide this column when iterating over the "sample view" of the table
writable=False,
)
schemas = {
"per_bb_luminosity": per_bb_luminosity_schema,
"per_image_luminosity": per_image_luminosity_schema,
}
schemas.update(input_table.row_schema.values) # Copy over the schema from the input table
Write the output Table#
We will use a TableWriter
to write the output table as a TableFromParquet
.
[13]:
from collections import defaultdict
table_writer = tlc.TableWriter(
project_name=PROJECT_NAME,
dataset_name=DATASET_NAME,
description="Table with added per-bb luminosity metrics",
table_name="added_luminosity_metrics",
column_schemas=schemas,
if_exists="overwrite",
input_tables=[input_table.url],
)
# TableWriter accepts data as a dictionary of column names to lists
data = defaultdict(list)
# Copy over all rows from the input table
for row in input_table.table_rows:
for column_name, column_value in row.items():
data[column_name].append(column_value)
# Add the luminosity metrics
data["per_image_luminosity"] = per_image_luminosity
data["per_bb_luminosity"] = per_bb_luminosity
table_writer.add_batch(data)
new_table = table_writer.finalize()
Inspect the properties of the output Table#
[14]:
128
['image_id', 'image', 'width', 'height', 'bbs', 'weight', 'per_image_luminosity', 'per_bb_luminosity']
../added_luminosity_metrics
Let’s check which columns are present in the sample view / table view of the input and output tables:
[15]:
# Sample view of input table
input_table[0].keys()
[15]:
dict_keys(['image_id', 'image', 'bbs', 'width', 'height'])
[16]:
# Table view of input table
input_table.table_rows[0].keys()
[16]:
dict_keys(['image_id', 'image', 'width', 'height', 'bbs', 'weight'])
[17]:
# Sample view of output table (does not contain the luminosity columns due to the sample_type="hidden" flag)
new_table[0].keys()
[17]:
dict_keys(['image_id', 'image', 'width', 'height', 'bbs'])
[18]:
# Table view of output table (contains the luminosity columns)
new_table.table_rows[0].keys()
[18]:
dict_keys(['image_id', 'image', 'width', 'height', 'bbs', 'weight', 'per_image_luminosity', 'per_bb_luminosity'])