Per Bounding Box Luminosity Calculation¶
This notebook demonstrates how to calculate the luminosity of images and their respective bounding boxes. We will write a new table combining the columns of the input table with the calculated luminosity properties.
Project Setup¶
[2]:
PROJECT_NAME = "Luminosity"
DATASET_NAME = "COCO128"
TEST_DATA_PATH = "./data"
TLC_PUBLIC_EXAMPLES_DEVELOPER_MODE = True
INSTALL_DEPENDENCIES = False
[4]:
%%capture
if INSTALL_DEPENDENCIES:
%pip --quiet install torch --index-url https://download.pytorch.org/whl/cu118
%pip --quiet install torchvision --index-url https://download.pytorch.org/whl/cu118
%pip --quiet install 3lc
Imports¶
Set Up Input Table¶
We will use a TableFromCoco
to load the input dataset from a annotations file and a folder of images.
[9]:
table_url = tlc.Url.create_table_url(
project_name=PROJECT_NAME,
dataset_name=DATASET_NAME,
table_name="table_from_coco",
)
annotations_file = tlc.Url(TEST_DATA_PATH + "/coco128/annotations.json").to_absolute()
images_dir = tlc.Url(TEST_DATA_PATH + "/coco128/images").to_absolute()
input_table = tlc.Table.from_coco(
table_url=table_url,
annotations_file=annotations_file,
image_folder=images_dir,
description="COCO 128 dataset",
if_exists="overwrite",
)
input_table.ensure_fully_defined()
Calculate the Luminosity of Images and Bounding Boxes¶
In this section, we will calculate the luminosity property for each image as well as for each bounding box within the images.
We build the variables per_image_luminosity
and per_bb_luminosity
to store the luminosity properties for each image and bounding box, respectively.
[10]:
[11]:
per_bb_luminosity: list[list[float]] = []
per_image_luminosity: list[float] = []
bb_schema = input_table.row_schema.values["bbs"].values["bb_list"]
for row in tqdm.tqdm(input_table, total=len(input_table), desc="Calculating luminosity"):
image_filename = row["image"]
image_bbs = row["bbs"]["bb_list"]
image_bytes = tlc.Url(image_filename).read()
image = Image.open(BytesIO(image_bytes))
image_luminosity = calculate_luminosity(image)
per_image_luminosity.append(image_luminosity)
bb_luminosity_list: list[float] = []
h, w = image.size
for bb in image_bbs:
bb_crop = tlc.BBCropInterface.crop(image, bb, bb_schema)
bb_luminosity = calculate_luminosity(bb_crop)
bb_luminosity_list.append(bb_luminosity)
per_bb_luminosity.append(bb_luminosity_list)
Calculating luminosity: 100%|██████████| 128/128 [00:01<00:00, 66.45it/s]
Create new Table containing luminosity properties¶
After calculating the luminosity, we will create a new table using a TableWriter
.
Setup the Schema of the output Table¶
[12]:
# Each entry in the list is a list of luminosity values for each bounding box in the image
per_bb_luminosity_schema = tlc.Schema(
value=tlc.Float32Value(
value_min=0,
value_max=1,
number_role=tlc.NUMBER_ROLE_FRACTION,
),
size0=tlc.DimensionNumericValue(value_min=0, value_max=1000), # Max 1000 bounding boxes
sample_type="hidden", # Hide this column when iterating over the "sample view" of the table
writable=False,
)
per_image_luminosity_schema = tlc.Schema(
value=tlc.Float32Value(
value_min=0,
value_max=1,
number_role=tlc.NUMBER_ROLE_FRACTION,
),
sample_type="hidden", # Hide this column when iterating over the "sample view" of the table
writable=False,
)
schemas = {
"per_bb_luminosity": per_bb_luminosity_schema,
"per_image_luminosity": per_image_luminosity_schema,
}
schemas.update(input_table.row_schema.values) # Copy over the schema from the input table
Write the output Table¶
We will use a TableWriter
to write the output table as a TableFromParquet
.
[13]:
from collections import defaultdict
table_writer = tlc.TableWriter(
project_name=PROJECT_NAME,
dataset_name=DATASET_NAME,
description="Table with added per-bb luminosity metrics",
table_name="added_luminosity_metrics",
column_schemas=schemas,
if_exists="overwrite",
input_tables=[input_table.url],
)
# TableWriter accepts data as a dictionary of column names to lists
data = defaultdict(list)
# Copy over all rows from the input table
for row in input_table.table_rows:
for column_name, column_value in row.items():
data[column_name].append(column_value)
# Add the luminosity metrics
data["per_image_luminosity"] = per_image_luminosity
data["per_bb_luminosity"] = per_bb_luminosity
table_writer.add_batch(data)
new_table = table_writer.finalize()
Inspect the properties of the output Table¶
[14]:
128
['image_id', 'image', 'width', 'height', 'bbs', 'weight', 'per_image_luminosity', 'per_bb_luminosity']
../added_luminosity_metrics
Let’s check which columns are present in the sample view / table view of the input and output tables:
[15]:
# Sample view of input table
input_table[0].keys()
[15]:
dict_keys(['image_id', 'image', 'bbs', 'width', 'height'])
[16]:
# Table view of input table
input_table.table_rows[0].keys()
[16]:
dict_keys(['image_id', 'image', 'width', 'height', 'bbs', 'weight'])
[17]:
# Sample view of output table (does not contain the luminosity columns due to the sample_type="hidden" flag)
new_table[0].keys()
[17]:
dict_keys(['image_id', 'image', 'width', 'height', 'bbs'])
[18]:
# Table view of output table (contains the luminosity columns)
new_table.table_rows[0].keys()
[18]:
dict_keys(['image_id', 'image', 'width', 'height', 'bbs', 'weight', 'per_image_luminosity', 'per_bb_luminosity'])