Images¶
Most Computer Vision applications in 3LC include at least one image column. Use
ImageSchema and pick a mode via its sample_type kwarg.
Schema¶
All four modes serialize to the same wire format — a URL string pointing at an image file. What varies is what the column accepts on write and returns on read.
Schema |
Input data |
Storage |
Sample view |
|---|---|---|---|
|
|
Saved as PNG bulk data files |
|
|
|
Saved as JPEG bulk data files |
|
|
|
Saved as WEBP bulk data files |
|
|
URL strings (paths to existing files) |
Stored inline as-is |
|
The three PIL-writing variants differ only in the encoding used when externalizing in-memory PIL images to disk; reading is format-agnostic (PIL sniffs the format from the bytes), so any PIL-decoding variant can read files written by any other.
Important
The sample_type variant is an encoder, not a re-encoder.
The variant’s format (PNG / JPEG / WEBP) only kicks in when a PIL.Image is being
saved to disk for the first time — i.e. it was constructed in memory and has no backing
file. In every other case, the variant is bypassed:
A
PIL.Imageloaded from disk (carrying afilenameor_tlc_urlattribute) is referenced verbatim. Its original file stays on disk untouched, regardless of the column’s declared format. AnImageSchema(sample_type="pil_jpeg")column does not re-encode a backed.pngto.jpeg.A
EncodedSample(pre-encoded image bytes plus an explicit extension) is written verbatim. The column’s format does not apply.Raw
bytesare written verbatim. They get the column’s file extension on disk, but no decode/re-encode happens — useEncodedSampleif extension mismatch matters.A URL string passes through unchanged.
In short: pick a variant for what you want newly-authored PIL images to be encoded as. Existing images on disk are referenced as-is.
Saving PIL Images¶
When you have PIL Image objects (e.g., from a data pipeline or augmentation), use the default
ImageSchema. The TableWriter saves each image as a file and stores
a URL reference in the row:
import tlc
from PIL import Image
writer = tlc.TableWriter(
project_name="My Project",
dataset_name="train",
schema={"image": tlc.schemas.ImageSchema()}, # PNG by default
)
writer.add_row({"image": Image.open("photo.jpg")})
table = writer.finalize()
Referencing Existing Images¶
When your images already exist on disk and you just want to point to them, pass
sample_type=None:
import tlc
table = tlc.Table.from_dict(
data={"image": ["path/to/image0.png", "path/to/image1.png"]},
schema={"image": tlc.schemas.ImageSchema(sample_type=None)},
)
This is the most common pattern for large datasets where images are already organized on disk. The Dashboard displays them directly — no copying or conversion needed.
Image Format Support
Since 3LC Tables store references to images, rather than the images themselves, any image format can be referenced in the Table. The Dashboard is however limited to displaying the formats supported by your web browser. In practice, this means that most common formats can be displayed. For a useful guideline, see the Wikipedia comparison of web browser image format support.
In addition to those supported by your web browser, 3LC supports on-the-fly transcoding of certain formats such that they can be visualized in the Dashboard. Currently, the following cases are supported:
TIFF: Single-frame TIFF images are transcoded to PNG format.
NumPy: Referenced
.npyfiles are instantiated, and when the array is 2D or 3D with a third dimension of 1 (grayscale), 3 (RGB), or 4 (RGBA) channels, it is transcoded to PNG format.
Note that in all cases, the Schema must declare that the column contains image URLs for transcoding to work. For example, transcoding is not invoked for referenced files without the Image URL role.
Exif Orientation
Exif (exchangeable image file format) is a standard for storing metadata in images. In the Dashboard, images with Exif orientation tags are always visualized in the canonical orientation, meaning rotations/flips are applied such that the images are displayed upright.
Therefore, when storing annotation columns, it is important that these (and any other data such as image dimensions) are
stored in the canonical orientation. For image dimensions, many commonly used methods like
PIL.Image.size are not based on Exif data, and will not return dimensions in the
canonical orientation for certain Exif orientations. To get image dimensions in the canonical orientation, you can use
the helper tlc.helpers.ImageHelper.get_exif_image_dimensions()
which we use internally.
As a further step it is possible to rotate/flip your images to the canonical orientation and remove the Exif orientation
tag from the file. This removes all ambiguity about the orientation of the image, and can be done using
PIL.ImageOps.exif_transpose().
Several importer methods create Tables with an image column,
notably tlc.Table.from_image_folder(),
tlc.Table.from_coco() and
tlc.Table.from_yolo_url(). The free-form methods can also be used
by declaring the image schema and providing the images as a list of URLs, like in the examples above.
Visualization¶
The Dashboard provides several ways to visualize image data.
Charts¶
To create an image chart, select the IMAGE column (click the column header),
RightClick on the IMAGE column header, and select Create 2D Chart on the popup
menu. Alternatively, you can press 2 after selecting the IMAGE column.
Viewing images in the Charts panel is useful when you want to visualize images from a single row, and is often visualized with an overlay of geometric labels or predictions, such as bounding boxes or keypoints.
Grid View¶
It is also possible to change the tabular view of the Rows panel and show the rows as a grid. This shows all of the images in a single grid, allowing you to view several at once.
Thumbnails¶
The Dashboard draws image cells much smaller than the images behind them, so a dataset or run can publish small copies and have the Object Service serve those instead of the full-resolution originals — worthwhile for large images, slow storage, or a Dashboard reached over a network link.
Thumbnails are published ahead of time with 3lc thumbnails create, per table or per run:
3lc thumbnails create "<root>/My Project/datasets/My Dataset/tables/train"
3lc thumbnails create "<root>/My Project/runs/My Run"
The same is available from Python, where tlc.Table.create_thumbnails() and
tlc.Run.create_thumbnails() publish for a single table or a whole run.
A table names itself; a run stands for every metrics table it records. Thumbnails are written beside the owning
dataset or run, under its cache/thumbnails: they share read access with the images, are removed together with their
owner, and the folder can be deleted at the cost of regenerating it.
Once published, the Object Service serves them with no further configuration — a thumbnail when the Dashboard requests a display size it covers, the original image otherwise. Thumbnails are WEBP with a longest edge of 768 pixels; a request larger than that is served the published thumbnail. See the CLI Reference for the available options.
Note
Thumbnails of aliased images need the alias registered where the Object Service runs.
A thumbnail’s location derives from the image URL in its most portable form: images named under an
alias get thumbnails that travel with the dataset, provided the Object Service can expand the
alias; images named by location get thumbnails found as long as the images stay put. 3lc thumbnails create reports
which case applies to each table and warns when one table mixes both.
Virtual Columns¶
While images are mostly used for creating charts alongside additional data, it can be useful to derive the string representation of the image as part of debugging where images are stored, why they are not being displayed or if they have aliases. Check the FAQ for how this can be used.