3LC Dashboard#

The 3LC Dashboard is where you will spend most of your time when using 3LC. The Dashboard lets you visualize your Tables and Runs, query your data based on filters, and modify samples within your dataset, all from your browser.

The following sections outline various aspects of the Dashboard: an overview of the three panels, operations you can apply to your data, keyboard shortcuts, and more.

First we will explore one possible way to use 3LC with an example - finding and fixing labeling issues in an object detection dataset.

Example#

For this example, we have trained an object detection model on 118k images from the COCO dataset. While 3LC can be used to collect metrics throughout the training process, for this example we have only collected metrics after training has finished.

The Run will appear on the front page of the 3LC Dashboard, under the Runs tab. <DoubleClick> on this row to open this Run in the Dashboard.

Each row of the now opened Run represents one inference pass over one sample of the COCO dataset. Each row contains data about the sample itself, like image and bbs (ground truth bounding boxes), but also data collected using our model, like bbs (predicted bounding boxes) and loss.

In your code, you can specify exactly which metrics you want 3LC to collect. Let’s try to find some labeling issues in the COCO dataset.




Most columns can be filtered on using the Filters panel. For numeric values, you can specify a value range to only show rows where the value falls within this range. Filtering on multiple columns at once narrows the search further. Let’s filter on rows which contain at least one false positive and at least one false negative.

We can now see that the 118,287 rows of the dataset have been reduced to 43,056 rows which contain both false positives and false negatives. These rows represent difficult samples for the model, but it’s hard to see obvious labelling issues.

Let’s try sorting the rows by the model’s confidence in its predictions. Since we don’t have a “confidence” column to sort by, we need to extract the confidence values trapped inside the “Predicted bounding boxes” column into a Virtual column which we can sort by. To do this, <RightClick> on the predicted bounding box column, hover over Create virtual column, then Bounding boxes, then confidence, then click Min.

A new column will appear, containing the lowest confidence out of all predictions for each image. By clicking the arrow next to the name of this column once, we can sort the rows by this minimum confidence in ascending order. Clicking it again sorts the rows in descending order.

The rows at the top of this table are samples where model predictions did not align with the labels (as known from the false positives and false negatives) but where the model was nonetheless very confident in its predictions. Flicking through these images, either by clicking the rows or using the arrow keys, we see a high concentration of labelling issues.

When looking at a mislabeled sample, we can either manually edit, remove, or add bounding boxes, or we can simply accept what the model predicted to be the new ground truth.


Once we have cleaned up our dataset and are ready to run training again with our changes, we press the green checkmark in the top-left corner to commit our changes and create a new dataset revision. The next time we run our training, the changes we made in the Dashboard will automatically be reflected in the data passed to the model. With improved labels, retraining will likely result in a better model, which can again be used for further dataset improvements.

Panels#

Most pages in the 3LC Dashboard are split into three panels: Rows, Filters, and Charts. These panels are connected, and changes within one of them will be reflected in the others. The panels can be resized by dragging the boundaries between them, and they can be hidden by pressing the “X” in their top right corner. They can also be toggled on and off using the three buttons at the top left of the Dashboard:

Rows panel#

The Rows panel is a comprehensive view of your tabular data, divided into rows and columns. The exact contents of those rows and columns will depend on which part of the Dashboard is currently being used. Depending on the context, each row in the Rows panel might represent a revision of a dataset, a Run, a sample in a dataset, or an inference pass over a sample.

Rows can be manually selected by <LeftClick>’ing on them. Alternatively, use <Ctrl><LeftClick> to select multiple rows at once. With one row selected, <Shift><LeftClick> another row to select all rows between them.

At the top of the Rows panel, you can see how many rows are currently selected at any time, as well as how many rows match the current filter criteria and hence are shown in the table.

Certain columns, such as weight, are editable. <LeftClick> an editable cell to change its value, with the exact editing mechanism depending on the type of data in the column. If you have multiple rows selected when making edits, the edits will apply to every selected row.

You can change which columns are shown in the Rows panel at any time. If a column is not useful to what you are doing right now, <RightClick> on it and click Hide selected column. The visibility of all available columns can be independently toggled on or off by clicking the Wrench symbol on the top bar of the Rows panel.

You can also create new Virtual columns, derived from a selection of existing columns. To do this, make a selection of one more columns, <RightClick>, and select Create virtual column. You will be presented with a list of Operations which can be performed on your selection to produce a new virtual column. The exact operations available will depend on the type of data in the selected columns. For example, if you select two columns containing integers, you will be able to create a new column containing the Sum of the two columns.

It is also possible to change the tabular view and show the rows as a grid, which can be useful for image datasets:

Filters panel#

The Filters panel lets you define which rows you want to show in the Rows panel. In addition to a general “Text filter” to search for specific text across all values within a row, each column in the Rows panel has its own widget in the Filters panel.

The type of widget depends on the type of data in the column. For example, a column containing numbers will have a range slider widget. These slider widgets also show a histogram of the distribution of values in the column. As you use other filters to narrow down the rows shown in the rows panel, the coloring of this histogram will update to show the distribution of values in the remaining rows. When multiple filter widgets are being used, only rows satisfying all of their constraints are shown. In other words, the filters are AND’ed together.

Filter widgets can be collapsed by clicking the arrow in the top right corner of the widget. This will hide the widget, but it will still be applied to the Rows panel and the views within the Charts panel that refer to this Table. The button next to this arrow resets the widget to its default state.

Another Reset button (at the very top of the Filters panel) resets all widgets to their default state, unless they have been locked using the button in their top right corner. Locked widgets will not be reset when the “Reset all” button is pressed.

For any filter setting, you can press the button at the top of the Filters panel labeled Create subset table to create a new table with exactly the rows that are currently filtered in. This new Table will be added to the Rows, and can be used in the same way as any other table. Also, a new boolean Virtual column will be created in your original table; the value for each row corresponds to whether or not it is included in the new subset Table.

Charts panel#

The Charts panel lets you visualize the data in the Rows panel. In most contexts, the Charts panel will initially be empty by default. To create charts, select one or more columns, <RightClick> and select Create 2D chart or Create 3D chart. If you select only one column with a single value per row and try to create a 2D chart, the values will be plotted against the sample_id column. Which columns are assigned to the X-, Y and Z-axes is determined by the order in which you select them.

If you have three columns selected when creating a 2D chart, the third column will be used to colorize the points in the chart. Similarly, with four columns selected when creating a 3D chart, the fourth column will be used to color the points.

To display an image in the charts panel, select an image column and create a 2D chart. If you have a column of bounding boxes selected along with an image column when creating the 2D chart, the bounding boxes will be overlaid on the image.

Front Page#

The front page lets you switch between three main tabs: Projects, Runs, and Tables.

Projects Tab

Projects Tab#

The dashboard starts in the Projects tab to let the user select a Project, after which the other tabs are populated with the corresponding Run and Tables data. Note that the (Any project)-project allows viewing all runs and tables at the same time.

The currently active project is shown at the center in the top tool bar.

Tables Tab#

The available Tables for the currently selected project and filter settings are shown in the Tables tab. Each row in the Rows panel represents a revision of a dataset registered with 3LC, which is an individual Table in 3LC, and by default shows the following columns:

  • Favorite

  • Lineage

  • Url

  • Dataset name

  • Location

  • Row count

  • Created

Tables Tab

Table Revisions and Lineage

When the user edits or otherwise makes a modification to a Table, 3LC creates a new revision which is itself a Table. The system keeps track of revisions and shows this connectivity as directed edges in the lineage column of the Table rows view. This makes it easy to navigate even a complex dataset with many revisions and branching lineage.

Tip: To open a specific Table revision in a notebook simply locate the revision in the Tables tab, hover the URL column to enable the copy-action, perform the copy and then simply do:

table = Table.from_url(...)

Open a Table#

<DoubleClick> a row will open a dedicated Table view in a tab called Examples. See the Views section for more info.

Runs Tab#

To give an overview, the Runs tab shows a set of charts with aggregate metrics for all the runs in the project in the Charts panel. Like in the Tables tab, the Runs Tab shows all the Runs for the current project in the Rows panel. The following columns are available for Runs:

  • Show

  • Url

  • Description

  • Status

  • Last modified

Just as with Tables, you can open a Run by <DoubleClick>’ing on it.

Whenever you run your Python code with 3LC metrics collection, a new Run will appear as a new row in the Runs panel.

Filtering Projects, Tables, and Runs#

Under Projects, Runs, and Tables you can filter on values associated with each row, using the Filters panel. Creating Virtual columns based on the Constants column is particularly useful for Runs, to find Runs with a particular set of hyperparameters, to sort based on a metric, or to e.g. visualize how loss varies across epoch.

<DoubleClick> a Run or a Table to open it in a new View.

Views#

Opening a specific Run or Table will create a new view in a dedicated tab called Examples or Metrics-....

Run#

When opening a Run you will see the same three panels as described earlier, with a rows panel named Metrics. When a 3LC Table was used as input to produce the metrics, that input data is joined and shown alongside the all the metrics data for the Run. This is at the core of the 3LC workflow, allowing you to make improvements to your data based on the model predictions and metrics.

Table#

When opening a Table you will see an instantiation of a particular revision of your dataset. Just like in a Run, you can modify your samples directly.

Operations#

Operations are functions that can be applied to one or more columns of a table to produce new Virtual columns. Filtering and sorting on these virtual columns can give you deeper insights into your model and training set and help you find issues with your data, your model, or perhaps your entire approach to the problem at hand. The exact operations available will depend on the type of data in the selected columns. In general we can divide operations into two main categories: Global operations and Local operations.

Global operations#

Global operations need the full context of every row of its input columns to produce a result for any individual row. The Occurrences operation, which counts the number of times each value(s) occur in one or more columns, is an example of a global operation. A complete list of the available global operations, along with their input requirements, can be seen below.

Operation

Input Type

Output type

Description

Traversal index

Any

Number

Traversal index maximizing the walk within the input column coordinate space

Rank

Any

Number

The rank of each row, as sorted by the input value(s)

Group

Any

Same as input

Group index shared by all rows with the same value(s) in the input column(s)

Nearest neighbor

Numbers

Number

Distance to closest neighbor within the input column coordinate space

Derivative

Number

Number

For a numeric input column evolving over time, this operation returns the difference between the next value and the previous one, divided by two (and handling first and last values appropriately).

Deviation

Number

Number

For a numeric input column evolving over time, this operation returns the difference between a value and the average between the next and previous one.

From previous

Number

Number

For a numeric input column evolving over time, this operation returns the difference between this value and the previous one.

To next

Number

Number

For a numeric input column evolving over time, this operation returns the difference between the next value and the current one.

Occurrence

Any

Number

Number of columns with the same value(s) in the input column(s)

Foreign table row

Number

Any

The entire row of a foreign table, as referenced by a foreign key in the input table row

Local operations#

Local operations only need the context of a single row to produce a result. We can further subdivide local operations into three categories: Unary operations, Order-independent operations, and Order-dependent operations.

Unary operations#

Unary operations take a single column as input, and produce a result for each row in that column. The Character count operation, which returns the length of a string, is an example of a unary operation. A complete list of the available unary operations, along with their input requirements, can be seen below.

Operation

Input Type

Output type

Description

Pick random

Any

Any

A random element picked from the input array (if any)

Abs

Number

Number

The absolute value of the numeric input value

Character count

String

Number

The number of characters in the input string (including whitespace).

Log

Number

Number

The log value of the numeric input value

Non-zero

Numbers

Boolean

Whether the numeric input value is non-zero

Not

Any

Boolean

The boolean ‘not’ of the input value(s)

Sign

Number

Number

The sign (i.e. -1, 0, or 1) of the input value(s).

Raw

Any

Same as input

The raw value of the input property, i.e. with value maps and/or string roles removed from the schema (recursive when required)

Word count

String

Number

The number of words in the input string

Zero

Number

Boolean

Whether the numeric input value is zero

Area

Bounding boxes

Numbers

The area of each bounding box within the input bounding box list

Week since epoch

Datetime string

Number

For input datetime string, returns the week since epoch.

Hour of day

Datetime string

Number

For input datetime string, returns the hour [0..23].

Day of week

Datetime string

Number

For input datetime string, returns the day of week.

Milliseconds since epoch

Datetime string

Number

For an input datetime string, returns the number of milliseconds since epoch.

Default boolean value

None

Any

Constant bool value

Default integer value

None

Any

Constant int32 value

Default floating-point value

None

Any

Constant float32 value

Default string value

None

Any

Constant string value

Order-independent operations#

Order-independent operations take two or more columns as input, and produce a result for each row in those columns. The order of the input columns does not matter. The Sum operation, which adds two or more columns together, is an example of an order-independent operation. A complete list of the available order-independent operations, along with their input requirements, can be seen below.

Operation

Input Type

Output type

Description

Sum

Numbers

Number

The sum of all numeric input values

Average

Numbers

Number

The arithmetic average of the numeric input values.

Common

Any

Any

The common value within the input array (if any)

Count

String

Number

The number of elements within the input array.

Length

Numbers

Number

The length of the input values (i.e. the square root of the sum of the squares of the input values)

Equal

Any

Boolean

Whether all input values are equal.

Not equal

Any

Boolean

Whether some of the input values differ.

Geometric average

Numbers

Number

The geometric average of all numeric input values (i.e. the nth root of the product of all input values).

Harmonic average

Numbers

Number

The harmonic average of the input values, i.e. the reciprocal of the arithmetic mean of the reciprocals of the input values.

Max

Numbers

Number

The largest numeric value of all input values

Min

Numbers

Number

The smallest numeric value of all input values

Multiply

Numbers

Number

The product of all numeric input values.

Range

Numbers

Number

The absolute range (i.e. ‘max value - min value’) across all numeric input values.

Unique

Any

Any

A list of all unique input values (i.e. with duplicates removed)

Sort

Numbers

Numbers

The input vector, sorted in ascending order

Median

Numbers

Number

The median of the input vector

Normalize

Numbers

Numbers

The input vector in normalized form

Order-dependent operations#

Order-dependent operations take two or more columns as input, and produce a result for each row in those columns. The order of the input columns affects the results of these operations. The “Subtract” operation, which subtracts one or more columns from another, is an example of an order-dependent operation. A complete list of the available order-dependent operations, along with their input requirements, can be seen below.

Operation

Input Type

Output type

Description

Divide

Numbers

Number

The first value divided by the next. In the case of more terms than two, each subsequent term is used as a divisor

To string

Any

String

The input value(s) converted to a string. In the case of multiple input columns, values are comma separated. In the case of nested input columns, the values are represented as JSON.

Hash

Any

Number

A numeric hash value calculated from all input values (including scalars, nested inputs and arrays

Subtract

Numbers

Number

The first numeric input value minus all following ones.

Delta

Numbers

Number

The difference between one column and the next (i.e. ‘next value - this value’

IoU

Bounding boxes

Numbers

For each bounding box in the first column, the maximum intersection-over-union against any bounding box in the second column

TP

Bounding boxes

Numbers

Whether each predicted bounding box is a True Positive, i.e. has a corresponding ground truth bounding box.

FP

Bounding boxes

Numbers

Whether each predicted bounding box is a False Positive, i.e. has no corresponding ground truth bounding box.

FN

Bounding boxes

Numbers

Whether each ground truth bounding box is a False Negative, i.e. has no corresponding predicted bounding box.

Greater than or equal

Numbers

Boolean

Whether each value is greater than or equal to the next.

Greater than

Numbers

Boolean

Whether each value is greater than the next.

Less than or equal

Numbers

Boolean

Whether each value is less than or equal to than the next.

Less than

Numbers

Boolean

Whether each value is less than the next.

Keyboard Shortcuts#

The 3LC Dashboard has a number of keyboard shortcuts for common operations. Some of these shortcuts are always available, while others are only available when you have one or more columns selected, rows selected, or charts windows selected.

Global shortcuts#

Modifiers

Key

Action

<0>

Show Settings window

<Alt>

<1>

Copy operation docs to clipboard

<Ctrl>

<1>

Create new editable column (bool)

<Ctrl>

<2>

Create new editable column (string)

<Ctrl>

<3>

Create new editable column (float32)

<Ctrl>

<4>

Create new editable column (int32)

<5>

Toggle images on/off

<Ctrl>

<5>

Create new value map item

<6>

Toggle manual labels on/off

<7>

Toggle predictions on/off

<9>

Show the keyboard cheat sheet

<Ctrl>

<9>

Show Dataflow diagram window

<A>

Select all table rows

<Ctrl>

<A>

Select all bounding boxes

<B>

Add selected predictions

<Ctrl>

<B>

Add predictions selected row

<Ctrl><Shift>

<B>

Add predictions all rows

<Alt>

<Backspace>

Undo last edit

<Shift><Alt>

<Backspace>

Redo last undone edit

<Ctrl>

<Backspace>

Discard all pending edits

<C>

Show/hide Rows panel

<D>

Clear all filters

<Ctrl>

<D>

Clear polygon selections

<Ctrl><Shift>

<D>

Toggle light/dark mode

<E>

Filter using Polygon tool

<Ctrl>

<Enter>

Commit all pending edits

<Esc>

Deselect all table elements

<F>

Create subset table from the currently filtered table elements

<G>

Create reduced table given the currently selected columns

<J>

Set List mode

<Ctrl>

<J>

Set Grid mode

<N>

Assign selected predictions

<Ctrl>

<N>

Assign predictions selected row

<Ctrl><Shift>

<N>

Assign predictions all rows

<P>

Refresh the contents of all tables

<Q>

Pan/rotate viewport

<Ctrl>

<Q>

Pan/rotate viewport (while keys pressed)

<R>

Filter using Lasso tool

<Ctrl>

<R>

Refresh the contents of all tables

<S>

Deselect all table elements

<Ctrl>

<S>

Deselect all table elements

<Space>

Pan/rotate viewport

<Ctrl>

<Space>

Pan/rotate viewport (while keys pressed)

<T>

Filter using Paint tool

<W>

Filter using Rectangle tool

<X>

Show/hide Charts panel

<Y>

Snap all 2D/3D windows to contents

<Ctrl>

<Y>

Redo last undone edit

<Z>

Show/hide Filter panel

<Ctrl>

<Z>

Undo last edit

<Ctrl><Shift>

<Z>

Redo last undone edit