Runs#

Runs are the basic unit of ML computation in 3LC, and can represent a training run, a metrics capture session, an experiment, or other data-related workflow. A Run brings together a collection of input tables (datasets), metric tables, and configuration parameters.

Multiple runs can be compared in the Dashboard.

Create a Run#

Runs can be explicitly created by constructing a Run object.

[1]:
import tlc

run = tlc.Run(
    url=tlc.Url("./run.json").to_absolute(),
    project_name="My Project",
)

print(run)
{
  "type":"Run",
  "created":"2023-11-24 12:28:13.326912+00:00",
  "last_modified":"2023-11-24 12:28:13.326912+00:00",
  "project_name":"My Project",
  "constants":{
    "parameters":{}
  }
}

When integrating with 3LC, it is most common to let a Run be implicitly created by a call to tlc.init(), which writes a new Run to the default Run location, and returns a client handle that can be used to interact with the Run.

[2]:
run = tlc.init(project_name="My Project")
[11/24 13:28:13 tlc.client.session]: Created run light-treble in project My Project

Record configuration parameters#

When working on machine learning projects, it’s crucial to keep track of the configuration parameters for each training run. Recording these parameters is important for several reasons:

  • Reproducibility: Keeping a record of the configuration parameters ensures that your experiments can be replicated. This is vital for validating results and for when you need to revisit old models.

  • Comparison and Analysis: By tracking different configurations, you can compare the performance of models under various settings. This helps in understanding which parameters are most effective for your model.

[3]:
# Example: how to record configuration parameters in your run:

config = {
    "epochs": 10,
    "batch_size": 32
}

run.set_parameters(config)
print(run)
{
  "type":"Run",
  "created":"2023-11-24 12:28:13.881718+00:00",
  "last_modified":"2023-11-24 12:28:13.924718+00:00",
  "project_name":"My Project",
  "constants":{
    "parameters":{
      "epochs":10,
      "batch_size":32
    }
  }
}

Adding input and output values#

This is mostly taken care of by the system, i.e. by calls to collect_metrics.

Input values point to tables (datasets) that were used as input to the run.

Output values point to metric tables that were produced by the run.

[4]:
run.add_input_value({"dataset_name": "Training Set", "dataset_url": "./training.json"})
run.add_input_value({"dataset_name": "Test Set", "dataset_url": "./test.json"})

run.add_output_value({"epoch": 1, "loss": 0.1, "accuracy": 0.9})

print(run)
{
  "type":"Run",
  "created":"2023-11-24 12:28:13.881718+00:00",
  "last_modified":"2023-11-24 12:28:13.950240+00:00",
  "project_name":"My Project",
  "constants":{
    "inputs":[
      {
        "dataset_name":"Training Set",
        "dataset_url":"./training.json"
      },
      {
        "dataset_name":"Test Set",
        "dataset_url":"./test.json"
      }
    ],
    "outputs":[
      {
        "epoch":1,
        "loss":0.1,
        "accuracy":0.9
      }
    ],
    "parameters":{
      "epochs":10,
      "batch_size":32
    }
  }
}

Adding a metrics table to a run#

For more complex metrics-collection workflows, collect_metrics() should be used. For simpler cases when you just want to record some metrics, you can use run.add_metrics_data() to add a free-form dictionary of metrics data.

[5]:
run.add_metrics_data({"epoch": [1], "loss": [0.1], "accuracy": [0.9]})

print(run)
{
  "type":"Run",
  "created":"2023-11-24 12:28:13.881718+00:00",
  "last_modified":"2023-11-24 12:28:14.007240+00:00",
  "project_name":"My Project",
  "metrics":[
    {
      "url":"../light-treble/metrics_0000.json",
      "file_size":1336,
      "stream_name":"default_stream",
      "row_count":1
    }
  ],
  "constants":{
    "inputs":[
      {
        "dataset_name":"Training Set",
        "dataset_url":"./training.json"
      },
      {
        "dataset_name":"Test Set",
        "dataset_url":"./test.json"
      }
    ],
    "outputs":[
      {
        "epoch":1,
        "loss":0.1,
        "accuracy":0.9
      }
    ],
    "parameters":{
      "epochs":10,
      "batch_size":32
    }
  }
}

The metrics tables of a Run can also be accessed as a list of info-dicts via run.metrics, or as a list of tables via run.metrics_tables.

[6]:
print(run.metrics)
print([t.__class__.__name__ for t in run.metrics_tables])
[{'url': '../light-treble/metrics_0000.json', 'file_size': 1336, 'stream_name': 'default_stream', 'row_count': 1}]
['TableFromParquet']