This feature is currently in Beta. You can request free access on the main page of the beta or contact us directly for more information.

Record a Training Run (Text Editor & CLI)

We'll walk through an example of instrumenting and executing training run code to make use of SigOpt.

How a Training Run is Recorded

A SigOpt training run is an object that records one or more of the following attributes:

the model type,
dataset identifier,
evaluation metrics,
hyperparameters,
logs, and
a code reference.

The SigOpt package provides a command line wrapper that allows you to run Python files as you normally would in the terminal.

If instrumented in the way described below, you can take advantage of SigOpt to use the same file for single training runs:

sigopt run foo.py

and SigOpt experiments:

sigopt optimize foo.py

Take a look at the following example.

Example

Install & Configure
If you haven't already, follow the installation and configuration instructions in our Get Started page.
Set Your Project
Training runs are created within projects. The project allows you to sort and filter through your training runs and view useful charts to gain insight into everything you've tried.
By default, the name of the current directory will be the project name and ID associated with the training run.
If you'd like to choose a different project, you can override the default by setting an environment variable.
Run the following code in your terminal, replacing run_example with your project ID.
```
export SIGOPT_PROJECT=run_example
```
If the valid Project ID doesn't already exist, a project will be created.

Instrument the Training Run

This example training run code shows how to integrate the key SigOpt calls to record your training run. Note: This example has been tested with Tensorflow 1.14.

# example.py
from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf

import sigopt

sigopt.log_dataset(name='mnist')
mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)

input_features = 784
sigopt.log_metadata('input_features', input_features)
x = tf.placeholder("float", [None, input_features])
output_classes = 10
sigopt.log_metadata('output_classes', output_classes)
y = tf.placeholder("float", [None, output_classes])

sigopt.log_model(type='Multi Layer Perceptron')

num_hidden_nodes = sigopt.get_parameter('num_hidden_nodes', default=10)
hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes]))
hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes]))
hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases))

activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes]))
activation_biases = tf.Variable(tf.random_normal([output_classes]))
activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases))

learning_rate = sigopt.get_parameter('learning_rate', default=1e-3)

cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=activation_layer, labels=y))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

with tf.Session() as sess:
  sess.run(tf.global_variables_initializer())

  batch_size = sigopt.get_parameter('batch_size', default=100)
  epochs = sigopt.get_parameter('epochs', default=5)
  num_batches = mnist.train.num_examples // batch_size
  for i in range(epochs):
    for _ in range(num_batches):
      batch_x, batch_y = mnist.train.next_batch(batch_size)
      sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})

  correct_prediction = tf.equal(tf.argmax(activation_layer, 1), tf.argmax(y, 1))
  accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
  accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels})
  sigopt.log_metric('accuracy', accuracy_value)

Run the Code

sigopt run example.py

Run started, view it on the SigOpt dashboard at https://app.sigopt.com/run/1
Extracting /tmp/data/train-images-idx3-ubyte.gz
Extracting /tmp/data/train-labels-idx1-ubyte.gz
Extracting /tmp/data/t10k-images-idx3-ubyte.gz
Extracting /tmp/data/t10k-labels-idx1-ubyte.gz
Run finished, view it on the SigOpt dashboard at https://app.sigopt.com/run/1

SigOpt will conveniently output links in the terminal to the run page on our web application.

Any extra arguments will be passed along to your Python executable.

Be aware, you can always run your Python file without the sigopt run command, but if you do so, no run information will be sent to SigOpt.

View the Record
Now, visit the project in the SigOpt web app to see the results. Here's a view of the training run record:
At the project level, you can compare the training runs.

Overview of the SigOpt Methods

The example above uses the following methods (links to the specifications):

Let's discuss a few notable features below.

sigopt.get_parameter

Parameters are any input to your model that you may change as you seek to refine the model. In our example, we have the number of hidden nodes, batch size, and number of epochs. If you think you might want to optimize on a parameter, that's a good reason to include it as a parameter.

During a single training run, sigopt.get_parameter utilizes the default value. If you're interested in optimizing or exploring a hyperparameter (or two), visit the Optimize & Explore section to learn more about how sigopt.get_parameter can return a suggested parameter SigOpt suggests within the search space. With a single method, sigopt.get_parameter, you can instrument your code once and use it to create a single training run or to automate an experiment.

sigopt.log_model

sigopt.log_model stores a text value that you can use to filter your runs in the SigOpt web view. In our example, you might want to filter by sklearn - logistic regression to compare models of the same type in the web charts.

sigopt.log_dataset

Name your dataset with sigopt.log_dataset so you can easily reference it later. Store the path to AWS S3 or a csv filename, whatever might be useful to you.

sigopt.log_metadata

Record any key value pairs you can imagine with sigopt.log_metadata.

In our example, we've used sigopt.log_metadata to store the input features and output classes, values we might want to change and compare across training runs but aren't data we'd consider parameters.

sigopt.log_metric

The most important information about a model is how it performed. With sigopt.log_metric you can take advantage of SigOpt's analysis dashboard of custom charts and advanced sorting and filtering.

Optimize & Explore

SigOpt has long been used to automate the tuning of a model. The next section, Optimize & Explore, shows how you can set up SigOpt experiments that will generate hyperparameter configurations with either a simple grid search or our Bayesian ensemble.

Documentation