This feature is currently in Beta. You can request free access on the main page of the beta or contact us directly for more information.

Record a Training Run (Notebook)

We'll walk through an example of instrumenting and executing training run code to make use of SigOpt.

How a Training Run is Recorded

A SigOpt training run is an object that records one or more of the following attributes:

the model type,
dataset identifier,
evaluation metrics,
hyperparameters,
logs, and
a code reference.

Take a look at the following example.

Example

Install & Configure
If you haven't already, follow the installation and configuration instructions in our Get Started page.
Set Your Project
Training runs are created within projects. The project allows you to sort and filter through your training runs and view useful charts to gain insight into everything you've tried.
We recommend setting the project name as an environment variable. In this example, we're naming our project run-examples. Note: This example has been tested with Tensorflow 1.14.
```
from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf
import sigopt
%load_ext sigopt
import os
os.environ['SIGOPT_PROJECT'] = 'run-examples'
```
If the valid Project ID doesn't already exist, a project will be created.

Instrument the Training Run

In another cell, run your training run code, with SigOpt methods integrated.

%%run Example Run
sigopt.log_dataset(name='mnist')
mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)

input_features = 784
sigopt.log_metadata('input_features', input_features)
x = tf.placeholder("float", [None, input_features])
output_classes = 10
sigopt.log_metadata('output_classes', output_classes)
y = tf.placeholder("float", [None, output_classes])

sigopt.log_model(type='Multi Layer Perceptron')

num_hidden_nodes = sigopt.get_parameter('num_hidden_nodes', default=10)
hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes]))
hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes]))
hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases))

activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes]))
activation_biases = tf.Variable(tf.random_normal([output_classes]))
activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases))

learning_rate = 10**sigopt.get_parameter('log_learning_rate', default=-3)

cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=activation_layer, labels=y))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

with tf.Session() as sess:
  sess.run(tf.global_variables_initializer())

  batch_size = sigopt.get_parameter('batch_size', default=100)
  epochs = sigopt.get_parameter('epochs', default=5)
  num_batches = mnist.train.num_examples // batch_size
  for i in range(epochs):
    for _ in range(num_batches):
      batch_x, batch_y = mnist.train.next_batch(batch_size)
      sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})

  correct_prediction = tf.equal(tf.argmax(activation_layer, 1), tf.argmax(y, 1))
  accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
  accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels})
  sigopt.log_metric('accuracy', accuracy_value)

Run the Code

Once you've run the code above, SigOpt will conveniently output links to the run page on our web application.

Run started, view it on the SigOpt dashboard at https://app.sigopt.com/run/1
Extracting /tmp/data/train-images-idx3-ubyte.gz
Extracting /tmp/data/train-labels-idx1-ubyte.gz
Extracting /tmp/data/t10k-images-idx3-ubyte.gz
Extracting /tmp/data/t10k-labels-idx1-ubyte.gz
Run finished, view it on the SigOpt dashboard at https://app.sigopt.com/run/1

View the Record
Now, visit the project in the SigOpt web app to see the results. Here's a view of the training run record:
At the project level, you can compare the training runs.

Overview of the SigOpt Methods

The example above uses the following methods (links to the specifications):

Let's discuss a few notable features below.

sigopt.get_parameter

Parameters are any input to your model that you may change as you seek to refine the model. In our example, we have the number of hidden nodes, batch size, and number of epochs. If you think you might want to optimize on a parameter, that's a good reason to include it as a parameter.

During a single training run, sigopt.get_parameter utilizes the default value. If you're interested in optimizing or exploring a hyperparameter (or two), visit the Optimize & Explore section to learn more about how sigopt.get_parameter can return a suggested parameter SigOpt suggests within the search space. With a single method, sigopt.get_parameter, you can instrument your code once and use it to create a single training run or to automate an experiment.

sigopt.log_model

sigopt.log_model stores a text value that you can use to filter your runs in the SigOpt web view. In our example, you might want to filter by sklearn - logistic regression to compare models of the same type in the web charts.

sigopt.log_dataset

Name your dataset with sigopt.log_dataset so you can easily reference it later. Store the path to AWS S3 or a csv filename, whatever might be useful to you.

sigopt.log_metadata

Record any key value pairs you can imagine with sigopt.log_metadata.

In our example, we've used sigopt.log_metadata to store the input features and output classes, values we might want to change and compare across training runs but aren't data we'd consider parameters.

sigopt.log_metric

The most important information about a model is how it performed. With sigopt.log_metric you can take advantage of SigOpt's analysis dashboard of custom charts and advanced sorting and filtering.

Optimize & Explore

SigOpt has long been used to automate the tuning of a model. The next section, Optimize & Explore, shows how you can set up SigOpt experiments that will generate hyperparameter configurations with either a simple grid search or our Bayesian ensemble.

Documentation