Documentation

Welcome to SigOpt’s developer documentation. If you have a question you can’t answer, feel free to contact us!
This feature is currently in Beta. You can request free access on the main page of the beta or feel free to contact us directly for more information.

Training Runs

Keep track of individual training runs in SigOpt as you explore data features and models. Easily run optimization experiments based on a training run.

Contents

Conceptual Overview

A SigOpt Run stores the training and evaluation of a model, so that modelers can see a history of their work.

Runs record everything you might need to understand how a model was built, reconstitute the model in the future, or explain the process to a colleague.

Common attributes of a Run include:

  • the model type,
  • dataset identifier,
  • evaluation metrics,
  • hyperparameters,
  • logs, and
  • a code reference.

For a complete list of attributes see API Reference.

Installation

Note: If you have installed SigOpt’s Orchestrate package, please uninstall it first: pip uninstall orchestrate.

Install the package using pip.

pip install --upgrade https://sigopt-python-mpm.s3-us-west-1.amazonaws.com/sigopt-7.1.1-py2.py3-none-any.whl

That’s all! You should be able to run sigopt --help.

First Time Setup

In order to interact with the SigOpt API, you’ll need to set up some general, basic configuration information.

Configure the SigOpt CLI with the command sigopt config.

This command will ask you for your API token and if you would like to opt into various features. You may override the config by setting the SIGOPT_API_TOKEN environment variable.

Find your SigOpt API token on the API tokens page.

Recording Training Runs

There are two common ways to record Runs:

  1. Command Line: Running a training and evaluation script from the command line.
  2. Jupyter Notebook: Iterative development in Jupyter notebooks.

An Example

# in an interactive notebook, use %%run
# on the command line, use $> sigopt run <file>
# log info about your run...
sigopt.log_dataset(name="my_training_data_v1")
sigopt.log_model(type="logisticRegression")
model = make_my_model()
training_loss = model.train(
    my_training_data,
    lr=sigopt.get_parameter('learning_rate', 0.1)
)
sigopt.log_metric('train-loss', training_loss)
accuracy = model.evaluate(my_validation_data)
sigopt.log_metric('train-accuracy', accuracy)

When you record runs, they will appear in your project in a table:

Everything that gets recorded will be stored within a project. The ID of this project is the name of your current directory by default. You may override the default project ID by setting the SIGOPT_PROJECT environment variable.

Seeing Results

To see your results on the web, visit https://app.sigopt.com/projects and select the project you've stored your Runs in. The default name for a project is the name of the directory in which your sigopt command was run.

You can also navigate directly to a run by following the link in your sigopt command output.

Runs on the Command Line

SigOpt supports training runs and evaluations which occur in a single Python executable. To record these runs:

  • Add sigopt logging calls about your model training run and evaluation to your Python code.
  • Use the sigopt run command to execute your Python executable. Any extra arguments will be passed along to your Python executable.

To start, import sigopt in your Python executable. Then use the API Reference section to add logging calls to record the important aspects of your model.

Once your code is instrumented, execute your model training and evaluation by running sigopt run yourmodel.py. This command will upload your logs and source code to SigOpt if you have opted in and enables the functions listed in the runs section below. A valid Python script is required as an argument.

You can always run your model file without the sigopt run command, but if you do so then no run information will be sent to SigOpt.

Example

# example.py
import sigopt
import sys
print(sys.argv)
sigopt.log_model('xgboost')
num_nodes = sigopt.get_parameters('num_nodes', default=10)
accuracy = 0.9
sigopt.log_metric('accuracy', accuracy)
# terminal commands
$ export SIGOPT_PROJECT=run-examples
$ sigopt run example.py arg1 arg2

Runs in a Notebook

The %%run magic command is the easiest way to record a run in a Python notebook (IPython and Jupyter). This will create a new run, execute your cell contents then complete the run.

To start using these sigopt functions, import sigopt and then run %load_ext sigopt in your notebook.

Example

import sigopt
%load_ext sigopt
import os
os.environ['SIGOPT_PROJECT'] = 'run-examples'
%%run
sigopt.log_model('xgboost')
num_nodes = sigopt.get_parameter('num_nodes', default=10)
accuracy = 0.9
sigopt.log_metric('accuracy', accuracy)

Optimization

After your model has been instrumented, it is easy to take advantage of SigOpt’s optimization features. Optimization helps find the parameters for your model that give you the best metric (eg. maximizing an accuracy metric). Optimize your model by running:

sigopt optimize yourmodel.py

You will need to create a sigopt.yml file in the same directory as your model file.

  • The sigopt.yml file should have an experiment section with the name, project, parameters, metrics and other options that you would like to run your experiment with.
  • The parameters should match the names provided to sigopt.get_parameter in your Python code and similarly the metrics should match the names of the sigopt.log_metric calls.

Here's an example which finds parameters to maximize test_accuracy for a Tensorflow model:

Command Line Run & Optimization Example

# example.py
from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf

import sigopt

sigopt.log_dataset(name='mnist')
mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)

input_features = 784
sigopt.log_metadata('input_features', input_features)
x = tf.placeholder("float", [None, input_features])
output_classes = 10
sigopt.log_metadata('output_classes', output_classes)
y = tf.placeholder("float", [None, output_classes])

sigopt.log_model(type='Multi Layer Perceptron')

num_hidden_nodes = sigopt.get_parameter('num_hidden_nodes', default=10)
hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes]))
hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes]))
hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases))

activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes]))
activation_biases = tf.Variable(tf.random_normal([output_classes]))
activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases))

learning_rate = 10**sigopt.get_parameter('log_learning_rate', default=-3)

cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=activation_layer, labels=y))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

with tf.Session() as sess:
  sess.run(tf.global_variables_initializer())

  batch_size = sigopt.get_parameter('batch_size', default=100)
  epochs = sigopt.get_parameter('epochs', default=5)
  num_batches = mnist.train.num_examples // batch_size
  for i in range(epochs):
    for _ in range(num_batches):
      batch_x, batch_y = mnist.train.next_batch(batch_size)
      sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})

  correct_prediction = tf.equal(tf.argmax(activation_layer, 1), tf.argmax(y, 1))
  accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
  accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels})
  sigopt.log_metric('accuracy', accuracy_value)
# sigopt.yml
experiment:
  name: Multi-Layer Perceptron
  parameters:
    - name: num_hidden_nodes
      bounds:
        min: 10
        max: 784
      type: int
    - name: log_learning_rate
      bounds:
        min: -8
        max: 0
      type: double
    - name: batch_size
      bounds:
        min: 50
        max: 400
      type: int
    - name: epochs
      bounds:
        min: 5
        max: 20
      type: int
  metrics:
    - name: accuracy
      objective: maximize
  observation_budget: 100
# terminal commands
$ export SIGOPT_PROJECT=run-examples
$ sigopt run example.py
$ sigopt optimize example.py

Notebook Run & Optimization Example

from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf
import sigopt
%load_ext sigopt
import os
os.environ['SIGOPT_PROJECT'] = 'run-examples'

mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)

input_features = 784
output_classes = 10
def create_model(x):
  num_hidden_nodes = sigopt.get_parameter('num_hidden_nodes', default=10)
  hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes]))
  hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes]))
  hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases))

  activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes]))
  activation_biases = tf.Variable(tf.random_normal([output_classes]))
  activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases))
  return activation_layer
def train_model(sess, optimizer, x, y):
  batch_size = sigopt.get_parameter('batch_size', default=100)
  epochs = sigopt.get_parameter('epochs', default=5)
  num_batches = mnist.train.num_examples // batch_size
  for i in range(epochs):
    for _ in range(num_batches):
      batch_x, batch_y = mnist.train.next_batch(batch_size)
      sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})
def evaluate_model(sess, model, x, y):
  correct_prediction = tf.equal(tf.argmax(model, 1), tf.argmax(y, 1))
  accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
  accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels})
  sigopt.log_metric('accuracy', accuracy_value)
%%experiment
# define the parameters, metrics and budget for your experiment
'''
name: Multi-Layer Perceptron
parameters:
  - name: num_hidden_nodes
    bounds:
      min: 10
      max: 784
    type: int
  - name: log_learning_rate
    bounds:
      min: -8
      max: 0
    type: double
  - name: batch_size
    bounds:
      min: 50
      max: 400
    type: int
  - name: epochs
    bounds:
      min: 5
      max: 20
    type: int
metrics:
  - name: accuracy
    objective: maximize
observation_budget: 100
'''
%%optimize
# optimize the model over multiple iterations
# use %%run instead to only record a single run using the default parameters

x = tf.placeholder("float", [None, input_features])
y = tf.placeholder("float", [None, output_classes])

sigopt.log_dataset(name='mnist')
sigopt.log_metadata('input_features', input_features)
sigopt.log_metadata('output_classes', output_classes)
sigopt.log_model(type='Multi Layer Perceptron')

model = create_model(x)

learning_rate = 10 ** sigopt.get_parameter('log_learning_rate', default=-3)
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=model, labels=y))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

with tf.Session() as sess:
  sess.run(tf.global_variables_initializer())

  train_model(sess, optimizer, x, y)

  evaluate_model(sess, model, x, y)

API Reference & Advanced Usage

For additional details on the API and advanced usage topics, please visit this page. Topics include:

  • sigopt.get_parameter
  • sigopt.log_dataset
  • sigopt.log_failure
  • sigopt.log_metadata
  • sigopt.log_metric
  • sigopt.log_model
  • sigopt.log_checkpoint
  • Manually creating and ending runs

Limitations

At this time, Training Monitor runs are not integrated with Project pages and thus will not appear alongside the runs created by the CLI in data tables and charts. Training Monitor runs will only be seen in Experiment pages.