This feature is currently in Beta. You can request free access on the main page of the beta or contact us directly for more information.

Optimization & Exploration

In the guides for notebook and command line development environments, we've covered the recommended way to instrument your code with SigOpt for a training run and shown how that can be easily extended to run SigOpt experiments.

Here we'll discuss more of the options for running a SigOpt experiment.

Optimization

After your model has been instrumented, it is easy to take advantage of SigOpt's optimization features. Optimization helps find the parameters for your model that give you the best metric (eg. maximizing an accuracy metric).

An experiment is a definition of the hyperparameter search space and the metrics you would like optimized.

The experiment can be defined in either a YAML or JSON format. In a notebook, this will be defined using the %%experiment magic command in a cell.

In a text editor, you'll create a sigopt.yml file in the directory from which you'll be running sigopt optimize example.py

Below, we have an example optimization which finds parameters to maximize test_accuracy for a Tensorflow model:

Text Editor & CLI
Notebook

Text Editor & CLI Optimization Example

Install & Configure SigOpt
If you haven't already, follow the installation and configuration instructions in our Get Started page.
Set Project
Add Sigopt methods to capture data about your training run.
See Record a Run for more details about setting the project ID. The API Reference is a definitive list of the SigOpt methods available to instrument your run.
If you don't set a $SIGOPT_PROJECT environment variable, SigOpt will use the current directory name as the project ID.

Instrument Your Model

See Record a Run for more details about how the following example run was instrumented.

# example.py
from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf

import sigopt

sigopt.log_dataset(name='mnist')
mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)

input_features = 784
sigopt.log_metadata('input_features', input_features)
x = tf.placeholder("float", [None, input_features])
output_classes = 10
sigopt.log_metadata('output_classes', output_classes)
y = tf.placeholder("float", [None, output_classes])

sigopt.log_model(type='Multi Layer Perceptron')

num_hidden_nodes = sigopt.get_parameter('num_hidden_nodes', default=10)
hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes]))
hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes]))
hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases))

activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes]))
activation_biases = tf.Variable(tf.random_normal([output_classes]))
activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases))

learning_rate = 10**sigopt.get_parameter('log_learning_rate', default=-3)

cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=activation_layer, labels=y))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

with tf.Session() as sess:
  sess.run(tf.global_variables_initializer())

  batch_size = sigopt.get_parameter('batch_size', default=100)
  epochs = sigopt.get_parameter('epochs', default=5)
  num_batches = mnist.train.num_examples // batch_size
  for i in range(epochs):
    for _ in range(num_batches):
      batch_x, batch_y = mnist.train.next_batch(batch_size)
      sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})

  correct_prediction = tf.equal(tf.argmax(activation_layer, 1), tf.argmax(y, 1))
  accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
  accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels})
  sigopt.log_metric('accuracy', accuracy_value)

Define the Experiment

Create a sigopt.yml file to define your experiment. The file should have an experiment section with the name, project, parameters, metrics and other options that you would like to run your experiment with.

The parameters should match the names provided to sigopt.get_parameter in your Python code and similarly the metrics should match the names of the sigopt.log_metric calls.

Here's an example file:

# sigopt.yml
experiment:
  name: Multi-Layer Perceptron
  parameters:
    - name: num_hidden_nodes
      bounds:
        min: 10
        max: 784
      type: int
    - name: log_learning_rate
      bounds:
        min: -8
        max: 0
      type: double
    - name: batch_size
      bounds:
        min: 50
        max: 400
      type: int
    - name: epochs
      bounds:
        min: 5
        max: 20
      type: int
  metrics:
    - name: accuracy
      objective: maximize
  observation_budget: 50

Note: If your sigopt.yml file is not in the directory where you are running sigopt optimize, you can specify the location of the sigopt.yml file with an additional --sigopt-file flag:

$ sigopt optimize example.py --sigopt-file example/sigopt.yml

Run the Optimization
```
# terminal
$ sigopt optimize example.py
```
Note: Any extra arguments will be passed along to your Python executable. For example:
```
$ sigopt optimize example.py arg1 arg2
```
View the Results
The output will provide links to the SigOpt web app where you can navigate to recent runs and experiments.

Notebook Optimization Example

Install & Configure SigOpt
If you haven't already, follow the installation and configuration instructions in our Get Started page.

Set the Project

from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf
import sigopt
%load_ext sigopt
import os
os.environ['SIGOPT_PROJECT'] = 'run-examples'

mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)

input_features = 784
output_classes = 10

Instrument Your Model

See Record a Run for more details about how the following example run was instrumented.

def create_model(x):
  num_hidden_nodes = sigopt.get_parameter('num_hidden_nodes', default=10)
  hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes]))
  hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes]))
  hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases))

  activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes]))
  activation_biases = tf.Variable(tf.random_normal([output_classes]))
  activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases))
  return activation_layer

def train_model(sess, optimizer, x, y):
  batch_size = sigopt.get_parameter('batch_size', default=100)
  epochs = sigopt.get_parameter('epochs', default=5)
  num_batches = mnist.train.num_examples // batch_size
  for i in range(epochs):
    for _ in range(num_batches):
      batch_x, batch_y = mnist.train.next_batch(batch_size)
      sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})

def evaluate_model(sess, model, x, y):
  correct_prediction = tf.equal(tf.argmax(model, 1), tf.argmax(y, 1))
  accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
  accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels})
  sigopt.log_metric('accuracy', accuracy_value)

Define the Experiment

The experiment definition will include the name, project, parameters, metrics and other options that you would like to run your experiment with.

The parameters should match the names provided to sigopt.get_parameter in your Python code and similarly the metrics should match the names of the sigopt.log_metric calls.

You can format the experiment definition in Python, YAML, or JSON. In this example, we're using Python. See below for YAML and JSON examples. With Python, you have the advantage of using local variables if you wish.

%%experiment
# define the parameters, metrics and budget for your experiment
{
  'metrics': [
    {
      'name': 'accuracy',
      'objective': 'maximize'
    }
   ],
  'name': 'Multi-Layer Perceptron',
  'observation_budget': 5,
  'parameters': [
    {
      'bounds':
        {
          'max': 784,
          'min': 10
        },
      'name': 'num_hidden_nodes',
      'type': 'int'
    },
    {
      'bounds':
        {
          'max': 0,
          'min': -8
        },
      'name': 'log_learning_rate',
      'type': 'double'
    },
    {
      'bounds':
        {
          'max': 400,
          'min': 50
        },
      'name': 'batch_size',
      'type': 'int'
    },
    {
      'bounds':
        {
          'max': 20,
          'min': 5
        },
      'name': 'epochs',
      'type': 'int'
    }
  ]
}

Note: The observation budget is the number of runs you would like created in the optimization. We recommend using 5 for testing. When you're ready, visit the observation budget page to learn our rule of thumb for the appropriate observation budget.

Run the Optimization

%%optimize
# optimize the model over multiple iterations
# use %%run instead to only record a single run using the default parameters

x = tf.placeholder("float", [None, input_features])
y = tf.placeholder("float", [None, output_classes])

sigopt.log_dataset(name='mnist')
sigopt.log_metadata('input_features', input_features)
sigopt.log_metadata('output_classes', output_classes)
sigopt.log_model(type='Multi Layer Perceptron')

model = create_model(x)

learning_rate = 10 ** sigopt.get_parameter('log_learning_rate', default=-3)
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=model, labels=y))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

with tf.Session() as sess:
  sess.run(tf.global_variables_initializer())

  train_model(sess, optimizer, x, y)

  evaluate_model(sess, model, x, y)

View the Results
The output will provide links to the SigOpt web app where you can navigate to recent runs and experiments.

Formatting

When working in a notebook environment, in addition to Python, you have the option to define your experiment in YAML or JSON. Here are examples of both:

YAML

%%experiment
# define the parameters, metrics and budget for your experiment
'''
name: Multi-Layer Perceptron
parameters:
  - name: num_hidden_nodes
    bounds:
      min: 10
      max: 784
    type: int
  - name: log_learning_rate
    bounds:
      min: -8
      max: 0
    type: double
  - name: batch_size
    bounds:
      min: 50
      max: 400
    type: int
  - name: epochs
    bounds:
      min: 5
      max: 20
    type: int
metrics:
  - name: accuracy
    objective: maximize
observation_budget: 5
'''

JSON

%%experiment
# define the parameters, metrics and budget for your experiment
'''
{
  "name": "Multi-Layer Perceptron",
  "parameters": [
    {
      "name": "num_hidden_nodes",
      "bounds": {
        "min": 10,
        "max": 784
      },
      "type": "int"
    },
    {
      "name": "log_learning_rate",
      "bounds": {
        "min": -8,
        "max": 0
      },
      "type": "double"
    },
    {
      "name": "batch_size",
      "bounds": {
        "min": 50,
        "max": 400
      },
      "type": "int"
    },
    {
      "name": "epochs",
      "bounds": {
        "min": 5,
        "max": 20
      },
      "type": "int"
    }
  ],
  "metrics": [
    {
      "name": "accuracy",
      "objective": "maximize"
    }
  ],
  "observation_budget": 5
}
'''

Exploration

You might want to use grid or random searches to explore a space. Please contact our team to learn how to use these experiment options.

Documentation

Optimization & Exploration

Optimization

Text Editor & CLI Optimization Example

Install & Configure SigOpt

Set Project

Instrument Your Model

Define the Experiment

Run the Optimization

View the Results

Notebook Optimization Example

Install & Configure SigOpt

Set the Project

Instrument Your Model

Define the Experiment

Run the Optimization

View the Results

Formatting

Exploration