This feature is currently in Beta. You can request free access on the main page of the beta or contact us directly for more information.

Convert Existing SigOpt Code

If you've been using SigOpt already, this article will help you convert your existing SigOpt experiment code to use the new SigOpt methods, using a Python file as an example. Variations in converting code in a notebook environment are called out.

However, if you wish, your old code will still work with the newest version of our Python package. Multitask and Parallelism are not currently compatible with the beta command line tool.

Overview

Instead of executing a Python file that contains an optimization loop, we pass a YAML file and a Python file to the SigOpt command line tool. The SigOpt command line tool will create the experiment and execute the Python file once for each iteration of the experiment. This way, the Python file can be used both to execute a single training run and an entire experiment, without modification.

In this article, we'll start with an example of a SigOpt experiment and walk through the following steps of converting it to the new SigOpt method:

Update SigOpt
Define the Experiment
Remove the Optimization Loop
Get Parameters
Record Metrics
Record Additional Information
Set the Project
Set the API Token
Run an Experiment

Example Code

Here's our example Python code that uses the older version of SigOpt. Note: This example has been tested with Tensorflow 1.14.

# example.py
from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf

from sigopt import Connection

conn = Connection(client_token=SIGOPT_API_TOKEN)

mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)

input_features = 784
x = tf.placeholder("float", [None, input_features])
output_classes = 10
y = tf.placeholder("float", [None, output_classes])

experiment = conn.experiments().create(
  name='Multi-Layer Perceptron',
  parameters=[
    dict(name='num_hidden_nodes', type='int', bounds=dict(min=10, max=784)),
    dict(name='learning_rate', type='double', bounds=dict(min=1.0e-8, max=1.0), transformation='log'),
    dict(name='batch_size', type='int', bounds=dict(min=50, max=400)),
    dict(name='epochs', type='int', bounds=dict(min=5, max=20)),
  ],
  metrics=[dict(name='accuracy', objective='maximize')],
  observation_budget=50,
  project='sigopt-examples',
  metadata=dict(
      input_features=input_features,
      output_classes=output_classes,
      model='Multi Layer Perceptron',
      dataset='mnist'
    ),
)

with tf.Session() as sess:
  while experiment.progress.observation_count < experiment.observation_budget:
    suggestion = conn.experiments(experiment.id).suggestions().create()
    batch_size = suggestion.assignments['batch_size']
    epochs = suggestion.assignments['epochs']
    num_batches = mnist.train.num_examples // batch_size

    num_hidden_nodes = suggestion.assignments['num_hidden_nodes']
    hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes]))
    hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes]))
    hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases))

    activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes]))
    activation_biases = tf.Variable(tf.random_normal([output_classes]))
    activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases))

    learning_rate = suggestion.assignments['learning_rate']
    cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=activation_layer, labels=y))
    optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

    sess.run(tf.global_variables_initializer())

    for i in range(epochs):
      for _ in range(num_batches):
        batch_x, batch_y = mnist.train.next_batch(batch_size)
        sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})

    correct_prediction = tf.equal(tf.argmax(activation_layer, 1), tf.argmax(y, 1))
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
    accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels})
    conn.experiments(experiment.id).observations().create(
      suggestion=suggestion.id,
      values=[
        dict(
          name='accuracy',
          value=accuracy_value,
        )
      ],
    )
    experiment = conn.experiments(experiment.id).fetch()

Conversion Steps

Update SigOpt Back to Top
If you haven't already, please install the latest version of the SigOpt Python client.
```
pip install --upgrade https://public.sigopt.com/experiment-management/sigopt-7.1.5-py2.py3-none-any.whl
```

Define the Experiment Back to Top

Create a sigopt.yml file to define your experiment. This will include:

the experiment name,
metric(s),
parameters,
the observation budget (the anticipated number of iterations), and
additional experiment features.

# sigopt.yml
experiment:
  name: Multi-Layer Perceptron
  parameters:
    - name: num_hidden_nodes
      bounds:
        min: 10
        max: 784
      type: int
    - name: learning_rate
      bounds:
        min: 1.0e-8
        max: 1.0
      type: double
      transformation: log
    - name: batch_size
      bounds:
        min: 50
        max: 400
      type: int
    - name: epochs
      bounds:
        min: 5
        max: 20
      type: int
  metrics:
    - name: accuracy
      objective: maximize
  observation_budget: 50

A few notes:

Multitask is not currently compatible with the Beta.
If your sigopt.yml file is not in the directory where you are running sigopt optimize, you can specify the location of the sigopt.yml file with an additional --sigopt-file flag.
If you're working in an IPython notebook, you can define the experiment using YAML, JSON, or a Python dictionary in a cell, as shown on this page.

Remove the Optimization Loop Back to Top

Next, remove the optimization loop that relies on the observation budget.

This will include removing:

the Connection import and object,
the while or for loop that relies on the experiment.progress.count and the experiment.observation_budget,
the suggestion creation experiments(experiment.id).suggestions().create(), and
the observation creation experiments(experiment.id).observations().create(), and
any updates to the experiment object.

Make sure you have imported SigOpt, import sigopt.

We've left a gap in our code now where we used the suggestion to provide parameters. We'll fix that in the next step.

# example.py
from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf

import sigopt

mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)

input_features = 784
x = tf.placeholder("float", [None, input_features])
output_classes = 10
y = tf.placeholder("float", [None, output_classes])

with tf.Session() as sess:
  batch_size = suggestion.assignments['batch_size']
  epochs = suggestion.assignments['epochs']
  num_batches = mnist.train.num_examples // batch_size

  num_hidden_nodes = suggestion.assignments['num_hidden_nodes']
  hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes]))
  hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes]))
  hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases))

  activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes]))
  activation_biases = tf.Variable(tf.random_normal([output_classes]))
  activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases))

  learning_rate = suggestion.assignments['learning_rate']
  cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=activation_layer, labels=y))
  optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

  sess.run(tf.global_variables_initializer())

  for i in range(epochs):
    for _ in range(num_batches):
      batch_x, batch_y = mnist.train.next_batch(batch_size)
      sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})
  correct_prediction = tf.equal(tf.argmax(activation_layer, 1), tf.argmax(y, 1))
  accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
  accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels})

Get Parameters Back to Top

To make this Python file usable for both experiments and single training runs, sigopt.get_parameter will fetch the appropriate parameter for a given iteration of an experiment, or, for a single training run, use the default parameter.

We'll do that here for batch_size, epochs, num_hidden_nodes, and learning_rate.

For the learning_rate, you can see we're making use of the logarithmic Parameter Transformation feature.

# example.py
from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf

import sigopt

mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)

input_features = 784
x = tf.placeholder("float", [None, input_features])
output_classes = 10
y = tf.placeholder("float", [None, output_classes])

with tf.Session() as sess:
  batch_size = sigopt.get_parameter('batch_size', default=100)
  epochs = sigopt.get_parameter('epochs', default=5)
  num_batches = mnist.train.num_examples // batch_size

  num_hidden_nodes = sigopt.get_parameter('num_hidden_nodes', default=10)
  hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes]))
  hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes]))
  hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases))

  activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes]))
  activation_biases = tf.Variable(tf.random_normal([output_classes]))
  activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases))

  learning_rate = sigopt.get_parameter('learning_rate', default=1e-3)
  cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=activation_layer, labels=y))
  optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

  sess.run(tf.global_variables_initializer())

  for i in range(epochs):
    for _ in range(num_batches):
      batch_x, batch_y = mnist.train.next_batch(batch_size)
      sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})
  correct_prediction = tf.equal(tf.argmax(activation_layer, 1), tf.argmax(y, 1))
  accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
  accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels})

Record Metrics Back to Top

Next, we need to report the metric(s) for a training run or an experiment iteration. For this example, we report one metric with sigopt.log_metric('accuracy', accuracy_value) .

# example.py
from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf

import sigopt

mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)

input_features = 784
x = tf.placeholder("float", [None, input_features])
output_classes = 10
y = tf.placeholder("float", [None, output_classes])

with tf.Session() as sess:
  batch_size = sigopt.get_parameter('batch_size', default=100)
  epochs = sigopt.get_parameter('epochs', default=5)
  num_batches = mnist.train.num_examples // batch_size

  num_hidden_nodes = sigopt.get_parameter('num_hidden_nodes', default=10)
  hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes]))
  hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes]))
  hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases))

  activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes]))
  activation_biases = tf.Variable(tf.random_normal([output_classes]))
  activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases))

  learning_rate = sigopt.get_parameter('learning_rate', default=1e-3)
  cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=activation_layer, labels=y))
  optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

  sess.run(tf.global_variables_initializer())

  for i in range(epochs):
    for _ in range(num_batches):
      batch_x, batch_y = mnist.train.next_batch(batch_size)
      sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})
  correct_prediction = tf.equal(tf.argmax(activation_layer, 1), tf.argmax(y, 1))
  accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
  accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels})
  sigopt.log_metric('accuracy', accuracy_value)

Record Additional Information Back to Top

We recommend defining the following optional fields inline so that this data will be included when a single training run is created with the SigOpt command line tool, not only in experiments.

sigopt.log_metadata('key', 'value') allows you to log metadata anywhere it is convenient.
sigopt.log_model(type='xgboost') will store a string for identifying the model used.
sigopt.log_dataset(name='minst') will store an identifier for the dataset used.

# example.py
from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf

import sigopt

sigopt.log_dataset(name='mnist')
mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)

input_features = 784
sigopt.log_metadata('input_features', input_features)
x = tf.placeholder("float", [None, input_features])
output_classes = 10
sigopt.log_metadata('output_classes', output_classes)
y = tf.placeholder("float", [None, output_classes])

sigopt.log_model(type='Multi Layer Perceptron')

with tf.Session() as sess:
  batch_size = sigopt.get_parameter('batch_size', default=100)
  epochs = sigopt.get_parameter('epochs', default=5)
  num_batches = mnist.train.num_examples // batch_size

  num_hidden_nodes = sigopt.get_parameter('num_hidden_nodes', default=10)
  hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes]))
  hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes]))
  hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases))

  activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes]))
  activation_biases = tf.Variable(tf.random_normal([output_classes]))
  activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases))

  learning_rate = sigopt.get_parameter('learning_rate', default=1e-3)
  cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=activation_layer, labels=y))
  optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

  sess.run(tf.global_variables_initializer())

  for i in range(epochs):
    for _ in range(num_batches):
      batch_x, batch_y = mnist.train.next_batch(batch_size)
      sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})
  correct_prediction = tf.equal(tf.argmax(activation_layer, 1), tf.argmax(y, 1))
  accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
  accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels})
  sigopt.log_metric('accuracy', accuracy_value)

Set the Project Back to Top
Training runs are created within projects. The project allows you to sort and filter through your training runs and view useful charts to gain insight into everything you've tried.
By default, the name of the current directory will be the project name and ID associated with the training run.
If you'd like to choose a different project, you can override the default by setting an environment variable.
Run the following code in your terminal, replacing run_example with your project ID.
```
export SIGOPT_PROJECT=run_example
```
If the valid Project ID doesn't already exist, a project will be created.
See our notebook environment page for instructions on setting the project in a notebook.
Set the API Token Back to Top
Set your token via sigopt config. More details are available on Get Started.
Run an Experiment Back to Top
Now you're ready to give your code a test run!
```
sigopt optimize example.py
```
If you have arguments to pass to your Python file, the SigOpt client will pass them along, for example:
```
sigopt optimize example.py arg1 arg2
```

Limitations

Multitask is not supported.
Parallelism is not supported.

Conclusion

We hope this article has helped you convert your existing SigOpt experiment files with little fuss. Please reach out to us with any further questions.

Documentation