Convert Existing SigOpt Code

If you've been using SigOpt already, this article will help you convert your existing SigOpt experiment code to use the new SigOpt methods, using a Python file as an example. Variations in converting code in a notebook environment are called out.

However, if you wish, your old code will still work with the newest version of our Python package. Multitask and Parallelism are not currently compatible with the beta command line tool.

Overview

Instead of executing a Python file that contains an optimization loop, we pass a YAML file and a Python file to the SigOpt command line tool. The SigOpt command line tool will create the experiment and execute the Python file once for each iteration of the experiment. This way, the Python file can be used both to execute a single training run and an entire experiment, without modification.

In this article, we'll start with an example of a SigOpt experiment and walk through the following steps of converting it to the new SigOpt method:

1. Update SigOpt
2. Define the Experiment
3. Remove the Optimization Loop
4. Get Parameters
5. Record Metrics
6. Record Additional Information
7. Set the Project
8. Set the API Token
9. Run an Experiment

Example Code

Here's our example Python code that uses the older version of SigOpt. Note: This example has been tested with Tensorflow 1.14.

# example.py
from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf

from sigopt import Connection

conn = Connection(client_token=SIGOPT_API_TOKEN)

mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)

input_features = 784
x = tf.placeholder("float", [None, input_features])
output_classes = 10
y = tf.placeholder("float", [None, output_classes])

experiment = conn.experiments().create(
  name='Multi-Layer Perceptron',
  parameters=[
    dict(name='num_hidden_nodes', type='int', bounds=dict(min=10, max=784)),
    dict(name='learning_rate', type='double', bounds=dict(min=1.0e-8, max=1.0), transformation='log'),
    dict(name='batch_size', type='int', bounds=dict(min=50, max=400)),
    dict(name='epochs', type='int', bounds=dict(min=5, max=20)),
  ],
  metrics=[dict(name='accuracy', objective='maximize')],
  observation_budget=50,
  project='sigopt-examples',
  metadata=dict(
      input_features=input_features,
      output_classes=output_classes,
      model='Multi Layer Perceptron',
      dataset='mnist'
    ),
)

with tf.Session() as sess:
  while experiment.progress.observation_count < experiment.observation_budget:
    suggestion = conn.experiments(experiment.id).suggestions().create()
    batch_size = suggestion.assignments['batch_size']
    epochs = suggestion.assignments['epochs']
    num_batches = mnist.train.num_examples // batch_size

    num_hidden_nodes = suggestion.assignments['num_hidden_nodes']
    hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes]))
    hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes]))
    hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases))

    activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes]))
    activation_biases = tf.Variable(tf.random_normal([output_classes]))
    activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases))

    learning_rate = suggestion.assignments['learning_rate']
    cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=activation_layer, labels=y))
    optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

    sess.run(tf.global_variables_initializer())

    for i in range(epochs):
      for _ in range(num_batches):
        batch_x, batch_y = mnist.train.next_batch(batch_size)
        sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})

    correct_prediction = tf.equal(tf.argmax(activation_layer, 1), tf.argmax(y, 1))
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
    accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels})
    conn.experiments(experiment.id).observations().create(
      suggestion=suggestion.id,
      values=[
        dict(
          name='accuracy',
          value=accuracy_value,
        )
      ],
    )
    experiment = conn.experiments(experiment.id).fetch()

Conversion Steps

1. Update SigOpt Back to Top

   If you haven't already, please install the latest version of the SigOpt Python client.

   pip install --upgrade sigopt==7.4.0

2. Define the Experiment Back to Top

   Create a sigopt.yml file to define your experiment. This will include:

   • the experiment name,
   • metric(s),
   • parameters,
   • the observation budget (the anticipated number of iterations), and
   • additional experiment features.

   # sigopt.yml
   experiment:
     name: Multi-Layer Perceptron
     parameters:
       - name: num_hidden_nodes
         bounds:
           min: 10
           max: 784
         type: int
       - name: learning_rate
         bounds:
           min: 1.0e-8
           max: 1.0
         type: double
         transformation: log
       - name: batch_size
         bounds:
           min: 50
           max: 400
         type: int
       - name: epochs
         bounds:
           min: 5
           max: 20
         type: int
     metrics:
       - name: accuracy
         objective: maximize
     observation_budget: 50
   

   A few notes:

   • Multitask is not currently compatible with the Beta.
   • If your sigopt.yml file is not in the directory where you are running sigopt optimize, you can specify the location of the sigopt.yml file with an additional --sigopt-file flag.
   • If you're working in an IPython notebook, you can define the experiment using YAML, JSON, or a Python dictionary in a cell, as shown on this page.

3. Remove the Optimization Loop Back to Top

   Next, remove the optimization loop that relies on the observation budget.

   This will include removing:

   • the Connection import and object,
   • the while or for loop that relies on the experiment.progress.count and the experiment.observation_budget,
   • the suggestion creation experiments(experiment.id).suggestions().create(), and
   • the observation creation experiments(experiment.id).observations().create(), and
   • any updates to the experiment object.

   Make sure you have imported SigOpt, import sigopt.

   We've left a gap in our code now where we used the suggestion to provide parameters. We'll fix that in the next step.

   # example.py
   from tensorflow.examples.tutorials.mnist import input_data
   import tensorflow as tf
   
   import sigopt
   
   mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
   
   input_features = 784
   x = tf.placeholder("float", [None, input_features])
   output_classes = 10
   y = tf.placeholder("float", [None, output_classes])
   
   with tf.Session() as sess:
     batch_size = suggestion.assignments['batch_size']
     epochs = suggestion.assignments['epochs']
     num_batches = mnist.train.num_examples // batch_size
   
     num_hidden_nodes = suggestion.assignments['num_hidden_nodes']
     hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes]))
     hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes]))
     hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases))
   
     activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes]))
     activation_biases = tf.Variable(tf.random_normal([output_classes]))
     activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases))
   
     learning_rate = suggestion.assignments['learning_rate']
     cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=activation_layer, labels=y))
     optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
   
     sess.run(tf.global_variables_initializer())
   
     for i in range(epochs):
       for _ in range(num_batches):
         batch_x, batch_y = mnist.train.next_batch(batch_size)
         sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})
     correct_prediction = tf.equal(tf.argmax(activation_layer, 1), tf.argmax(y, 1))
     accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
     accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels})
   

4. Get Parameters Back to Top

   To make this Python file usable for both experiments and single training runs, sigopt.get_parameter will fetch the appropriate parameter for a given iteration of an experiment, or, for a single training run, use the default parameter.

   We'll do that here for batch_size, epochs, num_hidden_nodes, and learning_rate.

   For the learning_rate, you can see we're making use of the logarithmic Parameter Transformation feature.

   # example.py
   from tensorflow.examples.tutorials.mnist import input_data
   import tensorflow as tf
   
   import sigopt
   
   mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
   
   input_features = 784
   x = tf.placeholder("float", [None, input_features])
   output_classes = 10
   y = tf.placeholder("float", [None, output_classes])
   
   with tf.Session() as sess:
     batch_size = sigopt.get_parameter('batch_size', default=100)
     epochs = sigopt.get_parameter('epochs', default=5)
     num_batches = mnist.train.num_examples // batch_size
   
     num_hidden_nodes = sigopt.get_parameter('num_hidden_nodes', default=10)
     hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes]))
     hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes]))
     hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases))
   
     activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes]))
     activation_biases = tf.Variable(tf.random_normal([output_classes]))
     activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases))
   
     learning_rate = sigopt.get_parameter('learning_rate', default=1e-3)
     cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=activation_layer, labels=y))
     optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
   
     sess.run(tf.global_variables_initializer())
   
     for i in range(epochs):
       for _ in range(num_batches):
         batch_x, batch_y = mnist.train.next_batch(batch_size)
         sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})
     correct_prediction = tf.equal(tf.argmax(activation_layer, 1), tf.argmax(y, 1))
     accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
     accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels})
   

5. Record Metrics Back to Top

   Next, we need to report the metric(s) for a training run or an experiment iteration. For this example, we report one metric with sigopt.log_metric('accuracy', accuracy_value) .

   # example.py
   from tensorflow.examples.tutorials.mnist import input_data
   import tensorflow as tf
   
   import sigopt
   
   mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
   
   input_features = 784
   x = tf.placeholder("float", [None, input_features])
   output_classes = 10
   y = tf.placeholder("float", [None, output_classes])
   
   with tf.Session() as sess:
     batch_size = sigopt.get_parameter('batch_size', default=100)
     epochs = sigopt.get_parameter('epochs', default=5)
     num_batches = mnist.train.num_examples // batch_size
   
     num_hidden_nodes = sigopt.get_parameter('num_hidden_nodes', default=10)
     hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes]))
     hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes]))
     hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases))
   
     activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes]))
     activation_biases = tf.Variable(tf.random_normal([output_classes]))
     activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases))
   
     learning_rate = sigopt.get_parameter('learning_rate', default=1e-3)
     cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=activation_layer, labels=y))
     optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
   
     sess.run(tf.global_variables_initializer())
   
     for i in range(epochs):
       for _ in range(num_batches):
         batch_x, batch_y = mnist.train.next_batch(batch_size)
         sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})
     correct_prediction = tf.equal(tf.argmax(activation_layer, 1), tf.argmax(y, 1))
     accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
     accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels})
     sigopt.log_metric('accuracy', accuracy_value)
   

6. Record Additional Information Back to Top

   We recommend defining the following optional fields inline so that this data will be included when a single training run is created with the SigOpt command line tool, not only in experiments.

   • sigopt.log_metadata('key', 'value') allows you to log metadata anywhere it is convenient.
   • sigopt.log_model(type='xgboost') will store a string for identifying the model used.
   • sigopt.log_dataset(name='minst') will store an identifier for the dataset used.

   # example.py
   from tensorflow.examples.tutorials.mnist import input_data
   import tensorflow as tf
   
   import sigopt
   
   sigopt.log_dataset(name='mnist')
   mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
   
   input_features = 784
   sigopt.log_metadata('input_features', input_features)
   x = tf.placeholder("float", [None, input_features])
   output_classes = 10
   sigopt.log_metadata('output_classes', output_classes)
   y = tf.placeholder("float", [None, output_classes])
   
   sigopt.log_model(type='Multi Layer Perceptron')
   
   with tf.Session() as sess:
     batch_size = sigopt.get_parameter('batch_size', default=100)
     epochs = sigopt.get_parameter('epochs', default=5)
     num_batches = mnist.train.num_examples // batch_size
   
     num_hidden_nodes = sigopt.get_parameter('num_hidden_nodes', default=10)
     hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes]))
     hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes]))
     hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases))
   
     activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes]))
     activation_biases = tf.Variable(tf.random_normal([output_classes]))
     activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases))
   
     learning_rate = sigopt.get_parameter('learning_rate', default=1e-3)
     cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=activation_layer, labels=y))
     optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
   
     sess.run(tf.global_variables_initializer())
   
     for i in range(epochs):
       for _ in range(num_batches):
         batch_x, batch_y = mnist.train.next_batch(batch_size)
         sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})
     correct_prediction = tf.equal(tf.argmax(activation_layer, 1), tf.argmax(y, 1))
     accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
     accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels})
     sigopt.log_metric('accuracy', accuracy_value)
   

7. Set the Project Back to Top

   Training runs are created within projects. The project allows you to sort and filter through your training runs and view useful charts to gain insight into everything you've tried.

   By default, the name of the current directory will be the project name and ID associated with the training run.

   If you'd like to choose a different project, you can override the default by setting an environment variable.

   Run the following code in your terminal, replacing run_example with your project ID.

   export SIGOPT_PROJECT=run_example

   If the valid Project ID doesn't already exist, a project will be created.

   See our notebook environment page for instructions on setting the project in a notebook.

8. Set the API Token Back to Top

   Set your token via sigopt config. More details are available on Get Started.

9. Run an Experiment Back to Top

   Now you're ready to give your code a test run!

   sigopt optimize example.py

   If you have arguments to pass to your Python file, the SigOpt client will pass them along, for example:

   sigopt optimize example.py arg1 arg2

Limitations

• Multitask is not supported.
• Parallelism is not supported.

Conclusion

We hope this article has helped you convert your existing SigOpt experiment files with little fuss. Please reach out to us with any further questions.