Documentation

Welcome to SigOpt’s developer documentation. If you have a question you can’t answer, feel free to contact us!
This feature is currently in Beta. You can request free access on the main page of the beta or contact us directly for more information.

Convert Existing SigOpt Code

If you've been using SigOpt already, this article will help you convert your existing SigOpt experiment code to use the new SigOpt methods, using a Python file as an example. Variations in converting code in a notebook environment are called out.

However, if you wish, your old code will still work with the newest version of our Python package. Multitask and Parallelism are not currently compatible with the beta command line tool.

Overview

Instead of executing a Python file that contains an optimization loop, we pass a YAML file and a Python file to the SigOpt command line tool. The SigOpt command line tool will create the experiment and execute the Python file once for each iteration of the experiment. This way, the Python file can be used both to execute a single training run and an entire experiment, without modification.

In this article, we'll start with an example of a SigOpt experiment and walk through the following steps of converting it to the new SigOpt method:

  1. Update SigOpt
  2. Define the Experiment
  3. Remove the Optimization Loop
  4. Get Parameters
  5. Record Metrics
  6. Record Additional Information
  7. Set the Project
  8. Set the API Token
  9. Run an Experiment

Example Code

Here's our example Python code that uses the older version of SigOpt. Note: This example has been tested with Tensorflow 1.14.

# example.py
from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf

from sigopt import Connection

conn = Connection(client_token=SIGOPT_API_TOKEN)

mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)

input_features = 784
x = tf.placeholder("float", [None, input_features])
output_classes = 10
y = tf.placeholder("float", [None, output_classes])

experiment = conn.experiments().create(
  name='Multi-Layer Perceptron',
  parameters=[
    dict(name='num_hidden_nodes', type='int', bounds=dict(min=10, max=784)),
    dict(name='learning_rate', type='double', bounds=dict(min=1.0e-8, max=1.0), transformation='log'),
    dict(name='batch_size', type='int', bounds=dict(min=50, max=400)),
    dict(name='epochs', type='int', bounds=dict(min=5, max=20)),
  ],
  metrics=[dict(name='accuracy', objective='maximize')],
  observation_budget=50,
  project='sigopt-examples',
  metadata=dict(
      input_features=input_features,
      output_classes=output_classes,
      model='Multi Layer Perceptron',
      dataset='mnist'
    ),
)

with tf.Session() as sess:
  while experiment.progress.observation_count < experiment.observation_budget:
    suggestion = conn.experiments(experiment.id).suggestions().create()
    batch_size = suggestion.assignments['batch_size']
    epochs = suggestion.assignments['epochs']
    num_batches = mnist.train.num_examples // batch_size

    num_hidden_nodes = suggestion.assignments['num_hidden_nodes']
    hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes]))
    hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes]))
    hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases))

    activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes]))
    activation_biases = tf.Variable(tf.random_normal([output_classes]))
    activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases))

    learning_rate = suggestion.assignments['learning_rate']
    cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=activation_layer, labels=y))
    optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

    sess.run(tf.global_variables_initializer())

    for i in range(epochs):
      for _ in range(num_batches):
        batch_x, batch_y = mnist.train.next_batch(batch_size)
        sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})

    correct_prediction = tf.equal(tf.argmax(activation_layer, 1), tf.argmax(y, 1))
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
    accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels})
    conn.experiments(experiment.id).observations().create(
      suggestion=suggestion.id,
      values=[
        dict(
          name='accuracy',
          value=accuracy_value,
        )
      ],
    )
    experiment = conn.experiments(experiment.id).fetch()

Conversion Steps

  1. Update SigOpt Back to Top

    If you haven't already, please install the latest version of the SigOpt Python client.

    pip install --upgrade https://sigopt-python-mpm.s3-us-west-1.amazonaws.com/sigopt-7.1.4-py2.py3-none-any.whl
  2. Define the Experiment Back to Top

    Create a sigopt.yml file to define your experiment. This will include:

    • the experiment name,
    • metric(s),
    • parameters,
    • the observation budget (the anticipated number of iterations), and
    • additional experiment features.
    # sigopt.yml
    experiment:
      name: Multi-Layer Perceptron
      parameters:
        - name: num_hidden_nodes
          bounds:
            min: 10
            max: 784
          type: int
        - name: learning_rate
          bounds:
            min: 1.0e-8
            max: 1.0
          type: double
          transformation: log
        - name: batch_size
          bounds:
            min: 50
            max: 400
          type: int
        - name: epochs
          bounds:
            min: 5
            max: 20
          type: int
      metrics:
        - name: accuracy
          objective: maximize
      observation_budget: 50
    

    A few notes:

    • Multitask is not currently compatible with the Beta.
    • If your sigopt.yml file is not in the directory where you are running sigopt optimize, you can specify the location of the sigopt.yml file with an additional --sigopt-file flag.
    • If you're working in an IPython notebook, you can define the experiment using YAML, JSON, or a Python dictionary in a cell, as shown on this page.
  3. Remove the Optimization Loop Back to Top

    Next, remove the optimization loop that relies on the observation budget.

    This will include removing:

    • the Connection import and object,
    • the while or for loop that relies on the experiment.progress.count and the experiment.observation_budget,
    • the suggestion creation experiments(experiment.id).suggestions().create(), and
    • the observation creation experiments(experiment.id).observations().create(), and
    • any updates to the experiment object.

    Make sure you have imported SigOpt, import sigopt.

    We've left a gap in our code now where we used the suggestion to provide parameters. We'll fix that in the next step.

    # example.py
    from tensorflow.examples.tutorials.mnist import input_data
    import tensorflow as tf
    
    import sigopt
    
    mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
    
    input_features = 784
    x = tf.placeholder("float", [None, input_features])
    output_classes = 10
    y = tf.placeholder("float", [None, output_classes])
    
    with tf.Session() as sess:
      batch_size = suggestion.assignments['batch_size']
      epochs = suggestion.assignments['epochs']
      num_batches = mnist.train.num_examples // batch_size
    
      num_hidden_nodes = suggestion.assignments['num_hidden_nodes']
      hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes]))
      hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes]))
      hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases))
    
      activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes]))
      activation_biases = tf.Variable(tf.random_normal([output_classes]))
      activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases))
    
      learning_rate = suggestion.assignments['learning_rate']
      cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=activation_layer, labels=y))
      optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
    
      sess.run(tf.global_variables_initializer())
    
      for i in range(epochs):
        for _ in range(num_batches):
          batch_x, batch_y = mnist.train.next_batch(batch_size)
          sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})
      correct_prediction = tf.equal(tf.argmax(activation_layer, 1), tf.argmax(y, 1))
      accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
      accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels})
    
  4. Get Parameters Back to Top

    To make this Python file usable for both experiments and single training runs, sigopt.get_parameter will fetch the appropriate parameter for a given iteration of an experiment, or, for a single training run, use the default parameter.

    We'll do that here for batch_size, epochs, num_hidden_nodes, and learning_rate.

    For the learning_rate, you can see we're making use of the logarithmic Parameter Transformation feature.

    # example.py
    from tensorflow.examples.tutorials.mnist import input_data
    import tensorflow as tf
    
    import sigopt
    
    mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
    
    input_features = 784
    x = tf.placeholder("float", [None, input_features])
    output_classes = 10
    y = tf.placeholder("float", [None, output_classes])
    
    with tf.Session() as sess:
      batch_size = sigopt.get_parameter('batch_size', default=100)
      epochs = sigopt.get_parameter('epochs', default=5)
      num_batches = mnist.train.num_examples // batch_size
    
      num_hidden_nodes = sigopt.get_parameter('num_hidden_nodes', default=10)
      hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes]))
      hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes]))
      hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases))
    
      activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes]))
      activation_biases = tf.Variable(tf.random_normal([output_classes]))
      activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases))
    
      learning_rate = sigopt.get_parameter('learning_rate', default=1e-3)
      cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=activation_layer, labels=y))
      optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
    
      sess.run(tf.global_variables_initializer())
    
      for i in range(epochs):
        for _ in range(num_batches):
          batch_x, batch_y = mnist.train.next_batch(batch_size)
          sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})
      correct_prediction = tf.equal(tf.argmax(activation_layer, 1), tf.argmax(y, 1))
      accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
      accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels})
    
  5. Record Metrics Back to Top

    Next, we need to report the metric(s) for a training run or an experiment iteration. For this example, we report one metric with sigopt.log_metric('accuracy', accuracy_value) .

    # example.py
    from tensorflow.examples.tutorials.mnist import input_data
    import tensorflow as tf
    
    import sigopt
    
    mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
    
    input_features = 784
    x = tf.placeholder("float", [None, input_features])
    output_classes = 10
    y = tf.placeholder("float", [None, output_classes])
    
    with tf.Session() as sess:
      batch_size = sigopt.get_parameter('batch_size', default=100)
      epochs = sigopt.get_parameter('epochs', default=5)
      num_batches = mnist.train.num_examples // batch_size
    
      num_hidden_nodes = sigopt.get_parameter('num_hidden_nodes', default=10)
      hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes]))
      hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes]))
      hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases))
    
      activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes]))
      activation_biases = tf.Variable(tf.random_normal([output_classes]))
      activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases))
    
      learning_rate = sigopt.get_parameter('learning_rate', default=1e-3)
      cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=activation_layer, labels=y))
      optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
    
      sess.run(tf.global_variables_initializer())
    
      for i in range(epochs):
        for _ in range(num_batches):
          batch_x, batch_y = mnist.train.next_batch(batch_size)
          sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})
      correct_prediction = tf.equal(tf.argmax(activation_layer, 1), tf.argmax(y, 1))
      accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
      accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels})
      sigopt.log_metric('accuracy', accuracy_value)
    
  6. Record Additional Information Back to Top

    We recommend defining the following optional fields inline so that this data will be included when a single training run is created with the SigOpt command line tool, not only in experiments.

    • sigopt.log_metadata('key', 'value') allows you to log metadata anywhere it is convenient.
    • sigopt.log_model(type='xgboost') will store a string for identifying the model used.
    • sigopt.log_dataset(name='minst') will store an identifier for the dataset used.
    # example.py
    from tensorflow.examples.tutorials.mnist import input_data
    import tensorflow as tf
    
    import sigopt
    
    sigopt.log_dataset(name='mnist')
    mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
    
    input_features = 784
    sigopt.log_metadata('input_features', input_features)
    x = tf.placeholder("float", [None, input_features])
    output_classes = 10
    sigopt.log_metadata('output_classes', output_classes)
    y = tf.placeholder("float", [None, output_classes])
    
    sigopt.log_model(type='Multi Layer Perceptron')
    
    with tf.Session() as sess:
      batch_size = sigopt.get_parameter('batch_size', default=100)
      epochs = sigopt.get_parameter('epochs', default=5)
      num_batches = mnist.train.num_examples // batch_size
    
      num_hidden_nodes = sigopt.get_parameter('num_hidden_nodes', default=10)
      hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes]))
      hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes]))
      hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases))
    
      activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes]))
      activation_biases = tf.Variable(tf.random_normal([output_classes]))
      activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases))
    
      learning_rate = sigopt.get_parameter('learning_rate', default=1e-3)
      cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=activation_layer, labels=y))
      optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
    
      sess.run(tf.global_variables_initializer())
    
      for i in range(epochs):
        for _ in range(num_batches):
          batch_x, batch_y = mnist.train.next_batch(batch_size)
          sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})
      correct_prediction = tf.equal(tf.argmax(activation_layer, 1), tf.argmax(y, 1))
      accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
      accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels})
      sigopt.log_metric('accuracy', accuracy_value)
    
  7. Set the Project Back to Top

    Training runs are created within projects. The project allows you to sort and filter through your training runs and view useful charts to gain insight into everything you've tried.

    By default, the name of the current directory will be the project name and ID associated with the training run.

    If you'd like to choose a different project, you can override the default by setting an environment variable.

    Run the following code in your terminal, replacing run_example with your project ID.

    export SIGOPT_PROJECT=run_example

    If the valid Project ID doesn't already exist, a project will be created.

    See our notebook environment page for instructions on setting the project in a notebook.

  8. Set the API Token Back to Top

    Set your token via sigopt config. More details are available on Get Started.

  9. Run an Experiment Back to Top

    Now you're ready to give your code a test run!

    sigopt optimize example.py

    If you have arguments to pass to your Python file, the SigOpt client will pass them along, for example:

    sigopt optimize example.py arg1 arg2

Limitations

Conclusion

We hope this article has helped you convert your existing SigOpt experiment files with little fuss. Please reach out to us with any further questions.