Convert Existing SigOpt Code
If you've been using SigOpt already, this article will help you convert your existing SigOpt experiment code to use the new SigOpt methods, using a Python file as an example. Variations in converting code in a notebook environment are called out.
However, if you wish, your old code will still work with the newest version of our Python package. Multitask and Parallelism are not currently compatible with the beta command line tool.
Overview
Instead of executing a Python file that contains an optimization loop, we pass a YAML file and a Python file to the SigOpt command line tool. The SigOpt command line tool will create the experiment and execute the Python file once for each iteration of the experiment. This way, the Python file can be used both to execute a single training run and an entire experiment, without modification.
In this article, we'll start with an example of a SigOpt experiment and walk through the following steps of converting it to the new SigOpt method:
- Update SigOpt
- Define the Experiment
- Remove the Optimization Loop
- Get Parameters
- Record Metrics
- Record Additional Information
- Set the Project
- Set the API Token
- Run an Experiment
Example Code
Here's our example Python code that uses the older version of SigOpt. Note: This example has been tested with Tensorflow 1.14.
# example.py
from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf
from sigopt import Connection
conn = Connection(client_token=SIGOPT_API_TOKEN)
mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
input_features = 784
x = tf.placeholder("float", [None, input_features])
output_classes = 10
y = tf.placeholder("float", [None, output_classes])
experiment = conn.experiments().create(
name='Multi-Layer Perceptron',
parameters=[
dict(name='num_hidden_nodes', type='int', bounds=dict(min=10, max=784)),
dict(name='learning_rate', type='double', bounds=dict(min=1.0e-8, max=1.0), transformation='log'),
dict(name='batch_size', type='int', bounds=dict(min=50, max=400)),
dict(name='epochs', type='int', bounds=dict(min=5, max=20)),
],
metrics=[dict(name='accuracy', objective='maximize')],
observation_budget=50,
project='sigopt-examples',
metadata=dict(
input_features=input_features,
output_classes=output_classes,
model='Multi Layer Perceptron',
dataset='mnist'
),
)
with tf.Session() as sess:
while experiment.progress.observation_count < experiment.observation_budget:
suggestion = conn.experiments(experiment.id).suggestions().create()
batch_size = suggestion.assignments['batch_size']
epochs = suggestion.assignments['epochs']
num_batches = mnist.train.num_examples // batch_size
num_hidden_nodes = suggestion.assignments['num_hidden_nodes']
hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes]))
hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes]))
hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases))
activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes]))
activation_biases = tf.Variable(tf.random_normal([output_classes]))
activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases))
learning_rate = suggestion.assignments['learning_rate']
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=activation_layer, labels=y))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
sess.run(tf.global_variables_initializer())
for i in range(epochs):
for _ in range(num_batches):
batch_x, batch_y = mnist.train.next_batch(batch_size)
sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})
correct_prediction = tf.equal(tf.argmax(activation_layer, 1), tf.argmax(y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels})
conn.experiments(experiment.id).observations().create(
suggestion=suggestion.id,
values=[
dict(
name='accuracy',
value=accuracy_value,
)
],
)
experiment = conn.experiments(experiment.id).fetch()
Conversion Steps
Update SigOpt Back to Top
If you haven't already, please install the latest version of the SigOpt Python client.
pip install --upgrade https://public.sigopt.com/experiment-management/sigopt-7.1.5-py2.py3-none-any.whl
Define the Experiment Back to Top
Create a
sigopt.yml
file to define your experiment. This will include:- the experiment name,
- metric(s),
- parameters,
- the observation budget (the anticipated number of iterations), and
- additional experiment features.
# sigopt.yml experiment: name: Multi-Layer Perceptron parameters: - name: num_hidden_nodes bounds: min: 10 max: 784 type: int - name: learning_rate bounds: min: 1.0e-8 max: 1.0 type: double transformation: log - name: batch_size bounds: min: 50 max: 400 type: int - name: epochs bounds: min: 5 max: 20 type: int metrics: - name: accuracy objective: maximize observation_budget: 50
A few notes:
- Multitask is not currently compatible with the Beta.
- If your
sigopt.yml
file is not in the directory where you are runningsigopt optimize
, you can specify the location of thesigopt.yml
file with an additional--sigopt-file
flag. - If you're working in an IPython notebook, you can define the experiment using YAML, JSON, or a Python dictionary in a cell, as shown on this page.
Remove the Optimization Loop Back to Top
Next, remove the optimization loop that relies on the observation budget.
This will include removing:
- the
Connection
import and object, - the
while
orfor
loop that relies on theexperiment.progress.count
and theexperiment.observation_budget
, - the suggestion creation
experiments(experiment.id).suggestions().create()
, and - the observation creation
experiments(experiment.id).observations().create()
, and - any updates to the experiment object.
Make sure you have imported SigOpt,
import sigopt
.We've left a gap in our code now where we used the
suggestion
to provide parameters. We'll fix that in the next step.# example.py from tensorflow.examples.tutorials.mnist import input_data import tensorflow as tf import sigopt mnist = input_data.read_data_sets("/tmp/data/", one_hot=True) input_features = 784 x = tf.placeholder("float", [None, input_features]) output_classes = 10 y = tf.placeholder("float", [None, output_classes]) with tf.Session() as sess: batch_size = suggestion.assignments['batch_size'] epochs = suggestion.assignments['epochs'] num_batches = mnist.train.num_examples // batch_size num_hidden_nodes = suggestion.assignments['num_hidden_nodes'] hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes])) hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes])) hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases)) activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes])) activation_biases = tf.Variable(tf.random_normal([output_classes])) activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases)) learning_rate = suggestion.assignments['learning_rate'] cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=activation_layer, labels=y)) optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost) sess.run(tf.global_variables_initializer()) for i in range(epochs): for _ in range(num_batches): batch_x, batch_y = mnist.train.next_batch(batch_size) sess.run(optimizer, feed_dict={x: batch_x, y: batch_y}) correct_prediction = tf.equal(tf.argmax(activation_layer, 1), tf.argmax(y, 1)) accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float")) accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels})
- the
Get Parameters Back to Top
To make this Python file usable for both experiments and single training runs,
sigopt.get_parameter
will fetch the appropriate parameter for a given iteration of an experiment, or, for a single training run, use the default parameter.We'll do that here for
batch_size
,epochs
,num_hidden_nodes
, andlearning_rate
.For the
learning_rate
, you can see we're making use of the logarithmic Parameter Transformation feature.# example.py from tensorflow.examples.tutorials.mnist import input_data import tensorflow as tf import sigopt mnist = input_data.read_data_sets("/tmp/data/", one_hot=True) input_features = 784 x = tf.placeholder("float", [None, input_features]) output_classes = 10 y = tf.placeholder("float", [None, output_classes]) with tf.Session() as sess: batch_size = sigopt.get_parameter('batch_size', default=100) epochs = sigopt.get_parameter('epochs', default=5) num_batches = mnist.train.num_examples // batch_size num_hidden_nodes = sigopt.get_parameter('num_hidden_nodes', default=10) hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes])) hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes])) hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases)) activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes])) activation_biases = tf.Variable(tf.random_normal([output_classes])) activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases)) learning_rate = sigopt.get_parameter('learning_rate', default=1e-3) cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=activation_layer, labels=y)) optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost) sess.run(tf.global_variables_initializer()) for i in range(epochs): for _ in range(num_batches): batch_x, batch_y = mnist.train.next_batch(batch_size) sess.run(optimizer, feed_dict={x: batch_x, y: batch_y}) correct_prediction = tf.equal(tf.argmax(activation_layer, 1), tf.argmax(y, 1)) accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float")) accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels})
Record Metrics Back to Top
Next, we need to report the metric(s) for a training run or an experiment iteration. For this example, we report one metric with
sigopt.log_metric('accuracy', accuracy_value)
.# example.py from tensorflow.examples.tutorials.mnist import input_data import tensorflow as tf import sigopt mnist = input_data.read_data_sets("/tmp/data/", one_hot=True) input_features = 784 x = tf.placeholder("float", [None, input_features]) output_classes = 10 y = tf.placeholder("float", [None, output_classes]) with tf.Session() as sess: batch_size = sigopt.get_parameter('batch_size', default=100) epochs = sigopt.get_parameter('epochs', default=5) num_batches = mnist.train.num_examples // batch_size num_hidden_nodes = sigopt.get_parameter('num_hidden_nodes', default=10) hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes])) hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes])) hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases)) activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes])) activation_biases = tf.Variable(tf.random_normal([output_classes])) activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases)) learning_rate = sigopt.get_parameter('learning_rate', default=1e-3) cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=activation_layer, labels=y)) optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost) sess.run(tf.global_variables_initializer()) for i in range(epochs): for _ in range(num_batches): batch_x, batch_y = mnist.train.next_batch(batch_size) sess.run(optimizer, feed_dict={x: batch_x, y: batch_y}) correct_prediction = tf.equal(tf.argmax(activation_layer, 1), tf.argmax(y, 1)) accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float")) accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels}) sigopt.log_metric('accuracy', accuracy_value)
Record Additional Information Back to Top
We recommend defining the following optional fields inline so that this data will be included when a single training run is created with the SigOpt command line tool, not only in experiments.
sigopt.log_metadata('key', 'value')
allows you to log metadata anywhere it is convenient.sigopt.log_model(type='xgboost')
will store a string for identifying the model used.sigopt.log_dataset(name='minst')
will store an identifier for the dataset used.
# example.py from tensorflow.examples.tutorials.mnist import input_data import tensorflow as tf import sigopt sigopt.log_dataset(name='mnist') mnist = input_data.read_data_sets("/tmp/data/", one_hot=True) input_features = 784 sigopt.log_metadata('input_features', input_features) x = tf.placeholder("float", [None, input_features]) output_classes = 10 sigopt.log_metadata('output_classes', output_classes) y = tf.placeholder("float", [None, output_classes]) sigopt.log_model(type='Multi Layer Perceptron') with tf.Session() as sess: batch_size = sigopt.get_parameter('batch_size', default=100) epochs = sigopt.get_parameter('epochs', default=5) num_batches = mnist.train.num_examples // batch_size num_hidden_nodes = sigopt.get_parameter('num_hidden_nodes', default=10) hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes])) hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes])) hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases)) activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes])) activation_biases = tf.Variable(tf.random_normal([output_classes])) activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases)) learning_rate = sigopt.get_parameter('learning_rate', default=1e-3) cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=activation_layer, labels=y)) optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost) sess.run(tf.global_variables_initializer()) for i in range(epochs): for _ in range(num_batches): batch_x, batch_y = mnist.train.next_batch(batch_size) sess.run(optimizer, feed_dict={x: batch_x, y: batch_y}) correct_prediction = tf.equal(tf.argmax(activation_layer, 1), tf.argmax(y, 1)) accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float")) accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels}) sigopt.log_metric('accuracy', accuracy_value)
Set the Project Back to Top
Training runs are created within projects. The project allows you to sort and filter through your training runs and view useful charts to gain insight into everything you've tried.
By default, the name of the current directory will be the project name and ID associated with the training run.
If you'd like to choose a different project, you can override the default by setting an environment variable.
Run the following code in your terminal, replacing
run_example
with your project ID.export SIGOPT_PROJECT=run_example
If the valid Project ID doesn't already exist, a project will be created.
See our notebook environment page for instructions on setting the project in a notebook.
Set the API Token Back to Top
Set your token via
sigopt config
. More details are available on Get Started.Run an Experiment Back to Top
Now you're ready to give your code a test run!
sigopt optimize example.py
If you have arguments to pass to your Python file, the SigOpt client will pass them along, for example:
sigopt optimize example.py arg1 arg2
Limitations
- Multitask is not supported.
- Parallelism is not supported.
Conclusion
We hope this article has helped you convert your existing SigOpt experiment files with little fuss. Please reach out to us with any further questions.