Optimization & Exploration
In the guides for notebook and command line development environments, we've covered the recommended way to instrument your code with SigOpt for a training run and shown how that can be easily extended to run SigOpt experiments. In this article, we'll discuss more of the options for running a SigOpt experiment.
If you've used SigOpt before, you will find our Convert an Existing Experiment guide useful.
Optimization
After your model has been instrumented, it is easy to take advantage of SigOpt's optimization features. Optimization helps find the parameters for your model that give you the best metric (eg. maximizing an accuracy metric).
An experiment is a definition of the hyperparameter search space and the metrics you would like optimized.
The experiment can be defined in either a YAML
or JSON
format. In a notebook, this will be defined using the %%experiment
magic command in a cell.
In a text editor, you'll create a sigopt.yml
file in the directory from which you'll be running sigopt optimize example.py
Below, we have an example optimization which finds parameters to maximize test_accuracy
for a Tensorflow model:
Text Editor & CLI Optimization Example
Install & Configure SigOpt
If you haven't already, follow the installation and configuration instructions in our Get Started page.
Set Project
Add Sigopt methods to capture data about your training run.
See Record a Run for more details about setting the project ID. The API Reference is a definitive list of the SigOpt methods available to instrument your run.
If you don't set a
$SIGOPT_PROJECT
environment variable, SigOpt will use the current directory name as the project ID.Instrument Your Model
See Record a Run for more details about how the following example run was instrumented.
# example.py from tensorflow.examples.tutorials.mnist import input_data import tensorflow as tf import sigopt sigopt.log_dataset(name='mnist') mnist = input_data.read_data_sets("/tmp/data/", one_hot=True) input_features = 784 sigopt.log_metadata('input_features', input_features) x = tf.placeholder("float", [None, input_features]) output_classes = 10 sigopt.log_metadata('output_classes', output_classes) y = tf.placeholder("float", [None, output_classes]) sigopt.log_model(type='Multi Layer Perceptron') num_hidden_nodes = sigopt.get_parameter('num_hidden_nodes', default=10) hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes])) hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes])) hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases)) activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes])) activation_biases = tf.Variable(tf.random_normal([output_classes])) activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases)) learning_rate = sigopt.get_parameter('learning_rate', default=1e-3) cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=activation_layer, labels=y)) optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost) with tf.Session() as sess: sess.run(tf.global_variables_initializer()) batch_size = sigopt.get_parameter('batch_size', default=100) epochs = sigopt.get_parameter('epochs', default=5) num_batches = mnist.train.num_examples // batch_size for i in range(epochs): for _ in range(num_batches): batch_x, batch_y = mnist.train.next_batch(batch_size) sess.run(optimizer, feed_dict={x: batch_x, y: batch_y}) correct_prediction = tf.equal(tf.argmax(activation_layer, 1), tf.argmax(y, 1)) accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float")) accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels}) sigopt.log_metric('accuracy', accuracy_value)
Define the Experiment
Create a
sigopt.yml
file to define your experiment. The file should have an experiment section with the name, project, parameters, metrics and other options that you would like to run your experiment with.The parameters should match the names provided to
sigopt.get_parameter
in your Python code and similarly the metrics should match the names of thesigopt.log_metric
calls.Here's an example file:
# sigopt.yml experiment: name: Multi-Layer Perceptron parameters: - name: num_hidden_nodes bounds: min: 10 max: 784 type: int - name: learning_rate bounds: min: 1.0e-8 max: 1.0 type: double transformation: log - name: batch_size bounds: min: 50 max: 400 type: int - name: epochs bounds: min: 5 max: 20 type: int metrics: - name: accuracy objective: maximize observation_budget: 50
Note: If your
sigopt.yml
file is not in the directory where you are runningsigopt optimize
, you can specify the location of thesigopt.yml
file with an additional--sigopt-file
flag:$ sigopt optimize example.py --sigopt-file example/sigopt.yml
Run the Optimization
# terminal $ sigopt optimize example.py
Note: Any extra arguments will be passed along to your Python executable. For example:
$ sigopt optimize example.py arg1 arg2
View the Results
The output will provide links to the SigOpt web app where you can navigate to recent runs and experiments.
Notebook Optimization Example
Install & Configure SigOpt
If you haven't already, follow the installation and configuration instructions in our Get Started page.
Set the Project
from tensorflow.examples.tutorials.mnist import input_data import tensorflow as tf import sigopt %load_ext sigopt import os os.environ['SIGOPT_PROJECT'] = 'run-examples' mnist = input_data.read_data_sets("/tmp/data/", one_hot=True) input_features = 784 output_classes = 10
Instrument Your Model
See Record a Run for more details about how the following example run was instrumented.
def create_model(x): num_hidden_nodes = sigopt.get_parameter('num_hidden_nodes', default=10) hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes])) hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes])) hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases)) activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes])) activation_biases = tf.Variable(tf.random_normal([output_classes])) activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases)) return activation_layer
def train_model(sess, optimizer, x, y): batch_size = sigopt.get_parameter('batch_size', default=100) epochs = sigopt.get_parameter('epochs', default=5) num_batches = mnist.train.num_examples // batch_size for i in range(epochs): for _ in range(num_batches): batch_x, batch_y = mnist.train.next_batch(batch_size) sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})
def evaluate_model(sess, model, x, y): correct_prediction = tf.equal(tf.argmax(model, 1), tf.argmax(y, 1)) accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float")) accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels}) sigopt.log_metric('accuracy', accuracy_value)
Define the Experiment
The experiment definition will include the name, project, parameters, metrics and other options that you would like to run your experiment with.
The parameters should match the names provided to
sigopt.get_parameter
in your Python code and similarly the metrics should match the names of thesigopt.log_metric
calls.You can format the experiment definition in Python, YAML, or JSON. In this example, we're using Python. See below for YAML and JSON examples. With Python, you have the advantage of using local variables if you wish.
%%experiment # define the parameters, metrics and budget for your experiment { 'metrics': [ { 'name': 'accuracy', 'objective': 'maximize' } ], 'name': 'Multi-Layer Perceptron', 'observation_budget': 5, 'parameters': [ { 'bounds': { 'max': 784, 'min': 10 }, 'name': 'num_hidden_nodes', 'type': 'int' }, { 'bounds': { 'max': 1.0, 'min': 1.0e-8 }, 'name': 'learning_rate', 'type': 'double', 'transformation': 'log' }, { 'bounds': { 'max': 400, 'min': 50 }, 'name': 'batch_size', 'type': 'int' }, { 'bounds': { 'max': 20, 'min': 5 }, 'name': 'epochs', 'type': 'int' } ] }
Note: The observation budget is the number of runs you would like created in the optimization. We recommend using 5 for testing. When you're ready, visit the observation budget page to learn our rule of thumb for the appropriate observation budget.
Run the Optimization
%%optimize # optimize the model over multiple iterations # use %%run instead to only record a single run using the default parameters x = tf.placeholder("float", [None, input_features]) y = tf.placeholder("float", [None, output_classes]) sigopt.log_dataset(name='mnist') sigopt.log_metadata('input_features', input_features) sigopt.log_metadata('output_classes', output_classes) sigopt.log_model(type='Multi Layer Perceptron') model = create_model(x) learning_rate = sigopt.get_parameter('learning_rate', default=1e-3) cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=model, labels=y)) optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost) with tf.Session() as sess: sess.run(tf.global_variables_initializer()) train_model(sess, optimizer, x, y) evaluate_model(sess, model, x, y)
View the Results
The output will provide links to the SigOpt web app where you can navigate to recent runs and experiments.
Formatting
When working in a notebook environment, in addition to Python, you have the option to define your experiment in YAML or JSON. Here are examples of both:
YAML
%%experiment # define the parameters, metrics and budget for your experiment ''' name: Multi-Layer Perceptron parameters: - name: num_hidden_nodes bounds: min: 10 max: 784 type: int - name: learning_rate bounds: min: 1.0e-8 max: 1.0 type: double transformation: log - name: batch_size bounds: min: 50 max: 400 type: int - name: epochs bounds: min: 5 max: 20 type: int metrics: - name: accuracy objective: maximize observation_budget: 5 '''
JSON
%%experiment # define the parameters, metrics and budget for your experiment ''' { "name": "Multi-Layer Perceptron", "parameters": [ { "name": "num_hidden_nodes", "bounds": { "min": 10, "max": 784 }, "type": "int" }, { "name": "learning_rate", "bounds": { "min": 1.0e-8, "max": 1.0 }, "type": "double", "transformation": "log" }, { "name": "batch_size", "bounds": { "min": 50, "max": 400 }, "type": "int" }, { "name": "epochs", "bounds": { "min": 5, "max": 20 }, "type": "int" } ], "metrics": [ { "name": "accuracy", "objective": "maximize" } ], "observation_budget": 5 } '''
Exploration
You might want to use grid or random searches to explore a space. Please contact our team to learn how to use these experiment options.