Record a Training Run (Notebook)
We'll walk through an example of instrumenting and executing training run code to make use of SigOpt.
How a Training Run is Recorded
A SigOpt training run is an object that records one or more of the following attributes:
- the model type,
- dataset identifier,
- evaluation metrics,
- hyperparameters,
- logs, and
- a code reference.
Take a look at the following example.
Example
Install & Configure
If you haven't already, follow the installation and configuration instructions in our Get Started page.
Set Your Project
Training runs are created within projects. The project allows you to sort and filter through your training runs and view useful charts to gain insight into everything you've tried.
We recommend setting the project name as an environment variable. In this example, we're naming our project
run-examples
. Note: This example has been tested with Tensorflow 1.14.from tensorflow.examples.tutorials.mnist import input_data import tensorflow as tf import sigopt %load_ext sigopt import os os.environ['SIGOPT_PROJECT'] = 'run-examples'
If the valid Project ID doesn't already exist, a project will be created.
Instrument the Training Run
In another cell, run your training run code, with SigOpt methods integrated.
%%run Example Run sigopt.log_dataset(name='mnist') mnist = input_data.read_data_sets("/tmp/data/", one_hot=True) input_features = 784 sigopt.log_metadata('input_features', input_features) x = tf.placeholder("float", [None, input_features]) output_classes = 10 sigopt.log_metadata('output_classes', output_classes) y = tf.placeholder("float", [None, output_classes]) sigopt.log_model(type='Multi Layer Perceptron') num_hidden_nodes = sigopt.get_parameter('num_hidden_nodes', default=10) hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes])) hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes])) hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases)) activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes])) activation_biases = tf.Variable(tf.random_normal([output_classes])) activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases)) learning_rate = sigopt.get_parameter('learning_rate', default=1e-3) cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=activation_layer, labels=y)) optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost) with tf.Session() as sess: sess.run(tf.global_variables_initializer()) batch_size = sigopt.get_parameter('batch_size', default=100) epochs = sigopt.get_parameter('epochs', default=5) num_batches = mnist.train.num_examples // batch_size for i in range(epochs): for _ in range(num_batches): batch_x, batch_y = mnist.train.next_batch(batch_size) sess.run(optimizer, feed_dict={x: batch_x, y: batch_y}) correct_prediction = tf.equal(tf.argmax(activation_layer, 1), tf.argmax(y, 1)) accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float")) accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels}) sigopt.log_metric('accuracy', accuracy_value)
Run the Code
Once you've run the code above, SigOpt will conveniently output links to the run page on our web application.
Run started, view it on the SigOpt dashboard at https://app.sigopt.com/run/1 Extracting /tmp/data/train-images-idx3-ubyte.gz Extracting /tmp/data/train-labels-idx1-ubyte.gz Extracting /tmp/data/t10k-images-idx3-ubyte.gz Extracting /tmp/data/t10k-labels-idx1-ubyte.gz Run finished, view it on the SigOpt dashboard at https://app.sigopt.com/run/1
View the Record
Now, visit the project in the SigOpt web app to see the results. Here's a view of the training run record:
At the project level, you can compare the training runs.
Overview of the SigOpt Methods
The example above uses the following methods (links to the specifications):
Let's discuss a few notable features below.
sigopt.get_parameter
Parameters are any input to your model that you may change as you seek to refine the model. In our example, we have the number of hidden nodes, batch size, and number of epochs. If you think you might want to optimize on a parameter, that's a good reason to include it as a parameter.
During a single training run, sigopt.get_parameter
utilizes the default
value. If you're interested in optimizing or exploring a hyperparameter (or two), visit the Optimize & Explore section to learn more about how sigopt.get_parameter
can return a suggested parameter SigOpt suggests within the search space. With a single method, sigopt.get_parameter
, you can instrument your code once and use it to create a single training run or to automate an experiment.
sigopt.log_model
sigopt.log_model
stores a text value that you can use to filter your runs in the SigOpt web view. In our example, you might want to filter by sklearn - logistic regression
to compare models of the same type in the web charts.
sigopt.log_dataset
Name your dataset with sigopt.log_dataset
so you can easily reference it later. Store the path to AWS S3 or a csv filename, whatever might be useful to you.
sigopt.log_metadata
Record any key value pairs you can imagine with sigopt.log_metadata
.
In our example, we've used sigopt.log_metadata
to store the input features and output classes, values we might want to change and compare across training runs but aren't data we'd consider parameters.
sigopt.log_metric
The most important information about a model is how it performed. With sigopt.log_metric
you can take advantage of SigOpt's analysis dashboard of custom charts and advanced sorting and filtering.
Next
Optimize & ExploreSigOpt has long been used to automate the tuning of a model. The next section, Optimize & Explore, shows how you can set up SigOpt experiments that will generate hyperparameter configurations with either a simple grid search or our Bayesian ensemble.