Record a Training Run (Text Editor & CLI)
We'll walk through an example of instrumenting and executing training run code to make use of SigOpt.
How a Training Run is Recorded
A SigOpt training run is an object that records one or more of the following attributes:
- the model type,
- dataset identifier,
- evaluation metrics,
- hyperparameters,
- logs, and
- a code reference.
The SigOpt package provides a command line wrapper that allows you to run Python files as you normally would in the terminal.
If instrumented in the way described below, you can take advantage of SigOpt to use the same file for single training runs:
sigopt run foo.py
and SigOpt experiments:
sigopt optimize foo.py
Take a look at the following example.
Example
Install & Configure
If you haven't already, follow the installation and configuration instructions in our Get Started page.
Set Your Project
Training runs are created within projects. The project allows you to sort and filter through your training runs and view useful charts to gain insight into everything you've tried.
By default, the name of the current directory will be the project name and ID associated with the training run.
If you'd like to choose a different project, you can override the default by setting an environment variable.
Run the following code in your terminal, replacing
run_example
with your project ID.export SIGOPT_PROJECT=run_example
If the valid Project ID doesn't already exist, a project will be created.
Instrument the Training Run
This example training run code shows how to integrate the key SigOpt calls to record your training run. Note: This example has been tested with Tensorflow 1.14.
# example.py from tensorflow.examples.tutorials.mnist import input_data import tensorflow as tf import sigopt sigopt.log_dataset(name='mnist') mnist = input_data.read_data_sets("/tmp/data/", one_hot=True) input_features = 784 sigopt.log_metadata('input_features', input_features) x = tf.placeholder("float", [None, input_features]) output_classes = 10 sigopt.log_metadata('output_classes', output_classes) y = tf.placeholder("float", [None, output_classes]) sigopt.log_model(type='Multi Layer Perceptron') num_hidden_nodes = sigopt.get_parameter('num_hidden_nodes', default=10) hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes])) hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes])) hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases)) activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes])) activation_biases = tf.Variable(tf.random_normal([output_classes])) activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases)) learning_rate = sigopt.get_parameter('learning_rate', default=1e-3) cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=activation_layer, labels=y)) optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost) with tf.Session() as sess: sess.run(tf.global_variables_initializer()) batch_size = sigopt.get_parameter('batch_size', default=100) epochs = sigopt.get_parameter('epochs', default=5) num_batches = mnist.train.num_examples // batch_size for i in range(epochs): for _ in range(num_batches): batch_x, batch_y = mnist.train.next_batch(batch_size) sess.run(optimizer, feed_dict={x: batch_x, y: batch_y}) correct_prediction = tf.equal(tf.argmax(activation_layer, 1), tf.argmax(y, 1)) accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float")) accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels}) sigopt.log_metric('accuracy', accuracy_value)
Run the Code
sigopt run example.py
Run started, view it on the SigOpt dashboard at https://app.sigopt.com/run/1 Extracting /tmp/data/train-images-idx3-ubyte.gz Extracting /tmp/data/train-labels-idx1-ubyte.gz Extracting /tmp/data/t10k-images-idx3-ubyte.gz Extracting /tmp/data/t10k-labels-idx1-ubyte.gz Run finished, view it on the SigOpt dashboard at https://app.sigopt.com/run/1
SigOpt will conveniently output links in the terminal to the run page on our web application.
Any extra arguments will be passed along to your Python executable.
Be aware, you can always run your Python file without the
sigopt run
command, but if you do so, no run information will be sent to SigOpt.View the Record
Now, visit the project in the SigOpt web app to see the results. Here's a view of the training run record:
At the project level, you can compare the training runs.
Overview of the SigOpt Methods
The example above uses the following methods (links to the specifications):
Let's discuss a few notable features below.
sigopt.get_parameter
Parameters are any input to your model that you may change as you seek to refine the model. In our example, we have the number of hidden nodes, batch size, and number of epochs. If you think you might want to optimize on a parameter, that's a good reason to include it as a parameter.
During a single training run, sigopt.get_parameter
utilizes the default
value. If you're interested in optimizing or exploring a hyperparameter (or two), visit the Optimize & Explore section to learn more about how sigopt.get_parameter
can return a suggested parameter SigOpt suggests within the search space. With a single method, sigopt.get_parameter
, you can instrument your code once and use it to create a single training run or to automate an experiment.
sigopt.log_model
sigopt.log_model
stores a text value that you can use to filter your runs in the SigOpt web view. In our example, you might want to filter by sklearn - logistic regression
to compare models of the same type in the web charts.
sigopt.log_dataset
Name your dataset with sigopt.log_dataset
so you can easily reference it later. Store the path to AWS S3 or a csv filename, whatever might be useful to you.
sigopt.log_metadata
Record any key value pairs you can imagine with sigopt.log_metadata
.
In our example, we've used sigopt.log_metadata
to store the input features and output classes, values we might want to change and compare across training runs but aren't data we'd consider parameters.
sigopt.log_metric
The most important information about a model is how it performed. With sigopt.log_metric
you can take advantage of SigOpt's analysis dashboard of custom charts and advanced sorting and filtering.
Next
Optimize & ExploreSigOpt has long been used to automate the tuning of a model. The next section, Optimize & Explore, shows how you can set up SigOpt experiments that will generate hyperparameter configurations with either a simple grid search or our Bayesian ensemble.