Documentation

Welcome to SigOpt’s developer documentation. If you have a question you can’t answer, feel free to contact us!
This feature is currently in Beta. You can request free access on the main page of the beta or contact us directly for more information.

Optimization & Exploration

In the guides for notebook and command line development environments, we've covered the recommended way to instrument your code with SigOpt for a training run and shown how that can be easily extended to run SigOpt experiments.

Here we'll discuss more of the options for running a SigOpt experiment.

Optimization

After your model has been instrumented, it is easy to take advantage of SigOpt's optimization features. Optimization helps find the parameters for your model that give you the best metric (eg. maximizing an accuracy metric).

An experiment is a definition of the hyperparameter search space and the metrics you would like optimized.

The experiment can be defined in either a YAML or JSON format. In a notebook, this will be defined using the %%experiment magic command in a cell.

In a text editor, you'll create a sigopt.yml file in the directory from which you'll be running sigopt optimize example.py

Below, we have an example optimization which finds parameters to maximize test_accuracy for a Tensorflow model:

Text Editor & CLI Optimization Example

  1. Install & Configure SigOpt

    If you haven't already, follow the installation and configuration instructions in our Get Started page.

  2. Set Project

    Add Sigopt methods to capture data about your training run.

    See Record a Run for more details about setting the project ID. The API Reference is a definitive list of the SigOpt methods available to instrument your run.

    If you don't set a $SIGOPT_PROJECT environment variable, SigOpt will use the current directory name as the project ID.

  3. Instrument Your Model

    See Record a Run for more details about how the following example run was instrumented.

    # example.py
    from tensorflow.examples.tutorials.mnist import input_data
    import tensorflow as tf
    
    import sigopt
    
    sigopt.log_dataset(name='mnist')
    mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
    
    input_features = 784
    sigopt.log_metadata('input_features', input_features)
    x = tf.placeholder("float", [None, input_features])
    output_classes = 10
    sigopt.log_metadata('output_classes', output_classes)
    y = tf.placeholder("float", [None, output_classes])
    
    sigopt.log_model(type='Multi Layer Perceptron')
    
    num_hidden_nodes = sigopt.get_parameter('num_hidden_nodes', default=10)
    hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes]))
    hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes]))
    hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases))
    
    activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes]))
    activation_biases = tf.Variable(tf.random_normal([output_classes]))
    activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases))
    
    learning_rate = 10**sigopt.get_parameter('log_learning_rate', default=-3)
    
    cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=activation_layer, labels=y))
    optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
    
    with tf.Session() as sess:
      sess.run(tf.global_variables_initializer())
    
      batch_size = sigopt.get_parameter('batch_size', default=100)
      epochs = sigopt.get_parameter('epochs', default=5)
      num_batches = mnist.train.num_examples // batch_size
      for i in range(epochs):
        for _ in range(num_batches):
          batch_x, batch_y = mnist.train.next_batch(batch_size)
          sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})
    
      correct_prediction = tf.equal(tf.argmax(activation_layer, 1), tf.argmax(y, 1))
      accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
      accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels})
      sigopt.log_metric('accuracy', accuracy_value)
    
  4. Define the Experiment

    Create a sigopt.yml file to define your experiment. The file should have an experiment section with the name, project, parameters, metrics and other options that you would like to run your experiment with.

    The parameters should match the names provided to sigopt.get_parameter in your Python code and similarly the metrics should match the names of the sigopt.log_metric calls.

    Here's an example file:

    # sigopt.yml
    experiment:
      name: Multi-Layer Perceptron
      parameters:
        - name: num_hidden_nodes
          bounds:
            min: 10
            max: 784
          type: int
        - name: log_learning_rate
          bounds:
            min: -8
            max: 0
          type: double
        - name: batch_size
          bounds:
            min: 50
            max: 400
          type: int
        - name: epochs
          bounds:
            min: 5
            max: 20
          type: int
      metrics:
        - name: accuracy
          objective: maximize
      observation_budget: 50
    

    Note: If your sigopt.yml file is not in the directory where you are running sigopt optimize, you can specify the location of the sigopt.yml file with an additional --sigopt-file flag:

    $ sigopt optimize example.py --sigopt-file example/sigopt.yml
  5. Run the Optimization

    # terminal
    $ sigopt optimize example.py
    

    Note: Any extra arguments will be passed along to your Python executable. For example:

    $ sigopt optimize example.py arg1 arg2
  6. View the Results

    The output will provide links to the SigOpt web app where you can navigate to recent runs and experiments.

Notebook Optimization Example

  1. Install & Configure SigOpt

    If you haven't already, follow the installation and configuration instructions in our Get Started page.

  2. Set the Project

    from tensorflow.examples.tutorials.mnist import input_data
    import tensorflow as tf
    import sigopt
    %load_ext sigopt
    import os
    os.environ['SIGOPT_PROJECT'] = 'run-examples'
    
    mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
    
    input_features = 784
    output_classes = 10
    
  3. Instrument Your Model

    See Record a Run for more details about how the following example run was instrumented.

    def create_model(x):
      num_hidden_nodes = sigopt.get_parameter('num_hidden_nodes', default=10)
      hidden_weights = tf.Variable(tf.random_normal([input_features, num_hidden_nodes]))
      hidden_biases = tf.Variable(tf.random_normal([num_hidden_nodes]))
      hidden_layer = tf.nn.tanh(tf.add(tf.matmul(x, hidden_weights), hidden_biases))
    
      activation_weights = tf.Variable(tf.random_normal([num_hidden_nodes, output_classes]))
      activation_biases = tf.Variable(tf.random_normal([output_classes]))
      activation_layer = tf.nn.sigmoid(tf.add(tf.matmul(hidden_layer, activation_weights), activation_biases))
      return activation_layer
    
    def train_model(sess, optimizer, x, y):
      batch_size = sigopt.get_parameter('batch_size', default=100)
      epochs = sigopt.get_parameter('epochs', default=5)
      num_batches = mnist.train.num_examples // batch_size
      for i in range(epochs):
        for _ in range(num_batches):
          batch_x, batch_y = mnist.train.next_batch(batch_size)
          sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})
    
    def evaluate_model(sess, model, x, y):
      correct_prediction = tf.equal(tf.argmax(model, 1), tf.argmax(y, 1))
      accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
      accuracy_value = sess.run(accuracy, feed_dict={x: mnist.validation.images, y: mnist.validation.labels})
      sigopt.log_metric('accuracy', accuracy_value)
    
  4. Define the Experiment

    The experiment definition will include the name, project, parameters, metrics and other options that you would like to run your experiment with.

    The parameters should match the names provided to sigopt.get_parameter in your Python code and similarly the metrics should match the names of the sigopt.log_metric calls.

    You can format the experiment definition in Python, YAML, or JSON. In this example, we're using Python. See below for YAML and JSON examples. With Python, you have the advantage of using local variables if you wish.

    %%experiment
    # define the parameters, metrics and budget for your experiment
    {
      'metrics': [
        {
          'name': 'accuracy',
          'objective': 'maximize'
        }
       ],
      'name': 'Multi-Layer Perceptron',
      'observation_budget': 5,
      'parameters': [
        {
          'bounds':
            {
              'max': 784,
              'min': 10
            },
          'name': 'num_hidden_nodes',
          'type': 'int'
        },
        {
          'bounds':
            {
              'max': 0,
              'min': -8
            },
          'name': 'log_learning_rate',
          'type': 'double'
        },
        {
          'bounds':
            {
              'max': 400,
              'min': 50
            },
          'name': 'batch_size',
          'type': 'int'
        },
        {
          'bounds':
            {
              'max': 20,
              'min': 5
            },
          'name': 'epochs',
          'type': 'int'
        }
      ]
    }
    

    Note: The observation budget is the number of runs you would like created in the optimization. We recommend using 5 for testing. When you're ready, visit the observation budget page to learn our rule of thumb for the appropriate observation budget.

  5. Run the Optimization

    %%optimize
    # optimize the model over multiple iterations
    # use %%run instead to only record a single run using the default parameters
    
    x = tf.placeholder("float", [None, input_features])
    y = tf.placeholder("float", [None, output_classes])
    
    sigopt.log_dataset(name='mnist')
    sigopt.log_metadata('input_features', input_features)
    sigopt.log_metadata('output_classes', output_classes)
    sigopt.log_model(type='Multi Layer Perceptron')
    
    model = create_model(x)
    
    learning_rate = 10 ** sigopt.get_parameter('log_learning_rate', default=-3)
    cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=model, labels=y))
    optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
    
    with tf.Session() as sess:
      sess.run(tf.global_variables_initializer())
    
      train_model(sess, optimizer, x, y)
    
      evaluate_model(sess, model, x, y)
    
  6. View the Results

    The output will provide links to the SigOpt web app where you can navigate to recent runs and experiments.

Formatting

When working in a notebook environment, in addition to Python, you have the option to define your experiment in YAML or JSON. Here are examples of both:

  • YAML

    %%experiment
    # define the parameters, metrics and budget for your experiment
    '''
    name: Multi-Layer Perceptron
    parameters:
      - name: num_hidden_nodes
        bounds:
          min: 10
          max: 784
        type: int
      - name: log_learning_rate
        bounds:
          min: -8
          max: 0
        type: double
      - name: batch_size
        bounds:
          min: 50
          max: 400
        type: int
      - name: epochs
        bounds:
          min: 5
          max: 20
        type: int
    metrics:
      - name: accuracy
        objective: maximize
    observation_budget: 5
    '''
    
  • JSON

    %%experiment
    # define the parameters, metrics and budget for your experiment
    '''
    {
      "name": "Multi-Layer Perceptron",
      "parameters": [
        {
          "name": "num_hidden_nodes",
          "bounds": {
            "min": 10,
            "max": 784
          },
          "type": "int"
        },
        {
          "name": "log_learning_rate",
          "bounds": {
            "min": -8,
            "max": 0
          },
          "type": "double"
        },
        {
          "name": "batch_size",
          "bounds": {
            "min": 50,
            "max": 400
          },
          "type": "int"
        },
        {
          "name": "epochs",
          "bounds": {
            "min": 5,
            "max": 20
          },
          "type": "int"
        }
      ],
      "metrics": [
        {
          "name": "accuracy",
          "objective": "maximize"
        }
      ],
      "observation_budget": 5
    }
    '''
    

Exploration

You might want to use grid or random searches to explore a space. Please contact our team to learn how to use these experiment options.