Metric Strategy

By default, SigOpt will aim to optimize your metric or search for the efficient frontier when more than one metric is being optimized. However, it can also be helpful to track metrics even when they are not being optimized. For example, some metrics may be very strongly correlated so you want to focus optimization on one. Another case is when investigating a new field while still needing to report an industry–standard one.

The goal of Metric Strategy is to enable you to more easily track these metrics. Now you can store up to 50 metrics on an experiment while focusing optimization on one or two. Stored metrics allow you to better track all the relevant information you uncover in your experiment and enable more informative, more flexible graphs on the Analysis Page.

Metric Strategy works with the Metric Objective field to help make analysis more intuitive. Whether the strategy is set to optimize or store, the objective can still be set to maximize or minimize. The objective defines the direction of improvement for the metric, whether or not it gets optimized.

Example: Storing Infrastructure Measurements

Neural networks can perform well in very complicated situations but may need to balance model complexity with inference time, ensuring viable solutions can be reached within business constraints for prediction time. While accuracy and inference time need to be optimized, it may be beneficial to track the memory usage, to help monitor the expected load on the system and how that impacts the current infrastructure.

Now, we can analyze whether memory usage is correlated, positively or negatively, with our optimized metrics, giving us a more natural way to explore this interaction.

Defining the Metric Strategy

To assign a strategy to one of your metrics, specify the strategy field in the desired Metric object when creating your experiment. You can specify optimize, which is the default, or store when setting the strategy.

Creating the Experiment

Below, we create a new experiment, using the above example of storing memory usage while optimizing accuracy and inference time.

from sigopt import Connection

conn = Connection(client_token="SIGOPT_API_TOKEN")
conn.set_api_url("https://api.sigopt.com")

experiment = conn.experiments().create(
  name="Neural network",
  parameters=[
    dict(
      name="log10_learning_rate",
      bounds=dict(
        min=-4,
        max=0
        ),
      type="double"
      ),
    dict(
      name="nodes_per_layer",
      bounds=dict(
        min=5,
        max=20
        ),
      type="int"
      )
    ],
  metrics=[
    dict(
      name="inference_time_milliseconds",
      objective="minimize",
      strategy="optimize"
      ),
    dict(
      name="validation_accuracy",
      objective="maximize",
      strategy="optimize"
      ),
    dict(
      name="memory_usage",
      objective="minimize",
      strategy="store"
      )
    ],
  observation_budget=65,
  parallel_bandwidth=2,
  project="sigopt-examples",
  type="offline"
  )

Limitations

  • Metrics whose strategy is set to store will not be optimized in any way.
  • There can be up to 50 metrics of any kind for an experiment.
  • All stored metrics must be reported for successful Observations.
  • No stored metrics can be reported for Failed Observations.
  • Metric Thresholds can only be applied to optimize metrics.