Metric Thresholds

In Multimetric experiments, SigOpt searches for the efficient frontier; these are the parameter assignments whose metric values cannot be improved uniformly. The final outcome of such an experiment is the subset of all observations which have efficient metric values. In general, however, it may be the case that only a subset of the full efficient frontier is actually relevant for a given model. For example, some observations on the efficient frontier may not be relevant because unique business requirements force one or both of your metrics to meet custom criteria.

The goal of this Metric Thresholds feature is to allow you to provide a region of metric values which are of most relevance. SigOpt’s optimization engine will use this information to focus on this region, with the intent of better resolving the efficient frontier which satisfies these stated thresholds.

Example: Balancing Accuracy with Speed

Neural networks have great capacity to perform well in complicated machine learning situations; however, this may come at the cost of creating a very complex model with a slow inference time. We discussed this situation in an earlier blog. Suppose that the neural network would only be acceptable if it could make predictions in less than 100ms. The Metric Thresholds feature allows users to state this limitation at experiment creation to help inform SigOpt of the practical restrictions in this problem. Factoring in this limitation directs SigOpt to focus on the region of interest and thus increase the proportion of models which satisfy the inference latency restriction.

Example: Zooming in on Viable Trading Strategies

Many financial trading models must balance a desired high performance with a need to limit risk or maximize liquidity. However, at the start of a SigOpt experiment for model tuning, it may be the case that the feasible performance and risk quantities for the upcoming trading session are unknown. The SigOpt Metric Thresholds feature allows for experiment updating, to alert SigOpt which parts of the frontier are of most relevance after some of the efficient frontier has been exposed. Providing this information to SigOpt can allow the service to increase the proportion of models which align with the desired tradeoff between performance and risk.

Defining the Metric Thresholds

To assign a threshold to one of your metrics, specify the threshold field in the desired Metric object when creating your experiment. You can specify a threshold for any number of metrics in your multimetric experiment. To assign thresholds after an experiment has been created, update the metrics field, as shown below. SigOpt will interpret the specified threshold in the following ways.

  • By default, SigOpt will assume you want to maximize each metric and try to find observations that are greater than or equal to the specified threshold.
  • If you provide an objective of “minimize” for a metric then SigOpt will try to find observations that are less than or equal to the specified threshold.

Creating the Experiment

Below we create a new experiment, mimicking the earlier example of limiting neural network inference time.

from sigopt import Connection

conn = Connection(client_token="SIGOPT_API_TOKEN")
conn.set_api_url("https://api.sigopt.com")

experiment = conn.experiments().create(
  name="Neural network",
  parameters=[
    dict(
      name="log10_learning_rate",
      bounds=dict(
        min=-4,
        max=0
        ),
      type="double"
      ),
    dict(
      name="nodes_per_layer",
      bounds=dict(
        min=5,
        max=20
        ),
      type="int"
      )
    ],
  metrics=[
    dict(
      name="inference_time_milliseconds",
      objective="minimize",
      threshold=100
      ),
    dict(
      name="validation_accuracy",
      objective="maximize"
      )
    ],
  observation_budget=65,
  parallel_bandwidth=2,
  project="sigopt-examples",
  type="offline"
  )

Updating Metric Thresholds

Metric thresholds can be updated on the properties page of an experiment at any time while the experiment is in progress. The thresholds can also be updated directly through our API. An example of this is given below.

experiment = conn.experiments().create(
  name="My Thresholds Experiment",
  project="sigopt-examples",
  parameters=[
    dict(
      name="investment_0",
      bounds=dict(
        min=0.0,
        max=1.0
        ),
      type="double"
      ),
    dict(
      name="investment_1",
      bounds=dict(
        min=0.0,
        max=1.0
        ),
      type="double"
      )
    ],
  metrics=[
    dict(
      name="expected_return",
      objective="maximize"
      ),
    dict(
      name="uncertainty",
      objective="minimize"
      )
    ],
  observation_budget=65,
  parallel_bandwidth=2
  )

### Run the experiment for some number of observations and identify an interesting region

experiment = conn.experiments().update(
  metrics=[
    dict(
      name="expected_return",
      threshold=6.2
      ),
    dict(
      name="uncertainty",
      threshold=0.79
      )
    ]
  )

Feasibility

If the metric thresholds which are defined for SigOpt cannot be satisfied anywhere in the domain, this feature will perform unpredictably. For example, if the accuracy threshold for a machine learning model is set to 984, when the actual accuracy is bounded between [0, 1], this feature will assume the value 984 is actually feasible and explore the domain erratically trying to find an observation that satisfies this threshold. When defining thresholds, it is best: to state thresholds at the start of an experiment which are well-established based on previous experiments; or to update the experiment after some feasible observations have been observed.

Limitations

  • Metric thresholds can only be applied to Multimetric experiments. Therefore it has the same limitations as Multimetric experiments.
  • Metric thresholds can only be applied to optimized metrics.