Picking a Metric
The SigOpt optimization loop can maximize or minimize any real-valued objective function which outputs a metric. The most important decision when creating a new SigOpt experiment is defining a good objective function. This article will detail best practices for defining a good objective function to get the most out of SigOpt.
SigOpt is able to optimize a process or model for any underlying objective function, whether it is noisy, discontinuous, non-convex, or the combination of many other metrics. This is the value that SigOpt seeks to maximize so it is important to make sure it is the value that corresponds to a better process and model for your application.
1. Connect your objective to your business goals
The best objective function is one that, when maximized, will have a direct and provable impact on your overall business goals. For a predictive analytics system, such as fraud detection, this might be the accuracy of the prediction. For an industrial process, this might be maximizing the measured output strength or stability of the finished product.
We strongly encourage customers to identify a real-world objective function, and are happy to provide guidance on how to do this well - just!
2. Compute your objective using cross-validation to avoid overfitting
(for machine learning optimization)
When tuning hyperparameters and feature parameters in a machine learning context, the objective function should be a cross-validated accuracy, AUC, or similar metric. This prevents overfitting of the model to the training data. Learn More.
3. Compute the standard deviation in your underlying process
(for physical and industrial processes)
SigOpt supports objective functions that are the output of an underlying process that may have variance, or uncertainty, associated with it. Computing the standard deviation of the objective function output for each suggested parameter configuration and reporting it to SigOpt is highly recommended. This allows SigOpt’s optimization platform to adapt its exploration and exploitation based on the known uncertainty at different configurations of the system, leading you to better results, faster.
In some cases it might be necessary to run the underlying process 5-10 times with a single parameter configuration to compute a normal standard deviation before reporting an Observation to SigOpt.
4. Create a composite objective function to combine multiple goals
If your objective function is a composite (say, a combination of performance and cost), you must compute the composite value before reporting to SigOpt. In this case, we recommend using our Multimetric Experiments.
We are happy to provide guidance on how to create a good composite objective - just!
While this document discusses metric maximization, you can also minimize your metric.