Interpret Multimetric Experiment Solutions
In Executing Multimetric Experiments you learned how to create a multimetric experiment and implement the optimization loop for at least
observation_budget many iterations. Now, the results of your Experiment must be interpreted. Calling Best Assignments List on a multimetric experiment will return a pagination of many Best Assignments results.
More Than One Solution?
These points represent a Pareto efficient set of observations, where one metric cannot be improved without another metric suffering. This set of points forms an approximate Pareto frontier, from which the domain experiment (you) must select the optimal solution. Further discussion on this topic appears in our blog here and here.
Extracting the Pareto Efficient Points
We revisit Executing Multimetric Experiments where the creation and execution of a multimetric experiment is demonstrated. The contour plots below show two metrics,
f2, which cannot both be maximized with the same parameters.
After the optimization loop has completed, the efficient metric values can be extracted along with the corresponding parameters that generated them. The code block below shows how to access these results from the
best_assignments object. Note that the Metric Evaluations in the
efficient_results pagination are ordered alphabetically by metric name.
efficient_results = conn.experiments(experiment.id).best_assignments().fetch() efficient_metrics = [ [d.value for d in data.values] for data in efficient_results.data ] efficient_parameters = [ [data.assignments['x1'], data.assignments['x2']] for data in efficient_results.data ]
Interpreting the Solution
After extracting the efficient observations, a decision must be made regarding which of the parameter choices is best for your purposes. This decision often requires expert insight, so as to appropriately balance the competing metrics. Given that, it may be possible to produce graphics to help inform the decision. Below is an example of possible outcomes of an experiment involving the metrics from the contour plots above.
The graph on the left shows the region of feasible points with the blue circles representing the efficient points that SigOpt found on the Pareto frontier, the red circles representing other suggested parameters from SigOpt, and the black dots representing randomly chosen parameters (to show the outline of the region). Each of the efficient blue circles in the left graph corresponds to an efficient parameter choice represented by a blue circle in the right graph. A domain expert may be able to judge which of the results in the left graph is preferred, and then use the parameters from the right graph (or simply accessed through
best_assignments) to create the model in production.