Track Suggestion and Observation Using Metadata
When running SigOpt in parallel, one may be interested in tracking which worker (e.g., an EC2 instance on AWS) a Suggestion or Observation is running on. This can be useful, for instance, when a failed observation appears and the user wants to identify the corresponding failed worker.
metadata provides a convenient way to do that. Information about the worker can be passed in when creating a Suggestion or Observation.
metadata at Suggestion creation can be done via
conn.experiments(experiment.id).suggestions().create( metadata=dict(instance_id=instance_id), )
or at Observation creation via
conn.experiments(experiment.id).observations().create( suggestion=suggestion.id, value=value, metadata=dict(instance_id=instance_id), )
Information stored in
metadata will be displayed on the SigOpt Experiment webpage. If
instance_id is passed in, then under the Suggestion tab each Suggestion entry will include the associated
instance_id. Likewise, under the History tab, the user can see all
metadata passed in with an Observation in the modal when clicking on that entry. In particular if the user clicks on entries of failed Observations, they can see
instance_id for that Observation (see example below).
To find the
instance_id of an EC2 instance, the UNIX command is
or in Python
import boto.utils boto.utils.get_instance_metadata()["instance-id"]
If, for some reason, an instance crashes during computing a Suggestion, the user can reconnect to that instance and run the following commands to restart the open Suggestion(s) associated with that instance:
suggestions = conn.experiments(experiment_id).suggestion().fetch(state="open") for suggestion in suggestions.iterate_pages(): if suggestion.metadata["instance_id"] == current_instance_id: value = evaluate_metric(suggestion.assignments) conn.experiments(experiment_id).observations().create( suggestion=suggestion.id, value=value, )
When generating plots on SigOpt webpage, the user can choose fields from
metadata as coordinates for plots. In particular, under the Analysis tab, Experiment History, Full Experiment History and Parallel Coordinates plots all include fields from
metadata as possible choices for coordinates. The user can select and combine them with other coordinates to generate plots (see below for an example where
instance_id is included as one coordinate).
Note that plotting metric value against
instance_id is unlikely to show any correlation since Suggestions are served out independently of the
More information about
metadata can be found here. For more details on running SigOpt in parallel, please refer to the documentation running SigOpt in parallel. There is also an example python file on parallel computation in the git repo