Documentation

Welcome to SigOpt’s developer documentation. If you have a question you can’t answer, feel free to contact us!

Track Suggestion and Observation Using Metadata

When running SigOpt in parallel, one may be interested in tracking which worker (e.g., an EC2 instance on AWS) a Suggestion or Observation is running on. This can be useful, for instance, when a failed observation appears and the user wants to identify the corresponding failed worker. metadata provides a convenient way to do that. Information about the worker can be passed in when creating a Suggestion or Observation.

Including metadata at Suggestion creation can be done via

conn.experiments(experiment.id).suggestions().create(
  metadata=dict(instance_id=instance_id),
)

or at Observation creation via

conn.experiments(experiment.id).observations().create(
  suggestion=suggestion.id,
  value=value,
  metadata=dict(instance_id=instance_id),
)

Information stored in metadata will be displayed on the SigOpt Experiment webpage. If instance_id is passed in, then under the Suggestion tab each Suggestion entry will include the associated instance_id. Likewise, under the History tab, the user can see all metadata passed in with an Observation in the modal when clicking on that entry. In particular if the user clicks on entries of failed Observations, they can see instance_id for that Observation (see example below).

To find the instance_id of an EC2 instance, the UNIX command is

ec2metadata --instance-id

or in Python

import boto.utils
boto.utils.get_instance_metadata()["instance-id"]

If, for some reason, an instance crashes during computing a Suggestion, the user can reconnect to that instance and run the following commands to restart the open Suggestion(s) associated with that instance:

suggestions = conn.experiments(experiment_id).suggestion().fetch(state="open")
for suggestion in suggestions.iterate_pages():
  if suggestion.metadata["instance_id"] == current_instance_id:
    value = evaluate_metric(suggestion.assignments)
    conn.experiments(experiment_id).observations().create(
      suggestion=suggestion.id,
      value=value,
    )

When generating plots on SigOpt webpage, the user can choose fields from metadata as coordinates for plots. In particular, under the Analysis tab, Experiment History, Full Experiment History and Parallel Coordinates plots all include fields from metadata as possible choices for coordinates. The user can select and combine them with other coordinates to generate plots (see below for an example where instance_id is included as one coordinate).

Note that plotting metric value against instance_id is unlikely to show any correlation since Suggestions are served out independently of the instance_id.

More information about metadata can be found here. For more details on running SigOpt in parallel, please refer to the documentation running SigOpt in parallel. There is also an example python file on parallel computation in the git repo sigopt-examples.