Multitask Experiments (Alpha)
SigOpt supports multitask experiments through the API. Multitask experiments are useful in situations where fast approximations to the true metric under consideration are available. We refer to these approximations as
tasks, with the full cost task associated with the true metric that SigOpt is trying to maximize. The goal of this experiment type is to leverage data which is available at less cost to more efficiently acquire and interpret data regarding the true objective.
The chosen task names (along with an estimated cost) should be provided at experiment creation. Subsequent observations will contain both a set of parameter assignments and the appropriate task to consider. If the faster tasks provide useful insights regarding the true metric, SigOpt should be able to search the parameter space in less wall-clock time.
Example - Partial Number of Epochs
One example where faster tasks naturally arise is in maximizing the accuracy resulting from a gradient descent algorithm. If we hope to conduct at most 10 epochs, one might choose to define four tasks: a
cheapest task consisting of 1 epochs at cost
cheaper task consisting of 2 epochs at cost
cheap task consisting of 5 epochs at cost
0.5, and the
true task consisting of all 10 epochs at cost
Example - Subset of Data
Another example of how faster tasks can be defined in a supervised machine learning setting is by training on subsets of the data. Suppose 10000 labeled examples are available, and the goal of the SigOpt experiment is to maximize the training accuracy. One might choose to define two tasks: a
true task which involves building a model on all the data at cost
1.0 and a
cheap task which involves a balanced subsampling of only 1000 labeled examples at cost
Creating the Experiment
A multitask experiment must have its
type set to
offline and must have a field
tasks set to a positive integer.
tasks should contain a list of objects stating the
cost associated with each task (including the full cost task for which SigOpt is trying to find the optimum). Multitask experiments also require the
observation_budget to be set; in this setting,
observation_budget represents the cumulative cost to be expended throughout the entire course of the experiment (discussed further below).
from sigopt import Connection conn = Connection(client_token="SIGOPT_API_TOKEN") experiment = conn.experiments().create( name="Classifier Accuracy", parameters=[ dict( name="gamma", bounds=dict( min=0.001, max=1 ), type="double" ) ], observation_budget=47, parallel_bandwidth=1, project="sigopt-examples", tasks=[ dict( name="cheap", cost=0.1 ), dict( name="true", cost=1 ) ], type="offline" ) print("Created experiment: https://app.sigopt.com/experiment/" + experiment.id)
This feature is currently in alpha. Some API client libraries do not support multitask experiments.
Interpreting Suggestions and Reporting Observations
Suggestions in multitask experiments have one new field:
task contains the
cost of the task for which to execute the given assignments. Observations must be reported using the suggestion
id provided from SigOpt; manually reporting data without a corresponding suggestion is not presently supported.
SigOpt will suggest many partial cost suggestions, especially in the beginning of the experiment, but also guarantees at least one full cost observation (often multiple) by the end of the experiment (once the
observation_budget is expended).
Reviewing Reported Observations
Best Assignments List will only consider full cost observations for multitask experiments. Additionally, it is possible that there will be no full cost observations in the early stages of multitask experiments, and thus no best assignments. In this situation, Best Assignments List will return no best assignments (an empty Pagination), and we recommend waiting until the experiment is further along to retrieve best assignments.
Setting Observation Budget
observation_budget would correspond to the expected number of observations created during an experiment. For a multitask experiment, the
observation_budget represents the expected cost accumulated over all observations created. You should set
observation_budget based on how much time or compute you would like to allocate to your experiment. If you were previously using standard SigOpt (non-multitask) to tune a model, you may be able to decrease
observation_budget when running multitask experiments.
When defining an
observation_budget for a multitask experiment consider that all of the following circumstances would yield cumulative cost 100:
- 90 observations with cost 1.0 and 40 with cost 0.25
- 50 observations with cost 1.0 and 200 with cost 0.25
- 10 observations with cost 1.0 and 360 with cost 0.25
SigOpt will balance exploration and exploitation over the course of the experiment to determine which of these circumstances is best for the given problem. We will not, however, know a priori, so the
observation_budget should most appropriately be set as a rough guideline of the total cost with which SigOpt can work. As is the case with all SigOpt experiments, the budget is meant only as guidance to SigOpt and not as a promise.
Observation Budget Consumed
For multitask experiments, the Experiment object has an
observation_budget_consumed field that corresponds to the cost accumulated over all observations so far (current cumulative cost). We recommend running the optimization loop until
observation_budget_consumed hits the defined
observation_budget. Do not use the
observation_count field to determine when to stop, as with standard SigOpt experiments, since that will lead to an incomplete experiment (if you run your experiment until
observation_budget_consumed will be less than your
while experiment.progress.observation_budget_consumed < experiment.observation_budget: # ...
- Experiment Metric Importances, in the API and on the web dashboard, are calculated including both full and partial cost observations for multitask experiments.
Multitask experiments have some limitations in order to allow their complicated functionality. This list is likely to change as this feature develops during its beta release.
- Suggestions cannot be enqueued, updated or deleted.
- Observations must be created with a
suggestionfield, i.e., without an
- Only one metric is permitted and the number of solutions must be one.
- Despite the fractional nature of the cumulative cost, the
observation_budgetis still required to be an integer.
- Tasks must all have unique names and costs.
- The true task under consideration in the experiment must have cost 1.0, and all other costs should be less than that.
During development of this feature, we have produced some guidance regarding the successful use of multiple tasks to accelerate optimization of the true, full cost, task.
- Limiting the number of tasks can help SigOpt develop a more consistent understanding of the relationship between less expensive and more expensive tasks. As a result, we recommend using less than 5 tasks in initial experimenting to gauge the benefit of more on your experimental setting.
- Limiting the gap in cost between the cheapest and true tasks can help SigOpt understand the relationship across tasks. In practice, we have seen that setting the cheapest task to a cost no less than 0.03 can improve the effectiveness of balancing of sampling cheaper and more expensive tasks.
- Parallelism is another powerful tool for accelerating tuning by testing multiple suggestions simultaneously. It can be used in conjunction with multitask, but because of the starkly different durations required to train models at different tasks (by design) we recommend limiting parallelism to no more than 5 simultaneously open suggestions.
While this document discusses metric maximization, you can also minimize your metric.