Documentation

Welcome to the developer documentation for SigOpt. If you have a question you can’t answer, feel free to contact us!
Welcome to the new SigOpt docs! If you're looking for the classic SigOpt documentation then you can find that here. Otherwise, happy optimizing!

Tuning XGBoost Models with sigopt.xgboost.experiment

The sigopt.xgboost.experiment function simplifies the hyperparameter tuning process of an XGBoost model, by automatically creating and running a SigOpt optimization Experiment. This function also extends the automatic parameter, metric, and metadata logging of our sigopt.xgboost.run API to the SigOpt experimentation platform.

However, this automatic logging is only one of the features, andsigopt.xgboost.experiment offers the following crucial improvements over the existing SigOpt Experiment API when tuning an XGBoost model:

  • A simplified and streamlined API that knows the exact problem it is tuning: XGBoost, and makes intelligent decisions accordingly.
  • Automatic selection of the parameter search space, optimization metric, and the tuning budget.
  • A preset list of standard optimization metrics to choose from.
  • An improved hyperparameter optimization routine that leverages advanced methods in metalearning and multi-fidelity optimization to learn a more performant model in less time.

This API has been designed with ease-of-use in mind, so that you may run an XGBoost Experiment as effortlessly as possible.

Examples

To give you an initial feel for how you might use the sigopt.xgboost.experiment API, we provide multiple examples showcasing its simplicity and flexibility. Our API aims to reduce the overall complexity of intelligent experimentation and hyperparameter optimization by automatically selecting parameters, metrics, and even the budget where needed.

The sequence of examples are provided in the tabs below, increases in complexity.

Automatic Experiment Configuration

The parameter search space, metric, and budget are determined by SigOpt based on the training data provided.

from sklearn import datasets
from sklearn.model_selection import train_test_split
import xgboost
import sigopt.xgboost

bc = datasets.load_breast_cancer()
(Xtrain, Xtest, ytrain, ytest) = train_test_split(bc.data, bc.target, test_size=0.5, random_state=42)
dtrain = xgboost.DMatrix(data=Xtrain, label=ytrain, feature_names=bc.feature_names)
dtest = xgboost.DMatrix(data=Xtest, label=ytest, feature_names=bc.feature_names)

my_config = dict(
  name="My XGBoost Experiment", # Let SigOpt set the tuning parameters, metrics, and budget.
)
experiment = sigopt.xgboost.experiment(
  experiment_config=my_config,
  dtrain=dtrain,
  evals=[(dtest, "val_set")],
  params = {"objective": "binary:logistic"}, # XGB parameters to be fixed for all runs
)

The simple examples are made possible because of key research advances made by SigOpt research. It is worth noting that the decrease in simplicity corresponds to a decrease in flexibility; if you opt to omit the metric for example, you will be forced to optimize the metric we select for you.

Input Arguments for sigopt.xgboost.experiment

The API for an XGBoost Experiment follows:

ArgumentTypeDescription
experiment_configdictThe configuration of the Experiment. See the following section for more information.
dtrainxgboost.DMatrixThe training dataset.
evalsxgboost.DMatrix or array<(xgboost.DMatrix, string)>These are the validation set(s). If it is a list, the first dataset will be used to compute the optimization metric.
paramsdictThese are the XGBoost parameters, e.g., tree_method, you plan on fixing throughout the course of the Experiment. See the description of these parameters at the XGBoost Parameters documentation for more information.
num_boost_roundintOptional. The number of boosting rounds. Leave this argument as blank if num_bost_round is specified in parameters field in the experiment_config.
run_optionsdictOptional. A dictionary specifying the autologging capabilities. See the SigOpt XGBoost Run documenation for more information.

The experiment_config is most important to understand, since it not only determines how your Experiment executes, but also possesses the most flexibility and extensibility out of all XGBoost Experiment API arguments. Thus, we explain it next.

Specifying sigopt.xgboost.experiment through experiment_config

An experiment config has the following keys:

KeyTypeValue
namestringName of the Experiment.
parametersarray<Parameter>Optional. An array of Parameter objects. See Parameter Space subsection below for more information.
metricsarray<Metric> or stringOptional. An array of Metric objects. See Metric Space subsection below for more information.
budgetintOptional. An integer defining the minimum number of SigOpt Runs in a given SigOpt Experiment.

Parameter Space

We show an illustrative example below on how to set the parameter space.

sigopt.xgboost.experiment(...,
  parameters = [
    {"name": "eta"}, # name only
    {"name": "num_boost_round", "bounds": {"min": 10, "max": 100}}, # name and bounds only
    {"name": "tree_method", "type": "categorical", "categorical_values": ["exact", "hist"]},
  ]
)

There are three different ways of specifying the Experiment parameters:

  • name only: SigOpt autoselects the bounds and type.
  • name and type: SigOpt autoselects the bounds.
  • name, type, and bounds/categorical_values: Explicit parameter specification.

These specifications may be mixed as in the example above. Currently, SigOpt only autoselects the bounds for the following parameters:

eta, max_delta_step, alpha, gamma, lambda, max_depth, min_child_weight, num_boost_round, colsample_bylevel, colsample_bynode, colsample_bytree

Any parameter that is not on this list must have its bounds or categorical_values explicitly stated.

Metric Space

The metric space of an Experiment is defined by both the metrics argument of the experiment_config and the datasets listed in the evals argument.

There are two ways of specifying the metric space.

# Option 1: Fully specified like a SigOpt Experiment
metrics = [{"name": "accuracy", "strategy": "optimize", "objective": "maximize"}]

# Option 2: Using only a String
metrics = "F1"

Below is a table of the metrics we natively support for classification and regression.

TaskOptionsDefault
Classificationaccuracy, F1, precision, recallaccuracy
Regressionmean absolute error, mean squared errormean squared error