Manage Open Suggestions

If you run into an error, you may have some outstanding suggestions for which you have not reported observations (which are called open suggestions in the API). Here are some strategies you can use to recover in this situation.

Iterate through all open suggestions and reevaluate

suggestions = conn.experiments(experiment_id).suggestions().fetch(state="open")
for suggestion in suggestions.iterate_pages():
    value = evaluate_metric(suggestion.assignments)  # implement this
    conn.experiments(experiment_id).observations().create(
        suggestion=suggestion.id,
        value=value,
    )

If you know the suggestion ID and it is something you wish to report it as a failure

conn.experiments(experiment_id).observations().create(
    suggestion=suggestion_id,
    failed=True,
)

If you know the suggestion ID and it is something you wish to retry to evaluate and report

suggestion = conn.experiments(experiment_id).suggestions(suggestion_id).fetch()
value = evaluate_metric(suggestion.assignments)  # Implement this
conn.experiments(experiment_id).observations().create(
    suggestion=suggestion_id,
    value=value,
)

If you know the suggestion ID and you just want to delete the suggestion

conn.experiments(experiment_id).suggestions(suggestion_id).delete()

If you just want to delete all open suggestions

conn.experiments(experiment_id).suggestions().delete(state="open")