Hyperparameter Tuning

VAMOS can tune algorithm configuration: operator probabilities, distribution indices, population-related settings, and other parameters that control a MOEA. A tuning campaign evaluates candidate configurations over explicit problem and seed blocks and ranks them by a scalar quality score.

Experimental surface in VAMOS 1.1.0

The tuning/racing implementation and the vamos tune and vamos ablation commands are experimental. They are supported for evaluation but may change incompatibly in a minor release. Stable optimization calls used by the tuning runtime remain governed by the stability policy.

This page is about configuring optimization algorithms. If the variables in your optimization problem are model hyperparameters (for example SVM C and gamma), that is a different workflow; see examples/tuning/hyperparam_tuning.py for that formulation.

Changes included in 1.0.1

The install command below selects VAMOS 1.1.0, which keeps the corrected scoring, archive-search, and persistence behavior introduced in 1.0.1 and described on this page:

Behavior	VAMOS 1.0.0	VAMOS 1.0.1
HV input	Top-level result.F; may be population, archive, or SMPSO leaders depending on the algorithm/configuration	Feasible non-dominated final population
External-archive search parameters	Included where defined by the search-space builder	Removed from ordinary CLI tuning
Optuna persistent study name	User-supplied name, without the scoring-contract suffix	Adds __vamos_cli_final_population_hv_v1
AGE-MOEA/RVEA constraint handling	Constraints are not used in survival	Feasibility-aware behavior when enabled

Record the package version and any source commit with a tuning campaign. Do not mix persisted trials from the two scoring contracts: their result sources and search spaces differ. See also the algorithm changes.

Programmatic API status

VAMOS 1.1.0 does not expose a curated public programmatic facade for the algorithm-configuration tuner. Maintained user workflows should therefore use vamos tune rather than import deep vamos.engine.* implementation modules.

The implementation-level tuning and racing classes remain documented for advanced evaluation and contributors under the experimental tuning API reference, but those deep imports are not a compatibility commitment or a copy-paste learning path.

What a tuning experiment must define

Before tuning, fix the experimental protocol just as you would for an algorithm comparison:

1. Parameter space — what may change and its legal range.
2. Training instances — the problems on which candidate configurations are selected.
3. Algorithm seeds — stochastic replicates used for every candidate.
4. Evaluation-budget schedule — the optimization budget or fidelity levels given to each candidate/problem/seed run.
5. Scoring contract — the metric, reference point, result source, direction, runtime penalty, and failure handling actually used by the selected tuning interface.
6. Aggregation rule — how repeated problem/seed scores become one scalar tuning score.

CLI scoring contract: hypervolume

In VAMOS 1.1.0, the maintained vamos tune CLI has no metric selector. It scores candidate runs with hypervolume (HV) and maximizes that score.

The score source is fixed: VAMOS takes the final population, removes infeasible rows when constraint values are present, Pareto-filters the remaining objective vectors, and computes HV on that feasible non-dominated population. It does not score top-level result.F, because that field can refer to an external archive for archive-enabled algorithm configurations.

The maintained CLI also removes use_external_archive, archive_unbounded, and archive_prune_policy from its built-in search space. This keeps the optimized parameter space aligned with the set actually used for scoring. Archive design can still be studied explicitly in a separate controlled experiment, but it is not mixed into ordinary vamos tune comparisons.

Constrained AGE-MOEA and RVEA (1.0.1)

Since VAMOS 1.0.1, AGE-MOEA and RVEA preserve population-aligned constraint values whenever constraint handling is active, so constrained runs can use the same final-population HV scoring contract as the other maintained algorithms. Their stable constraint_mode="none" setting remains an explicit opt-out: in that mode constraints are ignored and no G values participate in survival, archives, or population results.

• --ref-point supplies one HV reference point for the tuning run.
• If it is omitted (or cannot be parsed with the required dimensionality), the current default is [10.0, ..., 10.0], one value per objective.
• --runtime-penalty changes the scalar score to HV - lambda * log1p(runtime_seconds); its default is 0.0, so the default score is plain HV.
• --failure-score has a narrower scope than its name may suggest. Its default is 0.0, and it is used when the evaluator catches a failure while running the candidate algorithm and therefore has no usable result. It is not a universal failure policy: exceptions raised later while scoring HV can propagate in the random backend, while racing may substitute its own backend-level sentinel instead of --failure-score.
• --aggregate-mode then combines the per-block scores using mean, median, p25, or p10.

Choose and report a reference point that is meaningful for all problems in the tuning campaign. The CLI uses the same supplied reference point across the selected instances. In particular, validate that the reference point is valid for the objective values produced by every selected problem before a long run; --failure-score should not be relied on to rescue an invalid HV scoring setup. If you need IGD, IGD+, epsilon indicators, or another selection metric, the current maintained CLI cannot select it; that requires a custom/experimental workflow rather than a vamos tune flag.

Current CLI seed coupling

The current CLI also has an important seed-coupling constraint: --seed is the global tuning seed and the base used to derive the training algorithm-seed schedule. --n-seeds changes how many training seeds are derived; it does not let you provide an independent training-seed list.

Therefore changing --seed changes both configuration-search randomness and training evaluation randomness. Record --seed and --n-seeds together and do not interpret them as independently controlled factors. --split-seed separately controls the problem split, while validation/test seed lists can be overridden with --validation-seeds and --test-seeds.

For scientific use, reserve held-out problems and/or seeds for validation or final testing. Do not choose a configuration on the same test blocks used for the final claim.

Install

The built-in random and racing backends use the core installation.

Model-based backends (optuna, smac3, bohb, and bohb_optuna) require the optional tuning dependencies:

python -m pip install "vamos-optimization[tuning]==1.1.0"

For an editable development installation, follow Install from a source checkout and select the tuning extra from the repository root.

Start with a core-installation run

The vamos tune CLI currently defaults to optuna, so omitting --backend requires the optional tuning extra. For a command that works with a core installation, choose a built-in backend explicitly:

vamos tune \
  --algorithm nsgaii \
  --problem zdt1 \
  --backend random \
  --budget 1000 \
  --tune-budget 20 \
  --n-jobs 1

The important budgets are different quantities:

• --budget is the candidate-run MOEA evaluation budget for random and the ordinary non-racing path; it also supplies the baseline budget used by downstream validation/test stages unless those stages override it.
• --tune-budget is the configuration-search budget (trials/experiments, depending on backend).

Do not report only --tune-budget when estimating compute cost. Candidate configurations are evaluated across the selected instances and seeds, and each of those runs consumes its own algorithm budget.

Racing has its own fidelity-budget schedule

racing is a special case. The CLI enables multi-fidelity racing by default. If --fidelity-levels is omitted, the current racing schedule is 1000,3000,10000 evaluations. Those values are passed directly as the candidate-run budgets at successive fidelity levels; they are not capped by --budget. Therefore, for example, --backend racing --budget 5000 can still execute promoted candidate blocks with a 10,000-evaluation budget.

For a fixed-budget racing experiment, disable multi-fidelity explicitly:

vamos tune \
  --algorithm nsgaii \
  --problem zdt1 \
  --backend racing \
  --budget 5000 \
  --no-multi-fidelity

For multi-fidelity racing, specify the schedule explicitly and include it in the experimental record. If 5000 evaluations is intended to be the maximum fidelity, for example:

vamos tune \
  --algorithm nsgaii \
  --problem zdt1 \
  --backend racing \
  --budget 5000 \
  --fidelity-levels 1000,3000,5000

Treat --fidelity-levels, rather than --budget, as the authoritative tuning budget schedule while multi-fidelity racing is enabled. --fidelity-promotion-ratio and --fidelity-min-configs additionally affect how many configurations reach each level, so compute estimates should account for the schedule and promotion policy together.

Cheap verification path

Use --smoke when you only want to verify the CLI, evaluator, and artifact pipeline:

vamos tune \
  --instances zdt1,zdt2,zdt3,dtlz1,dtlz2,wfg1 \
  --algorithm nsgaii \
  --backend random \
  --smoke \
  --output-dir results/tuning_smoke

--smoke clamps budgets and workers and disables validation, test, and statistical-finisher stages. It is a real execution path and is exercised by the documentation smoke suite, but its tiny design is not intended for scientific conclusions.

Choose a backend deliberately

The current backend families are:

• random — built-in random search;
• racing — built-in racing that progressively allocates experiments and can eliminate weak candidates; the CLI enables its multi-fidelity schedule by default;
• optuna — current CLI default; requires the tuning extra;
• bohb_optuna, smac3, and bohb — optional model-based backends from the tuning extra.

A racing run uses the same scientific ingredients as any other tuning run: problem blocks, algorithm seeds, an explicit fidelity/budget schedule, the CLI HV score, and an aggregation rule. Its statistical elimination is an allocation mechanism during tuning; it is not a substitute for an independently designed final comparison on held-out blocks.

Conditional parameters

A trial contains only the parameters that are active for the rest of its assignment: crossover_eta only for SBX, archive_prune_policy only when the external archive is enabled and bounded, and, since VAMOS 1.1.0 for real-valued operator spaces, mutation_eta only for polynomial and linked polynomial mutation. The evaluation callback, the trial history, and the returned best configuration contain that active assignment. Before VAMOS 1.1.0, the model-based backends kept every parameter, and random and racing could keep a parameter whose own conditions held while its parent was inactive. They now still draw every value in declaration order, so their seeded values do not change, and keep a parameter only when its parents are kept.

optuna and bohb_optuna suggest parents before the parameters that depend on them, whatever their declaration order, and record only active suggestions in trial.params. smac3 and bohb receive the dependencies as ConfigSpace conditions, so their models see them too. Several conditions on one parameter are combined with AND, and descendants of an inactive parameter stay inactive. The ConfigSpace translation supports comparisons of cfg['parent'] with a scalar literal (==, !=, <, <=, >, >=), boolean parents, and and/or/not; any other condition, an unknown parent, or a dependency cycle fails with an explicit error before sampling starts. AlgorithmConfigSpace samples and encodes parameters in the same dependency order, so a parameter may be declared before its parent.

HpBandSter's native BOHB accepts no seed, and its configuration generator draws from NumPy's global random state. The bohb backend gives the generator of each search a private random stream derived from the tuning seed instead. NumPy's global state therefore remains entirely the caller's, including inside the evaluation callback. With one worker (--n-jobs 1) a native BOHB search is reproducible for a seed; with several workers, the order in which results arrive can still change it.

Inactive parameters never changed how a candidate algorithm was built, so the scores of studies persisted by earlier versions remain comparable and keep their namespace. Their stored trials may record inactive values; when such a study is resumed, the history and the best configuration leave them out. Seeded random and racing searches propose the same candidates as VAMOS 1.0.2, without an inactive mutation_eta, except for MOEA/D: its mutation_eta moved into the polynomial-mutation block, so a seeded MOEA/D search proposes different candidates.

Persistent Optuna studies are scoring-contract versioned

In VAMOS 1.1.0, when optuna or bohb_optuna uses --optuna-storage, the maintained CLI treats --optuna-study-name as a base name and appends __vamos_cli_final_population_hv_v1 to the effective persisted study name. The same versioned namespace is used whether --optuna-load-if-exists is enabled or disabled.

This isolation is intentional: trials persisted by an older VAMOS tuning CLI may contain archive-derived scores and a different search space, so they must not compete with trials scored under the current final-population HV contract. Reusing the same base study name resumes only studies from this contract version. A future incompatible scoring/search-space change must use a new namespace rather than mixing historical trial values.

Split-based tuning

For a larger campaign, use explicit instance splitting and post-tuning stages:

vamos tune \
  --instances zdt1,zdt2,zdt3,dtlz1,dtlz2,wfg1 \
  --algorithm nsgaii \
  --backend optuna \
  --backend-fallback random \
  --split-strategy suite_stratified \
  --budget 5000 \
  --tune-budget 200 \
  --n-jobs -1

The key controls include:

• --algorithm: algorithm family to tune;
• --instances: comma-separated problem list; overrides --problem;
• --backend: racing, random, optuna, bohb_optuna, smac3, or bohb;
• --backend-fallback: fallback when an optional model backend is unavailable;
• --split-strategy: suite_stratified or random instance splitting;
• --split-seed: random seed for the instance split;
• --seed: coupled global tuner seed and base for training algorithm seeds;
• --n-seeds: number of training algorithm seeds derived from --seed;
• --validation-seeds, --test-seeds: optional explicit post-selection seed schedules;
• --ref-point: global HV reference point used by the CLI scorer;
• --budget: ordinary candidate-run budget; not a cap on enabled racing multi-fidelity levels;
• --tune-budget: racing experiments or model trials;
• --multi-fidelity / --no-multi-fidelity: enable or disable racing multi-fidelity execution;
• --fidelity-levels: explicit increasing candidate-run budgets for the multi-fidelity schedule;
• --aggregate-mode: aggregation across instance/seed scores;
• --n-jobs: parallel workers (-1 means CPU cores minus one);
• --optuna-storage, --optuna-study-name: persistent Optuna storage and the base study name used by the contract-versioned namespace;
• --run-validation, --run-test: optional post-tuning evaluation stages;
• --run-statistical-finisher: optional paired-test selection on the training split top-k.

Define the split and budget schedule before interpreting results. Tuning chooses among candidate configurations; validation can support model-selection decisions; the final test split should answer the pre-specified performance question without being fed back into another tuning round.

Artifacts and provenance

Tuning output includes:

• best_config_raw.json and best_config_active.json;
• tuning_history.json and tuning_history.csv;
• tuning_summary.json;
• split_instances.csv and split_seeds.json;
• optional finisher/validation/test artifacts when those stages are enabled.

Keep these together with the command/configuration and environment used for the campaign. A best configuration without its search space, seeds, HV reference point, fixed final-population score source, aggregation rule, budget/fidelity schedule, and split is not a reproducible tuning result.

Archive studies are separate from ordinary CLI tuning

The 1.1.0 CLI intentionally excludes external-archive controls from its ordinary algorithm-configuration search so every candidate is compared on the same final-population basis. If archive capacity, pruning, or archive-enabled result semantics are themselves the research question, define them explicitly as experimental variants and compare them under a pre-specified, common metric and result-source protocol. For publication-grade claims, run the selected variants on held-out blocks through the stable Study lifecycle and retain the canonical run provenance.

Ablation planning

vamos ablation is also experimental. For a durable scientific comparison of already chosen variants, the stable study lifecycle is preferable: represent each scientifically distinct configuration explicitly, plan the matrix before execution, and retain canonical run provenance.

from pathlib import Path

from vamos import StudySpec, create_study, plan_study

output_root = Path("results/ablation_demo")
summaries = {}
for variant, population_size in {"baseline": 50, "tuned": 80}.items():
    spec = StudySpec(
        problems=["zdt1", "dtlz2"],
        algorithms=["nsgaii"],
        seeds=[1, 2, 3],
        max_evaluations=20_000,
        pop_size=population_size,
        algorithm_configs={"nsgaii": {"pop_size": population_size}},
        labels={"workflow": "ablation", "variant": variant},
    )
    planned = plan_study(spec)
    completed = create_study(spec, output=output_root / variant).run()
    assert completed.plan_id == planned.plan_id
    summaries[variant] = completed.summarize()

Build caller-specific analysis tables from StudySummary.rows, but retain the canonical provenance fields (study_id, plan_id, task_id, selected_run_id, run_manifest_path, and run_manifest_sha256). A derived table is an analysis artifact, not resume authority.

For executable ablation material, see:

• examples/tuning/ablation_runner.py;
• notebooks/2_advanced/32_ablation_planning.ipynb;
• examples/configs/study_nsgaii.json for a configuration whose keys map to StudySpec.

When interpreting variant contributions, define the metric, replicate aggregation, and statistical procedure before comparing deltas against the baseline. Tuning, validation, final testing, and ablation answer different questions; keep their data partitions and provenance explicit.

A bounded worked experiment

Follow the tuning tutorial for a verified standalone CLI wrapper, two recorded search trials and separate validation/test seeds. For model parameters as decision variables, use the mixed SVC example. For a controlled component removal using stable studies, use the ablation lesson.