Hysteresis Compensation

TFT hyperparameter search

1 min read

Optimal hyperparameters

We perform a hyperparameter search for the TFT on simulated data to determine the optimal hyperparameters to learn hysteresis.

A grid search was initiated on 2024-11-28 on ml4, that will take approximately 2 weeks.

Grid:

GRID = {
    "num_lstm_layers": ray.tune.choice([1, 2, 3]),
    "n_dim_model": ray.tune.choice([64, 128, 256, 300, 500]),
    "ctxt_seq_len": ray.tune.choice([100, 200, 300, 600, 1200]),
    "tgt_seq_len": ray.tune.choice([100, 200, 300, 400, 600]),
    "quantiles": ray.tune.choice([
        [0.1, 0.5, 0.9],
        [0.25, 0.5, 0.75],
        [0.02, 0.1, 0.25, 0.5, 0.75, 0.9, 0.98],
        [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9],
        [
            0.067,
            0.133,
            0.2,
            0.267,
            0.333,
            0.4,
            0.467,
            0.5,
            0.533,
            0.6,
            0.667,
            0.733,
            0.8,
            0.867,
            0.933,
        ],
        # 15 quantiles
        [0.05 * i for i in range(1, 21)],
    ]),
    "hidden_continuous_dim": ray.tune.choice([32, 64, 128, 256]),
    "num_heads": ray.tune.choice([2, 4, 8]),
    "dropout": ray.tune.uniform(0.0, 0.4),
    "weight_decay": ray.tune.loguniform(1e-5, 1e-2),
    "lr": ray.tune.loguniform(1e-5, 1e-3),
    "time_format": ray.tune.choice(["relative", "absolute"])
}

PARAM2KEY = {
    "num_lstm_layers": "model.init_args.num_lstm_layers",
    "n_dim_model": "model.init_args.n_dim_model",
    "ctxt_seq_len": "data.init_args.ctxt_seq_len",
    "tgt_seq_len": "data.init_args.tgt_seq_len",
    "quantiles": "model.init_args.criterion.init_args.quantiles",
    "hidden_continuous_dim": "model.init_args.hidden_continuous_dim",
    "num_heads": "model.init_args.num_heads",
    "dropout": "model.init_args.dropout",
    "lr": "optimizer.init_args.lr",
    "weight_decay": "optimizer.init_args.weight_decay",
    "time_format": "data.init_args.time_format"
}
  • Evaluate TFT hyperparameter search [priority:: medium] [due:: 2024-12-12] [completion:: 2025-01-23]

Graph View