Foundation model for hysteresis modeling

Following the work of Jiles-Atherton model for SPS main magnetswe simulate additional waveforms, but this time remembering to smooth the current waveform before simulating, to avoid artifacts.

Previous attempts to avoid the artifacts were to remove the random small changes during the ramp and plateaus. However this proved to make the model overfit very early during training (since the all ramps and plateaus were completely straight). We solve this by adding back the random changes in di/dt, and smoothing using a gaussian filter instead of the roundoff function provided by Marco, which requires the previous rounding to be completed before being able to do the next (and therefore introducting artifacts if areas are overlapping).

We apply RDPwith epsilons $\epsilon \in [2e-5,5e-5,8e-5,1e-4,2e-4,5e-4,1e-3,2e-3]$ , which yields the following diagram showing number of points kept and RMSE w.r.t. the original signal

We therefore choose the validation RDP $\epsilon =5e-4$ to be the best balance between sample efficiency and accuracy, and $\epsilon \in [1e-4,2e-4,5e-4]$ for the training set. We additionally decrease the maximum time step from 6000+ms to around 350 ms to give the normalizer enough resolution to work with, and reduce outliers. This adds between 0.1 % and 0.8 % of points, still allowing us to stay below 5% points kept.

To avoid numerical errors accumulating, we simulate over 2h, 10 times, for training, and over 1h 3 times for validation, before applying RDP.

The datasets used for training were the following:

data:
  init_args:
    train_df_paths:
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_0_2h_rdp_eps_1e-04.parquet
    -~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_0_2h_rdp_eps_2e-04.parquet
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_0_2h_rdp_eps_5e-04.parquet
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_1_2h_rdp_eps_1e-04.parquet
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_1_2h_rdp_eps_2e-04.parquet
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_1_2h_rdp_eps_5e-04.parquet
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_2_2h_rdp_eps_1e-04.parquet
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_2_2h_rdp_eps_2e-04.parquet
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_2_2h_rdp_eps_5e-04.parquet
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_3_2h_rdp_eps_1e-04.parquet
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_3_2h_rdp_eps_2e-04.parquet
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_3_2h_rdp_eps_5e-04.parquet
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_4_2h_rdp_eps_1e-04.parquet
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_4_2h_rdp_eps_2e-04.parquet
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_4_2h_rdp_eps_5e-04.parquet
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_5_2h_rdp_eps_1e-04.parquet
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_5_2h_rdp_eps_2e-04.parquet
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_5_2h_rdp_eps_5e-04.parquet
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_6_2h_rdp_eps_1e-04.parquet
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_6_2h_rdp_eps_2e-04.parquet
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_6_2h_rdp_eps_5e-04.parquet
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_7_2h_rdp_eps_1e-04.parquet
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_7_2h_rdp_eps_2e-04.parquet
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_7_2h_rdp_eps_5e-04.parquet
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_8_2h_rdp_eps_1e-04.parquet
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_8_2h_rdp_eps_2e-04.parquet
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_8_2h_rdp_eps_5e-04.parquet
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_9_2h_rdp_eps_1e-04.parquet
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_9_2h_rdp_eps_2e-04.parquet
    - ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/train_sim_v2_9_2h_rdp_eps_5e-04.parquet
    val_df_paths: ~/cernbox/hysteresis/dipole/datasets/simulation/rdp/val_sim_v2_0_1h_rdp_eps_5e-04.parquet

Hyperparameter tuning

We run hyperparameter tuning on the different datasets.

GRID = {
    "num_lstm_layers": ray.tune.choice([1, 2, 3]),
    "n_dim_model": ray.tune.choice([64, 128, 256, 300, 500]),
    "ctxt_seq_len": ray.tune.choice([100, 200, 300, 600, 1200]),
    "tgt_seq_len": ray.tune.choice([100, 200, 300, 400, 600]),
    "quantiles": ray.tune.choice([
        [0.1, 0.5, 0.9],
        [0.25, 0.5, 0.75],
        [0.02, 0.1, 0.25, 0.5, 0.75, 0.9, 0.98],
        [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9],
        [
            0.067,
            0.133,
            0.2,
            0.267,
            0.333,
            0.4,
            0.467,
            0.5,
            0.533,
            0.6,
            0.667,
            0.733,
            0.8,
            0.867,
            0.933,
        ],
        # 15 quantiles
        [0.05 * i for i in range(1, 21)],
    ]),
    "hidden_continuous_dim": ray.tune.choice([32, 64, 128, 256]),
    "num_heads": ray.tune.choice([2, 4, 8]),
    "dropout": ray.tune.uniform(0.0, 0.4),
}

• Finetune foundation model on SPS MBIs. [priority:: high] [due:: 2024-11-12] [completion:: 2024-11-26]

Best parameters:

num_lstm_layers: 3
n_dim_model: 256
ctxt_seq_len: 100
tgt_seq_len: 300
quantiles: [0.02, 0.1, 0.25, 0.5, 0.75, 0.9, 0.98]
hidden_continuous_dim: 128
num_heads: 8
dropout: 0.036