Normalization of variables can be either of
- MinMax-scaling
- Max-scaling
- Standard scaling
Reasons to normalize data
Normalization is important for several reasons when training neural networks:
-
Faster Convergence: Normalization helps gradient-based optimization algorithms converge faster. When features have different scales, the loss landscape can be elongated or skewed, making it difficult for the optimizer to find the optimal solution efficiently. Normalization creates a more spherical loss landscape, leading to faster convergence.
-
Improved Stability: Normalization helps prevent issues like vanishing or exploding gradients, which can occur when the scales of the features or the weights of the network are significantly different. These issues can make training unstable and lead to poor performance.
-
Fairer Feature Comparison: When features are on different scales, features with larger scales can dominate the learning process, even if they are not more informative. Normalization ensures that all features are treated equally, allowing the network to learn from all features effectively.
-
Better Generalization: In some cases, normalization can improve the generalization performance of the model by preventing it from overfitting to the specific scale of the training data.