Abdulmohsen Saleh

Artificial Intelligence Specialization · Cohort 2026

Batch Normalization using an SVD Layer

Regularization in its various forms has been extensively used in neural networks, particularly in unsupervised learning models such as autoencoders. Recently, Mounayer et al. introduced Rank Reduction Autoencoders (RRAEs), demonstrating both theoretically and empirically that the Singular Value Decomposition (SVD) can serve as a powerful regularizer for autoencoders without requiring the optimization of additional loss functions. While this archiitecture shows many advantages, the effect of the SVD by itself has not been studied seperately. In this work, we present a new type of layers called SVD layer. We derive an analytical expression of the distribution imposed by the truncated SVD and show how to normalizes batches using the singular values of the SVD. Our method is to be compared with batch norm and studied to see what advantages it can have. We will also provide a PyTorch code where the layer can be included in Torch modules and be used by others easily.

Read the final report (PDF)