AVM In-loop Filtering using Adaptive Hardware-friendly Neural Networks

Learning-based methods have demonstrated significant efficacy in video codecs in recent years. However, practical implementation necessitates fulfilling several constraints. The neural networks used must be exceptionally low in computational complexity and require minimal area cost in hardware. They must be designed to employ fixed-point operations rather than floating-point operations to ensure decoder-side compliance. Practical decoder hardware implementations also require block-by-block processing to facilitate pipelining. Crucially, even after these simplifications, the coding gain resulting from such a tool must be big enough to justify the implementation cost. In this study, we present a neural network based in-loop filtering tool for AVM that adheres to all these constraints while still achieving about -1.12% PSNR-Y improvement and -1.89% VMAF improvement in a random-access setting, with as much as -4.71% VMAF improvement for 4K content. We also present a lighter version of the neural network which reduces the complexity by more than half while retaining proportional improvements.