Switchable CNNs for In-loop Restoration and Super-resolution for AV2

Two related tasks where CNN-based models have shown huge promise are in-loop restoration in videos and frame super-resolution. In our previous work, we presented a framework that uses a common CNN architecture with downloadable model parameters (weights and biases) for both these tasks, for incorporation into a next-generation video codec. The advantage of a common architecture with switchable parameters is that a single hardware inference engine can be utilized in all cases of same-resolution and super-resolution restoration, thereby limiting implementation costs. In this paper, we fully integrate this framework into the under-development AV2 video codec from the Alliance for Open Media (AOM). We also implement an algorithm for encoder-side selection of the super-resolution scale factor. With this implementation, we present the total compression improvement results for in-loop restoration and super-resolution tasks together, to showcase the combined benefit of this framework. As in our previous work, we constrain the model architecture and the encoder implementation to be as simple as possible.