RATE-DISTORTION OPTIMIZED LORA FOR EFFICIENT POST-FILTERING IN AV2
Abstract
This paper proposes RD-LoRA, a rate-distortion (R-D) optimized low-rank adaptation (LoRA) framework for neural post-filtering in the upcoming AV2 coding standard. RD-LoRA adapts a pre-trained base neural model to diverse input content via online updating of LoRA parameters, whose quantity is governed by the matrix ranks. The updated parameters are then quantized, transmitted to the decoder, and merged with the pre-trained weights for post-filtering. While more parameters generally improve coding performance, they also increase transmission bitrate.
To balance distortion reduction against transmission cost, we propose to dynamically allocate rank budget to each layer of the base model in a closed-loop R-D manner. Specifically, it incorporates a cost-aware importance assessment to discourage parameter-heavy updates, together with an RD-rank allocator to guide pruning based on global R-D optimization. To maintain robustness when certain layers are pruned to zero rank, we further introduce a lightweight fallback modulation mechanism. Experimental results show that, when deployed on a lightweight 25 kMACs ResNet model, RD-LoRA achieves a BD-rate reduction of 2.556% over the AV2 anchor, significantly outperforming the base model with only limited decoding overhead.
To balance distortion reduction against transmission cost, we propose to dynamically allocate rank budget to each layer of the base model in a closed-loop R-D manner. Specifically, it incorporates a cost-aware importance assessment to discourage parameter-heavy updates, together with an RD-rank allocator to guide pruning based on global R-D optimization. To maintain robustness when certain layers are pruned to zero rank, we further introduce a lightweight fallback modulation mechanism. Experimental results show that, when deployed on a lightweight 25 kMACs ResNet model, RD-LoRA achieves a BD-rate reduction of 2.556% over the AV2 anchor, significantly outperforming the base model with only limited decoding overhead.
Research Areas
