RefAM: Attention Magnets for Zero-Shot Referral Segmentation
Abstract
Internal features from large-scale pre-trained diffusion models are known to encode rich semantic information. However, standard approaches extract features by feeding noisy images into the model, degrading the quality of the representation. We propose InvertDIFT, a method that leverages diffusion inversion to recover clean representations without modifying the model. By inverting the noisy forward process to retrieve clean or intermediate states, we extract timestep-independent features from the diffusion model as is. This approach preserves the model's pre-trained knowledge and eliminates the need for additional training, while achieving performance on par or better than fine-tuned approaches across several tasks.
