Covid Vaccine Search Classification with Pretrained Transformers and Dense Feature Memory

With the devastating outbreak of Covid, vaccines are the biggest line of defense that provide hope of bringing things back to normal. Given the safeguard that they provide, vaccines are becoming mandatory in certain social and professional settings. As such, governments all around the world have been been trying to increase the reach of these vaccines to protect their respective communities. Trends of search queries pertaining to vaccination in different geographies can provide very valuable information and insights to stakeholders. This paper presents a model for Covid-19 vaccination related search query classification, a machine learned model that is used to generate Covid-19 Vaccine Search insights (VSI). The proposed method combines and leverages advances from modern state-of-the-art natural language understanding (NLU) techniques such as pretrained Transformers with traditional dense features. In order to combine SOTA NLU models with traditional features, we propose a novel approach of considering dense features as memory tokens. Specifically, we learn a parameterized transformation that maps dense features to a memory store, allowing the model to not only learn latent query-feature interactions but also dense feature-conditioned query representations. We show that this new modeling advance enables a significant improvement to the VSI task, improving a strong well-established gradient-boosting baseline by +15\% in terms of F1 score.