Multi-Grained Topological Pre-Training of Language Models in Sponsored Search

Relevance models measure the semantic closeness between queries and the candidate ads, widely recognized as the nucleus of sponsored search systems. Conventional relevance models solely rely on the textual data within the queries and ads, whose performance is hindered by the scarce semantic information in these short texts. Recently, user behavior graphs have been incorporated to provide complementary information beyond pure textual semantics. Despite the promising performance, behavior-enhanced models suffer from exhausting resource costs due to the extra computations introduced by explicit topological aggregations. In this paper, we propose a novel topological pre-training paradigm MGTLM to enjoy the merits of behavior graphs while avoiding explicit topological operations. MGTLM is capable of teaching language models to understand multi-grained topological information, which contributes to eliminating explicit graph aggregations and avoiding information loss. Empirically, our proposal is extensively evaluated over both online and offline settings, and the experimental results demonstrate the superiority of our proposal.