NL4AI 2024 Paper

Our paper ‘Fantastic Labels and Where to Find Them: Attention-Based Label Selection for Text-to-Text Classification’ (with Michele Papucci and Felice Dell’Orletta) has been accepted to CLiC-it 2024! In this work, we introduce a novel method for selecting well-performing label representations by leveraging the attention mechanisms of Transformer models. We used an Italian T5 model fine-tuned on a topic classification task, trained on posts extracted from online forums and categorized into 11 classes, to evaluate different label representation selection strategies. We’ve employed a context-mixing score called Value Zeroing to assess each token’s impact to select possible representations from the training set. Our results include a detailed qualitative analysis to identify which label choices most significantly affect classification outcomes, suggesting that using our approach to select label representations can enhance performance.