EMNLP 2024 Paper

Our paper ‘Evaluating Large Language Models via Linguistic Profiling’ (with Felice Dell’Orletta and Giulia Venturi) has been accepted at EMNLP 2024! In this paper, we introduce a novel evaluation methodology designed to test LLMs' sentence generation abilities under specific linguistic constraints. Drawing on the `linguistic profiling' approach, we rigorously investigate the extent to which five LLMs of varying sizes, tested in both zero- and few-shot scenarios, effectively adhere to (morpho)syntactic constraints. Our findings shed light on the linguistic proficiency of LLMs, revealing both their capabilities and limitations in generating linguistically-constrained sentences.