What Makes My Model Perplexed? A Linguistic Investigation on Neural Language Models Perplexity

@inproceedings{miaschi-etal-2021-makes,
    title = "What Makes My Model Perplexed? A Linguistic Investigation on Neural Language Models Perplexity",
    author = "Miaschi, Alessio  and
      Brunato, Dominique  and
      Dell{'}Orletta, Felice  and
      Venturi, Giulia",
    booktitle = "Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures",
    month = jun,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2021.deelio-1.5",
    pages = "40--47",
    abstract = "This paper presents an investigation aimed at studying how the linguistic structure of a sentence affects the perplexity of two of the most popular Neural Language Models (NLMs), BERT and GPT-2. We first compare the sentence-level likelihood computed with BERT and the GPT-2{'}s perplexity showing that the two metrics are correlated. In addition, we exploit linguistic features capturing a wide set of morpho-syntactic and syntactic phenomena showing how they contribute to predict the perplexity of the two NLMs.",
}