Friday September 29 at the Bright Night, we presented the activity of our laboratory. You can find a brief interview about the lab and our activities at the following link: https://www.cnrweb.tv/il-bright-del-cnr-in-centro-citta-a-pisa/.

Two papers accepted at CLiC-it 2023! In ‘Lost in Labels’ (with Michele Papucci and Felice Dell’Orletta) we present an evaluation of the influence of label selection on the performance of a Sequence-to-Sequence Transformer model in a classification task. Our study investigates whether the choice of words used to represent classification categories affects the model’s performance, and if there exists a relationship between the model’s performance and the selected words. To achieve this, we fine-tuned an Italian T5 model on topic classification using various labels. Our results indicate that the different label choices can significantly impact the model’s performance. That being said, we did not find a clear answer on how these choices affect the model performances, highlighting the need for further research in optimizing label selection. In ‘Unmasking the Wordsmith: Revealing Author Identity through Reader Reviews’ (with Chiara Alzetta, Felice Dell’Orletta, Chiara Fazzone and Giulia Venturi) we propose a novel task called Book Author Prediction, where we predict the author of a book based on user-generated reviews’ writing style. To this aim, we first introduce the Literary Voices Corpus (LVC), a dataset of Italian book reviews, and use it to train and test machine learning models. Our study contributes valuable insights for developing user-centric systems that recommend leisure readings based on individual readers’ interests and writing styles.

Our paper ‘Tell me how you write and I’ll tell you what you read: a study on the writing style of book reviews’ (with Chiara Alzetta, Felice Dell’Orletta, Elena Prat and Giulia Venturi) has been accepted for publication in the next issue of the journal of Documentation. In this work we investigate variations in the writing style of book reviews published on different social reading platforms and referring to books of different genres. In particular, we propose a corpus-based study focused on the analysis of A Good Review, a novel corpus of online book reviews written in Italian, posted on Amazon and Goodreads, and covering six literary fiction genres. We rely on stylometric analysis to explore the linguistic properties and lexicon of reviews and we conducted automatic classification experiments using multiple approaches and feature configurations to predict either the review’s platform or the literary genre.

I am glad to announce that on Friday, June 9th, I will give a talk at the DCP23 Workshop in Pisa. DCP is an inter-disciplinary workshop focused on non-linear dynamics, statistical mechanics and complexity in multiple areas, from mathematics to philosophy, biology, physiology, economy and social sciences, among others. Title: Opening Large Language Models Abstract: As language models become increasingly complex and sophisticated, the processes leading to their predictions are growing increasingly difficult to understand. Research in NLP interpretability focuses on explaining the rationales driving model predictions and is crucial for building trust and transparency in the usage of these systems in real-world scenarios. In this talk, we will first introduce state-of-the-art Neural Language Models (NLMs) and discuss their characteristics. Then we will cover the most commonly applied analysis methods for understanding the inner behaviour of NLMs based on Transformer architectures and how they implicitly encode linguistic knowledge.

I am glad to announce that on May 31 I will be hosting with Gabriele Sarti a laboratory focused on the interpretability of Neural Language Models (NLMs) at the 2023 edition of the Lectures on Computational Linguistics. Below you can find title and abstract of the lab: Title: Explaining Neural Language Models from Internal Representations to Model Predictions Abstract: As language models become increasingly complex and sophisticated, the processes leading to their predictions are growing increasingly difficult to understand. Research in NLP interpretability focuses on explaining the rationales driving model predictions and is crucial for building trust and transparency in the usage of these systems in real-world scenarios. ...

Our paper ‘Testing the Effectiveness of the Diagnostic Probing Paradigm on Italian Treebankss’ (with Chiara Alzetta, Dominique Brunato, Felice Dell’Orletta and Giulia Venturi) has been accepted for publication in the next issue of the Information journal. In this work we contribute to the debate on the effectiveness of the linguistic probing paradigm by presenting an approach to assessing the effectiveness of a suite of probing tasks aimed at testing the linguistic knowledge implicitly encoded by one of the most prominent NLMs, BERT. To this aim, we compared the performance of probes when predicting gold and automatically altered values of a set of linguistic features. Our experiments were performed on Italian and were evaluated across BERT’s layers and for sentences with different lengths. As a general result, we observed higher performance in the prediction of gold values, thus suggesting that the probing model is sensitive to the distortion of feature values. However, our experiments also showed that the length of a sentence is a highly influential factor that is able to confound the probing model’s predictions.

I am happy to announce that I will be co-organizing a shared task at EVALITA 2023, the evaluation campaign of NLP and Speech Tools for Italian, that will have place in Parma on September 7-8 2023. For more information please visit the shared task web page: LangLearn: Language Learning Development at EVALITA 2023

Our paper ‘On Robustness and Sensitivity of a Neural Language Model: A Case Study on Italian L1 Learner Errors’ (with Dominique Brunato, Felice Dell’Orletta and Giulia Venturi) has been accepted for publication in the next issue of the IEEE/ACM Transactions on Audio, Speech and Language Processing journal. In this work, we propose a comprehensive linguistic study aimed at assessing the implicit behaviour of one of the most prominent Neural Language Model (NLM) based on Transformer architectures, BERT (Devlin et al., 2019), when dealing with a particular source of noisy data, namely essays written by L1 Italian learners containing a variety of errors targeting grammar, orthography and lexicon. Differently from previous works, we focus on the pre-training stage and we devise two evaluation tasks aimed at assessing the impact of errors on sentence-level inner representations from two complementary perspectives, i.e. robustness and sensitivity. Our experiments show that BERT’s ability to compute sentence similarity and to correctly encode a set of raw and morpho-syntactic properties of a sentence are differently modulated by the category of errors and that the error hierarchies in terms of robustness and sensitivity change across layer-wise representations.

Our paper ‘Evaluating Text-To-Text Framework for Topic and Style Classification of Italian texts’ (with Michele Papucci, Chiara De Nigris and Felice Dell’Orletta) has been accepted at the NL4AI Workshop (AIxIA Conference). In this paper, we propose an extensive evaluation of the first text-to-text Italian Neural Language Model (NLM), IT5, on a classification scenario. In particular, we test the performance of IT5 on several tasks involving both the classification of the topic and the style of a set of Italian posts. We assess the model in two different configurations, single- and multi-task classification, and we compare it with a more traditional NLM based on the Transformer architecture (i.e. BERT). Moreover, we test its performance in a few-shot learning scenario. We also perform a qualitative investigation on the impact of label representations in modeling the classification of the IT5 model. Results show that IT5 could achieve good results, although generally lower than the BERT model. Nevertheless, we observe a significant performance improvement of the Text-to-text model in a multi-task classification scenario. Finally, we found that altering the representation of the labels mainly impacts the classification of the topic.

I am glad to announce that on October 27 I will give a tech talk at Pi School. Title: Interpreting Neural Language Models Abstract: The field of Natural Language Processing (NLP) has seen an unprecedented progress in the last few years. Much of this progress is due to the replacement of traditional systems with newer and more powerful algorithms based on neural networks and deep learning. This improvement, however, comes at the cost of interpretability, since deep neural models offer little transparency about their inner workings and their abilities. Therefore, in the last few years, an increasingly large body of work has been devoted to the analysis and interpretation of these models. This talk will be divided into two parts. In the first part, we will briefly introduce Neural Language Models (NLMs) and the main techniques developed for interpreting their decisions and their inner linguistic knowledge. In the second part, we will see how to fine-tune one of the most popular NLM and then analyze its decisions according to two different interpretability methods: integrated gradients and analysis of attention matrices. ...

I am glad to announce that on September 21 I will give a talk at the International Summer School on “Advances in Artificial Intelligence” (see below for the details). The main purpose of the school is to gather scholars, researchers and PhD students to learn and explore the main advanced topics offered by AI with a wide look towards new perspectives coming by innovative technological scenarios. Title: Profiling Neural Language Models Abstract: The field of Natural Language Processing (NLP) has seen an unprecedented progress in the last years. Much of this progress is due to the replacement of traditional systems with newer and more powerful algorithms based on neural networks and deep learning. This improvement, however, comes at the cost of interpretability, since deep neural models offer little transparency about their inner workings and their abilities. Therefore, in the last few years, an increasingly large body of work has been devoted to the analysis and interpretation of these models. ...

I am glad to announce that on May 24 I have successfully defended my PhD thesis, ‘Tracking Linguistic Abilities in Neural Language Models’. You can find the pdf of my thesis at the following link: https://etd.adm.unipi.it/theses/available/etd-05062022-162420/.

Our paper ‘Evaluating Transformer Models for Punctuation Restoration in Italian’ (with Andrea Amelio Ravelli and Felice Dell’Orletta) has been accepted at the NL4AI Workshop (AIxIA Conference). In this paper, we propose an evaluation of a Transformer-based punctuation restoration model for the Italian language. Experimenting with a BERT-base model, we perform several fine-tuning with different training data and sizes and tested them in an in- and cross-domain scenario. Moreover, we offer a comparison in a multilingual setting with the same model fine-tuned on English transcriptions. Finally, we conclude with an error analysis of the main weaknesses of the model related to specific punctuation marks.

Two papers accepted at CLiC-it 2021! In ‘Probing Tasks Under Stress’ (with Chiara Alzetta, Dominique Brunato, Felice Dell’Orletta and Giulia Venturi) we introduced a new approach to put increasingly under pressure the effectiveness of a suite of probing tasks to test the linguistic knowledge implicitly encoded by a BERT Italian model. To achieve this goal, we set up a number of experiments aimed at comparing the performance of a regression model trained with BERT representations to predict the values of a set of linguistic properties extracted from the Italian Universal Dependency Treebank and from a suite of control datasets we specifically built for the purpose of this study. In ‘On the role of Textual Connectives in Sentence Comprehension: a new Dataset for Italian’ (with Giorgia Albertin, Alessio Miaschi and Dominique Brunato) we presented a new evaluation resource for Italian aimed at assessing the role of textual connectives in the comprehension of the meaning of a sentence. The resource is arranged in two sections, corresponding to a distinct challenge task conceived to test how subtle modifications involving connectives in real usage sentences influence the perceived acceptability of the sentence by native speakers and Neural Language Models (NLMs). Although the main focus is the presentation of the dataset, we also provided some preliminary data comparing human judgments and NLMs performance in the two tasks.

Last saturday at the Science Web Festival we presented our educational workshop ‘Ehi Siri, che cos’è la Linguistica Computazionale?’ organized in collaboration with AILC (Associazione Italiana di Linguistica Computazionale). You can find the video of the presentation (in Italian) at the following link: https://www.youtube.com/watch?v=HGTpAXXRkWA.

4 papers accepted at NAACL 2021 workshops! 'What Makes My Model Perplexed? A Linguistic Investigation on Neural Language Models Perplexity' (with Dominique Brunato, Felice Dell'Orletta and Giulia Venturi): We studied how the linguistic structure of a sentence affects the perplexity of BERT and GPT-2 models (accepted at DeeLIO 2021). 'How Do BERT Embeddings Organize Linguistic Knowledge?' (with Giovanni Puccetti e Felice Dell'Orletta): We proposed a study, based on Lasso regression, to understand how the information encoded by BERT sentence-level representations is arrange within its hidden units (accepted at DeeLIO 2021). 'Teaching NLP with Bracelets and Restaurant Menus: An Interactive Workshop for Italian Students' (with Ludovica Pannitto, Lucia Busso, Claudia Roberta Combei, Lucio Messina, Gabriele Sarti e Malvina Nissim): We illustrated an interactive workshop designed to delinate the basic principles of NLP and computational linguistics to high school Italian students aged between 13 and 18 years (accepted at Teaching NLP 2021). 'A dissemination workshop for introducing young Italian students to NLP' (with Lucio Messina, Lucia Busso, Claudia Roberta Combei, Ludovica Pannitto, Gabriele Sarti e Malvina Nissim): We described and made available the game-based material developed for the laboratory mentioned in 'A dissemination workshop for introducing young Italian students to NLP' (accepted at Teaching NLP 2021).

Our paper ‘A NLP-based stylometric approach for tracking the evolution of L1 written language compentece’ (with Dominique Brunato and Felice Dell’Orletta) is finally out and will feature in the next issue of JoWR! In this paper we demonstrated that linguistic features automatically extracted from text not only allow making explicit the relevant transformations occurring in L1 learners’ writing competence but can be exploited as effective predictors in the automatic classification of the chronological order of essays written by the same student, especially at more distant temporal spans. We showed that features related to the error annotation, as well as features belonging to the use of grammatical categories and to the inflectional properties of verbs, acquire much more relevance as the temporal span increase. Finally, we found that the student’slearning curve varies according at least to the geographical area where the school is located: when a higher temporal span is considered, the classifier is more confident about its decision for texts written by students who belong to suburban schools.

I am very proud to announce that our paper ‘Linguistic Profiling of a Neural Language Model’ (with Dominique Brunato, Felice Dell’Orletta and Giulia Venturi) has been awarded as Outstanding Paper for COLING 2020! Here’s the official announcement: https://coling2020.org/2020/11/29/outstanding-papers.html The paper will be presented on Tuesday 8 at 17:00 - 17:30 (CET).

Two papers accepted at CLiC-it 2020! In ‘Italian Transformers Under the Linguistic Lens’ (with Gabriele Sarti, Dominique Brunato, Felice Dell’Orletta and Giulia Venturi) we present an in-depth investigation of the lingusitic knowledge encoded by the Transformer models currently available for the Italian language. In particular, we showed that Multilayer Perceptron is the best model for inferring the amount of information implicitly encoded in the Transformers representations. We also observed that BERT-base-italian achieved best scores in average, but the linguistic generalization abilities of the examined models vary according to specific groups of linguistic phenomena and according to distinct textual genres. In ‘Is Neural Language Model Perplexity Related to Readability?’ (with Chiara Alzetta, Dominique Brunato, Felice Dell’Orletta and Giulia Venturi) we explore the relationship between Neural Language Model (NLM) perplexity and (automatically assessed) sentence readability. Starting from the evidence that NLMs implicitly acquire sophisticated linguistic knowledge from a huge amount of training data, our goal is to investigate whether perplexity is affected by linguistic features used to automatically assess sentence readability and if there is a correlation between the two metrics. Our findings highlight that no significant correlation can be found, either between the two metrics and the set of linguistic features that mostly impact their values.

Our paper ‘Linguistic Profiling of a Neural Language Model’ (with Dominique Brunato, Felice Dell’Orletta and Giulia Venturi) has been accepted at COLING 2020! In this paper we investigate the linguistic knowledge learned by a Neural Language Model (BERT) before and after a fine-tuning process and how this knowledge affects its predictions during several classification problems. We use a wide set of probing tasks, each of which corresponds toa distinct sentence-level feature extracted from different levels of linguistic annotation. In particular, we showed that BERT encodes a wide range of linguistic properites, but the order in which they are stored in the internal representations does not necessarily reflect the traditional division with respect to the linguistic annotation levels. We also found that BERT tends to lose its precision in encoding linguistic features after a fine-tuning process (Native Language Identification), probably because it is storing more task–related information for solving the task. Finally, we showed that the implicit linguistic knowledge encoded by the NLM positively affects its ability to solve the tested downstream tasks.