Abstract:
In morphology, the type frequencies of the different patterns in existing forms have a considerable influence in the way speakers create new forms, i.e., on the productivity of those patterns. However, type frequencies are sometimes insufficient to fully account for the productivity, or lack thereof, of some morphological patterns. This raises the question of which other factors may play a role in the abstraction and updating of the representations that guide morphological productivity. This thesis addresses this question by exploring linguistic situations in which the influence of the type frequencies is limited by a low number of existing forms and/or may conflict with the new input speakers encounter in an interaction.
Our results show that the type frequencies indeed play a major role in the way speakers create novel forms. However, speakers sometimes fail to abstract a pattern when the number of existing forms is too low. In this situation, certain patterns are over- or under-represented in novel forms as compared to their type frequencies among existing forms. The presence of preferences which are unmotivated by type frequencies suggests that speakers have inherent biases towards certain patterns.
Besides, speakers are able to quickly converge with an interaction partner, even when this goes against the tendencies observed in existing forms. Thus, it follows that speakers can gradually modify their preferences through the accumulated effect of morphological convergence in multiple interactions. Consequently, these findings suggest an important role for convergence in the emergence and evolution of morphological patterns.
A data intensive approach for characterizing speech interpersonal dynamics in natural conversations
Under the direction of Laurent Prévot (LPL) and Benoit Favre (LIS)
Jury members:
Prof. Julia Hirschberg, Columbia University
Prof. Giuseppe Riccardi, Università degli Studi di Trento
Prof. Stefan Benus, Constantine the Philosopher University
Cr. Roxane Bertrand, LPL
Director: Prof. Laurent Prevot, LPL
Co-Director: Prof. Benoit Favre, LIS
Abstract:
During a conversation, participants tend to tune, consciously or not, their communicative production in regards to their interlocutor. It is generally admitted, that under standard circumstances, these phenomena result in convergence of the two participants’ speech parameters.
Past literature offers a large part of studies describing the effects of convergence in interpersonal dynamics but there are still some unclear aspects.
These concerns firstly the mechanisms that rule the phenomenon in natural conversations. These are hard to be studied due to the spontaneous flow of the conversants that results to be noisy and variable. In second place in this kind of conversation is still not well known how participants modify their speech style (the dynamics i.e.) in the course of the conversation.
In this thesis, we aim to validate previous results in acoustic-prosodic convergence and provide novel approaches to have a partial a posteriori filter on natural conversations and to track the interpersonal dynamics.
We firstly perform a replication study on the speech rate, confirming that speaker speech rate in the entire conversation converge to their interlocutor speech rate baseline (average speech rate they have in other conversations) even if we perform the analysis on smaller subsets of the original dataset. On the other side, we raised that convergence effects are less reliable in magnitude and significance when reducing the size of the dataset.
In the second part, we explore the dynamics of convergence effects by comparing the distances of average acoustic-prosodic features in the two halves of each conversation (interval of the same temporal length) between the speaker and interlocutor. Results exhibit that both energy and speech rate show convergence in the second half of the conversations of the corpus. In addition, we extend this approach by proposing to study natural conversations comparing similar speech activities. This approach has the advantage to have a posteriori control of the natural flow of the speaker and interlocutor in spontaneous conversation. We observed that the comparison of speaker and interlocutor in more homogeneous speech activities leads to having convergent effects even if the size of the sample is much smaller than the uncontrolled dataset. Based on this idea the thesis proposes a way to automatically tag speech activities for unlabelled data of this kind with the use of a recent LSTM net for classification.
Besides measuring distances between speaker and interlocutor we propose a prediction classifier paradigm to explore the speaker and interlocutor position in the second half of the conversation.
By the use of a Random Classifier, we correlate the use of linguistics variables that describes the trend of speech style of speaker and interlocutor with profile information with the increase of accuracy score in predicting the speech rate variation in the second half of the conversation.
In the last part, we deepen the study of the dynamics in a more fine grain segments of the conversations. The goal is the prediction of mean variables (energy, range F0 and speech rate) in the upcoming turn by the use of previous turns history information that include speech style and lexical information; results, achieved by the use of separately LSTM and LSTM with word embeddings layer, exhibit that the use
of interlocutor and speaker speech style in the previous turns reduce the prediction error of the upcoming turn compared to the case of using just past turns of the speaker.
These results extend the landscape of convergence effects in the not controlled dataset and offer novel approaches, concerning the method to control the variability of natural conversations and the prediction task paradigm to evaluate the interpersonal dynamics, consisting in evaluating the influence of the speaker and interlocutor on each other speech style.
Abstract:
While neuroimaging and behavioral studies have shown that sensory-motor systems are recruited during semantic processing, how and when this occurs has not yet been clearly established. Our purpose was to observe the different contexts in which motor activation can contribute to language comprehension and learning, using interactive and ecologically valid environments. In our first study, novice learners acquired a reduced second language (L2) lexicon through interactive computer games. Behavioral and electroencephalography (EEG) results indexed rapid L2 word learning. Interestingly, even-related potential (ERP) results revealed a gender congruency effect such that only words that had the same grammatical gender across participants’ L1 and L2 gave rise to an N400 effect for match vs mismatch auditory word and image pairs, indicating that these words were better encoded. In a second study, we used an action-sentence compatibility effect (ACE) paradigm to evaluate how motor preparation affects language processing. ERP results showed greater N400 amplitude for congruent compared to incongruent action-sentence trials, suggesting that compatibility between motor and language processes produced interference. In studies 3 and 4, we combined virtual reality (VR) and EEG to investigate interactions between language processing and motor activation. In the first of these studies, participants heard action verbs in their native language and performed varied actions on a virtual object in a Cave automatic virtual environment (CAVE) during a Go-Nogo task. Time frequency analysis showed motor activation for both Go and Nogo conditions during action verb processing and prior to movement proper. In addition, greater motor activation for Go versus Nogo trials. Our final (projected) study is a registered report that aims to determine the neural correlates of embodied L2 learning by having participants encode auditory action verbs using an interactive virtual reality head-mounted display system and specific real-life actions on a virtual object. Using behavioral and EEG measures in a pre-post training design, this condition will be compared to a control condition in which participants will simply point to the virtual object.
The body of the work reported in this dissertation represents a significant step towards better understanding the subtle relationship between motor and semantic processes. By making use of new technologies that allow for manipulating and controlling the environment, our work opens up fresh perspectives for taking into account the contextual nature of how we learn and understand language.
Dysrhythmia in Parkinson's Disease: The serious game as a remedy for coordination disorders
Under the supervision of Simone Dalla Bella and Serge Pinto