Research projects
Boosting children's narrative and pragmatic abilities
As part of my PhD thesis, we have designed the MultiModal Narrative (MMN) intervention program, which is a multi-tiered intervention that can be applied either at a classroom level (i.e., universal support) or at an individual level with children with linguistic and communicative difficulties (i.e., intensive support). The MMN program trains narrative macrostructure (i.e., structural elements of the narrative) as well as pragmatic and perspective-taking abilities (i.e., understanding characters’ perspectives and emotions) with the aid of multimodal strategies (i.e., embodying the different actions and emotions of the stories).
In collaboration with:
Dr. Alfonso Igualada (UOC)
Dr. Pilar Prieto (ICREA/UPF)
Project supported by the Education Department (Generalitat de Catalunya)
Related research:
Florit-Pons, J. (in prep.). Multimodality as a tool for boosting narrative and pragmatic abilities in typical and non-typical language development [PhD thesis]. Universitat Pompeu Fabra. Directed by Pilar Prieto (ICREA-UPF) and Alfonso Igualada (UOC).
Florit-Pons, J., Prieto, P., & Igualada, A. (2022, July 13-15). Can multimodality boost narrative and pragmatic abilities of preschool ASD and DLD children? Evidences from a preliminary study. Poster presentation at the 9th Conference of the International Society for Gesture Studies (ISGS). University of Chicago, Chicago, IL, USA.
The StoryTIC project
As part of this project, we are currently working on transforming the MMN intervention into a digital format, which will also incorporate activities with Augmented Reality.
In collaboration with:
Dr. Florence Baills (UdL)
Dr. Ïo Valls (SDU)
Dr. Alfonso Igualada (UOC)
Dr. Pilar Prieto (ICREA/UPF)
The relationship between multimodal skills (versus motor skills) and narrative and pragmatic skills
This project has two main aims: a) to determine whether multimodal skills can predict narrative and pragmatic skills in development by comparing them to the relationship with motor skills; and b) to compare between typically-developing (TD) children and children with neurodevelopmental disorders affecting language and communication (NDD).
In collaboration with:
Dr. Mariia Pronina (UIB)
Dr. Alfonso Igualada (UOC)
Dr. Pilar Prieto (ICREA/UPF)
Dr. Courtenay Norbury (UCL)
Related research
Florit-Pons, J., Pronina, M., Igualada, A., Prieto, P., & Norbury, C. F. (2023, October 25-27). Multimodal skills, but not motor skills, predict narrative and pragmatic skills in typically and non-typically developing preschoolers. Poster presentation at the Many Paths to Language Conference. Max Planck Institute for Psycholinguistics: Nijmegen, The Netherlands.
Improving Catalan use in preschoolers
The goal of this project is to determine whether the MultiModal Narrative (MMN) intervention can also help boost the use of Catalan language and improve Catalan phonology of Spanish-dominant preschoolers.
In collaboration with:
Joel Espejo-Álvarez (UPF)
Dr. Alfonso Igualada (UOC)
Dr. Pilar Prieto (ICREA/UPF)
Previous collaborators: Mireia Gómez (University College Cork), Claire Luong (UPF), Alba Buendía (UPF), Dr. Josefina Carrera-Sabaté (UB)
Related research:
Florit-Pons, J., Gómez, M., Buendía, A., Carrera-Sabaté, J., Igualada, A., & Prieto, P. (2022, July 7). L'efectivitat del programa d'intervenció Narrativa MultiModal (NMM): Resultats de tres estudis realitzats a educació infantil. Oral presentation at XIIIè Workshop sobre la prosòdia del català. Universitat de Barcelona: Barcelona, Spain.
The development of gesture, speech and information structure
This project is concerned about children's multimodal development, specifically on the temporal relationship between gesture and prosodic prominence and on how children use gesture to mark information structure in discourse.
In collaboration with:
Dr. Ingrid Vilà-Giménez (UdG/UPF)
Dr. Patrick Louis Rohrer (UPF/UN)
Sara Muñoz-Coego (UPF)
Dr. Pilar Prieto (ICREA/UPF)
Main findings:
The use of non-referential gestures increases significantly from ages 5-6 to 7-9.
By age five, children have acquired the ability to temporally integrate gesture and speech in narrative discourse.
Children use non-referential gestures to mark information that updates speaker's and listener's common ground (i.e., new referents, topic and comment constituents).
Related research:
Florit-Pons, J., Vilà-Giménez, I., Rohrer, P. L. & Prieto, P. (2023). Multimodal development in children’s narrative speech: Evidence for tight gesture-speech temporal alignment patterns as early as 5 years old. Journal of Speech, Language and Hearing Research, 66(3), 888-900. doi: 10.1044/2022_JSLHR-22-00451
Rohrer*, P. L., Florit-Pons*, J., Vilà-Giménez, I., & Prieto, P. (2022). Children use non-referential gestures in narrative speech to mark discourse elements which update common ground. Frontiers in Psychology, 12: 661339. doi: 10.3389/fpsyg.2021.661339
* Shared first authorship
Muñoz-Coego, S., Florit-Pons, J., Rohrer, P. L., Vilà-Giménez, I., & Prieto, P. (2022). The prosodic and gestural marking of information status in children's narrative speech: A longitudinal study. Proceedings of the 11th International Conference on Speech Prosody. Lisbon, Portugal.
Videos related to this project
20-minute video presented at XIIè Workshop sobre la prosòdia del català
10-minute video presented at Gesture and Speech in Interaction (GESPIN)
5-minute video presented at Budapest CEU Conference on Cognitive Development (BCCCD21)
MultiModal MultiDimensional (M3D) labeling scheme
The team has developed the MultiModal MultiDimensional (M3D) labeling scheme for the annotation of multimodal corpora, which offers a dimensionalized approach to the annotation of communicative body movements (i.e., manual gestures, head movements, and other articulators) in terms of their form, their semantic and pragmatic contributions to speech, and their relationship to speech prosody.
In collaboration with:
Dr. Patrick Louis Rohrer (UPF/UN)
Dr. Ingrid Vilà-Giménez (UdG/UPF)
Dr. Núria Esteve-Gibert (UOC)
Ada Ren (MIT)
Dr. Stefanie Shattuck-Hufnagel (MIT)
Dr. Pilar Prieto (ICREA/UPF)
Check the M3D training materials here! https://m3d.upf.edu/
Related research:
Rohrer, P. L., Vilà-Giménez, I., Florit-Pons, J., Gibert, N. E., Ren, P., Shattuck-Hufnagel, S., & Prieto, P. (2021). The MultiModal MultiDimensional (M3D) labeling system. https://doi.org/10.17605/OSF.IO/ANKDX. Accessible at OSF
Rohrer, P. L., Vilà-Giménez, I., Florit-Pons, J., Esteve-Gibert, N., Ren, A., Shattuck-Hufnagel, S., and Prieto, P. (2020, September 7-9). The MultiModal MultiDimensional (M3D) labelling scheme for the annotation of audiovisual corpora. Oral presentation at Gesture and Speech in Interaction (GESPIN). KTH Speech, Music & Hearing and Språkbanken Tal: Stockholm, Sweden.
Videos related to this project
20-minute video presented at Gesture and Speech in Interaction (GESPIN)
The predictive role of pragmatic skills on language development
This longitudinal project's aim is to analyze whether early expressive pragmatic skills at 3-4 years of age can be predictive of later linguistic skills at 5-6.
In collaboration with:
Dr. Núria Esteve-Gibert (UOC)
Dr. Nadia Ahufinger (UOC)
Dr. Mariia Pronina (UPF)
Dr. Pilar Prieto (ICREA/UPF)
Related research:
Esteve-Gibert, N., Ahufinger, N., Pronina, M., Florit-Pons, J., & Prieto, P. (2023, July 9-14). Children’s linguistic abilities at 5-6 years of age can be predicted by expressive pragmatic skills at 3-4 years of age. Oral presentation at the 18th International Pragmatics Conference. Université Libre de Bruxelles: Brussels, Belgium.