Multimodalidade sensorial como auxílio ou obstáculo à compreensão linguística.

Abstract

Reading comprehension, the ability that combines general language comprehension with the visual recognition and decoding of words (Hoover; Gough, 1990; Scarborough, 2001), is a fundamental skill for living fully in our literate society. Despite this status, reading activities, which demand a large amount of cognitive processing, are constantly carried out in suboptimal scenarios, almost always competing against the myriad of distractions of the contemporary world. Attention is the gateway that defines which stimuli – among them, reading – receive privileged cognitive treatment and can reach consciousness and also memory. When it comes to linguistic, or more broadly, semiotic materials, these stimuli are usually mixed in multiple simultaneous sensory modalities. With this in mind, this study aims to analyze possible interferences, ranging from providing assistance to generating obstacles, caused by the association, in a single text, of visual and auditory sensory modalities over the language comprehension of college students. In order to conduct an investigation on an undergraduate population (n = 25), a test was designed consisting of nine short narratives (with approximately 150 words each), presented in nine different formats (written, spoken, written and spoken, in subtitled video, in narrated video, sung, with instrumental background music, with sound distractions, and with visual distractions) to measure language comprehension as a process, applying eye tracking, and as a product, using questionnaires with short open-ended questions. Additionally, a profile questionnaire was applied, aimed at understanding the participants' reading and media consumption habits, and a perception questionnaire about perceived preferences during the reading comprehension test. The reaction time metrics for answering the comprehension questionnaires and the accuracy scores did not show any statistically significant differences among the experimental conditions; however, the latter measure revealed a tendency for silent reading to be more efficient than the spoken, sung, and instrumental music-accompanied text formats. Eye tracking data showed leveled effects across all experimental conditions based on static text, masking possible variations in working memory processing in this segment. Differences were observed in audio and video formats, particularly larger pupil dilation, whose interpretation is not conclusive. Considering all data, including participant preferences, the unimodal visual text format performed best and most consistently. In general, the results indicate that undergraduate students process language comprehension with great flexibility in dealing with multimodality and distractions. The number of sensory modes occupied by the communicated information seems to have less impact than parameters such as conciseness, structuring, and native effectiveness of the applied encoding. Future studies with more informationally dense texts, instruments calibrated to measure more complex reading comprehension processes, or considering audiences with less proficient reading and less adaptability to distractions will likely present different conclusions that will help put the findings of this research into perspective.