Vous êtes sur la nouvelle plateforme d’Érudit. Bonne visite! Retour à l’ancien site

Captioning and Subtitling: Undervalued Language Learning Strategies

  • Martine Danan

…plus d’informations

Corps de l’article

In countries where viewers rarely watch subtitled programs, for example in the United States, language students often experience feelings of guilt or annoyance when first exposed to subtitles, while language teachers themselves tend to be openly hostile to their use. In such cases, subtitles are most commonly viewed as distracting: They are accused of encouraging viewers to rely on the written text, taking attention away from the actual spoken language, and even fostering a form of laziness bordering on cheating. Yet, many Europeans claim to have learned English from their regular exposure to subtitled American films and television programs. So how much of this learning can be attributed to the help of subtitles, and should greater effort be made to encourage the use of subtitled audiovisual material to foster second – or foreign-language acquisition in and outside the classroom?

A partial answer to this question can be found in a number of recent empirical studies examining ways in which standard subtitling (i.e. interlingual, translation subtitles in the native language), but also other forms of subtitling, can actually improve the effectiveness of audiovisual presentations and develop viewers’ language skills. The bulk of the research has focused on the use of captions (called teletext subtitles in Europe, with subtitles in the same language as the sound track), which are primarily used in the context of the hard-of-hearing population. We will first review the findings of the most significant experiments dealing with the relationship between second-language acquisition and captions (also labeled bimodal, same-language, unilingual, or intralingual subtitles in scholarly literature), especially in relation to the processing of the spoken language. But we will also point to the limitations of captions in some instances and argue that standard subtitling can provide language learners with additional valuable assistance, as in the case of the incidental language learning occurring in Europe with spectators of American films. However, to derive the greatest benefits of both captions and subtitles, we will finally stress the importance of teaching students how to consciously adopt effective learning strategies, which ultimately play a fundamental role in improved listening skills and successful language acquisition.

1. Benefits and Limitations of Audiovisual Material

It is now commonplace to say that audiovisual material, with its rich context, is a powerful instructional tool known to have a motivational, attentional, and affective impact on viewers, which in turn facilitates auditory processing (Baltova, 1994: 510-1). In addition, film, television, video, and now digitized images usually expose students to larger amounts of authentic oral language input, which in the long run should improve listening comprehension in face-to-face interaction with native speakers (Herron et. al., 1995).

Indeed, audiovisual media are closer to real life because visual clues and context make it possible to “view” the message as much as listen to it (Baltova, 1994: 508). As a number of researchers have shown, listening comprehension is an “active cognitive process” involving “speculating and predicting” rather than individual sound deciphering because incomplete acoustic input often necessitates filling in missing information (Noblitt, 1995). Comprehension is also influenced by visual information and cultural knowledge (for example interpreting facial expressions), as lip-reading research has demonstrated (Baltova, 1994: 509). The positive effect of visual clues was confirmed by an experiment involving 53 intermediate-level Grade 8 Canadian pupils in a core French program. Those who watched a 15-minute clip in the video-and-sound condition obtained scores almost twice as good as in the sound-only condition (Baltova, 1994: 511, 513).

However, beyond global understanding, visual clues do not necessarily assist with the comprehension of the actual spoken text, as this experiment also demonstrated. In fact, students’ performance in the video-and-sound condition was very similar to results in the silent viewing condition (Baltova, 1994: 513). In order to analyze actual listening comprehension gains, Baltova set up a second experiment comparing the same video-and-sound condition with the sound-only treatment but focusing on a more text-dependent comprehension test. This particular test revealed no significant difference between the two groups and confirmed the poor level of distinct textual understanding (Baltova, 1994: 516). These results point to the limitations of video alone and the need to find techniques, such as captioning and subtitling, to improve the pedagogical effectiveness of the medium.

2. Listening Comprehension Benefits Through Captioning

Various studies have demonstrated the positive effects of captioning on productive skills such as verbatim recall and retention, reuse of vocabulary in the proper context, as well as communicative performance in specific oral and written communication tasks (Vanderplank, 1988: 276; Baltova, 1999: 38; Garza, 1991: 245; Borras & Lafayette, 1994: 63, 65, 68; Neuman & Koskinen, 1992: 102). However, since the main objection to the pedagogical use of subtitles stems from the perception that they hinder the development of receptive skills, the main goal of this article is to examine whether captions and subtitles can also improve listening comprehension.

General ideas can often be made understandable through images alone or even advanced organizers such as oral and written summaries or video clips, but captions have proven to be more beneficial for the comprehension of details pertaining to characters and plot (Chiquito, 1995: 219; Chung, 1999: 300-1). In terms of comprehension, captions can in fact assist students at different levels of linguistic ability. Markham had 76 advanced, intermediate, and beginning ESL students in an American university watch two 2 H – and 4 H-minute-long educational television programs. The results of the multiple-choice comprehension tests based on the vocabulary and syntax of the captions showed that within each level, responses were more accurate when captions had been available. Thus captions helped students perform beyond their proficiency level (Markham, 1989: 39, 41).

In addition to comprehension, captions can help with word recognition and vocabulary building. Neuman and Koskinen conducted a nine-week experiment with 129 seventh and eighth grade ESL students (mostly at an advanced level) watching nine 5- to 8-minute long segments of an American children-oriented science production. The researchers found that captioning was more beneficial to vocabulary recognition and acquisition than traditional television watching, or reading while listening. A series of increasingly complex tests demonstrated the beneficial effects of captions. These tests ranged from weekly word recognition exercises which entailed distinguishing written target words from nonword distractors, to sentence anomaly exercises testing word comprehension in context, and on the most difficult level, meaning identification of words presented in isolation (Neuman & Koskinen, 1992: 101).

Even audio text strongly supported by images that undoubtedly clarify and contextualize it can become more comprehensible with captions. Garza conducted an experiment comparing the comprehension ability of 70 high-intermediate/low-advanced ESL learners as well as 40 third/fourth year American university students of Russian, who viewed five discursive types of 2- to 4-minute-long videos with and without captions. Subjects were tested through multiple-choice content-based questionnaires requiring the identification of target-language “informational paraphrases, basic deductions, or synonym identification” of a term made visually explicit in the video segment. Results revealed significantly improved performance for the captioned condition in both language groups. Thus, captions may help make the audio input more intelligible by bridging the gap between reading comprehension skills, which are usually more developed, and listening comprehension (Garza, 1991: 241-243, 246).

Skeptics may still contend that even if captioning allows for language gains and improved comprehension, students are not being truly trained to develop their listening skills without written support. To respond to these objections, we will now turn to a few other studies which have attempted to measure pure listening comprehension gains, tested independently of any written components.

To examine the effect of captioning on aural word recognition skills, Markham designed another experiment involving multiple-choice tests administered orally. 118 advanced ESL students watched two short video programs (12 and 13 minutes) with or without captions. In the subsequent listening tests, subjects heard sentences directly taken from the script and immediately followed by four single words (one key word which belonged to a sentence just heard and three distractors) presented orally too. The tests showed that the availability of subtitles during the screening significantly improved the students’ ability to identify the key words when they subsequently heard them again (Markham, 1999: 323-4).

Some may still argue that the improved listening comprehension resulting from the specific context of a captioned audiovisual program does not necessarily prove students’ ability to better comprehend new scenes without captions. To test how subtitling affected listening ability regardless of semantic information, so as to assess recognition memory in relation to sound alone, Bird and Williams focused on the implicit and explicit learning of spoken words and nonwords. Implicit learning pertained to auditory word recognition, while explicit learning referred to the intentional recollection and conscious retention of aural stimuli. A first experiment with 16 English native and 16 advanced nonnative speakers demonstrated that subjects in the captioned condition were better able to implicitly retain the phonological information they had just processed. They also showed superior explicit recognition memory when asked to aurally identify words that had been presented in a previous phase. A second experiment with 24 advanced ESL students found that captioning had a beneficial effect on word recognition and implicit learning of nonword targets paired with two rhyming and two nonrhyming aural cues, especially in the rhyme condition. Thus, captioning clearly aids with the phonological visualization of aural cues in the minds of listeners, who become more certain of ambiguous input, can more accurately form a memory trace of the words, and can later more easily identify identical sounds without textual support (Bird & Williams, 2002).

In short, captioning seems to improve the actual language processing ability of second-language learners, who could be described as “hard of listening” (Vanderplank, 1988: 272). The benefits of this “hearing aid” become even clearer over time, as shown by longer-term experiments. One study over an 11-week period involved grades 5 and 6 Canadian pupils who had attended a French immersion program since kindergarten. Tests measuring phrase form, contextual meaning, and comprehension demonstrated that captioning increased in effectiveness through time (Lambert & Holobow, 1984: 3). A long-term protocol analysis in another study may explain this improved effectiveness. 15 European ESL students ranging from high-intermediate to superior, and 8 Arabic ESL students from low-intermediate to advanced, were asked to watch captioned programs one hour a week over a nine-week period while reflecting on the experience. After a few hours of practice with the captions, the subjects felt that they were able to process longer stretches of verbal and written texts, i.e. their “chunking ability” had improved (Vanderplank, 1988: 274-5). Thanks to improved processing, subjects also remarked on their ability to “find,” note, ask about phrases totally new to them, and extract terminology (Vanderplank, 1988: 275; Vanderplank, 1990: 225).

3. Limitations of Captioning and Need for Comprehensible Input

In spite of the beneficial aspects described above, captioning may not be suitable for all materials and viewers at all levels of language proficiency. In particular, it may be helpful to beginners only if the material is carefully adapted to their level and contains many familiar phrases that can be activated and reinforced by the audiovisual presentation. This point was illustrated by an experiment in which full text or keyword captioning was added to the authentic language video clips designed to accompany a French beginning textbook. Guillory, who tested 202 American subjects in second semester French classes, found that even with captions the comprehension mean scores for two video clips from this program were at best 72%. If the material is too advanced, as in this case with nearly 28% of the words used in the video not listed in the textbook glossary, captions cannot sufficiently compensate for the fast rate of speech and the difficult vocabulary level (Guillory, 1998: 95, 102).

Thus, even with captions, visual input which is too far beyond the linguistic competency of the viewers may yield poor language gains. For example, in an experiment discussed earlier, Neuman and Koskinen found that in spite of the additional contextual support provided by the video, students with limited linguistic abilities learned less from the captions than viewers at the mastery level (Neuman & Koskinen, 1992: 103-4). Another experiment involving “relative beginners” (grade 7 Canadian pupils who had had 45 minutes a day of French-as-a-second language instruction since grade 1) indicated that captioning offered limited benefits compared to forms of subtitling incorporating native language input (Lambert & Holobow, 1984: 6). These experiments may suggest a minimum language competency threshold for learners to derive benefits from captioning, or the need for careful matching of competency level and linguistic difficulty of the audiovisual material.

The findings of these studies support Krashen’s hypothesis that we acquire language “by understanding messages or by receiving ‘comprehensible input’” (Krashen, 1985: 2). The concept of comprehensible input is linked to recent cognitive theories which stress the role of active individual learners in the effective acquisition, organization, and storage of knowledge into a meaningful cognitive network (Hummel, 1995: 448). But Krashen also warns that to be comprehensible, input can only be slightly above the students’ level of proficiency, and he advocates a stage by stage approach to the introduction of comprehensible input by language teachers (through, for example, the choice of somewhat simplified language or limited topics) (Krashen, 1985: 72-73).

Therefore, even with captions, massive exposure to authentic audiovisual material which has not been carefully enough selected or made accessible to non-native viewers can be a very inefficient pedagogical approach. In the subsequent section of this article, we will argue that interlingual subtitling may be another way of making incomprehensible visual input comprehensible.

4. Advantages of Interlingual Subtitles

The functional-notional and communicative approaches most widely used today in language teaching fit into the comprehensible input model since they encourage active understanding and help students organize new information into meaningful units (Hummel, 1995: 448-9). Current methodology, however, tends to discourage any use of the first language, although recent research in language and memory processing shows that selective recourse to translation can also lead to cognitive benefits. In particular, the effort of establishing paired equivalents through translation, and the use of a redundant and interconnected network of encoded propositions, lead to greater elaborateness of processing, which usually results in better recognition and recall (Hummel, 1995: 449, 450, 453).

Even greater depth of processing and interconnectedness occur in the case of subtitled audiovisual material, when both audio and visual channels have to be processed simultaneously (hence the feeling of difficulty experienced by some viewers). According to Paivio’s bilingual dual coding theory, the verbal system and the imagery system, comprised of nonverbal objects and events, are functionally independent but linked by referential connections. In addition, bilingual situations rely on two separate verbal systems related by associative connections. In the case of subtitled visual input, these three independent systems are interconnected through triple associations between image, sound in one language, and text in another, which may lead to better processing and recall because of the additive effects of both image and translation. Once translation has linked the two verbal systems, viewers have established more paths for retrieval and may benefit from visual traces as well as from two distinct sets of verbal traces (Paivio, 1986: chapter 11; Danan, 1992: 504, 522).

Detractors of interlingual subtitles in the context of second-language acquisition may still argue that if students can simply read their native language, they will stop listening to the foreign dialogue. Indeed, the reading of subtitles tends to be an automatic behavior, as demonstrated by a series of cognitive experiments measuring eye movement patterns. These experiments showed that subtitles were consistently read, whether the sound was on or off, regardless of the subjects’ knowledge of the spoken language or their familiarity with subtitling (d’Ydewalle & Gielen, 1992: 416-7; d’Ydewalle & Pavakanum, 1992: 193-4).

Automatic reading of subtitles, however, does not prevent the processing of the soundtrack. To demonstrate this point, another group of cognitive experiments relied on a double task technique measuring reaction times to a flashing light during a television program. The slower reactions in the presence of both sound and subtitles suggested that more complex, simultaneous processing of the sound track and the subtitles was occurring (d’Ydewalle & Pavakanum, 1997: 146-7). Another experiment confirmed that subjects processed the sound when available since slightly more time was devoted to the subtitles in the absence of sound (d’Ydewalle & Gielen, 1992: 417-8). With both subtitles and sound, attention seemed in fact to be divided between the two according to the viewers’ needs, with more time usually devoted to subtitles for the processing of complex information (d’Ydewalle & Gielen, 1992: 425).

Anecdotal reports of incongruities and discrepancies between the spoken and written texts support the notion that both channels can be processed simultaneously. But to go beyond anecdotal evidence of attention to both sources of input, a group of researchers set up a news program with subtitles occasionally deviating from speech on the phonological, grammatical, lexical, or informational levels (De Bot et. al., 1986: 74-75). Two groups of subjects, 50 secondary school pupils learning English at school, and 20 advanced university students who were no longer studying English, watched this program in English with subtitles in their native Dutch language. The subjects responded to a multiple-choice test about each news item, with questions equally divided between deviations and non-deviations. The results disproved the notion of exclusive subtitle orientation and showed that all viewers made use of the audio input, although this particular experiment was unable to quantify the extent of learning directly resulting from the spoken text (De Bot et. al., 1986: 78, 80).

Language acquisition, with the help of interlingual subtitling, is most frequently incidental since many viewers may learn the language heard in subtitled programs without a conscious or systematic effort to do so (van de Poel & d’Ydewalle, 1999). This may be particularly true of children in countries where subtitling is the norm. For example, Dutch children devote half of their television viewing time on average to subtitled programs (Koostra & Beentjes, 1999: 59). In Belgium also, many children can speak and understand some English even before they start learning English at school, presumably because of their frequent exposure to English-language subtitled television programs (d’Ydewalle & Pavakanum, 1997: 146). As for adults, they often view subtitling as a perk allowing them to learn or maintain their knowledge of a foreign language, especially English, thanks to preference for subtitled programs in many countries. For example, a 1977 survey conducted by the Dutch Broadcasting Service (NOS) revealed that 70% of their spectators favored subtitling, most often because it allowed them to increase their language proficiency (De Bot et. al., 1986: 74).

Assessing how much English these viewers would learn if they watched the same programs without any subtitles is of course complex, considering the repeated and extensive presence of English-language programs. However, a few short-term experiments have attempted to measure whether actual incidental learning takes place from watching subtitled television. One study by Koostra and Beentjes focused on 246 Dutch children in grade 4 (before any formal instruction in English) and grade 6 (following one year of English at school), after they watched a 15-minute American documentary shown twice with or without subtitles. This study demonstrated that children acquired more English vocabulary from watching subtitled television, although even children in the condition without subtitles learned some new words. The subjects had been asked to match target test items, played aurally, with four possible Dutch written translations, all chosen among words that had appeared in the subtitles. Children in the subtitled condition also performed significantly better on a word recognition test, consisting of words heard in the soundtrack and words that could have been used in the context of the particular program. Thus, far from distracting the viewers, the written input helped them recognize English words in the audio channel; as a result, they were better identified aurally later. Older children performed better than the younger ones without any formal training in English, but high frequency of subtitled television viewing at home proved to be a more significant factor on performance results (Koostra & Beentjes, 1999: 56-8).

These last findings may point to the fact that a key to the beneficial use of subtitled programs for language learning may in part be familiarity with them. Lack of familiarity may lead to a sense of distraction and poor language gains because the foreign language can be bypassed for comprehension, as noted in earlier experiments (Lambert & Holobow, 1984: 9; Danan, 1992: 512). But over time, as with captioning, viewers may more or less intentionally develop strategies to process subtitles efficiently and derive the most benefits from them.

5. Successful Learner Strategies

Recent research on second-language acquisition is also stressing the significance of successful learner strategies, which can apply to the optimal use of both captions and subtitles. A good learner is now perceived as an “active processor” of information,” while learning is described as “a process of ‘successive approximation,’” which involves making and testing hypotheses about meaning. Hence, guessing and inferencing, clarification (through questions), and verification of meaning are essential cognitive strategies in this process (Rubin, 1995: 15, 16).

In terms of listening skills, more specifically, Thompson and Rubin have shown that listening comprehension improved significantly for students who were systematically taught cognitive and metacognitive strategies, compared to learners who were simply provided opportunities for listening (Thompson & Rubin, 1996: 337). The rich audiovisual context allows for an even greater range of strategies. Cognitive strategies during a screening include prediction based on visual clues or background knowledge; reliance on known elements such as familiar words and cognates; and “resourcing,” i.e. jotting down unknown words and looking them up (Rubin, 1995: 12; Thompson & Rubin, 1996: 335). Metacognitive strategies are “management techniques” to control and reflect on the learning process. In the case of audiovisual material, such strategies include “planning,” i.e. how to watch a section (how many replays and at what pace, how to use the sound, etc.); “defining goals,” i.e. deciding on what to listen for and how much is needed for basic comprehension; “monitoring,” i.e. self-assessing comprehension, identifying difficulties, judging strategy effectiveness, and choosing strategies in a flexible manner (Thompson & Rubin, 1996: 331, 335).

A few empirical studies have shown that captioning allows for more active viewing and better learner strategies, especially metacognitive ones. In Vanderplank’s nine-week-long protocol analysis mentioned above, subjects reported that after an initial sense of disturbance, they had developed strategies to derive the greatest benefits from the captions, for example by switching back and forth between the sound and the subtitles, or learning to process all three channels simultaneously without feeling overburdened (Vanderplank, 1988: 275). Vanderplank’s follow-up study over a three-month period involved a core group of 15 European and Arabic students with intermediate to superior English skills. The subjects noted that they had developed additional flexible strategies, as they only turned to captions when they needed the support or wanted to test their listening ability (Vanderplank, 1990: 224-5).

Current pedagogical theory is also stressing the significant role of affective factors in learning, and captioning has been proven to have a positive effect in this regard. Krashen, in particular, has theorized the beneficial effect of a low “affective filter” (for example, due to the learners’ low-anxiety environment or involvement in the message) in allowing input “in” instead of blocking it (Krashen, 1985: 3-4). An empirical study conducted by Borras and Lafayette demonstrated that students working with captions had a much more positive attitude than those who did not have access to them (Borras & Lafayette, 1994: 68). Captions relieve some of the anxiety experienced by students who sense they have missed an important element and sometimes give up watching because they feel lost. Instead, captioning allows learners to relax, grow more confident in their ability to understand, and direct their mental energy to actual learning. In addition, captions provide instant feedback that reinforces the positive learning experience (Vanderplank, 1988: 277). They also seem to be clearly motivational since in the experiment set up by Borras and Lafayette (1994: 67), students using captions spent more time working on higher-level tasks than all the other groups.

The ability to use the written support strategically and the positive learning environment discussed in the context of captioning research can certainly apply to subtitling. Thus, these factors may be the key to the high degree of incidental learning that takes place among motivated, habitual viewers of subtitled programs, even if adequate strategies are used subconsciously. In fact, in one of his empirical studies Vanderplank noted that subjects who came from countries where regular subtitling is the norm were most adept at quickly developing strategies to take advantage of captioning (Vanderplank, 1988: 276).

But since many students are inexperienced users of subtitles or captions, it is crucial that intentional, effective use of strategies be taught. Teachers must introduce them systematically as they encourage reflective attention to both oral and written text instead of simply urging pupils to listen attentively (Vanderplank, 1990: 226, 228-9; Vanderplank, 1994: 125, 128; Vanderplank, 1999: 266). Moreover, educators can expand on the range of strategies made available to learners by selecting appropriate tasks and material enhanced with either captions or subtitles. For example, captioning may be best with easier to understand audio input supplementing in-class instruction or for advanced students. Subtitling, on the other hand, may be most useful for less skilled learners needing first language input or with complex films and news broadcasts assigned for independent, out-of-class viewing in relation to more holistic goals.

In some cases a multi-step or need-based approach incorporating both captions and subtitles may also be beneficial. Such flexibility may become more readily available to learners with the growth of multimedia programs (and to some extent DVDs), which allow users to select different modes of audio and text presentations, use subtitling selectively, interact with on-screen information in a self-paced fashion, and benefit from instant feedback as in face-to-face exchanges (Brett, 1997: 49; Brett, 1998). Not surprisingly, much of the current research on captioning has originated in a multimedia environment, which can give learners the opportunity to deliberately select their preferred tasks and strategies. Mastery of these strategies, which simulate real-life communication, may be taught in schools, but they can also ultimately lead to independent, life-long language acquisition.

6. A Call for Long-Term Research

All over the world access to vast amounts of culturally-rich, enjoyable material is bound to increase with the expansion of satellite television and the Internet, as well as the development of multimedia and DVDs. However, immersion in a flow of foreign utterances, especially if they are far above the students’ listening comprehension level, may do little to improve the viewers’ language skills. Without the availability of captions and subtitles, the complexity and transient nature of the audiovisual input often make it difficult for language learners to truly comprehend foreign utterances. Captions or subtitles can lead to significant improvement in listening skills as long as viewers learn to take advantage of relevant strategies. Although empirical research has demonstrated the beneficial effect of captions and, to a lesser extent, standard subtitles, more data still need to be systematically collected on their long-term role and ways to successfully implement strategies adapted to the linguistic difficulty of the material and the learning environment. It is hoped that current interest in multimedia will lead to the development of language curricula and self-learning programs integrating captions and subtitles while encouraging in-depth pedagogical research on their most effective use.

Parties annexes