Corps de l’article

Introduction

The interdependence of the different modes of expression in multimodal genres (which, arguably, account for the whole range of textual genres down to the printed page, Baldry 2000: 41-43) has already been highlighted in translation studies and related disciplines. For instance, the semiotician Van Leeuwen (1999) has pointed out the artificiality of the division of speech from other acoustic means of expression, such as music and other sounds which are meaningful in a given context (such as a doorbell to announce somebody or the sound of footsteps in a radio play or film to describe or highlight the action of walking). It is the visual, however, that seems to channel attention towards multimodality. According to Lepilová, the physical dimension is inherent in language: any lexical or syntactical item (a word, phrase or sentence) remains “an abstract phenomenon as long as it is isolated from a concretising feature,” which can be visual (i.e., the written word, or the sign in sign language) or aural (i.e., the spoken word). Even an aural concretisation, however, “necessarily brings to mind a picture” that visualises the content of the verbal item. This is inevitable in today’s communication, which is “under the influence of the mass media, multimedia and the combination of text and picture,” even more so in the case of the younger generations, whose mentality and language “has been taken over by the TV screen and its action movies and computer games, its video clips and advertisements” (Lepilová 2002: 157-158).

Multimodality, however, is not necessarily a sign of our times, but can be seen as an inherent trait of human communication. As Kress (1997) points out, children’s first approach to the bidimensionality and abstracting potential of the page is through drawing, which should not be seen as a mere means of expression, but also of communication and meaning-making. In children’s literature, especially that for children below school age, the pictures thus become an important interpretive tool (Oittinen 2000, 2003).

This paper will take the role of the visual in the translation of print advertisements and other printed texts as its principal focus. But these texts will be used as examples of the application of theories of social semiotics to all types of textual transfer across languages, cultures and/or communities. According to such theories, communication is achieved not only, and not even primarily, through verbal language, but also through other sign systems. The visual mode of expression, including (among others) graphics, pictures and photographic images, is one such system. In the next section, I will attempt to explore the reasons why the incorporation of such systems into practices – and theories – of translation still encounters strong resistance. In section 2, I will examine the issue of intersemiotic translation as applied to print advertisements, while in section 3, I will try to provide practical examples from other fields showing that a social semiotic approach to translation is not only possible and at times necessary, but also simpler and much more natural than it sounds, also suggesting some operational implications of a semiotic approach to multimodal translation practice.

1. A few common perceptions about the untranslatability of images

1.1. Translation training and professional practice

In its non-metaphorical usages, the term “translation” is usually defined as a verbal-only practice, as the act of changing “sth spoken or esp written into another language” or, focusing on the product of such an act, “a text or word that is translated” – where the association of “text” with “word” makes it difficult to expand the meaning of “translation” to cover non-verbal modes of expression (Crowther et al. 1995: 1270 s.v. “translate” and “translation”; see also Hawkins and Allen 1991: 1534 s.v. “translation”; Devoto and Oli 1990: 2004-2005 s.v. “tradurre” and “traduzione”). This common understanding is mirrored by the kind of training that is usually provided by translation schools, which – to the best of my knowledge – tends to focus on the verbal dimensions of the text that are relevant for the science of linguistics, treating as incidental, if at all, any non-verbal elements that may nonetheless contribute to the construction of the meaning of the text as would be understood in a broader semiotic perspective. Exceptions to “verbocentric” didactic practice can be seen in training in highly multimodal fields where non-verbal, non-linguistic dimensions are prominent (e.g., multimedia translation – see for instance Neves 2004: 131-133). This kind of translation, however, is seldom included in mainstream translation courses, but is usually treated in separate, highly specialised ones.

This verbal-only approach to translation training is not unmotivated – rather, one must admit that it is backed by very practical and sensible reasons. First of all, it provides teachers and students with a clear focus, reducing at the same time the textual variables that would require separate, or special, treatment. In other words, defining text genres by linguistic means alone, focusing on the verbal dimension, makes things much easier. For instance, in a module devoted to short stories, analysing the illustrations that went with one or more editions of the source text, and reflecting on how the target text might be illustrated, would take up precious time. Moreover, somebody might argue that it would “distract” the students from the primary skills they will be required to use on the translation market, which leads us to a second very reasonable argument in favour of the verbal-only approach to translator training: the division of labour.

In real professional life, the translation of the verbal content is just one of the several compartments of the production of the final target text, just as an assembly line is made up of workers who specialise in one aspect of production, but never get a general overview of the process as a whole. Those who deal with words do not usually know, or work in close cooperation with, those who are in charge of the visual aspects of the text, with the result that when the two realms overlap, conflict is likely to arise. For instance, graphic designers may omit a final paragraph without consulting the translator, because “it didn’t fit into the space” of the pdf file; or translators may dismiss as irrelevant the graphic designer’s warning to keep the target text of the same length as the source text, or else “it won’t fit into the space.” (I am afraid that both situations have actually taken place in my professional experience.) Thus, in their training years translators learn that it is natural for them to be left out of comprehensive text-making processes that take into account all the modes in which the text is encoded; and in professional life, they consciously or unconsciously perpetuate this status quo, until they find a market for – and consequently acquire – skills that are seen as outside the traditional realm of translation: page layout, revision or editing of page-proof files, web editing, etc.

As we will see more in detail in sections 2 and 3, another, less compartmental, approach to professional translation and teaching practice is nonetheless possible. The inclusion of semiotics into translation does not necessarily have to pass through theory; rather, it emerges as a natural necessity when one shifts the focus from the translation process (and the possibility of breaking it down into compartments, each defined by its enclosure into a single mode of expression) to the final text as the product of this process, but already detached from it (which can be broken down only artificially, for analytical purposes). In this section, however, we will concentrate on the reasons why translators are foreigners to the non-verbal domains, particularly the visual.

1.2. Barthes’ paradox, and the visual as universal

The reasons why translation is usually seen as a verbal-only activity are not limited to the need of facilitating the production of the target text by compartmentalising it; the very possibility of translating, modifying the visual dimension of a text is seldom taken into consideration anywhere along the target-text-making process. This appears to be the natural result of the common perception that visual signs, unlike verbal ones, do not constitute an obstacle to interlinguistic and intercultural communication, thanks to their alleged universal, immediate nature – meaning that they are not encoded in an arbitrary set of signs which can only be understood by those who belong to a certain community of users. Let us look at what may seem the most immediate, therefore universal, kind of visual signs: photographs, which depict external reality rather than images produced directly by the author (as opposed, for instance, to paintings, which arguably do respond to different codes depending on the culture and artistic community in which they were produced).

In Image-Music-Text, Roland Barthes points out what he calls a “paradox”: the photographic image does convey a meaning, but, since the photograph is (or was in his time) merely analogical in nature and the processes involved in its creation do not transform or encode the reality it depicts, that message is “a message without a code” (Barthes 1961 [1977: 17]). He repeats this several times, stating that all that we need to read the message it conveys denotatively is “the knowledge bound up with our perception […], an almost anthropological knowledge” (Barthes 1964 [1977: 36]), thus providing an argument for the untranslatability of photographs, and actually pointing out that photographs cannot be described in words (i.e., translated intersemiotically; for a more insightful explanation of “intersemiotic” translation, see Jakobson 1959 and Toury 1986), because verbal language inevitably superimposes connotation upon denotation (Barthes 1961 [1977: 18]). But Barthes also highlights that the denotative and objective status of photographs is “mythical (these are the characteristics that common sense attributes to the photograph)” (Barthes 1961 [1977: 19]) and this myth is perpetuated by the mechanical means of production of the image (Barthes 1964 [1977: 44]), while in fact photographic images are naturally connoted and, especially in advertising, are the instruments of intentional processes of signification that are established a priori, therefore encoded in the photograph (Barthes 1964 [1977: 33]). The paradox is apparently solved by postulating that the iconic (i.e., non-verbal) dimension of the image can carry two kinds of message, one of which is not coded (denotation, based on perception) while the other is (connotation, based on culture). This distinction is in fact only operational, since it “is not made spontaneously in ordinary reading: the viewer of the image receives at one and the same time the perceptual message and the cultural message” (Barthes 1964 [1977: 36], emphasis in the original). Similarly, in Mythologies Barthes indicates that photographs can be the material for mythological systems, that is to say, secondary (i.e., complex) semiotic systems (Barthes 1957: 199-201), which should be seen as inseparable wholes and where distinctions among signifier, signified and sign can be drawn only artificially, for analytical purposes (Barthes 1957: 197).

One might in fact wonder whether Barthes’ distinction between the denotation and connotation of photographs – the first pointing to the absence of a code, the second implying the presence thereof – has the side-effect of reiterating the common-sense perception that at least at some theoretical primary level, photographs can still be seen as truthful, objective and immediate (not encoded in any system), therefore universal and untranslatable. In the following section, however, we will see that this common perception is clearly illusory in advertising and its translation.

2. The localization of visual elements in advertising

Advertising is one of the multimodal genres which are most suitable for analysing the interplay of verbal and visual elements. In the case of print ads, advertisers need to condense in one page, or a few pages at best, a complex referencing system that allows for the reader’s understanding of, and identification with, the represented world and its inherent values (Eco 2002 and 1965 [1992: 72], Williamson 1978: 170, De Mooij 2004: 190-192). For this reason, they usually build up rich and highly structured multimodal frameworks, where redundancy plays an important part as it not only reiterates the message across time and space (e.g., repetition of the same ad, or different ads of the same campaign, in different magazines or consecutive issues of the same magazine) but also ensures that the message gets through to the reader by simultaneously repeating it, or scattering its components, across several co-occurring sensory channels and modes of expression. For instance, ads for perfumes may contain a scented band in addition to verbal and pictorial elements that synaesthetically recall, or signify, the scent. The reader may even engage in an active tactile experience by lifting the flap of the page in order to reveal the band imbued with the product, perhaps rubbing it against her/his wrist. Usually, however, the two main modes of communication in print ads are the verbal and the visual, articulated in complex interactions that make them “a good starting point for studying aspects of visual communication” (Van Leeuwen 2005: 8).

Evidence of the existence of a visual code, or system, or even “grammar” of visual signs has already been provided by ground-breaking semiotic work (Kress and Van Leeuwen 1996, Van Leeuwen and Jewitt 2001, Thibault 1997). In such work, Barthes’ paradox is solved by the discovery of a code of signification which is embedded in all forms of visual communication, even the photographic image (not just “artificial nonverbal … codes” such as graphs, Sebeok and Danesi 2000: 65). Pure denotation simply ceases to be an option, because it is never possible in real communication, but can be detached from connotation and meaning-making only in formal analysis (as Barthes himself admitted, see above). The core of the issue here is the question whether the elements of such visual code are translatable from one culture or semio-linguistic community to another, and if so, how this may concern translators. Were this code truly universal, then not only would there be no need to translate it, but the very preconditions for translation would not exist.

A few other considerations regarding the translatability of meaningful visual elements are necessary here. First, visual elements, like language, convey cultural values and stereotypes, and sometimes clear indexical and symbolic relationships can be recognised. For instance, cultures have different “high modality” colours (Scollon and Scollon 2003: 91), and colours may take up political, social or even commercial indexical values that are relevant in one community only (see Munday 2004: 212 on orange and green in Northern Ireland). We may recall here the importance of the use of recognisable colours for the identity of trade marks or bank offices (which may have a national or even local diffusion), or the notion of colour as applied to dress codes: black shoes are strongly recommended in the banks and offices of the City of London, and in this community they may well match a dark blue suit, while the combination of black and dark blue would be seen as a sign of bad taste in an Italian workplace. This would seem to be an argument in favour of the translatability of visual elements.

A second argument in favour of even more structural intersemiotic intercultural translation is that of high-context and low-context cultures: in cultures that depend highly on context, a significant part of the information that would normally be expressed verbally in low-context cultures must be retrieved from the non-verbal environment, including non-verbal communication conventions (Hall 1966, in Infante et al. 1993: 432. Examples of low-context cultures provided by Hall include North-Americans, whereas most Asian cultures would tend to be high-context). A third argument, again relating to structures, not single elements, is that the norms of visual composition may differ substantially across cultures, for instance depending on reading directions (left to right, right to left, top to bottom) or traditions concerning the visualization of religious hierarchies (Kress and Van Leeuwen 1996: 199 and 203-206). Fourth, visuals can be used to convey the image of a culture into other cultures: for instance, visual renderings of Italianness or “Italianicity” in advertising for agro-food products make extensive use of photographic techniques and lights to convey an idea of genuineness and authenticity – even when the products are not really made in Italy (Barthes 1964 [1977], Chiaro 2004). Such visualizations represent a first stage of intersemiotic “translation” aimed at marketing – translating a given culture in visual terms in order to make it more appealing for commercial purposes.

These arguments point to the need to take the visual mode into account when translating advertisements. It may be superfluous here to point out that in advertising (and advertising translation), adjusting the message to the addressee’s social and cultural values is vital for the ad or campaign to be successful. Domestication is often inevitable; if the addressee does not fully understand, or feel moved or compelled by, the translated advertisement, this would drastically reduce the effectiveness of a presumably substantial investment in marketing. The success, or failure, of an advertising translation is assessed in terms of its commercial outcome; traditional linguistic quality assessment criteria, such as faithfulness and “respect” for the source text, disappear altogether (Guidère 2000: 280). This, the arguments above, and the heavy reliance of the advertising genre on visuals all point to the need to treat the advertising text as a semiotic whole, and to the impossibility of separating the copy from other non-verbal components, if the result is to be a text that functions well in the target culture and society.

I have already treated elsewhere the importance of potentially invisible, and easily overlooked, cultural stereotypes in the translation of advertising material (Torresi 2004). I have also already provided examples of advertisements containing visual translations, i.e., where the visual was actually changed – either in its meaning in a given context, or in its physical form – when the ad was transferred across cultures (Torresi forthcoming). Similar examples, however, abound in all campaigns realized following the localizing approach as opposed to global or “glocalized” campaigns (Adab 1998: 224). Intersemiotic translation is a particularly effective instrument when the very image, not only of a given product but of a whole brand and the values it aspires to embody, must be adapted to different target cultures. For instance, Figures 1 to 3 advertise the same product in the same period and in three women’s monthly magazines; however, while the first two ads were circulated in the UK and Italy, respectively, the third was distributed in the United States. The benefits of the product are the same: it is an anti-age cream that contains a mild disinfectant against blemishes. The two models, too, do not seem to have a dramatically different physical appearance: they are both white, brown-haired, and look about the same age and weight. Both the visuals and the copies, however, are encoded in such a manner that Fig. 1 and 2 convey a different image of woman than Figure 3, which in turn reflects different brand positioning (or aspirations) on the respective local markets.

In Fig. 1 and 2, the model is set against an indeterminate cream-coloured background which decontextualizes her figure, at the same time generalizing her, turning her into the icon of the kind of woman the perspective buyer is called to identify with (cf. Kress and Van Leeuwen 1996: 165-166). (The differences between Fig. 1 and Fig. 2 will not be taken into consideration here.) The decontextualization effect is further enhanced by the contrast between the model’s skin (which is highlighted, because it is the area of application of the product) and her white sweater. The dominant colour range is pastel, spanning from white to pale pinks through cream. The general impression is one of refined, quiet elegance, without any trace of excess – which is mirrored by the model’s measured smile, expressing mild self-satisfaction rather than happiness or irony. The Italian copy informs us that this woman, just like her skin, feels “young under some respects and mature under others” [“giovani per alcune cose e donne mature per altre”].

Youth and maturity are visualized more explicitly in Fig. 3, through two clear symbols that recur across both the headline and the body-copy – “bifocals and ripped jeans,” which in turn refer indexically to two other symptoms of old age and youth – respectively, “wrinkles and pimples.” Several visual and verbal cues stand against this woman’s transformation into an icon of measured elegance. First of all, the background points to a structured physical environment such as a room, which provides some kind of context, although it is indistinct and, being dark, matches the model’s black blouse, thus achieving the same kind of contrast of Fig. 1 and 2, only in negative. Second, there is an element of irony in the model’s facial expression and angle: whereas in Fig. 1 and 2 the angle of the model’s face is almost frontal with respect to the reader and the eye-contact vector is straight, in Fig. 3 the model looks at the reader from above her glasses and although the vector of her gaze is perpendicular to the page, the same cannot be said with respect to the axis of her body and face, which are positioned three-quarters to the reader. This gives her a slanted glance and makes the inclination of her smiling lips more evident, building a distance between her and the reader that makes identification less immediate (Kress and Van Leeuwen 1996: 144). Third, the humorous look of the model’s face is matched by the frivolous and inconsistent juxtaposition of old-fashioned glasses for long-sightedness and ripped jeans with embroidered multi-coloured flowers; the element of inconsistency is reinforced and hyperbolically described as bordering with folly by the payoff, “the perfect remedy for your split-personality skin” (my emphasis), where split personality can be metonymically attributed to the woman herself. Fourth, “pimples” – which, like the trousers in the photo, are hardly recognizable as tokens of refined elegance, and do not point to the notion of balance, but rather recall hormonal imbalance and overindulgence in unhealthy food – are openly referred to in the verbal part of the ad, whereas the British and Italian ones only mention “blemishes” (“impurità” and “imperfezioni” in Italian). In conclusion, the interaction of the verbal and visual modes in Fig. 1-2 attributes to the product and its brand an image of a well-balanced, elegant beauty, thus suggesting that Oil of Olay aims at achieving an equal footing with more established (and expensive) brands. Conversely, Fig. 3 conveys the image of a woman who seems to acknowledge her hectic, neurotic, excessive and rebellious nature, without, however, ceasing to “fight” and find a “remedy” to it. This points to an entirely different brand image; after all, stating that this cream is as effective “as leading department store brands” marks its implicit exclusion from top-of-the-range positioning.

Complex intersemiotic adaptations of advertisements and brand images as in the Oil of Olay example can be defined as the result of a translation process, but are obviously not carried out by translators alone. More probably, they are the product of a long decision-making process that starts with market analysis and brand positioning, and ends with the production of fresh art and copy, rather than involving interlinguistic translation proper. Such examples, however, show that the intersemiotic translation of advertising material is not only theoretically possible, but it is a reality. It may not be as frequent as interlinguistic, intrasemiotic translation (i.e., re-encoding only the verbal elements of the source text in a different verbal language), but it is nonetheless a resource that translators may resort to if they judge it necessary or appropriate to do so. How to do it in practice is probably less complicated and sounds much more familiar than one might expect, as we will see in the examples provided in the following section.

3. Translating the visual in non-fiction publications

In this last section, I would like to explain with a few very simple practical examples how it is in fact possible (and sometimes inevitable) to find a use for the “semiotic competence” that is required when translating multimodal genres (Kadric and Kaindl 1997: 136). I hope these examples, taken from real-life professional practice, can illustrate how a translator who is oriented towards the text as a semiotic whole rather than as an exclusively verbal phenomenon is not necessarily a competitor for other professional figures traditionally responsible for the visual and graphic rendering of the text (although a translator offering “turnkey” solutions including translation and typesetting would surely find a better market for his/her services). Rather, a translator who is aware of the importance of nonverbal elements, and the resource they represent for translation, proves a more reliable team-worker and produces better target texts. In order to make this argument relevant for as many readers as possible, I will leave the field of advertising for the moment, and apply what has been discussed in the previous sections to non-fiction publications – a genre that is perhaps more reassuringly recognisable for everybody, where decisions about translation do not come from some corporate marketing department, but are mainly limited to familiar figures such as translators, reviewers, editors, and a few members of the staff of a publishing house.

In Fig. 4, the visual rendering or “inscription” (Kress and Van Leeuwen 1996: 230-232) is clearly relevant in the construction of the meaning of the word “splat,” which is an instance of multimodality and intersemiotic translation in itself, as it is an onomatopoeia used to “verbalize” and “auralize” an originally visual element (Jackson Pollock’s art, which was shown on the next page in the original book). The force with which the paint is thrown on the canvas, and the loudness and saturation of the resulting sound, are rendered through the use of large-sized, thick bold capital letters, while the chaotic look of the painter’s art and its disruption of traditional artistic precepts is reproduced by scattering the letters across the page – perfect evidence for Schopp’s (2002) argument for treating layout and typography as translation issues. In this particular case (Fig. 5), the Italian translator consciously selected an onomatopoeia that matched the conditions for a graphic rendering that was similar to that of the source text, choosing “sciaf!” (pronounced /st∫af/) instead of other tempting options, such as “sciàffete,” linguistically more humorous because hyperbolic, but too long to allow an equally large print (the hyperbolic element is partly recovered by the exclamation mark). The mix of verbs and onomatopoeias crumpled together in Fig. 6 provides an even clearer example of the extent to which verbal, visual and aural elements can be intertwined in one semiotic unit, which must certainly be translated linguistically, but taking necessarily the other elements into account, reconstructing sounds and references to the picture it alludes to and meeting the criteria for graphic layout, such as word length (the result can be seen in Fig. 7). Albeit to a much lesser extent, the translation of such “complicated” scripts poses some of the problems raised by the translation of concrete poetry (Brotherston 1998).

Another example in which the translator’s semiotic awareness proved useful is represented by Fig. 8 and 9. Here the translator, being aware of the need to preserve the complementarity of the photographs with their respective captions, and pursuing a domesticating strategy which was in line with the content of the book (structured as a prolonged advertising copy), contacted the publishing studio to suggest that the text in the two notices be replaced with an Italian text by means of a photo editing software. The studio then contacted the publishing house (the end customer), who conversely decided to adopt a “low-context” solution and leave the visuals as they were, but adding the translation of the notices to the captions. The same solution had already been adopted for the French translation and the same policy was followed for the Spanish, German and Japanese translations of the book. This, however, does not constitute a “defeat” of the translator’s semiotic orientation, nor does it refute the fact that visuals can be altered for translation purposes (as might have been the case here). Rather, the translator’s awareness helped save time and might in turn have raised the publisher’s awareness of the problem, thus indirectly contributing to the harmonization of all translated versions. As is often the case on the translation market, it is the client who has the last word; it is vital, however, that translators raise problems and stimulate the client or the professionals they work with in order to find solutions together in the quickest possible way.

If the translator chooses or is able to deal with words only, in situations such as those described here and in other multimodal texts such as advertising material, s/he would actually be an obstacle to the production of a functional target text, because the task of matching the verbal mode with the non-verbal ones would fall to other professionals, who might not be in a position to fulfil this task (not all translation jobs are backed by a solid network of reviewers and editors – think about technical handbooks, where the coordination of the verbal and visual elements is nonetheless vital for the text to function properly). Moreover, the text production chain is made of several links, each of them usually communicating only with the immediately preceding and immediately following ones; if the publisher’s internal staff (the last link) detected a malfunctioning of the text imputable to the translator’s work (if, for instance, the translator had not provided a suitable list of words for the re-encoding of Fig. 6, which “looks like,” and partly is, a picture and might therefore be overlooked), the text must go back along the chain, one step at a time, until it reaches the translator again, who hopefully corrects the problem and sends the corrections back again, with a significant waste of time and energy that might have been saved if s/he had been aware of, and signalled, such a possibility in the first place.

In conclusion, this paper does not suggest “trespassing” on other professionals’ fields of action and markets, or taking independent action to change visual elements without the client’s consent. There is an ethical side to the de-compartmentalization of the translation process that cannot be ignored. But developing full semiotic awareness is vital for grasping the global meaning of source texts, for creating target texts that function well in their formal, social and cultural contexts, and for smoothing out the translation process, which otherwise risks to be hindered by the lack of communication and mutual understanding among the professionals involved.

No text can be said to be exclusively verbal: even words printed on paper or viewed on a computer screen have a visual dimension (the layout) and a tactile one (the paper, or the pressure of the fingers on the keyboard). Moreover, new text genres are emerging that rely on a high level of multimodality. This is often ignored, not only by translators, but also by clients: the inconsistencies that are still recurrent in the adaptation of some videogames or user’s instructions for homecare appliances or industrial equipment, for instance, can often be imputed to the practice of providing translators with the verbal text without the visuals that accompany and complete it, perhaps in order to avoid espionage or to make it easier to exchange the material via e-mail. Gaining more semiotic skills and confidence is therefore only the first step towards what might become political action – persuading clients that translators are not just “word-mongers,” but text- and meaning-makers.