Multimodality

Overview

Multimodality refers to the complex process involved in transmitting and receiving information, that is, how one individual or group designs a method for sharing an idea or a concept with others. After all, any idea must find some form of transmittal or it will remain within the isolated mind of the person who has conceived it. Whether the idea is something basic—calling for roadside assistance, for example, or asking a person on a date—or far more complicated—a public speech, a poem, a researched paper that explores some radical insight into theoretical physics, an advertisement intended to sell toothpaste—that idea must find an appropriate form to maximize the opportunity for others to share it. Multimodality explores the wide variety of strategies a person can use to design effective communication. For example, ideas can be shared in a range of communication dynamics: in language (that is, writing and speaking), aurally (in sound), and visually (through images). Each strategy or mode is a complicated method through which meaning—that is, the original idea or concept that is to be shared—is first constructed and then shared.

Multimodality is itself a relatively new concept in communications theory. Beginning in the 1980s and the dawn of computer communication, digital data, and global networking, communication theorists began to appreciate the dimensions of communication itself, how information itself can be moved effectively and efficiently through modes other than language. Before the advent of computer technology and image technologies, knowledge was assumed best communicated through words, most often the printed word but in the case of academic settings and/or professional network settings, through spoken word, often enhanced and supplemented by the printed word, such as some sort of handout or an outline. In either case, communication was essentially language-centric. Communication, thus, was unimodal, that is, it relied largely on a single mode or method of representation and dissemination, privileging specifically the use of language, printed and/or spoken.

For nearly seven centuries, dating from the invention of the printing press in 1440, words, sentences, paragraphs were the constituent elements essential to communication. Nonfiction books and textbooks; popular magazines and specialized (academic) journals; newspapers; transcriptions of speeches; flyers; court records; letter writing; journals and diaries; and all the forms of creative writing (poems, narratives, short stories, dramas) defined the most reliable and most trusted medium for communication. In addition, the language mode was manifested in the robust and vibrant exchange of ideas and/or information represented by speaking and listening. Entire cultures were premised on language. Indeed, until the later decades of the twentieth century and the emergence of image technologies, so vested were cultures in this single mode of communication (language) that an entire culture thought of itself as a manifestation of its language. Individual identity within that culture was tied directly to language.

Communication theorists, beginning with the groundbreaking work done in the early 1990s by Gunther Kress and Carey Jewitt, both of the communication and media studies faculty of the University College in London, started to question whether language was in fact how knowledge was actually moved. They asked whether the human brain was actually far more complex and responsive to other modes of communication.

Children's books are not limited to printed words. As Kim and Kim argued in 2017, multimodality greatly enhances a toddler's education. Very young children are far more engaged by a book that uses multimodal strategies: The book still has words, but the book also has images—often large-sized, carefully detailed, and colorful. Preschool "touch and feel books" may have few words but come loaded with other exploratory modalities: The pages may have flaps that the child can open and close; swatches of material for the child to touch; buttons to press to trigger sound effects and even snippets of music; scented strips that can be scratched; reflective discs that act like mirrors; tactile effects such as yarn or zippers or buttons. These books still move the information (that is, a storyline with characters or an interesting topic), but through engaging a variety of modes it makes the communication process more interactive, more engaging, and the material more memorable. Add to that the communication system provided by a parent/guardian reading the book with the child—facial expressions, gestures, intonations, inflections—and the child is now engaged within an immersive multimodal learning environment, far removed from the unimodular, essentially passive experience of earlier generations in which the child simply listened. Multimodality then is about an inventory of strategies. As Kress argued in 2001, "[T]he different modes have technically become the same at some level of representation, and they can be operated by one multi-skilled person, using one interface, one mode of physical manipulation, so that he or she can ask, at every point: 'Shall I express this with sound or music?', 'Shall l say this visually or verbally?'"

Thus, with the advent of image technologies beginning with the revolution in television in the late 1940s and continuing into the exponential growth of the image technologies available on the Internet, through social media and the camera-technologies available in smartphones, communications theorists have begun to challenge the conventional assumptions about the primacy of language as the most reliable and most efficient system of meaning-transmission, which is central to all communication protocols. Previous to the Internet, a proud parent might watch his children build a snowman. The data point would be "My children built a snowman." That parent would only have two options to move that information: in writing by describing the information in a sentence or two in a letter or verbally by conveying the same information in a conversation. With the advantage of digital technologies, the parent is able to use multimodal strategies to illustrate and even animate the snowman event for the amusement of his social and family circles. To the textual mode he can add a visual (a photo of the event) or record the event and post it as a YouTube or TikTok video, which adds visual and aural information and a sense of immediacy. The information being moved carries a packet of effects not available through words alone, allowing friends and grandparents to "triangulate" input from the various modes for a richer experience (Martens, Martens, Doyle, Loomis & Aghalarov, 2012).

Multimodality thus acknowledges what people already intuitively understand: The brain engages new data through a variety of stimulations and sensual experiences, that indeed language (and words) are among the least effective methods. Stand on a street corner, sit in a classroom, stroll around a mall, drive down a street—the brain thrives in a world not of words, sentences, and paragraphs but of images and noises, textures, and colors; it receives and processes new data through all the senses; the brain responds to shapes, shadows, odors, motion, proximity, and sounds. That basic concept of complex engagement has shaped the research on multimodality, a revolutionary concept in communications theory that, in reaction to the unprecedented reach of digital technology, has begun to reshape assumptions about how information is moved in a range of basic cultural and social environments, most directly classrooms and the workplace. Multimodality has come to shape the social profile of the millennial generation, the first generation raised entirely within the digital age and a generation for whom technologies are an integral part of their schooling, their home life, and their professional careers. Australian media researchers Rowsell and Walsh concluded in 2011 that digital literacy had become essential to students in order for them to be prepared for the workplace.

Applications

Multimodality, as a communications theory, examines the impact and potential of four different modes: 1) print; 2) digital texts; 3) sound; and 4) images. More specifically, multimodality explores ways in which information is transmitted through a sort of mix-and-match approach in which no one mode is given primacy. Rather, an individual designs a system that draws on some or all of the modes to effect clear, efficient, and impactful communication.

Of course, print remains very much a viable mode. There is still in the digital age a foundational reliance on language, the transmission of ideas into hard copy; ink pressed on paper still carries with it the aura of stability and durability. Although industries that rely on paper-based printed communication have dramatically suffered financially during and against the emergence of web-based information sources, there is a viable market in paper for magazines, newspapers, books, legal correspondence, official documents and records, reference materials, and works of the creative imagination. A strong preference for printed modes—a morning newspaper, a volume of cherished poems, a how-to handbook, a tourist guidebook, a magazine for a long plane ride—remains for many consumers despite the advantages of digital media, in part because of all nostalgia or resistance to new forms, but in part because print media has its own modal advantages, including tactility. All genres of the printed mode that have remained staples in contemporary cultural communication.

Computer-based communication is a mode that uses a complex of transmission strategies to move information digitally. Software apps, platforms, blogs, social media, texting all combine a variety of modes. A basic website, for example, creates its impact not only through what information it presents but also in how that information is presented: colors, typography, images, frames, headers, shapes, and graphic effects are available for creating a visually stimulating package, much more like a magazine spread than a plain typewritten page, as a way to maximize the reception of that information. Computer mode is visually stimulating in that reading a screen involves light and color; computer mode is aurally stimulating as a website can provide musical backgrounds or sound effects; computer mode is arguably tactile in that a user has to engage keys and scroll around or negotiate a touchscreen. Indeed, even the printed word has found new life online; publishers, citing the advent of digital reading apps, have termed the first decades of the twenty-first century as something of a Golden Era for language-based communication.

Texts can also be created entirely through the manipulation of tones. Meaning can be delivered through a variety of heard media: in music, through spoken word, through sound effects, through recordings. Communication that engages the aural faculties creates a rich and potentially more personal experience than words, indeed a response that is beyond words. The aural mode does not rely exclusively on the impact of orchestrated and pre-designed sounds. The everyday world teems with aural modes of communication: ambulances, dogs, car horns, thunder, airplanes, doorbells, phone tones—the interaction with aural texts routinely establishes and sustains a rich environment of information.

The visual mode long ago gave rise to the expression, "A picture is worth a thousand words." Images, whether still or in motion, create an interactive communication system that inevitably creates impact. Further, there is a hierarchy of impact; most immediate are animated images with high-end resolution and color. A still photo of a wrecked car, for example, does not carry the same impact as a car wreck caught by a cellphone, and low-resolution phone footage does not impact as much as the same accident recorded by a sophisticated television camera. Bazalgette and Buckingham (2013) studied the impact of images in motion rather than still images as a vehicle for effective communication to conclude that moving images are most often the images saved in the memory. YouTube has indeed provided a generation with a virtually limitless trove of examples of the immediacy and effectiveness of the visual mode: a convenience store robbery recorded on a surveillance camera; a remarkable athletic play recorded by a phalanx of cameras and drones; a concert; a wedding or graduation; vacation videos: professional webinars; classroom presentations. The range and reach of the animated visual mode is integral to contemporary information transmission.

Modern educators have attempted to increase students' involvement and retention of material through multimodal strategies. This became particularly necessary during the global COVID-19 pandemic, when many schools and universities turned to remote learning. Message boards, chat apps, PowerPoint, videos, quiz apps, podcasts, and other options became essential tools for educators.

Viewpoints

Because at its core, multimodality concerns how new information is transmitted, it has profoundly changed conceptions on how both the classroom and the workplace operate; after all, both settings are constantly changing, high-pressure environments in which new information must routinely be moved efficiently and coherently to optimize results. In the end, a teacher introducing colors and a human resource officer onboarding a new employee, or a college literature professor exploring the implication of irony in Renaissance poetry and a project manager eager to instruct a team on the goals and protocols of a new project are all engaged in the process of constructing and then sharing abstract ideas through accepted channels. In each case, language-based communication, although helpful, is limited. Multimodality provides teachers and project managers with the opportunity to choose from an inventory of communications modes and then package and deliver that data in strategies designed to maximize the impact: handouts enhanced with images and shaped by decisions about eye relief and page design; lectures and/or verbal explanations that emphasize delivery, volume, hand movements, and body postures; interactive discussion pods that involve animated discussions among students or coworkers, that involve eye contact, gestures, inflection, and body language as well as environmental decisions ranging from furniture selections to light design; PowerPoint presentations graphically enhanced (color, font size, image placement) and layered with animation sequences, musical snippets, and/or sound clips. Within the theory of multimodality, because the brain learns through a variety of modes, the person who possesses the knowledge to be shared can orchestrate a delivery that is potentially the most effective, the most immediate, and the most useful transmission of that information. In turn, recipients of this information data are encouraged to produce documents of their own that show the value of drawing on a variety of modes. "The texts students produce often mimic the digital documents they may create in the workforce, including blogs or Web sites. Much more than a trendy new way of writing, a multimodal pedagogy insists that students make choices and have control over what they create" (Bourelle, Bourelle & Jones, 2015). Choice is what multimodality offers as knowledge itself can be packaged in ways that are both stimulating and consequential.

Bibliography

Bazalgette, C., & Buckingham, D. (2013). Literacy, media and multimodality: A critical response. Literacy, 47(2), 95–102. Retrieved May 23, 2018 from EBSCO Online Database Academic Source Ultimate. http://search.ebscohost.com/login.aspx?direct=true&db=asn&AN=89151002&site=ehost-live

Bloomberg, L. D. (n.d.) Employing Multimodal Strategies in Online Teaching. Sage. us.sagepub.com/en-us/nam/employing-multimodal-strategies-in-online-teaching

Bourelle, A., Bourelle, T., & Jones, N. (2015). Multimodality in the technical communication classroom: Viewing classical rhetoric through a 21st century lens. Technical Communication Quarterly, 24(4), 306–327. Retrieved May 23, 2018 from EBSCO Online Database Academic Source Ultimate. http://search.ebscohost.com/login.aspx?direct=true&db=asn&AN=110320224&site=ehost-live

Kim, K., & Kim, K. (2017). Multimodality play-literacy: A preschooler's storytelling, drawing, and performing of dinosaur extinction theories. Early Childhood Development, 187(3/4), 568–582. Retrieved May 23, 2018 from EBSCO Online Database Academic Source Ultimate. http://search.ebscohost.com/login.aspx?direct=true&db=asn&AN=121520072&site=ehost-live

Kress, G. (2001). Kress and van Leeuwen on multimodality. In Multimodal Discourse: The Modes and Media of Contemporary Communication (pp. 1–2). Oxford, UK: Oxford University Press.

Martens, P., Martens, R., Doyle, M. H., Loomis, J., & Aghalarov, S. (2012). Learning from picturebooks: Reading and writing multimodally in first grade. Reading Teacher, 66(4), 285–294. Retrieved May 23, 2018 from EBSCO Online Database Academic Source Ultimate. http://search.ebscohost.com/login.aspx?direct=true&db=asn&AN=83711251&site=ehost-live

Mondada, L. (2018). Multiple temporalities of language and body in interaction: Challenges for transcribing multimodality. Research on Language & Social Interaction, 51(1), 85–106. doi:10.1080/08351813.2018.1413878

Philippe, S. (2020, Oct. 5). Multimodal teaching, learning and training in virtual reality: A review and a case study. Virtual Reality & Intelligent Hardware, 2, 421-42, https://doi.org/10.1016/j.vrih.2020.07.008

Rowsell, J., & Walsh, M. (2011). Rethinking literacy education in new times: Multimodality, multiliteracies, & new literacies. Brock Education, 21(1), 53–62.