Item Details

A functional projection from visual contrast to mode: Creating building blocks for reading multimodal visual narrative

Issue: Vol 13 No. 1-2 (2017)

Journal: Linguistics and the Human Sciences

Subject Areas: Writing and Composition Linguistics

DOI: 10.1558/lhs.36177

Abstract:

In multimodal visual narratives such as Japanese manga, meaning is created through components of different modes - from verbals to onomatopoeias, emblems, and graphics. With the aim of integrating different modes (from text to image) so as to suggest an integrated solution that collectively defines the composing elements of comics, this paper proposes a Function of Mode, which applies semiotic and mathematical principles to expound on the subconscious process of recognizing visual contrasts that are drawn, printed or digitally displayed into various modes that form the building blocks of multimodal discourse. This function sits at the base of multiliteracy for understanding multimodal visual narrative and enables the reader to distinguish different modes by accentuating relevant semiotic channels of meaningmaking - iconicity for imagistic description, symbolization for linguistic features, entitativity for metaphorizing and visualizing non-graphic images, and diegesis for demarcating which elements are meaningful, ideationally or interpersonally.

Author: Xiran Yang

View Full Text

References :

Atkin, A. (2005). Peirce on the index and indexical reference. Transactions of the Charles S. Peirce Society, 41 (1): 161–188.

Atkin, A. (2006/2013). Peirce’s Theory of Signs, The Stanford Encyclopedia of Philosophy (Summer 2013 Edition), Edward N. Zalta (Ed.), URL = <http://plato.stanford.edu/archives/sum2013/entries/peirce-semiotics/>. Retrieved 10-06-2016.

Atkin, A. (2008). Peirce’s final account of signs and the philosophy of language. Transactions of the Charles S. Peirce Society: A Quarterly Journal in American Philosophy, 44 (1): 63–85.

Bateman, J. (2014). Text and Image: A Critical Introduction to the Visual/verbal Divide. Abingdon: Routledge. https://doi.org/10.4324/9781315773971

Bateman, J. A. and Veloso, F. O. (2013). The semiotic resources of comics in movie adaptation: Ang Lee’s Hulk (2003) as a case study. Studies in Comics, 4 (1): 135–157. https://doi.org/10.1386/stic.4.1.135_1

Bordwell, D. and Thompson, K. (2008). Film Art: An Introduction (8th Edition). New York: McGraw-Hill.

Camurri, A., Mazzarino, B., Ricchetti, M., Timmers, R., and Volpe, G. (2004). Multimodal analysis of expressive gesture in music and dance performances. In Gesture-based Communication in Human-computer Interaction, 20–39. Berlin and Heidelberg: Springer. https://doi.org/10.1007/978-3-540-24598-8_3

Feng, D. and O’Halloran, K. L. (2012). Representing emotive meaning in visual images: A social semiotic approach. Journal of Pragmatics, 44 (14): 2067–2084. https://doi.org/10.1016/j.pragma.2012.10.003

Forceville, C. (2005). Visual representations of the idealized cognitive model of anger in the Asterix album La Zizanie. Journal of Pragmatics, 37 (1): 69–88. https://doi.org/10.1016/j.pragma.2003.10.002

Fuse, H. (2004). Manga wo Kaibousuru [Anatomizing manga]. Tokyo: Chikumashobo.

Halliday, M. A. K. and Hasan, R. (1985/1976). Cohesion in English. London and New York: Longman.

Halliday, M. A. K. (1994). An Introduction to Functional Grammar (2nd edition). London: Edward Arnold.

Halliday, M. A. K. (2002a). The Collected Works of M. A. K. Halliday. Vol. 1. (J. J. Webster, Ed.). London: Continuum.

Halliday, M. A. K. (2002b). The Collected Works of M. A. K. Halliday.Vol. 2. (J. J. Webster, Ed.). London: Continuum.

Halliday, M. A. K. and Webster, J. J. (2014). Text Linguistics: The How and Why of Meaning. Sheffield: Equinox Publishing.

Hatfield, C. (2009). An art of tensions. In J. Heer and K. Worcester (Eds), A Comic Studies Reader. 132–148. Jackson, MS: University Press of Mississippi.

Jacobs, D. (2013). Graphic Encounters: Comics and the Sponsorship of Multimodal Literacy. New York: Bloomsbury.

Jewitt, C. (2009). The Routledge Handbook of Multimodal Analysis. London: Routledge

Kaitani, S. (2005/2010). Liar Game-1. Tokyo: Shueisha.

Kirby, J. T. (1996). Classical Greek Origins of Western Aesthetic Theory. In B. Allert (Ed.) Languages of Visuality, 29–45. Detroit, MI: Wayne State University Press.

Kishimoto, M. (2000/2013). Naruto-1. Tokyo: Shueisha.

Kress, G. and Van Leeuwen, T. (2001). Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Arnold.

Kress, G. and Van Leeuwen, T. (2006). Reading Images: The Grammar of Visual Design (2nd Ed.). London: Routledge. https://doi.org/10.4324/9780203619728

Kress, G., and Van Leeuwen, T. (2002). Colour as a semiotic mode: Notes for a grammar of colour. Visual Communication, 1 (3): 343–368. https://doi.org/10.1177/147035720200100306

Martin, J. R. and Rose, D. (2008). Genre Relations: Mapping Culture. London and Oakville, CN.: Equinox.

Matthiessen, C., Teruya, K., and Lam, M. (2010). Key Terms in Systemic Functional Linguistics. London: A&C Black.

McCloud, S. (1994). Understanding Comics: The Invisible Art. New York: HarperPerennial.

Mitchell, W. J. T. (1984). What is an Image?. New Literary History, 15 (3): 503–537. https://doi.org/10.2307/468718

Mitchell, W. J. T. (2009). Beyond comparison. In J. Heer, and K. Worcester (Eds), A Comic Studies Reader, 116–123. Jackson, MI: University Press of Mississippi.

MODE. (2012). Glossary of multimodal terms. URL=<https://multimodalityglossary.wordpress.com/>. Retrieved 25-10-2016.

O’Halloran, K. L. (Ed.). (2004). Multimodal Discourse Analysis: Systemic Functional Perspectives. London and New York: Continuum.

Oda, E. (1997/2013). One Piece-1. Tokyo: Shueisha.

Peirce, C. S. (1932). Collected Papers of Charles Sanders Peirce. Volume II: Elements of Logic. Cambridge: Harvard University Press. https://doi.org/10.2307/2270913

Peirce, C. S. and Houser, N. (1998). The Essential Peirce: Selected Philosophical Writings (Vol. 2). Bloomington, IN: Indiana University Press.

Quek, F., McNeill, D., Bryll, R., Duncan, S., Ma, X. F., Kirbas, C., … and Ansari, R. (2002). Multimodal human discourse: gesture and speech. ACM Transactions on Computer-Human Interaction (TOCHI), 9 (3): 171–193. https://doi.org/10.1145/568513.568514

Rosen, K. H. (2007). Discrete Mathematics and Its Application (6th Ed, International Edition). New York: McGraw-Hill.

Short, T. L. (2007). Peirce’s Theory of Signs. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9780511498350

Stewart, J. (2009). Multivariable Calculus (6th Ed., Metric International Version). Thomson Brooks/Cole.

Stöckl, H. (2004). In between modes: Language and image in printed media. In E. Ventola, C. Charles, and M. Kaltenbacher (Eds). Perspectives on Multimodality, 9–30. Amsterdam and Philadelphia, PA: John Benjamins.

Togashi, Y. (1998/2012). Hunter×Hunter-1. Tokyo: Shueisha.

Van Leeuwen, T. (2006). Towards a semiotics of typography. Information Design Journal, 14 (2): 139–155. https://doi.org/10.1075/idj.14.2.06lee

Veloso, F. and Bateman, J. (2013). The multimodal construction of acceptability: Marvel’s Civil War comic books and the PATRIOT Act. Critical Discourse Studies, 10 (4): 427–443. https://doi.org/10.1080/17405904.2013.813776

Yang, X. and Webster, J. (2015). To be continued: Meaning-making in serialized manga as functional-multimodal narrative. Semiotica, 2015 (207): 583–606. https://doi.org/10.1515/sem-2015-0066

Yang, X. and Webster, J. (2017). Meaning-making in multimodal visual narrative: patterns of visual weaving. In J. Webster and X. Peng (Eds). Applying Systemic Functional Linguistics: the State of the Art in China Today. London and New York: Bloomsbury Publishing.