A Systems Theory of Multimodality

James Gee


This paper offers a tentative theory of how to approach the analysis of multimodal “texts” (using the word loosely for any integrated set of symbols that resemble written texts in communicative power).  Much work on multimodality is new and many key issues are not yet broached, let alone settled.  The theory I produce treats modes as semiotic systems (sets of social conventions about meaning) and multimodality as a system of systems.  A theory in a new domain helps us understand what we take the “things” we are going analyze to be (our “ontology”).  In turn, our choice of “things” determines the sorts of tools we will need for analysis and the sorts of questions we will ask, a number of which are delineated in this paper.  The theory developed here is based on the correspondences between linguistic systems and non-linguistic modal systems.  The paper closes with a discussion of the implications of the approach I propose for work on design and learning.

Paraules clau

multimodality, semiotic systems, communiative modes, design, learning.

Text complet:

PDF (English)


Gee, J. P. (2004). Situational language and learning: A critique of traditional schooling. London: Routledge.

Gee, J. P. (2007). What video games have to teach us about learning and literacy. Second Edition. New York: Palgrave/Macmillan.

Gee, J. P. (2014a). An Introduction to discourse analysis: Theory and method. Fourth Edition. London: Routledge, 2014

Gee, J. P. (2014b). Unified Discourse Analysis: Language, Reality, Virtual Worlds, and Video Games. New York: Routledge, 2014

Gee, J. P. (2017a). Introducing discourse analysis: From grammar to society. London: Routledge.

Gee, J. P. (2017b). Teaching, learning, literacy in our high-risk high-tech world: A framework for becoming human. New York: Teachers College Press.

Halliday, M. A. K. (1978), Language as social semiotic: the social interpretation of language and meaning. London: Edward Arnold.

Halliday, M. A. K. (1985). Written and spoken language. Geelong, Victoria, Australia; Deakin University.

Halliday, M. A. K. (2013). Halliday’s introduction to functional grammar. Fourth Edition. Revised by Christian M.I.M. Matthiessen. New York: Routledge.

Halliday, M. A. K. & Greaves, W. S. (2004). Intonation in the grammar of English. London: Equinox.

Halliday< M. A. K. and Hasan, R. (1976). Cohesion in English. London: Longman.

Halliday, M. A. K. & Matthiessen, C. M. I. M. (1999). Construing experience through meaning: A language-based approach to cognition. New York: Continuum.

Jewitt, C. (ed.) (2009). The Routledge handbook of multimodal analysis. London: Routledge.

Jewitt, C. and Kress, G. (2003). Multimodal literacy. New York: P. Lang.

Kress, G. (2010). Multimodality: A social semiotic approach to contemporary communication. London: Routledge.

Kress, G. and Van Leeuwen, T. (1996). Reading images: The grammar of visual design. London: Routledge.

Kress, G. and Van Leeuwen, T. (2001). Multimodal discourse: The modes and media of contemporary communication. New York: Oxford University Press.

Napier, S. L. (2005). Anime from Akira to Howl’s Moving Castle, Updated Edition: Experiencing contemporary Japanese animation. New York: Palgrave/Macmillan.

Saussure, F. de (1986). Course in general linguistics. Chicago: Open Court. Originally published in French in 1916.

DOI: https://doi.org/10.17345/ute.2018.2.2489

Enllaços refback

  • No hi ha cap enllaç refback.