Tag Archives: 2009 Congress of the Humanities and Social Sciences

Kay O’Halloran interview on multimodal discourse: Part 1 of 3

I am thrilled to announce that Kay O’Halloran an expert on multimodal discourse analysis has given me an interview. She recently spoke at the 2009 Congress of the Humanities and Social Sciences in Ottawa as a featured speaker (invited by the Canadian Association for the Study of Discourse and Writing). Kay is an Associate Professor in the Department of English Language and Literature at the National University of Singapore and she is the Director of the Multimodal Analysis Lab. (more details about Kay in future installments)

Before going with the introduction and the interview, I want to explain why I think this work is important. (Forgive me if I gush?) We have so much media coming at us at any one time and it is increasingly being ‘mashed up’, remixed, reused, and repurposed. How important is text going to be when we have icons and videos and audio materials to choose from? Take for example, the bubble charts on Andrew Maynard’s 2020 blog which are a means of representing science Twitters. How do you interpret the information? Could they be used for in-depth analysis? (I commented earlier about the bubble charts on June 23 and 24, 2009 and Maynard’s post is here. You might also want to check out the comments where Maynard explains few things that puzzled me.)

As Kay points out in her responses to my questions, we have more to interpret than just a new type of chart or data visualization.

1. I was quite intrigued by the title of your talk (A Multimodal Approach to Discourse Studies: A paradigm with new research questions, agendas and directions for the digital age) at the 2009 Congress for the Humanities and Social Sciences held in Ottawa, Canada this May. Could you briefly describe a multimodal approach for people who aren’t necessarily in the field of education?

Traditionally, language has been studied in isolation, largely due to an emphasis on the study of printed linguistic texts and existing technologies such as print media, telephone and radio where language was the primary resource which was used. However, various forms of images, animations and videos form the basis for sharing information in the digital age, and thus it has become necessary to move beyond the study of language to understand contemporary communicative practices. In a sense, the study of language alone was never really sufficient because analysing what people wrote or said missed significant choices such as typography, layout and the images which appeared in the written texts, and the intonation, actions and gestures which accompanied spoken language. In addition, disciplinary knowledge (e.g. mathematics, science and social science disciplines) involves mathematical symbolism and various kinds of images, in addition to language. Therefore, researchers in language studies and education are moving beyond the study of language to multimodal approaches in order to investigate how linguistic choices combine with choices from other meaning-making resources.

Basically multimodal research explores the various roles which language, visual images, movement, gesture, sound, music and other resources play, and the ways those resources integrate across modalities (visual, auditory, tactile, olfactory etc) to create meaning in artefacts and events which form and transform culture. For example, the focus may be written texts, day-to-day interactions, internet sites, videos and films and 3-D objects and sites. In fact, one can think of knowledge and culture as specific choices from meaning-making resources which combine and unfold in patterns which are familiar to members of groups and communities.

Moreover, there is now explicit acknowledgement in educational research that disciplinary knowledge is multimodal and that literacy extends beyond language.

The shift to multimodal research has taken place as a result of digital media which not only serves as the object of study, but also because digital media technologies offer new research tools to study multimodal texts. Such technologies have become available and affordable, and increasingly they are being utilised by multimodal researchers in order to make complex multimodal analysis possible. Lastly, scientists and engineers are increasingly looking to social scientists to solve important problems involving multimodal phenomena, for example, data analysis, search and retrieval and human computer interface design. Computer scientists and social sciences face similar problems in today’s world of digital media, and interdisciplinary collaboration is the promise of the future in what has become the age of information.

Have a nice weekend. There’ll be more of the interview next week, including a bibliography that Kay very kindly provided.

Nanotechnology metaphors and understanding visual data

Leave a reply

I found a typesetting metaphor today in a media release titled, ‘Molecular typesetting — proofreading without a proofreader‘. The number of publishing, writing, and reading metaphors associated with nanotechnology has always startled me. As for the article, it is about how proteins are built with a minimal number of errors in a process that researchers compare to typesetting. If you want to read more, you can go here to Nanowerk News.

I looked at Andrew Maynard’s 2020 Science blog and found a posting that presents some visual data about science twittering. He has three spheres made of bubbles or smaller spheres representing the number of followers that science twittering attracts. He’s done this before and I’m still not sure how to interpret the data and I mean that from two perspectives. I don’t understand the visual data being presented very well (Maynard does provide an explanation in a screencast) and while I find the whole Twitter scene interesting I’m waiting to see if it becomes something more substantive (which seems to be Maynard’s stance as well).

With regard to visual data, I think this will become increasingly important and it was one of the reasons I was so interested in Kay O’Halloran’s talk at the 2009 Congress of Humanities and Social Sciences about mathematics and using visual data to communicate about it. Unfortunately, the organizers were not able to arrange a webcast but I’ll see if I can dig up so more information about what she’s doing.

As for the Twitter phenomenon, it seems interesting to me that MySpace has just downsized itself (more here) as I can recall when it was as a big trend as Twitter is now. I’m not sure what conclusions can be drawn from the popularity of any social networking phenomenon. I think it is clear that people are interested in each other (and sometimes for the oddest of reasons) as for anything else I need more data.

One brief note, I had occasion to email Andrew Maynard last week and during the exhange I asked him why the Project on Emerging Nanotechnologies is having fewer events. (I figured their Chief Science Advisor would know why.) He says there is a reorganization taking place.

Science and multimodal media approaches

Leave a reply

There’s an interesting article on an experiment being conducted at Fortune magazine. For anyone who’s not aware, the publishing industry is in a serious quandary and many publishers are struggling for survival. This explains why Fortune magazine has a multimodal media version of its print cover story available on the web. From the article by Andrew Vanacore on the Physorg.com site here,

Dispensing advice on finding a job during a recession, the piece had a soundtrack, a troupe of improv actors from Chicago and about 4,000 fewer words than your average magazine feature. Instead of scrolling through a column of text, readers (if the term can be applied) flipped through nine pages that told the story with a mix of text, photo-illustrations, interactive graphics and video clips.

I like that bit about “readers (if the term can be applied)” because I’ve been coming to the conclusion that with less and less text (think Twitter) that we may be returning to a more oral society as opposed to our still literate-dominant society. I’ve been thinking about this since some time in the early 1990’s when a communications professor (Paul Heyer) at Simon Fraser University first made the suggestion to us in class.

Following on this idea that we will be less and less text oriented, the work that Kay O’Halloran is doing at her Mulimodal Lab (situated at the National University of Singapore) casts an interesting light on where this all may be going with regard to science communication. An associate professor in the Dept. of English Language and Literature, O’Halloran is speaking tomorrow (in Ottawa, Canada) at the 2009 Congress of Humanities and Social Science about reading, mathematics, and digital media. I hope there will be a webcast of her talk available afterwards (I suggested it to the folks from the Canadian Association for the Study of Discourse and Writing (CASDW) who are sponsoring her talk. If there is a webcast, I’ll post a link.

Meanwhile, for those of us not lucky enough to be there, from the programme,

To understand digital texts we need theories that study more than words alone. This talk will show how images, mathematical and scientific symbols, gestures, actions, music, and sound can all be studied along with words using examples from the classroom, digital media, and mathematics.

I believe that more and more of our communication, science and otherwise, is moving in a multimodal direction. It seems so obvious to me that it surprises me that it’s not commonly accepted wisdom.

Later this week, I will have more about science funding and I have notice of another sythetic biology event coming up at the Project on Emerging Nanotechnologie.