Category Archives: multimodal discourse

Is the medium the message? Virtual museums and the the user’s experience technology experience

A Sept. 21, 2015 Pennsylvania State University (Penn State) news release by Matt Swayne (also on EurekAlert) puts a different spin on art/science (Note: Links have been removed),

Museum curators planning to develop virtual exhibits online should choose communication and navigation technologies that match the experience they want to offer their visitors, according to a team of researchers.

“When curators think about creating a real-world exhibit, they are thinking about what the theme is and what they want their visitors to get out of the exhibit,” said S. Shyam Sundar, Distinguished Professor of Communications and co-director of the Media Effects Research Laboratory. “What this study suggests is that, just like curators need to be coherent in the content of the exhibit, they need to be conscious of the tools that they employ in their virtual museums.” [emphasis mine]

For some reason that phrase “need to be conscious of the tools that they employ” reminds of Marshall McLuhan and his dictum “the medium is the message.” Here’s more about study from the news release,

Many museum curators hope to create an authentic experience in their online museums by using technology to mimic aspects of the social, personal and physical aspects of a real-world museum experience. However, a more-is-better approach to technology may actually hinder that authentic experience, the researchers suggest.

In a study, visitors to an online virtual art museum found that technology tools used to communicate about and navigate through the exhibits were considered helpful when they were available separately, but less so when they were offered together. The researchers tested customization tools that helped the participants create their own art gallery, live-chat technology to facilitate communication with other visitors and 3-D tool navigation tools that some participants used to explore the museum.

The participants’ experiences often depended on what tools and what combinations of tools they used, according to the researchers, who released their findings in a recent issue of the International Journal of Human-Computer Interaction.

The news release goes on to provide some examples of when technologies do not mesh together for a good experience,

“When live chat and customization are offered together, for example, the combination of tools may be perceived to have increased usability, but it turns out using either customization or live chat separately was greater than either both functions together, or neither of the functions,” said Sundar. “We saw similar results not just with perceived usability, but also with sense of control and agency.”

The live chatting tool gave participants a feeling of social presence in the museum, but when live chatting was used in conjunction with the 3D navigation tool, the visitor had less of a sense of control, said Sundar, who worked with Eun Go, assistant professor of broadcasting and journalism, Western Illinois University; Hyang-Sook Kim, assistant professor of mass communication and media communication studies, Towson University and Bo Zhang, doctoral candidate in mass communications, Penn State.

Similarly, participants indicated the live chatting function lessened the realistic experience of the 3D tool, according to the researchers, who suggested that chatting may increase the user’s cognitive burden as they try to navigate through the site.

Each of these tools carries unique meaning for users, Sundar said. While customization provides an individualized experience, live-chatting signals a social experience of the site.

“Our data also suggest that expert users prefer tools that offer more agency or control to users whereas novices appreciate a variety of tools on the interface,” he added.

Users may react to these tools on other online platforms, not just during visits to online museums, Sundar said.

“We might be able to apply this research on tools you might add to news sites, for example, or it could be used to improve educational sites and long-distance learning,” he added. “You just have to be careful about how you deploy the tools because more is not always better.”

The researchers recruited 126 participants for the study. The subjects were assigned one of eight different website variations that tested their reactions to customization, live chat, 3D navigation and combinations of those tools during their visit to a virtual version of the Museum of Modern Art. The museum’s artworks were made available through the Google Art Project.

Here’s a link to and a citation for the paper,

Communicating Art, Virtually! Psychological Effects of Technological Affordances in a Virtual Museum by S. Shyam Sundar, Eun Goc, Hyang-Sook Kim, & Bo Zhang. International Journal of Human-Computer Interaction
Volume 31, Issue 6, 2015 pages 385-401 DOI: 10.1080/10447318.2015.1033912 Accepted author version posted online: 15 Apr 2015

This paper is behind a paywall.

Cool science; where are the women?; biology discovers graphical notations

Popular Science’s Future of .., a programme [developed in response to a question “What’s missing from science programming?” posed by Debbie Myers, {US} Science Channel general manager] , was launched last night (Aug. 11, 2009). From the Fast Company posting by Lynne D. Johnston,

The overall response from the 50-plus room full of mostly New York digerati, was resoundingly, “a show that was both entertaining and smart–not dumbed down.”

Their host, Baratunde Thurston, offers an interesting combination of skills as he is a comedian, political pundit, and author. If you go to the posting, you can find the trailer. (It’s gorgeous and, I suspect, quite expensive due to the effects, and as you’d expect from a teaser, it’s short on science content.)

It does seem as if there’s some sort of campaign to make science ‘cool’ in the US. I say campaign because there was also, a few months ago, the World Science Festival in New York (mentioned in my June 12, 2009 posting). Thanks to Darren Barefoot’s blog I see they have posted some highlights and videos from the festival. Barefoot features one of musician Bobby McFerrin’s presentations here.

Barefoot comments on the oddity of having a musician presenting at a science event. The clip doesn’t clarify why McFerrin would be on the panel but neuroscientists have been expressing a lot of interest in musician’s brains and I noticed that there was at least one neuroscientist on the panel. Still, it would have been nice to have understood the thinking behind the panel composition. If you’re interested in more clips and information about the World Science Festival, go here.

Back to my thoughts on the ‘cool’ science campaign, there have been other initiatives including the ‘Dancing with scientists’ video contest put on by the American Association for the Advancement of Science and the nanotechnology video contests put on by the American Chemical Society. All of these initiatives have taken place this year. By contrast, nothing of a similar nature appears to be taking place in Canada. (If you know of a ‘cool science’ project in Canada, please do contact me as I’d be happy to feature it here.)

On the subject of putting together panels, there’s an interesting blog posting by Allyson Kapin (Fast Company) on the dearth of women on technology and/or social media panels. She points out that the problem has many aspects and requires more than one tactic for viable solutions.

She starts by talking about the lack of diversity and she very quickly shifts her primary focus to women. (I’ve seen this before in other writing and I think it happens because the diversity topic is huge so writers want to acknowledge the breadth but have time and expertise to discuss only a small piece of it.) On another tack altogether, I’ve been in the position of assembling a panel and trying to get a diverse group of people can be incredibly difficult. That said, I think more work needs to be done to make sure that panels are as diverse as possible.

Following on my interest in multimodal discourse and new ways of communicating science, a new set of standards for graphically representing biology has been announced. From,

Researchers at the European Molecular Biology Laboratory’s European Bioinformatics Institute (EMBL-EBI) and their colleagues in 30 labs worldwide have released a new set of standards for graphically representing biological information – the biology equivalent of the circuit diagram in electronics. This visual language should make it easier to exchange complex information, so that models are accurate, efficient and readily understandable. The new standard, called the Systems Biology Graphical Notation (SBGN), is published today (August 11, 2009) in Nature Biotechnology.

There’s more here and the article in Nature Biotechnology is here (keep scrolling).

Reimagining prosthetic arms; touchable holograms and brief thoughts on multimodal science communication; and nanoscience conference in Seattle

Reimagining the prosthetic arm, an article by Cliff Kuang in Fast Company (here) highlights a student design project at New York’s School of Visual Arts. Students were asked to improve prosthetic arms and were given four categories: decorative, playful, utilitarian, and awareness. This one by Tonya Douraghey and Carli Pierce caught my fancy, after all, who hasn’t thought of growing wings? (Rrom the Fast Company website),

Feathered cuff and wing arm

Feathered cuff and wing arm

I suggest reading Kuang’s article before heading off to the project website to see more student projects.

At the end of yesterday’s posting about MICA and multidimensional data visualization in spaces with up to 12 dimensions (here)  in virtual worlds such as Second Life, I made a comment about multimodal discourse which is something I think will become increasingly important. I’m not sure I can imagine 12 dimensions but I don’t expect that our usual means of visualizing or understanding data are going to be sufficient for the task. Consequently, I’ve been noticing more projects which engage some of our other senses, notably touch. For example, the SIGGRAPH 2009 conference in New Orleans featured a hologram that you can touch. This is another article by Cliff Kuang in Fast Company, Holograms that you can touch and feel. For anyone unfamiliar with SIGGRAPH, the show has introduced a number of important innovations, notably, clickable icons. It’s hard to believe but there was a time when everything was done by keyboard.

My August newsletter from NISE Net (Nanoscale Informal Science Education Network) brings news of a conference in Seattle, WA at the Pacific Science Centre, Sept. 8 – 11, 2009. It will feature (from the NISE Net blog),

Members of the NISE Net Program group and faculty and students at the Center for Nanotechnology in Society at Arizona State University are teaming up to demonstrate and discuss potential collaborations between the social science community and the informal science education community at a conference of the Society for the Study of Nanoscience and Emerging Technologies in Seattle in early September.

There’s more at the NISE Net blog here including a link to the conference site. (I gather the Society for the Study of Nanoscience and Emerging Nanotechnologies is in its very early stages of organizing so this is a fairly informal call for registrants.)

The NISE Net nano haiku this month is,


Surface plasmon resonance
Silver looks yellow

by Dr. Katie D. Cadwell of the University of Wisconsin-Madison MRSEC.

Have a nice weekend!

Teaching nano the haptic way; Brownian motion ain’t what we thought; EPA issues final new rules for carbon nanotubes

In keeping with my interest in the multimodal communication of science, I have found a slide show about teaching nanotechnology using haptics here. The technique is intended for the visually impaired but as the authors point out visual contact at the nano scale is impossible. So, everyone, visually impaired or not, makes haptic contact with material at the nano scale with the consequence that the teaching technique is suitable for everybody.

As suggested in my July 27, 2009 blog posting (part 4 of the robots and human enhancement series), developments such as these suggest that the notion of physical impairment may change significantly or disappear.

In a media release, on the Azonano site, detailing new revelations about Brownian motion,  Steve Granick, Founder Professor of Engineering at the University of Illinois, describes how many of us are taught about Brownian motion,

“In high school science classes, students are often assigned the task of using a microscope to watch a particle of dust sitting in a drop of water,” Granick said. “The dust particle seems alive, moving back and forth, never in the same way. The motion of the dust particle is caused by the random ‘kicks’ of surrounding water molecules.”

Granick goes on to describe what he and his researchers have observed,

“Like Einstein, we used to think we could describe Brownian motion with a standard bell-shaped curve,” Granick said. “But now, with the ability to measure very small distances much more precisely than was possible 100 years ago, we have found that we can have extremes much farther than previously imagined.”

Please do take a look at the story on the Azonano site for more about the significance of this discovery.

Nanowerk News has posted a media release from the US Environmental Protection Agency (EPA) about new rules which allow for commercialization of carbon nanotubes under limited conditions.  The EPA document is here and pages 9 (multi-walled carbon nanotubes) and 10 (single-walled nanotubes) are the relevant pages.

Visualizing innovation and the ACS’s second nanotube contest

I’ve found more material on visualizing data, this time the data is about innovation. An article by Cliff Kuang in Fast Company comments on the WAINOVA (World Alliance for Innovation) and its interactive atlas of innovation. From the article,

Bestario, a Spanish infographics firm, designs Web sites that attempt to find new relationships in a teeming mass of data. Sometimes, the results are interesting, as examples, if nothing else, of data porn; other times, it’s merely confounding. Its new project is a great deal easier to explain: The Wainova World Atlas of Innovation attempts to map the world’s major science and business incubators, as well as the professional associations linking them.

Kuang goes on to point out some of the difficulties associated with visualizing data when you get beyond using bar graphs and pie charts. The atlas can be found here on the WAINOVA site. If you’re interested in looking at more data visualization projects, you can check out the infosthetics site mentioned in Kuang’s article.

Rob Annan at the Don’t leave Canada behind blog has picked up on an article in the Financial Post which, based on an American Express survey, states that Canadian business is being very innovative despite the economic downturn. You can read Annan’s comments and get a link to the Financial Post article here. As for my take on it all, I concede that it takes nerve to keep investing in your business when everything is so uncertain but I agree with Annan (if I may take the liberty of rephrasing his comment slightly) there’s no real innovation in the examples  given in the Financial Post article.

The American Chemical Society (ACS) has announced its second nano video contest. From the announcement on Azonano,

In our last video contest “What is Nano?”, you showed us that nano is a way of making things smaller, lighter and more efficient, making it possible to build better machines, solar cells, materials and radios. But another question remains: how exactly is “nano” going to impact both us and the world? We want you to think big about nano and show us how nano will address the challenges we face today.

The contest is being run by ACS Nanotation NanoTube. There’s a cash prize of $500USD and submissions must be made between July 6, 2009 and August 9, 2009.  (Sorry, I kept forgetting to put this up.) You must be a registered user to make a submission but registration is free here. The Nano Song (complete with puppets!) that was making the rounds a few months ago was a video submission for the first contest.

Elsevier has announced a new project, the Article of the Future. The beta site is here. From the announcement on Nanowerk News,

Elsevier, a leading publisher of scientific, technical and medical information products and services, today announces the ‘Article of the Future’ project, an ongoing collaboration with the scientific community to redefine how a scientific article is presented online. The project takes full advantage of online capabilities, allowing readers individualized entry points and routes through content, while exploiting the latest advances in visualization techniques.

Yes, it’s back to visualization and, eventually, multimodal discourse analysis and one of the big questions (for me) how is all this visualizing of data going to affect our knowledge? More tomorrow.

Alberta welcomes a new nanotechnology product and research centre plus some news on a kissing phone

The new facility will be called the Hitachi Electron Microscopy Products Development Centre (HEMiC) at Canada’s National Institute of Nanotechnology (NINT) at the University of Alberta, Edmonton. From the media release (on Azonano),

“Alberta’s strength in nanotechnologies, and the province’s coordinated strategy for nanotechnology made our decision to seek a partnership here easy,” said John Cole, President of Hitachi High-Technologies Canada, Inc. “This initiative engages Hitachi with Alberta’s nanotechnology community at the leading edge of research while contributing to commercial opportunities.”

The Centre will house three new electron microscopes valued at $7 million, including the first-ever Hitachi environmental transmission electron microscope Model H-9500 in operation outside of Japan.

There are many quotes in the media release, surprisingly, none from Dr. Nils Petersen, NINT’s  Director General.

Fast Company is featuring an article by Kit Eaton about phones that won’t require buttons for control (more touch screen-type technology but introducing a new level of innovation). As it turns out, these phones will be coming from Nokia. Kissing the phone as a gesture that you want to contact a loved is just one of the ideas being explored. More here including a Nokia video about the project. The product designers are looking at how people gesture and, depending on your culture, the meaning behind gestures can vary greatly as the Nokia designer notes in the video. Anyway, this type of project relates to my interest in multimodal discourse and my suspicion that we won’t be writing (or for the matter reading) as much as we do now.

Rob Annan over at Don’t leave Canada behind has picked up on my series of last week’s about innovation in Canada, in his posting Canada not simply hewers and drawers.

Science’s exquisite corpse and other interesting science communication developments

The ‘exquisite corpse’ is a game that surrealists started playing in the earlyish part of the 20th century, according to the wikipedia essay here. I first came across the game in a poetry context. I was part of an online poetry organization and someone suggested (as I recall) that we start an exquisite corpse project on our website. Nothing much of came of it but I’ve always found the phrase quite intriguing. The idea is that a group of people play with words or images individually then put the pieces together to construct a final work.

Andrew Maynard’s 2020 Science blog has been featuring an art/science exquisite corpse project by Tim Jones. Billed as an experiment in science engagement, Jones and his colleagues (at the Imperial College) have created videos of two  members of the public, a science communicator, and a scientist talking about a drawing they’ve each created that expresses what they each think is important abou science.  What you’ll see are the interviews, the pictures that the people drew, and an exquisite corpse of science, if you go here.

Tim Jones has now invited more people to participate for the biggest art/science project in history (maybe) to create a bigger exquisite corpse of science. If you’re interested go here to Tim Jones’s site or you can read about it here at 2020 Science.

I came across a way for scientists to publish workflows and experiment plans  at myExperiment.

BBC4 has been conducting an experiment of their own, visualising radio. In this case, it’s a science show that’s cast over the internet. They’ve blogged about the project here.

All of this makes me think back to the interview that Kay O’Halloran (July 3, 6, and 7, 2009 postings) gave me on multimodal discourse analysis and Andrew Maynard’s bubble charts (June 24 and 29, 2009). It’s exciting to explore these new and rediscovered techniques and to think about how we perceive the information being conveyed to us.

One last bit, there’s been an announcement from Lord Drayson, UK’s Science and Innovation Minister and Chair of Ministerial Group on Nanotechnologies that the government is seeking advice for a national nanotechnology strategy. From the announcement on Nanowerks News,

Industry, academia and consumer groups were invited to use a new website to help develop the strategy, building on and consolidating the existing research and consultations that have already taken place. The website will gather views on core issues including research, regulation, innovation and commercialisation, measurement and standards and information as well as on the anticipated impact of nanotechnologies on a wide range of sectors. The aim of the strategy is to describe the actions necessary to ensure that the UK obtains maximum economic, environmental and societal benefit from nanotechnologies while keeping the risks properly managed.

The rest of the announcement is  here and the project website is here.  (NOTE: Consumer groups will have their own website although members of the public are welcome the new website is really intended for academia, industry, and NGOs.)

Happy weekend!

The other side of the multimodal discourse coin

Bill Thompson has an article, Giving life a shape, on BBC News which touches tangentially on approaching the world in a multimodal fashion. He takes a kind of digital approach i.e.when he uses the word technology he actually means digital technology and his examples come from social networking, Second Life, social gaming and other activities mediated through the Internet and computers. From the article,

… because in working through the creative potential of new technologies artists of all types are helping us to find new ways to think about these tools and working out how to integrate them into our wider cultural and commercial practice.

They are helping us to explore the latest chapter in the ongoing conversation between human psychology and the capabilities of modern technology, something which will matter more and more as the network becomes pervasive and digital devices penetrate every area of our lives.

Different modalities (audio files, graphics files, animation (Second Life), and others are referred to indirectly in the course of Thompson’s article, which is why I’ve picked up on it. In  light of the Kay O’Halloran interviews (on this website blog July 3, 6, and 7, 2009) Thompson’s description of how “artists help find us new ways to think about things” reveals the other side of the multimodal discourse coin.

While O’Halloran and her colleagues develop a framework for analyzing and understanding multimodal discourse, it’s artists (I define that word broadly) who enact and explore that discourse through their work.

One quibble, I think Thompson’s definition could be broadened so that technology  includes nanotechnology, biotechnology, synthetic biology and other emerging technologies. Now back to Thompson and a comment that works no matter how you define technology,

One problem in talking about this is that relatively few people understand the underlying technology sufficiently well to be comfortable with it. We have few stories that talk about technology and few workable metaphors or analogies that let us convey complex technological issues in ways that people really grasp.

Metaphors came up in the O’Halloran interview (July 6, 2009 posting) too and I got this in the comments (from inkbat),

I was struck by the point on metaphor. When you come right down to it, isn’t it sad that so many of our concepts are the result of some designer or advertiser or whoever deciding to create some kind of shortcut for us .. which would work if it was just in the one instance but then it takes on a life of its own and suddenly we no longer think of the heart AS IF it is a pump but as though it IS a pump. Or the brain as a computer. …

Unfortunately as inkbat points out, we forget we’ve created a metaphor and we treat it ‘as if it were so’ to results that can be disastrous. Still, I think that creating metaphors and then having to ‘break’ or ‘see through’ them ultimately discarding the old metaphor and developing a new one is part of the human condition.

Back to my nanotech ways tomorrow.

Kay O’Halloran interview on multimodal discourse: Part 3 of 3

Thanks to Kay O’Halloran for kindly giving me this interview and here’s the last part which also includes a bibliography.

3. I notice that you have a project examining PowerPoint in the classroom and in corporate settings which you are conducting for the Australian Research Council. Could you explain a little bit about the project?

The project ‘Towards a Social Theory of Semiotic Technology: Exploring PowerPoint’s Design and its Use Higher Education and Corporate Settings awarded by the Australian Research Council (ARC) (Discovery Grant No. DP09889939) is a collaborative project between Chief Investigator Professor Theo van Leeuwen (Dean for Faculty of Arts and Social Sciences, University of Technology, Sydney), Dr Emilia Djonov (Post-doctoral Fellow, University of Technology, Sydney) and myself. The following description of the project is drawn from our research proposal.

PowerPoint has become the dominant technology for designing and delivering presentations, particularly in education and business settings where success often depends on skills in the use of the application. Powerpoint is the subject of much debate and it creates strong reactions, both positive and negative. It’s either praised for increasing presenters’ confidence and eloquence (e.g. Gold 2002) or it’s condemned for limiting users’ ability to present complex ideas through an over-simplification of information presented in bullet points, linear slide-by-slide formats and illegible graphics (e.g. Tufte 2003).

From the multimodal perspective, Powerpoint is a semiotic technology which has a range of options (i.e. grammar) from which presenters make selections with regards to the linguistic text, images, animations and sounds. There are default themes which the presenter may choose as well. These choices integrate in multimodal presentations which are recontextualised by the speaker during the presentation. Most studies of Powerpoint adopt a different approach, however, by either exploring lecturers’ and students’ perceptions of PowerPoint to support learning, or alternatively they are experimental studies which investigate the effects of PowerPoint versus transparency-supported lectures on learning.

Our project adopts a multimodal approach to (a) conceptualise the grammar of Powerpoint through the study of its systems of meaning; (b) analyse and compare the choices which are made in higher education and corporate settings; and (c) investigate how these choices are contextualised in presentations. In this way, we will explore how the design of PowerPoint supports or hinders the achievement of the various goals of the presenters. At the moment, there are no studies which investigate differences in the use of Powerpoint across educational and corporate settings, and furthermore, there is no evidence for arguments that PowerPoint cannot support the representation of knowledge in technical disciplines such as engineering (Tufte, 2003) or the rich narrative and interpretative skills required for social science disciplines (Adams, 2006), nor is there evidence that PowerPoint has introduced corporate rhetoric into educational practices (Turkle, 2004). In addition, the study will provide guidelines for evaluating and improving the design and use of PowerPoint and other similar presentation software.


Adams, C. (2006). PowerPoint, habits of mind, and classroom culture. Journal of Curriculum Studies, 38(4), 389 – 411.

Gold, R. (2002). Reading PowerPoint. In N. J. Allen (Ed.), Working with words and images: New steps in an old dance. (pp. 256-270). Westport, Connecticut: Ablex.

Tufte, E. R. (2003). The cognitive style of PowerPoint (2nd edition). Cheshire, Connecticut: Graphics Press.

Turkle, S. (2004). The fellowship of the microchip: global technologies as evocative objects. In M. Suárez-Orozco & D.B. Qin-Hilliard (Eds.), Globalization: Culture and Education in the New Millennium (pp. 97-113). Berkeley, CA: University of California Press.

Kay O’Halloran interview on multimodal discourse: Part 2 of 3

Before going on to the second part of her interview, here’s a little more about Kay O’Halloran. She has a Ph.D. in Communication Studies from Murdoch University (Australia), a B.Sc. in Mathematics and a Dip. Ed. and B.Ed. (First Class Honours) from the University of Western Australia.

The Multimodal Analysis Lab of which she is the Director brings together researchers from engineering, the performing arts, medicine, computer science, arts and social sciences, architecture, and science working together in an interdisciplinary environment. (This is the first instance where I’ve seen the word interdisciplinary and can wholeheartedly agree with its use. As I have found, interdisciplinary can mean that an organic chemist is having to collaborate with an inorganic chemist or an historian is working with an anthropologist. I understand that there are leaps between, for example, history and anthropology but by comparison with engineering and the performing arts, the leap just isn’t that big.)

There’s more on Kay O’Halloran’s page here and more on the Multimodal Analysis Lab here.

2. Could you describe the research  questions, agendas and directions that are most compelling to you at this  time?

Multimodal research involves new questions and problems such as:

– What are the functionalities of the resources (e.g. language versus image)?

– How do choices combine to make meaning in artefacts and events?

– What types of reconstruals take place within and across semiotic artefacts and events and what type of metaphors consequently arise?

– How is digital meaning expanding our meaning-making potential?

The most compelling agendas and directions in multimodal research include developing new approaches to annotating, analysing, modeling, and interpreting semiotic patterns using digital media technologies, particularly in dynamic contexts (e.g. videos, film, website browsing, online learning materials). The development of new practices for multimodal analysis (e.g. multimodal corpus approaches) means we can investigate social cultural patterns and trends and the nature of knowledge and contemporary life in the age of digital media, together with its limitations. Surely new media offers us the potential for new research paradigms and making new types of meanings which will lead us to new ways of thinking about the world. Also, multimodal approaches offer the promise of new paradigms for educational research where classroom and pedagogical practices and disciplinary knowledge can be investigated in their entirety. Multimodal research opens up a new exciting world, one which is being eagerly embraced by academic researchers and postgraduate students as the way forward (in my experience at least).