Olfaction and its combination with visual stimuli in the creation of interactive and immersive environments

【下記の日本語訳をご参照ください】

Lead Researchers and authors: Prof Takamichi Nakamoto, Saya Onai, IIR Laboratory for Future Interdisciplinary Research of Science and Technology, School of Engineering, Tokyo Institute of Technology; Nathan Cohen, Tokyo Institute of Technology WRHI Visiting Professor, Central Saint Martins, University of the Arts London.

ARTICLE’S FULL TITLE: Olfaction and its combination with visual stimuli in the creation of interactive and immersive environments, with the potential to enhance personal engagement and well-being – a transdisciplinary collaborative research project integrating science, technology and art.

Since March 2020 we have been working on an olfactory project investigating how smell, in combination with visual stimuli, influences our impression of an environment and how this could impact Virtual Reality (VR) and Augmented Reality (AR) users when creating immersive, interactive scenes.

Smell is a complex medium to work with, posing challenges for developers, technologists and creatives in its identification, application and handling. Invisible yet, to varying degrees, influential in how we interpret our environment, it has been a source of fascination and inspiration for centuries. Samples of aromatic substances are to be found in the graves of early Egyptians and, in refined forms, it has been a rare and sought after commodity across cultures through to the present day.

Smell, or more particularly the odours which form it, can be distributed artificially in different ways. Perfume is worn directly on the body or clothing. Aromatic oil or alcohol based solutions can be sniffed from a simple container or using smelling sticks that absorb the sample. Moved under the nose the aromatic compounds within the odour samples are released into the air and we absorb them through our nasal passages where smell receptors translate the experience to the brain. This requires a simple haptic approach to encountering the smell sample. Alternatives include burning incense, warming essential oils, and the use of aerosol sprays and gel dispensers to disseminate odours more widely within a space.

A high-speed solenoid valve open/close olfactory display setup in the  Nakamoto Laboratory (Credit: authors).
Diagram illustrating use of the high-speed solenoid valve open/close olfactory display set up in the Nakamoto Laboratory. (Credit: authors).

However, to offer a range of odours in a way that is more intimate and corresponds to other stimuli within a controlled and time based environment requires a different approach. Over recent years Takamichi Nakamoto has developed, with his team in the Nakamoto Laboratory, a technical and computerised approach to delivering odour samples, either linked to a headset or in close proximity, enabling their combination with visual and auditory media in immersive digital environments. Many technical difficulties had to be overcome in this construction including how to enable several odour samples to be delivered through one device without their contaminating each other, and how these odours could be linked to different aspects of the immersive environment in which they can be encountered over varying durations of time. The device currently being used for our experiments in the creation of interactive virtual spacio-olfactory games is an high-speed solenoid valve open/close olfactory display.

Scentscape (2019, Nathan Cohen, Reiko Kubota) an interactive olfactory artwork with digital display (Credit: authors).

One area of investigation we are exploring is how smell can encourage memory and enhance narrative association. An earlier example of this may be seen in the boxed artwork Scentscape (2019, Nathan Cohen, Reiko Kubota) where odour samples were presented in small glass screw capped containers which, when handled, triggered sequences of still images on a video screen corresponding to particular places with which the odours are associated. Other objects relating to memories of these places could also be placed within the box, the intention being that the user could combine their own imagery, odour samples and objects to create their own personalised memory box.

The olfactory displays developed by Takamichi Nakamoto offer a more technical and differently immersive approach. In this current research we will be exploring ways in which the combination of imagery and olfaction can create an enhanced narrative experience for the user through the development of scenes that complement and are complemented by odours. This will also take the form of an animated interactive environment that users engage with.

As Saya Onai states in a co-published paper ‘Significant research has already been done in relation to memory and scene recall that can be induced by scent (odours), but information transmission and scene recall by scent alone has not yet been realised effectively. It has been difficult to achieve a common perception of a particular scene from a certain odour due to the influence of prior experience and differences in individual perceptions.’ * In this research, we are seeking ways to enhance scene recollection by combining olfactory, visual, and possibly auditory, stimuli within an immersive environment.

Pair of images displayed sequentially with a 10 second blank screen and odour sample between the first and second versions of the image (Credit: authors).

To better understand the links between smell and memory recall a supervised experiment was conceived by the group working on this research and conducted in the Nakamoto Laboratory by Saya Onai in December 2020. 18 volunteers were divided into 2 separate groups.† Group 1 were presented with the first of 2 related images on a monitor screen, the screen then went blank and an odour related to the image was dispensed through the linked headset for 7 seconds with an overall pause of 10 seconds. Following this the screen displayed the second related image. There were a sequence of 11 sets of visual image pairs with corresponding odours, referred to as ‘scenes’, that all the volunteers experienced consecutively. After viewing all 11 scenes a multiple choice questionnaire was presented asking which odour is the correct one in relation to each scene. For group 1 the questionnaire presented sets of 4 visual images per odour to select from, inviting the participant to identify which image correctly corresponds to the odour for each scene.

Group 1 multiple choice selection for an test odour 
Group 2 multiple choice selection for an test odour (Credit: authors).

Group 2 separately experienced the same test, only the multiple choice options for selecting the correct odour were text based only and not images.

Initial analysis of the experiment’s results reveal mixed rates of identification, with some odour – image associations stronger than others. We will be exploring this further to identify which odour image combinations are more readily identifiable, together with investigating the temporal nature of odour and odour combination in the creation of animated smell scenes that induce narrative association. We will also be considering the aesthetic aspects of the game design required to engage users and enhance their experience of interaction. This will form the basis for the research during April 2021 – March 2022, with the intention to produce a working prototype for display at Siggraph Asia in December 2021.

*香りによる情景想起の基礎的研究  Fundamental Study of Association of Scenes with Scents Nakamoto, T., Onai, S., Iseki, M., Cohen, N. (2021) IEICE General conference

† This was an initial experiment conducted under Covid-19 pandemic restrictions which meant numbers of participants were restricted.

S T A D H I – Science & Technology + Art & Design Hybrid Innovation

This research is supported by the Tokyo Tech World Research Hub Initiative (WRHI), Program of the Institute of Innovative Research (IIR), Tokyo Institute of Technology.

© 2021 Photographs and intellectual content – all rights reserved by the authors.


インタラクティブで没入型の環境の作成における嗅覚と視覚刺激の組み合わせ

2020年3月以来、私たちは嗅覚プロジェクトに取り組んでおり、視覚刺激と組み合わせて、嗅覚が環境の印象にどのように影響し、没入型のインタラクティブなシーンを作成するときに仮想現実(VR)と拡張現実(AR)のユーザーにどのように影響するかを調査しています。

香りは複雑なメディアであり、その識別、応用、処理において、開発者、技術者、クリエイターに課題をもたらします。香りは、香水、アロマオイル、お香、またはエアゾールスプレーやジェルなど、さまざまな方法で人工的に分散させることができます。ただし、制御された時間・環境内で、より緊密で他の刺激に対応する方法でさまざまな香りを提供するには、別のアプローチが必要です。近年、中本高道教授は中本研究所のチームとともに、ヘッドセットにリンクされた、または近接した香りサンプルを拡散するための技術的かつコンピュータ化されたアプローチを開発し、没入型デジタル環境で視覚および聴覚メディアとの組み合わせを可能にしました。

このインタラクティブ嗅覚ディスプレイは、香りがどのように記憶を促し、物語の関連性を高めることができるかを探求するために使用されました。実験は、2020年12月に中本研究室のメンバーによって18人のボランティアに対して実施されました。初期分析結果はさまざまで、強い関連をしめす香りと画像があることが明らかになっています。2021年4月から2022年3月までのさらなる研究は、2021年12月のSIG-GRAPH Asiaで展示するための実用的なプロトタイプを作成することを目指します。

Jan 7th – Feb 5th 2021: “Technology and Product in Context” course by Dr Betti Marenko

【下記の日本語訳をご参照ください】

This article introduces the “Technology and Product in Context” course by Dr Betti Marenko held in the 2020/21 autumn term for GSEC, the Global Scientists and Engineers Course. The classes included a series of 6 lectures and a workshop with the students on the third week. Design theorist Dr Marenko is WRHI Specially Appointed Professor at Tokyo Tech and Reader in Design and Techno-Digital Futures at Central Saint Martins (CSM), University of the Arts London, UK.

What does it mean to be human in a world designed to be smart? How well can we get along with machines that are unpredictable and inscrutable? How do we think about ‘hybrid futures’? These were some of the questions raised in the Technology and Product in Context lecture series by design theorist, academic and educator Dr Betti Marenko. The course – ended in February 2021 – was attended by about 15 students from various branches of engineering, social and life sciences, who share an interest in the future of technology, philosophical issues around design and making, design theory and science communication. The sessions were conducted entirely in English and online, using Zoom, PowerPoint and Miro boards. This article follows the structure of the course and outlines some of the key topics, references and examples discussed each week.

Dr Marenko’s publications focus on Design, Philosophy and Digital Futures (Credit: Marenko, 2021)

Dr Marenko has written extensively about technological futures and the role of design in the Post-Anthropocene, a future geological era that does not presuppose the presence of humans on Earth. Her “tools for thinking in the Post-Anthropocene” lie at the intersection of design, philosophy and technology. In her view, the development of future technologies needs to engage with complexity, and design can benefit from a shift “from problem solving to problem finding”. The first lecture explored the question of hybrid futures from a historical perspective, tracing the origins of the human-machine encounter back to the automata that emerged in Europe in the Renaissance period. 

Dr Marenko discussed the history of automata, including The boy writer by Jaquet Droz (1770s) (Credit: unknown; slide by Marenko, 2021)

Prompted by questions on their views on “technology” and “context” – two keywords in the course title – the students proposed ideas such as “the unknown”, “a more harmonious and convenient society”, abstract and changeable ideas of “hope”, “cooperation” and “unpredictability”. For Marenko, the context of design is not simply a background to a project, but “mutually constituted ecologies” of interactions that retain an ability to ask better questions. She highlights the undivided nature of theory and practice through the image of the Moebius strip, a continuous form that is both inside and outside. Similarly, the contrast between what is human and non-human, or post-human, is dissolved in philosopher Giorgio Agamben’s definition of the human as “the machine that produces the notion of the human”. For design theorists Colomina and Wigley, “being human means being able to design”, and design is about changing the world. For Marenko, the boundaries between human and non-human need constant reassessing, and technology is what we use to address this instability. The lecture included numerous examples of artworks and writings that illustrate or embody her philosophical narratives.

Dr Marenko’s slides included striking images from popular culture, advertising and art projects (Credits: Apple Inc., 2015, left; Andy Taylor, 2012, right)

The course continued with an exploration of the concept of future through three keywords: expectation, imagination and anticipation. Anticipation is the capacity to imagine the non-existent future in the present, leading to the idea of ‘future proofing’. However, as Marenko puts it, “the conditions for change do change”. The simplistic assumption that future proofing is possible, let alone desirable, underpins some of the failed philosophies of modern design: planned obsolescence (the design of failure to stimulate future sales), solutionism (the idea that design is all about finding solutions to existing problems) and linear progress (the vision of a world constantly improving thanks to science and technology). Design brings better solutions, but better for whom? And, better for what? Dr Marenko proposes a view based on French philosopher Gilles Deleuze’s idea that building the future is not about predicting but “being attentive to the unknown knocking at the door”.

This set the basis for a workshop conducted on Week 3 using a Miro board and a set of cards developed by Dr Marenko and colleagues at CSM. Working in four small groups, the students were asked to propose a scenario for 2050 that addressed one of four ‘pills’ provided: animism, counterfactuals, decolonization and post-Anthropocene. These were read through selected ‘affective mode cards’, which summarised the attitude performed in the discussion, i.e. the anxious, the optimist, the resilient, the survivalist, the nihilist and the Zen master. Guided by this participatory strategy, the groups offered their visions of the future in short presentations, anticipating a few aspects that would be analysed in the subsequent weeks.

A screenshot of the Miro board used by Marenko and her students (Credits: Marenko, 2021; Miro.com, 2021)

The course went on to question “received notions of technology” as having to do with the latest innovations, and stressed the continuity with historical developments. The term and notion of android, for instance, go back to Pierre Jaquet Droz’s writing automaton from the 1770s and its use of the power of technology to “enchant” its audience. Similarly, the term automaton today to some extent maintains the original meaning (from Diderot’s Encyclopaedia of 1751) of a machine that can move by itself, following a sequence of operations or responding to encoded instructions. The conversation continued on the topic of “digital enchantment” (based on texts by anthropologist Alfred Gell) and the relationship of technology with magic. The lecture material was grounded in historical and philosophical developments but made more accessible by recurrent references to well-known techno-gadgets, and visual and popular culture: from iPhones and Blade Runner, to Amazon and the latest Android firmware.

Following the steps of French philosopher F. Guattari, Dr Marenko discussed digital uncertainty in contemporary society, one that is seeing “a fundamental repositioning of human beings in relation to both their machinic and natural environments”. Information and computation are not simply mediating our lives, they constitute a large part of what we do every day. But the outcomes of these digital encounters are not fully predicted or programmed, hence the emergence of uncertainty. Examples include the algorithmic automation that drives financial services and much of our interaction online. These considerations are driving AI innovations and constitute a new “technological unconsciousness” that contrasts with 20th century views of technology. Marenko therefore asks, “Can AI get smarter by becoming more uncertain?”.

Dr Marenko reflected on the impact of planetary computation on contemporary and future societies (Credits: unknown, slide by Marenko, 2021)

Through old and new theories of cybernetics, uncertainty was explored both as an accident and as a glitch. A fundamental concept is von Foerster’s “non-trivial machines”, deterministic systems producing unpredictable outcomes. Digital models, for example, can work by iterations and design strategies can operate by a fast succession of trial and error, as described by historian and critic Mario Carpo (2013). This poses interesting questions on what constitutes digital craft and how it relates to the idea of “risk”, an essential aspect of handmade production.

The next lecture started by pointing out the paradox of innovation: any new products must retain familiarity, so people can comprehend and recognise them. For example, the first car in the 1870s was named “the horseless carriage” and very much looked like one. Design theorists D. Norman and R. Verganti discussed this issue in their 2014 paper on “incremental and radical innovation”, a critique of the same human-centred design (UCD) that Normal had helped developing in the 1980s and 90s. For them, UCD can provide incremental innovation to “users” but only focuses on things people already know. For Marenko, instead, design can assume a more rhizomatic nature and embrace its role as interface between the making of objects and that of concepts. According to this view, the design process is simultaneously thing-making, concept-making and future-building.

The discussion followed on the concept of future crafting and the role of fiction in producing reality. This was linked to other design strategies and methods of future crafting, such as cultural probes (embracing risk and uncertainty) and defamiliarization (embracing strangeness).

The horseless carriage, an early model of car (right), still closely resembled a horse-powered carriage (left). (Credit: unknown; slide by Marenko, 2021)

The series concluded with Dr Marenko’s original reflections on technology and animism. As surprising as it may sound, we already live in a world that has seen a shift from “talking about things to talking with things” (her italics). If from a technological perspective we are seeing the rise of the ‘internet of things’, theoretical developments also attempt to question outdated (Western) notions of animism for our new age. Following Bruno Latour’s thinking, the focus is not just on drawing parallels between consumerist and religious practices, but to rethink about the “agency” of objects as a relational property. Philosopher Jane Bennett has also discussed “thing-power”, the curious ability of inanimate things to produce effects. Referencing multiple recent studies on the subject, Dr Marenko discussed the role of animism in creativity and design. She provides a definition of “animistic design” as one that operates in a post-user (or post-UCD) scenario and maintains “mental elbowroom” to generate new, non-linear forms of knowledge. But why is uncertainty so important? Because it establishes perceptions, it shows what might happen and focuses on ranges of possibility, including those that were not thought of. It depends on elements that are not fully controllable, are random and not fully predicted. Uncertainty has to do with creativity.

Through her often surprising and always inspiring lectures, Dr Marenko opens new views on technology and its deployment in crafting humanity’s future. Her arguments on science and technology stand out as seamlessly built on a diverse range of references across disparate disciplines. The discussion was made more accurate and relevant by drawing from philosophy and design theory, but also science fiction, critical design, art practice, advertising and popular culture. The hope is that students’ accepted views of technology could be shaken by all this unorthodox transdisciplinarity, leading them to wider-open reflection, inspiration and future-shaping innovation.

Dr Betti Marenko’s forthcoming book, Designing Smart Objects in Everyday Life. Intelligences. Agencies. Ecologies (co-edited with Marco Rozendaal and Will Odom), is a collection of essays developing a new research framework for interaction design. For more information on this and other projects, visit bettimarenko.org


2021年1月7日から2月5日:ベティ・マレンコ博士による「物語のあるものつくり」

スマートになるように設計された世界で人間でいるとはどういう意味でしょうか?予測不可能で不可解なマシンとどれだけうまくやっていけるでしょうか?「ハイブリッド未来」についてどう思いますか?これらは、デザイン理論家、学者、教育者であるベティ・マレンコ博士(http://bettimarenko.org/)による「物語のあるものつくり」講義シリーズで提起された質問の一部でした。

コースは2020/21後期に開催されました。このコースは、Zoom、PowerPoint、Miroボードを使用して英語・オンラインで行われた6回の講義と3週目に行われる学生によるワークショップのシリーズでした。

一連の講義は、以下のトピックに基づいて展開されました。

  1. Post-Anthropoceneを推測するための理論的道具
  2. 将来の期待
  3. 「未来の哲学のピル」ワークショップ
  4. 人間と機械の出会い:アンドロイドからアルゴリズムへ
  5. 惑星計算におけるデジタルの不確実性
  6. スペキュレイティブ、フィクション、未来のクラフトをデザインする
  7. アニミズム2.0

マレンコ博士の驚きに溢れ、常に刺激的な講義を通じて、テクノロジーと人類の未来を創造する上でのテクノロジーの展開についての新しい見解が開かれます。学生が受け入れているテクノロジーに対する見方が、この非正統的な学際的研究によって揺さぶられ、より広い省察、インスピレーション、未来を形作るイノベーションにつながることを願っています。