Conversations is an experimental Distributed Multi-User Virtual Environment forming the central focus of an Australian Research Council funded study investigating the reformulation of narrative within digital cinema. It offers viewers an immersive multi-modal interactive narrative experience dealing with the events leading up to the escape, recapture, trial and hanging of Ronald Ryan at Pentridge Prison, Melbourne in 1967, the last person publicly executed in Australia. It allows viewers “to inhabit the landscape of the escape, to experience the confusion of the crime scene first hand; then it enables us to encounter the key protagonists of the subsequent trial. Instead of being presented with ‘facts’, viewers are asked to negotiate the historical landscape themselves, to draw conclusions from their encounter with both the scene of the disputed events and the opinions of the characters who dominated their aftermath”.* It has been developed in the form of two demonstrators. The first for the Powerhouse Museum, Sydney where it was premiered in November 2004 and the second at iCinema’s Scientia Laboratory where it is available for presentation as part of a portfolio of experimental works. The project has been developed in collaboration with the University of Technology, Sydney, the University of Sydney and the ZKM Centre for Art and Media, Germany.

 

* McQuire, S. and Papastergiadis, N. (2006). Conversations: The Parallax Effect. iCinema: Sydney.

3D axonometric view

Background

The rise of digital media has led to a decline in the use of traditional single-layered narrative and the corresponding loss of a major instrument of cinematic representation. This study investigates the reformulation of narrative within digital cinema through the integration of three models of interactivity so as to produce a new emergent digital narrative form. The study tests the proposition predicted in the cinematic theory of Gilles Deleuze and the systems theory of Manuel DeLanda that narrative, when generated as a complex of digitally interactive forms, provides the opportunity to recapture the significance of narrative within digital cinema, through its enactment within a multi-layered, emergent virtual space.

Objectives

  • Effect the synthesis of three prior experimental interactive narrative models: the multi-branched, the navigable and the algorithmic.
  • Demonstrate the expressive structure of a digital narrative as reformulated within the synthesis of the three prior models through its experimental application in the Distributed Multi-User Virtual Environment entitled Conversations.
  • Test through critical interpretation the representational character of these new forms of narrative emergent within a distributed multi-user virtual environment as a type of social dialogue emergent within a digital space.

Demonstrators

Conversations is composed of three electronically linked installations that are positioned at remote locations within the museum. The visitor at each installations wears a head mounted display (HMD) whose twin LCD screens present an immersive stereoscopic display. A gyroscopic tracking device, (Intersense) attached to each HMD, monitors the movements of the viewer’s head, so that it can generate images which correspond to the direction and angle of view. Microphones and headphones attached to the HMD enable the viewers at the three remote locations to hear and speak to each other. The microphones are also linked to a software based voice recognition system that enables the viewer to also interact with the environment by means of spoken commands. Each of the installations is equipped with a PC and high performance real time computer graphics hardware (nVidia). Custom application graphics software enables the construction of a shared distributed virtual environment in which the three viewers are simultaneously immersed and can individually navigate and interact with the features of this hybrid space of representation.

Installation at the Powerhouse Museum, Head Mounted Display

Its main attributes are:

A contemporary stereoscopic panoramic photograph of the Pentridge Prison and its environs. The viewers, wearing the HMDS, find themselves located within this scene and can freely look about it in any direction. The viewers also move between two different renditions of this scene – one is a daytime setting where the sequence of Ryan’s escape from prison is enacted, and the other is a nighttime setting inhabited by the ghosts of the protagonists in Ryan’s story.

Actors, who are the central characters in Ryan’s escape and subsequent murder trial, are three dimensionally placed within this photographic setting. These narrative agents are video-graphic entities that have been previously recorded in a Chromakey studio. In the Pentridge daytime setting, these narrative agents have been scripted and composited into the virtual environment and present the viewers with a linear narrative experience. In the nighttime setting, where Pentridge has become a ghost world, these ethereal narrative agents float about and individually respond to each of the three visitors, both when they are purposely looked at, as detected by the tracker on the HMD, and when they are talked to, as interpreted by the voice recognition system.

Computer generated avatars of the three visitors, who become aware of one another, are positioned within the Pentride setting. Each is identified by a different color, with which each corresponding installation is also coded. The real time behavior of these avatars depicts the viewers’ navigation of the virtual environment, their interactions with the narrative agents, and with one other. The trackers on the visitors’ heads control the corresponding behavior of their avatars’ heads, and when voice detection is activated, these avatars are animated by typical conversational gestures.

Panoramic view, Pentridge in Daytime
Panoramic view, Ghost world – Pentridge at Nighttime

In this configuration, an innovative new component has been added to the three HMD installations. It is a large immersive hemispherical projection, set within a three-meter diameter half dome, mounted vertically in front of the viewer. Whereas each of the HMD viewpoints expresses the natural point of view of a visitor within the shared virtual environment, the hemispherical projection presents a global view of this environment, where one can simultaneously see all the multimedia assets – the Pentridge prison setting, the videographic narrative agents and the three visitor avatars. Using simple trackball control the user can rotate this scene and move back and forth between distant to close-up viewing. This immersive visualisation system provides a highly effective technique for rendering a complete audio-visual observatory of the Conversations multi-user, multi-modal, virtual environment with all of its narrative and interactive components, and is a significant advance on the multi-screen monitor setups that have been used until now to elucidate the workings of such complex mixed reality situations.

Research Outcomes

  • This project realised a number of groundbreaking technological advances in the field of Virtual Reality. In particular, the project saw the development of a high-resolution stereo spherical video production and playback system and a high fidelity audio system capable of spatially rendering an unlimited number of sound sources. Both of these advances were world firsts. In addition, considerable effort was made to ensure that the maximum amount of the software developed was independent and so applicable to future research. The audio stream of this research was undertaken in collaboration with the Computer Audio Research Laboratory (CARLab) at the University of Sydney. This research lead to the identification that acoustic control of distance in the near-field (locations within arm’s reach) can provide a substantially improved audio user interface. As part of this research CARLab developed a method for mathematically computing near-field acoustic filter functions from a given set of far-field acoustic filter functions.

  • A Ph.D scholarship was provided to Keir Smith for the development of a number of software components. Research support was provided to APA Ph.D candidate Greg Ferris to be involved in development of the stereo spherical video production.

  • Re-entering History

    Having been the subject of numerous newspaper articles, books and public debates, the events leading up to Ryan’s hanging remain relatively well-known in Australia. Conversations takes us into this terrain differently. Firstly, it allows us to inhabit the landscape of the escape, to experience the confusion of the crime scene first hand; then it enables us to encounter the key protagonists of the subsequent trial. Conversations offers history reconstructed as a type of narrative. But it is a narrative history made up less of a linear account of events pinned to their authoritative meaning than of a loose web of overlapping actions from which interpretive guidelines are largely withdrawn. Instead of being presented with ‘facts’, viewers are asked to negotiate the historical landscape themselves, to draw conclusions from their encounter with both the scene of the disputed events and the opinions of the characters who dominated their aftermath.

    On one level, the work raises the problem of seeing what ‘actually happened’. The key scene in Conversations consists of an interactive 3D environment of the surrounds of Pentridge, comprising 180 still images, each 2 degrees wide by one hundred and eighty degrees vertical, taken with a masked fish-eye lens. These separate slices are stitched into a 360 degree panorama forming a seamless environment onto which moving elements such as people and vehicles, filmed in a green box studio shoot, are composited. Users enter the space using head-mounted displays and navigate it via individual head-tracking systems. While the user’s position in space is fixed, the direction of their look is not; they can choose to pay attention to peripheral phenomena such as the reactions of various bystanders, or they can focus fully on the central events: The actions of the guard up on the outer wall; the paths taken by the two escapees; the arrival of the vehicle they eventually commandeer; and, of course, the movements of the guard who pursues them outside the prison and who is eventually shot. Sound cues are important for establishing spatial depth and orienting users to the action. But being positioned as eyewitnesses to the crime does not resolve all questions. Critically, the origin of the fatal shot is left ambiguous. No matter how often users examine the scene, the direction from which the bullet arrives remains unclear.

    Users seeking clarification are thus directed to the second level of the work, which consists of a series of conversations or rather monologues – delivered by a variety of ghostly characters from the past: the Chaplain who consecrated Ryan’s soul before his hanging; Ryan’s Mother who pleaded with the Premier for her son’s life and finally for his body; the Judge who pronounced the death sentence; the Premier who refused to commute it; the Prosecutor who convinced the jurors of Ryan’s guilt beyond reasonable doubt; the Defence lawyer who sought to raise doubt surrounding the fatal shot; the warder who was shot; and the two escapees themselves, Walker and Ryan, who deny responsibility for the shooting. This level is navigated via voice cues, allowing users to choose whether or not to meet particular characters. All the ‘ghosts’ speak in snippets reminiscent of news grabs, offering insights into both the evidence presented in the case but also the different motivations of those fighting it. In the intersection of detail, observation, reasoning and prejudice, a picture is gradually built up; not the stable image of supposedly objective history but a mosaic which users must evaluate and reconcile for themselves. Competing voices are not organised into a harmonious consensus of guilt or innocence, but rather do battle. What ensues is a cacophony of voices with doubt ricocheting off each other and conflicting perspectives that jostle for the place of truth.

    The task of reconciliation is assisted by the waiting room to the installation which includes extensive historical documentation of the events, and also a large screen projection of the events seen from the perspective of one user’s head set. Assessing what one has witnessed is also the subject of the third level of interaction, which enables conversations to take place between different users. Up to three users can inhabit the space simultaneously, appearing in the landscape as avatars. Since users can also talk in real time to each other, discrepancies and differing accounts of the events that each witnesses can become the subject of discussion and contestation.

    • Scott McQuire and Nikos Papastergiadis. 2006. Conversations: The Parallex Effect. iCinema: Sydney.
  • On occasion, cinema has exploited the overlap of character and camera point of view to construct narratives where the events are refracted through the perspectives of different protagonists. Kurosawa’s Rashamon (1950) is perhaps the best known example, where the recollections of events by different characters challenge the plausibility of earlier versions of the same events. Bernardo Bertolucci’s first film La Commare Secca (The Grim Reaper, 1962) famously begins with a shot of a leaf drifting down from a bridge to alight on a dead body; the film subsequently retraces the course of events leading to this outcome. The different characters we meet during the police investigations into the crime offer versions of events which differ materially. Gradually we realise that not all can be telling the truth.

    Conversations begins from a similar premise, which heightens our sensitivity to the constructed nature of the past, but with the difference that the order of the user’s encounter is not decided in advance. The need for each viewer of Kurosawa’s and Bertolucci’s films to evaluate the events portrayed and interpret the veracity of different accounts is here matched by the demand that each user negotiate the scene of the crime and the sequence in which the various stories stemming from it are told. If it has become more common to understand History as a function of point of view, the technological embodiment of this sensibility in a database narrative such as Conversations should be read less as the proposal that a single event can be witnessed from many different sides, but rather as alerting us to a more radical diffraction of the event. It suggests that the apparent unity of history is only ever imposed retrospectively, a fictive act legislated by power. It remains a prismatic representation where the lines not only fail to unify but insist upon their distinctive and decisive trajectories. Perspectives split like the crossfire of parallax bullet lines. The corollary of such an understanding is that presenting history as a set of elements open to re-interpretation is a political undertaking in the broadest sense.

  • Crime scenarios seem peculiarly suited to database narratives.

    The detective story was itself a distinctive invention of 19th century urban culture. Influential figures such as Poe’s Auguste Dupin and Conan-Doyle’s Sherlock Holmes exemplify the mobilisation of reason with the aim of deciphering the secret life of the city. Recourse to ‘scientific’ logic to map the new forms of urban space and identity reveals the extent to which the older urban milieu of the quartier was becoming a nostalgic point of reference. In the place of the fixity of class-based ‘character’, abstract systems were needed to comprehend the more fluid exigencies of urban life. As Joan Copjec points out: “The origins of detective fiction coincide […] with what Ian Hacking has termed ‘the avalanche of [printed] numbers’. Unlike the priest, or classical ideas of determinism, the scientific detective deals less in truth than probability. Modern complexities lend themselves less to absolute certainty than to the relative certainty of the balance of probabilities.”

    In the 21st century cities are becoming hybrid compositions of architectural and media space. If detective fiction first offered a means of making sense of the social anonymity and abstract relations of the modern city, database narrative offers the detective scenario in a reflexive mode and corresponds to the hybrid forms of the material and the immaterial in contemporary culture. Conversations enables one to stand back from the conventions of the crime scene and pull them apart, to evaluate them even while enjoying the narrative pulsion of the quest for truth – a quest which is endlessly deferred, and endlessly repeated.

ARC InvestigatorsDennis Del FaveroRoss GibsonIan Howard, Jeffrey Shaw
Programmers: Matthew McGinity, Keir Smith, Joachim Tesch, Teewon Tan, Craig Jin, Andre Van Schaik, Dennis Lin, Alan Kan
Project Funding: ARC DP0209550

2004-06

Distributed Multi-User Virtual Environment

  • Scientia, UNSW, Sydney, 2006
  • Powerhouse Museum, Sydney, 2004/05

Directors: Dennis Del Favero, Ross Gibson, Ian Howard, Jeffrey Shaw

Writer and Video Director: Ross Gibson

Video Editor/Compositor/ Cinematographer: Greg Ferris

Project Manager: Kate Dennis

Production Manager: Damian Leonard

Production and Technical Co-Ordinator: Volker Kuchelmeister

Production Assistant: Allison Bisshop

Software Developer/Co-Ordinator: Matt McGinity

Software Developer: Keir Smith, Joachim Tesch

Audio Engine Software Developer: Teewon Tan, Craig Jin, Andre Van Schaik, Dennis Lin, Alan Kan

The Computing and Audio Research Laboratory: CARlab, University of Sydney

Character Animator/3D Modeller: Guido Wolters, Steve Weymouth, Andy Homan

Sound Designer: Robert Hindley

Panoramic Still Photographer: Peter Murphy

Technical Director/Sound: Max Harrison

Props and Costumes Designer: Imogen Ross

Hair and Makeup Stylist: Sue Carroll

Line Producer: Sue Midgley

Camera Operators: Katharina Klodt, Jodie Smith

NIDA Team: Amanda Morris, James Tait, Lisa Lipson, Jeavons

Ronald Ryan: Tim Burns

Peter Walker: Scott Williams

Ryan’s Mother: Mercia Deane-Johns

Prison Chaplain: Craig Coventry

Premier Bolte: Tony Girdler

Justice Starke: Ted Bennett

Leader Of The Prosecution: David Robbins

Leader Of The Defence: Stephen Holt

State Hangman: Damien Parker

Prison Officer: Kristen Throw

George Hodgson’s Ghost: Simon Mills

Anti-Hanging Campaigner: Kath Parle

Extras: Daniel Heckenburg, Casey Whitelaw, Nick Mariette

 

The iCinema Centre gratefully acknowledges the financial assistance of:

The Australian Research Council;

University of New South Wales – College of Fine Arts and Faculty of Engineering, School of Computer Science and Engineering;

National Institute of Dramatic Art;

CARlab at the University of Sydney;

ZKM, Centre for Art and Media, Karlsruhe, Germany;

the Powerhouse Museum.

 

It acknowledges the support of:

dedece;

Xenon Systems;

Harbour Street Management;

3 Arts Make-Up Centre;

Penny Hagan;

Anthony Speed;

Jackie Farkis;

Milvia Harder;

Chameleon Touring Systems.

 

Funded by:

Australian Research Council Discovery Grant: The reformulation of narrative within digital cinema as the integration of three models of interactivity.