Interactive Visualisation
If a visualisation is to be perceived as realistic, is it increasingly required to respond to the viewer’s actions? Is static visualisation becoming old hat? Has interactivity become a necessary part of engendering perception, action, and emotion in our response to a visualisation? And what do we mean by interactivity?
Of course, interactivity may take various forms. For instance, it may entail navigation facilities: an ability to change the viewpoint, to move through the visualisation. It may also entail manipulation facilities: the ability to modify the visualisation, to move and re-organise elements. But what are we actually interacting with?
Evidently we see a visual representation or simulation of an environment so we are interacting with that simulation. But this implies a single interface, between us as the physical embodied viewer/actor and the visualisation. Indeed, Virtual Reality is characterised as the transparent invisible interface which is all-encompassing and three-dimensional; the user is surrounded by an immersive, total simulation in which the interface both disappears and becomes the experienced simulation at one and the same time (Pold 2005). But is this true?
Consider the Vive controllers, or the Oculus equivalent. These handsets incorporate triggers, buttons, trackpads, haptic feedback etc. through which we interact with the virtual world. First we need to interact with the physical devices themselves – to map our body movement and intended action within the virtual world to the physical interface. Then there is the interface between the physical devices and the devices in the virtual world – the tools that they turn into or the virtual body itself, and the translation between the bodily manipulation of the physical device and the virtual action which results.
So interaction involves a layering of simulations – a simulation of an interaction with a simulation of an interaction with a simulated object, for example. In reality (!) we interact through a series of interfaces, multiple layers of simulation, that are inserted between ourselves and the visual simulation in the attempt to create our sense of embodied cognition within the virtual world. A sense of embodiment that is constrained by the specific spatial frame of reference built into the interface, one which is not natural, or at least, not the only one. Of course, it could be argued in this case that the simulation is not truly immersive because of its reliance on external, manually operated devices which require this simulation within a simulation to translate physical action into virtual action. But is the situation greatly different in a fully immersive simulation which entails data gloves, omni-directional treadmills and the like which facilitate a more realistic sense of embodiment? While the translation of action and the feedback from it are certainly significantly improved, there still remains the interface(s) involved in transferring actions and reactions between the physical and virtual.
Transparency is at stake here, transparency in the sense of being easy to see rather than see through. The function of the interactive interface is superficially enabling – it is, after all, what allows us to explore and manipulate the visualisation. It is superficial because while it enables, it simultaneously determines, disguises, and limits the interactive experience. It gives the illusion of freedom – some have characterised this as ‘creating a better kind of mousetrap’, a form of hyperselectivity (Elsaesser 2014, 303) – while keeping us firmly at arm’s length. Consequently, the sense of empowerment we experience through interacting with and controlling a virtual space is illusory – we are being controlled by the possibilities that are released through the interface. We interact with an interface which is a device of capture, which confines us, and defines our agency. The challenge is to make it possible to break down this virtual equivalent of the fourth wall. This then raises a host of further questions. Should we be capturing and representing our interactions? Is there such a thing as inappropriate or unsuitable interactivity? Can there be too much or too little interactivity? What do we actually learn from interactivity? And does an interactivity encapsulated within a 21st century technical environment have any real value for our understanding of the past?
References
Elsaesser, T. 2014 ‘Pushing the contradictions of the digital: ‘virtual reality’ and ‘interactive narrative’ as oxymorons between narrative and gaming’, New Review of Film and Television Studies 12 (2), 295-311. http://dx.doi.org/10.1080/17400309.2014.927182
Pold, S. 2005 ‘Interface Realisms – The Interface as Aesthetic Form’, Postmodern Culture 15 (2). https://dx.doi.org/10.1353/pmc.2005.0013