Return to Article Details The augmented subject - technological interfaces of subjectivity, geography and the moving image

Introduction

Hollis put the visor on. She could see through it, though only dimly. She looked toward the corner of Clark and Sunset, making out the marquee of the Whiskey. Alberto reached out and gently fumbled with a cable, at the side of the visor.
"This way," he said, leading her along the sidewalk to a low, windowless, black-painted facade. She squinted up at the sign. The Viper Room.
"Now," he said, and she heard him tap the laptop’s keyboard. Something shivered, in her field of vision.
"Look. Look here." She turned, following his gesture, and saw a slender, dark-haired body, facedown on the sidewalk.
"Alloween night, 1993," said Odile. Hollis approached the body. That wasn’t there. But was.
(William Gibson, Spook Country)

In William Gibson’s Spook Country, Hollis Henry, the protagonist of Gibson’s latest trilogy of novels, is invited to a spooky installation in the very first chapter: with a virtual reality headset device she can revisit the death of the actor, River Phoenix, on the spot (Gibson 2008, 9). The on-location virtual artefact is part of the trend once known as "locative art" – today it is not really an art form, and we simply use a more general term, augmented reality instead.

 

The Viper Room – the spot for the locative art installation in Spook Country – viewed in Google’s Street View

 

Today we are – more and more without the clumsy goggles and devices that accompany them – like Hollis: not sure of what we see, how we see, and of the flimsy boundary between virtual and real, online and offline; and a bit like the body that is there and not there at the very same time, mapped and unmapped in an uncanny cartographic way.

I would like to address the specific problem the emplacement and the positioning of the subject presents in today’s increasingly augmented everyday realities. My particular interest lies in the technological aspect of the augmentation of space, of geography that results in what Nanna Verhoeff calls “performative cartography” that questions not only our modern understanding of – mostly urban – spaces, but also reveals a problematic aspect of subjectivity. If the space that the subject finds itself in is augmented, it is inevitable to pose the question what this augmentation brings for the subject. I will identify a common element, a digital-material glue that helps us understand how the subject is mapped onto the very cartography it pretends to master: the interface. While theorists have recently paid much attention to both the question of the subject and of space in the digital world separately, I would like to refocus the discussion on what connects the two, and claim that it is the technological aspect of this connection that defines the augmentation of space and subject alike.

Augmented Reality

In general terms, augmented reality can be thought of as a digital layer on our reality comprised of geo-registered data retrieved and visualized through the screen of a mobile interface. As such, augmented reality, through contemporary mobile screen technology, not only takes place, but it also makes space (Verhoeff 2012, 134): the screen of the interface reconfigures the geographical space around us and creates a data-filled vision of it in front of us, thus making us capable of not only understanding our present situation (the deictic “You are here” type of positioning of traditional cartographic logic), but also of navigating this space. In other words, augmented reality fills the space with visualized data connected to geo-registered points, thus augmenting (but not completely overlaying or overtaking) our perception of reality.

Let me illustrate my point with a somewhat unique application that might even raise questions about the medial definition of the cinema, as well as show the basic principles on which augmented reality operates. The Augmented Reality Cinema application takes geo-registered data of well-known film sequences and allows the user to activate the digital layer in their reality on-location. This way it is not the subject who enters the diegetic realm, but the diegesis enters the subject’s reality, inviting the subject to the location of the particular shots, outside the institution of the cinema (or rather, the user-subject navigates the filmic sequence to its location).

 

 

I could have chosen other applications as well, of course, as this is an evolving technology, but I think the diegetic space of the filmic sequence can more spectacularly integrate with the representation of our reality than any other data-driven apps.

What is pivotal in augmented reality is an upgraded cartography. Traditionally, cartography is concerned with making and reading maps, in two-dimensional meaning-making attempts, separating the observer and the observed (Verhoeff 2012, 139). Today, however, with the introduction of interactive, digital mapping tools featured by mobile and essentially networked media technologies, the function and the working mechanism of mapping have fundamentally changed. New mapping practices blur the boundary between observer and observed, and put the user in the role of navigator with the use of Global Positioning System (GPS), Geographic Information Systems (GIS), geo-browsers (like Google Earth), map-hacking, geo-tagging, and allowing the creation of mash-ups. According to Verhoeff, this not only leads to new representations of space, but also provides “tools to modify maps” and “to fill maps with different kinds of knowledge” (139).

It is thus a new way of putting oneself onto the map: it does not depend on the cartographer, but rather on how connected and networked one is, since contemporary digital cartography allows the user to manipulate and navigate the datalayers comprising the ever evolving map, thus allows for acts of creating and recreating space in a process of re-locating oneself.

According to Verhoeff, this means that we have moved from representational cartography to what she coins as performative cartography. Performative cartography is “a procedural form of simultaneous making and reading space by exploring a hybrid space of atoms and bits, both the physical and the virtual, through interaction between on- and off-screen navigable space” (137). This practice brings together three domains central to the visual regime of navigation: screen, space, and mobility. Performative cartography emerges “during movement, as a particular form of interactive navigation: fixed maps do not dictate the itinerary, but rather maps and views evolve and emerge along the way” (145).

Verhoeff compares the process of performative cartography to bricolage, that is, a type of “creation by recombining ready-made bits and pieces” in the way Claude Lévi-Strauss or Gerard Gennette talk about it (146), but I think this is not the case here. Performative cartography and digital mash-ups are examples of what Lev Manovich calls deep remix, which is a truly hybrid media format (Manovich 2008). The difference between a bricolage or remix and the deep remix is that while in a remix the combined elements preserve their logic of representation (an image is an image even if it is transposed to text or music), in a deep remix the logic of representation is brought to a common platform: that of the programmable language of software.

Let me refer to a mash-up example to illustrate the hybridity of new media that is also at play in performative cartography. Canadian indie band Arcade Fire teamed up with Chris Milk to create an intriguing Google Experiment project called The Wilderness Downtown, which takes the band’s track, "We Used to Wait," and turns it into a hybrid mash-up of video footage, geo-imagery and Google’s map application, most notably its street-view option. The user navigates to the website the project is located at and enters the name of his or her home town – the algorithm then searches for the coordinates and assembles a personalized show. In a curious act of digital flânerie, the user gets multiple points of view shots of video and street-view imagery stitched together, adjusted to the movement of the faceless protagonist of the clip, using several, dynamically changing browser windows as a multiplication of screens within the screen.

 

 

A nostalgic gesture indeed, the clip in fact utilizes the spatial memory of the user-subject, connecting the psychological factors with an algorithmic rendition of space. It dynamically, in a procedural manner, repositions the subject through the interface of the mash-up to recreate the once visited spatial coordinates that allow the subject not only the recollection of psychic, but also of bodily engagement within the given spatio-temporal setting. The subject’s spatial memory thus comes to be aligned (and overseen) by digital and performative cartography – in André Nusselder’s reading of Lacan, a fantasy screen navigated through the interface.

Another project for browser rendition by Milk, called RO.ME, illustrates the fact that one does not need real coordinates to create and maintain visual representations of space, or to navigate a diegetic space. The dream-like, travellesque, fantastic rollercoaster is in fact a momentarily rendering of a bunch of code written in WebGL inside the browser. The user can interact with the source code, change its properties to reposition itself or to redesign the spatial coordinates in which the short story unfolds. While the project obviously does not refer to any real world cartographic element, it perfectly shows the way performative cartography is embedded in representations crafted on new media bases.

 

 

The Moving Image

The concept of the Augmented Reality Cinema application is indicative in terms of how we tend to think about the cinema in particular, and about the moving image in general today. Cinema, a cultural interface, dominant in the twentieth century as Manovich asserts, is moving out of its institutionalized home, freeing itself of its brick and mortar roots with the theatrical institution we used to go to see the latest and greates releases the motion picture industry produced. With the advance of digital technology in production and, a bit later, in distribution, cinema is no longer the well-definable medium theorists thought it to be. According to Manovich, if you look up the definition of the leading industry standard of motion graphics on Wikipedia, you will get a concise and more or less precise understanding of the background (Manovich 2008). However, it is no longer so easy to define what cinema is on strictly medial terms.

The above examples of new media make it essential to rethink and augment the notion of spectatorship. On the one hand, we need to investigate what happens when the kinetic presence of the spectator precedes that of the moving image, i.e. when the network of geolocated cinematic sequences structures the movement or path of the spectator in space. On the other hand, we also have to look at the phenomenon of what might be termed as digital flânerie: a kind of wandering or strolling reminiscent of a leisure walk in reality, but generated through a technical apparatus by the augmentation of the spatial structure of the moving image with maps and data.

In post-structuralist film theory, the spectator subject is a kind of receptacle, sitting inert and immobile in the darkness of the auditorium. Its position recalls the imprint of Plato’s simile of the cave for the theorists of the apparatus (as seen in Jean-Louis Baudry’s work for instance). Similarly influential is the idea of the dream analogy (utilized by theorists from Christian Metz to Bruce Kawin), according to which the spectator encounters with the flickering images on the screen just as it would in a dream situation, where it would also be a passive recipient or witness to the sequences, like in the case of a film (Cristian and Dragon 2008, 37-38).

All of these theories of the filmic spectator enlarge upon the technology and characteristics of modern cinema. But what happens if we reach back to the original position the movie spectator inhabited: the mobile spectator? The spread of nickelodeons and movie palaces were indeed preceded by the invention of the technology, called the kinetoscope, invented under the auspicies of Thomas Alva Edison by William K. L. Dickson, that was later taken over by more sophisticated projection technologies (Belton 2005, 6-8). The machinery, introduced in 1892, found its home under street arcades. The individual kinetoscopes, arranged in line, presented the viewer with short, repeated, looped sequences. These clips were sometimes connected and made up a story of some sorts that could be assembled by walking from machine to machine, and peeping into the viewer on the top of the box. According to Manovich, this method is strikingly similar to the logic of representation advocated by the QuickTime video player released in the 1990s: reception is provate as opposed to collective in both cases; the size of the screens are relatively small (as compared to the cinematic one); and both techniques feed on the structure and logic of the loop (Manovich 2001, 315).

I think the difference betwen the two modes of representation and reception can best be grasped in view of contemporary mobile interface technology: it not only returns the moving image to its originally urban home, but also puts a range of portable devices into the hands of the spectators which free them from standing still in the shade of the arcades. By this augmentation of reality, film and the moving image find a new place, as well.

The beginning of the 21st century brought about a significant shift in thinking about the experiencing of reality: blunt reality can no longer be seen solely as to be experienced through the ususal sensorial percptorial avenues. Perception today is defined by multi-layered, georegistered, dynamically updated and changing database: moving image is one data layer among the many. However, this spectacular augmentation of the space of reality has serious impact on the notion of the spectator subject, too.

The X-Subject

Steve Pile and Nigel Thrift define subjectivity as “rooted in the spatial home of the body … united and orchestrated to a greater or lesser extent by narrative, and as registered through a whole series of senses” (Pile and Thrift 1996, 11). Locating the “spatial home of the body” and registering through senses recalls the Freudian definition of the ego, as “first and foremost a bodily ego,” that is “not merely a surface entity, but is itself the projection of a surface” (Freud 1961, 27). Jacques Lacan takes his cue from the projective quality of the surface and suggests that the three-dimensional space in which the subject resides and functions is not a given, we are not born into space: it is the result of identification in the field of the Other that prepares the subject to comprehend and navigate space (Adams 2003, 148). Lacan’s idea is instructive here: from his famous mirror stage idea on, he emphasizes the presence of a screen, a projective surface on which various scenarios of identification can play out.

André Nusselder, in Interface Fantasy: A Lacanian Cyborg Ontology, argues that the subject in the age of information technology lives on the screen (Nusselder 2009, 2), which echoes among many critical views, Nicholas Mirzoeff’s claim that “modern life takes place onscreen” (Mirzoeff 1999, 1), but he enlarges on this statement in a different way. For Nusselder, the computer screen is essentially a psychological space, the screen of fantasy, which distorts “the ideal relationship” of the user-subject and the sign (Nusselder 2009, 3). “Therefore,” he says, “the relation of user and sign is … cybernetic” (3).

While I agree with the claim that the introduction and use of digital computing devices distort the way the user-subject and the signs of representation engage, I think the subject does not “live on the screen.” As I will discuss it later, the cybernetic relation of user-subject and sign can work only if we stay close to the Lacanian graph of visual regime: the screen needs to be seen as a digital interface through which the relation of the user and the sign can play out. In other words, life does not take place on the screen but rather through it. It is an important distinction, as I do not think the subject has ultimately become completely virtual – though I will keep arguing for it being ultimately augmented.

To think about subjectivity in the age of Web 2.0, we need to consider that the subject is co-present in blunt reality as much as it can be online. It means that the digital aspect of subjectivity needs to be taken into consideration as well in defining the subject of the 21st century. What it entails is an intricate process of tracing one’s avatar – not only the visually composed set of images (let alone the blue Na’vis of James Cameron’s groundbreaking film) that we use as funny (or not so funny) identifications of ourselves on social networking sites or personal websites, but also the invisible digital footprints that we inevitably leave behind when browsing the World Wide Web.

This online subjectivity layer is in fact more than what we may think it is: we know what sites we have visited, what videos we have just watched on YouTube or whom we befriended on Facebook – but the internet knows a lot more about us. When you log into Amazon.com, the algorithm will instantly recommend you a wide range of items: how did it know that I am interested in science fiction? Well, it tracks my online route, but it uses not only the data I leave behind when I navigate away from its pages, but collects the digital footprints I may take afterwards as well. It knows where I come from, it knows what I do there, and it knows exactly whither I leave. A lot of companies out there do nothing but trace my footsteps and compile a virtual map of my online travels. My online avatar thus is an ever growing dataset composed of my navigation, my browsing habits, which then “pictures me” so that I can be identified when entering the site of commerce.

To see who follows your invisible digital footprints, one handy tool to refer to is the Collision plugin for Mozilla’s Firefox browser: it gives you an interactive infographic representation of those sites that you connect with without actually clicking on their hyperlinks by accident or on purpose. To illustrate how intricate this graph can turn, see my 5-minutes long browsing on regular sites: I start out to read my emails, check my friends on Twitter and Facebook, navigate to a couple of Hungarian news portals, watch a YouTube video, and the like – and see how more bubbles pop up adjacent to the particular sites: sites I have never ever visited, sites that follow me nonetheless. When talking about avatars or online representations of subjectivity, this is what should be meant: the intentioned plus the unintentional pieces of navigation that one performs when browsing. (Note that as I am talking, more and more companies announce their DNT strategies, such as Twitter or even Microsoft with the introduction of its new Internet Explorer browser: they allow the user to opt out or even to block 3rd party sites and services tracking their online browsing.)

 

 

In line with Nusselder’s discussion of Lacanian psychoanalysis in the digital age, one could assert that the unintentioned part of the dataset that builds my avatar is basically the digital unconscious that symptomatically returns in the most unexpected corners of the World Wide Web. He suggests that “information technologies … seem to design or create a second, parallel world” (an idea that corresponds to the main hypothesis of virtual domains like Second Life), and we can communicate, interact with this world in a way that user-data relation becomes transparent (Nusselder 2009, 8), an idea that could easily be linked to what Jay David Bolter and Richard Grusin calls “immediacy” (Bolter and Grusin 2000).

Recently, Beth Coleman investigated how our perception of reality is changing by the minute and how our online persona is getting more and more emphasis in how we see ourselves. In Hello Avatar: Rise of the Networked Generation, she elucidates the difference between offline and online dimensions that is usually at the core of discussion of contemporary digital identity, and contests the use of such a binary opposition. Instead, she points out how the two have really become one inseparable experience and goes on to offer the term “x-reality” (can be read either as “ex-“ or “cross-reality”) that is the quasi-virtual realm intertwining the two domains of experience.

While Coleman improves on the subject-object relationship in the digital age compared to many of the contemporary considerations of the topic by adding the programming, technological layer into the equation at some point (even attributing some kind of disruptive aspect to technological agency), I think that by putting the ultimate emphasis on the agency of the user-subject in interaction, the immediacy of the interface becomes a ghostly non-presence. Moreover, I also question the disruptive aspect of technology in augmented reality – especially in arguments that aim at connecting the on- and offline domains. I agree with both Nusselder and Coleman that transparency for the interface is pivotal, but I consider it problematic to locate agency on either the side of representation or that of the subject of representation.

As I have outlined above, it is through the interface that space is made: it is the performative cartography that uses geo-registered data, user interaction and interface technology that creates spatial representation. However, it is only through the programmable interface that the user can locate and constitute itself as subject who then is able to partake of the process of creating space. It is as if the Lacanian big Other got manifested in the mobile screen technology of the interface: the user-subject identifies itself through the device that helps it locate itself – a gesture that also makes space available to navigate in.

The Interface

In his discussion of the position of the Lacanian Real in the cyperspace, Slavoj Žižek argues that "[w]hat defines the properly ‘human dimension’ is the presence of a screen, a frame, through which we communicate with the ‘suprasensible’ virtual universe to be found nowhere in reality" (Žižek 1997, 60). In this model he refers to the diagram Lacan uses to define the difference between the look and the gaze in The Four Fundamental Concept of Psycho-analysis, where the subject of representation is ultimately deprived of all of its imaginary agency. Interestingly, Lacan also denies this power of the gaze, i.e., the position of the big Other, thus challenging the thereto hierarchically organized relationship between the subject and object of representation (Lacan 1998, 67-78). The structure Lacan proposes and Žižek adapts to a cyberspace context is thus very similar to the situation I am describing.

The only difference that Žižek does not address is indicative, however. Both Lacan and Žižek presuppose the structuring role of representation as such: for instance, a painting designates the position of its spectator (think of perspectival organization) that leaves him or her with the feeling of mastery over the visual field. However, the smallest detail or point of disorientation (such as the strange blot in Hans Holbein’s infamous The Ambassadors) annihilates this impression and reveals the “other side” of representation, the gaze, the collision with which is the flimsy structural pane of the screen. The screen is then a mediating instance necessary to keep the balance of the look and the gaze, a pivotal part of the structure of representation, of the visual regime in which the subject comes to be, nonetheless, itself part of the scheme of representation.

In augmented reality situations, however, the screen of the interface (today in quite a material format) pre-presents rather than behaves as part of the representation. I am referring to the technological necessity Verhoeff calls attention to: in digital cartography space is continuously recreated by the user-subject through the interface. While, as we have seen in the example of comparing the practice of performative cartography to bricolage, Verhoeff does not emphasize the role of the interface as much as I think it should be done, it is clear that this pre-presentational turn could not operate without the fundamental presence of the interface, of the mobile screen.

Moreover, we need to acknowledge the agency of the interface as well – as taken over from either the agency of the visual regime of representation or from the subject. William Uricchio has recently announced the algorithmic turn in visual culture (Uricchio 2011) in his analysis of Microsoft’s Photosynth app. Photosynth basically takes user-generated, geo-registered imagery, finds the matches among the myriad of photos and stitches them together in such a way that it does not become a simple 3D panoramic representation of space, but rather a fluid, navigable one. One example is the collaboration with CNN’s broadcasting Barack Obama’s inauguration, when thousands of images were uploaded by professionals and amateurs alike, while Photosynth’s algorithm was busy at work to create a dynamic, navigable experience of the Washingtonian space.

 

 

While I think the algorithmic turn came a bit earlier (think of Animoto’s service, for instance, that matches color, editing and even music to the “feeling” of the photos you upload and thus creates a video montage), this turn clearly shows how the agency is now ultimately on the side of the interface and technology. It even forces its own digital, non-linear logic on us: in Photosynth there is no fixed point of view, set frame of representation, as it works with multiple authorities, fluid structuring of imagery based solely on data.

The interface, however, is not innocent, and while strives to be transparent, it is not so successful. In Lev Manovich’s definition, the interface “acts as a code that carries cultural messages in a variety of media” (Manovich 2001, 64). The interface is the mode of displaying, receiving, and transmitting information, an electronic surface that allows users interactive engagement with any form of content: the screen, the software and hardware used, and the special logic they entail. However, “far from being a transparent window into the data inside a computer,” Manovich argues, “the interface brings with it strong messages of its own” (65). It means that the logic with which an interface operates is already coded in the transmission of data.

As the technological insertion between the user-subject and space, the interface enables performative cartography. As Verhoeff explains,

The basic principle of screen-based navigation is that we see how we move, while how we move enables vision. This mutually constructive, discursive relationship between seeing and moving is a new principle in real-time, digital cartography. It is the movement that establishes the map; reading space requires navigation, rather than the other way around. Digital maps make use of the logic of tagging, plotting, stitching as forms of interaction. (Verhoeff 2012, 153)

Tagging is labeling objects and locations with metadata (this is what the organizers of the conference did with the map we were given: they augmented our reality, re-mapping a Szeged for us). Plotting is the practice of tagging objects in space, and inserting tagged objects in the map (like live-streamed videos, or photographs you could see in Photosynth, plotting a whole inauguration ceremony in a special way). Finally, stitching is when these practices are integrated into a navigable whole (153).

Stitching is like a 4D model of filmic suture – not the standard suture procedure criticized so severely, but the short-circuited mode as discussed by Žižek who argues that when shot and counter-shot (the basic constituents of the suture operation on a formal level) appear within the same shot, subject and object of the look become co-present in the same virtual, impossible space, creating what he calls, the interface (Žižek 2001, 40). To refer back to the Augmented Reality Cinema app, I could say that what it presents us with is the suture operation translated into the interfaced, short-circuited mode of proto-filmic and post-production sequences in which scenario, curiously enough, the film sequence comes before, precedes, pre-presents the real space for the subject: stitches it into a new, augmented type of diegetic realm.

It is precisely this aspect, the tagging, plotting, and finally, the stitching of the user-subject within performative cartography that is missing from Verhoeff’s discussion: it is as if – contrary what she states about the interactive nature of such a process – the subject could distance itself from the scene it is navigating, exactly like the Cartesian subject of representational cartography. The point, however, is that by this process, the subject itself comes to be: tagged in space, plotted in the map, and stitched into its physical and augmented environment. It is the digital version of the Lacanian scenario of identification through the field of the Other as an establishing gesture of making space: here, however, the augmentative act of the interface takes the place of the field of the Other.

To refer back to Manovich’s definition of the interface, and to augment Verhoeff’s performative cartography with its help, one can detect a crucial change in how our perception of our spatial setting is structured: in Lacanese, the visual field (including the spatial coordinates) is born out of the split between the gaze and the look, with the intrusion of the screen. However, in augmented reality, and especially with regard to performative cartography, it is the screen that gives birth to both subject and space. This is, I would say, the techno-logic of digital representation in spatial terms.

The consequences of this cannot be overemphasized: the interface augments reality and the subject alike, thus taking the ideological power of representation with it, for it is no longer the map that acts as the locus of knowledge, of politics, of power, of instruction, i.e., it is no longer the representation that designates the place of object and subject, but the pre-presentational interface that – as Nusselder showed – distorts the relation between user and sign.

What all this amounts to finally, is the need to reconsider how we think about subjectivity in the age of digitally augmented reality. As our space comes to be augmented more and more by the day, inevitably, the subject – caught up in the flux of augmenting reality – cannot but be part of this interactive gesture, and augment itself by re-locating itself, renewing its self-recognition through the digital interface that instead of plotting our narratives in space, invite us to the procedure of continuous tagging and mapping. With the use of contemporary augmenting scenarios, therefore, the augmented subject is born through the interface.

 

Works Cited


Copyright (c) 2013 Zoltán Dragon

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.