SkyLineDroid: An Outdoor Mobile Augmented Reality Application for Virtual Heritage

Recent Researches in Engineering Mechanics, Urban & Naval Transportation and Tourism

SkyLineDroid: An Outdoor Mobile Augmented Reality Application for Virtual Heritage GIUSEPPE ARMANNO, ANDREA BOTTINO, ANDREA MARTINA Dipartimento di Automatica e Informatica Politecnico di Torino Corso Duca degli Abruzzi 24, 10129 Torino ITALY [email protected], {andrea.bottino,andrea.martina}@polito.it http://evip.polito.it Abstract: - Cultural tourism could greatly benefit from recent advances in technology, in particular in the field of mobile devices. This paper presents SkyLineDroid, an application of Virtual Heritage where Augmented Reality is used on mobile phones to support visitors of outdoor cultural heritage sites. Virtual and real world are overlaid on the device screen, according to device position and orientation, in order to immerse users in the 3D historical reconstruction of ancient buildings. Furthermore, a wealth of available multimedia information, related to each virtual element, is available and can be accessed by visitors according to their own interest and curiosity. Therefore, users are provided with a complete multimedia cultural guide which allows them to assume an active role in their journey through history. A prototypal implementation, based on the history of one of the most famous boulevards of Las Vegas, the Strip, is presented in this article. However, the structure of SkyLineDroid is very flexible and can be easily adapted to other cultural heritage sites.

Key-Words: - Virtual Heritage, Augmented Reality, Mobile devices, Multimedia, Cultural Tourism

commentaries related to the virtual elements. Therefore, visitors of a cultural heritage site can be provided with a large amount of information about objects and monuments within the site, which can be consumed according to their interest and curiosity, improving their cultural, historical and scientific understanding.

1 Introduction Cultural tourism is gaining increasing attention in recent decades. Although the numbers of cultural tourists represents a minimal quantity compared to the total, 9% according to the ATLAS survey [1], that number is growing thanks to the multitude of services and products offered. Tourism authorities all over the world have pointed to the improvement of the attractiveness of cultural sites, developing novel and interesting tourist information services. In this context, technologies play a very important role, thanks to their enormous development and progress.

Recent advances in mobile devices, which have evolved to provide powerful computational and connectivity capabilities, makes the adoption of AR particularly appealing for such devices. Recent products offer enhanced multimedia features, like high quality displays, video cameras, touch screens, advanced audio support and 3D rendering. Current communication networks (GPRS, UMTS, HSDPA) allow them to receive large amount of data in a short period of time. Furthermore, mobile phones are personal devices that users carry with them most of the time and are, therefore, specially suited for developing applications supporting moving users with interactive multimedia information and, in the specific context, with cultural contents.

In the field of Virtual Heritage, meant as the integration of ICT technologies and cultural heritage, Augmented Reality (AR) is receiving a growing interest. AR combines real objects and computer-generated data, which are blended together into the real world view. It means that user could see virtual and real object living in the same space [11], [12]. Another advantage is the possibility to integrate into the system video, images, text and audio

ISBN: 978-1-61804-071-8

91

Recent Researches in Engineering Mechanics, Urban & Naval Transportation and Tourism

the mobile technologies. Some projects, as [6], [13], [14] and [16], offer real-time AR applications on Ultra-Mobile PCs, which guarantees good computing performances. However, normally, tourist do not carry with them such devices and cultural institutions would be required to provide them to visitors. Instead, applications for smartphones, like SkyLineDroid, can exploit HW commonly held by visitors. Concerning contents, several applications, like [15], [19], [18] and [17] as well as the so-called AR browsers, like Lyar, Wikitude or AcrossAir, handle augmentation only in 2D, showing 2D multimedia contents related to georeferenced POIs. The project described in [16] tries to mimic the delivery of 3D contents using a set of billboard-like textured objects at different depths. Two recent projects ([7], [5]) have some similarities with SkyLineDroid. The first ([7]) presents an iPhone mobile application for the AR tour of the National Palace Museum of Korea. Additional information is provided as 2D content, but 3D is also used to present a virtual character giving information about the object seen by the device camera. MobiAR [5] is a mobile AR application that can support tourists during their visit. 2D and 3D contents are presented to the user, but here 3D is not used to reproduce the virtual reconstruction of a site, which is, instead, the very principle of our application.

In this paper we present SkyLineDroid, an outdoor Augmented Reality VH application for mobile phones that immerses users in the virtual reconstruction of a cultural site by showing how the location looked at different periods in history. The 3D models of ancient buildings are rendered in their original locations and overlaid on the real world view, acquired with the device camera. Therefore, user can wander around to see the buildings from every point of view and navigate in a space where real and virtual elements coexist (Figure 1).

Figure 1. An example of augmented view

Historical multimedia information related to the places visited can be accessed from the application by simply tapping on the displayed objects. This freedom of interaction with the system and with the available historical contents, allows visitors to play an active role in their journey through history. The content of the paper is organized as follows. Previous works are reviewed in Section 2. In section 3 we describe the application and its prototypal implementation. Conclusions and future works are in section 4.

3 Description of the application As already stated in the introduction, SkyLineDroid is an application of VH that exploits Augmented Reality on mobile phones to immerse visitors in the historical reconstruction of an outdoor cultural heritage site. The system is designed to handle multiple 2D and 3D content, in order to provide visitors a true timeline of the evolution of the site. Virtual models of ancient buildings and monuments of the selected period are shown on the device display on their original location. Therefore, SkyLineDroid allows users to customize their visit by choosing what to see, from which point of view and to interact with history, observing the architectural and social

2 Previous works Among different techniques, Augmented Reality (AR) has been used in a variety of applications for enhancing visitors experience at cultural heritage sites. Surveys on mobile AR technologies and their use in the context of VH can be found in [4] and [8]. Since the introduction of the concept of mobile virtual tour with the Ancient Pompeii project [3] and the ARCHEOGUIDE [2] system, which were based on wearable computers, the number of mobile AR cultural heritage applications have increased thanks, in particular, to advances in

ISBN: 978-1-61804-071-8

92

Recent Researches in Engineering Mechanics, Urban & Naval Transportation and Tourism

Riviera. The timeline goes from 1952 to 2006. During this period, these casinos underwent several renewals and expansions, which have been all virtually rebuilt (two exapmles are shown in Figure 2). In total, there are 16 timeslots through which visitors can appreciate the temporal changes of (a part of) the Strip skyline. For each building and each period, a wealth of historical original documents is available, like pictures, advertising billboards and restaurant menus, video footage and radio shows, maps and postcards. This original archive material, which has been also used to model the different versions of the casinos, has been granted by several institutions and private persons: UNLV, holder of the UNLV Special Collections, the Liberace Foundation’s music archive and Tom Askew’s Private Collection.

evolution of the site through a straightforward comparison with its actual state. The structure of SkyLineDroid is the following. When the application starts, it shows a welcome message that explains its functionalities, followed by a short introduction with some general information about the heritage site the user is going to visit. Touching the display, the application switches to the navigation mode, where the user can select the historical period he/she is interested in. Then, when the device is pointed at the surrounding world, the application renders, according to their location w.r.t. the observer, the 3D models of nearby ancient buildings on top of the live video frame, creating the augmented view. Tourist can click on the preferred building and the application will display a range of additional multimedia contents related to that building in that particular historical period. Each virtual building is associated with a visibility region, that is the area from which its 3D representation can be viewed by the user without being occluded by other buildings or without being too far for appreciating the model details. When no virtual buildings are visible, they are treated as Point Of Interest (POI) and their location is shown in AR as balloons. Navigation in the augmented world is facilitated by a radar displayed on the monitor, showing the locations, with respect to user’s position, of the virtual elements surrounding him. In addition, displacing the device parallel to the ground plane, a full screen 2D map of the area, which is interactively rotated according to user’s orientation, appears on screen. Icons highlight on the map user’s position and locations where augmented views are available. Hence, the application performs a dual role: a virtual driver through the past and a guide for visiting the actual places.

We underline the fact that SkyLineDroid is an open platform that can be easily adapted to any other cultural heritage site by simply changing the collection of 3D models and digital contents handled by the application.

Figure 2. The virtual reconstruction of the Stardust in the 1958 (top) and in 1972 (bottom)

A prototypal version of SkyLineDroid has been implemented on the Android platform. As a test case, it has been designed to immerse visitors of Las Vegas in the historical evolution of “the Strip”, the main city boulevard, focusing in particular on three of its most prestigious and oldest casinos, the Stardust, the Sands and the

ISBN: 978-1-61804-071-8

In the following subsections we will describe in details the main components of SkyLineDroid.

93

Recent Researches in Engineering Mechanics, Urban & Naval Transportation and Tourism

and Z. Virtual objects are then rendered using OpenGL ES library according to position and orientation of the virtual camera. The picture on the OpenGL surface is finally overlaid on the image taken with the device camera, which represents the current world view. On the augmented view, the user is able to change smoothly in real time the transparency level of the rendered objects, from full transparent to opaque, with a slider. This allows a straightforward comparison between real and augmented view and an easier way to highlight their differences.

3.1 AR contents management A block diagram, outlining the operations required for handling AR contents, is shown in Figure 3.

As for lighting, during daytime, position, orientation and intensity of lights in the virtual environment are set in order to mimic that of the real world. Given the current date - taken from the phone’s calendar - and the location of the observer, the application checks in a table the Right Ascension and Declination of the Sun on that day, which, combined with the current clock time, provides in real time Sun azimuth and altitude. This table has been created with the algorithm described in [9], which guarantees position errors of few arc minutes. At night, lighting has been studied to recreate that of the original buildings, according to an interpretation of the available iconographic archive material.

Figure 3. Management of AR contents

SkylineDroid supports location based tracking using device positional data, obtained through mobile phone GPS, digital compass and accelerometer. These data are used to evaluate the location and orientation of the virtual camera that will provide the view on the augmented world. In order to guarantee a seamless integration of real and virtual elements, device positional values are queried at high frequency (around 25 Hz) and are, therefore, affected by a certain amount of noise. To reduce it and to ensure greater stability of the displayed objects, a moving average filter, taking into account the last 10 samples, is applied to the data stream. Such filter introduces a certain latency between the changes in the device position and the display of their effect, which, however, remains negligible for smooth movements.

Figure 4. AR view of POIs as balloons

When no virtual models can be seen from the current position, that is when the user is not inside any of the model visibility regions, his/her position and orientation are used to display the location of virtual buildings as POI on the AR view (Figure 4). Each POI is represented as a balloon, containing the POI description and its distance from the actual position. A series of concentric circles are also

Virtual elements are reconstructed to scale with the real ones. They are positioned and oriented, according to original buildings’ locations, along a global reference system, where GPS units are used as metric. The Z-axis of the reference system is the normal to the ground plane, its Yaxis points towards the North Magnetic Pole and X-axis is obtained as the cross product of Y

ISBN: 978-1-61804-071-8

94

Recent Researches in Engineering Mechanics, Urban & Naval Transportation and Tourism

models corresponding to that exact period. The models are cached on the device so that a further request of the same model does not require its download from the server. Models are time stamped in order to invalidate cache entries that become stale after a model update on the server.

present to improve the user’s perception of distances. Another feature provided by the application is the ability to access at any moment a full screen 2D map of the area showing the current user’s position and the location of the surrounding POIs (Figure 5). The map is activated by simply displacing the device parallel to the ground plane and is rotated according to the user’s orientation.

Models descriptions, including their geometry and associated textures, are stored in XML format using an optimized subset of COLLADA features [10], in order to reduce size of the transferred data and their decoding time on the client. Currently, all necessary models (at most three new models for our test case) are loaded at the same time and stored on the mobile device. We are planning, for more complex scenarios, to implement smarter strategies, like initially downloading only objects in a predefined neighbourhood of the observer, and then requesting new models, in background, according to user’s movements.

Figure 5. The 2D map showing user’s position and POIs locations

Finally, the user can access and browse the additional multimedia information related to any object simply tapping on it, either in “augmented reality” or in “map” mode. The historical contents (text, images, videos, audio, etc…), related to the chosen historical period, are presented in a gallery where users can select and open the items they are interested into (see Figure 6).

The description of the multimedia content related to each model is included in a separate XML file. Along with the list of available material and their thumbnail sketches, the way it will be presented to the user (in terms of its organization in tabs or folders) is defined as well. This file is downloaded only when the user taps on a model (on the 3D view) or on an icon (on the map view). Finally if the user taps on a thumbnail, the referred content (e.g. a high resolution image, a video or an audio file) is downloaded from the server and activated. The described client/server architecture is necessary for handling the large amount of multimedia data that can be used to support visitors during their journey. Another advantage of this structure is that contents, in terms of both available POIs and their related data, can be added or modified at any time on the server without requiring users to download un update of the application.

Figure 6. The content gallery related to the 1952 model of the Sands

3.2 The client-server architecture SkyLineDroid is based on a client-server architecture. The server handles requests from the client (the user’s mobile phone) and stores in a database all the 3D models and multimedia information used during the augmented tour. Anytime the user chooses an historical period to visit, a request is sent to the server for obtaining the POI list and for downloading only the

ISBN: 978-1-61804-071-8

4 Conclusion In this paper we have presented an application for mobile phones, capable of supporting visitors of an outdoor cultural heritage site.

95

Recent Researches in Engineering Mechanics, Urban & Naval Transportation and Tourism

Mobile Devices”, in Proc. MMM, 2010, pp. 588-602. [7] J. Kim, C. Park, “Development of Mobile AR Tour Application for the National Palace Museum of Korea”, Proc. HCI (13), 2011, pp. 55-60 [8] Z. Noh, M.S. Sunar, and Z. Pan, “A review on augmented reality for virtual heritage system”, Proc. of Int. Conf. on E-Learning and Games, 2009, pp. 50-61 [9] T. C. van Flandern, K. F. Pulkkinen, “Low precision formulae for planetary positions”, Astrophysical Journal Supplement Series, vol. 41, Nov. 1979, p. 391-411 [10] COLLADA 1.5.0 Specification, available online at www.khronos.org/collada, last accessed July 2011 [11] R. Azuma, "A Survey of Augmented Reality". Presence:Teleoperators and Virtual Environemnt 6(4), 1997, pp.355-385. [12] R. Azuma, T. Ronald , "Augmented Reality: Approaches and Technical Challenges". In Fundamentals of Wearable Computers and Augmented Reality, pp. 27-63. [13] M. Zöllner , J. Keil, H. Wuest, D. Pletinckx, "An Augmented Reality Presentation System for Remote Cultural Heritage Sites", Proc. VAST-STAR 2009, pp. 112-116 [14] K. Wang, L. Chen, P. Chu, and Y. Cheng, "A Study on the Design of Augmented Reality User Interfaces for Mobile Learning Systems in Heritage Temples", in Proc. HCI (13), 2009, pp.282-290. [15] Y. Tokusho and S. Feiner, “Prototyping an Outdoor Mobile Augmented Reality Street View Application,” in Int’l Symp. Mixed and Augmented Reality (ISMAR), 2009. [16] M. Zoellner, J. Keil, T. Drevensek, H. Wuest, "Cultural Heritage Layers: Integrating Historic Media in Augmented Reality," Proc. VSMM 2009, pp.193-196 [17] C. Yu and H. Chang, "Personalized LocationBased Recommendation Services for Tour Planning in Mobile Tourism Applications", in Proc. EC-Web, 2009, pp.38-49. [18] M. Kenteris, D. Gavalas, and D. Economou, "An innovative mobile electronic tourist guide application", presented at Personal and Ubiquitous Computing, 2009, pp.103-118. [19] M. Rey-López, A.B. Barragáns-Martínez, A. Peleteiro, F.A. Mikic-Fonte, J.C. Burguillo: "MoreTourism: Mobile recommendations for tourism," Proc. ICCE 2011 pp.347-348

SkyLineDroid lets the user navigate in an augmented space where real objects and virtual reconstructions of ancient buildings coexist. Historical multimedia material is associated to each virtual object and visitors can access it according to their interests and curiosity, improving their cultural, historical and scientific understanding of the site. A prototypal version of the application, based on the historical evolution of the Las Vegas Strip, has been developed. However, the proposed application is versatile, since its structure can be easily adapted to different contexts, like the Imperial Forum in Rome, the Acropolis in Athens or even the 19th century suburbs of Manhattan, simply changing its digital contents.

5 Acknowledgements We would like to thank the UNLV academic staff and the people of the Department of Special Collection as well as the students of the Politecnico di Torino who created, during their Master’s Thesis, the 3D models used in this application. This work has been partially supported by Seat PG with the project LiveItaly. References: [1] G. Richards, “ATLAS Cultural Tourism Survey Summary Report 2007”. ATLAS Cultural Tourism [2] V. Vlahakis et al., “Archeoguide: An Augmented Reality Guide for Archaeological Sites”. IEEE Comput. Graph. Appl., 2002, pp. 52-60 [3] G. Papagiannakis et al., “Mixing virtual and real scenes in the site of ancient Pompeii”. Journal of Vis. and Comp. Animation, 2005, pp.11-24 [4] G. Papagiannakis, G. Singh, and N. MagnenatThalmann, “A survey of mobile and wireless technologies for augmented reality systems”, Comput. Animat. Virtual Worlds, vol 19(1), 2008, pp. 3-22. [5] D. Marimon et al., “MobiAR: Tourist Experiences through Mobile Augmented Reality”, Proc. 2010 NEM Summit, 2010. [6] M.E. Choubassi, O. Nestares, Y. Wu, I. Kozintsev, and H.W. Haussecker, “An Augmented Reality Tourist Guide on Your

ISBN: 978-1-61804-071-8

96