Augmented Reality. Topic relevant selected content from the highest rated wiki entries, typeset, printed and shipped.

Augmented Reality

Topic relevant selected content from the highest rated wiki entries, typeset, printed and shipped. Combine the advantages of up-to-date and in-depth knowledge with the convenience of printed books. A portion of the proceeds of each book will be donated to the Wikimedia Foundation to support their mission: to empower and engage people around the world to collect and develop educational content under a free license or in the public domain, and to disseminate it effectively and globally. The content within this book was generated collaboratively by volunteers. Please be advised that nothing found here has necessarily been reviewed by people with the expertise required to provide you with complete, accurate or reliable information. Some information in this book maybe misleading or simply wrong. The publisher does not guarantee the validity of the information found here. If you need specific advice (for example, medical, legal, financial, or risk management) please seek a professional who is licensed or knowledgeable in that area. Sources, licenses and contributors of the articles and images are listed in the section entitled “References”. Parts of the books may be licensed under the GNU Free Documentation License. A copy of this license is included in the section entitled “GNU Free Documentation License” All used third-party trademarks belong to their respective owners.

Contents Articles Augmented reality

1

Alternate reality game

12

ARQuake

25

Augmented browsing

26

Augmented virtuality

26

Augmented Reality-based testing

28

Bionic contact lens

29

Brain in a vat

30

Camera resectioning

33

Augmented GeoTravel

35

Junaio

37

Layar

38

Spectrek

39

Total Immersion (augmented reality)

40

Wikitude

41

Zombie ShootAR

43

3D computer vision

43

Agent Vi (Agent Video Intelligence)

45

Automated optical inspection

47

Automatic image annotation

52

Automatic number plate recognition

56

Automatic target recognition

69

Check weigher

70

Closed-circuit television

74

Computer stereo vision

83

Content-based image retrieval

84

Digital video fingerprinting

89

GazoPa

92

Gesture recognition

93

Google Goggles

98

Image retrieval

99

Image-based modeling and rendering

101

Intelligent character recognition

102

Iris recognition

103

Machine vision

110

Object detection

112

Optical character recognition

112

Pedestrian detection

115

People counter

116

Physical computing

120

Red light camera

122

Remote sensing

126

Smart camera

132

Traffic enforcement camera

134

Traffic sign recognition

142

Vehicle infrastructure integration

143

Video Content Analysis

148

View synthesis

150

Visual sensor network

151

3D Interaction

152

Accelerator (Internet Explorer)

156

Accelerator table

158

Adjustment handles

158

Alt-Tab

159

Attentive user interface

165

Balloon help

166

Bounce keys

168

Brace matching

168

Brain–computer interface

169

Breadcrumb (navigation)

182

Canned response

184

Capacitive sensing

184

Caret navigation

187

User:Cklokmose/Instrumental Interaction

188

Clipboard (software)

189

Command-line completion

192

Command-line interface

196

Computer-mediated reality

203

Context awareness

205

Cover Flow

208

Crossing-based interface

210

Cursor (computers)

211

Cut, copy, and paste

214

Dasher

218

Delimited search

219

Desktop metaphor

220

Digital puppetry

222

Direct manipulation interface

225

Direct Voice Input

227

Dock (computing)

228

Dock (Mac OS X)

229

Docky

233

Double-click

234

Drag-and-drop

235

Exposé (Mac OS X)

238

Flip page

241

Focus (computing)

242

Form (web)

244

Graffiti (Palm OS)

249

Graffiti 2

251

Grayed out

252

Hands-free computing

253

Incremental search

253

Input method

257

Inspector window

258

Intelligent form

259

Interaction technique

259

Interactive voice response

262

Keyboard shortcut

268

List of dock applications

271

Location awareness

272

Lock key

276

Marching ants

276

Media space

277

Eric Michelman

278

Miller Columns

280

Mixed reality

281

Mnemonics (keyboard)

285

Mode (computer interface)

286

Modifier key

290

Moodbar

292

Mouse chording

293

Mouse keys

295

Multi-factor authentication

298

Multi-touch

299

Multi-touch gestures

304

MultiFinder

305

Multiple document interface

307

Multiple frames interface

311

Navigation controls

312

Optogenetics

312

Page zooming

321

Pen computing

322

Personalization

327

Pie menu

332

Point-and-click

335

Pointing device gesture

337

Post-WIMP

341

Progress indicator

342

Progressive disclosure

343

Projection augmented model

346

Publish and Subscribe

353

Query by Example

354

Responsiveness

356

Rhizome Navigation

357

Screen labeled function keys

358

Scroll lock

359

Scroll wheel

361

Scrolling

363

Selection (user interface)

365

ShapeWriter

366

Silent speech interface

367

SlideIT

369

Slow keys

371

Smart tag (Microsoft)

371

SmartAction

373

Spatial Contextual Awareness

375

Spatial file manager

381

Spatial navigation

384

Speech recognition

385

Split screen (computer graphics)

392

Spoken dialog system

393

Stacks (software)

394

State Bag

395

StickyKeys

395

Stylus (computing)

396

Swiflet

397

Switch access

397

Swype

400

Syntax highlighting

403

T9 (predictive text)

405

Tab (GUI)

407

Tabbing navigation

412

Table of keyboard shortcuts

413

Tangible User Interface

423

Taskbar

427

Telexistence

433

Text entry interface

433

Text-based (computing)

434

Three-pane interface

436

Tiling window manager

436

Timed Text

441

Tip of the day

443

Tooltip

443

Treemapping

444

Triple-click

448

Typeahead

450

The Unfinished Revolution

451

Universal Scrolling

452

User interface

453

User persona

458

Virtual desktop

461

Virtual reality

465

Voice user interface

473

WIMP (computing)

475

Wizard (software)

477

Zooming user interface

478

ARToolKit

482

Bamzooki

484

Cave Automatic Virtual Environment

486

Compositing

492

EyeToy

497

FightBox

502

Gbanga

503

Gbanga Famiglia

505

Head-mounted display

507

Head-up display

512

Immersive technology

521

Interactive art

523

Interactive video compositing

527

Real-time computer graphics

528

Reality–virtuality continuum

530

Simulated reality

532

Simulated reality in fiction

544

Video tracking

549

Virtual (computing)

550

Virtual retinal display

551

Vuzix

555

Xi (alternate reality game)

557

Eon Reality

566

3D Stereo View

567

3DML

567

3DUI

569

Advanced Disaster Management Simulator

569

Advanced Flight Simulation devices

573

Affective Haptics

576

Amateur flight simulation

580

Anarchy Online

584

Ancient Qumran: A Virtual Reality Tour

595

Aspen Movie Map

597

Avatar (computing)

600

BattleTech Centers

606

Battlezone (1980 video game)

609

Maurice Benayoun

614

Bhuvan

618

Blaxxun

621

Blink 3D

622

Blue Brain Project

622

Bump mapping

625

Cal Ripken's Real Baseball

627

Cave5D

631

Cityspace

631

Collaborative Virtual Environments

632

Computer simulation

634

Conquista de Titã

641

Cover system

642

CryEngine

647

CryEngine 2

648

Cyberspace

650

Cyberwar (video game)

657

Cyberworlds

658

Char Davies

658

User:Dheijman/Mirror Worlds

659

Digimask

660

Digital environment

661

Digital Molecular Matter

663

DirectX

665

Do3D

675

Dolby Surround

675

Draw distance

676

DualShock

677

Endocentric environment

684

Environmental audio extensions

684

Eve Online

688

Evolver (3D Avatar Web Portal)

706

Flight simulator

708

Forterra Systems

721

Free look

722

FreeTrack

724

GeoVector

730

GeoWall

733

William Gibson

736

Raymond Goertz

756

Goggles

756

Gouraud shading

759

Graphics processing unit

761

Graphics Turing Test

767

GT Force

767

Haptic technology

772

Havok (software)

778

Eric Howlett

781

Hyperland

783

id Tech 3

786

id Tech 4

793

id Tech 5

797

Imagination Age

799

Immersion (virtual reality)

803

Immersive virtual musical instrument

807

International Stereoscopic Union

808

Internet

809

Inverse kinematics

822

Iwerks Entertainment

825

Kinematic chain

826

Knowledge Machine

827

Languagelab.com

828

Leadwerks Engine

830

LifeClipper

833

List of video games that support EAX

834

User:MBoardmanSC/Motion simulator

839

Mark Stephen Meadows

845

Methods of virtual reality

847

Micro-revenue

848

Mirror world

849

Motion simulator

850

MotionVR

860

mscape

861

Multiverse Network

866

Myst Online: Uru Live

869

NearGlobal

874

NECA Project

875

Next Limit Technologies

877

Normal mapping

877

Nowheremen

879

Object Locative Environment Coordinate System

880

Online community

881

Open Cobalt

885

OpenGL

891

OZ Virtual

907

Tony Parisi (software developer)

908

Particle system

909

Persistent world

912

Mark Pesce

913

Vortex (physics engine)

917

PhysX

919

Polygon mesh

923

Procedural animation

931

Psychoacoustics

932

Quake II engine

936

Ray tracing (graphics)

937

Real Virtuality

944

RealFlow

945

Red Planet (game)

948

RenderWare

951

Rumble Pak

952

Screenless

957

Sculpted prim

958

Sensorama

960

Sensorial transposition

961

Shading

961

SIMNET

964

SixthSense

967

Smeet

969

Social presence theory

972

Space Shuttle Mission 2007

974

SpeedTree

979

StarQuest Online

984

Nicole Stenger

986

Stereotaxy

988

Google Street View

989

Stroker Serpentine

1000

Surround sound

1002

Technology and mental health issues

1010

Telepointer

1011

Terrafly

1012

The Sword of Damocles (virtual reality)

1013

Three-Dimensional Virtual Tourism

1014

Train simulator

1015

Transformed social interaction

1017

Tremor Pak

1019

The Tunnel under the Atlantic

1019

Twinity

1021

Ty Girlz

1025

Unigine

1029

Unreal Engine

1032

Unreal Engine 3

1036

Uru: Ages Beyond Myst

1038

V-business

1045

Vesuite

1045

Virtual Acoustic Space

1047

Virtual art

1048

Virtual artifact

1050

Virtual avatar

1052

Virtual body

1058

Virtual Cocoon

1059

Virtual community

1060

Virtual engineering

1068

Virtual environment software

1070

Virtual fixture

1072

Virtual globe

1075

Virtual graffiti

1081

Virtual Heritage

1084

Virtual Reality and Education Laboratory

1085

Virtual reality cue reactivity

1086

Virtual reality in telerehabilitation

1089

Virtual Reality in the schools

1091

Virtual reality simulation

1092

Virtual reality therapy

1092

Virtual studio

1096

Virtual surgery

1096

Virtual tour

1097

Virtual user interface

1100

Virtual War

1101

Virtual world

1101

Virtual world language learning

1112

Virtuality (gaming)

1126

VirtuSphere

1128

Virtway

1129

Visroom

1131

Volumetric lighting

1131

VR photography

1132

VRML

1133

Web3D

1136

Web3D Consortium

1137

Wire-frame model

1138

Wired glove

1139

X3D

1142

xVRML

1145

XVROS

1146

YoVille (game)

1147

Z800 3DVisor

1148

References Article Sources and Contributors

1149

Image Sources, Licenses and Contributors

1173

Article Licenses License

1183

Augmented reality

1

Augmented reality Augmented reality (AR) is a term for a live direct or an indirect view of a physical, real-world environment whose elements are augmented by computer-generated sensory input, such as sound or graphics. It is related to a more general concept called mediated reality, in which a view of reality is modified (possibly even diminished rather than augmented) by a computer. As a result, the technology functions by enhancing one’s current perception of reality. By contrast, virtual reality replaces the real world with a simulated one. Augmentation is conventionally in real-time and in semantic context with environmental elements, such as sports scores on TV during a match. With the help of advanced AR technology (e.g. adding computer vision and object recognition) the information about the surrounding real world of the user becomes interactive and digitally manipulable. Artificial information about the environment and its objects can be overlaid on the real world. The term augmented reality is believed to have been coined in 1990 by Thomas Caudell, working at Boeing.[1] Research explores the application of computer-generated imagery in live-video streams as a way to enhance the perception of the real world. AR technology includes head-mounted displays and virtual retinal displays for visualization purposes, and construction of controlled environments containing sensors and actuators.

Wikitude World Browser on the iPhone 3GS uses GPS and a solid state compass

AR Tower Defense game on the Nokia N95 smartphone (Symbian OS) uses fiduciary markers

Definition Ronald Azuma offered a definition in 1997.[2] Azuma's definition says that Augmented Reality combines real and virtual, is interactive in real time and is registered in 3D. Additionally Paul Milgram and Fumio Kishino defined Milgram's Reality-Virtuality Continuum in 1994.[3] They describe a continuum that spans an entirely real environment to a purely virtual environment. In between are Augmented Reality (closer to the real environment) and Augmented Virtuality (closer to the virtual environment).

Taxonomy of reality, virtuality, mediality This continuum has been extended into a second dimension that incorporates Mediality.[4] On a graph, the origin R at the bottom Milgram's Continuum left denotes unmodified reality. A continuum across the Virtuality axis V includes reality augmented with additional information (AR), as well as virtual reality augmented by reality (Augmented Virtuality or AV). Unmediated AV simulations are constrained to match the real world behaviorally if not in contents. The mediality axis measures modification of AV, AR and mixes thereof. Moving away from the origin on this axis, the depicted

Augmented reality

2

world becomes increasingly different from reality. Diagonally opposite from R are virtual worlds that have no connection to reality. (at right) It includes the virtuality reality continuum (mixing) but also, in addition to additive effects, also includes modulation and/or diminishment of reality. Mediation encompasses deliberate and/or unintentional modifications.

Examples

Mediated Reality continuum showing four points: Augmented Reality, Augmented Virtuality, Mediated Reality, and Mediated Virtuality on the Virtuality and Mediality axes

Sports AR has become common in sports telecasting. The yellow "first down" line seen in television broadcasts of American football games shows the line the offensive team must cross to receive a first down using the 1st & Ten system. The real-world elements are the football field and players, and the virtual element is the yellow line, which augment the image in real time. Similarly, in ice hockey an AR colored trail shows location and direction of the puck. Sections of Rugby fields and cricket pitches display sponsored images. Swimming telecasts often add a line across the lanes to indicate the position of the current record holder as a race proceeds to allow viewers to compare the current race to the best performance. As an example of mediated (diminished) reality, the network may hide a real message or replace a real ad message with a virtual message.

Other First-person shooter video games can simulate a player's viewpoint using AR to give visual directions to a location, mark the direction distance of another person who is not in line of sight and give information about equipment such as remaining ammunition. This is done using a virtual head-up display. Heads-up displays in AR cars such as some BMW 7 Series models or within airplanes are typically integrated into the windshield. The F-35 Lightning II instead display information in the pilot's helmet mounted display, which allows the pilot to look through the aircraft's walls as if he was floating in space.[5]

Augmented reality

History • 1957-62: Morton Heilig, a cinematographer, creates and patents a simulator called Sensorama with visuals, sound, vibration, and smell.[6] • 1966: Ivan Sutherland invents the head-mounted display and positions it as a window into a virtual world. • 1975: Myron Krueger creates Videoplace to allow users to interact with virtual objects for the first time. • 1989: Jaron Lanier coins the phrase Virtual Reality and creates the first commercial business around virtual worlds. • 1990: Tom Caudell coins the phrase Augmented Reality while at Boeing helping workers assemble cables into aircraft.[7] • 1992: L.B. Rosenberg develops one of the first functioning AR systems, called VIRTUAL FIXTURES, at the U.S. Air Force Research Laboratory—Armstrong, and demonstrates benefits to human performance.[8] [9] • 1992: Steven Feiner, Blair MacIntyre and Doree Seligmann present the first major paper on an AR system prototype, KARMA, at the Graphics Interface conference. A widely cited version of the paper is published in Communications of the ACM in 1993. • 1993: Loral WDL, with sponsorship from STRICOM, performed the first demonstration combining live AR-equipped vehicles and manned simulators. Unpublished paper, J. Barrilleaux, "Experiences and Observations in Applying Augmented Reality to Live Training", 1999.[10] • 1994: Julie Martin creates first Augmented Reality Theater production, Dancing In Cyberspace, funded by the Australia Council for the Arts, features dancers and acrobats manipulating body–sized virtual object in real time, projected into the same physical space and performance plane. The acrobats appeared immersed within the virtual object and environments. The installation used Silicon Graphics computers and Polhemus sensing system. • 1998: Spatial Augmented Reality introduced at University of North Carolina at Chapel Hill by Raskar, Welch, Fuchs.[11] • 1999: Hirokazu Kato (加 藤 博 一 ) created ARToolKit at HITLab, where AR later was further developed by other HITLab scientists, demonstrating it at SIGGRAPH. • 2000: Bruce H. Thomas develops ARQuake, the first outdoor mobile AR game, demonstrating it in the International Symposium on Wearable Computers. • 2008: Wikitude AR Travel Guide launches on Oct. 20, 2008 with the G1 Android phone.[12] • 2009: Wikitude Drive, AR navigation system launched on Oct. 28, 2009 for the Android platform. • 2009: AR Toolkit was ported to Adobe Flash (FLARToolkit) by Saqoosha, bringing augmented reality to the web browser.[13] • 2011: The Nintendo 3DS comes packaged with 6 AR cards which could be used as fiduciary markers. That allow for various mini-games to be played involving virtual objects appearing in camera view.

Technology Hardware The main hardware components for augmented reality are: processor, display, sensors and input devices. These elements, specifically CPU, display, camera and MEMS sensors such as accelerometer, GPS, solid state compass are often present in modern smartphones, which make them prospective AR platforms. Display There are three major display techniques for Augmented Reality: head–mounted displays, handheld displays and spatial displays.

3

Augmented reality Head–mounted A Head Mounted Display (HMD) places images of both the physical world and registered virtual graphical objects over the user's view of the world. The HMD's are either optical see–through or video see–through. Optical see-through employs half-silver mirrors to pass images through the lens and overlay information to be reflected into the user's eyes. The HMD must be tracked with sensor that provides six degrees of freedom. This tracking allows the system to align virtual information to the physical world. The main advantage of HMD AR is the user's immersive experience. The graphical information is slaved to the view of the user.[14] Handheld Handheld displays employ a small display that fits in a user's hand. All handheld AR solutions to date opt for video see-through. Initially handheld AR employed fiduciary markers, and later GPS units and MEMS sensors such as digital compasses and six degrees of freedom accelerometer–gyroscope. Today SLAM markerless trackers such as PTAM are starting to come into use. Handheld display AR promises to be the first commercial success for AR technologies. The two main advantages of handheld AR is the portable nature of handheld devices and ubiquitous nature of camera phones. The disadvantages are the physical constraints of the user having to hold the handheld device out in front of them at all times as well as distorting effect of classically wide-angled mobile phone cameras when compared to the real world as viewed through the eye. [15] Spatial Instead of the user wearing or carrying the display such as with head mounted displays or handheld devices, Spatial Augmented Reality (SAR) [11] makes use of digital projectors to display graphical information onto physical objects. The key difference in SAR is that the display is separated from the users of the system. Because the displays are not associated with each user, SAR scales naturally up to groups of users, thus allowing for collocated collaboration between users. SAR has several advantages over traditional head mounted displays and handheld devices. The user is not required to carry equipment or wear the display over their eyes. This makes spatial AR a good candidate for collaborative work, as the users can see each other’s faces. A system can be used by multiple people at the same time without each having to wear a head mounted display. Spatial AR does not suffer from the limited display resolution of current head mounted displays and portable devices. A projector based display system can simply incorporate more projectors to expand the display area. Where portable devices have a small window into the world for drawing, a SAR system can display on any number of surfaces of an indoor setting at once. The drawbacks, however, are that SAR systems of projectors do not work so well in sunlight and also require a surface on which to project the computer-generated graphics. Augmentations cannot simply hang in the air as they do with handheld and HMD-based AR. The tangible nature of SAR, though, makes this an ideal technology to support design, as SAR supports both a graphical visualisation and passive haptic sensation for the end users. People are able to touch physical objects, and it is this process that provides the passive haptic sensation. [2] [11] [16] [17] [18] Tracking Modern mobile augmented reality systems use one or more of the following tracking technologies: digital cameras and/or other optical sensors, accelerometers, GPS, gyroscopes, solid state compasses, RFID and wireless sensors. These technologies offer varying levels of accuracy and precision. Most important is the position and orientation of the user's head. Tracking the user's hand(s) or a handheld input device can provide a 6DOF interaction technique.[19] Input devices Techniques include the pinch glove,[20] a wand with a button and a smartphone that signals its position and orientation from camera images.

4

Augmented reality Computer The computer analyzes the sensed visual and other data to synthesize and position augmentations.

Software and algorithms A key measure of AR systems is how realistically they integrate augmentations with the real world. The software must derive real world coordinates, independent from the camera, from camera images. That process is called image registration and is part of Azuma's definition of Augmented Reality. Image registration uses different methods of computer vision, mostly related to video tracking. Many computer vision methods of augmented reality are inherited from visual odometry. Usually those methods consist of two parts. First detect interest points, or fiduciary markers, or optical flow in the camera images. First stage can use feature detection methods like corner detection, blob detection, edge detection or thresholding and/or other image processing methods. The second stage restores a real world coordinate system from the data obtained in the first stage. Some methods assume objects with known geometry (or fiduciary markers) present in the scene. In some of those cases the scene 3D structure should be precalculated beforehand. If part of the scene is unknown simultaneous localization and mapping (SLAM) can map relative positions. If no information about scene geometry is available, structure from motion methods like bundle adjustment are used. Mathematical methods used in the second stage include projective(epipolar) geometry, geometric algebra, rotation representation with exponential map, kalman and particle filters, nonlinear optimization, robust statistics.

Applications Applications as of 2011 Advertising: Marketers started to use AR to promote products via interactive AR applications. For example, at the 2008 LA Auto Show, Nissan unveiled the concept vehicle Cube and presented visitors with a brochure which, when held against a webcam, showed alternate versions of the vehicle.[21] In August 2009, Best Buy ran a circular with an augmented reality code that allowed users with a webcam to interact with the product in 3D.[22] In 2010 Walt Disney used mobile AR to connect a movie experience to outdoor advertising.[23] Task support: Complex tasks such as assembly, maintenance, and surgery can be simplified by inserting additional information into the field of view. For example, labels can be displayed on parts of a system to clarify operating instructions for a mechanic who is performing maintenance on the system.[24] [25] AR can include images of hidden objects, which can be particularly effective for medical diagnostics or surgery. Examples include a virtual X-ray view based on prior tomography or on real time images from ultrasound and microconfocal probes[26] or open NMR devices. AR can enhance viewing a fetus inside a mother's womb.[27] See also Mixed reality. Navigation: AR can augment the effectiveness of navigation devices. For example, building navigation can be enhanced to aid in maintaining industrial plants. Outdoor navigation can be augmented for military operations or disaster management. Head-up displays or personal display glasses in automobiles can provide navigation and traffic information. Head-up displays are currently used in fighter jets. These systems include full interactivity, including gaze tracking. Industrial: AR can be used to compare digital mock-ups with physical mock-ups for efficiently finding discrepancies between them. It can safeguard digital data together with existing real prototypes, and thus reduce the number of real prototypes and improve the quality of the final product. Military and emergency services: Wearble AR can provide information such as instructions, maps, enemy locations, and fire cells.

5

Augmented reality Art: AR can help create art in real time integrating reality such as painting, drawing and modeling. AR art technology has helped disabled individuals to continue pursuing their passion.[28] Architecture: AR can simulate planned construction projects.[29] Sightseeing: Guides can include labels or text related to the objects/places visited. With AR, users can rebuild ruins, buildings, or even landscapes as they previously existed.[30] Collaboration: AR can help facilitate collaboration among distributed team members via conferences with real and virtual participants.[31] Entertainment and education: AR can create virtual objects in museums and exhibitions, theme park attractions,[32] games[33] [34] and books.[35] Performance: AR can enhance concert and theater performances. For example, artists can allow listeners to augment their listening experience by adding their performance to that of other bands/groups of users.[36] [37] [38] Translation: AR systems can provide dynamic subtitles in the user's language.[39] [40]

Potential applications Possible extensions include: • Devices: Create new applications that are physically impossible in "real" hardware, such as 3D objects interactively changing their shape and appearance based on the current task or need. • Multi-screen simulation: Display multiple application windows as virtual monitors in real space and switch among them with gestures and/or redirecting head and eyes. A single pair of glasses could "surround" a user with application windows. • Holodecks: Enhanced media applications, like pseudo holographic virtual screens and virtual surround cinema. • Automotive: eye-dialing, navigation arrows on roadways • "X-ray vision" • Furnishings: plants, wallpapers, panoramic views, artwork, decorations, posters, illumination etc. For example, a virtual window could show a live feed of a camera placed on the exterior of the building, thus allowing the user to toggle a wall's transparency. • Public displays: Window dressings, traffic signs, Christmas decorations, advertisements. • Gadgets: Clock, radio, PC, arrival/departure board at an airport, stock ticker, PDA, PMP, informational posters/fliers/billboards. • Group-specific feeds: For example, a construction manager could display instructions including diagrams at specific locations. Patrons at a public event could subscribe to a feed of directions and/or program notes. • Speech synthesis: Render location/context-specific information via spoken words. • Prospecting: In hydrology, ecology, and geology, AR can be used to display an interactive analysis of terrain characteristics. Users can collaboratively modify and analyze, interactive three-dimensional maps.

Notable researchers • Ivan Sutherland invented the first AR head mounted display at Harvard University. • Steven Feiner, Professor at Columbia University, is a leading pioneer of augmented reality, and author of the first paper on an AR system prototype, KARMA (the Knowledge-based Augmented Reality Maintenance Assistant), along with Blair MacIntyre and Doree Seligmann.[41] • L.B. Rosenberg developed one of the first known AR systems, called Virtual Fixtures, while working at the U.S. Air Force Armstrong Labs in 1991, and published first study of how an AR system can enhance human performance.[8] [9] • Mark Billinghurst and Daniel Wagner jump started the field of AR on mobile phones. They developed the first marker tracking systems for mobile phones and PDAs.[42]

6

Augmented reality • Bruce H. Thomas and Wayne Piekarski develop the Tinmith system in 1998.[43] They along with Steve Feiner with his MARS system pioneer outdoor augmented reality.

Conferences • • • • • • • • • • • • •

1st International Workshop on Augmented Reality, IWAR'98, San Francisco, Nov. 1998. 2nd International Workshop on Augmented Reality (IWAR'99 [44]), San Francisco, Oct. 1999. 1st International Symposium on Mixed Reality (ISMR'99), Yokohama, Japan, March 1999. 2nd International Symposium on Mixed Reality (ISMR'01), Yokohama, Japan, March 2001. 1st International Symposium on Augmented Reality (ISAR 2000 [45]), Munich, Oct. 2000. 2nd International Symposium on Augmented Reality (ISAR 2001 [46]), New York, Oct. 2001. 1st International Symposium on Mixed and Augmented Reality (ISMAR 2002 [47]), Darmstadt, Oct. 2002. 2nd International Symposium on Mixed and Augmented Reality (ISMAR 2003 [48]), Tokyo, Oct. 2003. 3rd International Symposium on Mixed and Augmented Reality (ISMAR 2004 [49]), Arlington, VA, Nov. 2004. 4th International Symposium on Mixed and Augmented Reality (ISMAR 2005 [50]), Vienna, Oct. 2005. 5th International Symposium on Mixed and Augmented Reality (ISMAR 2006 [51]) Santa Barbara, Oct. 2006. 6th International Symposium on Mixed and Augmented Reality (ISMAR 2007 [52]) Nara, Japan, Nov. 2007. 7th International Symposium on Mixed and Augmented Reality (ISMAR 2008 [53]) Cambridge, United Kingdom, Sep. 2008. • 8th International Symposium on Mixed and Augmented Reality (ISMAR 2009 [54]) Orlando, Oct. 2009. • Augmented Reality Developer Camp (AR DevCamp [55]) in Mountain View, Dec. 2009. • 9th International Symposium on Mixed and Augmented Reality (ISMAR 2010 [56]) Seoul, Korea, Oct. 2010. • 10th International Symposium on Mixed and Augmented Reality (ISMAR 2011 [57]) Basel, Switzerland Oct. 2011

Software Open source software • ARToolKit, an open source (dual-license: GPL, commercial) C-library to create augmented reality applications; was ported to many different languages and platforms like Android, Flash or Silverlight; very widely used in augmented reality related projects • ArUco [58], a minimal library for augmented reality applications based on OpenCv; licenses: BSD, Linux, Windows • mixare [59], Open-source (GPLv3) augmented reality engine for Android and iPhone; works as an autonomous application and for developing other implementations [60] , Open Mobile Augmented Reality component framework for the Symbian platform, released under • OpenMAR EPL; website is down but there is some information here [61] • Argon [62], Augemented reality browser by Georgia Tech's GVU Center that uses a mix of KML and HTML/JavaScript/CSS to allow developing AR applications; any web content (with appropriate meta-data and properly formatted) can be converted into AR content; currently available only for the iPhone [63], website is down • Goblin [64], BSD licensed, Microsoft XNA based • PTAM [65], Non-commercial use only • ARTag [66], Downloads unavailable after 12/21/2010 due to licensing restrictions

7

Augmented reality

Books • Woodrow Barfield, and Thomas Caudell, eds. Fundamentals of Wearable Computers and Augmented Reality. Mahwah, NJ: Lawrence Erlbaum, 2001. ISBN 0-8058-2901-6. • Oliver Bimber and Ramesh Raskar. Spatial Augmented Reality: Merging Real and Virtual Worlds. A K Peters, 2005. ISBN 1-56881-230-2. • Michael Haller, Mark Billinghurst and Bruce H. Thomas. Emerging Technologies of Augmented Reality: Interfaces and Design. Idea Group Publishing, 2006. ISBN 1-59904-066-2 , publisher listing [67] • Rolf R. Hainich. "The end of Hardware : A Novel Approach to Augmented Reality" [68] 2nd ed.: Booksurge, 2006. ISBN 1-4196-5218-4. 3rd ed. ("Augmented Reality and Beyond"): Booksurge, 2009, ISBN 1-4392-3602-X. • Stephen Cawood and Mark Fiala. Augmented Reality: A Practical Guide, 2008, ISBN 1-934356-03-4

In popular culture Television, film • The television series Dennō Coil depicts a near-future where children use AR glasses to enhance their environment with games and virtual pets. • In the Terminator movie series, all Terminator models, beginning with T-800 series, use augmented reality systems to "see". • The television series Firefly depicts numerous AR applications, including a real-time medical scanner which allows a doctor to use his hands to manipulate a detailed and labeled projection of a patient's brain. • In the 1993 ABC miniseries Wild Palms, a Scientology-like organization used holographic projectors to overlay virtual reality images over physical reality. • In the movie Iron Man, Tony Stark (Robert Downey Jr.) uses an augmented reality system to design his super-powered suit. The suit itself also uses augmented reality technology. • In the Philippines, during their first automated elections (2010), ABS-CBN News and Current Affairs used augmented reality during the counting of votes for all National and Local Candidates and in delivering news reports. ABS-CBN still uses augmented reality in its TV Patrol news programs. • In Minority Report Tom Cruise stands in front of a supercomputer using AR technology. • In the movie Mission Impossible 2, Tom Cruise uses Augmented Reality technology via a set of sunglasses he wears to debrief himself of his forthcoming mission,Chimera,after he completes climbing a mountain at the very outset of the movie. • In the movie,"RoboCop", RoboCop uses Augmented Reality tech via his Head-mounted display to get into the details of a particular person or status quo. • In the movie, They Live, aliens on Earth use a hypnotic radio frequency causing the human population to see generated images and adverts which mask billboards that actually contain subliminal messaging. Curiously, it takes wearing a Head-mounted display (in this case, a pair of sunglasses) in order not to be able to see the AR.[69] • NBC School Pride debuts AR alive: Letters alive in the Communication & Media Arts High School in Detroit, Michigan [70]

Literature • The books Halting State by Charles Stross and Rainbows End by Vernor Vinge and the Daemon series by Daniel Suarez include augmented reality primarily in the form of virtual overlays over the real world. Halting State mentions Copspace, which is used by cops, and the use by gamers to overlay their characters onto themselves during a gaming convention. Rainbows End mentions outdoor overlays based on popular fictional universes from H. P. Lovecraft and Terry Pratchett among others. The Daemon series features the "Darknet", which connects human followers and allows them to create their own ranking system and economy among other features.

8

Augmented reality • The term "Geohacking" has been coined by William Gibson in his book Spook Country, where artists use a mix of GPS and 3D graphics technology to embed rendered meshes in real world landscapes. • In The Risen Empire, by Scott Westerfeld, most - if not all - people have their own "synesthesia". An AR menu unique to the user that is projected in front of them, but they can only see their own synesthesia menus. It is controlled by hand gestures, blink patterns, where the user is looking, clicks of the tongue, etc. • In the Greg Egan novel Distress, the 'Witness' software used to record sights and sounds experienced by the user can be set-up to scan what the user is seeing and highlight people the user is looking out for. • In the Revelation Space series of novels, Alastair Reynolds characters frequently employ "Entoptics" which are essentially a highly developed form of augmented reality, going so far as to entirely substitute natural perception. • The book The California Voodoo Game by Larry Niven and Steve Barnes, the game players use LCD displays for what the book calls Dreamtime technology to add virtual overlays to the real world.

Games • The table top role-playing game, Shadowrun, introduced AR into its game world. Most of the characters in the game use viewing devices to interact with the AR world most of the time. • Cybergeneration, a table top role-playing game by R. Talsorian, includes "virtuality", an augmented reality created through v-trodes, cheap, widely available devices people wear at their temples. • In the video game Heavy Rain, Norman Jayden, an FBI profiler, possesses a set of experimental augmented reality glasses called an "Added Reality Interface", or ARI. It allows him to rapidly investigate crime scenes and analyze evidence. • In Dead Space the RIG worn by Isaac Clarke is thoroughly equipped with augmented reality technology, including a navigation system that projects a line along the best route to his destination, and a system that displays images, video and text in front of him.

Comics • The free minicomic Le Sketch - with sketches by Jorge Alderete, Matt Madden, Nick Bertozzi, Peter Kuper, Marcellus Hall, Steven Weissman, Craig Atkinson and Matthew Thurber - published an Augmented Reality issue in February 2011.

Tools see Augmented reality#Software

References [1] "The interactive system is no longer a precise location, but the whole environment; interaction is no longer simply a face-to-screen exchange, but dissolves itself in the surrounding space and objects. Using an information system is no longer exclusively a conscious and intentional act."Brian X. Chen (2009-08-25). "If You’re Not Seeing Data, You’re Not Seeing" (http:/ / www. wired. com/ gadgetlab/ tag/ augmented-reality/ ). Wired Magazine. . Retrieved 2009-08-26. [2] R. Azuma, A Survey of Augmented Reality (http:/ / www. cs. unc. edu/ ~azuma/ ARpresence. pdf) Presence: Teleoperators and Virtual Environments, pp. 355–385, August 1997. [3] P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays (http:/ / vered. rose. utoronto. ca/ people/ paul_dir/ IEICE94/ ieice. html) IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994. [4] Mediated Reality with implementations for everyday life (http:/ / wearcam. org/ presence_connect/ ), 2002 August 6th, Presence Connect, the on line companion to the MIT Press journal PRESENCE: Teleoperators and Virtual Environments, MIT Press [5] "F-35 Distributed Aperture System EO DAS." (http:/ / www. youtube. com/ user/ F35JSFVideos#play/ uploads/ 7/ CwvnhFgzIKI) Youtube.com. Retrieved: 07 October 2010. [6] http:/ / www. google. com/ patents?q=3050870 [7] Tom Caudell (http:/ / www. ece. unm. edu/ morenews/ profile_caudell. html) [8] L. B. Rosenberg. The Use of Virtual Fixtures As Perceptual Overlays to Enhance Operator Performance in Remote Environments. Technical Report AL-TR-0089, USAF Armstrong Laboratory, Wright-Patterson AFB OH, 1992.

9

Augmented reality [9] L. B. Rosenberg, "The Use of Virtual Fixtures to Enhance Operator Performance in Telepresence Environments" SPIE Telemanipulator Technology, 1993. [10] Experiences and Observations in Applying Augmented Reality to Live Training (http:/ / jmbaai. com/ vwsim99/ vwsim99. html) [11] Ramesh Raskar, Greg Welch, Henry Fuchs Spatially Augmented Reality (http:/ / www. cs. unc. edu/ ~raskar/ Office), First International Workshop on Augmented Reality, Sept 1998 [12] Wikitude AR Travel Guide (http:/ / www. youtube. com/ watch?v=8EA8xlicmT8) [13] Saqoosha (http:/ / saqoosha. net/ en/ flartoolkit/ start-up-guide/ ) [14] The most common products employed are as follows: MicroVision Nomad, Sony Glasstron, Vuzix AR and I/O Displays. Vuzix AR (http:/ / www. vuzix. com/ AR_Site/ default. asp) [15] Feiner, Steve. "Augmented reality: a long way off?" (http:/ / www. pocket-lint. com/ news/ 38869/ augmented-reality-interview-steve-feiner). AR Week. Pocket-lint. . Retrieved 3 March 2011. [16] David Drascic of the University of Toronto is a developer of ARGOS: A Display System for Augmenting Reality (http:/ / vered. rose. utoronto. ca/ people/ david_dir/ CHI93/ CHI93. full. html). David also has a number of AR related papers on line, accessible from his home page (http:/ / vered. rose. utoronto. ca/ people/ David. html). [17] Augmented reality brings maps to life (http:/ / www. newscientist. com/ article/ dn7695) July 19, 2005 [18] Feiner, Steve. "Augmented reality: a long way off?" (http:/ / www. pocket-lint. com/ news/ 38802/ augmented-reality-maintenance-and-repair). AR Week. Pocket-lint.com. . Retrieved 3 March 2011. [19] Stationary systems can employ 6DOF track systems such as Polhemus, ViCON, A.R.T, or Ascension. [20] Tinmith [21] company website (http:/ / www. nissanusa. com/ cube/ ) [22] Vlad Savov. "Best Buy goes 3D, even augmented reality isn't safe from advertising" (http:/ / www. engadget. com/ 2009/ 08/ 06/ best-buy-goes-3d-even-augmented-reality-isnt-safe-from-adverti). . [23] AR at Disney (http:/ / adsoftheworld. com/ media/ outdoor/ walt_disney_prince_of_persia_the_gods_have_a_plan_for_you) [24] The big idea:Augmented Reality (http:/ / ngm. nationalgeographic. com/ big-idea/ 14/ augmented-reality-pg1) [25] Steve Henderson, Steven Feiner. "ARMAR:Augmented Reality for Maintenance and Repair (ARMAR)" (http:/ / graphics. cs. columbia. edu/ projects/ armar/ index. htm). . Retrieved 2010-01-06. [26] Peter Mountney, Stamatia Giannarou, Daniel Elson and Guang-Zhong Yang. "Optical Biopsy Mapping for Minimally Invasive Cancer Screening. In proc MICCAI(1), 2009, pp. 483-490" (http:/ / www. sciweavers. org/ external. php?u=http:/ / www. doc. ic. ac. uk/ ~pmountne/ publications/ MICCAI%202009. pdf& p=springer). . Retrieved 2010-07-07. [27] "UNC Ultrasound/Medical Augmented Reality Research" (http:/ / www. cs. unc. edu/ Research/ us/ ). . Retrieved 2010-01-06. [28] One such example of this phenomenon is called Eyewriter that was developed in 2009 by Zachary Lieberman and a group formed by members of Free Art and Technology (FAT), OpenFrameworks and the Graffiti Research Lab to help a graffiti artist, who became paralyzed, draw again. Zachary Lieberman. "The Eyewriter" (http:/ / www. eyewriter. org/ ). . Retrieved 2010-04-27. [29] Anish Tripathi. "Augmented Reality: An Application for Architecture" (http:/ / www. usc. edu/ dept/ architecture/ mbs/ thesis/ anish/ ). . Retrieved 2010-01-06. [30] Patrick Dähne, John N. Karigiannis. "Archeoguide: System Architecture of a Mobile Outdoor Augmented Reality System" (http:/ / portal. acm. org/ citation. cfm?id=854948). . Retrieved 2010-01-06. [31] The Hand of God is a good example of a collaboration system. Aaron Stafford, Wayne Piekarski, and Bruce H. Thomas. "Hand of God" (http:/ / www. hog3d. net/ ). . Retrieved 2009-12-18. [32] Theme park attraction:Cadbury World [33] ARQuake [34] Eye of Judgement [35] Jose Fermoso. "Make Books ‘Pop’ With New Augmented Reality Tech" (http:/ / www. wired. com/ gadgetlab/ 2008/ 10/ im-in-yur-physi/ ). . Retrieved 2010-10-01. [36] Pop group Duran Duran included interactive AR projections into their stage show during their 2000 Pop Trash concert tour. Pair, J., Wilson, J., Chastine, J., Gandy, M. " The Duran Duran Project: The Augmented Reality Toolkit in Live Performance (http:/ / ieeexplore. ieee. org/ xpl/ freeabs_all. jsp?tp=& arnumber=1107010)". The First IEEE International Augmented Reality Toolkit Workshop, 2002. ( photos and video (http:/ / www. jarrellpair. com/ ddar/ index. html)) [37] Sydney band Lost Valentinoslaunched the world's first interactive AR music video on 16 October 2009, where users could print out 5 markers representing a pre-recorded performance from each band member which they could interact with live and in real-time via their computer webcam and record as their own unique music video clips to share viaYouTube Gizmodo: Sydney Band Uses Augmented Reality For Video Clip (http:/ / www. gizmodo. com. au/ 2009/ 10/ sydney-band-uses-augmented-reality-for-video-clip/ ) [38] cnet: Augmented reality in Aussie film clip (http:/ / www. cnet. com. au/ augmented-reality-in-aussie-film-clip-339299097. htm) [39] iPhone applicationWord Lens injects subtitles into the desired language in video. (http:/ / techcrunch. com/ 2010/ 12/ 16/ world-lens-translates-words-inside-of-images-yes-really) Alexia Tsotsis "Word Lens Translates Words Inside of Images. Yes Really." TechCrunch (December 16, 2010) [40] (http:/ / www. economist. com/ blogs/ gulliver/ 2010/ 12/ instant_translation) N.B. "Word Lens: This changes everything" The Economist: Gulliver blog (December 18, 2010) [41] "Knowledge-based augmented reality" (http:/ / portal. acm. org/ citation. cfm?id=159587). ACM. July, 1993. .

10

Augmented reality [42] Wagner, Daniel (September 29, 2009). "First Steps Towards Handheld Augmented Reality" (http:/ / portal. acm. org/ citation. cfm?id=946910). ACM. . Retrieved 2009-09-29. [43] Pikarski, Wayne and Thomas, Bruce (October 1, 2001). "Tinmith-Metro: New Outdoor Techniques for Creating City Models with an Augmented Reality Wearable Computer" (http:/ / www. computer. org/ portal/ web/ csdl/ doi/ 10. 1109/ ISWC. 2001. 962093). IEEE. . Retrieved 2010-11-09. [44] http:/ / ieeexplore. ieee. org/ xpl/ tocresult. jsp?isNumber=17413 [45] http:/ / ieeexplore. ieee. org/ xpl/ tocresult. jsp?isNumber=19065 [46] http:/ / ieeexplore. ieee. org/ xpl/ tocresult. jsp?isNumber=20930 [47] http:/ / ieeexplore. ieee. org/ xpl/ tocresult. jsp?isNumber=24594 [48] http:/ / ieeexplore. ieee. org/ xpl/ tocresult. jsp?isNumber=27815 [49] http:/ / ieeexplore. ieee. org/ xpl/ tocresult. jsp?isnumber=30137 [50] http:/ / ieeexplore. ieee. org/ xpl/ tocresult. jsp?isnumber=32972 [51] http:/ / www. ismar06. org [52] http:/ / www. ismar07. org/ [53] http:/ / ismar08. org/ wiki/ doku. php [54] http:/ / www. ismar09. org [55] http:/ / www. ardevcamp. org [56] http:/ / www. ismar10. org [57] http:/ / www. ismar11. org [58] http:/ / www. uco. es/ investiga/ grupos/ ava/ node/ 26 [59] http:/ / www. mixare. org/ [60] http:/ / OpenMAR. org [61] http:/ / www. symlab. org/ wiki/ index. php/ OpenMAR [62] http:/ / argon. gatech. edu/ [63] http:/ / itunes. apple. com/ app/ argon/ id396105088 [64] http:/ / goblinxna. codeplex. com/ [65] http:/ / www. robots. ox. ac. uk/ ~gk/ PTAM/ [66] http:/ / www. artag. net/ [67] http:/ / www. igi-pub. com/ books/ details. asp?id=6456 [68] http:/ / www. theendofhardware. org [69] Miles, Stuart. "Top 10 uses of augmented reality in the movies" (http:/ / www. pocket-lint. com/ news/ 38805/ best-augmented-reality-in-movies). AR Week. Pocket-lint.com. . Retrieved 1 March 2011. [70] Dybis, Karen (August 6, 2010). "Some Genuine Detroit 'School Pride'" (http:/ / detroit. blogs. time. com/ 2010/ 08/ 06/ some-genuine-detroit-school-pride/ ). Time. . Retrieved October 7, 2010.

11

Alternate reality game

Alternate reality game An alternate reality game (ARG) is an interactive narrative that uses the real world as a platform, often involving multiple media and game elements, to tell a story that may be affected by participants' ideas or actions. The form is defined by intense player involvement with a story that takes place in real-time and evolves according to participants' responses, and characters that are actively controlled by the game's designers, as opposed to being controlled by artificial intelligence as in a computer or console video game. Players interact directly with characters in the game, solve plot-based challenges and puzzles, and often work together with a community to analyze the story and coordinate real-life and online activities. ARGs generally use multimedia, such as telephones, email and mail but rely on the Internet as the central binding medium. ARGs are growing in popularity, with new games appearing regularly and an increasing amount of experimentation with new models and subgenres. They tend to be free to play, with costs absorbed either through supporting products (e.g. collectible puzzle cards fund Perplex City) or through promotional relationships with existing products (for example, I Love Bees was a promotion for Halo 2, and the Lost Experience and FIND815 promoted the television show Lost). However, pay-to-play models are not unheard of. ARGs are now being recognized by the mainstream entertainment world: The Fallen Alternate Reality game [1], produced in the fall of 2007 by Xenophile Media Inc.[2] was awarded a Primetime Emmy for Outstanding Achievement for an Interactive Television Program. Xenophile Media Inc.'s ReGenesis Extended Reality Game [3] won an International Interactive Emmy Award in 2007 and in April 2008 The Truth About Marika won the iEmmy for Best interactive TV service.[4] The British Academy of Film and Television Arts recognises Interactivity as a category in the British Academy Television Awards.

Defining alternate reality gaming There is a great deal of debate about how to define the term "alternate reality game" and what should be included or excluded by the definition. Although it doesn't really make sense to use the word "alternate" (the correct word should be "alternative"), it is generally accepted that "alternate reality" is entrenched. Sean Stacey, founder of the website Unfiction [5], has suggested that the best way to define the genre was not to define it, and instead locate each game on three axes (ruleset, authorship and coherence) in a sphere of "chaotic fiction" that would include works such as [6] the Uncyclopedia and street games like SF0 as well. While addressing all of the various attempts at definitions and arguments for and against them is beyond the scope of this article, defining a few terms unique to ARG parlance, identifying precursors and influences on the development of the genre, and comparing and contrasting ARGs to other similar forms of entertainment may be helpful in aiding understanding of the form.

Unique terminology Among the terms essential to understand discussions about ARGs are: • Puppetmaster - A puppetmaster or "PM" is an individual involved in designing and/or running an ARG. Puppetmasters are simultaneously allies and adversaries to the player base, creating obstacles and providing resources for overcoming them in the course of telling the game's story. Puppetmasters generally remain behind the curtain while a game is running. The real identity of puppet masters may or may not be known ahead of time. • The Curtain - The curtain is generally a metaphor for the separation between the puppetmasters and the players. This can take the traditional form of absolute secrecy regarding the puppetmasters' identities and involvement with the production, or refer merely to the convention that puppetmasters do not communicate directly with players through the game, interacting instead through the characters and the game's design.

12

Alternate reality game • Rabbithole - Also known as a Trailhead. A Rabbithole marks the first website, contact, or puzzle that starts off the ARG. • Trailhead - A deliberate clue which enables a player to discover a way into the game. Most ARGs employ a number of trailheads in several media, to maximise the probability of people discovering the game. Some trailheads may be covert, others may be thinly-disguised adverts. • This Is Not A Game (TINAG) - Setting the ARG form apart from other games is the This Is Not A Game aesthetic, which dictates that the game doesn't behave like a game: phone numbers mentioned in the ARG, for example, should actually work, and the game should not provide an overtly-designated playspace or ruleset to the players.

Similarities to and differences from other forms of entertainment • Computer/console/video games. While ARGs generally use the internet as a central binding medium, they are not played exclusively on a computer and usually do not require the use of special software or interfaces. Non-player characters in ARGs are controlled in real-time by the puppetmasters, not computer AI. • Role-playing games (RPGs) and Live action role-playing games (LARPs). The role of the puppetmaster in creating ARG narratives and the puppetmaster's relationship with an ARG's players bears a great deal of similarity to the role of a game master, gamemaster or referee in a role-playing game. However, the role of the players is quite different. Most ARGs do not have any fixed rules—players discover the rules and the boundaries of the game through trial and error—and do not require players to assume fictional identities or roleplay beyond feigning belief in the reality of the characters they interact with (even if games where players play 'themselves' are a long standing variant on the genre).[7] • Massively multiplayer online role-playing games (MMORPGs). As outlined above with computer games and traditional role-playing games, non-player characters in ARGs are controlled by real people in real time, not by computer AI; ARGs do not generally require special software or interfaces to play; the games do not require players to roleplay or create characters or avatars; and ARGs generally use multiple media and real life in addition to the internet to distribute their narratives. • Viral marketing/internet hoaxes. While ARGs are often used as a type of viral marketing, they diverge sharply from the philosophy behind "sponsored consumers" or other viral marketing practices that attempt to trick consumers into believing that planted shills for a product are other independent consumers. Similarly, they also diverge from sites or narratives that genuinely try to convince visitors that they are what they claim to be. Puppetmasters generally leave both subtle and overt clues to the game's fictional nature and boundaries where players can find them (e.g. through clearly fictional names on site registrations) and many ARGs openly flaunt obviously fictional plots. The puppetmasters of the genre's seminal example, the Beast,(see below)[8] made it a point of pride never to pretend to be players in order to solicit publicity or nudge players along, and the Terms of Service of Unfiction, the central community site for the ARG genre, strictly prohibit individuals involved in [9] creating games from posting about them without disclosing their involvement.

Influences and precursors Due to factors like the curtain, attempts to begin games with "stealth launches" to fulfill the TINAG aesthetic, and the restrictive non-disclosure agreements governing how much information may be revealed by the puppetmasters of promotional games, the design process for many ARGs is often shrouded in secrecy, making it difficult to discern the extent to which they have been influenced by other works. In addition, the cross-media nature of the form allows ARGs to incorporate elements of so many other art forms and works that attempting to identify them all would be a nearly-impossible task.

13

Alternate reality game Possible inspirations from fiction and other art forms G. K. Chesterton's 1905 short story "The Tremendous Adventures of Major Brown" [10] (part of a collection entitled The Club of Queer Trades [11]) seems to predict the ARG concept, as does John Fowles' 1965 novel The Magus. The performance artists in Delany's science fiction novel Triton (published in 1976) appear to be playing a type of ARG. Ludic texts such as the popular Choose Your Own Adventure children's novels may also have provided some inspiration. Reader-influenced online fiction such as AOL's QuantumLink Serial provides a model that incorporates audience influence into the storytelling in a manner similar to that of ARGs, as do promotional online games like Wizards of the Coast's Webrunner games. Other possible antecedents include performance art and other theatrical forms that attempt to directly engage the audience. Due to the influence the Beast exerted over the form of later ARGs and the willingness of its creators to talk about its development, its sources of inspiration are both particularly relevant to the evolution of the modern ARG and somewhat more verifiable than other possible antecedents. Elan Lee, one of its creative principals, cites the 1997 movie The Game as an inspiration, as well as the Beatles' "Paul is dead" phenomenon. Sean Stewart, another of the three principal designers, notes that designing and running an ARG bears some similarities to running an RPG, and the influence of that particular game form is further suggested by the fact that Jordan Weisman, the game's third main designer, was also the founder of leading RPG company FASA. Stewart also noted that the sort of "creative, collaborative, enthusiastic scavengering behavior"[12] upon which the Beast depended has its antecedents outside the arts: the Beast just "accidentally re-invented Science as pop culture entertainment."[13] The conspiracy in Thomas Pynchon's The Crying of Lot 49 may be an ARG set up by Pierce Inverarity to bedevil Oedipa Maas, as may be the hallucinatory Turkish frontier across which A.W. Hill's Stephan Raszer tracks his quarry in the current literary thriller Nowhere-Land.

Basic design principles of ARGs ARGs are sometimes described as the first narrative art form native to the internet, because their storytelling relies on the two main activities conducted there: searching for information, and sharing information. • Storytelling as archaeology. Instead of presenting a chronologically unified, coherent narrative, designers scatter pieces of the story across the Internet and other media, allowing players to reassemble it, supply connective tissue and determine what it means. • Platformless narrative. Stories are not bound to a single medium, but exist independently and use whatever media is available to make itself heard. • Designing for a hive mind. While it might be possible to follow games individually, designs are directed at a collective of players that share information and solutions almost instantly, and incorporate individuals possessing almost every conceivable area of expertise. While games might initially attract a small group of participants, as the participants come across new challenges they try to find others with the knowledge needed to overcome an obstacle. • A whisper is sometimes louder than a shout. Rather than openly promoting games and trying to attract participation by "pushing" it toward potential players, designers attempt to "pull" players to the story by engaging in over-the-top secrecy, have elements of the game "warn" players away from them, and eschew traditional marketing channels. Designers do not communicate about the game with players or press while it is in play. • The "this is not a game" (TINAG) aesthetic. ARGs themselves do not acknowledge that they are games. They do not have an acknowledged ruleset for players; as in real-life, they determine the "rules" either through trial and error or by setting their own boundaries. Narratives present a fully-realized world: any phone number or email address mentioned works, and any website acknowledged exists. Games take place in real-time and are not replayable. Characters function like real people, not game pieces, respond authentically, and are controlled by real people, not by computer AI. Some events involve meetings or live phone calls between players and actors.

14

Alternate reality game • Real life as a medium. Games use players' lives as a platform. Players are not required to build a character or role-play being someone other than themselves. They might unexpectedly overcome a challenge for the community simply because of the real-life knowledge and background they possessed. Participants are constantly on the lookout for clues embedded in everyday life. • Collaborative storytelling. While the puppetmasters control most of the story, they incorporate player content and respond to players' actions, analysis and speculation by adapting the narrative and intentionally leave "white space" for the players to fill in. • Not a hoax. While the TINAG aesthetic might seem on the surface to be an attempt to make something indistinguishable from real life, there are both subtle and overt metacommunications in place to reveal a game's framework and most of its boundaries.

Development and history Early examples The first example of an ARG style game was Dreadnot,[14] a (non-commercial) web game produced with a grant from the San Francisco Chronicle and published on sfgate.com in 1996. It included most of the techniques above that would, in upcoming years, become the standard for most ARG games. The game included working voice mail phone numbers for characters, clues in the source code, character email addresses, off-site websites, real locations in San Francisco, real people (including then Mayor Willie Brown), and of course a fictional mystery. In 1997, a year prior to the release of the Douglas Adams computer game Starship Titanic, The Digital Village launched a web site purporting to be that of an intergalactic travel agency called Starlight Travel, which in the game is the Starship Titanic's parent company. The site combined copious amounts of Monty Python-esque writing (by Michael Bywater) with ARG-type interactivity. The marketing for the 1999 movie The Blair Witch Project resembled ARGs in many ways (and some of its makers went on to create the 2005 Audi promotional ARG The Art of the Heist), expanding the world of the movie online, adding backstory, and treating the fiction as reality through real-world media such as fliers and a fake documentary on the Sci-Fi Channel. However, perhaps in part due to the subject material and the absence of overt metacommunications that this was fiction, it also resembles an internet hoax or attempt to create an urban legend. Pervasive play games like the Go Game and the Nokia Game also incorporated many elements similar to ARGs (although they tended to lack the narrative element central to ARGs) and prefigured the public play components of large-scale corporate ARGs like I Love Bees, The Art of the Heist and Last Call Poker. Electronic Arts' Majestic began development in 1999, although it didn't launch until after the Beast had concluded, in 2001. Featuring phone calls, emails and other media that involved players in a multiplatform narrative, the game was eventually cancelled due to lack of players. This was due to many factors, ranging from the monthly subscription fee (as part of Electronic Arts' EA Online venture) to Majestic's unfortunate timing and subject matter in relation to the September 11 attacks on the World Trade Center. Many players also criticized the absence of the TINAG principle (e.g. in-game phone calls were preceded by an announcement that they were part of the game, although these announcements were optional based on user preference).

The Beast In 2001, in order to market the movie A.I.: Artificial Intelligence directed by Steven Spielberg and based on Stanley Kubrick's unfinished project but also a planned series of Microsoft computer games based on the film, Microsoft's Creative Director Jordan Weisman and another Microsoft game designer, Elan Lee, conceived of an elaborate murder mystery played out across hundreds of websites, email messages, faxes, fake ads, and voicemail messages. They hired Sean Stewart, an award-winning science fiction/fantasy author, to write the story. The game, dubbed "the Citizen Kane of online entertainment" by Internet Life,[15] was a runaway success[16] that involved over three million

15

Alternate reality game active participants [17] from all over the world during its run and would become the seminal example of the nascent ARG genre. An early asset list for the project contained 666 files, prompting the game's puppetmasters to dub it "the Beast", a name which was later adopted by players.[18] A large and extremely active fan community called the Cloudmakers formed to analyze and participate in solving the game,[19] and the combined intellect, tenacity and engagement of the group soon forced the puppetmasters to create new subplots, devise new puzzles, and alter elements of the design to keep ahead of the player base.[20] Somewhat unusual for a computer-based game, the production engaged equal numbers of male and female participants,[21] and drew players from a wide spectrum of age groups and backgrounds. Although the Beast ran for only three months, it prompted the formation of a highly organized and intensely engaged community that remains active[22] years after the game concluded. Perhaps more significantly, it inspired a number of its participants to create games adapting and expanding the model, extending it from an anomalous one-time occurrence to a new genre of entertainment and allowing the community to grow even after the Beast itself concluded. Members of the Cloudmakers group went on to form ARGN, the primary news source for the genre, and Unfiction, its central community hub, as well as designing the first successful and widely-played indie ARGs, such as LockJaw and Metacortechs, and corporate efforts such as Perplex City.

Community and genre growth The years immediately after the Beast saw independent developers who had played it extend the form from a one-time occurrence to a new genre of gaming, and the formation of an ever-growing community devoted to playing, designing and discussing ARGs. Grassroots development Influenced heavily by the Beast and enthusiastic about the power of collaboration, several Cloudmakers came together with the idea that they could create a similar game. The first effort to make an independent Beast-like game, Ravenwatchers, failed,[23] but another team soon assembled and met with greater success. With very little experience behind them, the group managed, after nine months of development, to create a viable game that was soon seized upon eagerly by the Cloudmakers group and featured in WIRED Magazine.[24] As players of the Beast, members of the Lockjaw development team were extremely aware of the community playing the game and took steps to encourage the tight bonding of the player base through highly collaborative puzzles, weekly Euchre games, and the inclusion of player personas in the game. While the numbers never rivaled those of The Beast, the game proved both that it was possible for developers to create these games without corporate funding or promotion, and that there was interest in the ARG form beyond a one-time audience for a production on the Beast's scale. Lockjaw marked the start of the ARG as a genre of gaming, rather than simply a one-time occurrence. Shortly before Lockjaw's conclusion, players discovered a game that seemed to revolve around the movie Minority Report. Despite speculation to the contrary, the game (known as Exocog) was not an official promotion for the film, [25] Inspired by the independent Lockjaw effort, Dave but an experiment in interactive storytelling by Jim Miller. Szulborski introduced ChangeAgents, a spinoff of EA's failed Majestic ARG, to the ARGN audience, then followed it with two additional installments. During this time, Szulborski also created a successful grassroots game not based on the Majestic universe, called Chasing the Wish. Just before the release of the third and the final Matrix movie, the team that developed Lockjaw launched Metacortechs [26], an ARG based on that universe. The fan fiction effort was very successful, reached a larger and more active player base than many professionally produced games, and was at first assumed by many to be an officially-sanctioned promotion for the movie. Metacortechs was followed by an ever-increasing number of grassroots ARGs. In the wake of these successful, low-budget independent ARGs, an active "grassroots" development community began to evolve within the genre. While the quality of the grassroots games varies wildly, amateur storytellers, web designers, and puzzle creators continue to provide independently-developed ARGs for the active player community.

16

Alternate reality game Community development The term Alternate Reality Gaming was first used by Sean Stacey, one of the moderators of the Lockjaw player community, in the Trail for that game. Stacey and Steve Peters, another of the moderators, created the two websites that have become the central hub of the ARG community: ARGN [27] and UnFiction [28]. Due to their efforts, when Lockjaw ended, the players had a new community resource allowing them to assemble to play the games that were soon to follow. Unfiction now boasts over 26,000 members, and ARGN employs a staff of 15 volunteer writers to report on new games and other topics of interest to the community, as well as producing a weekly netcast. A first experience in video games Although not considered as a pure Alternate Reality Game, Missing Since January ("In Memoriam" in Europe) is a video game based on the same principles that appear in an ARG: an online enquiry, the game entering into the players real life environment, willingly confusing reality and fiction (real fact-based sites, emails…). Developed from 1999 onwards by the French studio Lexis Numérique [29], Missing Since January was launched by Ubisoft in Europe in October 2003 and by Dreamcatcher in the US in January 2004. In Missing Since January, using the internet, the player must attempt to decode a mysterious CD ROM broadcast by the police in order to find two missing people abducted by a serial killer. More than a hundred sites were created for this purpose. By and large, as the player advances in the enquiry, they are contacted by different characters that send emails. The follow-up, which appeared in 2006 under the title Evidence: The Last Ritual ("In Memoriam 2, The Last Ritual" in Europe) also allowed players to receive text messages and to speak on the phone with certain characters in the game.

Massive-scale commercial games and mainstream attention After the success of the first major entries in the nascent ARG genre, a number of large corporations looked to ARGs to both promote their products, and to enhance their companies' images by demonstrating their interest in innovative and fan-friendly marketing methods. To create buzz for the launch of the Xbox game Halo 2,[30] Microsoft hired the team that had created the Beast, now operating independently as 42 Entertainment. The result, I Love Bees, departed radically from the website-hunting and puzzle-solving that had been the focus of the Beast. I Love Bees wove together an interactive narrative set in 2004, and a War Of The Worlds-style radio drama set in the future, the latter of which was broken into 30-60 second segments and broadcast over ringing payphones worldwide.[31] The game pushed players outdoors to answer phones, create and submit content, and recruit others, and received as much or [32] more mainstream notice than its predecessor, finding its way onto television during a presidential debate, and becoming one of the New York Times' catchphrases of 2004.[33] A slew of imitators,[34] [35] fan tributes[36] and parodies[37] [38] followed. In 2005, a pair of articles profiling 42 Entertainment appeared in Game Developer magazine and the East Bay Express, both of which tied into an ARG[39] created by the journalist and his editors.[40] The following spring, Audi launched The Art of the Heist, developed by Audi ad agency McKinney+Silver, Haxan Films (creators of The Blair Witch Project), to promote its new A3. Roughly a year after I Love Bees, 42 Entertainment produced Last Call Poker, a promotion for Activision's video game Gun. Designed to help modern audiences connect with the Western genre, Last Call Poker centered on a working poker site, held games of "Tombstone Hold 'Em" in cemeteries around the United States—as well as in at [41] -- and sent players to their own local least one digital venue, World of Warcraft's own virtual reality cemetery [42] cemeteries to clean up neglected grave sites and perform other tasks. At the end of 2005, the International Game Developers Association ARG Special Interest Group was formed "to bring together those already designing, building, and running ARGs, in order to share knowledge, experience, and ideas for the future." More recently, an ARG was created by THQ for the game Frontlines: Fuel of War around peak oil theories where the world is in a crisis over diminishing oil resources.

17

Alternate reality game

The rise of the self-supporting ARG As the genre has grown, there has been increasing interest in exploring models that provide funding for large-scale ARGs that are neither promotions for other products nor limited by the generally small budget of grassroots/indie games. The two major trends that have emerged in this area are support through the sale of products related to the game, and fees for participation in the game. A third possible model is one using in-game advertising for other products, as in The LOST Experience, but at this time no large-scale game has attempted to fund itself solely through in-game advertising. The first major attempt (other than EA's failed Majestic) to create a self-supporting ARG was Perplex City, which launched in 2005 after a year's worth of teasers. The ARG offered a $200,000 prize to the first player to locate the buried Receda Cube and was funded by the sale of puzzle cards. The first season of the game ended in January 2007, when Andy Darley found the Receda Cube at Wakerly Great Wood in Northamptonshire, UK. Mind Candy, the production company, has also produced a board game related to the ARG and plans to continue it with a second season beginning March 1, 2007. This model was delayed till June 1, and has again, been delayed to an unspecified date. Mind Candy's acceptance of corporate sponsorship and venture capital suggests that the puzzle cards alone are not enough to fully fund the ARG at this time. In March 2006, Elan Lee and Dawne Weisman founded edoc laundry [43], a company designed to produce ARGs using clothes as the primary platform. Consumers decipher the codes hidden within the garments and input the results into the game's main website to reveal pieces of a story about the murder of a band manager. Reviving the pay-to-play model, Studio Cypher launched the first chapter of its "multiplayer novel" in May 2006. Each "chapter" is a mini-ARG for which participants who pay the $10 registration fee receive earlier access to information and greater opportunities to interact with characters than non-paying participants. VirtuQuest, a well-known corporate team, also attempted a pay-to-play model with Township Heights later in the year, but despite initial enthusiasm on the part of the ARG community, the game was not well-received due to the design team's use of player Hybrid-Names based on their real life names. Also the short run time frame was not appreciated by some seasoned players. In June 2006, Catching the Wish launched from an in-game website about comic books based on its predecessor, 2003's Chasing the Wish. 42 Entertainment released Cathy's Book, by Sean Stewart and Jordan Weisman, in October 2006, shifting the central medium of this ARG from the internet to the printed page. The young-adult novel contains an "evidence packet" and expands its universe through websites and working phone numbers, but is also a stand-alone novel that essentially functions as an individually-playable ARG. Neither the cost of creating the book nor sales figures are available (although it made both American[44] and British bestseller lists) to determine whether the project was successfully self-funded. It is difficult to judge the efficacy of self-funded ARG models at this time, but it seems likely that exploration of ways to fund large-scale ARGs without using them as marketing for other products will continue as the genre grows.

The Serious ARG In a 2007 article, columnist Chris Dahlen (of Pitchfork Media) voiced a much-discussed ARG concept: if ARGs can spark players to solve very hard fictional problems, could the games be used to solve real-world problems?[45] Dahlen was writing about World Without Oil, the first ARG centered on a serious near-future scenario: a global oil shortage.[46] Another ARG, Tomorrow Calling, appears to be a testbed for a future project focused on environmental themes and activism.[47] Serious ARGs introduce plausibility as a narrative feature to pull players into the game. People participate to experience, prepare for or shape an alternative life or future.[48] The games thus have the potential to attract casual or non-players, because ’what if’ is a game anyone can play.[49] Serious ARGs may therefore be sponsored by organizations with activist or educational goals; World Without Oil was a joint project of the Public Broadcasting

18

Alternate reality game Service's Independent Lens and its Electric Shadows Web-original programming.[50] Their serious subject matter may lead Serious ARGs to diverge from mainstream ARGs in design. Instead of challenging collective intelligence to solve a gamemastered puzzle, World Without Oil’s puppetmasters acted as players to guide the “collective imagination” to create a multi-authored chronicle of the alternative future, purportedly as it was happening.[51] By asking players to chronicle their lives in the oil-shocked alternative reality, the WWO game relinquished narrative control to players to a degree not seen before in an ARG.[52] In October 2008 The British Red Cross created a serious ARG called Traces of Hope to promote their campaign about civilians caught up in conflict.[53] There are possible future Serious ARGs described in fiction. In his novel Halting State, Charles Stross foresightedly describes a number of possible ARGs, where players engage in seemingly fictional covert spy operations. In 2008 the European Union funded an ARG to support motivation for multilingualism within European secondary school students called ARGuing for Multilingual Motivation in Web 2.0 [54]. As noted above in World Without Oil, to complete this ARG it was necessary to move away from the strict definitions of an ARG as listed. The ARG was by invitation only and players (students) knew they were going to play a game. This project is now completed and papers on the project and the resources produced for education (a Methodology and Teacher Training guides)are available and have been presented at the 3rd European Conference on Games Based Learning [55]. In 2008-2009 the MacArthur Foundation supported an ARG The Black Cloud [56] to teach US high-school students about indoor air quality. The project is active and allows teachers to rent sophisticated air quality sensors to run the game locally.

New developments 2006 produced fewer large-scale corporate ARGs than past years, but the ARG form continued to spread and be adapted for promotional uses, as an increasing number of TV shows and movies extended their universes onto the internet through such means as character blogs and ARG-like puzzle trails, and as an increasing number of independent/grassroots games launched, with varying levels of success.[57] One of the more popular indie ARGs to launch in the fall of 2006 was Jan Libby's dark yet whimsical "Sammeeeees". Lonelygirl15, a popular series of videos on YouTube, relinquished an unprecedented amount of control to its audience by recognizing a fan-created game as the "official" ARG. In August 2006, Hoodlum produced 'PSTRIXI' for Yahoo!7 Australia. PSTRIXI was designed around a young DJ Trixi and her boyfriend Hamish. Players were engaged across all of Yahoo!7's platforms and asked to help solve the mystery of Trixi's missing sister Max. The multiplatform ARG ran for 12 weeks and used websites, email, Yahoo!360 forums, Yahoo Radio and viral television to engage the audience in the game. PSTRIXI was a major success with the Yahoo!7 community; players spent an average of 16 minutes per session on the websites and returned more than once a week. 2007 got off to a strong start immediately, with Microsoft's Vanishing Point to promote the launch of Windows Vista. The game was designed by 42 Entertainment and, due in part to many large-scale real world events, such as a [58] and having a winner's lavish show at the Bellagio Fountain in Las Vegas as well as a prizes of a trip into space [59] name engraved on all AMD Athlon 64 FX chips for a certain period of time, received large media attention.[60] It was followed almost immediately by another 42 Entertainment production for the release of the Nine Inch Nails album Year Zero, in which fans discovered leaked songs on thumb drives in washrooms at concerts,[61] as well as clues to websites that describe a dystopian future. Monster Hunter Club, a promotion for the U.S. release of the movie The Host, launched by sending action figures and other items to prominent members of the ARG community.[62] Perplex City concluded its first season by awarding a $200,000 prize to a player who found the game's missing cube.[63] They plan to continue the ARG into a second "season" under the name Perplex City Stories, although they have said that there will not be a large grand prize this time around.[64] Meigeist, produced by a new professional puppetmaster team, garnered a great deal of community attention and affection with a light, humorous

19

Alternate reality game storyline and numerous references to past ARGs. The teaser site for World Without Oil, the first major "Serious ARG," was unveiled in March 2007; the game itself launched on April 30 and ran through June 1, gathering over 1500 videos, images, blog entries and voice mails to document the "Oil Crisis of 2007."[50] In May 2007, 42 Entertainment launched Why So Serious, an ARG to promote the feature film The Dark Knight. Hailed as being the single most impressive viral marketing campaign of all-time,[65] it played out over 15 months, concluding in July 2008. Millions of players in 177 countries participated both online and taking part in live events, and it reached hundreds of millions through internet buzz and exposure.[66] In March 2008 McDonalds and the IOC [67] launched Find The Lost Ring, a global ARG promoting the 2008 Summer Olympics in Beijing, China. The game was run simultaneously in six languages with new story lines developing in each, encouraging players to communicate with residents of other countries to facilitate sharing of clues and details of the game as a whole. American track and field athlete Edwin Moses acted as a celebrity Game Master, and McDonalds Corporation promised to donate $100,000 (USD) to Ronald McDonald House Charities China on behalf of the players. February 2009 saw the launch of the ARG Something In The Sea [68], designed to promote the videogame Bioshock 2 by immersing players in character Mark Meltzer's quest to find his missing daughter. In addition to the messages, documents, photos and puzzles on the website, those lucky enough to be following along on August 8, 2009, were given the coordinates of 10 beaches worldwide and told to go there at dawn. Those who did found objects planted by the game runners designed to look like they had washed ashore from Bioshock's fictional underwater city of Rapture. Players who wrote letters to Mark, whose address was advertised on the website, also sometimes received items such as wine bottles, records, or masks. June 2009 also saw the release of another ARG, known as Marble Hornets, centering around the SomethingAwful myth of Slenderman, a paranormal entity. March 1, 2010, Valve Corporation released an update via Steam to their game Portal, adding a nondescript new achievement and some .wav files hidden within the game GCFs. The .wav files actually contained morse code and SSTV encoded images, some including certain numbers and letters. When pieced together in the correct order, these numbers and letters formed a 32-bit MD5 hash of a BBS phone number. When traced, it was found to originate from Kirkland, Washington, where Valve was based before moving to Bellevue, Washington in 2003. Accessing the number as a bulletin board system yielded large ASCII art images. Also launched in March 2010, an ARG produced by David Varela at nDreams featured the 2008 Formula 1 World Champion Lewis Hamilton; entitled Lewis Hamilton: Secret Life, the game ran throughout the 2010 Formula 1 season, in nine languages, with live events in a dozen cities around the world. Television tie-ins and "Extended Experiences" Even before the development of the ARG genre, television sought to extend the reality of its shows onto the web with websites that treated their world as real, rather than discussing it as fiction. An early example was Fox's Freakylinks, developed by Haxan, creators of The Blair Witch Project, who would later go on to develop the well-known ARGs The Art of the Heist and Who Is Benjamin Stove. Freakylinks employed a website designed to look like it had been created by amateur paranormal enthusiasts to generate internet interest in the show, which [69] In September 2002, following a successful initial gathered a cult following but was canceled after 13 episodes. [70] foray into ARG-like territory with 2001's Alias web game, ABC brought alternate reality gaming more definitively to the television screen with the show Push, Nevada. Produced and co-written by Ben Affleck, the show created a fictional city in Nevada, named Push. When advertising the show, LivePlanet advertised the city instead, with billboards, news reports, company sponsors, and other realistic life-intruding forms.[71] During each episode of the show, highly cryptic clues would be revealed on screen, while other hidden clues could be found on the city's website. Unfortunately, the show was cancelled mid-season, and all of the remaining clues were released to the public. Clever watchers eventually figured out that the show would still be paying out its $1 million prize during

20

Alternate reality game Monday Night Football. The last clue was revealed during half-time, prompting those fortunate enough to have solved the puzzle to call a telephone number. The first person to call received $1 million.[72] In October 2004, the ReGenesis Extended Reality game launched in tandem with the Canadian television series ReGenesis. Produced by Xenophile Media in association with Shaftesbury Films, clues and stories from the series sent players online to stop a bioterrorist attack.[73] In 2006, the TV tie-in ARG began to come into its own when there was a surge of ARGs that extended the worlds of related television shows onto the internet and into the real world. As with Push, Nevada, ABC led the way, launching three TV tie-in ARGs in 2006: Kyle XY,[74] Ocular Effect (for the show Fallen)[75] and The LOST Experience (for the show LOST).[76] ABC joined with Channel 4 in the UK and Australia's Channel 7 in promoting a revamped web site for The Hanso Foundation. The site was focused on a fictitious company prevalent in the storyline of the TV series, and the game was promoted through television advertisements run during LOST episodes. The Fallen Alternate Reality Game was launched in tandem with the Fallen TV movie for ABC Family and was originally conceived by Matt Wolf and created by Matt Wolf (Double Twenty Productions) in association with Xenophile Media. "I am humbled by this honor..." said Wolf when accepting the Emmy for The Fallen Alternate Reality Game at the 59th Annual Primetime Creative Arts Emmy Awards, live at the Shrine Auditorium in Los Angeles on September 8, 2007. NBC followed suit in January 2007, beginning an ARG for its hit TV series Heroes[77] launched through an in-show reference to the website for Primatech Paper [78], a company from the show, which turned out to be real. Text

messages and emails led players who applied for "employment" at the site to secret files on the show's characters.[79]

In May 2007, the BBC commissioned Kudos and Hoodlum to produce an interactive ARG for their flagship drama series Spooks, "Spooks Interactive." The game enlists players to become MI5 agents who join the Section D team on missions crucial to the security of the UK, and launched on September 26. In 2008 it won the Interactivity Award at the British Academy Television Awards and the Interactive Innovation -Content Award at the British Academy Craft Awards. The November 9, 2007 episode of Numb3rs entitled "Primacy" featured alternate reality gaming, and launched the ARG Chain Factor [80], which centered on players using a flash-based puzzle game to unknowingly destroy the world's economy on the whim of one of the characters from the "Primacy" episode. In January 2008, BBC launched "Whack the Mole" [81] for the CBBC show M.I. High in which viewers are asked to become M.I. High field agents and complete tasks to capture a mole that has infiltrated the organization. CBS made an ARG for Jericho to promote the series in 2007.

Notes [1] [2] [3] [4]

http:/ / web. archive. org/ web/ 20070317204354/ http:/ / www. occulareffect. com/ http:/ / www. xenophile. ca/ http:/ / web. archive. org/ web/ 20060219055154/ http:/ / www. regenesistv. com/ "SCANDINAVIA LEADS INTERNATIONAL INTERACTIVE EMMY AWARDS AT MIPTV" (http:/ / www. iemmys. tv/ news_item. aspx?id=60). International Academy of Television Arts and Sciences. 8 April 2008. . Retrieved 25 August 2008. [5] http:/ / www. unfiction. com/ about/ [6] Stacey, Sean (10 November 2006). "Undefining ARG" (http:/ / www. unfiction. com/ compendium/ 2006/ 11/ 10/ undefining-arg/ ). . Retrieved 19 February 2007. [7] McGonigal, Jane (2003). "Digital Games Research Association (DiGRA) "Level Up" Conference Proceedings" (http:/ / www. avantgame. com/ MCGONIGAL A Real Little Game DiGRA 2003. pdf) (PDF). [8] Cloudmakers. "Puppetmaster FAQ" (http:/ / familiasalla-es. cloudmakers. org/ credits/ note/ faq. html). . Retrieved 19 February 2007. [9] Stacey, Sean (22 September 2002). "Unfiction Terms of Service" (http:/ / forums. unfiction. com/ forums/ viewtopic. php?t=5). . Retrieved 13 February 2007. [10] http:/ / www. cse. dmu. ac. uk/ ~mward/ gkc/ books/ queertrades/ cqtchap1. html [11] http:/ / www. cse. dmu. ac. uk/ ~mward/ gkc/ books/ queertrades/ index. html [12] Hanas, Jim (25 January 2006). "The Story Doesn't Care: An Interview With Sean Stewart" (http:/ / www. hanasiana. com/ archives/ 001117. html). . Retrieved 20 February 2007.

21

Alternate reality game [13] Stewart, Sean. "alternate Reality Games" (http:/ / www. seanstewart. org/ interactive/ args/ ). . Retrieved 20 February 2007. [14] "Dreadnot" (http:/ / web. archive. org/ web/ 20000229151210/ www. sfgate. com/ dreadnot/ index. html). SFGate. . [15] Dena, Christy (22 May 2006). "Designing Cross-Media Entertainment" (http:/ / www. lamp. edu. au/ media/ pdf/ dena_LAMP3. pdf) (PDF). p. 27. . Retrieved 13 February 2007. [16] "TIME Best & Worst of 2001" (http:/ / www. time. com/ time/ magazine/ article/ 0,9171,1001509,00. html). TIME Magazine. 24 December 2001. . Retrieved 13 February 2007. [17] Dena, Christy. "Top ARGs, With Stats" (http:/ / www. cross-mediaentertainment. com/ index. php/ 2006/ 03/ 04/ top-args-with-stats/ ). . Retrieved 13 February 2007. [18] "The Buzzmakers" (http:/ / web. archive. org/ web/ 20070407060201/ http:/ / www. eastbayexpress. com/ 2005-05-18/ news/ the-buzzmakers/ ). East Bay Express. 18 May 2005. Archived from the original (http:/ / www. eastbayexpress. com/ 2005-05-18/ news/ the-buzzmakers/ ) on 2007-04-07. . Retrieved 13 February 2007. [19] "Signs of Intelligent Life: A.I.'s mysterious and masterful promotional campaign" (http:/ / www. slate. com/ id/ 106028/ ). Slate. 15 May 2001. . Retrieved 13 February 2007. [20] Stewart, Sean. "The A.I. Web Game" (http:/ / www. seanstewart. org/ interactive/ aiintro/ ). . Retrieved 13 February 2007. [21] Lee, Elan (2006). "Check Your Joystick At The Door". Montréal International Game Summit [22] "Cloudmakers Yahoo! List" (http:/ / games. groups. yahoo. com/ group/ cloudmakers/ ). . Retrieved 13 February 2007. [23] "Testing the Waters" (http:/ / www. unfiction. com/ history/ testing-the-waters/ ). Unfiction. . Retrieved 19 February 2007. [24] "A Conspiracy of Conspiracy Gamers" (http:/ / www. wired. com/ culture/ lifestyle/ news/ 2001/ 09/ 46672). WIRED. 19 September 2001. . Retrieved 19 February 2007. [25] Miller, Jim (November 2004). "Exocog: A case study of a new genre in storytelling" (http:/ / www. miramontes. com/ writing/ exocog/ ). . Retrieved 19 February 2007. [26] http:/ / www. metacortechs. com/ [27] http:/ / www. argn. com/ [28] http:/ / www. unfiction. com/ [29] http:/ / www. lexis-numerique. fr/ [30] "Ilovebees.com Link to Halo 2 Release Confirmed" (http:/ / www. argn. com/ archive/ 000050ilovebeescom_link_to_halo_2_release_confirmed. php). Alternate Reality Gaming Network. 23 July 2004. . Retrieved 19 February 2007. [31] "42 Entertainment: I Love Bees" (http:/ / www. 42entertainment. com/ bees. html). . Retrieved 19 February 2007. [32] "I Love Bees Game A Surprise Hit" (http:/ / www. wired. com/ culture/ lifestyle/ news/ 2004/ 10/ 65365). WIRED. 18 October 2004. . Retrieved 19 February 2007. [33] Mcgrath, Charles (26 December 2004). "2004: In a Word; The Year of (Your Catchphrase Here)" (http:/ / select. nytimes. com/ gst/ abstract. html?res=F70F16FD35540C758EDDAB0994DC404482). The New York Times. . Retrieved 19 February 2007. [34] "Metroid Prime ARGishness" (http:/ / www. argn. com/ archive/ 000172metroid_prime_argishness. php). Alternate Reality Gaming Network. 20 October 2004. . Retrieved 19 February 2007. [35] "I Love Bees Two" (http:/ / www. argn. com/ archive/ 000389i_love_bees_two. php). Alternate Reality Gaming Network. 7 March 2006. . Retrieved 19 February 2007. [36] "Ilovebees-Inspired Artwork to Raise Money for Charity" (http:/ / www. argn. com/ archive/ 000185ilovebeesinspired_artwork_to_raise_money_for_charity. php). Alternate Reality Gaming Network. 9 December 2004. . Retrieved 19 February 2007. [37] "I Love Beer" (http:/ / www. ilovebeer. org/ ). 2004. . Retrieved 19 February 2007. [38] "We Love Beef" (http:/ / www. welovebeef. co. uk/ ). 2007. . Retrieved 19 February 2007. [39] "Where's Handy?" (http:/ / www. argn. com/ archive/ 000265wheres_handy. php). Alternate Reality Gaming Network. 18 May 2005. . Retrieved 22 July 2007. [40] "The Buzzmakers" (http:/ / web. archive. org/ web/ 20070407060201/ http:/ / www. eastbayexpress. com/ 2005-05-18/ news/ the-buzzmakers/ ). The East Bay Express. 18 May 2005. Archived from the original (http:/ / www. eastbayexpress. com/ 2005-05-18/ news/ the-buzzmakers/ ) on April 7, 2007. . Retrieved 22 July 2007. [41] "Last Call Poker PM Chat Transcript" (http:/ / www. argn. com/ archive/ 000347last_call_poker_pm_chat_transcript. php). Alternate Reality Gaming Network. 30 November 2005. . Retrieved 19 February 2007. [42] "'Last Call Poker' celebrates cemeteries" (http:/ / news. cnet. com/ Last-Call-Poker-celebrates-cemeteries/ 2100-1043_3-5963346. html?tag=nefd. top). CNet. 20 November 2005. . Retrieved 19 February 2007. [43] http:/ / www. edoclaundry. com/ [44] "Bestseller List" (http:/ / www. nytimes. com/ 2006/ 11/ 12/ books/ bestseller/ 1112bestchildren. html?ex=1172120400& en=be7b8042fc5994ad& ei=5070). The New York Times. 12 November 2006. . Retrieved 20 February 2007. [45] "Surviving A World Without Oil" (http:/ / pitchfork. com/ features/ get-that-out-of-your-mouth/ 6586-get-that-out-of-your-mouth-34/ ). Pitchfork Media. 13 April 2007. . Retrieved 7 September 2007. [46] "Slick Way To Address Oil Thirst" (http:/ / www. mercurynews. com/ businessheadlines/ ci_5783396?nclick_check=1). San Jose Mercury News. 30 April 2007. . Retrieved 7 September 2007.

22

Alternate reality game [47] "It's Tomorrow Calling: Do You Accept The Charges?" (http:/ / www. argn. com/ archive/ 000634its_tomorrow_calling_do_you_accept_the_charges. php). ARGN. 2 August 2007. . Retrieved 7 September 2007. [48] "WWO: Serious Games For Lower-Consumption Practices" (http:/ / elianealhadeff. blogspot. com/ 2007/ 05/ wwo-serious-games-for-lower-consumption. html). Future-Making Serious Games. 2 May 2007. . Retrieved 7 September 2007. [49] "Game Friday: Aftermath of the ARG World Without Oil" (http:/ / museumtwo. blogspot. com/ 2007/ 07/ game-friday-aftermath-of-arg-world. html). Museum 2.0. 27 July 2007. . Retrieved 7 September 2007. [50] "World Without Oil: The Post-Game Press Release" (http:/ / www. argn. com/ archive/ 000615world_without_oil_the_postgame_press_release. php). ARGN. 13 July 2007. . Retrieved 7 September 2007. [51] "World Without Oil Launches" (http:/ / radar. oreilly. com/ archives/ 2007/ 04/ world_without_o. html). O'Reilly Radar. 30 April 2007. . Retrieved 7 September 2007. [52] "The Real Reason World Without Oil Is So Interesting" (http:/ / retext. blogspot. com/ 2007/ 04/ real-reason-that-wwo-is-so-interesting. html). RE:TEXT. 30 April 2007. . Retrieved 7 September 2007. [53] "Internet game for victims of war" (http:/ / news. bbc. co. uk/ 2/ hi/ africa/ 7638581. stm). BBC News. 29 September 2008. . Retrieved 1 October 2008. [54] http:/ / arg. paisley. ac. uk/ [55] http:/ / academic-conferences. org/ ecgbl/ ecgbl2009/ ecgbl09-home. htm [56] http:/ / www. blackcloud. org/ [57] "2006 In Review: Alternate Reality Gaming" (http:/ / www. argn. com/ archive/ 0005132006_in_review_alternate_reality_gaming. php). Alternate Reality Gaming Network. 6 January 2007. . Retrieved 19 February 2007. [58] "Beam me up, Bill: Network technician wins Vista 'rocketplane' ride" (http:/ / www. computerworld. com/ action/ article. do?command=viewArticleBasic& articleId=9011149& intsrc=hm_list). Computer News. 12 February 2007. . Retrieved 19 February 2007. [59] Lohr, Steve (30 January 2007). "First the Wait for Microsoft Vista; Now the Marketing Barrage" (http:/ / select. nytimes. com/ gst/ abstract. html?res=F60F1EF83F5B0C738FDDA80894DF404482& oref=login). The New York Times. . Retrieved 19 February 2007. [60] "Playing Now: A game that wants you" (http:/ / web. archive. org/ web/ 20070216162329/ http:/ / www. mercurynews. com/ mld/ mercurynews/ 16680141. htm). Mercury News. 12 February 2007. Archived from the original (http:/ / www. mercurynews. com/ mld/ mercurynews/ 16680141. htm) on 2007-02-16. . Retrieved 19 February 2007. [61] "Nine Inch Nails Sparks Web Marketing Conspiracy" (http:/ / www. adotas. com/ 2007/ 02/ nine-inch-nails-sparks-web-marketing-conspiracy/ ). Adotas. 16 February 2007. . Retrieved 19 February 2007. [62] "Dude, Where's My Monster?" (http:/ / www. argn. com/ archive/ 000534dude_wheres_my_monster. php). Alternate Reality Gaming Network. 1 February 2007. . Retrieved 19 February 2007. [63] "£100,000 prize for digital hunter" (http:/ / news. bbc. co. uk/ 2/ hi/ technology/ 6344375. stm?ls). BBC News. 8 February 2007. . Retrieved 19 February 2007. [64] Post-Game PM Chat Logs (http:/ / appliedirc. com/ logs/ pxo-chat/ 20Feb2007/ 1100/ ) Accessed 2/21/2007. [65] FilmSchoolRejects.com (http:/ / www. filmschoolrejects. com/ news/ when-will-the-dark-knight-trailer-be-online. php) [66] Reuters (http:/ / www. reuters. com/ article/ idUS212237+ 28-Jul-2008+ PRN20080728) [67] http:/ / www. olympic. org/ [68] http:/ / www. somethinginthesea. com/ [69] Bach, Robert (12 October 2000). "Fox TV's Freakylinks" (http:/ / onthebox. netfirms. com/ Articles/ Freakylinks/ Freakylinks. html). . Retrieved 19 February 2007. [70] "A Brief History of the Alias v1.0 Web Puzzle" (http:/ / www. unfiction. com/ compendium/ 2002/ 10/ 01/ alias/ ). Unfiction. 1 October 2002. . Retrieved 19 February 2007. [71] "Push, NV" (http:/ / www. unfiction. com/ compendium/ 2002/ 09/ 01/ 172/ ). Unfiction. 1 September 2002. . Retrieved 19 February 2007. [72] "ABC Primetime: Push, Nevada" (http:/ / web. archive. org/ web/ 20070311010408/ http:/ / abc. go. com/ primetime/ push/ ). Archived from the original (http:/ / abc. go. com/ primetime/ push/ ) on 2007-03-11. . Retrieved 19 February 2007. [73] "ReGenesis: Relaunch and Award Nomination!" (http:/ / www. argn. com/ archive/ 000317regenesis_relaunch_and_award_nomination. php). Alternate Reality Gaming Network. 12 September 2005. . Retrieved 19 February 2007. [74] "Kyle XY: Why, why, why?" (http:/ / www. argn. com/ archive/ 000437kyle_xy_why_why_why. php). Alternate Reality Gaming Network. 30 July 2006. . Retrieved 19 February 2007. [75] "It's Staring at Me, Mommy! Make the Oculus Stop!" (http:/ / www. argn. com/ archive/ 000442its_staring_at_me_mommy_make_the_oculus_stop. php). Alternate Reality Gaming Network. 3 August 2006. . Retrieved 19 February 2007. [76] Manly, Lorne (1 October 2006). "Running the Really Big Show: ‘Lost’ Inc." (http:/ / www. nytimes. com/ 2006/ 10/ 01/ arts/ television/ 01manl. html?ex=1317355200& en=9a89c6ab5bf568c9& ei=5088& partner=rssnyt& emc=rss). The New York Times. . Retrieved 19 February 2007. [77] "NBC Launches Digital Extensions for Heroes" (http:/ / web. archive. org/ web/ 20070711063909/ http:/ / www. mediaweek. com/ mw/ news/ interactive/ article_display. jsp?vnu_content_id=1003535631). Mediaweek. 22 January 2007. Archived from the original (http:/ / www. mediaweek. com/ mw/ news/ interactive/ article_display. jsp?vnu_content_id=1003535631) on 11 July 2007. . Retrieved 19 February 2007. [78] http:/ / www. primatechpaper. com/

23

Alternate reality game [79] "I Need a Hero! NBC ventures into ARGish territory with Heroes 360" (http:/ / web. archive. org/ web/ 20070927063048/ http:/ / www. argn. com/ archive/ 000526i_need_a_hero_nbc_ventures_into_argish_territory_with_heroes_360. phpd=1003535631). Alternate Reality Gaming Network. 24 January 2007. Archived from the original (http:/ / www. argn. com/ archive/ 000526i_need_a_hero_nbc_ventures_into_argish_territory_with_heroes_360. phpd=1003535631) on 2007-09-27. . Retrieved 19 February 2007. [80] http:/ / www. chainfactor. com/ [81] Bbc - Cbbc - Mi High (http:/ / www. bbc. co. uk/ cbbc/ mihigh/ )

External links • Alternate Reality Gaming - A Quickstart Guide (http://www.giantmice.com/features/arg-quickstart/) - A simple explanation on how to get started playing an ARG • Design-in-play: improving the variability of indoor pervasive games (http://www.springerlink.com/content/ c50217p160272h44/) - A design strategy which allows end users to redesign indoor pervasive games • Serious fun (http://www.economist.com/science-technology/technology-quarterly/displaystory. cfm?story_id=13174355), The Economist, Technology Quarterly, Mar 5th 2009 • ARGology (http://www.argology.org/) - Key information about ARGs by the International Game Developers Association Alternate Reality Game Special Interest Group (IGDA ARG SIG). • Alternate Reality Gaming Network (http://www.argn.com/) - the hub of a network of sites dedicated to Alternate Reality Gaming. • ARG Stats (http://www.christydena.com/online-essays/arg-stats/) A comprehensive list of ARG's by Christy Dena. • Storytelling in new media: The case of alternate reality games, 2001-2009 (http://www.uic.edu/htbin/cgiwrap/ bin/ojs/index.php/fm/article/view/2484/2199) First Monday (Volume 14, Number 6–1 June 2009) by Jeffrey Kim, Elan Lee, Thimoty Thomas and Caroline Dombrowski • Authorized free excerpt from THIS IS NOT A GAME by Dave Szulborski (http://www.incunabula.org/PDFS/ TINAGQUA.pdf) Authorized free excerpt from THIS IS NOT A GAME by Dave Szulborski as a PDF

24

ARQuake

ARQuake ARQuake is an Augmented Reality version of the popular Quake game by id Software. Created in the Wearable Computer Lab at the University of South Australia, ARQuake provides a first-person shooter that allows the user to run around in the real world whilst playing a game in the computer generated world. The system uses GPS, a hybrid magnetic and inertial orientation sensor, a custom made gun controller, and a standard laptop carried on a backpack. ARQuake was the first fully working Augmented Reality game created for outdoor use. The ARQuake project was started by Prof. Bruce H. Thomas. It An example of what ARQuake looks like was initially implemented by a group of honours students Benjamin Close, John Donoghue, John Squires and Philip DeBondi in the year 2000. Dr. Wayne Piekarski has further improved the game to work with the latest mobile AR technology. The game has never become commercial, existing only in its prototype state. However, it has generated some interest in the augmented reality world.

References • Thomas, B., Close, B., Donoghue, J., Squires, J., De Bondi, P., Morris, M., and Piekarski, W. "ARQuake: An Outdoor/Indoor Augmented Reality First Person Application." In 4th International Symposium on Wearable Computers, pp 139–146, Atlanta, Ga, Oct 2000. • Thomas, B. H., Close, B., Donoghue, J., Squires, J., De Bondi, P., and Piekarski, W. "First Person Indoor/Outdoor Augmented Reality Application: ARQuake." Personal and Ubiquitous Computing, Vol. 6, No. 2, 2002. • Thomas, B. H., Krul, N., Close, B., and Piekarski, W. "Usability and Playability Issues for ARQuake." In 1st International Workshop on Entertainment Computing, Tokyo, Japan, May 2002.

External links • • • •

ARQuake Home Page [1] Wearable Computer Lab [2] University of South Australia [3] a_rage [4]: Augmented Reality Gaming Company (Closed in 2008)

References [1] [2] [3] [4]

http:/ / wearables. unisa. edu. au/ arquake http:/ / wearables. unisa. edu. au/ http:/ / www. unisa. edu. au/ http:/ / a-rage. com/

25

Augmented browsing

Augmented browsing Augmented browsing describes the experience of using a system that can automatically augment or improve the information in web pages. For example, augmented browsing could be used to automatically add definitions for all scientific or technical keywords that occur in a document [1] [2] . A popular example of an augmented browsing technology is the Firefox add-on Greasemonkey, which allows end-users to install scripts that make on-the-fly changes to HTML-based web pages. Augmented browsing allows end-users to personalize how they view web documents, and is believed by some academics to be an important emerging technology[3] [4] . Usage of this term dates back to at least 1997[5] , and is likely to have been derived by analogy to the concept of augmented reality.

References [1] RDFa Use Cases: Scenarios for Embedding RDF in HTML, W3C Working Draft 30 March 2007 (http:/ / www. w3. org/ TR/ xhtml-rdfa-scenarios/ #use-case-7) [2] Pafilis et al. (2009) "Reflect: augmented browsing for the life scientist" Nature Biotechnology 27:508-510 (http:/ / www. nature. com/ nbt/ journal/ v27/ n6/ full/ nbt0609-508. html) [3] Ankolekar & Vrandečić (2008) "Kalpana - enabling client-side web personalization" Proc. 19th ACM Conf. on Hypertext and Hypermedia, p21-26. (http:/ / portal. acm. org/ citation. cfm?id=1379100) [4] Bowen, J.P. and Filippini-Fantoni, S., Personalisation and the Web from a Museum Perspective (http:/ / www. archimuse. com/ mw2004/ papers/ bowen/ bowen. html). In David Bearman and Jennifer Trant (eds.), Museums and the Web 2004: Selected Papers from an International Conference, Arlington, Virginia, USA, 31 March – 3 April 2004. Archives & Museum Informatics, pages 63–78, 2004. [5] Cunliffe et al. (1997) "Query-based navigation in semantically indexed hypermedia" Proc. 8th ACM Conf. on Hypertext, p87-95. (http:/ / portal. acm. org/ citation. cfm?id=267447)

Augmented virtuality Augmented virtuality (AV) (also referred to as Mixed reality) refers to the merging of real world objects into [1] virtual worlds . As an intermediate case in the Virtuality Continuum, it refers to predominantly virtual spaces, where physical elements, e.g. physical objects or people, are dynamically integrated into, and can interact with the virtual world in real-time. This integration is achieved Sony EyeToy with the use of various techniques. Often streaming video from physical spaces, e.g. via webcam, (see The [2] Distributed Interactive Virtual Environment (DIVE) ), or using 3-dimensional digitalisation of physical objects (see Tele-Immersion@UC Berkeley [3]).

26

Augmented virtuality

27

Popular culture • The children's television show Knightmare was based around a contestant entering an augmented reality game show and interacting with both real and virtual actors, objects and puzzles.

References [1] P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays (http:/ / vered. rose. utoronto. ca/ people/ paul_dir/ IEICE94/ ieice. html) IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994. [2] http:/ / www. sics. se/ dive/ [3] http:/ / tele-immersion. citris-uc. org/

A screenshot from the game show Knightmare showing a live contestant inserted into a computer-generated room.

External links • H. Regenbrecht and C. Ott and M. Wagner and T. Lum and P. Kohler and W. Wilke and E. Mueller, An Augmented Virtuality Approach to 3D Videoconferencing, Proceedings of the The 2nd IEEE and ACM International Symposium on Mixed and Augmented Reality, pp 290-291, 2003 (http://csdl2.computer.org/ comp/proceedings/ismar/2003/2006/00/20060290.pdf) • Kristian Simsarian and Karl-Petter Akesson, Windows on the World: An example of Augmented Virtuality, Interface Sixth International Conference Montpellier, Man-machine interaction, pp 68-71, 1997 (http://www. sics.se/~kalle/published/wow.pdf) • Mixed Reality Project: experimental applications on Mixed Reality (Augmented Reality, Augmented Virtuality) and Virtual Reality. (http://www.architecturemixedreality.com)

Augmented Reality-based testing

Augmented Reality-based testing Augmented Reality-based testing (ARBT) is a test method that combines Augmented Reality and Software testing to enhance testing by inserting an additional dimension into the testers field of view. For example, a tester wearing a Head Mounted Display (HMD) or Augmented reality contact lenses [1] that places images of both the physical world and registered virtual graphical objects over the user's view of the world can detect virtual labels on areas of a system to clarify test operating instructions for a tester who is performing tests on a complex system. In 2009 as a spin-off to Augmented Reality for Maintenance and Repair (ARMAR) [2] Alexander Andelkovic coined the idea 'Augmented Reality-based testing' introducing the idea of using Augmented Reality together with software testing.

Overview The test environment of technology is becoming more complex, this puts higher demand on test engineers to have higher knowledge, testing skills and work effective. A powerful unexplored dimension that can be utilized is the Virtual environment, a lot of information and data that today is available but unpractical to use due to overhead in time needed to gather and present can with ARBT be used instantly.

Application ARBT can be of help in following test environments: Support Assembling and disassembling a test object [3] can be learned out and practice scenarios can be run through to learn how to fix fault scenarios that may occur. Guidance Minimizing risk of misunderstanding complex test procedures can be done by virtually describing test steps in front of the tester on the actual test object. Educational Background information about test scenario with earlier bugs found pointed out on the test object and reminders to avoid repeating previous mistakes made during testing of selected test area. Training Junior testers can learn complex test scenarios with less supervision. Test steps will be pointed out and information about pass criteria need to be confirmed the junior tester can train before the functionality is finished and do some regression testing. Informational Tester can point at a physical object and get detailed updated technical data and information needed to perform selected test task.

28

Augmented Reality-based testing Inspire Testers performing exploratory testing that need inspiration of areas to explore can get instant information about earlier exploratory test sessions gathered through Session-based testing.

References [1] Babak A. Parviz, Augmented Reality in a Contact Lens (http:/ / spectrum. ieee. org/ biomedical/ bionics/ augmented-reality-in-a-contact-lens/ ) IEEE Spectrum insde technology biomedical bionics, Sep 2009. [2] Steve Henderson, Steven Feiner. ARMAR:Augmented Reality for Maintenance and Repair (ARMAR) (http:/ / graphics. cs. columbia. edu/ projects/ armar/ index. htm). Columbia University Computer Graphics & User Interfaces Lab. [3] BMW research labs, Augmented reality BMW car repair (http:/ / www. youtube. com/ watch?v=P9KPJlA5yds) Video Clip, Oct 2007.

Bionic contact lens The bionic contact lens are being developed to provide a virtual display that could have a variety of uses from assisting the visually impaired to the video game industry.[1] The device will have the form of a conventional contact lens with added bionics technology.[2] The lens will eventually have functional electronic circuits and infrared lights to create a virtual display. Babak Parviz, a University of Washington assistant professor of electrical engineering is quoted as saying "Looking through a completed lens, you would see what the display is generating superimposed on the world outside.”[3]

Manufacture The lenses require organic materials that are biologically safe and also use inorganic material for the electronic circuits. The electronic circuits are built from a layer of metal a few nanometres Picture of a bionic contact lens. thick. The light-emitting diodes are one third of a millimetre across. A grey powder is sprinkled onto the lens. Then a technique called microfabrication or 'self-assembly' is used to shape each tiny component. Capillary forces pull the pieces into their final position.

Development Harvey Ho, a former graduate student of Mr. Parviz who is now working at Sandia National Laboratories in Livermore, California presented the results in January 2008 at the Institute of Electrical and Electronics Engineers' International Conference on Micro Electro Mechanical Systems (or microbotics) in Tucson, Arizona.[4] The lens is expected to have more electronics and capabilities on the areas where the eye does not see. Wireless communication, radio frequency power transmission and solar cells are expected in future developments.[5]

29

Bionic contact lens

30

Prototype and testing The prototype does not light up or display information; however, it is proof that it is possible to create a biologically safe electronic lens that does not obstruct a person’s view. Engineers have tested the finished lenses on rabbits for up to 20 minutes and the animals showed no problems.[6]

References [1] "bionic-eyes-could-change-the-face-of-gaming" (http:/ / kotaku. com/ 346081/ bionic-eyes-could-change-the-face-of-gaming). . Retrieved 2008-01-23. [2] "'Bionic Lens' Adds Computing Power to Sight" (http:/ / dsc. discovery. com/ news/ 2008/ 02/ 05/ bionic-contact-lens. html). discovery.com. . Retrieved 2008-02-08. [3] "Bionic eyes: Contact lenses with circuits, lights a possible platform for superhuman vision" (http:/ / web. archive. org/ web/ 20080120125715/ http:/ / uwnews. org/ uweek/ uweekarticle. asp?visitsource=uwkmail& articleID=39100). Archived from the original (http:/ / uwnews. org/ uweek/ uweekarticle. asp?visitsource=uwkmail& articleID=39100) on 2008-01-20. . Retrieved 2008-01-23. [4] "Researchers Develop Bionic Contact Lens" (http:/ / www. foxnews. com/ story/ 0,2933,323929,00. html). Fox News. 2008-01-18. . Retrieved 2008-01-23. [5] "Bionic Vision" (http:/ / www. scenta. co. uk/ health/ news/ 1713965/ bionic-vision. htm). . Retrieved 2008-01-23. [6] "Vision of the future seen in bionic contact lens" (http:/ / www. msnbc. msn. com/ id/ 22731631/ ). . Retrieved 2008-01-23.

Brain in a vat In philosophy, the brain in a vat is an element used in a variety of thought experiments intended to draw out certain features of our ideas of knowledge, reality, truth, mind, and meaning. It is drawn from the idea, common to many science fiction stories, that a mad scientist, machine or other entity might remove a person's brain from the body, suspend it in a vat of life-sustaining liquid, and connect its neurons by wires to a supercomputer which would provide it with electrical impulses identical to those the brain normally receives. According to such stories, the computer would then be simulating reality (including appropriate responses to the brain's own output) and the person with the "disembodied" brain would continue to have perfectly normal conscious experiences without these being related to objects or events in the real world.

A brain in a vat believing to be rowing

The simplest use of brain-in-a-vat scenarios is as an argument for philosophical skepticism and solipsism. A simple version of this runs as follows: Since the brain in a vat gives and receives exactly the same impulses as it would if it were in a skull, and since these are its only way of interacting with its environment, then it is not possible to tell, from the perspective of that brain, whether it is in a skull or a vat. Yet in the first case most of the person's beliefs may be true (if he believes, say, that he is walking down the street, or eating ice-cream); in the latter case they are false. Since the argument says one cannot know whether he or she is a brain in a vat, then he or she cannot know whether most of his or her beliefs might be completely false. Since, in principle, it is impossible to rule out oneself

Brain in a vat being a brain in a vat, there cannot be good grounds for believing any of the things one believes; one certainly cannot know them. The brain-in-a-vat is a contemporary version of the argument given in Plato's Allegory of the Cave, Zhuangzi's "Zhuangzi dreamed he was a butterfly", and the deceiving demon in René Descartes' Meditations on First Philosophy.

Philosophical responses Such puzzles have been worked over in many variations by philosophers in recent decades. American philosopher's Kevin pepper aka DaPeppa and Hilary Putnam popularized the modern terminology over Descartes's "evil daemon," although it brings up such complications and objections as whether the mind is reducible to the workings of a brain. Some, including Barry Stroud, continue to insist that such puzzles constitute an unanswerable objection to any knowledge claims.[1] Hilary Putnam, in his 1981 book Reason, Truth, and History, argued against the special case of a brain born in a vat.[2] In the first chapter of his book, Putnam claims that the thought experiment is inconsistent on the grounds that a brain born in a vat could not have the sort of history and interaction with the world that would allow its thoughts or words to be about the vat that it is in. In other words, if a brain in a vat stated "I am a brain in a vat", it would always be stating a falsehood. If the brain making this statement lives in the "real" world, then it is not a brain in a vat. On the other hand, if the brain making this statement is really just a brain in the vat then by stating "I am a brain in a vat" what the brain is really stating is "I am what nerve stimuli have convinced me is a 'brain,' and I reside in an image that I have been convinced is called a 'vat'." That is, a brain in a vat would never be thinking about real brains or real vats, but rather about images sent into it that resemble real brains or real vats. This of course makes our definition of "real" even more muddled. This refutation of the vat theory is a consequence of his endorsement, at that time, of the causal theory of reference. Roughly, in this case: if you've never experienced the real world, then you can't have thoughts about it, whether to deny or affirm them. Putnam contends that by "brain" and "vat" the brain in a vat must be referring not to things in the "outside" world but to elements of its own "virtual world"; and it is clearly not a brain in a vat in that sense. One of the other problems is that the supposed brain in a vat cannot have any evidence for being a brain in a vat, because that would be saying "I have what nerve stimuli have convinced me is evidence to my being a brain in a vat" and also "Nerve stimuli have convinced me of the fact that I am a brain in a vat". Many writers have found Putnam's proposed solution unsatisfying, as it appears, in this regard at least, to depend on a shaky theory of meaning: that we cannot meaningfully talk or think about the "external" world because we cannot experience it; sounds like a version of the outmoded verification principle.[3] Consider the following quote: "How can the fact that, in the case of the brains in a vat, the language is connected by the program with sensory inputs which do not intrinsically or extrinsically represent trees (or anything external) possibly bring it about that the whole system of representations, the language in use, does refer to or represent trees or any thing external?" Putnam here argues from the lack of sensory inputs representing (real world) trees to our inability to meaningfully think about trees. But it is not clear why the referents of our terms must be accessible to us in experience. One cannot, for example, have experience of other people's private states of consciousness; does this imply that one cannot [4] In effect, Putnam demonstrates that the state of being an envatted meaningfully ascribe mental states to others? brain is invisible and indescribable from within, but it is unclear that this semantic victory goes far to address the problem in relation to knowledge. [5] Subsequent writers on the topic have been particularly interested in the problems it presents for content: that is, how - if at all - can the brain's thoughts be about a person or place with whom it has never interacted and which perhaps does not exist.

31

Brain in a vat

In Popular Culture Ideas such as the brain in a vat have lead to numerous popular incarnations. The popular 1999 film "The Matrix" is loosely based on this concept, in which a computer hacker Neo discovers that the world of 1999 America is in fact a virtual simulation created by a malign cyber-intelligence and that he and other humans are kept within fluid-filled pods, wired up to a vast computer.

References [1] Skeptical Hypotheses and the Skeptical Argument (http:/ / plato. stanford. edu/ entries/ brain-vat/ #1), from Brain in a Vat. Tony Brueckner, Stanford Encyclopedia of Philosophy, 2004. [2] Brains in a vat (http:/ / www. cavehill. uwi. edu/ bnccde/ ph29a/ putnam. html), Reason, Truth, and History, 1982 ch. 1, Hilary Putnam [3] Significance of the Argument (http:/ / www. iep. utm. edu/ b/ brainvat. htm#H5), from The "Brain in a Vat" Argument. Lance P. Hickey, Internet Encyclopedia of Philosophy. [4] The foundations of knowledge. By Reiner Grundmann, Nico Stehr. p. 201 Google Books (http:/ / books. google. com/ books?id=VI0PK7934AEC& pg=PA201& lpg=PA201& dq=How+ can+ the+ fact+ that,+ in+ the+ case+ of+ the+ brains+ in+ a+ vat,+ the+ language+ is+ connected+ by+ the+ program+ with+ sensory+ inputs+ which+ do+ not+ intrinsically+ or+ extrinsically+ represent+ trees+ (or+ anything+ external)+ possibly+ bring+ it+ about+ that+ the+ whole+ system+ of+ representations,+ the+ language+ in+ use,+ does+ refer+ to+ or+ represent+ trees+ or+ any+ thing+ external& source=bl& ots=I--x6lUrC6& sig=OH2Mq19WbOpziX8CBha1YKH5Fuc& hl=en& ei=XHugTf75H4W4tgehlOCaAw& sa=X& oi=book_result& ct=result& resnum=2& ved=0CBwQ6AEwAQ#) [5] Ben Dupre. 50 Philosophy Ideas. Quercus.

External links • Brains in a Vat (http://plato.stanford.edu/entries/brain-vat) entry by Tony Brueckner in the Stanford Encyclopedia of Philosophy • Putnam's discussion (http://www.cavehill.uwi.edu/bnccde/ph29a/putnam.html) of the "brain in a vat" in chapter one of Reason, Truth, and History • 'Where am I?' (http://www.sechumscm.org/WhereAmI.html) by Daniel Dennett • Thomas DeMarse, Daniel Wagenaar, Axel Blau, Steve Potter The Neurally Controlled Animat: Biological Brains Acting with Simulated Bodies (http://www.neuro.gatech.edu/groups/potter/papers/AutonRobots.pdf) • Thomas DeMarse, Karl Dockendorf, Adaptive flight control with living neuronal networks on microelectrode arrays (http://neural.bme.ufl.edu/page13/assets/NeuroFlght2.pdf) • Dimitris Xydas, Daniel Norcott, Kevin Warwick, Benjamin Whalley, Slawomir Nasuto, Victor Becerra, Mark Hammond, Julia Downes, Simon Marshall Architecture for Neuronal Cell Control of a Mobile Robot (http://dx. doi.org/10.1007/978-3-540-78317-6_3)

32

Camera resectioning

33

Camera resectioning Camera resectioning is the process of finding the true parameters of the camera that produced a given photograph or video. Usually, the camera parameters are represented in a 3 × 4 matrix called the camera matrix. This process is often called camera calibration, but "camera calibration" can also mean photometric camera calibration.

Parameters of camera model Often, we use

to represent a 2D point position in Pixel coordinates.

is used to represent a

3D point position in World coordinates.Note: they were expressed in augmented notation of Homogeneous coordinates which is most common notation in robotics and rigid body transforms. Referring to the pinhole camera model, a camera matrix is used to denote a projective mapping from World coordinates to Pixel coordinates.

Intrinsic parameters

The intrinsic matrix containing 5 intrinsic parameters. These parameters encompass focal length, image format, and principal point. The parameters and represent focal length in terms of pixels, where and

are the scale factors relating pixels to distance. [1]

represents the skew coefficient between the x and

the y axis, and is often 0. and represent the focal point, which would be ideally in the centre of the image. Nonlinear intrinsic parameters such as lens distortion are also important although they cannot be included in the linear camera model described by the intrinsic parameter matrix. Many modern camera calibration algorithms estimate these intrinsic parameters as well.

Extrinsic parameters are the extrinsic parameters which denote the coordinate system transformations from 3D world coordinates to 3D camera coordinates. Equivalently, the extrinsic parameters define the position of the camera center and the camera's heading in world coordinates (although T is not the position of the camera). Camera calibration is often used as an early stage in computer vision and especially in the field of augmented reality. When a camera is used, light from the environment is focused on an image plane and captured. This process reduces the dimensions of the data taken in by the camera from three to two (light from a 3D scene is stored on a 2D image). Each pixel on the image plane therefore corresponds to a shaft of light from the original scene. Camera resectioning determines which incoming light is associated with each pixel on the resulting image. In an ideal pinhole camera, a simple projection matrix is enough to do this. With more complex camera systems, errors resulting from misaligned lenses and deformations in their structures can result in more complex distortions in the final image. The camera projection matrix is derived from the intrinsic and extrinsic parameters of the camera, and is often represented by the series of transformations; e.g., a matrix of camera intrinsic parameters, a 3 × 3 rotation matrix, and a translation vector. The camera projection matrix can be used to associate points in a camera's image space with locations in 3D world space.

Camera resectioning

34

Camera resectioning is often used in the application of stereo vision where the camera projection matrices of two cameras are used to calculate the 3D world coordinates of a point viewed by both cameras. Some people call this camera calibration, but many restrict the term camera calibration for the estimation of internal or intrinsic parameters only.

Algorithms There are many different approaches to calculate the intrinsic and extrinsic parameters for a specific camera setup. 1. Direct linear transformation (DLT) method 2. A classical approach is "Roger Y. Tsai Algorithm".It is a 2-stage algorithm, calculating the pose (3D Orientation, and x-axis and y-axis translation) in first stage. In second stage it computes the focal length, distortion coefficients and the z-axis translation. 3. Zhengyou Zhang's "a flexible new technique for camera calibration" based on a planar chess board. It is based on constrains on homography

Zhang's method Zhang's camera calibration method[2] employs abstract concepts like the image of the absolute conic and circular points. Derivation Assume we have a homography

that maps points

The circular points

on a "probe plane"

lie on both our probe plane

to points

and on the absolute conic

of course means they are also projected onto the image of the absolute conic (IAC) . The circular points project as

.

We can actually ignore

while substituting our new expression for

which, when separating real and imaginary parts gives us

Since conics are symmetric matrices,

and...

on the image.

as follows:

, thus

. Lying on and

Camera resectioning

35

External links • • • • • • • • •

C++ Camera Calibration Toolbox with source code [3] Camera Calibration Toolbox for Matlab [4] Zhang's Camera Calibration Method with Software [5] Zhang's Camera Calibration and Tsai's Calibration Softwares on LGPL licence [6] Camera Calibration [7] - Augmented reality lecture at TU Muenchen, Germany Tsai's Approach [8] Camera calibration [9] (using ARToolKit) A Four-step Camera Calibration Procedure with Implicit Image Correction [10] Free software for distortion correction [11]

References [1] Richard Hartley and Andrew Zisserman (2003). Multiple View Geometry in Computer Vision. Cambridge University Press. pp. 155–157. ISBN 0-521-54051-8. [2] Z. Zhang, "A flexible new technique for camera calibration'" (http:/ / research. microsoft. com/ en-us/ um/ people/ zhang/ Papers/ TR98-71. pdf), IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.22, No.11, pages 1330–1334, 2000 [3] http:/ / graphics. cs. msu. ru/ en/ science/ research/ calibration/ cpp [4] http:/ / www. vision. caltech. edu/ bouguetj/ calib_doc/ [5] http:/ / research. microsoft. com/ en-us/ um/ people/ zhang/ Calib/ [6] http:/ / www. tecgraf. puc-rio. br/ ~mgattass/ calibration/ [7] http:/ / campar. in. tum. de/ twiki/ pub/ Far/ AugmentedRealityIISoSe2004/ L3-CamCalib. pdf [8] http:/ / www. cs. cmu. edu/ ~rgw/ TsaiDesc. html [9] http:/ / www. hitl. washington. edu/ artoolkit/ documentation/ usercalibration. htm [10] http:/ / www. vision. caltech. edu/ bouguetj/ calib_doc/ papers/ heikkila97. pdf [11] http:/ / graphics. cs. msu. ru/ en/ science/ research/ imageprocessing/ undistorter

Augmented GeoTravel Augmented GeoTravel is a mobile application which uses the Augmented reality (AR) platform, a technique that overlays virtual images and its information on the real world to enhance human visual perception. Current version of Augmented GeoTravel runs on iPhone 3GS.[1] and is used as a world travel guide. Users can get Wikipedia based information about points of interest all over the world.

Functions[1]

Augmented GeoTravel on the iPhone 3GS uses GPS and solid state compass to display the augmented reality view

Travel Guide Augmented GeoTravel [2] application displays information about users' surroundings in a mobile camera view[3] [4] . The application calculates users' current positions by using the Global Positioning System (GPS), a compass, and an accelerometer and accesses the Wikipedia data set to provide geographic information (e.g. longitude, latitude, distance), history, and contact details of points of interest. Augmented GeoTravel overlays the virtual 3-dimensional (3D) image and its information on real-time view.

Augmented GeoTravel

Planning the trip Users can plan their trip by inserting the name of the city they want to visit and choosing the points of interest provided by the application by viewing the related Wikipedia articles. The application provides the option to save the articles, so a Internet connection is no more necessary.

Car Finder From version 2.0, the application includes a car finder feature, which allow the user to save the position of the parked car and to get back to it when needed, always using augmented reality technology.

Internationalization Actually Augmented GeoTravel supports 22 languages for the Wikipedia articles. The supported languages are: English, Italian, German, French, Spanish, Japanese, Dutch, Portuguese, Russian, Swedish, Chinese, Norwegian, Finnish, Catalan, Ukrainian, Hungarian, Turkish, Romanian, Korean, Danish, Vietnamese, Serbian.

References [1] "Product: Augmented GeoTravel" (http:/ / www. augmentedworks. com). AugmentedWorks. . Retrieved 24 May 2010. [2] http:/ / www. youtube. com/ watch?v=DpmAyen4sv0 [3] James, Morgan; Julie, Notato; Michelle, Floris. "Augmented Reality in iPhone Apps" (http:/ / www2. hn. psu. edu/ faculty/ morgannotatofloris/ ar. pdf) (PDF). . Retrieved 24 May 2010. [4] "Augmented Reality - Towards the Future" (http:/ / www. demre. cl/ text/ publicaciones2010/ mayo/ publicacion23(03_05_10). pdf) (PDF). . Retrieved 24 May 2010.

36

Junaio

37

Junaio junaio

Original author(s)

metaio GmbH

Developer(s)

metaio GmbH

Initial release

November 11, 2009

Stable release

2.5 / December 15, 2010

Development status Active Operating system

Android, iPhone OS

Available in

English, German, Spanish, Russian, Japanese

Type

Augmented Reality

Website

[1]

junaio [2] is an augmented reality platform designed for 3G and 4G mobile devices. It was developed by Munich-based company metaio GmbH [3] . It provides an API for developers and content providers to generate mobile augmented reality experiences for end-users. Currently, it is available for iPhone and Android platforms. junaio is the first augmented reality browser that has overcome the accuracy limitations of GPS navigation through LLA Markers (latitude, longitude, altitude marker, patent pending).[4]

References [1] [2] [3] [4]

http:/ / www. junaio. com junaio (http:/ / www. junaio. com) metaio GmbH (http:/ / www. metaio. com) metaio: Indoor Usage of junaio®: the KIOSK EUROPE EXPO 2010 Channel (http:/ / newsletter. metaio. com/ index. php?id=1048)

Layar

38

Layar Layar is a Dutch company based in Amsterdam, founded in 2009 by Raimo van der Klein, Claire Boonstra and Maarten Lens-FitzGerald. They have created a mobile browser called Layar. The browser allows users to find various items based upon augmented reality technology. On September 1, 2010, the World Economic Forum announced the company as a Technology Pioneer for 2011.[1]

Technology The browser makes use of the following: • • • •

In-built camera Compass GPS Accelerometer

These are used together to identify the user’s location and field of view. From the geographical position, the various forms of data are laid over the camera view like inserting an additional layer.

Content Data in the browser comes in the form of layers. Layers are REST web services serving geo-located points of interest in the vicinity of the user. Layers are developed and maintained by third-parties using a free API[2] ; Layar as a company is responsible for their validation in the publication process. As of July 2010, Layar had 1000 layers[3] .

External links • http://www.layar.com/- Official page • Dutch Layar signs global augmented reality deals - http://www.reuters.com/article/ idUSTRE65H45B20100618?type=technologyNews • Layar to bring its augmented reality to one-third of global smartphones - http://mobile.venturebeat.com/2010/ 06/18/layars-augmented-reality-footprint-grows-to-one-third-of-global-smartphones/ • site with news about Layar - http://www.layarnews.com/

References [1] Thirty-One Visionary Companies Selected as Technology Pioneers 2011 (http:/ / www. weforum. org/ en/ media/ Latest News Releases/ NR_TP2011) [2] Layar Developer Wiki (http:/ / layar. pbworks. com/ ) [3] Layar Company Blog: 1000th layer published (http:/ / site. layar. com/ company/ blog/ 1000th-layer-published/ )

Spectrek

Spectrek SpecTrek[1] is an augmented reality ghost hunting game. The game won second prize in the "Android Developer Challenge II"[2] lifestyle category and is now also available for the iPhone. SpecTrek was designed to have the user work-out whilst playing the game, the tag line for the game is "protect the world, stay in shape". There are three default games to play, short which lasts 15 minutes, medium which lasts for 45 minutes, and long which lasts for 75 minutes. SpecTrek projects ghosts at various locations on a Google map in either a predetermined search radius or a user defined search radius. To play the user must walk to these ghosts, if within range the user can scan and find out what kind of ghost is nearby as well as how far said ghost is from their current position. If the user so chooses they can blow a ghost horn once every 10 minutes which makes all nearby ghosts flee. The user catches ghosts by tilting their phone to the "camera-position". Through the camera the user can scan the ghosts, see the ghosts in augmented reality and of course catch the ghosts.[3]

References [1] http:/ / www. spectrekking. com [2] "SpecTrek - [[Android (operating system)|Android (http:/ / www. appbrain. com/ app/ com. spectrekking. full)] app"]. Appbrain.com. . Retrieved 2010-08-15. [3] "SpecTrek" (http:/ / www. pcworld. com/ article/ 183760/ spectrek. html). PCWorld. 2009-12-29. . Retrieved 2010-08-15.

39

Total Immersion (augmented reality)

40

Total Immersion (augmented reality) Total Immersion Founded

Suresnes, Paris, France (1999)

Area served Worldwide Website

www.t-immersion.com

[1]

Total Immersion is an augmented reality company based in Suresnes, France[2] . Its patented D'Fusion[3] technology "integrates real time interactive 3D graphics into a live video stream."[4] The company maintains offices in Europe, North America, and Asia. Total Immersion supports the world's largest augmented reality partner network, with over 80 solution providers[5] .

References [1] http:/ / www. t-immersion. com/ [2] "Total Immersion Private Company Information" (http:/ / investing. businessweek. com/ research/ stocks/ private/ snapshot. asp?privcapId=1624447). . Retrieved 2010-12-28. [3] "Total Immersion LinkedIn Company Profile: Overview" (http:/ / www. linkedin. com/ company/ total-immersion). . Retrieved 2010-12-28. [4] Sterling, Bruce (15 September 2009). "Augmented Reality: an actual AR business announcement" (http:/ / www. wired. com/ beyond_the_beyond/ 2009/ 09/ augmented-reality-an-actual-ar-business-announcement/ ). Wired.com. . Retrieved 26 January 2011. [5] "Adobe MAX 2010" (http:/ / 2010. max. adobe. com/ sponsor/ sponsors/ ). . Retrieved 2010-12-28.

External links • Total Immersion Augmented Reality News Blog (http://blog.t-immersion.com/) • Wired Magazine, Total Immersion Augmented Reality Standards Proposal, AR+ (http://www.wired.com/ beyond_the_beyond/2010/06/augmented-reality-total-immersion-standards-proposal/) • Engage Digital, Total Immersion Announces Adobe Alliance (http://www.engagedigital.com/2010/11/09/ total-immersion-announces-adobe-alliance/)

Wikitude

41

Wikitude Wikitude is a mobile application that provides an Augmented reality (AR) platform. Augmented reality overlays virtual vision and information on the real world to enhance human visual perception. Current applications of Wikitude, such as Wikitude World Browser and Wikitude Drive, run on smartphones.[1] These applications can only be used on the iPhone, Android, and Symbian software platforms[2] as travel guides and personal navigation devices. Future applications of Wikitude can be developed for military, city modeling, and shopping.

Wikitude World Browser on the iPhone 3GS uses GPS and solid state compass

Wikitude World Browser Wikitude World Browser [3] application displays information about users' surroundings in a mobile camera view. The application calculates users' current positions by using the Global Positioning System (GPS), a compass, and an accelerometer and accesses the Wikitude data set to provide geographic information (e.g. longitude and latitude), history, and contact details of points of interest. Wikitude World Browser overlays the virtual 3-dimensional (3D) image and its information on real-time view. Travel guide [2] [4] Users can get web-based Currently, Wikitude World Browser is most commonly used as a travel guide. information about points of interest, where to eat, description of buildings and architectural images of historical places (architectural data provided by archINFORM).

Wikitude Drive Wikitude Drive [5] is a navigation application using AR technology. With an internet connection and GPS satellite reception, Wikitude Drive supports worldwide navigation in real-time, accessing the Wikitude data set.

Future applications [6] Military A mobile augmented reality system [7] (MARS) like Wikitude can be extended to military aviation and automobiles. Wikitude’s AR technology can be integrated into the windshield for pilots or drivers to provide geographic information of unknown or unfamiliar regions (e.g. altitude). In addition, Wikitude can update real-time information on weather and currents worldwide through internet access to prevent accidents.

City modeling By using Wikitude 3D AR technology, Wikitude applications can be developed for city modeling, a proposal which simulates and constructs cities. 3D visualization over the real world can be used for improving urban transportation network, determining the locations of pipes for water supply and drainage, and reconstructing buildings.

Wikitude

Shopping Wikitude’s online database can be used for shopping. Users can take an image of a product by use of a mobile camera. After the image recognition process, Wikitude can find the matched item within Wikitude database and show pricing information, shop locations, reviews, etc.

Restrictions [6] The activation limits Wikitude applications are restricted by internet connectivity and indoor disability. Since Wikitude World Browser is designed to collect information about users' surroundings through the Wikitude database, 3G (3rd Generation: international mobile communication standards) is necessary to run this application. In addition to the internet connectivity, GPS satellite reception is necessary for the Wikitude Drive to function as a navigation device.

The application limit Wikitude applications run only on iPhone, Android, Bada, and Symbian software platforms.[2] The internet and GPS connectivity depend on the specifications of smart-phones (e.g. the internet connectivity of Wikitude applications in iPhone will be limited by 802.1X, Port Based Network Access Control,[8] Wi-Fi network).

References [1] "Products: Wikitude World Browser and Drive" (http:/ / www. mobilizy. com/ produkte). Mobilizy Mobile Software. . Retrieved 05 February 2010. [2] Karpischek, Stephan; Magagna, Fabio; Michahelles, Florian; Sutanto, Juliana; Fleisch, Elgar. "Towards location-aware mobile web browsers" (http:/ / www. im. ethz. ch/ people/ karpischek/ mum09_location_aware_mobile_browser_final. pdf) (PDF). . Retrieved 22 November 2009. [3] http:/ / www. youtube. com/ watch?v=8EA8xlicmT8 [4] Kirkpatrick, Marshall (16 December 2009). "Lonely Planet Launches Augmented Reality Apps" (http:/ / www. nytimes. com/ external/ readwriteweb/ 2009/ 12/ 16/ 16readwriteweb-lonely-planet-launches-augmented-reality-ap-36792. html). The New York Times. . Retrieved 16 December 2009. [5] http:/ / www. youtube. com/ watch?v=ReH9dmqfOqA [6] Höllerer, Tobias H.; Feiner, Steven K.. "Mobile Augmented Reality" (http:/ / citeseerx. ist. psu. edu/ viewdoc/ download?doi=10. 1. 1. 131. 1679& rep=rep1& type=pdf) (PDF). . Retrieved January 2004. [7] http:/ / graphics. cs. columbia. edu/ projects/ mars/ [8] "Port Based Network Access Control" (http:/ / www. ieee802. org/ 1/ pages/ 802. 1x-2004. html). Institute of Electrical and Electronics Engineers. . Retrieved 23 May 2008.

42

Zombie ShootAR

43

Zombie ShootAR Zombie ShootAR Developer(s)

metaio GmbH

Platform(s)

Symbian

Release date(s) June 7, 2010 mobile Augmented Reality game

Genre(s)

Zombie ShootAR is an augmented reality video game for the Symbian platform, which utilizes the camera live-stream for optical tracking and 3D rendering. Junaio

3D computer vision 3D computer vision, or human-like computer vision, is the ability of double camera processor powered devices to acquire a real time picture of the world in three dimensions. It was not possible to achieve such instant speeds of 3D info capturing using the traditional technology of stereo cameras due to huge resources required to process and compare/combine images received from two misaligned image sensors. In 2006 Science Bureau Inc. came up with an idea how to seamlessly transition from 2D to 3D technology in personal computers and other mobile devices to enable them to see the world as humans. The idea behind the invention was quite simple. In order to avoid enormous processing resources for compensation of misalignment of two image sensors said image sensors need to be precisely aligned so that the rows of their sensing elements are parallel to the line connecting their optical centers. Then the rows can be easily compared on the fly. There is no further need in powerful image processors what makes the technology very inexpensive and suitable for its low budget mass implementation. The idea was introduced to all major companies playing in the market of home electronics and computer games back in 2007 but was not 3D Computer Vision System acquired by any of them. In 2010 US Patent 7,729,530 [1] was issued to protect the intellectual rights. The same year all kinds of 3D devices began flooding the market in the North America. Despite this recent breakthrough in 3D technologies there is still a lack of real time 3D vision computer systems on the market. There are a few high profile products that are close to achieving instant 3D image reconstruction. Nevertheless, they are still far from providing real time image and gesture recognition for computer games and device control. Let’s take a closer look at them. 1. Microsoft’s Kinect for xBox 360. The product uses the suggested advanced technology in part of having two image sensors with aligned rows of sensing elements. However, Microsoft utilizes a special source of light producing a large pattern on surrounding objects to get captured and recognized by the imaging part. Due to specifics of the pattern the image resolution is very low and the device is only capable of recognizing major body movements. The device uses low resolution image sensors and still not fast enough to process received images.

3D computer vision 2. Fuji’s stereo camera. Precisely aligned sensors with high grade optics. Could provide a great 3D real time image if connected and controlled by computer. 3. Panasonic’s 3D camcorder. Great idea with mechanically alignable sensors to get 3D video images. 4. HTC has unveiled the EVO 3D, a follow-up to Sprint Nextel's breakout smartphone. It has a 4.3-inch touchscreen, which can display eye-popping 3D without needing glasses. Users will also be able to capture photos and videos in 3D using a pair of cameras on the back. 5. LG Electronics has been working for a year and half on a 3D smartphone of its own.The Optimus 3D, as it's been called, will launch on AT&T Mobility's network with the name Thrill 4G. LG developers spent a great deal of time fine-tuning the pair of 5-megapixel cameras to accurately capture 3D media. Calibrating the cameras to produce good-looking stills and video is more difficult than pulling off a glasses-free display. 6. Nintendo's 3DS also has a pair of cameras for capturing scenes in 3D, and it works quite well. Being the first out of the gate to offer a mainstream glasses-free 3D gadget, Nintendo expected to find competitors, and it soon did when LG announced its phone. 7. Both LG and HTC are planning to debut tablet computers that should be able, like their phones, capture 3D with a pair of cameras. It is obvious that all of the above companies are on the right track building their products based on the technology to align two image sensors as precisely as possible. Therefore, if the technology keeps going in the defined direction, we are to soon witness computers recognizing and communicating with their users; robots being everywhere and doing everything from surgeries to driving cars; 3D virtual games with instant Avatar image creation of the players; 3D technologies everywhere from smartphones to TV.

References "United States Patent and Trademark Office: US Patent 7,729,530" [1] "Kinect xBox 360" [2] "Finepix Real 3DW1 Stereo Camera by Fuji" [3] "3D Camcorder by Panasonic" [4] "3-D smartphones ditch the glasses, CNN, 03/24/2011" [5] "Computer stereo vision" "3D reconstruction" [1] http:/ / patft. uspto. gov/ netacgi/ nph-Parser?Sect1=PTO1& Sect2=HITOFF& d=PALL& p=1& u=%2Fnetahtml%2FPTO%2Fsrchnum. htm& r=1& f=G& l=50& s1=7729530. PN. & OS=PN/ 7729530& RS=PN/ 7729530 [2] http:/ / www. xbox. com/ en-ca/ kinect/ ?WT. srch=1 [3] http:/ / www. fujifilm. com/ products/ 3d/ camera/ finepix_real3dw1/ [4] http:/ / www2. panasonic. com/ consumer-electronics/ shop/ Cameras-Camcorders/ Camcorders/ model. HDC-SDT750K_11002_7000000000000005702 [5] http:/ / www. cnn. com/ 2011/ TECH/ mobile/ 03/ 24/ 3d. phones. tablets/ index. html?hpt=Sbin

44

Agent Vi (Agent Video Intelligence)

45

Agent Vi (Agent Video Intelligence) Agent Video Intelligence Type Privately held company

Agent Vi (Agent Video Intelligence), previously known as Aspectus Video Intelligence [1] is a Video Analytics software company that aims to transform traditional surveillance systems into intelligent tools that can detect and analyze events involving people, objects and vehicles. Agent Vi’s solutions are meant to enable real-time detection and alerts, as well as forensic search and statistical analysis. Agent Vi is focused on introducing its technology to multiple markets including security, safety and business intelligence.

Technology Agent Vi’s solutions include Vi-System for real-time analysis of video surveillance networks and Vi-Search for post-event retrieval and analysis. Agent Vi uses a patented technology called Image Processing over IP (IPoIP) which distributes the video analysis task between a first stage which is performed inside the IP camera and a second stage which is performed at the server.[2] Agent Vi claims that this allows a single server to analyze hundreds of cameras while providing a higher probability of detection (POD) and lower false alarm rates (FAR) for a variety of video analytics tasks.

Vi-System Vi-System is Agent Vi's original product and is meant to perform real-time analysis of the video stream, by identifying and generating alerts for a variety of user-defined events relating to people, vehicles and objects. The company claims that Vi-System offers effective monitoring of multiple video sources in parallel, enabling automatic detections, alerts and responses to events, as they occur.

Vi-Search Vi-Search, is Agent Vi's latest offering and purportedly enables rapid and effective retrieval and presentation of specific video segments, events and data from large amounts of recorded video. The software is meant to analyze the video stream, generate metadata describing the scene content, and allow for later retrieval and analysis of the video through an automatic search within the stored metadata.

Integration Partners Agent Vi claims their open architecture solutions can be seamlessly integrated with a wide range of market-leading edge devices and video management systems, in both new and existing surveillance networks. Some of the edge device and video management system vendors whose products are integrated with Agent Vi include Axis Communications, Cisco, COE, DVTel, EVT, Genetec, IQinVision, LenSec, MangoDSP, Milestone, OnSSI, Proximex, Rontal, Salient Systems, Sony, Verint, Vicon, Vigilant, Vivotek , and others.

Agent Vi (Agent Video Intelligence)

Industries served Agent Vi’s products are deployed in multiple industries such as Mass Transportation Hubs, City Surveillance, Critical Infrastructure, Government & Military Facilities, Highway Surveillance, and Retail environments.

Applications Applications for Vi-System Applications for Vi-Systems include Security and Perimeter Protection, Safety, Traffic Monitoring, Asset Protection and Business Intelligence.

Applications for Vi-Search The market for video analytics in search and investigation is growing,[3] with increased applications for Vi-Search in the areas of Business Intelligence, Operational Efficiency, Forensic Analysis, as well as Time-Critical Security and Law Enforcement Operations.

References [1] http:/ / www. thefreelibrary. com/ Aspectus+ Changes+ Name+ to+ Agent+ Video+ Intelligence,+ Expands+ Global. . . -a0151539927 [2] http:/ / www. securityinfowatch. com/ root+ level/ 1295973 [3] http:/ / ipvideomarket. info/ report/ video_analytic_search_from_agentvi_examined

External links • http://www.agentvi.com

46

Automated optical inspection

Automated optical inspection Automated optical inspection (AOI) is an automated visual inspection of a wide range of products, such as printed circuit boards (PCBs), LCDs, transistors, automotive parts, lids and labels on product packages or agricultural products (seed corn or fruits). In case of PCB-inspection, a camera autonomously scans the device under test (DUT) for variety of surface feature defects such as scratches and stains, open circuits, short circuits, thinning of the solder as well as missing components, incorrect components, and incorrectly placed components. Agricultural inspections might check for variations in part color, perhaps to find ripe fruit. AOI is a type of white box testing. It is commonly used in the manufacturing process because it is a non-contact test method. AOI is able to perform most of the visual checks performed previously by manual operators, and far more swiftly and accurately. AOI systems are implemented at many stages through the manufacturing process. They are used for inspecting parts that have limited and known variations. For defect or flaw detection, the AOI system looks for differences from a perfect part.[1] There are systems capable of bare board inspection, Solder Paste inspection (SPI), as well as inspecting the component placement prior to reflow, the post-reflow component conditions, and post-reflow solder joints. These inspection devices all have some common attributes that affect capability, accuracy, and reliability. In this way AOI can be used to detect problems early in the production process. Since faults cost more to fix later in the production process, it is essential to notice problems early. For example, problems in the solder and assembly area of a PCB can be seen early in the production process and information used to feedback quickly to previous stages, avoiding the production of too many boards with the same problem.[2] Low costs and programming efforts make AOI a practical and powerful quality tool for both prototypes and high-volume assembles. It is often paired with the testing provided by boundary scan test, in-circuit test, x-ray test, and functional test. In many cases, smaller circuit board designs are driving up the demand for AOI versus in-circuit test.

Process A machine vision or an AOI system can acquire millions of data points (pixels) in a fraction of a second. These data points are used for visual inspection and precision measurement. AOI visually scans the surface of the PCB. The board is lit by several light sources and observed by a scanner or by a number of high definition cameras. This enables the monitoring of all areas of the board, even those hidden in one direction by other components. It should be noted that each manufacturer of AOI systems utilizes different inspection algorithms and lighting techniques, each of these systems may have varying strengths and weaknesses depending upon the item/product it is inspecting.

Light source Lighting preprocesses the image to amplify features that need to be inspected and suppress noise. Advances in lighting have improved the capabilities of vision systems, in part by reducing the computation required by the vision computer. In essence this means that the lighting combinations ideally will improve the image quality to improve the efficiency of the AOI system's decision making process. Most AOI systems will have predefined lighting combinations depending upon the mode of operation and type of product being inspected, these combinations will require no user interaction and the system software/algorithms will manipulate and choose the best image for analysis. However some customization is often required and with that these systems will provide interfacing for the end user. The test engineer has the option to choose which light source to use for the lighting: a LED, fluorescent light, or in certain industries other light sources such as IR or UV. This depends on the area to be inspected, industry, and other variables. Some users believe that LED light measures post-print solder brick height more accurately than a

47

Automated optical inspection fluorescent light source. This accuracy also makes LED light a good tool for post-reflow solder joint inspection. Fluorescent light, on the other hand is excellent for inspection of component placement pre-reflow but is typically used for location inspection only. The problem with it is that because of the frequent light changes the fluorescent lamps can degrade quickly. For these above reasons LED lighting has become very popular among most AOI companies in the PCB industry. The adoption of standard LED-based lighting has improved AOI systems because it is very stable and easily controlled when compared to older incandescent and fluorescent lighting. LEDs are not perfect: they become darker with age, but this can be easily compensated for by increasing the current to the LED or group of LEDs. Another lighting method projects a pattern of light on an object, often by using a laser with a holographic lens or a white light source with an attached projection grid. The distortions of this structured light pattern can be measured and processed to recover the object's 3D structure. AOI systems using structured light can, for example, compare complex objects such as engine blocks or PCBs to the designed shape in CAD files.[1] These techniques in the PCB industry are referred to as laser triangulation and phase shift profilometry Profilometer respectively. Both are valid techniques for acquiring three dimensional (3-D) information from the PCB The position of the light sources and laser inside the machine are important. Angles of approach are calculated into the inspection algorithms providing enhanced accuracy of the measurements. Also the angle of approach can be particularly important in certain applications because surrounding objects may interfere with the light approach to the target object. An example would be a tall component on a circuit board that blocks the visual approach of the light/camera system to the target component or solder deposit for example.

Capturing an image If a scanner is used it has to scan the surface of the PCB from above only once. If image cameras are used, one must first determine the number of cameras needed. There are systems with only one camera which scans the DUT from above and systems with a couple of cameras from all sides. To be able to scan the device from all points of view, the cameras should be able to move in both X- and Y-direction controlled by software. To program this software the test engineer must have the CAD data. Then the type of cameras must be chosen. There are several types in use today. Streaming video 2D frame grabbers are common. They utilize a motion capture video camera that extracts one frame from a streaming video and creates a still image. However, the system sacrifices image quality for speed and efficiency. A second type of camera imaging system is the Line Scan Still Image Camera. In this system, a still camera is placed relatively close to the target. Because of this, this system requires a very good lighting system. Unfortunately, the image can be distorted by subsystem imperfections such as transporter movement. This makes obtaining precise positioning and measurements difficult when compared to other types of systems. A benefit of the Line Scan Still Image Camera is the image acquisition speed, which is faster than a CCD camera. Another camera imaging system is the 2D Charge-coupled device or CCD. The CCD is used for high-end and special applications such as space and military technologies. This system creates high precision still images in color [3] that are more accurate than other systems. Each industry is different in how image acquisition signals are transferred to the camera. It can be hardware driven via a mechanical signal such as a proximity sensor, laser interruption, drive system encoder position, or software. Regardless of the signal the AOI system interprets the signal which triggers the vision assembly (which could be a single frame grabber and camera combination or more advanced as already discussed) that the object is in a known location and to begin image acquisition. The vision computer then triggers the camera/cameras to simultaneously acquire images of the device.

48

Automated optical inspection

Programming The AOI system takes time to "learn" the object which in the PCB industry is typically a circuit board. Several methods of learning exist but the two most common are image matching and algorithm based. Image matching is when a "golden board" is introduced to the system and the attributes of each component, solder deposit, etc. are learned into the system. This could include but not limited to color, white pixel count, dark pixel count, transition points, and relative position of transition points. Image based systems require a few example products typically to learn all possible variations. The other method is algorithm based programming; this is where the user can apply rules and measurement algorithms for inspection. Algorithm based programming typically also requires product examples for programming but not usually as many as image based programming. There are pros and cons for each technique which can include setup time, complexity of the system, program transportability, time to generate a program, and call effectiveness. The AOI system needs to be able to add the learned information acquired from the above inspection techniques to its memory. It has to "remember" the different types of components, their positions and also to check the quality of the soldered joints. It must be able to recognize and adapt to differences in the appearances of the board resulting from normal process variations, but must be able to recognize any that affect performance. To achieve this it is normal to run a number of good boards (golden boards) through the system before full production starts so that the system can "learn" the board. Currently the best AOI machines can be set up relatively quickly and then they are reliably able to inspect boards.[2] This can be done by offline programming. This reduces production downtime, and the programmer has time to enter accurate parameters without stopping an assembly line. These systems also have a component database that saves data so that it does not have to be entered every time a new board is produced. In case of automotive parts inspection for example, the system should know what features to measure or for sorting agricultural products the color of ripe fruit should be known.

Data collection To obtain the necessary data one can use 3D software imaging or LED light measurements, which are common methods for measuring solder joint parameters. Typical systems use both methods to obtain accurate measurements. These systems use a directed light source or refracting light to measure height, area, and volume. The vision computer and its software analyze data (images) and calculate statistical process control (SPC) results in these areas. The results of the inspection are used to reject defective parts. To reduce the computation required by the vision computer so that parts are quickly inspected it is a good idea to use a type of mechanical restraint known as staging or fixturing. The part is positioned in a known location and variability in where the parts are and how they look is decreased.[1] Other data collection concerns are factory software integration of inspection results, how long to keep the data/measurements, and what data is available. Each AOI system is going to vary in what is available and each industry and customer will have varying needs. For example military and aerospace industries may have the need to store data for years or until the expected end of life (EOL) of a product while commercial manufacturing such as mobile phones may only require data for a few weeks or possibly even just for that day or shift.

49

Automated optical inspection

System memory System memory is also important. Upgradeable systems that have the capacity for expanded memory are desirable. Databases expand every day particularly with the increase in lead-free components, solders, boards and assemblies that are rapidly entering the industry.

System speed System speed is influenced largely by the size and complexity of the boards being inspected and the level of inspection. Control over the pace of the production line can be influenced by the AOI system. Since one hundred percent inspection is not always necessary, valuable time can be saved by dialing in the appropriate inspection level.

System accuracy The accuracy for finding defects such as missing components, skewed components, reversed components, or wrong valued components depends not only on the capability of the inspection system but also on the accuracy of the program supplied by the user.[3] Accuracy of inspection is becoming even more important with the smaller devices especially in the PCB industry devices such as 0201 and 01005 resistors and capacitors are so small that the human eye can not detect their presence or absence on a PCB.

Types of AOI machines Stand-alone and inline AOI systems When using Stand-alone machines the PCB has to be inserted into and after the test taken out of the machine manually. Inline systems on the other hand are part of the production line so the PCBs move into and go out of the machine automatically.

Closed- and open-top AOI systems Inside Open-top machines the artificial light is distracted by light sources different than the ones used by the system itself (e.g. sunlight). Closed-top machines eliminate any light pollution because they are closed from all sides. In this way only artificial light is available inside the machine making these type of systems much more efficient.

AOI and combined AOI/AXI systems Today's AOI systems have the capability to inspect visible solder joints on capacitors, resistors, and other components. However, without additional capabilities such as Automated x-ray inspection (AXI), ball grid array [3] (BGA) and "J" leaded components are limited to polarity, missing, and placement error detection. That is why Combined AOI/AXI systems can provide the necessary mixture for high-performance and are becoming more and more popular.

Inspection in surface-mount technology The growing demand for Surface-mount technology equipment is reducing the need for expensive rework and repair while increasing throughput. As PCB assembly manufacturers aim for a zero-tolerance regime, the demand for pre-solder paste and pre-reflow optical inspection equipment that detect faults such as poor quality solder joints, tombstoning, and other post reflow defects is expected to rise significantly. Technological improvements in AOI equipment have resulted in higher throughput, repeatability, and reliability as well as increasing quality and production yields for the PCB assembly manufacturers.[4] AOI's for a PCB board with components may inspect the following features:

50

Automated optical inspection • • • • • • • • • • • • • • • • • • •

Area Defects Billboarding Component offset Component polarity Component presence/absence Component skew Excessive solder joints Flipped component Height Defects Insufficient paste around Leads Insufficient solder joints Lifted leads No-population tests Paste registration Severely damaged components Solder bridges Tombstoning Volume defects Wrong part

Bare PCB inspection AOI for a bare PCB board inspection may detect these features: • • • • • • •

Line width violations. Spacing violation. Access copper. Missing pad. I.e. a feature that should be on the board is missing. Shorts circuits. Cuts. Hole breakage. I.e. a drilled hole (via) is outside of its landing pad.

The triggering of a defects report may be either rule based (e.g. no lines on the board should be smaller than 50μ) or CAD based in which the board is locally compared with the intended design. This inspection is much more reliable and repeatable than manual visual inspection.

Typical vendors • • • • •

Orbotech Teradyne Viscom Omron Saki Corporation

51

Automated optical inspection

52

Related technologies The following are related technologies and are also used in electronic production to test for the correct operation of Electronics Printed Circuit boards • • • • •

AXI Automated x-ray inspection JTAG Joint Test Action Group ICT In-circuit test Functional testing under vehicle inspection

References [1] [2] [3] [4]

Applying Automated Optical Inspection (http:/ / 66. 11. 147. 102/ public/ ipd/ whitepapers/ EvalEng_Dawson. pdf) ATE Automatic test equipment (http:/ / www. radio-electronics. com/ info/ t_and_m/ ate/ automatic-test-equipment-basics. php) Kelly, Joe. Tech Tips...Automated Optical Inspection (http:/ / www. empf. org/ empfasis/ archive/ 1203aoi. htm) Growth Opportunities for the World SMT Inspection Equipment Markets (http:/ / www. frost. com/ prod/ servlet/ report-brochure. pag?id=F324-01-00-00-00)

Automatic image annotation Automatic image annotation (also known as automatic image tagging) is the process by which a computer system automatically assigns metadata in the form of captioning or keywords to a digital image. This application of computer vision techniques is used in image retrieval systems to organize and locate images of interest from a database. This method can be regarded as a type of multi-class image classification with a very large number of classes - as large as the vocabulary size. Typically, image analysis in the form of extracted feature vectors and the training annotation words are used by machine learning techniques to attempt to automatically apply annotations to new images. The first methods learned the correlations between image features and training annotations, then techniques were developed using machine translation to try and translate the textual vocabulary with the 'visual vocabulary', or clustered regions known as blobs. Work following these efforts have included classification approaches, relevance models and so on. The advantages of automatic image annotation versus content-based image retrieval are that queries can be more naturally specified by the user [1]. CBIR generally (at present) requires users to search by image concepts such as color and texture, or finding example queries. Certain image features in example images may override the concept that the user is really focusing on. The traditional methods of image retrieval such as those used by libraries have relied on manually annotated images, which is expensive and time-consuming, especially given the large and constantly-growing image databases in existence. [2]

Some annotation engines are online, including the ALIPR.com real-time tagging engine developed by Penn State researchers, and Behold [3] - an image search engine that indexes over 1 million Flickr images using automatically generated tags.

Automatic image annotation

53

Some major work • Word co-occurrence model Y Mori, H Takahashi, and R Oka (1999). "Image-to-word transformation based on dividing and vector quantizing images with words.". Proceedings of the International Workshop on Multimedia Intelligent Storage and Retrieval Management. • Annotation as machine translation P Duygulu, K Barnard, N de Fretias, and D Forsyth (2002). "Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary" [4]. Proceedings of the European Conference on Computer Vision. pp. 97–112. • Statistical models J Li and J Z Wang (2006). "Real-time Computerized Annotation of Pictures" pp. 911–920.

[5]

. Proc. ACM Multimedia.

J Z Wang and J Li (2002). "Learning-Based Linguistic Indexing of Pictures with 2-D MHMMs" ACM Multimedia. pp. 436–445.

[6]

. Proc.

• Automatic linguistic indexing of pictures J Li and J Z Wang (2008). "Real-time Computerized Annotation of Pictures" Analysis and Machine Intelligence.

[7]

. IEEE Trans. on Pattern

J Li and J Z Wang (2003). "Automatic Linguistic Indexing of Pictures by a Statistical Modeling Approach" [8]. IEEE Trans. on Pattern Analysis and Machine Intelligence. pp. 1075–1088. • Hierarchical Aspect Cluster Model K Barnard, D A Forsyth (2001). "Learning the Semantics of Words and Pictures" International Conference on Computer Vision. pp. 408–415.

[9]

. Proceedings of

• Latent Dirichlet Allocation model D Blei, A Ng, and M Jordan (2003). "Latent Dirichlet allocation" [10]. Journal of Machine Learning Research. pp. 3:993–1022. • Supervised multiclass labeling G Carneiro, A B Chan, P Moreno, and N Vasconcelos (2006). "Supervised Learning of Semantic Classes for Image Annotation and Retrieval" [11]. IEEE Trans. on Pattern Analysis and Machine Intelligence. pp. 394–410. • Texture similarity R W Picard and T P Minka (1995). "Vision Texture for Annotation" [12]. Multimedia Systems. • Support Vector Machines C Cusano, G Ciocca, and R Scettini (2004). "Image Annotation Using SVM" Imaging IV.

[13]

. Proceedings of Internet

• Ensemble of Decision Trees and Random Subwindows R Maree, P Geurts, J Piater, and L Wehenkel (2005). "Random Subwindows for Robust Image Classification" [14] . Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition. pp. 1:34-30. • Maximum Entropy J Jeon, R Manmatha (2004). "Using Maximum Entropy for Automatic Image Annotation" Image and Video Retrieval (CIVR 2004). pp. 24–32. • Relevance models

[15]

. Int'l Conf on

Automatic image annotation

54

J Jeon, V Lavrenko, and R Manmatha (2003). "Automatic image annotation and retrieval using cross-media relevance models" [16]. Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 119–126. • Relevance models using continuous probability density functions V Lavrenko, R Manmatha, and J Jeon (2003). "A model for learning the semantics of pictures" Proceedings of the 16th Conference on Advances in Neural Information Processing Systems NIPS.

[17]

.

• Coherent Language Model R Jin, J Y Chai, L Si (2004). "Effective Automatic Image Annotation via A Coherent Language Model and Active Learning" [18]. Proceedings of MM'04. • Inference networks D Metzler and R Manmatha (2004). "An inference network approach to image retrieval" the International Conference on Image and Video Retrieval. pp. 42–50.

[19]

. Proceedings of

• Multiple Bernoulli distribution S Feng, R Manmatha, and V Lavrenko (2004). "Multiple Bernoulli relevance models for image and video annotation" [20]. IEEE Conference on Computer Vision and Pattern Recognition. pp. 1002–1009. • Multiple design alternatives J Y Pan, H-J Yang, P Duygulu and C Faloutsos (2004). "Automatic Image Captioning" [21]. Proceedings of the 2004 IEEE International Conference on Multimedia and Expo (ICME'04). • Natural scene annotation J Fan, Y Gao, H Luo and G Xu (2004). "Automatic Image Annotation by Using Concept-Sensitive Salient Objects for Image Content Representation" [22]. Proceedings of the 27th annual international conference on Research and development in information retrieval. pp. 361–368. • Relevant low-level global filters A Oliva and A Torralba (2001). "Modeling the shape of the scene: a holistic representation of the spatial envelope" [23]. International Journal of Computer Vision. pp. 42:145–175. • Global image features and nonparametric density estimation A Yavlinsky, E Schofield and S Rüger (2005). "Automated Image Annotation Using Global Features and Robust Nonparametric Density Estimation" [24]. Int'l Conf on Image and Video Retrieval (CIVR, Singapore, Jul 2005). • Video semantics N Vasconcelos and A Lippman (2001). "Statistical Models of Video Structure for Content Analysis and Characterization" [25]. IEEE Transactions on Image Processing. pp. 1–17. • Image Annotation Refinement Yohan Jin, Latifur Khan, Lei Wang, and Mamoun Awad (2005). "Image annotations by combining multiple evidence & wordNet" [26]. 13th Annual ACM International Conference on Multimedia (MM 05). pp. 706–715. Changhu Wang, Feng Jing, Lei Zhang, and Hong-Jiang Zhang (2006). "Image annotation refinement using random walk with restarts" [27]. 14th Annual ACM International Conference on Multimedia (MM 06). Changhu Wang, Feng Jing, Lei Zhang, and Hong-Jiang Zhang (2007). "content-based image annotation refinement" [28]. IEEE Conference on Computer Vision and Pattern Recognition (CVPR 07). • Automatic Image Annotation by Ensemble of Visual Descriptors Emre Akbas and Fatos Y. Vural (2007). "Automatic Image Annotation by Ensemble of Visual Descriptors" [29] . Intl. Conf. on Computer Vision (CVPR) 2007, Workshop on Semantic Learning Applications in

Automatic image annotation Multimedia. • Simultaneous Image Classification and Annotation Chong Wang and David Blei and Li Fei-Fei (2009). "Simultaneous Image Classification and Annotation" [30]. Intl. Conf. on Computer Vision (CVPR).

References • Datta, Ritendra; Dhiraj Joshi, Jia Li, James Z. Wang (2008). "Image Retrieval: Ideas, Influences, and Trends of the New Age" [31]. ACM Computing Surveys 40: 1. doi:10.1145/1348246.1348248. • Nicolas Hervé; Nozha Boujemaa (2007). "Image annotation : which approach for realistic databases ?" [32]. ACM International Conference on Image and Video Retrieval. • M Inoue (2004). "On the need for annotation-based image retrieval" [1]. Workshop on Information Retrieval in Context. pp. 44–46.

External links • ALIPR.com [33] - Real-time automatic tagging engine developed by Penn State researchers. • Behold Image Search [3] - An image search engine that indexes over 1 million Flickr images using automatically generated tags. • SpiritTagger Global Photograph Annotation [34] - Annotation system from UCSB on 1.4 million images that predicts where a photo was taken and suggests tags.

References [1] http:/ / research. nii. ac. jp/ ~m-inoue/ paper/ inoue04irix. pdf [2] http:/ / www. alipr. com [3] http:/ / photo. beholdsearch. com/ search. jsp [4] http:/ / vision. cs. arizona. edu/ kobus/ research/ publications/ ECCV-02-1/ [5] http:/ / www-db. stanford. edu/ ~wangz/ project/ imsearch/ ALIP/ ACMMM06/ [6] http:/ / www-db. stanford. edu/ ~wangz/ project/ imsearch/ ALIP/ ACM02/ [7] http:/ / infolab. stanford. edu/ ~wangz/ project/ imsearch/ ALIP/ PAMI08/ [8] http:/ / www-db. stanford. edu/ ~wangz/ project/ imsearch/ ALIP/ PAMI03/ [9] http:/ / kobus. ca/ research/ publications/ ICCV-01/ [10] http:/ / www. ics. uci. edu/ ~liang/ seminars/ win05/ papers/ blei03-latent-dirichlet. pdf [11] http:/ / www. svcl. ucsd. edu/ publications/ journal/ 2007/ pami/ pami07-semantics. pdf [12] http:/ / citeseer. ist. psu. edu/ picard95vision. html [13] http:/ / adsabs. harvard. edu/ cgi-bin/ nph-bib_query?bibcode=2003SPIE. 5304. . 330C& amp;db_key=INST [14] http:/ / www. montefiore. ulg. ac. be/ ~maree/ #publications [15] http:/ / ciir. cs. umass. edu/ pubfiles/ mm-355. pdf [16] http:/ / ciir. cs. umass. edu/ pubfiles/ mm-41. pdf [17] http:/ / ciir. cs. umass. edu/ pubfiles/ mm-46. pdf [18] http:/ / www. cse. msu. edu/ ~rongjin/ publications/ acmmm04. jin. pdf [19] http:/ / ciir. cs. umass. edu/ pubfiles/ mm-346. pdf [20] http:/ / ciir. cs. umass. edu/ pubfiles/ mm-333. pdf [21] http:/ / www. informedia. cs. cmu. edu/ documents/ ICME04AutoICap. pdf [22] http:/ / portal. acm. org/ ft_gateway. cfm?id=1009055& type=pdf& coll=GUIDE& dl=GUIDE& CFID=1581830& CFTOKEN=99651762 [23] http:/ / cvcl. mit. edu/ Papers/ IJCV01-Oliva-Torralba. pdf [24] http:/ / km. doc. ic. ac. uk/ www-pub/ civr05-annotation. pdf [25] http:/ / www. svcl. ucsd. edu/ publications/ journal/ 2000/ ip/ ip00. pdf [26] http:/ / portal. acm. org/ citation. cfm?id=1101305& dl=GUIDE, [27] http:/ / portal. acm. org/ citation. cfm?id=1180639. 1180774#, [28] http:/ / ieeexplore. ieee. org/ xpl/ freeabs_all. jsp?arnumber=4270246 [29] http:/ / ieeexplore. ieee. org/ xpls/ abs_all. jsp?arnumber=4270482 [30] http:/ / cs. stanford. edu/ groups/ vision/ documents/ WangBleiFei-Fei_CVPR2009. pdf

55

Automatic image annotation [31] [32] [33] [34]

56

http:/ / infolab. stanford. edu/ ~wangz/ project/ imsearch/ review/ JOUR/ http:/ / www-rocq. inria. fr/ ~nherve/ nherve_civr2007. pdf http:/ / www. alipr. com/ http:/ / cortina. ece. ucsb. edu/ index. php

Automatic number plate recognition Automatic number plate recognition (ANPR; see also other names below) is a mass surveillance method that uses optical character recognition on images to read the license plates on vehicles. They can use existing closed-circuit television or road-rule enforcement cameras, or ones specifically designed for the task. They are used by various police forces and as a method of electronic toll collection on pay-per-use roads and cataloging the movements of traffic or individuals. ANPR can be used to store the images captured by the cameras as well as the text from the license plate, with some configurable to store a photograph of the driver. Systems commonly use infrared lighting to allow the camera to take the picture at any time of the day.[1] [2] ANPR technology tends to be region-specific, owing to plate variation from place to place.

The system must be able to deal with different styles of license plates

Concerns about these systems have centered on privacy fears of government tracking citizens' movements, misidentification and high error rates.

Other names ANPR is sometimes known by various other terms: • Automatic license plate recognition (ALPR) • Automatic vehicle identification (AVI) • Car plate recognition (CPR) • License plate recognition (LPR) • Lecture Automatique de Plaques d'Immatriculation (LAPI)

Development history The ANPR was invented in 1976 at the Police Scientific Development Branch in the UK. Prototype systems were working by 1979, and contracts were let to produce industrial systems, first at EMI Electronics,

Automatic number plate recognition

57

and then at Computer Recognition Systems (CRS) in Wokingham, UK. Early trial systems were deployed on the A1 road and at the Dartford Tunnel. The first arrest through detection of a stolen car was made in 1981.

Components The software aspect of the system runs on standard home computer hardware and can be linked to other applications or databases. It first uses a series of image manipulation techniques to detect, normalize and enhance the image of the number plate, and then optical character recognition (OCR) to extract the alphanumerics of the license plate. ANPR systems are generally deployed in one of two basic approaches: one allows for the entire process to be performed at the lane location in real-time, and the other transmits all the images from many lanes to a remote computer location and performs the OCR process there at some later point in time. When done at the lane site, the information captured of the plate alphanumeric, date-time, lane identification, and any other information that is required is completed in somewhere around 250 milliseconds. This information, now small data packets, can easily be transmitted to some remote computer for further processing if necessary, or stored at the lane for later retrieval. In the other arrangement, there are typically large numbers of PCs used in a server farm to handle high workloads, such as those found in the London congestion charge project. Often in such systems, there is a requirement to forward images to the remote server, and this can require larger bandwidth transmission media.

License Plate Recognition Process

Technology ANPR uses optical character recognition (OCR) on images taken by cameras. When Dutch vehicle registration plates switched to a different style in 2002, one of the changes made was to the font, introducing small gaps in some letters (such as P and R) to make them more distinct and therefore more legible to such systems. Some license plate arrangements use variations in font sizes and positioning—ANPR systems must be able to cope with such differences in order to be truly effective. More complicated systems can cope with international variants, though many programs are individually tailored to each country. The cameras used can include existing road-rule enforcement or closed-circuit television cameras, as well as mobile units, which are usually attached to vehicles. Some systems use [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] infrared cameras to take a clearer image of the plates.

The font on Dutch plates was changed to improve plate recognition.

Automatic number plate recognition

58

ANPR in mobile systems Recent advances in technology have taken automatic number plate recognition (ANPR) systems from fixed applications to mobile ones. Scaled-down components at more cost-effective price points have led to a record number of deployments by law enforcement agencies around the world. Smaller cameras with the ability to read license plates at high speeds, along with smaller, more durable processors that fit in the trunks of police vehicles, allow law enforcement officers to patrol daily with the benefit of license plate reading in real time, when they can interdict immediately.

The Dubai police use three ANPR cameras to monitor vehicles in front and either side of the patrol car

Despite their effectiveness, there are noteworthy challenges related with mobile ANPRs. One of the biggest is that the processor and the cameras must work fast enough to accommodate relative speeds of more than 100 mph (160 km/h), a likely scenario in the case of oncoming traffic. This equipment must also be very efficient since the power source is the vehicle battery, and equipment must be small to minimize the space it requires. Relative speed is only one issue that affects the camera's ability to actually read a license plate. Algorithms must be able to compensate for all the variables that can affect the ANPR's ability to produce an accurate read, such as time of day, weather and angles between the cameras and the license plates. A system's illumination wavelengths can also have a direct impact on the resolution and accuracy of a read in these conditions.

The MiniHawk 2i - one of the most used mobile ANPR cameras in the UK

Installing ANPR cameras on law enforcement vehicles requires careful consideration of the juxtaposition of the cameras to the license plates they are to read. Using the right number of cameras and positioning them accurately for optimal results can prove challenging, given the various missions and environments at hand. Highway patrol requires forward-looking cameras that A Merseyside Police car equipped with mobile span multiple lanes and are able to read license plates at ANPR. very high speeds. City patrol needs shorter range, lower focal length cameras for capturing plates on parked cars. Parking lots with perpendicularly parked cars often require a specialized camera with a very short focal length. Most technically advanced systems are flexible and can be configured with a number of cameras ranging from one to four which can easily be repositioned as needed. States with rear-only license plates have an additional challenge since a forward-looking camera is ineffective with incoming traffic. In this case one camera may be turned backwards.

Automatic number plate recognition

59

Algorithms There are six primary algorithms that the software requires for identifying a license plate: 1. Plate localization – responsible for finding and isolating the plate on the picture.

2. Plate orientation and sizing – compensates for the skew of the plate and adjusts the dimensions to the required size. 3. Normalization – adjusts the brightness and contrast of the image. 4. Character segmentation – finds the individual characters on the plates.

5. Optical character recognition. 6. Syntactical/Geometrical analysis – check characters and positions against country-specific rules.

Steps 2, 3 and 4: The license plate is normalized for brightness and contrast, and then the characters are segmented to be ready for OCR.

The complexity of each of these subsections of the program determines the accuracy of the system. During the third phase (normalization), some systems use edge detection techniques to increase the picture difference between the letters and the plate backing. A median filter may also be used to reduce the visual noise on the image.

Difficulties There are a number of possible difficulties that the software must be able to cope with. These include: • Poor image resolution, usually because the plate is too far away but sometimes resulting from the use of a low-quality camera. • Blurry images, particularly motion blur. • Poor lighting and low contrast due to overexposure, reflection or shadows. • • • •

Early ANPR systems were unable to read white or silver lettering on black background, as permitted on UK vehicles built prior to 1973.

An object obscuring (part of) the plate, quite often a tow bar, or dirt on the plate. A different font, popular for vanity plates (some countries do not allow such plates, eliminating the problem). Circumvention techniques. Lack of coordination between countries or states. Two cars from different countries or states can have the same number but different design of the plate.

While some of these problems can be corrected within the software, it is primarily left to the hardware side of the system to work out solutions to these difficulties. Increasing the height of the camera may avoid problems with objects (such as other vehicles) obscuring the plate but introduces and increases other problems, such as the adjusting for the increased skew of the plate. On some cars, tow bars may obscure one or two characters of the license plate. Bikes on bike racks can also obscure the number plate, though in some countries and jurisdictions, such as Victoria, Australia, "bike plates" are supposed to be fitted. Some small-scale systems allow for some errors in the

Must be able to recognize international license plates as such.

Automatic number plate recognition license plate. When used for giving specific vehicles access to a barricaded area, the decision may be made to have an acceptable error rate of one character. This is because the likelihood of an unauthorized car having such a similar license plate is seen as quite small. However, this level of inaccuracy would not be acceptable in most applications of an ANPR system.

Imaging Hardware At the front end of any ANPR system is the imaging hardware which captures the image of the license plates. The initial image capture forms a critically important part of the ANPR system which, in accordance to the Garbage In, Garbage Out principle of computing, will often determine the overall performance. License plate capture is typically performed by specialized cameras designed specifically for the task. Factors which pose difficulty for license plate imaging cameras include speed of the vehicles being recorded, varying ambient lighting conditions, headlight glare and harsh environmental conditions. Most dedicated license plate capture cameras will incorporate infrared illumination in order to solve the problems of lighting and plate reflectivity. Many countries now use license plates that are retroreflective.[13] This returns the light back to the source and thus improves the contrast of the image. In some countries, the characters on the plate are not reflective, giving a high level of contrast with the reflective background in any lighting conditions. A camera that makes use of active infrared imaging (with a normal colour filter over the lens and an infrared illuminator next to it) benefits greatly from this as the infrared waves are reflected back from the plate. This is only possible on dedicated ANPR cameras, however, and so cameras used for other purposes must rely more heavily on the software capabilities. Further, when a full-colour image is required as well as use of the ANPR-retrieved details it is necessary to have one infrared-enabled camera and one normal (colour) camera working together. To avoid blurring it is ideal to have the shutter speed of a dedicated camera set to 1/1000 of a second. Because the car is moving, slower shutter speeds could result in an image which is too blurred to read using the OCR software, especially if the camera is much higher up Blurry images make OCR difficult or impossible. ANPR systems should have fast shutter speeds to avoid motion blur than the vehicle. In slow-moving traffic, or when the camera is at a lower level and the vehicle is at an angle approaching the camera, the shutter speed does not need to be so fast. Shutter speeds of 1/500 of a second can cope with traffic moving up to 40 mph (64 km/h) and 1/250 of a second up to 5 mph (8 km/h). License plate capture cameras can now produce usable images from vehicles traveling at 120 mph (190 km/h). To maximize the chances of effective license plate capture, installers should carefully consider the positioning of the camera relative to the target capture area. Exceeding threshold angles of incidence between camera lens and license plate will greatly reduce the probability of obtaining usable images due to distortion. Manufacturers have developed tools to help eliminate errors from the physical installation of license plate capture cameras

Circumvention techniques Vehicle owners have used a variety of techniques in an attempt to evade ANPR systems and road-rule enforcement cameras in general. One method increases the reflective properties of the lettering and makes it more likely that the system will be unable to locate the plate or produce a high enough level of contrast to be able to read it. This is typically done by using a plate cover or a spray, though claims regarding the effectiveness of the latter are disputed. In most jurisdictions, the covers are illegal and covered under existing laws, while in most countries there is no law to disallow the use of the sprays.[14] Other users have attempted to smear their license plate with dirt or utilize covers to mask the plate.

60

Automatic number plate recognition

61

Novelty frames around Texas license plates were made illegal in Texas on 1 September 2003 by Texas Senate Bill 439 because they caused problems with ANPR devices. That law made it a Class C misdemeanor (punishable by a fine of up to US $200), or Class B (punishable by a fine of up to US $2,000 and 180 days in jail) if it can be proven that the owner did it to deliberately obscure their plates.[15] The law was later clarified in 2007 to allow Novelty frames. If an ANPR system cannot read the plate it can flag the image for attention, with the human operators looking to see if they are able to identify the alphanumerics. In order to avoid surveillance or penalty charges, there has been an upsurge in car cloning. This is usually achieved by copying registration plates from another car of a similar model and age. This can be difficult to detect, especially as cloners may change the registration plates and travel behavior to hinder investigations. Other possible options include IR emitting LEDs around the license plate which would serve to "blind" cameras.

Police enforcement Germany On 11 March 2008, the Federal Constitutional Court of Germany ruled that some areas of the laws permitting the use of automated number plate recognition systems [16] More in Germany violated the right to privacy. specifically, the court found that the retention of any sort of information (i.e. Number Plate data) which wasn't for any pre-destined use (e.g. for use tracking suspected terrorists or for enforcement of speeding laws) was in violation of German Law.

Ukraine

Closed-circuit television cameras such as these can be used to take the images scanned by automatic number plate recognition systems

The project of system integration «OLLI TECHNOLOGY» and Department of State Traffic Inspection (STI) of the Ministry of Internal Affairs of Ukraine continue experiment on introduction of a modern technical complex which is capable to search stolen cars, drivers deprived of driving license and other problem cars in online mode. The Ukrainian complex "Video control" [17] working by a principle of video fixing of the car with recognition of license plates with check under data base.

Hungary Several Hungarian Auxiliary Police units use a system called Matrix Police [18] in cooperation with the police. It consists of a portable computer equipped with a webcam that scans the stolen car database using automatic number plate recognition. The system is installed on the dashboard of selected patrol vehicles (PDA based handheld versions also exist) and is mainly used to control the license plate of parking cars. As the Auxiliary Police doesn't have the authority to order moving vehicles to stop, If a stolen car is found, the formal police is informed.

Automatic number plate recognition

62

United Kingdom The UK has an extensive (ANPR) automatic number plate recognition CCTV network. Effectively, the police and Security services track all car movements around the country and are able to track any car in close to real time. Vehicle movements are stored for 5 years in the National ANPR Data Center to be analyzed for intelligence and to be used as evidence. In 1997 a system of one hundred ANPR cameras, codenamed GLUTTON, was installed to feed into the automated British Military Intelligence Systems in Northern Ireland. Further cameras were also installed on the British mainland, including unspecified ports on the east and west coasts.

United States In the United States, ANPR systems are more commonly referred to as ALPR (Automatic License Plate Reader/Recognition) technology, due to differences in language (i.e. "number plates" are referred to as "license plates" in American English) Jurisdictions in the U.S. have stated a number of reasons for ALPR surveillance cameras, ranging from locating drivers with suspended licenses or no insurance, to finding stolen vehicles and "Amber Alerts". With funding from the insurance lobby, Oklahoma introduced ALPR with the promise of eliminating uninsured motorists, by integrating it with its existing PikePass hybrid RFID/OCR toll collection system, and unmarked police vehicles used for intelligence gathering. [19] [20]

A City of Alexandria police car equipped with mobile ALPR.

Oklahoma replaced all license tags with ALPR-compatible plates in 2009. In Arizona, insurance companies are helping to fund the purchase of ALPR systems for their local law enforcement agencies to aid in the recovery of stolen vehicles. Other ALPR uses include parking enforcement, and revenue collection from individuals who are delinquent on city or state taxes or fines. A recent initiative by New York State deployed ALPR systems to catch car thieves by tracing suspect plates back to forged documents. Albany, NY police also scan vehicles in their parking lot to check visitors for warrants.[21] In addition to the real-time processing of license plate numbers, ALPR systems in the US collect (and indefinitely store) data from each license plate capture. Images, dates, times and GPS coordinates can be stockpiled and can help place a suspect at a scene, aid in witness identification, pattern recognition or the tracking of individuals. Such data can be used to create specialized databases that can be shared among departments or individuals (such as insurers, banks or auto recovery "repo-men".[22] Specialized databases can also be used to compile personal information on [23] suspected gang members, employees of a business, patrons of a bar, etc., and be individuals such as journalists shared by E-mail or portable flash media. From time to time, states will make significant changes in their license plate protocol that will affect OCR accuracy. They may add a character or add a new license plate design. ALPR systems must adapt to these changes quickly in order to be effective. For the most part, however, the North American design will be based on a variation of the "Zurich Extra Condensed" font.[24] Another challenge with ALPR systems is that some states have the same license plate protocol. For example more than one state may use three letters followed by four numbers. So each time the ALPR systems alarms, it is the user’s responsibility to make sure that the plate which caused the alarm matches the state associated with the license plate listed on the in-car computer.

Automatic number plate recognition

Average Speed cameras ANPR is used for speed limit enforcement in Austria,[25] the UK,[26] Italy, The Netherlands[27] and Dubai (UAE). This works by tracking vehicles' travel time between two fixed points, and calculating the average speed. These cameras are claimed to have an advantage over traditional speed cameras in maintaining steady legal speeds over extended distances, rather than encouraging heavy braking on approach to specific camera locations and subsequent acceleration back to illegal speeds.[28]

The Netherlands Average speed cameras (trajectcontrole) are in place in the Netherlands since 2002. As of July 2009, 12 such cameras are operating, mostly in the west of the country and along the A12.[29] Some of these are divided in several “sections” to allow for cars leaving and entering the motorway. A first experimental system was tested on a short stretch of the A2 in 1997 and was deemed a big success by the police, reducing overspeeding to 0.66%, compared to 5 to 6% when regular speed cameras were used at the same location.[30] The first permanent average speed cameras were installed on the A13 in 2002, shortly after the speed limit was reduced to 80 km/h to limit noise and air pollution in the area.[31] In 2007, average speed cameras resulted in 1.7 million fines for overspeeding out of a total of 9.7 millions. According to the Dutch attorney general, the average number of violation of the speed limits on motorway sections equipped with average speed cameras is between 1 and 2%, compared to 10 to 15% elsewhere.[32]

UK One of the most notable stretches of average speed cameras in the UK is found on the A77 road in Scotland, with 32 miles (51 km) being monitored between Glasgow and Ayr.[33] In 2006 it was confirmed that speeding tickets could potentially be avoided from the 'SPECS' cameras by changing lanes and the RAC Foundation feared that people may play "Russian Roulette" changing from one lane to another to lessen their odds of being caught.[26] However, in 2007 the system was upgraded for multi-lane use and in 2008 the manufacturer described the "myth" as “categorically untrue”.[34] There exists evidence that implementation of systems such as SPECS has a considerable effect on the volume of drivers travelling at excessive speeds; on the strech of road mentioned above (A77 Between Glasgow and Ayr) there has been noted a "huge drop" in speeding violations since the introduction of a SPECS system.[33]

Italy In Italian Highways has developed a monitoring system named Tutor covering more than 1244 km (2007). Further extensions will add 900 km before the end of 2008. The Tutor system is also able to intercept cars while changing lanes.

China In some regions of China on hilly roads near rivers, a unique method is used to measure the average speed of vehicles without the use of cameras or other automated systems. There are checkpoints set up at regular intervals on the road and the shortest time it takes to travel between them is measured. At a checkpoint, a police officer provides the driver with a printout, this printout is needed to pass the next checkpoint and has information on when the driver passed the checkpoint and when they should be expected to arrive at the next. Upon arrival at the next checkpoint, the time of arrival is compared to the time on the printout and if it is less the driver is charged for each minute more they should have taken to reach the checkpoint. There are problems with this system however, as roadside businesses have appeared with the purpose of extending the time drivers spend between checkpoints.

63

Automatic number plate recognition

64

Traffic control Many cities and districts have developed traffic control systems to help monitor the movement and flow of vehicles around the road network. This had typically involved looking at historical data, estimates, observations and statistics such as: • • • • •

Car park usage Pedestrian crossing usage Number of vehicles along a road Areas of low and high congestion Frequency, location and cause of road works

CCTV cameras can be used to help traffic control centres by giving them live data, allowing for traffic management decisions to be made in real-time. By using ANPR on this footage it is possible to monitor the travel of individual vehicles, automatically providing information about the speed and flow of various routes. These details can highlight problem areas as and when they occur and helps the centre to make informed incident management decisions. Some counties of the United Kingdom have worked with Siemens Traffic to develop traffic monitoring systems for their own control centres and for the public.[35] Projects such as Hampshire County Council's ROMANSE [36] provide an interactive and real-time web site showing details about traffic in the city. The site shows information about car parks, ongoing road works, special events and footage taken from CCTV cameras. ANPR systems can be used to provide average driving times along particular routes, giving drivers the ability to choose which one to take. ROMANSE also allows travellers to see the current situation using a mobile device with an Internet connection (such as WAP, GPRS or 3G), thus allowing them to be alerted to any problems that are ahead. The UK company Trafficmaster has used ANPR since 1998 to estimate average traffic speeds on non-motorway roads without the results being skewed by local fluctuations caused by traffic lights and similar. The company now operates a network of over 4000 ANPR cameras, but claims that only the four most central digits are identified, and no numberplate data is retained.[37] [38] [39] • IEEE transactions on Intelligent Transportation Systems(IEEE Intelligent Transportation Systems Society) published some papers on the plate number recognition technologies and applications.

Electronic toll collection Toll roads Ontario's 407 ETR highway uses a combination of ANPR and radio transponders to toll vehicles entering and exiting the road. Radio antennas are located at each junction and detect the transponders, logging the unique identity of each vehicle in much the same way as the ANPR system does. Without ANPR as a second system it would not be possible to monitor all the traffic. Drivers who opt to rent a transponder for C$2.55 per month are not charged the "Video Toll Charge" of C$3.60 for using the road, with heavy vehicles (those with a gross weight of over 5,000 kg) being required to use one. Using either system, users of the highway are notified of the usage charges by post.

The FasTrak system in Orange County uses ANPR and radio transponders

There are numerous other electronic toll collection networks which use this combination of Radio frequency identification and ANPR. These include:

Automatic number plate recognition • • • • • • • • • • • • • • • •

65

Bridge Pass [40] for the Saint John Harbour Bridge in Saint John New Brunswick Quickpass [41] at the Golden Ears Bridge, crossing the Fraser River between Langley and Maple Ridge CityLink & Eastlink in Melbourne, Australia Gateway Motorway and Logan Motorway, Brisbane, Australia FasTrak in California, United States Highway 6 in Israel Tunnels in Hong Kong Autopista Central [42] in Santiago, Chile (site in Spanish) E-ZPass in New York, New Jersey, Massachusetts (as Fast Lane), Virginia (formerly Smart Tag), and other States. Maryland Route 200 uses a combination of E-ZPass and ANPR. TollTag in North Texas. I-Pass in Illinois Pike Pass in Oklahoma. OGS (Otomatik Geçiş Sistemi) used at Bosporus Bridges and Trans European Motorway entry points in İstanbul, Turkey. M50 motorway (Ireland) Westlink Toll in Dublin, Ireland Hi-pass in South Korea Northern Gateway, SH 1, Auckland, New Zealand

See also: List of electronic toll collection systems

Charge zones – the London congestion charge The London congestion charge is an example of a system that charges motorists entering a payment area. Transport for London (TfL) uses ANPR systems and charges motorists a daily fee of £10 paid before 10pm if they enter, leave or move around within the congestion charge zone between 7 a.m. and 6:00 p.m., Monday to Friday. A reduced fee of £9 is paid by vehicle owners who sign up for the automatic deduction scheme. Fines for traveling within the zone without paying the charge are £60 per infraction if paid before the deadline, doubling to £120 per infraction thereafter.

The London congestion charge scheme uses 230 cameras and ANPR to help monitor vehicles in the charging zone

There are currently 1,500 cameras, which use [43] There are also a number of mobile camera units Automatic Number Plate Recognition (ANPR) technology in use. which may be deployed anywhere in the zone. It is estimated that around 98% of vehicles moving within the zone are caught on camera. The video streams are transmitted to a data centre located in central London where the ANPR software deduces the registration plate of the vehicle. A second data centre provides a backup location for image data. Both front and back number plates are being captured, on vehicles going both in and out – this gives up to four chances to capture the number plates of a vehicle entering and exiting the zone. This list is then compared with a list of cars whose owners/operators have paid to enter the zone – those that have not paid are fined. The registered owner [44] of such a vehicle is looked up in a database provided by the DVLA.

Automatic number plate recognition

Stockholm congestion tax In Stockholm, Sweden, ANPR is used for the congestion tax, owners of cars driving into or out of the inner city must pay a charge, depending on the time of the day.

Usage Several companies and agencies use ANPR systems. These include Vehicle and Operator Services Agency (VOSA) [45] , Police Information Technology Organisation (PITO) [46] and Transport for London [47].

Controversy The introduction of ANPR systems has led to fears of misidentification and the furthering of 1984-style surveillance.[48] In the United States, some such as Gregg Easterbrook oppose what they call "machines that issue speeding tickets and red-light tickets" as the beginning of a slippery slope towards an automated justice system: "A machine classifies a person as an offender, and you can't confront your accuser because there is no accuser... can it be wise to establish a principle that when a machine says you did something illegal, you are presumed guilty?" Systems with a simple review step can eliminate this downside. Then the machine reports data - date, time, speed measurement and license plate - a good system records a photo of the event - so a person presented with the data is making an accusation. You will get a copy of the data when you go to court. Similar criticisms have been raised in other countries. Easterbrook also argues that this technology is employed to maximize revenue for the state, rather than to promote safety.[49] The electronic surveillance system produces tickets which in the US are often in excess of $100, and are virtually impossible for a citizen to contest in court without the help of an attorney. The revenues generated by these machines are shared generously with the private corporation that builds and operates them, creating a strong incentive to tweak the system to generate as many tickets as possible. Older systems had been notably unreliable; in the UK this has been known to lead to charges being made incorrectly with the vehicle owner having to pay £10 in order to be issued with proof (or not) of the offense. Improvements in technology have drastically decreased error rates, but false accusations are still frequent enough to be a problem. Perhaps the best known incident involving the abuse of an ANPR database in North America is the case of Edmonton Sun reporter Kerry Diotte in 2004. Diotte wrote an article critical of Edmonton police use of traffic cameras for revenue enhancement, and in retaliation was added to an ANPR database of "high-risk drivers" in an attempt to monitor his habits and create an opportunity to arrest him.[50] [51] The police chief and several officers were fired as a result, and The Office of the Privacy Commissioner of Canada expressed public concern over the "growing police use of technology to spy on motorists." [52] Other concerns include the storage of information that could be used to identify people and store details about their driving habits and daily life, contravening the Data Protection Act along with similar legislation (see personally identifiable information). The laws in the UK are strict for any system that uses CCTV footage and can identify individuals.[53] [54] [55] [56] [57] [58] [59] [60] [61] Also of concern is the safety of the data once it is mined, following the discovery of police surveillance records lost in a gutter.[62] [63] There is also a case in the UK for saying that use of ANPR cameras is against the law under the Regulation of Investigatory Powers Act 2000.[64] The breach exists, some say, in the fact that ANPR is used to monitor the activities of law-abiding citizens and treats everyone like the suspected criminals intended to be surveyed under the act. The police themselves have been known to refer to the system of ANPR as a "24/7 traffic movement database" which is a diversion from its intended purpose of identifying vehicles involved in criminal activities.[65]

66

Automatic number plate recognition

Other uses ANPR systems may also be used for/by: Section control, to measure average vehicle speed over longer distances.[66] Border crossings Automobile repossessions [67] [68] Filling stations to log when a motorist drives away without paying for their fuel. A marketing tool to log patterns of use Targeted advertising, a-la "Minority Report"-style billboards.[69] Traffic management systems, which determine traffic flow using the time it takes vehicles to pass two ANPR sites[70] • Drive Through Customer Recognition, to automatically recognize customers based on their license plate and offer them the items they ordered the last time they used the service, improving service to the customer. • To assist visitor management systems in recognizing guest vehicles. • • • • • • •

Related Research Society • IEEE Intelligent Transportation Systems Society

Measuring ANPR system performance It is not uncommon to read claims of Automatic Number Plate Recognition read rates in excess of 98%. The experience of system operators is that overall read rates for license plates are 90% to 94% in ideal conditions with excellent modern systems. In some older systems overall performance rates are rumoured to be between 60% and 80%. ANPR is a developing technology which is coming of age and functional applications are expanding on a steady basis. Although there has been a significant improvement in recent years in the performance of ANPR systems, how operators can assess and monitor ANPR/LPR systems has not advanced as much.[71]

References [1] [2] [3] [4] [5] [6]

ANPR Tutorial (http:/ / www. anpr-tutorial. com/ ) An introduction to ANPR (http:/ / www. cctv-information. co. uk/ i/ An_Introduction_to_ANPR) Plate Recognition (http:/ / www. photocop. com/ recognition. htm) at PhotoCop.com Algorithm For License Plate Recognition (http:/ / visl. technion. ac. il/ projects/ 2002w02/ ) at VISL, Technion "A Real-time vehicle License Plate Recognition (LPR)" (http:/ / visl. technion. ac. il/ projects/ 2003w24/ ) at visl.technion.ac.il "An Approach To License Plate Recognition" (http:/ / pages. cpsc. ucalgary. ca/ ~federl/ Publications/ licensePlate1996/ license-plate-1996. pdf) – a PDF file describing a University of Calgary project that looks at plate location in raster images [7] A neural network based artificial vision system for license plate recognition, 1997, Sorin Draghici, Dept. of Computer Science, Wayne State University (http:/ / vortex. cs. wayne. edu/ papers/ ijns1997. pdf) [8] License Plate Recognition in Turkey (Plaka Okuma Sistemi) (http:/ / www. grimedia. com/ ) [9] License plate localization and recognition in camera pictures, 2002, Halina Kwaśnicka and Bartosz Wawrzyniak (http:/ / www. ci. pwr. wroc. pl/ ~kwasnick/ download/ kwasnickawawrzyniak. pdf) [10] License Plate Character Segmentation Based on the Gabor Transform and Vector Quantization, 2003, Fatih Kahraman and Muhittin Gokmen (http:/ / www. be. itu. edu. tr/ ~kahraman/ License Plate Character Segmentation Based on the Gabor Transform and Vector Quantization. pdf) [11] Algorithmic and mathematical principles of automatic number plate recognition systems, 2007, Ondrej Martinsky, Brno University of Technology (http:/ / javaanpr. sourceforge. net/ anpr. pdf) [12] A License Plate Recognition algorithm for Intelligent Transportation System applications (http:/ / www. aegean. gr/ culturaltec/ canagnostopoulos/ cv/ new site/ journals. htm) at University of the Aegean and National Technical University of Athens [13] Automatic Number Plate Recognition (http:/ / www. cctv-information. co. uk/ constant3/ anpr. html) [14] Sexton, Steve. " License-plate spray foils traffic cameras (http:/ / www. phantomplate. com/ print_washingtontimes. html)". Retrieved 5 April 2005. [15] Wentworth, Jeff, " Obscured license plate could be motorists' ticket to fine (http:/ / www. senate. state. tx. us/ 75r/ senate/ members/ dist25/ pr03/ p092603a. htm)". Retrieved 5 April 2005.

67

Automatic number plate recognition [16] "Das Bundesverfassungsgericht" (http:/ / www. bverfg. de/ entscheidungen/ rs20080311_1bvr207405. html) (in German). Bverfg.de. 2008-11-03. . Retrieved 2009-02-16. [17] http:/ / ollie. com. ua/ eng/ content/ system-integration-field-search-vehicles [18] http:/ / www. matrixpolice. hu/ matrix/ index. php?Rendsz%26aacute%3Bmfelismer%26eacute%3Bs:Matrix_Police [19] Tulsa World," Cameras for insurance verification considered (http:/ / www. tulsaworld. com/ news/ article. aspx?subjectid=16& articleid=20091129_16_A20_OKLAHO172449)". [20] FOX23 KOKI, " Big Brother? License Plate Cameras - FOX23 News (http:/ / www. fox23. com/ news/ local/ story/ Big-Brother-License-Plate-Cameras/ WTC1ttvQuUaw6Z4SFVI1fw. cspx)". [21] FOX23news, " 27 arrested at jail after spot checks (http:/ / www. fox23news. com/ news/ local/ story/ 27-arrested-at-jail-after-spot-checks/ Fqi_t3B3aky91gTddH_FOw. cspx)". [22] New York Times, " The Wired Repo Man: He’s Not ‘As Seen on TV’ (http:/ / www. nytimes. com/ 2010/ 02/ 28/ automobiles/ 28REPO. html)" [23] Edmonton Sun, " Testimony Heard Regarding Edmonton Police Attempt to Arrest Journalist (http:/ / www. thenewspaper. com/ news/ 06/ 662. asp)" [24] " Embossed License Plate Fonts vs. 3M’s Default Font for Flat Digital Plates (http:/ / www. leewardpro. com/ articles/ licplatefonts/ licplate-fonts-nam-3m-2. html)" [25] See e.g. Section Control: 24.000 Raser angezeigt (http:/ / ooe. orf. at/ stories/ 448174/ ), ORF.at [26] "speeding tickets can potentially be avoided by changing lanes" (http:/ / www. dailymail. co. uk/ pages/ live/ articles/ news/ news. html?in_article_id=410539& in_page_id=1770). The Daily Mail (London). 15 October 2006. . "The Home Office admitted last night that drivers can avoid being caught the by hi-tech 'SPECS' cameras which calculate a car's average speed over a long distance." [27] Description of the system (http:/ / www. trajectcontrole. nl/ ) on the website of the Dutch attorney general. [28] See e.g., the Frequently Asked Question over Trajectcontrole (http:/ / www. om. nl/ onderwerpen/ verkeer/ veelgestelde_vragen/ trajectcontrole/ ) on the website of the Dutch attorney general. [29] Frequently Asked Question over Trajectcontrole (http:/ / www. om. nl/ onderwerpen/ verkeer/ veelgestelde_vragen/ trajectcontrole/ ) on the website of the attorney general. [30] Niemand rijdt meer te hard op de A2, Trouw, October 18, 1997. [31] Nummerborden lezen op de A13, Trouw, May 6, 2002. [32] Tom Kreling, Een duur foutje van de computer, NRC Handelsblad, August 28, 2008. [33] http:/ / www. a77safetygroup. com/ index. cfm/ page/ 40 [34] "Jeremy Clarkson tilts at windmills - Speed camera avoidance is an urban myth" (http:/ / www. theregister. co. uk/ 2008/ 07/ 21/ speed_camera_myth/ ). The Register. . [35] Recognising a new way to keep traffic moving (http:/ / www. siemenstraffic. com/ customcontent/ case_studies/ anpr/ anpr. html) [36] http:/ / www. romanse. org. uk/ [37] PIPS supplies Journey Time Measurement Systems to Trafficmaster (http:/ / www. pipstechnology. co. uk/ script/ download_document. php?file_id=9) [38] BLURA License Plate Recognition Engine (http:/ / www. blura. com/ ) [39] Trunk Roads - PTFM (http:/ / www. trafficmaster. co. uk/ page. cfm?key=network_ptfm-network) [40] http:/ / www. saintjohnharbourbridge. com/ request. htm [41] http:/ / www. translink. ca/ en/ Driving/ Golden-Ears-Bridge. aspx [42] http:/ / www. autopistacentral. cl/ [43] "Met given real time c-charge data" (http:/ / news. bbc. co. uk/ 1/ hi/ uk_politics/ 6902543. stm). BBC. 17 July 2007. . Retrieved 2007-09-20. [44] Transport for London (http:/ / www. cclondon. com/ ) [45] http:/ / www. vosa. gov. uk [46] http:/ / www. pito. org. uk [47] http:/ / www. cclondon. com/ [48] Keeping 1984 in the past (http:/ / www. guardian. co. uk/ technology/ 2003/ jun/ 19/ newmedia. media) 19 June 2003 [49] Lights, Camera, Action (https:/ / ssl. tnr. com/ p/ docsub. mhtml?i=w050228& s=easterbrook022805) 28 February 2005 [50] CBC News. http:/ / www. cbc. ca/ edmonton/ features/ police. [51] (http:/ / www. edmontonpolicewatch. org/ ?p=325) [52] Canada: Privacy Commissioner Concerned Over License Plate Spying (http:/ / www. thenewspaper. com/ news/ 29/ 2933. asp) [53] The London charge zone, the DP Act, and MS .NET (http:/ / www. theregister. co. uk/ 2003/ 02/ 21/ the_london_charge_zone/ ) 21 February 2003 [54] " ANPR Strategy for the Police Service 2005/2006 (http:/ / www. acpo. police. uk/ asp/ policies/ Data/ anpr_strat_2005-08_march05_12x04x05. doc)" Assn Chief Police officers (ACPO) Steering Group. Retrieved 28 September 2005. [55] " Driving crime down (http:/ / police. homeoffice. gov. uk/ news-and-publications/ publication/ operational-policing/ Driving_Crime_Down_-_Denyin1. pdf?view=Binary)". Home Office, October 2004. Retrieved 29 March 2005. [56] Constant, Mike. " CCTV Information – ANPR (http:/ / www. cctv-information. co. uk/ constant3/ anpr. html)". Retrieved 30 March 2005. [57] Hofman, Yoram. " License Plate Recognition - A Tutorial (http:/ / www. licenseplaterecognition. com/ )". Retrieved 28 March 2005. [58] Lucena, Raul. Automatic Number Plate Recognition Tutorial (http:/ / www. anpr-tutorial. com) 24 August 2006.

68

Automatic number plate recognition [59] Lettice, John. " The London charge zone, the DP Act, and MS .NET (http:/ / www. theregister. co. uk/ 2003/ 02/ 21/ the_london_charge_zone/ )". The Register, 21 February 2003. Retrieved 28 March 2005. [60] Lettice, John. " No hiding place? UK number plate cameras go national (http:/ / www. theregister. co. uk/ 2005/ 03/ 24/ anpr_national_system/ )". The Register, 24 March 2005. Retrieved 28 March 2005. [61] Siemens Traffic, " Recognizing a new way to keep traffic moving (http:/ / www. siemenstraffic. com/ customcontent/ case_studies/ anpr/ anpr. html)". Retrieved 3 April 2005. [62] UK: Traffic Camera Data Dropped in Gutter (http:/ / www. mirror. co. uk/ news/ top-stories/ 2008/ 03/ 11/ memory-loss-115875-20347364/ ) [63] Hertfordshire, UK police drop a memory stick containing sensitive personal information on motorists. (http:/ / www. thenewspaper. com/ news/ 22/ 2265. asp) [64] http:/ / www. legislation. gov. uk/ ukpga/ 2000/ 23/ contents [65] http:/ / www. theregister. co. uk/ 2006/ 07/ 17/ anpr_ripa_breach/ [66] Section control (http:/ / www. verkeershandhaving. nl/ index. php?s=44& id=113) [67] New York Times, The Wired Repo Man (http:/ / www. nytimes. com/ 2010/ 02/ 28/ automobiles/ 28REPO. html) [68] WFTV, High-Tech System Helps Repo Man Find Cars (http:/ / www. wftv. com/ news/ 21586788/ detail. html) [69] UK Billboards Equipped with License Plate Spy Cameras (http:/ / www. safespeed. org. uk/ forum/ viewtopic. php?f=13& t=20945) [70] Stockholm Traffic Cameras (http:/ / www. extremecctv. com/ pdf/ productnews/ CN040422-Extreme-CCTV-Announces-Contract-for-Stockholm-Traffic-Cameras. pdf) [71] Measuring ANPR System Performance (http:/ / www. parkingandtraffic. co. uk/ Measuring ANPR System Performance. pdf) ,Parking Trend International, June 2008

Automatic target recognition Automatic target recognition (ATR), is the ability for an algorithm or device to recognize targets or objects based on data obtained from sensors. The application of automatic target recognition technology is a critical element of robotic warfare. ATR systems are used in unmanned aerial vehicles and cruise missiles. General Electric provides an Automatic Target Recognition Unit (ATRU)[1] for the Standoff Land Attack Missile, which processes pre-launch and post-launch targeting data, allows high speed video comparison, and enables the SLAM-ER (Stand-Off Land Attack Missile - Expanded Response) Missile to be truly "Fire and Forget". The simplest version of an ATR system is the IFF transponder. Researchers at the University of Illinois at Urbana-Champaign with the support of DARPA have shown that it is possible to build a synthetic aperture radar image of an aircraft target using passive multistatic radar, possibly detailed enough to enable Automatic Target Recognition (ATR [2]).

References [1] GE - Automatic Target Recognition Unit (ATRU) (http:/ / www. geaviationsystems. com/ Products--/ Digital/ Mission-and-Stores-Management-Systems/ Missile-Subsystems/ Automatic-Target-Recognition-Unit--ATRU-/ index. asp) [2] http:/ / www. ifp. uiuc. edu/ %7Esmherman/ darpa/

69

Check weigher

70

Check weigher A checkweigher is an automatic machine for checking the weight of packaged commodities. It is normally found at the offgoing end of a production process and is used to ensure that the weight of a pack of the commodity is within specified limits. Any packs that are outside the tolerance are taken out of line automatically. A checkweigher can weigh in excess of 500 items per minute (depending on carton size and accuracy requirements). Checkweighers often incorporate additional checking devices such as metal detectors and X-ray machines to enable other attributes of the pack to be checked and acted upon accordingly.

Example checkweigher. Product passes on the conveyor belt where it is weighed

A typical machine A checkweigher incorporates a series of conveyor belts. Checkweighers are known also as belt weighers, in-motion scales, conveyor scales, dynamic scales, and in-line scales. In filler applications, they are known as check scales. Typically, there are three belts or chain beds: • An infeed belt that may change the speed of the package and to bring it up or down to a speed required for weighing. The infeed is also sometimes used as an indexer, which sets the gap between products to an optimal distance for weighing. It sometimes has special belts or chains to position the product for weighing. • A weigh belt. This is typically mounted on a weight transducer which can typically be a strain-gauge load cell or a servo-balance (also known as a force-balance), or sometimes known as a split-beam. Some older machines may pause the weigh bed belt before taking the weight measurement. This may limit line speed and throughput. • A reject belt that provides a method of removing an out-of-tolerance package from the conveyor line. The reject can vary by application. Some require an air-amplifier to blow small products off the belt, but heavier applications require a linear or radial actuator. Some fragile products are rejected by "dropping" the bed so that the product can slide gently into a bin or other conveyor. For high-speed precision scales, a load cell using electromagnetic force restoration(EMFR) is appropriate. This kind of system charges an inductive coil, effectively floating the weigh bed in an electromagnetic field. When the weight is added, the movement of a ferrous material through that coil causes a loss of ElectroMagnetic Force. A precision circuit charges the coil back to its original charge. The amount added to the coil is precisely measured. The voltage produced is filtered and sampled into digital data. That voltage is then passed through a Digital Signal Processor (DSP) filter and ring-buffer to further reduce ambient and digital noise and delivered to a computerized controller. It is usual for a built-in computer to take many weight readings from the transducer over the time that the package is on the weigh bed to ensure an accurate weight reading. Calibration is critical. A lab scale, which usually is in an isolated chamber pressurized with dry nitrogen(pressurized at sea level) can weigh an object within plus or minus 100th of a gram, but ambient air pressure is a factor. This is straightforward when there is no motion, but in motion there is a factor that is not obvious-noise from the motion of a weigh belt, vibration, air-conditioning or refrigeration which can cause drafts. Torque on a load cell causes erratic readings. A dynamic, in-motion checkweigher takes samples, and analyzes them to form an accurate weight over a given time period. In most cases, there is a trigger from an optical(or ultrasonic) device to signal the passing of a package. Once

Check weigher the trigger fires, there is a delay set to allow the package to move to the "sweet spot" (center) of the weigh bed to sample the weight. The weight is sampled for a given duration. If either of these times are wrong, the weight will be wrong. There seems to be no scientific method to predict these timings. Some systems have a "graphing" feature to do this, but it is generally more of an empirical method that works best. • A reject conveyor to enable the out-of-tolerance packages to be removed from the normal flow while still moving at the conveyor velocity. The reject mechanism can be one of several types. Among these are a simple pneumatic pusher to push the reject pack sideways from the belt, a diverting arm to sweep the pack sideways and a reject belt that lowers or lifts to divert the pack vertically. A typical checkweigher usually has a bin to collect the out-of-tolerance packs.

Tolerance methods There are several tolerance methods: • The traditional "minimum weight" system where weights below a specified weight are rejected. Normally the minimum weight is the weight that is printed on the pack or a weight level that exceeds that to allow for weight losses after production such as evaporation of commodities that have a moisture content. The larger wholesale companies have mandated that any product shipped to them have accurate weight checks such that a customer can be confident that they are getting the amount of product for which they paid. These wholesalers charge large fees for inaccurately filled packages. • The European Average Weight System which follows three specified rules known as the "Packers Rules".[1] • Other published standards and regulations such as NIST Handbook 133[2]

Data Collection There is also a requirement under the European Average Weight System that data collected by checkweighers is archived and is available for inspection. Most modern checkweighers are therefore equipped with communications ports to enable the actual pack weights and derived data to be uploaded to a host computer. This data can also be used for management information enabling processes to be fine-tuned and production performance monitored. Checkweighers that are equipped with high speed communications such as Ethernet ports are capable of integrating themselves in to groups such that a group of production lines that are producing identical products can be considered as one production line for the purposes of weight control. For example, a line that is running with a low average weight can be complemented by another that is running with a high average weight such that the aggregate of the two lines will still comply with rules. An alternative is to program the checkweigher to check bands of different weight tolerances. For instance, the total valid weight is 100 grams ±15 grams. This means that the product can weigh 85 g - 115 g. However, it is obvious that if you are producing 10,000 packs a day, and most of your packs are 110 g, you are losing 100 kg of product. If you try to run closer to 85 g, you may have a high rejection rate. EXAMPLE: A checkweigher is programmed to indicate 5 zones with resolution to 1 g: 1. 2. 3. 4. 5.

Under Reject.... the product weighs 84.9 g or less Under OK........ the product weighs 85 g, but less than 95 g Valid........... the product weighs 96 g, but less than 105 g Over OK......... the product weighs 105 g, and less than 114 g Over Reject..... the product weighs over the 115 g limit

With a check weigher programmed as a zone checkweigher, the data collection over the networks, as well as local statistics, can indicate the need to check the settings on the upstream equipment to better control flow into the packaging. In some cases the dynamic scale sends a signal to a filler, for instance, in real-time, controlling the actual flow into a barrel, can, bag, etc. In many cases a checkweigher has a light-tree with different lights to indicate the

71

Check weigher variation of the zone weight of each product.

Application considerations Speed and accuracy that can be achieved by a checkweigher is influenced by the following: • • • • • • • • • • • • • • • • • • • •

Pack length Pack weight Line speed required Pack content (solid or liquid) Motor technology Stabilization time of the weight transducer Airflow causing readings in error Vibrations from machinery causing unnecessary rejects Sensitivity to temperature, as the load cells can be temperature sensitive Quality control tape Band with the principles of quality control weight Weighing tape quality control principles Automatic weighing tape Weight control tape Food weight control tape Drug for weight control tape Detergent industry for the quality control tape Weight control tape for chemical industry Weight control tape for the pasta industry Flour weight control tape for the industry

Applications In-motion scales are dynamic machines that can be designed to perform thousands of tasks. Some are used as simple caseweighers at the end of the conveyor line to ensure the overall finished package product is within its target weight. An in motion conveyor checkweigher can be used to detect missing pieces of a kit, such as a cell phone package that is missing the manual, or other collateral. Checkweighers are typically used on the incoming conveyor chain, and the output pre-packaging conveyor chain in a poultry processing plant. The bird is weighed when it comes onto the conveyor, then after processing and washing at the end, the network computer can determine whether or not the bird absorbed too much water, which as it is further processed, will be drained, making the bird under its target weight. A high speed conveyor scale can be used to change the pacing, or pitch of the products on the line by speeding, or slowing the product speed to change the distance between packs before reaching a different speed going into a conveyor machine that is boxing multiple packs into a box. A checkweigher can be used to count packs, and the aggregate (total) weight of the boxes going onto a pallet for shipment, including the ability to read each package's weight and cubic dimensions. The controller computer can print a shipping label and a bar-code label to identify the weight, the cubic dimensions, ship-to address, and other data for machine ID through the shipment of the product. A receiving checkweigher for the shipment can read the label with a bar code scanner, and determine if the shipment is as it was before the transportation carrier received it from the shipper's loading dock, and determine if a box is missing, or something was pilfered or broken in transit. Checkweighers are also used for Quality management. For instance, raw material for machining a bearing is weighed prior to beginning the process, and after the process, the quality inspector expects that a certain amount of metal was

72

Check weigher removed in the finishing process. The finished bearings are checkweighed, and bearings over- or underweight are rejected for physical inspection. This is a benefit to the inspector, since he can have a high confidence that the ones not rejected are within machining tolerance.A common usage is for throttling plastic extruders such that a bottle used to package detergent meets that requirements of the finished packager. Quality management can use a checkweigher for Nondestructive testing to verify finished goods using common Evaluation methods to detect pieces missing from a "finished" product, such as grease from a bearing, or a missing roller within the housing. Checkweighers can be built with metal detectors, x-ray machines, open-flap detection, bar-code scanners, holographic scanners, temperature sensors, vision inspectors, timing screws to set the timing and spacing between product, indexing gates and concentrator ducts to line up the product into a designated area on the conveyor. An industrial motion checkweigher can sort products from a fraction of a gram to many, many kilograms. In English units, is this from less than 100th of an ounce to as much as 500 lbs or more. Specialized checkweighers can weigh commercial aircraft, and even find their center-of-gravity. Checkweighers can be very high speed, processing products weighing fractions of a gram at over 100m/m (meters per minute, such as pharmaceuticals, and 200 lb bags of produce at over 100fpm(feet per minute). They can be designed in many shapes and sizes, hung from ceilings, raised on mezzanines, operated in ovens or in refrigerators. Their conveying medium can be industrial belting, low-static belting, chains similar to bicycle chains(but much smaller), or interlocked chain belts of any width. They can have chain belts made of special materials, different polymers, metals, etc. Checkweighers are used in cleanrooms, dry atmosphere environments, wet environments, produce barns, food processing, drug processing, etc. Checkweighers are specified by the kind of environment, and the kind of cleaning will be used. Typically, a checkweigher for produce is made of mild steel, and one that will be cleaned with harsh chemicals, such as bleach, will be made with all stainless steel parts, even the Load cells. These machines are labeled "full washdown", and must have every part and component specified to survive the washdown environment. Checkweighers are operated in some applications for extremely long periods of time- 24/7 year round. Generally, conveyor lines are not stopped unless there is maintenance required, or there is an emergency stop, called an E-stop. Checkweighers operating in high density conveyor lines may have numerous special equipments in their design to ensure that if an E-stop occurs, all power going to all motors is removed until the E-stop is cleared and reset.

References [1] "The Weights and Measures(Packaged Goods)Regulations 2006" (http:/ / www. nmo. bis. gov. uk/ Documents/ PGR guidance 13 august 2007. pdf), NWML, Dept for Innovation, Universities & Skills URN 07/1343, 2006, [2] Checking the Net Contents of Packaged Goods, NIST Handbook 133 (http:/ / ts. nist. gov/ WeightsAndMeasures/ h1334-05. cfm), Fourth Edition, 2005,

Books • Yam, K. L., "Encyclopedia of Packaging Technology", John Wiley & Sons, 2009, ISBN 978-0-470-08704-6

73

Closed-circuit television

74

Closed-circuit television Closed-circuit television (CCTV) is the use of video cameras to transmit a signal to a specific place, on a limited set of monitors. It differs from broadcast television in that the signal is not openly transmitted, though it may employ point to point (P2P), point to multipoint, or mesh wireless links. Though almost all video cameras fit this definition, the term is most often applied to those used for surveillance in areas that may need monitoring such as banks, casinos, airports, military installations, and convenience stores. Videotelephony is seldom called "CCTV" but the use of video in distance education, where it is an important tool, is often so called.[1] [2]

Surveillance cameras.

In industrial plants, CCTV equipment may be used to observe parts of a process from a central control room, for example when the environment is not suitable for humans. CCTV systems may operate continuously or only as required to monitor a particular event. A more advanced form of CCTV, utilizing Digital Video Recorders (DVRs), provides recording for possibly many years, with a variety of quality and performance options and extra features (such as motion-detection and email alerts). More recently, decentralized IP-based CCTV cameras, some equipped with megapixel sensors, support recording directly to network-attached storage devices, or internal flash for completely stand-alone operation. Surveillance of the public using CCTV is particularly common in the U.K., where there are reportedly more cameras per person than in any other country in the world.[3] There and elsewhere, its increasing use has triggered a debate about security versus privacy.

History The first CCTV system was installed by Siemens AG at Test Stand VII in Peenemünde, Germany in 1942, for observing the launch of V-2 [4] rockets. The noted German engineer Walter Bruch was responsible for the design and installation of the system. In the U.S. the first commercial closed-circuit television system became available in 1949, called Vericon. Very little is known about Vericon except it was advertised as not requiring a government permit.[5] CCTV recording systems are still often used at modern launch sites to record the flight of the rockets, in order to find the possible causes of malfunctions,[6] [7] while larger rockets are often fitted with CCTV allowing pictures of stage separation to be transmitted back to earth by radio link.[8]

Sign warning that premises are watched by CCTV cameras.

In September 1968, Olean, New York was the first city in the United States to install video cameras along its main business street in an effort to fight crime. The use of closed-circuit TV cameras piping images into the Olean Police Department propelled Olean to the forefront of crime-fighting technology. The use of CCTV later on became very common in banks and stores to discourage theft, by recording evidence of criminal activity. Their use further popularized the concept. The first place to use CCTV in the United Kingdom was

Closed-circuit television

75

King's Lynn, Norfolk.[9] In recent decades, especially with general crime fears growing in the 1990s and 2000s, public space use of surveillance cameras has taken off, especially in some countries such as the United Kingdom.

Uses Crime prevention and prevalence Experiments in the U.K. during the 1970s and 1980s (including outdoor CCTV in Bournemouth in 1985), led to several larger trial programs later that decade.[9] These were deemed successful in the government report "CCTV: Looking Out For You", issued by the Home Office in 1994, and paved the way for a massive increase in the number of CCTV systems installed. Today, systems cover most town and city centers, and many stations, car-parks and estates. The exact number of CCTV cameras in the U.K. is not known for certain because there is no requirement to register CCTV cameras. However, research published in CCTV Image magazine estimates that the number of cameras in the U.K. is 1.85 million. The number is based on extrapolating from a comprehensive survey of public and private cameras within the Cheshire Constabulary jurisdiction.[10] This works out as an average of one

The two-year-old James Bulger being led away by his killers, recorded on shopping center CCTV.

camera for every 32 people in the U.K., although the density of cameras varies from place to place to such a degree as to make this figure almost meaningless. The Cheshire report also claims that the average person on a typical day would be seen by 70 CCTV cameras, although many of these sightings would be brief glimpses from cameras in shops. The Cheshire figure is regarded as more dependable than a previous study by Michael McCahill and Clive Norris of UrbanEye published in 2002.[11] Based on a small sample in Putney High Street, McCahill and Norris estimated the number of surveillance cameras in private premises in London at around 500,000 and the total number of cameras in the U.K. at around 4,200,000. According to their estimate the U.K. has one camera for every 14 people. Although it has been acknowledged for several years that the methodology behind this figure is somewhat dubious,[12] it has continued to be quoted in the absence of a better figure. The CCTV User Group estimates that there are around 1.5 million CCTV cameras in city centers, stations, airports, major retail areas and so forth. This figure does not include the smaller surveillance systems such as those that may [13] and is therefore broadly in line with the Cheshire report. be found in local corner shops Research conducted by the Scottish Center for Crime and Justice Research and based on a survey of all Scottish local authorities, identified that there are over 2,200 public space CCTV cameras in Scotland.[14] There is little evidence that CCTV deters crime; in fact, there is considerable evidence that it does not.[15] According to a Liberal Democrat analysis, in London "Police are no more likely to catch offenders in areas with hundreds of cameras than in those with hardly any."[16] A 2008 Report by U.K. Police Chiefs concluded that only 3% of crimes were solved by CCTV.[17] In London, a Metropolitan Police report showed that in 2008 only one crime was solved per 1000 cameras.[18] There are valid reasons for including CCTV as a component of a physical security program, but deterrence is not one of them. Cameras have also been installed on public transport in the hope of deterring crime,[19] [20] and in mobile police surveillance vans, often with automatic number plate recognition.[21] In some cases CCTV cameras have become a target of attacks themselves.[22]

Closed-circuit television On July 22, 2005, Jean Charles de Menezes was shot dead by police at Stockwell tube station. According to brother Giovani Menezes, "The film showed that Jean did not have suspicious behavior" .[23] Because of the bombing attempts the previous day, some of the tapes had been supposedly removed from CCTV cameras for study, and they were not functional.[24] An ongoing change to DVR-based technology may in future stop similar problems occurring.[25] In October 2009, an "Internet Eyes" website was announced which would pay members of the public to view CCTV camera images from their homes and report any crimes they witnessed. The site aimed to add "more eyes" to cameras which might be insufficiently monitored, but civil liberties campaigners criticized the idea as "a distasteful and a worrying development".[26]

Hacking and video art Hackers and guerrilla artists have exposed the vulnerabilities of the video systems in an act dubbed "video sniffing"[27] [28] They have crossed feeds, uploaded their own video feeds and used the video footage for artistic purposes.

Industrial processes Industrial processes that take place under conditions dangerous for humans are today often supervised by CCTV. These are mainly processes in the chemical industry, the interior of reactors or facilities for manufacture of nuclear fuel. Special cameras for some of these purposes include line-scan cameras and thermographic cameras which allow operators to measure the temperature of the processes. The usage of CCTV in such processes is sometimes required by law.

Traffic monitoring Many cities and motorway networks have extensive traffic-monitoring systems, using closed-circuit television to detect congestion and notice accidents. Many of these cameras however, are owned by private companies and transmit data to drivers' GPS systems. The U.K. Highways Agency has a publicly-owned CCTV network of over 1,200 cameras covering the English motorway and trunk road network. These cameras are primarily used to monitor traffic conditions and are not used as speed cameras. With the addition of fixed cameras for the Active Traffic Management system, the number of cameras on the Highways Agency's CCTV network is likely to increase significantly over the next few years. The London congestion charge is enforced by cameras positioned at the boundaries of and inside the congestion charge zone, which automatically read the license plates of cars. If the driver does not pay the charge then a fine will be imposed. Similar systems are being developed as a means of locating cars reported stolen. Other surveillance cameras serve as traffic enforcement cameras.

Transport safety A CCTV system may be installed where an operator of a machine cannot directly observe people who may be injured by some unexpected machine operation. For example, on a subway train, CCTV cameras may allow the operator to confirm that people are clear of doors before closing them and starting the train. Digital Video Recorder for Public Transport Operators of an amusement park ride may use a CCTV system to observe that people are not endangered by starting the ride. A CCTV camera and dashboard monitor can make reversing a vehicle safer, if it allows the driver to observe objects or people not otherwise visible.

76

Closed-circuit television

77

Outside the U.K. The use of CCTV in the United States is less common, though increasing, and generally meets stronger opposition. In 1998, 3,000 CCTV systems were in use in New York City.[29] There are more than 10,000 CCTV systems in Chicago.[30] In the last few years particularly, the percentage of people in the U.S. having installed a security-camera system has increased dramatically. Global Security Solutions with the help of Zone Tech Systems first announced the launch of IP surveillance in the U.S. security industry by partnering up with Axis Communications (an IP pioneer). Today's CCTV market has transformed the shift towards IP-based security products and systems. In Latin America, the CCTV market is growing rapidly with the increase of property crime.[31]

Criminal use Criminals may use surveillance cameras, for example a hidden camera at an ATM to capture people's PINs without their knowledge. The devices are small enough not to be noticed, and are placed where they can monitor the keypad of the machine as people enter their PINs. Images may be transmitted wirelessly to the criminal.[32]

Privacy Opponents of CCTV point out the loss of privacy of the people under surveillance, and the negative impact of surveillance on civil liberties. Furthermore, they argue that CCTV displaces crime, rather than reducing it. Critics often dub CCTV as "Big Brother surveillance", a reference to George Orwell's novel Nineteen Eighty-Four, which featured a two-way telescreen in every home through which The Party would monitor the populace. Civil liberties campaign group Big Brother Watch have published several research papers into CCTV systems. In December 2009, they released a report documenting council controlled CCTV cameras.[33]

A surveillance room

More positive views of CCTV cameras have argued that the cameras are not intruding into people's privacy, as they are not surveiling private, but public space, where an individual's right to privacy can reasonably be weighed against the public's need for protection from presumptively innocent people .[34] However, both the United States Supreme Court in Katz vs. The United States and anti-surveillance activists have held that there is a right to privacy in public areas.[35] [36] The recent growth of CCTV in housing areas also raises serious issues about the extent to which CCTV is being used as a social control A mobile closed-circuit TV van monitoring a street market measure rather than simply a deterrent to crime. However, since the September 11 attacks of 2001, many studies have suggested that public opinion of CCTV has grown more favorable. Many proponents of CCTV cite the attacks of the 2005 London Underground bombings as one example of how effective surveillance led to swift progress in post-event investigations. Quite apart from government-permitted use (or abuse), questions are also raised about illegal access to CCTV recordings. The Data Protection Act 1998 in the United Kingdom led to legal restrictions on the uses of CCTV recordings, and also mandated their registration with the Data Protection Agency. In 2004, the successor to the Data Protection Agency, the Information Commissioner's Office clarified that this required registration of all CCTV

Closed-circuit television

78

systems with the Commissioner, and prompt deletion of archived recordings. However, subsequent case law (Durant vs. FSA) has limited the scope of the protection provided by this law, and not all CCTV systems are currently regulated.[37] Private sector personnel in the U.K. who operate or monitor CCTV devices or systems are now considered security guards and have been made subject to state licensing. A 2007 report by the U.K.'s Information Commissioner's Office, highlighted the need for the public to be made more aware of the "creeping encroachment" into their civil liberties created by the growing use of surveillance apparatus. A year prior to the report Richard Thomas, the Information Commissioner, warned that Britain was "sleepwalking into a surveillance society". In 2007, the U.K. watchdog CameraWatch claimed that the majority of CCTV cameras in the U.K. are operated illegally or are in breach of privacy guidelines. In response, the Information Commissioner's Office denied the claim adding that any reported abuses of the Data Protection Act are swiftly investigated.[38] In the United States, there are no such data-protection mechanisms. It has been questioned whether CCTV evidence is allowable under the Fourth Amendment, which prohibits "unreasonable searches and seizures". The courts have generally not taken this view. In Canada, the use of video surveillance has grown very rapidly. In Ontario, both the municipal and provincial versions of the Freedom of Information and Protection of Privacy Act [39] outline very specific guidelines that control how images and information can be gathered by this method and/or released.

Technological developments Computer controlled analytics and identification Today’s High-definition CCTV cameras have many computer controlled technologies that allow them to identify, track, and categorize objects in their field of view. Video Content Analysis (VCA) is the capability of automatically analyzing video to detect and determine temporal events not based on a single image. As such, it can be seen as the automated equivalent of the biological visual cortex.

Surveillance camera at London (Heathrow)

A system using VCA can recognize changes in the environment and Airport with a wiper for clear images during rain even identify and compare objects in the database using size, speed, and sometimes color. The camera’s actions can be programmed based on what it is “seeing”. For example; an alarm can be issued if an object has moved in a certain area, or if a painting is missing from a wall, and if someone has spray painted the lens. VCA analytics can also be used to detect unusual patterns in a videos environment. The system can be set to detect anomalies in a crowd of people, for instance a person moving in the opposite direction in airports where passengers are only supposed to walk in one direction out of a plane or in a subway where people are not supposed to exit through the entrances. [40] VCA also has the ability to track people on a map by calculating their position from the images. It is then possible to link many cameras and track a person through an entire building or area. This can allow a person to be followed without having to analyze many hours of film. Currently the cameras have difficulty identifying individuals from video alone, but if connected to a key-card system, identities can be established and displayed as a tag over their heads on the video. There is also a significant difference in where the VCA technology is placed, either the data is being processed within the cameras (on the edge) or by a centralized server. Both technologies have their pros and cons. [41]

Closed-circuit television

79

Facial recognition system Is a computer application for automatically identifying or verifying a person from a digital image or a video frame from a video source. One of the ways to do this is by comparing selected facial features from the image and a facial database. The combination of CCTV and facial recognition has been tried as a form of mass surveillance, but has been ineffective because of the low discriminating power of facial recognition technology and the very high number of false positives generated. This type of system has been proposed to compare faces at airports and seaports with those of suspected terrorists or other undesirable entrants. Computerized monitoring of CCTV images is under development, so that a human CCTV operator does not have to endlessly look at all the screens, allowing an operator to observe many more CCTV cameras. These systems do not observe people directly. Instead, they track their behavior by looking for particular types of body-movement behavior, or particular types of clothing or baggage. To many, the development of CCTV in public areas, linked to computer databases of people's pictures and identity, presents a serious breach of civil liberties. Critics fear the possibility that one would not be able to meet anonymously in a public place or drive and walk anonymously around a city. Demonstrations or assemblies in public places could be affected as the state would be able to collate lists of those leading them, taking part, or even just talking with protesters in the street.

Retention, storage and preservation

Eye-in-the-sky surveillance dome camera watching from a high steel pole

Most CCTV systems record and store digital video and images to a Digital Video Recorder or in the case of IP cameras directly to a server, either on-site or offsite. The amount of data stored and the retention period of the video or pictures are subject to compression ratios, images stored per second, image size and duration of image retention before being overwritten. [42] Recordings are usually kept for a preset amount of time and then automatically archived, overwritten or deleted. The amount of time the videos are kept allow retrieval and review in the event a crime was committed or the information needs to be studied or any number of reasons.

Closed-circuit digital photography (CCDP) A development in the world of CCTV (October 2005) is in the use of megapixel digital still cameras that can take 1600 x 1200 pixel resolution images of the camera scene either on a time lapse or motion-detection basis. Images taken with a digital still camera have higher resolution than those taken with a typical video camera. Relatively low-cost digital still cameras can be used for CCTV purposes, using CCDP software that controls the camera from the PC. Images of the camera scene are transferred automatically to a computer every few seconds. Images may be monitored remotely if the computer is connected to a network. Combinations of PIR activated floodlights with 1.3Mpix and better digital cameras are now appearing. They save the images to a flash-memory card which is inserted into a slot on the device. The flash card can be removed for viewing on a computer if ever an incident happens. They are not intended for live viewing, but are a very simple and cheap "install and forget" approach to this issue. Closed-circuit digital photography (CCDP) is more suited for capturing and saving recorded photographs, whereas closed-circuit television (CCTV) is more suitable for live-monitoring purposes.

Closed-circuit television

80

IP cameras A growing branch in CCTV is internet protocol cameras (IP cameras). IP cameras use the Internet Protocol (IP) used by most Local Area Networks (LANs) to transmit video across data networks in digital form. IP can optionally be transmitted across the public internet, allowing users to view their camera(s) through any internet connection available through a computer or a 3G phone. For professional or public infrastructure security applications, IP video is restricted to within a private network or VPN.[43]

Networking CCTV cameras The city of Chicago operates a networked video surveillance system which combines CCTV video feeds of government agencies with those of the private sector, installed in city buses, businesses, public schools, subway stations, housing projects etc. Even home owners are able to contribute footage. It is estimated to incorporate the video feeds of a total of 15,000 cameras.

Easy Connect Wireless IP camera

The system is used by Chicago's Office of Emergency Management in case of an emergency call: it detects the caller's location and instantly displays the real-time video feed of the nearest security camera to the operator, not requiring any user intervention. While the system is far too vast to allow complete real-time monitoring, it stores the [44] video data for later usage in order to provide possible evidence in criminal cases. London also has a network of CCTV systems that allows multiple authorities to view and control CCTV cameras in real time. The system allows authorities including the Metropolitan Police Service, Transport for London and a number of London boroughs to share CCTV images between them. It uses a network protocol called Television Network Protocol to allow access to many more cameras than each individual system owner could afford to run and maintain. The Glynn County Police Department uses a wireless mesh-networked system of portable battery-powered tripods for live megapixel video surveillance and central monitoring of tactical police situations. The systems can be used either on a stand-alone basis with secure communications to nearby police laptops, or within a larger mesh system with multiple tripods feeding video back to the command vehicle via wireless, and to police headquarters via 3G.

Integrated systems Integrated systems allow users to connect remotely from the internet and view what their cameras are viewing remotely, similar to that of IP cameras. In one incident, a lady from Boynton Beach, Florida was able to watch her house get robbed and contacted police directly from her office at work.[45]

An integrated systems unit.

Closed-circuit television

81

Wireless security cameras Many consumers are turning to wireless security cameras for home surveillance. Wireless cameras do not require a video cable for video/audio transmission, simply a cable for power. Wireless cameras are also easy and inexpensive to install. Previous generations of wireless security cameras relied on analog technology; modern wireless cameras use digital technology which delivers crisper audio, sharper video, and a secure and interference-free signal.

CCTV camera vandalism

Wireless security camera

Unless physically protected, CCTV cameras have been found to be vulnerable against a variety of (mostly illegal) tactics: • Some people will deliberately destroy cameras. Some cameras can come with Dust-Tight, Pressurized, Explosion proof, and bullet-resistant housings.[46] • Spraying substances over the lens can make the image too blurry to be read. • Lasers can blind or damage them. However, since most lasers are monochromatic, color filters can reduce the effect of laser pointers. However, filters will also impair image quality and overall light sensitivity of cameras (see laser safety article for details on issues with filters). Also, complete protection from infrared, red, green, blue and UV lasers would require use of completely black filters, rendering the camera useless.

References [1] Verman, Romesh. Distance Education In Technological Age (http:/ / books. google. ca/ books?id=1VUU-eRAObAC), Anmol Publications Pvt. Ltd., 2005, pp.166, ISBN 8126122102, ISBN 9788126122103. [2] "Distance education in Asia and the Pacific: Proceedings Of The Regional Seminar On Distance Education, 26 November - 3 December 1986", Asian Development Bank, Bangkok, Thailand, Volume 2, 1987 [3] Lewis, Paul. "Every step you take: UK underground centre that is spy capital of the world" (http:/ / www. guardian. co. uk/ uk/ 2009/ mar/ 02/ westminster-cctv-system-privacy), The Guardian, March 2, 2009. [4] Dornberger, Walter: V-2, Ballantine Books 1954, ASIN: B000P6L1ES, page 14. [5] "Television Rides Wires" , February 1949, Popular Science (http:/ / books. google. com/ books?id=pCQDAAAAMBAJ& pg=PA179& dq=popular+ science+ 1949+ "Some+ time+ ago"& hl=en& ei=gZjhTI-wHZGUnweD79T6Dw& sa=X& oi=book_result& ct=result& resnum=1& ved=0CCoQ6AEwAA#v=onepage& q& f=true) small article, bottom of page 179 [6] "ET_SRB Cam FS.indd" (http:/ / www. nasa. gov/ centers/ marshall/ pdf/ 114016main_ET_SRB_Cam_FS. pdf) (PDF). . Retrieved 2009-07-22. [7] "Ecliptic Enterprises Corporation" (http:/ / web. archive. org/ web/ 20080705073920/ http:/ / www. eclipticenterprises. com/ products_rocketcam_avs. php). Eclipticenterprises.com. Archived from the original (http:/ / www. eclipticenterprises. com/ products_rocketcam_avs. php) on July 5, 2008. . Retrieved 2009-05-08. [8] Brent D. Johnson. "Cameras Monitor Rocket Launch" (http:/ / www. photonics. com/ content/ spectra/ 2003/ January/ applications/ 65734. aspx). Photonics.com. . Retrieved 2009-05-08. [9] Staff (August 2007). "CCTV" (http:/ / www. west-norfolk. gov. uk/ default. aspx?page=21697). Borough Council of King's Lynn & West Norfolk. . Retrieved 2008-12-14. [10] "Only 1.85 million cameras in UK, claims ACPO lead on CCTV - SecurityNewsDesk.com" (http:/ / www. securitynewsdesk. com/ 2011/ 03/ 01/ only-1-8-million-cameras-in-uk-claims-acpo-lead-on-cctv). SecurityNewsDesk.com. . Retrieved 2011-03-02. [11] "CCTV in London" (http:/ / www. urbaneye. net/ results/ ue_wp6. pdf) (PDF). . Retrieved 2009-07-22. [12] "FactCheck: how many CCTV cameras? - Channel 4 News" (http:/ / www. channel4. com/ news/ articles/ society/ factcheck+ how+ many+ cctv+ cameras/ 2291167). Channel4.com. . Retrieved 2009-05-08. [13] "How many cameras are there?" (http:/ / www. cctvusergroup. com/ art. php?art=94). CCTV User Group. 2008-06-18. . Retrieved 2009-05-08. [14] Bannister, J., Mackenzie, S. and Norris, P. Public Space CCTV in Scotland (http:/ / www. sccjr. ac. uk/ pubs/ Public-Space-CCTV-in-Scotland--Results-of-a-National-Survey-of-Scotlands-Local-Authorities/ 182)(2009), Scottish Centre for Crime and Justice Research (Research Report)

Closed-circuit television [15] Baram, Marcus (2007-07-09). "Eye on the City: Do Cameras Reduce Crime?" (http:/ / www. abcnews. go. com/ US/ Story?id=3360287& page=1). ABC News. . Retrieved 2007-07-10. [16] "Tens of thousands of CCTV cameras, yet 80% of crime unsolved" (http:/ / www. thisislondon. co. uk/ news/ article-23412867-details/ Tens+ of+ thousands+ of+ CCTV+ cameras,+ yet+ 80%+ of+ crime+ unsolved/ article. do) by Justin Davenport 2007 [17] "Are CCTV cameras a waste of money in the fight against crime?" (http:/ / www. independent. co. uk/ news/ uk/ crime/ the-big-question-are-cctv-cameras-a-waste-of-money-in-the-fight-against-crime-822079. html) The Independent, 7 May 2008 [18] Hughe, Mark (25 August 2009). "CCTV in the spotlight: one crime solved for every 1,000 cameras" (http:/ / www. independent. co. uk/ news/ uk/ crime/ cctv-in-the-spotlight-one-crime-solved-for-every-1000-cameras-1776774. html). Independent News and Media Limited. . Retrieved 2009-08-27. [19] " CCTV to drive down cab attacks (http:/ / news. bbc. co. uk/ 1/ hi/ england/ hereford/ worcs/ 3101016. stm)," BBC [20] Taxi CCTV cameras are installed (http:/ / news. bbc. co. uk/ 1/ hi/ england/ bristol/ 4295859. stm)," BBC [21] CCTV patrols to monitor estates (http:/ / news. bbc. co. uk/ 1/ hi/ england/ tees/ 3529305. stm)," BBC [22] " http:/ / news. bbc. co. uk/ (http:/ / news. bbc. co. uk/ 1/ hi/ wales/ south_east/ 3676550. stm)," BBC [23] " Menezes family view CCTV footage (http:/ / news. bbc. co. uk/ 1/ hi/ uk/ 4293462. stm)," BBC [24] " Menezes Death 'Cover-Up' Doubted (http:/ / news. bbc. co. uk/ 1/ hi/ uk/ 4175688. stm)," BBC [25] " Digital CCTV Scheme Switches On (http:/ / news. bbc. co. uk/ 1/ hi/ england/ 2070312. stm)," BBC [26] Public to Monitor CCTV From Home (http:/ / news. bbc. co. uk/ 1/ hi/ england/ london/ 8293784. stm), BBC [27] Christopher Werth To Watch the Watchers (http:/ / www. newsweek. com/ id/ 163113/ output/ print) October 20, 2008 Newsweek [28] Newsweek [29] " You're Being Watched, New York! (http:/ / news. bbc. co. uk/ 1/ hi/ world/ americas/ 1865828. stm)," 11 March 2002 BBC [30] "Chicago Links Police, Private Cameras" (http:/ / abclocal. go. com/ wls/ story?section=news/ local& id=7370352). WLS-TV. 2010. . Retrieved 2010-08-16. [31] " Latin American Physical Security Market Growing Rapidly (http:/ / www. securitymagazine. com/ Articles/ SEC_Newswire/ BNP_GUID_9-5-2006_A_10000000000000678960)," 8 October 2009 Security Magazine [32] "ATM Security" (http:/ / www. dedhamsavings. com/ index. php?option=com_content& task=view& id=83). Dedham Savings. . Retrieved 2009-04-18. [33] "Councils 'treble CCTV in decade'" (http:/ / news. bbc. co. uk/ 1/ hi/ uk/ 8419358. stm). BBC News. 2009-12-18. . [34] Smile, the cameras are here to watch over you - The New Zealand Herald, Tuesday 18 March 2008, Page A14 [36] . http:/ / www. annarbor. com/ community/ news/ opinion/ city_council_should_pass_the_freedom_from_surveillance_ordinance. [37] "Information Commissioner's Office" (http:/ / www. informationcommissioner. gov. uk/ eventual. aspx?id=5739). Informationcommissioner.gov.uk. . Retrieved 2009-05-08. [38] Majority of UK's CCTV cameras 'are illegal' (http:/ / www. telegraph. co. uk/ news/ main. jhtml?xml=/ news/ 2007/ 05/ 31/ ncamera131. xml) Telegraph.co.uk [39] Freedom of Information and Protection of Privacy Act (http:/ / www. ipc. on. ca/ index. asp?navid=73) Text [40] "MATE's Analytics Integrate with Hirsch Security Systems" (http:/ / www. mate. co. il/ page. asp?newsid=53& type=6& cat=28& lang=1) (HTML). . Retrieved 2011-03-28. [41] "Image Processing Techniques for Video Content Extraction" (http:/ / www. ercim. eu/ publication/ ws-proceedings/ DELOS4/ oliveira. pdf) (PDF). . Retrieved 2011-03-28. [42] "H.264 compression versus MPEG4 compression for cctv video storage" (http:/ / www. allbestarticles. com/ technology/ video/ h. 264-compression-versus-mpeg4-compression-for-cctv-video-storage. html). . Retrieved 2011-03-28. [43] "Some IP Cameras Can Be Remotely Monitored With An iPhone And Other Compatible 3G Devices" (http:/ / www. lorexcorp. com/ uploads/ LNE3003 - Release. pdf) (PDF). . Retrieved 2009-07-22. [44] " Chicago's Camera Network Is Everywhere (http:/ / online. wsj. com/ article/ SB10001424052748704538404574539910412824756. html)", The Wall Street Journal [45] By Kim Segal CNN (2009-04-10). "How IP Cameras can help protect your home. Real CNN report" (http:/ / www. cnn. com/ 2009/ CRIME/ 04/ 10/ webcam. home. invasion/ index. html#cnnSTCVideo). Cnn.com. . Retrieved 2009-05-08. [46] "Pelco Specialized Camera Housings" (http:/ / www. pelco. com/ sites/ global/ en/ products/ camera-solutions/ function-presentation. page?p_function_id=9506). . Retrieved 2008-06-02.

82

Closed-circuit television

83

External links • Space Shuttle External Tank and Solid Rocket Booster Camera Systems (http://www.nasa.gov/centers/ marshall/pdf/114016main_ET_SRB_Cam_FS.pdf) • UK Government pro-CCTV campaign (http://www.crimereduction.gov.uk/cctvminisite4.htm) • Assessing the Impact of CCTV, a UK Home office study on the effectiveness of closed-circuit television (http:// www.homeoffice.gov.uk/rds/pdfs05/hors292.pdf) • The Register story: Face recognition useless for crowd surveillance (http://www.theregister.co.uk/2001/09/ 27/face_recognition_useless_for_crowd/) • CCTV Guidance notes (http://www.ico.gov.uk/Home/for_organisations/topic_specific_guides/cctv.aspx) from the UK Information Commissioner's Office • CBC Digital Archives - The Long Lens of the Law (http://archives.cbc.ca/IDD-1-75-1299/ science_technology/police_cameras_privacy/) • The Urbaneye Project on CCTV in Europe (http://www.urbaneye.net/) • CCTV:Constant Cameras Track Violators (http://www.ncjrs.gov/pdffiles1/jr000249d.pdf) National Institute of Justice Journal 249 (2003). Washington, DC: U.S. Department of Justice. • Public Space CCTV in Scotland: Results of a National Survey of Scotland's Local Authorities (http://www. sccjr.ac.uk/pubs/ Public-Space-CCTV-in-Scotland--Results-of-a-National-Survey-of-Scotlands-Local-Authorities/182) • Opinion on Video Surveillance in Public Places by Public Authorities and the Protection of Human Rights (http:/ /www.venice.coe.int/docs/2007/CDL-AD(2007)014-e.asp) and Opinion on Video Surveillance by Private Operators in the Public and Private Spheres and by Public Authorities in the Private Sphere and the Protection of Human Rights (http://www.venice.coe.int/docs/2007/CDL-AD(2007)027-e.asp), Venice Commission, 2007

Computer stereo vision Computer stereo vision is the extraction of 3D information from digital images, such as obtained by a CCD camera. By comparing information about a scene from several camera perspectives in the scene, limited 3D information can be extracted by examination of the relative perspectives. This is similar to the biological process Stereopsis.

Outline In traditional stereo vision, two cameras, displaced horizontally from one another are used to obtain differing views on a scene, in a manner similar to human binocular vision. The cameras are then modelled as a perspective view whereby the two cameras will see slightly different projections of the world view. By comparing these two images, the relative depth information can be obtained, in the form of a disparity map, which is inversely proportional to the distance to the object. To compare the images, the two views must be transformed as if there were being observed from a common projective camera, this can be achieved, for example, by projecting the right camera to the left camera's position or vice versa, and the relative shifts between the two images can then be seen to be due to parallax. Alternately both camera views may be transformed to an arbitrary location in 3D space, as long as the front face of the images to be compared is visible from this location, and that occlusion or transparency does not interfere with the calculation. [1]

In a real camera system however, there are several steps required to make this calculation.

1. The image must first be removed of distortions, such as barrel distortion to ensure that the observed image is purely projectional. 2. The image must be projected back to a common plane to allow comparison of the image pairs, known as image rectification.

Computer stereo vision 3. The displacement of relative features is measured to calculate a disparity map 4. Optionally, the disparity as observed by the common projection, is converted back to the height map by inversion. Utilising the correct proportionality constant, the height map can be calibrated to provide exact distances.

Applications Stereo vision finds many applications in automated systems. Stereo vision is highly important in fields such as robotics, to extract information about the relative position of 3D objects in the vicinity of autonomous systems. Other applications for robotics include object recognition, where depth information allows for the system to separate occluding image components, such as one chair in front of another, which the robot may otherwise not be able to distinguish as a separate object by any other criteria. Scientific applications for digital stereo vision include the extraction of information from aerial surveys, for calculation of contour maps or even geometry extraction for 3D building mapping, or calculation of 3D heliographical information such as obtained by the NASA STEREO project.

References [1] Bradski, Gary and Kaehler, Adrian. Learning OpenCV: Computer Vision with the OpenCV Library. O'Reilly.

External links • Tutorial on uncalibrated stereo vision (http://pages.cs.wisc.edu/~chaol/cs766/)

Content-based image retrieval Content-based image retrieval (CBIR), also known as query by image content (QBIC) and content-based visual information retrieval (CBVIR) is the application of computer vision techniques to the image retrieval problem, that is, the problem of searching for digital images in large databases. (see this survey[1] for a recent scientific overview of the CBIR field) "Content-based" means that the search will analyze the actual contents of the image rather than the metadata such as keywords, tags, and/or descriptions associated with the image. The term 'content' in this context might refer to colors, shapes, textures, or any other information that can be derived from the image itself. CBIR is desirable because most web based image search engines rely purely on metadata and this produces a lot of garbage in the results. Also having humans manually enter keywords for images in a large database can be inefficient, expensive and may not capture every keyword that describes the image. Thus a system that can filter images based on their content would provide better indexing and return more accurate results.

History The term Content-Based Image Retrieval (CBIR) seems to have originated in 1992, when it was used by T. Kato to describe experiments into automatic retrieval of images from a database, based on the colors and shapes present.[2] Since then, the term has been used to describe the process of retrieving desired images from a large collection on the basis of syntactical image features. The techniques, tools and algorithms that are used originate from fields such as statistics, pattern recognition, signal processing, and computer vision.

84

Content-based image retrieval

Technical progress There is a growing interest in CBIR because of the limitations inherent in metadata-based systems, as well as the large range of possible uses for efficient image retrieval. Textual information about images can be easily searched using existing technology, but requires humans to personally describe every image in the database. This is impractical for very large databases, or for images that are generated automatically, e.g. from surveillance cameras. It is also possible to miss images that use different synonyms in their descriptions. Systems based on categorizing images in semantic classes like "cat" as a subclass of "animal" avoid this problem but still face the same scaling issues. Potential uses for CBIR include: • • • • • • • • •

Art collections Photograph archives Retail catalogs Medical diagnosis Crime prevention The military Intellectual property Architectural and engineering design Geographical information and remote sensing systems

CBIR software systems • • • • • • •

University of Washington FIDS Demo[3] CIRES: Content Based Image Retrieval System[4] LTU-Corbis Visual Search[5] TinEye[6] Cortina [7] Octagon[8] Windsurf[9]

See CBIR engines for other examples of publicly available and accessible CBIR systems.

CBIR techniques Many CBIR systems have been developed, but the problem of retrieving images on the basis of their pixel content remains largely unsolved.

Query techniques Different implementations of CBIR make use of different types of user queries. Query by example Query by example is a query technique that involves providing the CBIR system with an example image that it will then base its search upon. The underlying search algorithms may vary depending on the application, but result images should all share common elements with the provided example. Options for providing example images to the system include: • A preexisting image may be supplied by the user or chosen from a random set. • The user draws a rough approximation of the image they are looking for, for example with blobs of color or general shapes.[10]

85

Content-based image retrieval This query technique removes the difficulties that can arise when trying to describe images with words. Semantic retrieval The ideal CBIR system from a user perspective would involve what is referred to as semantic retrieval, where the user makes a request like "find pictures of dogs" or even "find pictures of Abraham Lincoln". This type of open-ended task is very difficult for computers to perform - pictures of chihuahuas and Great Danes look very different, and Lincoln may not always be facing the camera or in the same pose. Current CBIR systems therefore generally make use of lower-level features like texture, color, and shape, although some systems take advantage of very common higher-level features like faces (see facial recognition system). Not every CBIR system is generic. Some systems are designed for a specific domain, e.g. shape matching can be used for finding parts inside a CAD-CAM database. Other query methods Other query methods include browsing for example images, navigating customized/hierarchical categories, querying by image region (rather than the entire image), querying by multiple example images, querying by visual sketch, querying by direct specification of image features, and multimodal queries (e.g. combining touch, voice, etc.) [11]. CBIR systems can also make use of relevance feedback, where the user progressively refines the search results by marking images in the results as "relevant", "not relevant", or "neutral" to the search query, then repeating the search with the new information.

Content comparison using image distance measures The most common method for comparing two images in content based image retrieval (typically an example image and an image from the database) is using an image distance measure. An image distance measure compares the similarity of two images in various dimensions such as color, texture, shape, and others. For example a distance of 0 signifies an exact match with the query, with respect to the dimensions that were considered. As one may intuitively gather, a value greater than 0 indicates various degrees of similarities between the images. Search results then can be sorted based on their distance to the queried image.[10] Color Computing distance measures based on color similarity is achieved by computing a color histogram for each image that identifies the proportion of pixels within an image holding specific values (that humans express as colors). Current research is attempting to segment color proportion by region and by spatial relationship among several color regions. Examining images based on the colors they contain is one of the most widely used techniques because it does not depend on image size or orientation. Color searches will usually involve comparing color histograms, though this is not the only technique in practice. Texture Texture measures look for visual patterns in images and how they are spatially defined. Textures are represented by texels which are then placed into a number of sets, depending on how many textures are detected in the image. These sets not only define the texture, but also where in the image the texture is located. Texture is a difficult concept to represent. The identification of specific textures in an image is achieved primarily by modeling texture as a two-dimensional gray level variation. The relative brightness of pairs of pixels is computed such that degree of contrast, regularity, coarseness and directionality may be estimated (Tamura, Mori & Yamawaki, 1978). However, the problem is in identifying patterns of co-pixel variation and associating them with particular classes of textures such as silky, or rough.

86

Content-based image retrieval

87

Shape Shape does not refer to the shape of an image but to the shape of a particular region that is being sought out. Shapes will often be determined first applying segmentation or edge detection to an image. Other methods like [Tushabe and Wilkinson 2008] use shape filters to identify given shapes of an image. In some case accurate shape detection will require human intervention because methods like segmentation are very difficult to completely automate.

Applications Some software producers are trying to push CBIR based applications into the filtering and law enforcement markets for the purpose of identifying and censoring images with skin-tones and shapes that could indicate the presence of nudity, with controversial results.

Relevant research papers • Query by Image and Video Content: The QBIC System [12], (Flickner, 1995) • Finding Naked People [13] (Fleck et al., 1996) • Virage Video Engine [14], (Hampapur, 1997) • Library-based Coding: a Representation for Efficient Video Compression and Retrieval [15], (Vasconcelos & Lippman, 1997) • System for Screening Objectionable Images [16] (Wang et al., 1998) • Content-based Image Retrieval [17] (JISC Technology Applications Programme Report 39) (Eakins & Graham 1999) • Windsurf: Region-Based Image Retrieval Using Wavelets [18] (Ardizzoni, Bartolini, and Patella, 1999) • A Probabilistic Architecture for Content-based Image Retrieval [19], (Vasconcelos & Lippman, 2000) • A Unifying View of Image Similarity [20], (Vasconcelos & Lippman, 2000) • Next Generation Web Searches for Visual Content [21], (Lew, 2000) • Image Indexing with Mixture Hierarchies [22], (Vasconcelos, 2001) • SIMPLIcity: Semantics-Sensitive Integrated Matching for Picture Libraries [23] (Wang, Li, and Wiederhold, 2001) • FACERET: An Interactive Face Retrieval System Based on Self-Organizing Maps [24] (Ruiz-del-Solar et al., 2002) • Automatic Linguistic Indexing of Pictures by a Statistical Modeling Approach [8] (Li and Wang, 2003) • Video google: A text retrieval approach to object matching in videos [25] (Sivic & Zisserman, 2003) • Minimum Probability of Error Image Retrieval

[26]

(Vasconcelos, 2004)

• On the Efficient Evaluation of Probabilistic Similarity Functions for Image Retrieval [27] (Vasconcelos, 2004) • Extending image retrieval systems with a thesaurus for shapes [28] (Hove, 2004) • Names and Faces in the News [29] (Berg et al., 2004) • Cortina: a system for large-scale, content-based web image retrieval [30] (Quack et al., 2004) • A new perspective on Visual Information Retrieval [31] (Eidenberger 2004) • Language-based Querying of Image Collections on the basis of an Extensible Ontology [32] (Town and Sinclair, 2004) • Costume: A New Feature for Automatic Video Content Indexing [33] (Jaffre 2005)

Content-based image retrieval • Automatic Face Recognition for Film Character Retrieval in Feature-Length Films [34] (Arandjelovic & Zisserman, 2005) • Content-based Multimedia Information Retrieval: State of the Art and Challenges [35] (Lew et al 2006) • Algorithm on which Retrievr (Flickr search) and imgSeek is based on [36] (Jacobs, Finkelstein, Salesin) • Evaluating Use of Interfaces for Visual Query Specification. [37] (Hove, 2007) • From Pixels to Semantic Spaces: Advances in Content-Based Image Retrieval [38] (Vasconcelos, 2007) • Content-based Image Retrieval by Indexing Random Subwindows with Randomized Trees [39] (Maree et al., 2007) • Image Retrieval: Ideas, Influences, and Trends of the New Age [40] (Datta et al., 2008) • Real-Time Computerized Annotation of Pictures [7] (Li and Wang, 2008) • Query Processing Issues in Region-based Image Databases [41] (Bartolini, Ciaccia, and Patella, 2010)

References [1] Content-based Multimedia Information Retrieval: State of the Art and Challenges (http:/ / www. liacs. nl/ home/ mlew/ mir. survey16b. pdf), Michael Lew, et al., ACM Transactions on Multimedia Computing, Communications, and Applications, pp. 1-19, 2006. [2] Content-based Image Retrieval, John Eakins and Margaret Graham, University of Northumbria at Newcastle (http:/ / www. jisc. ac. uk/ uploaded_documents/ jtap-039. doc) [3] University of Washington FIDS Demo (http:/ / www. cs. washington. edu/ research/ imagedatabase/ demo/ fids/ ) [4] CIRES: Content Based Image Retrieval System (http:/ / amazon. ece. utexas. edu/ ~qasim/ research. htm) [5] LTU Technologies Corbis Visual Search (http:/ / corbis. demo. ltutech. com/ en/ demos/ corbis/ ) [6] Idée Inc. TinEye Reverse Image Search Engine. (http:/ / www. tineye. com/ ) [7] Vision Research Lab, UCSB (http:/ / vision. ece. ucsb. edu/ multimedia/ cortina. shtml) [8] Octagon by Viitala (http:/ / octagon. viitala. eu/ ) [9] Windsurf (University of Bologna, Italy) (http:/ / www-db. deis. unibo. it/ Windsurf/ ) [10] Shapiro, Linda; George Stockman (2001). Computer Vision. Upper Saddle River, NJ: Prentice Hall. ISBN 0-13-030796-3. [11] http:/ / mayron. net/ liam/ pub/ mayron_dissertation. pdf [12] http:/ / doi. ieeecomputersociety. org/ 10. 1109/ 2. 410146 [13] http:/ / www. cs. hmc. edu/ ~fleck/ naked-people. ps. gz [14] http:/ / spiedl. aip. org/ getabs/ servlet/ GetabsServlet?prog=normal& id=PSISDG003022000001000188000001& idtype=cvips& gifs=yes [15] http:/ / www. svcl. ucsd. edu/ publications/ conference/ 1997/ dcc97/ dcc97. pdf [16] http:/ / www-db. stanford. edu/ ~wangz/ project/ imscreen/ JCC98/ [17] http:/ / www. jisc. ac. uk/ uploaded_documents/ jtap-039. doc [18] http:/ / www-db. deis. unibo. it/ research/ papers/ IWOSS99. ABP. pdf [19] http:/ / www. svcl. ucsd. edu/ publications/ conference/ 2000/ cvpr00/ cvpr00. pdf [20] http:/ / www. svcl. ucsd. edu/ publications/ conference/ 2000/ icpr00/ icpr00. pdf [21] http:/ / www. liacs. nl/ home/ mlew/ comp2000. pdf [22] http:/ / www. svcl. ucsd. edu/ publications/ conference/ 2001/ cvpr01/ cvpr01. pdf [23] http:/ / www-db. stanford. edu/ ~wangz/ project/ imsearch/ SIMPLIcity/ TPAMI/ [24] http:/ / www. springerlink. com/ index/ 10FL41074LE5699P. pdf [25] http:/ / ieeexplore. ieee. org/ iel5/ 8769/ 27772/ 01238663. pdf [26] http:/ / www. svcl. ucsd. edu/ publications/ journal/ 2004/ sp04/ sp04. pdf [27] http:/ / www. svcl. ucsd. edu/ publications/ journal/ 2004/ it04/ it04. pdf [28] http:/ / www. nik. no/ 2004/ bidrag/ Hove. pdf [29] http:/ / www1. cs. columbia. edu/ CAVE/ NSF-ITR/ research/ publications/ Berkeley/ miller_cvpr_04. pdf [30] http:/ / portal. acm. org/ citation. cfm?id=1027650 [31] http:/ / www. ims. tuwien. ac. at/ ~hme/ papers/ ei2004-vir. pdf [32] http:/ / www. cl. cam. ac. uk/ ~cpt23/ papers/ TownIVC2004. pdf [33] http:/ / www. irit. fr/ ~Gael. Jaffre/ RECHERCHE/ thesis. html [34] http:/ / mi. eng. cam. ac. uk/ ~oa214/ academic/ publications/ 2005_CVPR_paper2. pdf [35] http:/ / www. liacs. nl/ ~mlew/ mir. survey16b. pdf [36] http:/ / grail. cs. washington. edu/ projects/ query/ [37] http:/ / caim. uib. no/ publications/ NOKOBIT07_ljh. pdf [38] http:/ / www. svcl. ucsd. edu/ publications/ journal/ 2007/ computer/ computer07. pdf [39] http:/ / www. montefiore. ulg. ac. be/ services/ stochastic/ pubs/ 2007/ MGW07b/ [40] http:/ / infolab. stanford. edu/ ~wangz/ project/ imsearch/ review/ JOUR/ datta. pdf

88

Content-based image retrieval [41] http:/ / www. springerlink. com/ content/ c212k8474x671648/

Bibliography • Bird, C.L.; P.J. Elliott, Griffiths (1996). User interfaces for content-based image retrieval. • Rui, Yong; Thomas S. Huang, Shih-Fu Chang (1999). Image Retrieval: Current Techniques, Promising Directions, and Open Issues. • Datta, Ritendra; Dhiraj Joshi, Jia Li, James Z. Wang (2008). "Image Retrieval: Ideas, Influences, and Trends of the New Age" (http://infolab.stanford.edu/~wangz/project/imsearch/review/JOUR/). ACM Computing Surveys 40 (2): 1–60. doi:10.1145/1348246.1348248. • Tushabe, F.; M.H.F. Wilkinson (2008). "Content-based Image Retrieval Using Combined 2D Attribute Pattern Spectra". Springer Lecture Notes in Computer Science.

External links • cbir.info (http://cbir.info/articles/) CBIR-related articles • Search by Drawing (http://www.sepham.com/)

Digital video fingerprinting Video fingerprinting is a technique in which software identifies, extracts and then compresses characteristic components of a video, enabling that video to be uniquely identified by its resultant “fingerprint”. Video fingerprinting is technology that has proven itself to be effective at identifying and comparing digital video data. Video fingerprinting analysis may be based on any number of visual video features including, but not limited to, key frame analysis, color and motion changes during a video sequence.

Principles behind video fingerprinting technology Video fingerprinting methods extract several unique features of a digital video that can be stored as a fingerprint of the video content. The evaluation and identification of video content is then performed by comparing the extracted video fingerprints. For digital video data, both audio and video fingerprints can be extracted, each having individual significance for different application areas. The creation of a video fingerprint involves the use of specialized software that decodes the video data and then applies several feature extraction algorithms. Video fingerprints are highly compressed when compared to the original source file and can therefore be easily stored in databases for later comparison. They may be seen as an extreme form of lossy compression and cannot be used to reconstruct the original video content. The huge number of videos currently available (thanks to the development of user generated content sites (UGC sites)) presents video fingerprinting technologies with a scalability challenge.

Compared to hash codes Normally, digital data are compared based on hash values that are directly derived from the digital components of a file. However, such methods are incomplete as they can only determine absolute equality or non-equality of video data files or parts. More often than not, differences in a video codec and digital processing artifacts may cause small differences in the digital components without changing the video perceptually. Thus, when employing hash methods, a comparison for absolute equality may fail even when two video segments are perceptually identical. Moreover, hash value comparisons are also of little value when one wishes to identify video segments that are similar (but not identical) to a given reference clip. The limitations of the equality / inequality dichotomy inherent to hash value

89

Digital video fingerprinting techniques render “similar searching” impossible. Also, digital video fingerprinting enables to recognize videos with a different resolution compared with the original (smaller or larger) as well as recognize videos that have been modified slightly (blurring, rotation, acceleration or decceleration, cropping, insertions of new elements in the video), and videos where the audio track has been modified.

Compared to watermarking Video fingerprinting should not be confused with digital watermarking which relies on inserting identifying features into the content itself, and therefore changing the nature of the content. Some watermarks can be inserted in a way that they remain imperceptible by a viewer. A robust watermark can be difficult to detect and remove, but this is a significant weakness in watermarks. Since watermarks must be inserted into the video, they only identify copies of the particular video made after that point in time. For example, if a watermark is inserted at broadcast it cannot be used to identify copies of the video made before the broadcast. Video fingerprinting does not rely on any addition to the video stream. A video fingerprint cannot be "removed" because it is not "added". In addition, a reference video fingerprint can be created at any point from any copy of the video. Watermarks offer some advantages over fingerprinting. A unique watermark can be added to the content at any stage in the distribution process and multiple independent watermarks can be inserted into the same video content. This can be particularly useful in tracing the history of a copy of a video. Detecting watermarks in a video can indicate the source of an unauthorized copy. While video fingerprinting systems must search a potentially large database of reference fingerprints, a watermark detection system only has to do the computation to detect the watermark. This computation can be significant and when multiple watermark keys must be tested then watermarking can fail to scale to UGV site volumes.

Video fingerprinting applications Video Fingerprinting is of interest in the Digital Rights Management (DRM) area, particularly regarding the distribution of unauthorized content on the Internet. Video Fingerprinting systems enable content providers (e.g. film studios) or publishers (e.g. UGC sites) to determine if any of the publisher's files contain content registered with the fingerprint service. If registered content is detected, the publisher can take the appropriate action - remove it from the site, monetize it, add correct attribution, etc. Video fingerprinting may be used for broadcast monitoring (e.g. advertisement monitoring, News monitoring) and general Media monitoring. Broadcast monitoring solutions can inform content providers and content owners with play lists of when and where their video content was used. Video fingerprinting is also used by authorities to track the distribution of illegal content such as happy slapping, terrorist and child abuse related videos. Another use is for companies to track the leak of confidential recordings or videos, or for celebrities to track the presence on the Internet of unauthorized videos (for instance videos of themselves taken by amateurs using a camcorder or a mobile phone). Fingerprinting visual content is similar to audio fingerprinting but uses a different technology. From a content provider's point of view, both video and audio fingerprinting need to be used in most applications. Consider the online publication of "mash-ups". Mash-ups can consist of content from several sources that is compiled together and is set to a unique audio track. Since the audio track is different from the original version, the copyrighted material in these mash-ups would go undetected using only audio fingerprinting techniques. In other cases, mash-ups consist of the soundtrack from a commercial video source, like a movie, used with a different video stream. In this case a video fingerprint would not match but an audio fingerprint would. When the audio and video streams are not

90

Digital video fingerprinting from the same master work, the question of fair-use may arise. This discrepancy has real applications in the global online community in terms of film distribution. Films shown in countries other than their country of origin are often dubbed into other languages. This change in audio renders the films virtually unrecognizable by audio fingerprinting technologies unless a copy of all known versions has been previously fingerprinted. Employing video fingerprinting, however, enables the content owner to fingerprint just once and have each subsequent version remain recognizable. Of course, if the customer wishes to know which language soundtrack is present on a particular video, then an audio fingerprint must be used.

External links • Video fingerprinting • • • • • • • • • • • •

Advestigo (audio, video and image fingerprinting) [1] Attributor (image and video fingerprinting) [2] Audible Magic (audio & video image fingerprinting) [3] Auditude Connect technology (audio and video fingerprinting) [4] INA (video fingerprinting) [5] Miranda (audio, video and image fingerprinting) [6] Civolution (audio and video fingerprinting) [7]

Vercury (audio, image and video fingerprinting) [8] Vidyatel Video conform, TV tracking and Management (frame accurate video fingerprinting) [9] Vobile Content Identification and Management (audio and video fingerprinting) [10] YUVsoft (video fingerprinting and search) [11] Zeitera (video fingerprinting) [12]

References [1] http:/ / www. advestigo. com [2] http:/ / www. attributor. com/ how_it_works/ overview. php [3] http:/ / www. audiblemagic. com [4] http:/ / www. auditude. com [5] http:/ / www. ina. fr/ to-know-ina/ signature. html [6] http:/ / www. miranda. com [7] http:/ / www. civolution. com [8] http:/ / www. vercury. com [9] http:/ / www. vidyatel. com [10] http:/ / www. vobileinc. com [11] http:/ / yuvsoft. com/ technologies/ video_matching/ [12] http:/ / www. zeitera. com

91

GazoPa

92

GazoPa gazopa URL

http:/ / www. gazopa. com/

Type of site

Image Search Engine

Available language(s) English Launched

September10, 2008

Current status

open beta

[1]

GazoPa is an image search engine that uses features from an image to search for and identify similar images. [2] GazoPa is listed in A new Top 100 Alternative Search Engines list. [3] GazoPa for iPhone is selected as a winner of Best Infotainment & Community at Mobile Content 2009. [4] GazoPa released a flower photo community site, GazoPa Bloom in private beta. This site is for exploring flower images and, if users need help identifying a flower, uploading images and letting other people try to identify them for users. [5]

References [1] Toto, Serkan (2009-10-27). "Similar Image Search Engine Gazopa Enters Open Beta" (http:/ / www. techcrunch. com/ 2009/ 10/ 27/ similar-image-search-engine-gazopa-enters-open-beta/ ). TechCrunch. . Retrieved 2010-01-17. [2] Toto, Serkan (2008-09-10). "TC50: Gazopa Searches for Images Similar To Other Images" (http:/ / www. techcrunch. com/ 2008/ 09/ 10/ tc50-gazopa-lets-users-search-for-images-without-typing-keywords/ ). TechCrunch. . Retrieved 2009-09-16. [3] Simpson, M.E. (2009-06-19). "A new Top 100 Alternative Search Engines list!" (http:/ / www. altsearchengines. com/ 2009/ 06/ 19/ a-new-top-100-alternative-search-engines-list/ ). Altsearchengines. . Retrieved 2009-09-23. [4] "Mobile Content 2009" (http:/ / www. mobilecontent. co. kr/ 03_awards/ awards08. htm). 2009-09-10. . Retrieved 2009-09-23. [5] "GazoPa Gets In Bloom" (http:/ / www. researchbuzz. org/ wp/ gazopa-gets-in-bloom/ ). ResearchBuzz. 2010-03-24. . Retrieved 2010-04-03.

External links • gazopa.com (http://www.gazopa.com) • gazopa bloom (http://bloom.gazopa.com)

Gesture recognition

93

Gesture recognition Gesture recognition is a topic in computer science and language technology with the goal of interpreting human gestures via mathematical algorithms. Gestures can originate from any bodily motion or state but commonly originate from the face or hand. Current focuses in the field include emotion recognition from the face and hand gesture recognition. Many approaches have been made using cameras and computer vision algorithms to interpret sign language. However, the identification and recognition of posture, gait, proxemics, and human behaviors is also the subject of gesture recognition techniques.[1]

A child being sensed by a simple gesture recognition algorithm detecting hand location and movement

Gesture recognition can be seen as a way for computers to begin to understand human body language, thus building a richer bridge between machines and humans than primitive text user interfaces or even GUIs (graphical user interfaces), which still limit the majority of input to keyboard and mouse. Gesture recognition enables humans to interface with the machine (HMI) and interact naturally without any mechanical devices. Using the concept of gesture recognition, it is possible to point a finger at the computer screen so that the cursor will move accordingly. This could potentially make conventional input devices such as mouse, keyboards and even touch-screens redundant. Gesture recognition can be conducted with techniques from computer vision and image processing. The literature includes ongoing work in the computer vision field on capturing gestures or more general human pose [2] [3] [4] [5] and movements by cameras connected to a computer. Gesture recognition and pen computing: • In some literature , the term gesture recognition has been used to refer more narrowly to non-text-input handwriting symbols, such as inking on a graphics tablet, multi-touch gestures, and mouse gesture recognition. This is computer interaction through the drawing of symbols with a pointing device cursor (see discussion at Pen computing).

Gesture types In computer interfaces, two types of gestures are distinguished:[6] • Offline gestures: Those gestures that are processed after the user interaction with the object. An example is the gesture to activate a menu. • Online gestures: Direct manipulation gestures. They are used to scale or rotate a tangible object.

Uses Gesture recognition is useful for processing information from humans which is not conveyed through speech or type. As well, there are various types of gestures which can be identified by computers. • Sign language recognition. Just as speech recognition can transcribe speech to text, certain types of gesture recognition software can transcribe the symbols represented through sign language into text.[7] • For socially assistive robotics. By using proper sensors (accelerometers and gyros) worn on the body of a patient and by reading the values from those sensors, robots can assist in patient rehabilitation. The best example can be stroke rehabilitation.

Gesture recognition • Directional indication through pointing. Pointing has a very specific purpose in our society, to reference an object or location based on its position relative to ourselves. The use of gesture recognition to determine where a person is pointing is useful for identifying the context of statements or instructions. This application is of particular interest in the field of robotics.[8] • Control through facial gestures. Controlling a computer through facial gestures is a useful application of gesture recognition for users who may not physically be able to use a mouse or keyboard. Eye tracking in particular may be of use for controlling cursor motion or focusing on elements of a display. • Alternative computer interfaces. Foregoing the traditional keyboard and mouse setup to interact with a computer, strong gesture recognition could allow users to accomplish frequent or common tasks using hand or face gestures to a camera.[9] [10] [11] [12] [13] • Immersive game technology. Gestures can be used to control interactions within video games to try and make the game player's experience more interactive or immersive. • Virtual controllers. For systems where the act of finding or acquiring a physical controller could require too much time, gestures can be used as an alternative control mechanism. Controlling secondary devices in a car, or controlling a television set are examples of such usage.[14] • Affective computing. In affective computing, gesture recognition is used in the process of identifying emotional expression through computer systems. • Remote control. Through the use of gesture recognition, "remote control with the wave of a hand" of various devices is possible. The signal must not only indicate the desired response, but also which device to be controlled.[15] [16] [17]

Input devices The ability to track a person's movements and determine what gestures they may be performing can be achieved through various tools. Although there is a large amount of research done in image/video based gesture recognition, there is some variation within the tools and environments used between implementations. • Depth-aware cameras. Using specialized cameras such as time-of-flight cameras, one can generate a depth map of what is being seen through the camera at a short range, and use this data to approximate a 3d representation of what is being seen. These can be effective for detection of hand gestures due to their short range capabilities.[18] • Stereo cameras. Using two cameras whose relations to one another are known, a 3d representation can be approximated by the output of the cameras. To get the cameras' relations, one can use a positioning reference such as a lexian-stripe or infrared emitters.[19] In combination with direct motion measurement (6D-Vision) gestures can directly be detected. • Controller-based gestures. These controllers act as an extension of the body so that when gestures are performed, some of their motion can be conveniently captured by software. Mouse gestures are one such example, where the motion of the mouse is correlated to a symbol being drawn by a person's hand, as is the Wii [20] [21] [22] Remote, which can study changes in acceleration over time to represent gestures. • Single camera. A normal camera can be used for gesture recognition where the resources/environment would not be convenient for other forms of image-based recognition. Although not necessarily as effective as stereo or depth aware cameras, using a single camera allows a greater possibility of accessibility to a wider audience.[23]

94

Gesture recognition

Challenges There are many challenges associated with the accuracy and usefulness of gesture recognition software. For image-based gesture recognition there are limitations on the equipment used and image noise. Images or video may not be under consistent lighting, or in the same location. Items in the background or distinct features of the users may make recognition more difficult. The variety of implementations for image-based gesture recognition may also cause issue for viability of the technology to general usage. For example, an algorithm calibrated for one camera may not work for a different camera. The amount of background noise also causes tracking and recognition difficulties, especially when occlusions (partial and full) occur. Furthermore, the distance from the camera, and the camera's resolution and quality, also cause variations in recognition accuracy. In order to capture human gestures by visual sensors, robust computer vision methods are also required, for example for hand tracking and hand posture recognition [24] [25] [26] [27] [28] [29] [30] [31] [32] or for capturing movements of the head, facial expressions or gaze direction.

"Gorilla arm" "Gorilla arm" was a side-effect that destroyed vertically-oriented touch-screens as a mainstream input technology despite a promising start in the early 1980s.[33] Designers of touch-menu systems failed to notice that humans are not designed to hold their arms in front of their faces making small motions. After more than a very few selections, the arm begins to feel sore, cramped, and oversized—the operator looks like a gorilla while using the touch screen and feels like one afterwards. This is now considered a classic cautionary tale to human-factors designers; "Remember the gorilla arm!" is shorthand for "How is this going to fly in real use?". Gorilla arm is not a problem for specialist short-term-use uses, since they only involve brief interactions which do not last long enough to cause gorilla arm.

References [1] Matthias Rehm, Nikolaus Bee, Elisabeth André, Wave Like an Egyptian - Accelerometer Based Gesture Recognition for Culture Specific Interactions (http:/ / mm-werkstatt. informatik. uni-augsburg. de/ files/ publications/ 199/ wave_like_an_egyptian_final. pdf), British Computer Society, 2007 [2] Pavlovic, V., Sharma, R. & Huang, T. (1997), "Visual interpretation of hand gestures for human-computer interaction: A review" (http:/ / www. cs. rutgers. edu/ ~vladimir/ pub/ pavlovic97pami. pdf), IEEE Trans. Pattern Analysis and Machine Intelligence., July, 1997. Vol. 19(7), pp. 677 -695. [3] R. Cipolla and A. Pentland, Computer Vision for Human-Machine Interaction (http:/ / books. google. com/ books?id=Pe7gG0LxEUIC& dq=pentland+ cipolla+ computer+ vision+ human+ interaction& printsec=frontcover& source=bl& ots=O2q5ExL8PU& sig=FMhom_f4h9dqeib-6pSSpjbsB38& hl=en& ei=uzvsSbruBdqIsAaq5PCKBw& sa=X& oi=book_result& ct=result& resnum=1), Cambridge University Press, 1998, ISBN 978-0521622530 [4] Ying Wu and Thomas S. Huang, "Vision-Based Gesture Recognition: A Review" (http:/ / reference. kfupm. edu. sa/ content/ v/ i/ vision_based_gesture_recognition__a_revi_291732. pdf), In: Gesture-Based Communication in Human-Computer Interaction, Volume 1739 of Springer Lecture Notes in Computer Science, pages 103-115, 1999, ISBN 978-3-540-66935-7, doi 10.1007/3-540-46616-9 [5] Alejandro Jaimesa and Nicu Sebe, Multimodal human–computer interaction: A survey (http:/ / staff. science. uva. nl/ ~nicu/ PUBS/ PDF/ 2005/ sebeHCI05. pdf), Computer Vision and Image Understanding Volume 108, Issues 1-2, October–November 2007, Pages 116-134 Special Issue on Vision for Human-Computer Interaction, doi:10.1016/j.cviu.2006.10.019 [6] We consider online gestures, which can also be regarded as direct manipulations like scaling and rotating. In contrast, offline gestures are usually processed after the interaction is finished; e. g. a circle is drawn to activate a context menu. (http:/ / vi-c. de/ vic/ sites/ default/ files/ Taxonomy_and_Overview_of_Multi-touch_Frameworks_Revised. pdf) [7] Thad Starner, Alex Pentland, Visual Recognition of American Sign Language Using Hidden Markov Models (http:/ / citeseer. comp. nus. edu. sg/ cache/ papers/ cs/ 405/ ftp:zSzzSzwhitechapel. media. mit. eduzSzpubzSztech-reportszSzTR-306. ps. gz/ starner95visual. ps. gz), Massachusetts Institute of Technology [8] Kai Nickel, Rainer Stiefelhagen, Visual recognition of pointing gestures for human-robot interaction (http:/ / isl. ira. uka. de/ ~stiefel/ papers/ nickel_journal_article_in_press. pdf), Image and Vision Computing, vol 25, Issue 12, December 2007, pp 1875-1884

95

Gesture recognition [9] Lars Bretzner and Tony Lindeberg "Use Your Hand as a 3-D Mouse ..." (http:/ / www. csc. kth. se/ cvap/ abstracts/ brelin-eccv98. html), Proc. 5th European Conference on Computer Vision (H. Burkhardt and B. Neumann, eds.), vol. 1406 of Lecture Notes in Computer Science, (Freiburg, Germany), pp. 141--157, Springer Verlag, Berlin, June 1998. [10] Matthew Turk and Mathias Kölsch, "Perceptual Interfaces" (https:/ / www. cs. ucsb. edu/ research/ tech_reports/ reports/ 2003-33. pdf), University of California, Santa Barbara UCSB Technical Report 2003-33 [11] M Porta "Vision-based user interfaces: methods and applications", International Journal of Human-Computer Studies, 57:11, 27-73, 2002. [12] Afshin Sepehri, Yaser Yacoob, Larry S. Davis "Employing the Hand as an Interface Device" (http:/ / academypublisher. com/ jmm/ vol01/ no07/ jmm01071829. pdf), Journal of Multimedia, vol 1, number 2, pages 18-29 [13] Henriksen, K. Sporring, J. Hornbaek, K. " Virtual trackballs revisited", IEEE Transactions on Visualization and Computer Graphics, Volume 10, Issue 2, paged 206-216, 2004 [14] William Freeman, Craig Weissman, Television control by hand gestures (http:/ / www. merl. com/ reports/ docs/ TR1994-024. pdf), Mitsubishi Electric Research Lab, 1995 [15] Do Jun-Hyeong, Jung Jin-Woo, Sung hoon Jung, Jang Hyoyoung, Bien Zeungnam, Advanced soft remote control system using hand gesture (http:/ / cat. inist. fr/ ?aModele=afficheN& cpsidt=19151757), Mexican International Conference on Artificial Intelligence, 2006 [16] K. Ouchi, N. Esaka, Y. Tamura, M. Hirahara, M. Doi, Magic Wand: an intuitive gesture remote control for home appliances (http:/ / ieeexplore. ieee. org/ xpl/ freeabs_all. jsp?arnumber=1505336), International Conference on Active Media Technology, 2005 (AMT 2005), 2005 [17] Lars Bretzner, Ivan Laptev, Tony Lindeberg, Sören Lenman, Yngve Sundblad "A Prototype System for Computer Vision Based Human Computer Interaction" (http:/ / www. nada. kth. se/ cvap/ abstracts/ cvap251. html), Technical report CVAP251, ISRN KTH NA/P--01/09--SE. Department of Numerical Analysis and Computer Science, KTH (Royal Institute of Technology), SE-100 44 Stockholm, Sweden, April 23–25, 2001. [18] Yang Liu, Yunde Jia, A Robust Hand Tracking and Gesture Recognition Method for Wearable Visual Interfaces and Its Applications, Proceedings of the Third International Conference on Image and Graphics (ICIG’04), 2004 [19] Kue-Bum Lee, Jung-Hyun Kim, Kwang-Seok Hong, An Implementation of Multi-Modal Game Interface Based on PDAs, Fifth International Conference on Software Engineering Research, Management and Applications, 2007 [20] Per Malmestig, Sofie Sundberg, SignWiiver - implementation of sign language technology (http:/ / www. tricomsolutions. com/ academic_reports. html) [21] Thomas Schlomer, Benjamin Poppinga, Niels Henze, Susanne Boll, Gesture Recognition with a Wii Controller (http:/ / www. wiigee. com/ download_files/ gesture_recognition_with_a_wii_controller-schloemer_poppinga_henze_boll. pdf), Proceedings of the 2nd international Conference on Tangible and Embedded interaction, 2008 [22] AiLive Inc., LiveMove White Paper (http:/ / www. ailive. net/ papers/ LiveMoveWhitePaper_en. pdf), 2006 [23] Wei Du, Hua Li, Vision based gesture recognition system with single camera, 5th International Conference on Signal Processing Proceedings, 2000 [24] Ivan Laptev and Tony Lindeberg "Tracking of Multi-state Hand Models Using Particle Filtering and a Hierarchy of Multi-scale Image Features" (http:/ / www. csc. kth. se/ cvap/ abstracts/ cvap245. html), Proceedings Scale-Space and Morphology in Computer Vision, Volume 2106 of Springer Lecture Notes in Computer Science, pages 63-74, Vancouver, BC, 1999. ISBN 978-3-540-42317-1, doi 10.1007/3-540-47778-0 [25] Christian von Hardenberg and François Bérard, "Bare-hand human-computer interaction" (http:/ / citeseerx. ist. psu. edu/ viewdoc/ download?doi=10. 1. 1. 23. 4541& rep=rep1& type=pdf), ACM International Conference Proceeding Series; Vol. 15 archive Proceedings of the 2001 workshop on Perceptive user interfaces, Orlando, Florida, Pages: 1 - 8, 2001 [26] Lars Bretzner, Ivan Laptev, Tony Lindeberg "Hand gesture recognition using multi-scale colour features, hierarchical models and particle filtering" (http:/ / www. csc. kth. se/ cvap/ abstracts/ BreLapLin-FG02. html), Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition, Washington, DC, USA, 21–21 May 2002, pages 423-428. ISBN 0-7695-1602-5, doi 10.1109/AFGR.2002.1004190 [27] Domitilla Del Vecchio, Richard M. Murray Pietro Perona, "Decomposition of human motion into dynamics-based primitives with application to drawing tasks" (http:/ / www. cds. caltech. edu/ ~ddomitilla/ reports/ AutomaticaReport. pdf), Automatica Volume 39, Issue 12, December 2003, Pages 2085-2098 , doi:10.1016/S0005-1098(03)00250-4. [28] Thomas B. Moeslund and Lau Nørgaard, "A Brief Overview of Hand Gestures used in Wearable Human Computer Interfaces" (http:/ / www. vision. auc. dk/ ~tbm/ Publications/ gesture-hci. pdf), Technical report: CVMT 03-02, ISSN: 1601-3646, Laboratory of Computer Vision and Media Technology, Aalborg University, Denmark. [29] M. Kolsch and M. Turk "Fast 2D Hand Tracking with Flocks of Features and Multi-Cue Integration" (http:/ / ilab. cs. ucsb. edu/ projects/ mathias/ KolschTurk2004Fast2DHandTrackingWithFlocksOfFeatures. pdf), CVPRW '04. Proceedings Computer Vision and Pattern Recognition Workshop, May 27-June 2, 2004, doi 10.1109/CVPR.2004.71 [30] Xia Liu Fujimura, K., "Hand gesture recognition using depth data", Proceedings of the Sixth IEEE International Conference on Automatic Face and Gesture Recognition, May 17–19, 2004 pages 529- 534, ISBN 0-7695-2122-3, doi 10.1109/AFGR.2004.1301587. [31] Stenger B, Thayananthan A, Torr PH, Cipolla R: "Model-based hand tracking using a hierarchical Bayesian filter" (http:/ / www. bmva. ac. uk/ sullivan/ prizethesis-2005. pdf), IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(9):1372-84, Sep 2006. [32] A Erol, G Bebis, M Nicolescu, RD Boyle, X Twombly, "Vision-based hand pose estimation: A review" (http:/ / www. cse. unr. edu/ ~bebis/ handposerev. pdf), Computer Vision and Image Understanding Volume 108, Issues 1-2, October–November 2007, Pages 52-73 Special Issue

96

Gesture recognition on Vision for Human-Computer Interaction, doi:10.1016/j.cviu.2006.10.012. [33] Windows 7? No arm in it - Mixed Signals - Rupert Goodwins's Blog at ZDNet.co.uk Community (http:/ / community. zdnet. co. uk/ blog/ 0,1000000567,10008314o-2000331777b,00. htm)

External links • A Gesture Recognition Review (http://perception.inrialpes.fr/people/Cuzzolin/review.html)--Compendium of references • The future, it is all a Gesture (http://www.bruceongames.com/2007/10/02/the-future-it-is-all-a-gesture/ )--Gesture interfaces and video gaming • Interactive Projection Systems (http://www.luminvision.co.uk/interactive.htm)--Commercial Motion Tracking and Gesture Recognition Products

97

Google Goggles

98

Google Goggles Google Goggles

Operating system Android, iOS Platform

Android phones, iPhone

Size

1.12 MB

Website

/mobile/goggles

[1]

Google Goggles is a downloadable image recognition application created by Google Inc. which can be currently found on the Mobile Apps page of Google Mobile.[2] It is used for searches based on pictures taken by handheld devices. For example, taking a picture of a famous landmark would search for information about it, or taking a picture of a product's barcode will search for information on the product.[3]

History Google Goggles was developed for use on Google's Android operating systems for mobile devices. While initially only available in a beta version for Android phones, Google announced its plans to enable the software to run on other platforms, notably iPhone and BlackBerry devices.[4] Google has not discussed a non-handheld format. On 5 October 2010, Google announced availability of Google Goggles for iPhone devices, that run iOS 4.0.[5]

Uses The program proposed will be able to identify virtually anything. Currently the system can identify various labels or landmarks, allowing users to learn about such items without needing a text-based search. The system can identify products barcodes or labels that allow users to search for similar products and prices, and save codes for future reference, similar to the failed CueCat of the late '90s, but with more functionality.[3] The system will also recognize printed text and using optical character recognition (OCR) produce a text snippet, and in some cases even translate the snippet into another language.[3]

Future uses Google is currently working to make the system able to recognize different plants and leaves, which can aid curious persons, those wishing to avoid toxic plants, and botanists and environmentalists searching for rare plants.[4]

Virtual worlds Goggle applications are in the process of being used for metaverse virtual world image indexing and catalog applications. Regional coordinates are cammed by avatar-based movement in virtual camera scripted vehicles through 3D simulators in pseudo-Levy patterns and indexed with image captures for recognition correlation of objects, avatars, and scripts in a virtual-GIS. Reported forecast is for searchable database results of the Second Life and/or OpenSimulator virtual worlds to be available as early as 3Q-2011 from imaging suppliers.

Google Goggles

99

Current version As of December 2010, Google Goggles is running on version 1.3.[6] Goggles is specifically developed to run on mobile devices running the Android operating system. Goggles runs on any phone running Android version 1.6 or higher and can be installed using the Android Market.[3] Although developed for Android there is now also an iPhone version, as part of the Google Search app, available from the iTunes Store or App Store. Goggles requires iPhone 3GS or iPhone 4 on iOS 4.0 or higher to run.[3] In January 2011, version 1.3 was released; it can solve Sudoku puzzles.[7]

References [1] [2] [3] [4]

http:/ / www. google. com/ mobile/ goggles Google Mobile (http:/ / www. google. com/ mobile/ ) Google mobile: Mainpage on Google Goggles (http:/ / www. google. com/ mobile/ goggles/ ), visited 8 December 2010 PCWorld: Goggles will reach other platforms (http:/ / www. pcworld. com/ article/ 184011/ confirmed_google_goggles_will_reach_other_platforms. html) [5] Post on Blogspot from user Googlemobile (http:/ / googlemobile. blogspot. com/ 2010/ 10/ open-your-eyes-google-goggles-now. html) [6] Google Goggles Release Notes (http:/ / www. google. com/ support/ mobile/ bin/ answer. py?hl=en& answer=181358), visited 8 December 2010 [7] http:/ / www. t3. com/ news/ google-goggles-can-now-solve-sudoku-puzzles?=52345

Image retrieval An image retrieval system is a computer system for browsing, searching and retrieving images from a large database of digital images. Most traditional and common methods of image retrieval utilize some method of adding metadata such as captioning, keywords, or descriptions to the images so that retrieval can be performed over the annotation words. Manual image annotation is time-consuming, laborious and expensive; to address this, there has been a large amount of research done on automatic image annotation. Additionally, the increase in social web applications and the semantic web have inspired the development of several web-based image annotation tools. The first microcomputer-based image database retrieval system was developed at MIT, in the 1980s, by Banireddy Prasaad, Amar Gupta, Hoo-min Toong, and Stuart Madnick.[1] [2]

A 2008 survey article documented progresses after 2007.

Search methods Image search is a specialized data search used to find images. To search for images, a user may provide query terms such as keyword, image file/link, or click on some image, and the system will return images "similar" to the query. The similarity used for search criteria could be meta tags, color distribution in images, region/shape attributes, etc. • Image meta search - search of images based on associated metadata such as keywords, text, etc. • Content-based image retrieval (CBIR) – the application of computer vision to the image retrieval. CBIR aims at avoiding the use of textual descriptions and instead retrieves images based on similarities in their contents (textures, colors, shapes etc.) to a user-supplied query image or user-specified image features. • List of CBIR Engines - list of engines which search for images based image visual content such as color, texture, shape/object, etc.

Image retrieval

Data Scope It is crucial to understand the scope and nature of image data in order to determine the complexity of image search system design. The design is also largely influenced by factors such as the diversity of user-base and expected user traffic for a search system. Along this dimension, search data can be classified into the following categories: • Archives - usually contain large volumes of structured or semi-structured homogeneous data pertaining to specific topics. • Domain-Specific Collection - this is a homogeneous collection providing access to controlled users with very specific objectives. Examples of such a collection are biomedical and satellite image databases. • Enterprise Collection - a heterogeneous collection of images that is accessible to users within an organization’s intranet. Pictures may be stored in many different locations. • Personal Collection - usually consists of a largely homogeneous collection and is generally small in size, accessible primarily to its owner, and usually stored on a local storage media. • Web - World Wide Web images are accessible to everyone with an Internet connection. These image collections are semi-structured, non-homogeneous and massive in volume, and are usually stored in large disk arrays.

Evaluations There are evaluation workshops for image retrieval systems aiming to investigate and improve the performance of such systems. • ImageCLEF [3] - a continuing track of the Cross Language Evaluation Forum [4] that evaluates systems using both textual and pure-image retrieval methods. • Content-based Access of Image and Video Libraries [5] - a series of IEEE workshops from 1998 to 2001.

References [1] Prasad, B E; A Gupta, H-M Toong, S.E. Madnick (February 1987). "A microcomputer-based image database management system". IEEE Transactions on Industrial Electronics IE-34 (1): 83–8. doi:10.1109/TIE.1987.350929. [2] Datta, Ritendra; Dhiraj Joshi, Jia Li, James Z. Wang (April 2008). "Image Retrieval: Ideas, Influences, and Trends of the New Age" (http:/ / infolab. stanford. edu/ ~wangz/ project/ imsearch/ review/ JOUR/ ). ACM Computing Surveys 40 (2): 1–60. doi:10.1145/1348246.1348248. . [3] http:/ / www. imageclef. org [4] http:/ / www. clef-campaign. org [5] http:/ / ieeexplore. ieee. org/ xpl/ RecentCon. jsp?punumber=4980

External links • (http://research.microsoft.com/en-us/um/people/larryz/zitnickcfir03.pdf) Content-Free Image Retrieval by C. Lawrence Zitnick and Takeo Kanade, May 2003 • PDF (http://www.asis.org/Bulletin/Jun-09/JunJul09_Uzwyshyn.pdf) Bulletin of the American Society for Information Science & Technology Special Issue on Visual Search (http://uwf.edu/ruzwyshyn/2009PDF/ Bulletin_JunJul09_Finaloptimized.pdf). June/July 2009. 35:5 ISSN: 1550-8366. • alipr.com (http://www.alipr.com) Automatic image tagging and visual image search. Developed with Stanford and Penn State technologies. • CIRES (http://amazon.ece.utexas.edu/~qasim/research.htm) Image retrieval system developed by the University of Texas at Austin. • Image Search (http://www.imagesbox.com) Image search engine with slide show future. • FIRE (http://thomas.deselaers.de/FIRE) Image retrieval system developed by the RWTH Aachen University, Aachen, Germany. • GIFT (http://www.gnu.org/software/gift/) GNU Image Finding Tool, originally developed at the University of Geneva, Switzerland.

100

Image retrieval

101

• ImageCLEF (http://www.imageclef.org) A benchmark to compare the performance of image retrieval systems. • imgSeek (http://www.imgseek.net) Open-source desktop photo collection manager and viewer with content-based search and many other features. • img(Anaktisi) (http://www.anaktisi.net) This Web-Solution implements a new family of CBIR descriptors. These descriptors combine in one histogram color and texture information and are suitable for accurately retrieving images. • Caliph & Emir (http://www.semanticmetadata.net/): Creation and Retrieval of images based on MPEG-7 (GPL). • img(Rummager) (http://www.img-rummager.com): Image retrieval Engine (Freeware Application). • Visual Similarity Duplicate Image Finder (http://www.mindgems.com/products/VS-Duplicate-Image-Finder/ VSDIF-About.htm): Photo collection manager and viewer with content-based image search. • Search by Drawing (http://www.sepham.com/)

Image-based modeling and rendering In computer graphics and computer vision, image-based modeling and rendering (IBMR) methods rely on a set of two-dimensional images of a scene to generate a three-dimensional model and then render some novel views of this scene. The traditional approach of computer graphics has been used to create a geometric model in 3D and try to reproject it onto a two-dimensional image. Computer vision, conversely, is mostly focused on detecting, grouping, and extracting features (edges, faces, etc.) present in a given picture and then trying to interpret them as three-dimensional clues. Image-based modeling and rendering allows the use of multiple two-dimensional images in order to generate directly novel two-dimensional images, skipping the manual modeling stage.

Light modeling Instead of considering only the physical model of a solid, IBMR methods usually focus more on light modeling. The fundamental concept behind IBMR is the plenoptic illumination function which is a parametrisation of the light field. The plenoptic function describes the light rays contained in a given volume. It can be represented with seven dimensions: a ray is defined by its position , its orientation , its wave length and its time : . IBMR methods try to approximate the plenoptic function to render a novel set of two-dimensional images from another. Given the high dimensionality of this function, practical methods place constraints on the parameters in order to reduce this number (typically to 2 to 4).

IBMR methods and algorithms • • • •

View morphing generates a transition between images Panoramic imaging renders panoramas using image mosaics of individual still images Lumigraph relies on a dense sampling of a scene Space carving generates a 3D model based on a photo-consistency check

External links • Mixed Reality Toolkit (MRT) [1] - University College London • insight3d [2] - open source image-based 3d modeling software

Image-based modeling and rendering

102

References [1] http:/ / www. cs. ucl. ac. uk/ staff/ r. freeman/ [2] http:/ / insight3d. sourceforge. net/

Intelligent character recognition In computer science, intelligent character recognition (ICR) is an advanced optical character recognition (OCR) or — rather more specific — handwriting recognition system that allows fonts and different styles of handwriting to be learned by a computer during processing to improve accuracy and recognition levels. Most ICR software has a self-learning system referred to as a neural network, which automatically updates the recognition database for new handwriting patterns. It extends the usefulness of scanning devices for the purpose of document processing, from printed character recognition (a function of OCR) to hand-written matter recognition. Because this process is involved in recognising hand writing, accuracy levels may, in some circumstances, not be very good but can achieve 97%+ accuracy rates in reading handwriting in structured forms. Often to achieve these high recognition rates several read engines are used within the software and each is given elective voting rights to determine the true reading of characters. In numeric fields, engines which are designed to read numbers take preference, while in alpha fields, engines designed to read hand written letters have higher elective rights. When used in conjunction with a bespoke interface hub, hand-written data can be automatically populated into a back office system avoiding laborious manual keying and can be more accurate than traditional human data entry. An important development of ICR was the invention of Automated Forms Processing in 1993. This involved a three stage process of capturing the image of the form to be processed by ICR and preparing it to enable the ICR engine to give best results, then capturing the information using the ICR engine and finally processing the results to automatically validate the output from the ICR engine. This application of ICR increased the usefulness of the technology and made it applicable for use with real world forms in normal business applications. Modern software applications use ICR as a technology of recognizing text in forms filled in by hand (hand-printed): Company

Products

ICR Languages Supported

A2IA

A2iA DocumentReader A2iA CheckReader A2iA AddressReader A2iA FieldReader

ExperVision

TypeReader OpenRTK English, French, German, Italian, Spanish, Portuguese, Danish, Dutch, Swedish, Norwegian, Hungarian, Polish, Simplified Chinese, Traditional Chinese, Russian, Finnish and Polynesian

ABBYY

ABBYY FlexiCapture ABBYY FlexiCapture Engine ABBYY FineReader Engine

I.R.I.S. Group IRISCapture Pro for Forms

English, French, German, Italian, Portuguese and Spanish

Afrikaans, Albanian, Aymara, Azerbaijani (Latin), Basque, Bemba, Blackfoot, Breton, Bugotu, Bulgarian, Cebuano, Chamorro, Corsican, Crimean Tatar, Croatian, Crow, Czech, Dakota (Sioux), Dutch (Belgium), Dutch (Netherlands), English, Estonian, Even, Evenki, Fijian, Finnish, French, Frisian, Friulian, Galician, Ganda, German, German (Luxembourg), German (new spelling), Greek, Guarani, Hani, Hausa, Hawaiian, Hungarian, Icelandic, Indonesian, Irish, Italian, Jingpo, Karachay-balkar, Kasub, Kawa, Kazakh, Kirghiz, Kongo, Kpelle, Kumyk, Kurdish, Latin, Latvian, Lithuanian, Luba, Malagasy, Malinke, Maori, Maya, Miao, Minangkabau, Mohawk, Moldavian, Mongol, Mordvin, Nahuatl, Nivkh, Nogay, Nyanja, Ojibway, OldFrench, OldGerman, OldItalian, OldSpanish, Papiamento, Polish, Quechua, Rhaeto-Romanic, Romanian, Romany, Rundi, Russian, Rwanda, Sami (Lappish), Samoan, Scottish Gaelic, Selkup, Serbian (Latin), Slovak, Slovenian, Somali, Sotho, Spanish, Swahili, Swazi, Tagalog, Tahitian, Tok Pisin, Tongan, Tswana, Tun, Turkish, Uigur (Latin), Ukrainian, Wolof, Xhosa, Zapotec, Ido, Interlingua Latin based languages

Intelligent character recognition LEADTOOLS LEADTOOLS ICR SDK Module

103 Catalan, Czech, Danish, Dutch, English, Finnish, French, German, Hungarian, Italian, Norwegian, Polish, Portuguese, Spanish, Swedish

Taking ICR to the Next Level Intelligent word recognition (IWR) can not only recognize and extract printed-handwritten information, but cursive handwriting as well. ICR recognizes on the character-level, whereas IWR works with full words or phrases. Capable of capturing unstructured information from every day pages, IWR is said to be more evolved than hand print ICR (according to the CCA (Committee for Capturing Abstractions)). Not meant to replace conventional ICR and OCR systems, IWR is optimized for processing real-world documents that contain mostly free-form, hard-to-recognize data fields that are inherently unsuitable for ICR. This means that the highest and best use of IWR is to eliminate a high percentage of the manual entry of handwritten data and run-on hand print fields on documents that otherwise could be keyed only by humans.

Iris recognition Iris recognition is a method of biometric authentication that uses pattern-recognition techniques based on high-resolution images of the irides of an individual's eyes. Not to be confused with another, less prevalent, ocular-based technology, retina scanning, iris recognition uses camera technology, with subtle infrared illumination reducing specular reflection from the convex cornea, to create images of the detail-rich, intricate structures of the iris. Converted into digital templates, these images provide mathematical representations of the iris that yield unambiguous positive identification of an individual. Iris scanners use pattern-recognition techniques based on images of

Iris recognition efficacy is rarely impeded by glasses or the irides of an individual's eyes. contact lenses. Iris technology has the smallest outlier (those who cannot use/enroll) group of all biometric technologies. Because of its speed of comparison, iris recognition is the only biometric technology well-suited for one-to-many identification. A key advantage of iris recognition is its stability, or template longevity, as, barring trauma, a single enrollment can last a lifetime. Breakthrough work to create the iris-recognition algorithms required for image acquisition and one-to-many matching was pioneered by John G. Daugman, Ph.D, OBE (University of Cambridge Computer Laboratory). These were utilized to

Iris recognition

104

effectively debut commercialization of the technology in conjunction with an early version of the IrisAccess system designed and manufactured by Korea's LG Electronics. Daugman's algorithms are the basis of almost all currently (as of 2006) commercially deployed iris-recognition systems. (In tests where the matching thresholds are—for better comparability—changed from their default settings to allow a false-accept rate in the region of 10−3 to 10−4 [1], the IrisCode false-reject rates are comparable to the most accurate single-finger fingerprint matchers [2].) Iris recognition system based on pattern matching

Visible Wavelength (VW) vs Near Infrared (NIR) Imaging The majority of iris recognition benchmarks are implemented in Near Infrared (NIR) imaging by emitting 750 nm wavelength light source. This is done to avoid light reflections from cornea in iris which makes the captured images very noisy. Such images are challenging for feature extraction procedures and consequently hard to recognize at the identification step. Although, NIR imaging provides good quality images, it loses pigment melanin information, which is a rich source of information for iris recognition. The melanin, also known as chromophore, mainly consists of two distinct heterogeneous macromolecules, called eumelanin (brown–black) and pheomelanin (yellow–reddish) [3], [4]. NIR imaging is not sensitive to these chromophores, and as a result they do not appear in the captured images. In contrast, visible wavelength (VW) imaging keeps the related chromophore information and, compared to NIR, provides rich sources of information mainly coded as shape patterns in iris. Hosseini et al. [5] provide a comparison between these two imaging modalities and fused the results to boost the recognition rate. An alternative feature extraction method to encode VW iris images was also introduced, which is highly robust to reflectivity terms in iris. Such fusion results are seemed to be alternative approach for multi-modal biometric systems which intend to reach high accuracies of recognition in large databanks. Visible Wavelength Iris Image

Near Infrared (NIR) version

Iris recognition

Operating principle An iris-recognition algorithm first has to identify the approximately concentric circular outer boundaries of the iris and the pupil in a photo of an eye. The set of pixels covering only the iris is then transformed into a bit pattern that preserves the information that is essential for a statistically meaningful comparison between two iris images. The mathematical methods used resemble those of modern lossy compression algorithms for photographic images. In the case of Daugman's algorithms, a Gabor wavelet transform is used in order to extract the spatial frequency range that contains a good best signal-to-noise ratio considering the focus quality of available cameras. The result is a set of complex numbers that carry local amplitude and phase information for the iris image. In Daugman's algorithms, all amplitude information is discarded, and the resulting 2048 bits that represent an iris consist only of the complex sign bits of the Gabor-domain representation of the iris image. Discarding the An IriScan model 2100 iris scanner amplitude information ensures that the template remains largely unaffected by changes in illumination and virtually negligibly by iris color, which contributes significantly to the long-term stability of the biometric template. To authenticate via identification (one-to-many template matching) or verification (one-to-one template matching), a template created by imaging the iris is compared to a stored value template in a database. If the Hamming distance is below the decision threshold, a positive identification has effectively been made. A practical problem of iris recognition is that the iris is usually partially covered by eyelids and eyelashes. In order to reduce the false-reject risk in such cases, additional algorithms are needed to identify the locations of eyelids and eyelashes and to exclude the bits in the resulting code from the comparison operation.

Advantages The iris of the eye has been described as the ideal part of the human body for biometric identification for several reasons: • It is an internal organ that is well protected against damage and wear by a highly transparent and sensitive membrane (the cornea). This distinguishes it from fingerprints, which can be difficult to recognize after years of certain types of manual labor. • The iris is mostly flat, and its geometric configuration is only controlled by two complementary muscles (the sphincter pupillae and dilator pupillae) that control the diameter of the pupil. This makes the iris shape far more predictable than, for instance, that of the face. • The iris has a fine texture that—like fingerprints—is determined randomly during embryonic gestation. Even genetically identical individuals have completely independent iris textures, whereas DNA (genetic "fingerprinting") is not unique for the about 0.2% of the human population who have a genetically identical twin. • An iris scan is similar to taking a photograph and can be performed from about 10 cm to a few meters away. There is no need for the person to be identified to touch any equipment that has recently been touched by a stranger, thereby eliminating an objection that has been raised in some cultures against fingerprint scanners, where a finger has to touch a surface, or retinal scanning, where the eye can be brought very close to a lens (like looking into a microscope lens).

105

Iris recognition • Some argue that a focused digital photograph with an iris diameter of about 200 pixels contains much more long-term stable information than a fingerprint. • The originally commercially deployed iris-recognition algorithm, John Daugman's IrisCode, has an unprecedented false match rate (better than 10−11). • While there are some medical and surgical procedures that can affect the colour and overall shape of the iris, the fine texture remains remarkably stable over many decades. Some iris identifications have succeeded over a period of about 30 years.

Disadvantages • Iris scanning is a relatively new technology and is incompatible with the very substantial investment that the law enforcement and immigration authorities of some countries have already made into fingerprint recognition. • Iris recognition is very difficult to perform at a distance larger than a few meters and if the person to be identified is not cooperating by holding the head still and looking into the camera. However, several academic institutions and biometric vendors are developing products that claim to be able to identify subjects at distances of up to 10 meters ("standoff iris" or "iris at a distance"). • As with other photographic biometric technologies, iris recognition is susceptible to poor image quality, with associated failure to enroll rates. [6] • As with other identification infrastructure (national residents databases, ID cards, etc.), civil rights activists have voiced concerns that iris-recognition technology might help governments to track individuals beyond their will.

Security considerations As with most other biometric identification technology, a still not satisfactorily solved problem with iris recognition is the problem of live-tissue verification. The reliability of any biometric identification depends on ensuring that the signal acquired and compared has actually been recorded from a live body part of the person to be identified and is not a manufactured template. Many commercially available iris-recognition systems are easily fooled by presenting a high-quality photograph of a face instead of a real face, which makes such devices unsuitable for unsupervised applications, such as door access-control systems. The problem of live-tissue verification is less of a concern in supervised applications (e.g., immigration control), where a human operator supervises the process of taking the picture. Methods that have been suggested to provide some defence against the use of fake eyes and irises include: • Changing ambient lighting during the identification (switching on a bright lamp), such that the pupillary reflex can be verified and the iris image be recorded at several different pupil diameters • Analysing the 2D spatial frequency spectrum of the iris image for the peaks caused by the printer dither patterns found on commercially available fake-iris contact lenses • Analysing the temporal frequency spectrum of the image for the peaks caused by computer displays • Using spectral analysis instead of merely monochromatic cameras to distinguish iris tissue from other material • Observing the characteristic natural movement of an eyeball (measuring nystagmus, tracking eye while text is read, etc.) • Testing for retinal retroreflection (red-eye effect) • Testing for reflections from the eye's four optical surfaces (front and back of both cornea and lens) to verify their presence, position and shape • Using 3D imaging (e.g., stereo cameras) to verify the position and shape of the iris relative to other eye features A 2004 report by the German Federal Office for Information Security noted that none of the iris-recognition systems commercially available at the time implemented any live-tissue verification technology. Like any pattern-recognition technology, live-tissue verifiers will have their own false-reject probability and will therefore further reduce the

106

Iris recognition

107

overall probability that a legitimate user is accepted by the sensor.

Deployed applications • United Arab Emirates IrisGuard's Homeland Security Border Control has been operating an expellee tracking system in the United Arab Emirates (UAE) since 2001, when the UAE launched a national border-crossing security initiative. Today, all of the UAE's land, air and sea ports of entry are equipped with systems. All foreign nationals who possess a visa to enter the UAE are processed through iris cameras installed at all primary and auxiliary immigration inspection points. To date, the system has apprehended over 330,000 persons re-entering the UAE with fraudulent travel documents.

IrisGuard Inc. UAE Enrollment Station

• Aadhar, India's UID project uses Iris scan along with fingerprints to uniquely identify people and allocate a Unique Identification Number. • One of three biometric identification technologies internationally standardized by ICAO for use in future passports (the other two are fingerprint and face recognition) • Iris recognition technology has been implemented by BioID Technologies SA in Pakistan for UNHCR repatriation project to control aid distribution for Afghan refugees. Refugees are repatriated by UNHCR in cooperation with Government of Pakistan, and they are paid for their travel. To make sure people do not get paid more than once, their irises are scanned, and the system will detect the refugees on next attempt. The database has more than 1.3 million iris code templates and around 4000 registrations per day. The one-to-many iris comparison takes place within 1.5 seconds against 1.3 million iris codes. • At Schiphol Airport, Netherlands, iris recognition has permitted passport-free immigration since 2001. • UK's IRIS — Iris Recognition Immigration System [7]

• Used to verify the recognition of the "Afghan Girl" (Sharbat Gula) by National Geographic photographer Steve McCurry. See http://www.cl. cam.ac.uk/~jgd1000/afghan.html • In a number of US and Canadian airports, as part of the NEXUS program that facilitates entry into the US and Canada for pre-approved, low-risk travelers. • In several Canadian airports, as part of the [8] CANPASS Air program that facilitates entry into Canada for pre-approved, low-risk air travelers.

Iris recognition in fiction

A U.S. Marine Corps Sergeant uses an iris scanner to positively identify a member of the Baghdadi city council prior to a meeting with local tribal leaders, sheiks, community leaders and U.S. service members.

Iris recognition • In Demolition Man (1993), a character played by Wesley Snipes uses the Warden's gouged eye to gain access through a security door. • In Dan Brown's 2000 novel Angels and Demons, an assassin gains access to a top secret CERN laboratory using a scientist's eye. • Steven Spielberg's 2002 science fiction film Minority Report depicts a society in which what appears to be a form of iris recognition has become daily practice. The principal character undergoes an eye transplant in order to change his identity but continues to use his original eyes to gain access to restricted locations. • In The Island (2005), a human clone character played by Ewan McGregor uses his eye to gain access through a security door of the original's house. • The Simpsons Movie (2007) features a scene that illustrates the difficulty of image acquisition in iris recognition.[9] • Numb3rs features a scene where a robber gets into the CalSci facility by cracking the code assigned to a specific iris. • NCIS uses an iris scanner in the garage, where forensic vehicle investigations are carried out and evidence is stored. There is another scanner at the entrance to MTAC. The sequence of Leroy Jethro Gibbs being verified is shown in the title sequence.

References • WO 8605018 [10] Leonard Flom, Aran Safir: Iris recognition system. 28 August 1986; also: US 4641349 [11] issued 2/3/1987. • US 5291560 [12] John Daugman: Biometric personal identification system based on iris analysis. 1 March 1994 • John Daugman: "How iris recognition works [13]". IEEE Transactions on Circuits and Systems for Video Technology 14(1), January 2004, pp. 21–30. • John Daugman: "The importance of being random: statistical principles of iris recognition [14]". Pattern Recognition 36, 2003, pp. 279–291. • John Daugman: "Results from 200 billion iris cross-comparisons [15]". Technical Report UCAM-CL-TR-635, University of Cambridge Computer Laboratory, June 2005. • Zhaofeng He, Tieniu Tan, Zhenan Sun and Xianchao Qiu, "Towards Accurate and Fast Iris Segmentation for Iris Biometrics [16]", In: IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 15 July 2008. IEEE Computer Society Digital Library. IEEE Computer Society, 2 September 2008 . • Zhaofeng He, Tieniu Tan, Zhenan Sun and Xianchao Qiu, "Boosting Ordinal Features for Accurate and Fast Iris Recognition [16]", In: Proc. of the 26th IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'08), pp. 1–8, Anchorage, Alaska, June 2008. • Kaushik Roy, Prabir Bhattacharya, Iris Recognition: A Machine Learning Approach, Publisher: VDM Verlag Dr. Müller (November 21, 2008),ISBN 3639082591,ISBN 978-3639082593. • K. Roy and P. Bhattacharya, "Variational level set method and game theory applied for nonideal iris recognition," 16th IEEE International Conference on Image Processing (ICIP'09), pp. 2721 – 2724, Cairo, Egypt, Nov. 7-11, 2009. Print ISBN 978-1-4244-5653-6. E-ISBN : 978-1-4244-5655-0. • Y Liu and JD Simon, “Metal-ion interactions and the structural organization of Sepia eume- lanin,” PIGMENT CELL RESEARCH, vol. 18, no. 1, pp. 42–48, FEB 2005. • Paul Meredith and Tadeusz Sarna, “The physical and chemical properties of eumelanin,” PIGMENT CELL RESEARCH, vol. 19, no. 6, pp. 572–594, DEC 2006. [5] • Hosseini, M.S.; Araabi, B.N.; Soltanian-Zadeh, H.; , "Pigment Melanin: Pattern for Iris Recognition ," Instrumentation and Measurement, IEEE Transactions on , vol.59, no.4, pp. 792–804, April 2010.

108