Impact of Event Filtering on OpenSimulator Server Performance

Impact of Event Filtering on OpenSimulator Server Performance Eugenia Gabrielova and Cristina V. Lopes Bren School of Information and Computer Sciences University of California, Irvine [email protected], [email protected] Keywords: Virtual Worlds, OpenSimulator

Abstract Virtual worlds can be used to create immersive 3D teleconferencing and collaboration spaces. One challenge to the scalability of virtual collaboration is the unpredictable behavior of virtual world client viewers. This paper describes the result of a performance study of server optimizations in OpenSimulator, a multi-platform, 3D virtual world server. We explore the effect of server-side optimizations on mitigating client viewer instability. Our results show that filtering out insignificant events decreases server load by at least 24%, restoring it to baseline load levels.

1.

INTRODUCTION

Virtual worlds offer a low-cost and immersive platform for teleconferencing and research collaboration. Virtual world server platforms like Second Life and OpenSimulator are increasingly used for applications in research, education, and technical collaboration. OpenSimulator is an open-source application server compatible with Linden Lab’s Second Life 3D virtual world application server. The OpenSimulator development community hosted a fully virtual conference in September 2013. The virtual venue included 12 separate virtual areas - four each for keynote sessions, breakout sessions, and landing zones. Up to 360 participants attended the virtual event. In addition to attending conference sessions, attendees could visit a virtual exhibition hall and socialize in a dance floor [1]. The conference took place without any technical glitches. This is remarkable because it was the first conference of its kind in OpenSimulator, and one of the largest virtual conferences ever held. In the months leading to the conference, OpenSimulator developers 1 implemented all kinds of serverside optimizations in order for OpenSimulator to be able to cope with so many concurrent users. One of such optimizations was related to client instability. OpenSimulator developers had previously observed unusually high CPU loads when certain client viewers connected to the server. Client viewers used to log into 3D virtual application servers can be unstable due to different physics implementations and event notification policies. 1 The second author is a core developer of OpenSimulator and was responsible for the optimizations described in this paper.

Figure 1. OpenSimulator Community Conference [16] One consequence of instability is that clients start to send a rapid stream of insignificant status updates. This caused performance to suffer. In order to scale a virtual world effectively, it is necessary to cope with client instability. One optimization to address this problem involved filtering the events generated by client viewers due to tiny avatar movements. Processing only the events representing changes above a certain threshold appeared to improve the performance of the simulator, and helped support a smooth conference experience without negatively affecting the perception of users. However, the precise effects of these filters haven’t been systematically measured at the time. In this paper, we describe our experiments that systematically measure those effects. Beyond OpenSimulator, this study is important in scenarios where clients are not trustworthy or have unreasonable resource demands. As virtual conferences scale to thousands of participants, the load on the server will become unmanageable. Mitigating the performance impact of insignificant agent updates is necessary to scale virtual conferences in the future. This paper is organized as follows. Section 2 describes the OpenSimulator system and agent updates. Section 3 details the methodology of our experiment. Section 4 describes results. Section 5 describes related work in virtual worlds and simulation research. Section 6 concludes the paper.

2.

AGENT UPDATES

The architecture of OpenSimulator is designed to scale to many simulators and their user-driven agents. OpenSimulator provides for managing user data and logins, assets, grid

Figure 3. Experimental setup

3.

Figure 2. Robust grid configuration separates some data services from region services [2].

connections, accounts, and inventory. It can be deployed as a standalone application containing all of these services, or in a modular grid mode. In the robust grid mode (Figure 2), data services (such as login and database communication) are executed in separate processes from region services (such as physics and inventory) [2]. The OpenSimulator server used for the conference, as well as the one used in our experiments, was configured in robust mode. This allowed us to measure the performance of the region server without the overhead of managing assets, inventories, user logins, and other services. Clients in large-scale online games usually produce streams of events about the users’ actions that may interest other players nearby. If many clients are connected to the same simulator or all subscribe to one another’s event streams, the quantity of event messages will negatively impact game performance [6]. In OpenSimulator, client viewers send about 10 agent updates each second. The clients will also ping the server once every two seconds. Hence, ten clients will send about 105 packets/s to the server, and a hundred clients will send 1050 packets/s to the server (including pings). If many of these packets contain insignificant updates, they will create extraneous server processing that doesn’t result in anything useful being shared.

EXPERIMENTAL SETUP

In order to emulate client viewers driven by real users, we used client bots. The pCampBot tool in OpenSimulator provides a bot management framework for Libopenmetaverse bots [9]. Libopenmetaverse is a library written in C# that provides a protocol for clients to communicate with virtual worlds. The pCampBot tool provides some behaviors for bots, such as walking around a region, grabbing nearby primitives, sending chat messages, and doing nothing. First, we implemented a pCampBot behavior to serve as the baseline for performance measurement. This baseline bot sends 10 agent updates each second. The bot remains stationary, sending always the exact same state (position, rotation, etc.). Second, we implemented a pCampBot behavior that also sends 10 agent updates per second, and includes a small change in body rotation, emulating one of the the unstable behaviors we observed in client viewers. We conducted experiments with three different configurations (see Figure 3): (1) bots with baseline behavior and server with no significance filter, (2) bots with unstable behavior and server without significance filter, and (3) bots with unstable behavior and server with significance filter. For each of these 3 configurations, we ran 10, 50, 100 and 200 bots, 3 times each. Our experiment included 12 configurations and 36 measurements in total. Figure 4 provides pseudocode describing the significance filtering. Significant1 corresponds to what OpenSimulator was doing before the optimizations, which was strict equality checks between the current state and the next state sent by the clients; Significant2 calculates the angle between the current and updated rotation and compares it with a threshold. Significant2 is one of the filters that was added to OpenSimulator before the conference event. It is now part of the core distribution. Bots were logged into a simulator with a 16x16 grid of cubes (shown in Figure 5). The cubes are phantom primitives; they do not collide with avatars and thus do not generate physics events that would interfere with the measurements. Bots were seated on the cubes with a command from

function S IGNIFICANT 1(current rotation, agent rotation) signi f icant ← current rotation 6= agent rotation return signi f icant end function function S IGNIFICANT 2(current rotation, agent rotation) delta ← 1 − (quaternion.dot(agent rotation, current rotation))2 threshold ← 0.00001 signi f icant ← delta > threshold return signi f icant end function Figure 4. Pseudocode for event thresholding in OpenSimulator server.

Figure 5. Bots sitting down. the region server. Being seated removes an avatar from the effects of the physics engine. It also emulates the conference scenario: all attendees were asked to find a seat as soon as they arrived at the sessions. Our grid for this experiment consisted of two machines - one to host the server, and one to deploy the bots. Having a separate machine for hosting the bots allowed us to measure network packets and verify the outcome of our experiment. Our OpenSimulator instances were deployed with mono, a multi-platform open source development framework for .NET client and server applications. The OpenSimulator monitoring module was used to record network load and processor time of the region server. The logging mechanism records the output of the ”show stats all” command to a log file every five seconds.

4.

RESULTS

We ran simulations for half an hour to allow the OpenSimulator server to reach a steady state performance level. We collected the last 60 seconds of the CPU log for each of

Figure 6. Distribution of baseline CPU usage for 50 bots.

the three experiments for each configuration. Each set of final minute measurements was then combined into a set of 180 measurements and each data point was considered independent for evaluation purposes. The measurements exhibit a normal distribution. This is shown for 50 bots in Figure 6; the data from all experiments follows the same normal distribution, therefore we omit the other distributions from the paper. Figure 7 shows the results for CPU usage in our experiments. The CPU usage was calculated from the OpenSimulator statistics monitoring log, using the Total Processor Time measurement (the sum of user and privileged processor time). Monitoring measurements are recorded to a log every five seconds. Our result is based on the average CPU usage every 5 seconds for the last 10 minutes of the experiment; 120 data points per run. We combine the data points for each run into a set of 360, and evaluate them as a bag of independent measurements. CPU usage in all experiments with unstable bot behavior and without the filters on the server-side increases the baseline CPU performance by at least 31%. As seen in the figure, the filters restore CPU load to baseline levels; the load decrease for each quantity of bots is shown in Table 1. Our results are statistically significant at 95% confidence level (Table 2). We also collected the last ten minutes of the OpenSimulator statistics log for three measurements; (1) Incoming UDP Packets Received (packets/s), (2) Incoming Packets Processed (packets/s), and (3) Outgoing UDP Packets sent (packets/s). These measurements are recorded every five seconds. As with the CPU measurements, we combine the 120 data points for each run into a set of 360 and evaluate them as independent data points. Figure 8 provides the average of these measurements. Only unfiltered, unstable configurations pass through our significance filter. Figure 8 shows the average quantities of packets sent per

Figure 8. Average packet rates over last 10 minutes of simulation. second in each of our experiments. These network measurements validate the correctness of our experiment. We expect OpenSimulator to receive the same number of events from client viewers; this measurement scales linearly with the quantity of bots. For example, for 100 bots we expected 1050 packets/s and we observed 1059 packets/s. The unfiltered configuration shows a similar pattern in rates of packets processed, as expected (there are no threshold filters). Although the baseline configuration does not include the filters, the server checks for equality of state; thus, the stationary state is filtered out and no packets are processed. In the filtered configuration, insignificant packets are discarded prior to processing. In all configurations, the server then sends the same quantity of packets each second.

Figure 7. CPU usage in the 12 configurations of our experiments.

Table 1. CPU Load Decrease with Filters Enabled Number of Bots

10

50

100

200

Load Decrease

47.4%

40.0%

24.1%

36.4%

These results show that it is possible to mitigate the effects of client instability on virtual world server performance, and measures the extent of that mitigation for the OpenSimulator conference in September, which, so far, had not been measured.

Though the load that our bots incur on the server is not exactly the same as user-driven client viewers, the differences between filtered and unfiltered behaviors are significant. Virtual world activity by real users may have a higher impact on server performance than suggested by a synthetic bot load. However, testing with synthetic bots allows us to pinpoint performance issues while controlling unpredictable client behavior.

Table 2. Differences of Means (∆) and Confidence Intervals (CI) of CPU Utilization 10 bots

5.

50 bots

200 bots

∆

CI

∆

CI

∆

CI

∆

CI

Baseline vs Unfiltered

6.876

0.128

12.373

0.474

13.030

1.332

33.408

3.864

Baseline vs Filtered

0.065

0.085

2.576

0.465

0.033

1.281

4.442

3.412

Unfiltered vs Filtered

6.941

0.128

14.949

0.460

12.997

1.312

37.850

3.710

RELATED WORK

We describe work related to collaboration in virtual environments and to performance and scalability of these virtual environments.

5.1.

100 bots

Collaboration in Virtual Environments

Second Life has been used as an effective platform for virtual conferences. In 2008, the IBM Academy of Technology held their annual general meeting in a secured virtual environment with the support of Linden Labs. The event included over 200 participants. The virtual venue included a conference space, community gathering spaces, and a simulated data center. Attendees responded positively and reported a feeling of immersion in the virtual conference. The event cost approximately 1/5 the cost of a real-world event [14]. Considerable savings are an advantage of virtual teleconferencing (e.g., lower travel costs, no jet lag) [4]. The Meta-Institute for Computational Astrophysics (MICA) was established to explore immersive virtual reality as a platform for scientific research. MICA hosted seminars, lecture series, and professional and outreach activities in Second Life. Participants joined from around the world and represented research science, technology, and education disciplines. The institute also offered a platform for virtual collaboration on numerical simulation (e.g., the gravitational n-body problem) [8]. Virtual venues for experimentation are gaining momentum in education [3]. Baldi and Lopes developed the Universal Campus, a 3-D virtual world containing laboratories, classrooms, meeting rooms, and lecture halls. In a virtual laboratory, users can perform experiments or observe phenomena they might not encounter in a traditional classroom. Building new virtual environments is a barrier to entry for users, and requires a high time investment. The Universal Campus technology decreases this start-up cost by providing an out-ofthe-box virtual campus that can be deployed on a local OpenSimulator server [4].The OpenSimulator Conference Center where the conference took place was based on the Universal Campus. Advances in ubiquitous computing have yielded novel technologies for immersion in virtual environments [7]. Mixed and augmented reality can enhance virtual world con-

ferencing. Stadon has studied social engagement in Second Life and identity in 3D virtual worlds [18]. Kantonen et. al. have developed the ACME system [12], which combines augmented reality and gesture tracking in the real world with a Second Life virtual environment. Their work utilizes both Second Life and OpenSimulator. Interaction design was a vital component of the success of the OpenSimulator Conference [16]. Modes of interaction in virtual worlds can be difficult for new users to understand [4]. One trade-off of virtual worlds is that it can be challenging to simulate reality, despite advancements in technology [4]. Existing virtual world client applications (viewers) are less realistic than other graphics application software, such as 3D modeling and video games [7].

5.2.

Performance

Scaling virtual worlds is necessary to provide immersive and realistic user experiences, and to support novel simulation usages. Liu and colleagues at Intel Labs recommend three simulation requirements for scaling virtual worlds: (1) large-scale, real-time simulations, (2) visualization support for many unique endpoints, and (3) multi-faceted simulation engine support [15]. The future of virtual world simulation could support thousands of users and millions of objects interacting in real time [5]. However, scalability remains challenging due to hardware and network limitations [5]. State-of-the-art world simulations can encounter performance issues with just 100 concurrent clients, due to the performance impact of avatar interactions with one another and objects in the world [7]. Massively Multiplayer Online Games leverage game design techniques and tiered software architectures to support many concurrent users. The commercial MMO EVE Online was able to achieve 3000 simultaneously interacting users through time dilation, a which slows down region time to compensate for server load. World of Warcraft regions can support about 120 concurrent users, and Second Life regions can support up to 100 [6]. Massively Multiplayer Online Games (MMOs) are popular for concurrent groups of up to 80 users. However, transactions from users arrive at a high rate, and a high computational load from a single user can negatively impact system

performance [11]. Servers must have strategies for managing latency to clients logging in from across the globe. Increasingly complex and realistic virtual artifacts such as cloth and fluids can slow down performance [13, 15]. Runtime execution overheads such as memory, computation, and network communication must be minimized to maintain reasonable performance [15, 17]. Exhaustive testing of these complex simulations is nearly impossible [19]. The growth of immersive virtual teleconferencing will rely on advances in this research area. Client-server architecture deployment and maintenance may be prohibitive with thousands of clients. Alternative virtual world architectures have emerged to support the increasing scale of virtual environments. Solipsis is designed with a peer-to-peer virtual architecture. Entities in the virtual world collaboratively maintain a common set of virtual worlds, and the infrastructure itself is self-adaptive [10]. Improvements in CPU utilization will support simulation responsiveness and consequently improve interactive experiences for virtual world users [5]. Thriving virtual world simulations depend on both scalable architectures and interaction design to support immersive experiences.

6.

CONCLUSION AND FUTURE WORK

Virtual worlds are growing increasingly popular to support the activities of distributed teams in many disciplines. Virtual worlds also provide venues for numeric simulation and support collaboration in scientific research communities. Future work will evaluate the performance impact of event filters when avatars are moving around and interacting in a conference environment. No framework exists within OpenSimulator for the automatic measurement of connection or simulator performance. An automated framework for performance measurement would be useful for a continuation of this work. More realistic bot behavior is difficult to implement, but could potentially be generated from anonymized logs of real user activity in OpenSimulator. Motivated by the success of the OpenSimulator Community Conference, we have demonstrated the positive impact of server-side optimizations on OpenSimulator performance. We are working on further optimizations to support the use of virtual worlds for large-scale teleconferencing and immersive collaboration.

REFERENCES [1] OpenSimulator Community Conference, September 2013. http://conference.opensimulator. org/2013/. [2] OpenSimulator Configuration, http://opensimulator.org/wiki/ Configuration/.

2014.

[3] William Sims Bainbridge. The scientific research potential of virtual worlds. Science, 317(5837):472–476, 2007. [4] Pierre Baldi and Crista Lopes. The universal campus: An open virtual 3-d world infrastructure for research and education. eLearn, 2012(4), April 2012. [5] CM Bowman, Dan Lake, and John Hurliman. Designing extensible and scalable virtual world platforms. In Extensible Virtual Worlds Workshop (X10), 2010. [6] Thomas Debeauvais, Arthur Valadares, and Cristina V. Lopes. Rcat: A scalable architecture for massively multiuser online environments. Technical Report UCI-ISR13-2, Institute for Software Research, University of California, Irvine, November 2013. [7] John David N. Dionisio, William G. Burns III, and Richard Gilbert. 3d virtual worlds and the metaverse: Current status and future possibilities. ACM Comput. Surv., 45(3):34:1–34:38, July 2013. [8] S George Djorgovski, Piet Hut, Steve McMillan, E Vesperini, R Knop, W Farr, and Matthew J Graham. Exploring the use of virtual worlds as a scientific research platform: The meta-institute for computational astrophysics (mica). In Facets of virtual environments, pages 29–43. Springer, 2010. [9] Open Metaverse Foundation. libopenmv. http:// openmetaverse.org/. [10] Davide Frey, J´erˆome Royan, Romain Piegay, AnneMarie Kermarrec, Emmanuelle Anceaume, Fabrice Le Fessant, et al. Solipsis: A decentralized architecture for virtual environments. In 1st International Workshop on Massively Multiuser Virtual Environments, 2008. [11] Nitin Gupta, Alan Demers, Johannes Gehrke, Philipp Unterbrunner, and Walker White. Scalability for virtual worlds. In Data Engineering, 2009. ICDE’09. IEEE 25th International Conference on, pages 1311–1314. IEEE, 2009. [12] Tuomas Kantonen, Charles Woodward, and Neil Katz. Mixed reality in virtual world teleconferencing. In Virtual Reality Conference (VR), 2010 IEEE, pages 179– 182. IEEE, 2010. [13] Sanjeev Kumar, Jatin Chhugani, Changkyu Kim, Daehyun Kim, Anthony D Nguyen, Pradeep Dubey, Christian Bienia, and Youngmin Kim. Second life and the new generation of virtual worlds. IEEE Computer, 41(9):46–53, 2008.

[14] Linden Lab. How meeting in second life transformed ibm’s technology elite into virtual world believers. Technical report, 2009. http: //secondlifegrid.net.s3.amazonaws. com/docs/Second_Life_Case_IBM_EN.pdf. [15] Huaiyu Liu, Mic Bowman, Robert Adams, John Hurliman, and Dan Lake. Scaling virtual worlds: Simulation requirements and challenges. In Simulation Conference (WSC), Proceedings of the 2010 Winter, pages 778–790. IEEE, 2010. [16] Cristina Lopes. The future of conferences, 2013. http://tagide.com/blog/2013/09/ the-future-of-conferences/. [17] Kalyan S. Perumalla. Parallel and distributed simulation: Traditional techniques and recent advances. In Proceedings of the 38th Conference on Winter Simulation, WSC ’06, pages 84–95. Winter Simulation Conference, 2006. [18] Julian Stadon. Project slarips: An investigation of mediated mixed reality. In Mixed and Augmented RealityArts, Media and Humanities, 2009. ISMAR-AMH 2009. IEEE International Symposium on, pages 43–48. IEEE, 2009. [19] Gwendolyn H Walton, Robert M Patton, and Douglas J Parsons. Usage testing of military simulation systems. In Simulation Conference, 2001. Proceedings of the Winter, volume 1, pages 771–779. IEEE, 2001.