Measuring Fun: A Case Study in Adapting to the Evolving Metrics of Player Experience

The gaming industry and the concept of gamification have altered the way many developers and users approach interactive products. As social gaming demographics expand to what was previously considered “casual” audiences, more users expect an enjoyable experience from their digital applications and games. Developers now request more detailed subjective descriptions of satisfaction and the player experience from user experience (UX) practitioners. Focusing on how fun a product is for users/players requires subjective, situationally dependent metrics rather than the traditional usability pillars: effectiveness, efficiency, and satisfaction. UX professionals are engaged in constructing a comprehensive ecology of the player experience and how to measure it. This article contributes to that ecology by detailing a case in which our team conducted a usability test on a new video game peripheral. Our client’s primary concern lay with how fun experienced gamers found the device. As our test progressed, we encountered a number of funrelated participant behaviors that led us to develop unique metrics beyond an initial systems survey. These new metrics helped our team and our client better define and discuss enjoyability. Our case, in conjunction with a detailed definition and review of player experience and UX scholarship, shows the importance of adopting metrics contextually specific to the video game product and player group when measuring fun or enjoyability is the primary goal.


Résumé de l'article
The gaming industry and the concept of gamification have altered the way many developers and users approach interactive products. As social gaming demographics expand to what was previously considered "casual" audiences, more users expect an enjoyable experience from their digital applications and games. Developers now request more detailed subjective descriptions of satisfaction and the player experience from user-experience (UX) practitioners. Focusing on how fun a product is for users/players requires subjective, situationally dependent metrics rather than traditional UX efficiency metrics. The UX discipline is still constructing a comprehensive ecology of the player experience and how to measure it. This article contributes to that ecology by detailing a case in which our team conducted a usability test on a new video game peripheral. Our client's primary concern dealt with how fun experienced gamers found the device. As our test progressed, we encountered a number of fun-related participant behaviors that led us to develop new metrics beyond our initial planned metrics. These new metrics helped us and our client better define and discuss enjoyability. Our case, in conjunction with a detailed definition and review of player experience and UX scholarship, shows the importance of adopting metrics contextually specific to the video-game product and player group when measuring fun is the primary goal.

Introduction
Usability, according to the International Organization for Standardization (ISO), focuses on the "extent to which a product can be used by specified users to achieve specific goals with effectiveness, efficiency, and satisfaction in a specified context of use." Most current definitions of usability include some variation of these critical measures: effectiveness, efficiency, and satisfaction (ISO 9241-11; Barnum, 2010). Others (Goodman, Kuniavsky, & Moed, 2012) define usability in similar terms: functionality, efficiency, and desirability. Usability testing methods, through refinement in lab settings as well as industry literature, have been solidified in standard understandings of user experience (UX). Though usability testing and UX are primary concerns for both game software and hardware developers, game studies and UX researchers have recently focused more closely on distinctions between these traditional usability foci and subjective player experiences with video games, especially on the distinct, but difficult to measure, differences in the concepts of satisfaction and enjoyability-the emotional experience of playing a video game-specifically fun.
Many researchers and practitioners have an innate sense of a difference between user satisfaction-based on the traditional ISO measures as they relate to using products-and the subjective emotional experiences of playing video games. However, as we discuss in this paper, it is important for researchers to separate the satisfaction of an efficient, effective, and usable product and the emotions users experience with the gaming medium when performing UX testing on gaming equipment. Current UX methods and the gamer-centric concept of the player experience offer researchers insights into how to approach UX testing when satisfaction and fun serve as important criteria.
The video game industry has experienced tremendous growth over the past 25 years. According to the Entertainment Software Association (ESA) (2019), the video game industry achieved retail sales of $43.4 billion in 2018 in the US alone. Worldwide, video game software and hardware make up a $152 billion industry (Wijman, 2019). Console gaming has seen great hardware innovation over the past 15 years during the previous two console generations from Nintendo, Microsoft, and Sony. Starting in 2006, the Nintendo Wii introduced motion controls to great financial success, and Microsoft and Sony followed with their own control innovations-the Kinect and PlayStation Move, respectively. At the time, these new control systems did much to immerse gamers in virtual worlds, giving longtime gamers new experiences while also bringing greater numbers of casual gamers into the market. Motion controls further led to gaming's integration into healthcare and wellness research and treatment (Kato, 2010;Tanaka et al., 2012). Eventually, the mainstream video game market moved on from these relatively rudimentary control schemes, but two of the major video game hardware developers have innovated on the technology further. Microsoft's Kinect peripheral vanished entirely, but Sony integrated its PlayStation Move technology into its virtual reality set, the PlayStation VR. Nintendo continues to support motion controls with its newest console, the Nintendo Switch.
The current, eighth generation, of video game consoles began during the Fall of 2012 when Nintendo released the Wii U with both the Xbox One and PlayStation 4 following in November 2013. At release, both Microsoft and Sony's consoles sold well, and Sony's PlayStation 4 recently became the fastest console to achieve 100 million sales (Warren, 2019). Though the Wii U was considered a sales flop, Nintendo's second eighth generation console the Switchreleased mid-generation in March 2017-has matched the pace of the PlayStation 4's sales (Minotti, 2019). The presumed ninth console generation is set to begin during the fall of 2020 when both Sony and Microsoft will release the next versions of their PlayStation and Xbox consoles. Amid our current video game console generation, computer gaming has also experienced sustained growth in digital distribution services like Steam and Origin, among others. Additionally, virtual reality technology, such as that created by Oculus (Oculus Rift and Go), Sony (PlayStation VR), and HTC (Vive) among others, is now widely available to a broad consumer base, particularly PC gamers.
The market for application-based gaming is also bigger than ever, with adults gaming on smartphones 60% of the time (ESA, 2019). In the recent past, mobile gamers would have been considered non-gamers or casual gamers by the larger, hardcore gamer culture, but the ubiquity of smartphones has led to mobile devices eclipsing consoles and PCs as the preferred gaming device for a majority of people who play games. Recent scholarship (McCauley et al., 2018) has used survey data and an adapted psychological needs scale to determine the intrinsic motivations of mobile gamers. Though users' in-depth psychological needs regarding gameplay and interfaces were not a major focus of the test presented in this paper, it is clear from the dedicated research out of both the gaming industry and academy (de Winter and Moeller, 2014), that mobile gaming is a major component of not just gaming culture but our larger cultural psyche as well.
The rise of mobile gaming and the gamification of the workplace means that more consumers than ever before are familiar with gaming systems and interfaces. Jesper Juul (2012) claims we live amid what he terms the casual revolution, a moment in time in which every individual is a gamer because of available and accessible technologies, especially our smartphones. Game developer and researcher Jane McGonigal (2011) makes similar proclamations in her book Reality is Broken, arguing that there are nearly 700 million console, PC, or mobile gamers in the world-that number is likely much higher now-and that game design is no longer solely a technical craft, but is a 21 st -century way of thinking.
As gaming technology advances and player bases grow, video games influence our culture and economy in significant ways. Gaming as a cultural zeitgeist propels technology forward. Kessler (2011) argues that the computer game industry has supplanted the military-industrial complex as the primary driver of technological innovation. De Winter and Moeller (2014) contend that games are ubiquitous, everyone plays them, and, as a result, games affect technical industries by entering into a "dialectic with other media," facilitating a remediation process that affects all kinds of digital applications, such as "website design, social media design, workplace training, education, experience design, and so on. These changes are sometimes obvious-gamified work or education-and sometimes they are less obvious, such as the transference of visual aesthetics in [Human-computer Interaction] design" (p. 7). In light of an ever-expanding multi-billiondollar industry, amid new and innovative gaming hardware, and in the face of changing user dynamics, UX and design students and professionals need to be prepared to respond to all kinds of clients, not just those based in the gaming industry. As gaming practices further influence a larger chunk of society's digital experiences, stakeholders in such applications will desire more complex, subjective measures based on fluctuating concepts of user satisfaction. This article details our team's experience user testing a small technology start-up company's gaming peripheral prototype, which is designed to increase player immersion by controlling aspects of the in-game avatar's movements using a headset with an integrated eye-tracking camera and gyroscope. We began by adapting a traditional systems usability scale that is often used to gauge user satisfaction with a product. Our adapted scale attempted to integrate more questions about enjoyability; however, we quickly realized that this scale did not offer enough insights into the player experience of engaging the innovative prototype. In the following section, we review the concept of player experience in the context of both game studies and UX research, and the ways in which UX is responding to evolving gaming technologies. Then, we provide an overview of our experience testing this innovative gaming peripheral. We end by discussing how UX professionals and game developers can and should be prepared to delve deeper into metrics that address the evolving player experience.

The Player Experience as User Experience
UX as a discipline certainly contains the tools to effectively measure users in dynamic situations for a variety of experiences with products or applications. That said, there exists a professional, academic, and public image of UX as focused more on product development and improvement in terms of stricter understandings of its ISO definition. In short, people often assume UX focuses primarily on a product's effective, efficient, and satisfactory use.
According to Geisler and Grice (2013), UX-and usability studies-historically focused on testing the usability of workplace and productivity tools. Dumas and Redish (1993) defined usability as people able to use a product and easily accomplish their own tasks, specifically assuming that "people use products to be productive" (p. 4). Jakob Nielsen, perhaps the most publicly recognizable usability researcher, has only recently moved away from defining usability solely in terms of productivity. As recently as 2009, Nielsen and Pernice restricted discussions of usability on the web to "sites that have a business goal and that aim to support users in getting something done" (p. xvii). Nielsen's cofounder Don Norman takes a broader focus on the experience of games (Isbister, 2008), and the Nielsen Norman Group (2020) collectively adopts a wider view of UX in 2020, but the more traditional opinion persists in academic and professional discussions of how UX functions (Geisler, 2017;Pellegrini, 2020;Stapelton, n.d.).
As gaming culture, hardware, and software permeate digital culture, it is important to explore the distinction between the usability of games and the experiences of playing them. Founder and president of XEODesign Nicole Lazzaro (2008) states, "Play experiences are not work experiences. Games are not used to achieve particular tasks; they are played for pure fun" (p. 317). Echoing this sentiment, professional UX writer Guido Pellegrini (2020) says, "You don't use videogames, you play them" (paragraph 6). Recent literature seeks to better understand how the player experience with video game software and hardware might be more carefully studied in UX testing. Though UX capably studies the experiences of users interfacing with products, systems, and services, a discrepancy still exists in the differences in the espoused primary purposes of video games and the primary function of more traditional digital tools. Pellegrini (2020) notes, "When it comes to videogames, it can be difficult to understand where the product ends and the art begins" (paragraph 5). Stapelton (n.d.) argues that video games must balance usability and functionality with visual appeal, enjoyability, and entertainment.
Recent scholarship compiled in a systematic review of gaming research in the field of humancomputer interaction (HCI) (Caroux et al., 2015) demonstrates a widely held view that studying usability alone does not sufficiently account for entertainment in game design. Though today's model of UX certainly offers the tools to study entertainment products, Caroux et al. (2015) also show that the field seeks-similar to our case-more precision in the ways in which we use and discuss UX when testing for player-video game interactions leading to fun. Fun or enjoyability do not overlap directly with the third pillar of usability, satisfaction. Don Norman (Isbister, 2008) claims that games are about the experience, not usability. "Games," Norman argues, "are very different than normal applications" (p. 354). Though satisfaction is tied to the pleasure of using a product, as we have shown above, so often UX's prime directive is often interpreted, and its methods have grown out of, productivity rather than entertainment and emotional experiences.
Game designer Alex Harbuzinski (2020) provides a UX framework for understanding game design; Harbuzinski's "UX Pyramid" contains product features-objective characteristics, such as usability, reliability, and functionality-at the bottom of the pyramid with a line demarcating the top of the pyramid-containing subjective characteristics, such as significance, enjoyability, and convenience. Of course, a game must function, but it also must deliver a highly subjective, emotionally satisfying experience. A review by Caroux et al. (2015) finds that enjoyment with a gaming product is linked to positive player reactions and emotional experiences with the concept of flow-a euphoric state of concentration and involvement-also linked to positive experiences (Caroux et al., 2015;Lazzaro, 2008). Marcus (2011) suggests that UX developers "may need cross-disciplinary assistance from the world of games" to be ready for changing content and shifts in user interfaces (paragraph 11). Nacke and Drachen (2011) similarly argue that standard usability metrics-effectiveness in task completion or efficiency in error rates, to name a couple-do not always coherently map to digital games and applications. Marcus sees this potential friction in a positive way: "The clash of cultures," he argues, "should be challenging, and also promote a vigorous and influential area for professional change" (paragraph 12). Designers now often focus on the aspects of games that work well; namely, a focus on engagement and entertainment is forcing developers to look at products in new ways (Rettig, 2011a).
Gaming as a practice and hobby has expanded from the hardcore, console and PC gamer demographics to include the social gamer using apps on mobile devices (Rettig, 2011b). As a result, digital media users share similar traits and expectations with traditional video-game players. Thus, user expectations for new products have changed due to the influence of video games and gamified applications; the experience of playing the game and identifying with the game and gamer culture (or not) is more important to the player than a game's functionality. Ferrara (2012a) argues, "Designers who are creating games must be centrally concerned with the quality of the player experience" (paragraph 11). In short, the concept of fun is not simply onethird of an equal triangle of the traditional usability definition, and fun does not equate with the satisfaction a user might get from using a tool. Instead, for gaming products, fun serves as the essential part of the user experience.
This focus on fun necessitates metrics that address the experience of the user with a product as well as the experience of the player in a game. Player experience is a situationally dependent, subjective response a player forms in relationship to a gaming experience. This response is a 6 complex component of the user experience for entertainment systems. The defining property of these entertainment systems is playability (Sanchez et al., 2012). According to Sanchez et al. (2012), "Playability represents the degree to which specified users can achieve specified goals with effectiveness, efficiency and, especially, satisfaction and fun in a playable context of use" (p. 1037). Pellegrini (2020) also homes in on this argument: "What videogames focus on, mostly, is conveying an idea or emotion, whether that's simple fun or profound joy and even sadness. Because of this, typical UX and UI questions like clarity, consistency, transparency, and usability-are much, much harder to answer" (paragraph 7). It is this player experience, this emphasis on feeling a particular emotion at a particular part of the game, that Ferrara (2012a) worries can be overlooked if developers and UX professionals alike adopt too narrow a definition of satisfaction during testing.
The player experience is measurable only through a mixture of qualitative and quantitative metrics aimed at determining how people interact with entertainment software.  describe the purpose of player experience methods as evaluating players to improve gaming. UX professionals use mixed-methods approaches in their work, but traditional usability testing especially in summative, quality assurance testing can involve studying efficiency and effectiveness, with satisfaction taking a backseat in such situations. When studying the usability of gaming or gamified systems, methods such as effectiveness and efficiency are subordinate to satisfaction (Federoff, 2002). Video games are highly interactive systems designed to entertain players who use them to have fun (Sanchez et al., 2012). "What you really want to do," says Don Norman, "is observe the people and whether they are enjoying the process…. That's why I advocate for emotions and total experience" (Isbister, 2008, p. 354). The now defunct gaming magazine GamePro evaluated a game's Fun Factor based on the criteria "Are you gonna have a good time?" Such a measure sounds simplistic because fun can manifest differently based on each player's experience. Still, there's no denying fun is an important part of deciding what players purchase, play, and revere, so how do we begin to measure it? Fun, as we have shown, is a highly subjective, slippery metric, one that can be difficult for UX professionals to pinpoint with traditional usability methods; the oft-used System Usability Scale (SUS) (Brooke, 1986) has been long used as a method for assessing user satisfaction with a particular tool. Yet, the SUS does not provide adequate insights to the player experience because of the differences in user satisfaction-based primarily on effectiveness, efficiency, and functionality-and fun-an emotional experience. Caroux et al. (2015) review research into the player-video game experience and define enjoyment broadly as a positive player reaction during a gameplay session, replete with many avenues to pursue measurement. The difficulty in quantifying enjoyment lies with the fact that players bring their individual experiences to each gameplay session, making each session unique. Renshaw et al. (2009), in an eye-tracking study designed to better measure player engagement, discuss the difficulties their participants had in communicating their emotional experiences during gameplay. Players have difficulty communicating these experiences for several reasons, including their personal gaming histories, the emotional responses elicited from the game in the moment, and the complex technical systems that comprise any given video game. An ecological model provides one way of understanding how emotional elements like fun are tied into the more technical elements of the user experience of video games, system satisfaction, controls, and task efficiency, among others.
Games, gaming peripherals, and gamified applications share characteristics with complex systems. Still (2011) argues, "In dynamic, complex systems there is never a set of finite problems. New problems emerge all the time or change their shape as multiple forces-user, product, environment, and so on-interact with each other" (p. 91). Taking an ecological approach to small-scale UX testing helps reveal more knowledge about an entire community (Still, 2011). In gaming, it is essential to comprehend how player-game interactions occur in context and from multiple perspectives because of the complexity of the system in which learnability and the gaming experience reciprocally influence each other (Law et al., 2012). The kinds of small-scale UX studies for which Still (2011) advocates are especially important when dealing with more subjective testing metrics related to gaming because they can contribute to a greater knowledge of how gamers' mental models work as they interact in gaming environments (Ferrara, 2013).
Mental models of the gaming population will likely expand in the face of new hardware innovations. Our client provided a prototype headset designed to sit on gamers' heads as they played a game. The headset was meant to enhance and more fully immerse players into the gameworld by augmenting the physical player body with more precise virtual control. Though used in conjunction with more typical hand-held controls-either console controllers or keyboard-and-mouse controls-the headset provided additional avenues for movement within the game world. Recent research in HCI (Caroux et al., 2015) notes that every type of movement that players perform during a gameplay session positively affects their perceptions of engagement, and augmenting gaming controls with physical movement makes gamers feel more engaged. Engagement contributes to a positive gaming experience, leading to enjoyment (Örtqvist & Liljedah, 2010). Further, researchers (Caroux et al., 2015) have linked immersion to engagement. Immersion is a psychological state in which one perceives one's self to be "enveloped by, included in, and interacting with an environment that provides a continuous stream of stimuli and experiences" (Witmer & Singer, 1998, p. 227). Ideally, such devices, as they are developed further, will continue to increase the ways in which players experience games and virtual worlds, leading to greater immersion, engagement, and enjoyability.
The issue of evolving mental models is also important as the development of social gaming and gamified applications continues to grow. Gamification has penetrated everyday workplaces and become a model to enhance productivity. What this means is that even individuals who do not play digital games are now familiar with gamified systems to some degree, including simple interfaces and competitive environments with scaling reward systems. The mental models of socalled "casual" gamers have also evolved across several facets of digital society. Users expect fun, immersive experiences, not only functionality, in all digital applications. Ferrara (2012b) argues that "user-centered design is closely related to the player-centric thinking that's the hallmark of all good games" (paragraph 7). User experience practitioners must account for contextually situated testing experiences when dealing with games, gaming peripherals, or gamified applications according to the product, the player, and market demands.
Our small UX test serves as a case that provides useful insights for UX practitioners, game and application designers, and game studies researchers of the ever-changing player-game dynamic. Our case adds to the growing ecological knowledge of the player experience-particularly how to understand each player's subjective experience of fun.

Overview
Our team was tasked with examining the usability of an advanced gaming peripheral developed by a small technology start-up company. The peripheral was specifically designed for this case to be used with a first-person shooter (FPS) video game. The client's goal is eventually to market the device as an optional peripheral addition to popular consoles from Microsoft and Sony, as well as PCs. The purpose of the device is to enable more immersive movement through a game world with a functional gyroscope and eye-tracking cameras, complementing the more traditional controller interface. For this test, only the integrated gyroscope was available for testing. Our team's goal, as directed by our client, was simple: determine how much fun experienced gamers had while playing the open-source online game Xonotic with the gyroscope in conjunction with either a traditional handheld controller or a keyboard, depending on participants' personal preferences.
This peripheral was an early prototype of the planned future device. We tested a 3D-printed, rapid-plastic prototype headset that housed a circular gyroscope enclosure that fit against the user's right ear. Our client's chief science officer programmed Xonotic to accept controls that linked the device to either an Xbox controller or computer keyboard. The client also programmed the game to accept inputs from the gyroscope, mapping it to move the avatar's head-typically the function of the right joystick on controllers or the mouse.
Because the early prototype of the device did not include the eye-tracking camera, the client was more interested in exploring if, and to what extent, an experienced player demographic found the device fun to use rather than uncovering statistically significant errors with its use. Our client's goal placed our team into a conflicted area within usability testing. As reviewed above, the subjective nature of the player experience, when testing gaming hardware and software, is challenging and highly subjective. By testing an early prototype of the device in a formative, exploratory way, the client engaged in iterative product development. Nielsen (1993) showed the efficacy of the iterative product development method as compared with others. Iterative design makes the user experience-in this case, the player experience-central to the development process, integrating user preferences into the very design of the device rather than relying solely on what the developer prefers.
We recruited from the student body at a large research university in Texas. All potential participants completed a pre-screening survey via email, and ten of those individuals were sent formal participation requests with the goal that at least five individuals would participate in the study. The participants consisted of both male and female students ranging in age from 18 to 35 years old. To qualify for research participation, individuals were required to satisfy the following requirements: • Define the abbreviation FPS.
• Play FPS games at least five hours per week.
• Play FPS games on console and/or computer.
We ultimately recruited and tested ten participants during the study.
These tests took place in a two-room usability lab located on the campus of the university in Texas where participants were recruited. The usability lab held a blanket Institutional Review Board (IRB) approval for such testing and all player data was kept confidential in accordance with the IRB's guidelines. The four authors of this article served as the UX researchers for these tests, and none of the authors was at the time or is now affiliated with the client's company in any way.
The facilitator and participant sat in one room at a computer station running the FPS game and the prototype. Two observers sat in a control room at a workstation connected via network to the testing room's workstation. The observers recorded all user tests with Morae-an application developed by TechSmith that facilitates setting up, recording, observing, and analyzing usability testing, both in person and remotely. Morae allowed our team to log notes and record participant audio, video, and keyboard-use data. The Morae software provided us with time stamps for key data points discussed in our procedure section as well as the chance to view participants' facial expressions and gameplay footage in concert. The team viewed the tests through a one-way mirror separating the testing room and the control room.

Procedure
After participants were scheduled, our team conducted a small pilot study to ensure the systems and methods worked. During the primary study, the facilitator greeted the participants, relayed information from the script/protocol, and had participants read and sign a consent form.
Scripts for all tasks were read orally by the facilitator, and a paper copy was given to the participant. In order to minimize potential technical errors, Xonotic was already running on the testing room's workstation when the participant arrived, and Morae was running in the background to record participant actions. All questions posed by participants were answered directly. When participants asked questions whose answers would directly or indirectly influence how they worked with the peripheral device, they were answered with "I'm sorry, I can answer questions about the process, but I can't answer questions about how to perform tasks." Each participant performed the play session twice to evaluate whether experience improved player results. All participants had experience using WASD keyboard controls and console controls. Because of this control experience, players were given an Xbox controller to use in conjunction with the headset prototype.
Following each observation, participants were asked to complete an Enjoyability Survey (see Figure 1). While participants completed this survey, the facilitator met with observers in the control room of the usability lab. Observers relayed any additional questions for the participant to the facilitator. The facilitator then returned to the testing room and conducted a post-test interview in which the facilitator asked the player about the individual's gameplay experience, a technique known as retrospective probing or retrospective recall. We adapted the Enjoyability Survey from the more traditional SUS. The traditional SUS asks users to rate their satisfaction with a system using a Likert scale to respond to ten questions. We initially used the survey because the SUS was originally intended to measure ease of use, and it has been shown to provide deeper insights into a user's satisfaction with a system or tool (Sauro, 2011). We based our enjoyability survey on the SUS with the goal of expanding the notion of satisfaction and enjoyability rather than ease of use with the system. As we discuss in the next section, though we believed our adapting the SUS to measure enjoyability would provide adequate insights, we learned during our analysis that the survey simply did not provide the kinds of insights that answered our client's inquiry about how fun the prototype was for gamers. Thus, we had to evolve our analysis to account for metrics unique to this particular situation.

Adapting Unique Metrics to the Situation
Our team initially considered more traditional methods related to system satisfaction-hence, the Enjoyability Survey's similarity to the more standard SUS. The traditional SUS is used to rate user satisfaction of a system. Sauro (2011) demonstrated that the SUS could provide insights into user satisfaction as well, so we decided to try adapting the SUS into an Enjoyability Survey, which we hoped would provide us and our client with data about how fun the prototype was to use. But a survey, regardless of its goals, can only account for the designer's preconceived notions. As the test progressed, we noticed certain unanticipated player behaviors that clearly affected properties like playability and engagement with the game. We then developed performative metrics that could account for and better indicate enjoyability with a peripheral designed to immerse gamers further in the gameplay experience.
With the Enjoyability Survey yielding data that fell more within the traditional realm of usability and not providing enough insights into fun, we developed situation-specific metrics based on the experience of these particular players, using this particular device, with this particular game. UX professionals contend that studying the player experience in terms of gaming requires contextually situated metrics related to the unique user group, hardware, software, and environment (Ferrara, 2012a(Ferrara, , 2012b(Ferrara, , 2013Law & Sun, 2012;Marcus 2011;Sanchez et al., 2012). Following these exemplars, we sought metrics that could provide better indicators for our client about how much fun players were having while using the prototype. We began focusing on these metrics after reviewing the participants' interviews and gameplay, many of which included comments about the performance of the prototype not meeting their expectations.
Users responded verbally both during gameplay and post-test interviews. Representative samples of participants' verbal frustrations included the following comments that participants made both during their gameplay sessions and the post-test interviews: • "[The device] is impossible to use with an Xbox controller." • "The [gyroscope] didn't follow my head movements." • "There are weird, jerky movements with [the device]. There's a lot of lag." • "[The device] made trying to move around the map really difficult." • "I had to adjust my head. It worked better when I kept my head down a bit." • "It was too hard to move, so I was like 'Oh, I'll just use the mouse.'" • "Maybe if it was more sensitive, it would be easier…. It just wasn't sensitive enough. It was hard for me to turn around without breaking my neck trying to spin going this way [participant mimics exaggerated head movement]." Users' verbal utterances were especially important when gauging the enjoyability, memorability, and learnability of the prototype. Players' interpretations of their experiences helped us better understand their play choices and behaviors, but we needed further contextualization.
After reviewing the gameplay and interviews, we recognized a pattern of behaviors-both physically and virtually-that coincided with these verbalized frustrations. Based on our observations of participants' in-game behaviors, we established a series of metrics: we hypothesized that each instance of exaggerated head movement with the device-each head movement beyond the slight bobbing and weaving necessary to use the device's gyroscopeindicated an attempt to compensate for a slower-than-expected response time from the device. We also adopted kills as an in-game metric for our participants' relative success or failure. Measuring in-game self-kills helped our group gauge the user's level of comfort and control using the device. In-game self-kills frequently occurred when players moved their avatar off the edge of Xonotic's space station-themed level where each test took place, but a few were the result of misusing an in-game weapon. Often, both kinds of self-kills occurred in conjunction with an exaggerated head movement and/or a verbalized frustration. We triangulated the participants' verbalizations, observed interactions with the device, and observed interactions within the game world itself to arrive at these contextually specific metrics. Considering these data points, and, given that our users all identified as experienced FPS gamers, we interpreted both sets of self-kills as signs of discomfort with the eye-tracking device's gyroscope.
In developing this self-kill metric, we realized that adapting to the testing situation provides a prototypical approach based on adapting to the needs of the player, client, and technology. Due to the unique nature of the player experience with this hardware, we used other observationbased metrics to better understand our participants' engagement with the game. To measure how well players adapted to the peripheral and used it as intended, we had our participants play through the same level of the game twice, gauging their relative success and failure with the device based on the number of in-game kills. Measuring the total number of in-game kills, specifically contrasting the total number of kills from the first and second rounds of the game, provided a useful comparison point as our participants became more skilled with the mechanics of Xonotic as well as the hardware. Measuring the number of exaggerated head movements per user from session 1 to session 2 also helped us gauge the relative memorability of using the device. Incidentally, some players improved between their two play sessions and some did not.
Beyond that basic, unremarkable observation, though, some players made unique adaptations of their own during their second play session to improve their experience and make it more normal or familiar, as they applied their previous experiences from other FPS games. These adaptations, along with the self-kills and exaggerated head movements, showed the frustration and difficulty our participants experienced while playing. Some players simply stopped using the headset by keeping their head still to avoid putting themselves in more vulnerable positions in the game. Others put the controller down between sessions, switching to the WASD keyboard controls. Allowing for user-generated "tasks" during a usability test is a common practice, especially in formative testing. Thus, we allowed the users to change their control preferences as they deemed fit. During participant interviews, the players who did make this adjustment confirmed that they made the switch between control schemes to better align with their play experiences and to overcompensate for the frustrations the prototype headset caused them.
Participants also responded with several positive verbal utterances regarding the device, showing a genuine interest in its future directions, including the following statements: • "This is a pretty cool idea. It's gonna be awesome when all the bugs are fixed." • "It's much less cumbersome than a VR headset." • "I could see it being really useful." We also provided the client with a new insight they had not considered when one participant specifically mentioned the device's potential for players with disabilities: • "I can see the potential for other applications, like maybe if the user has a disability that limits hand movement." Finally, of note, we recorded a particularly memorable moment featuring a participant who adapted well to the device using a combination of controlled jumping and strafing using the controller along with a slowed head movement (to account for the input lag): The participant 13 smiled widely, said to himself, "Somebody's about to get daddied," and then leapt from a platform, strafed midair, and performed a long head turn to bring his avatar face-to-face with an enemy upon landing, at which point the participant eliminated the enemy.
Overall, users found this early prototype difficult to use but expressed interest in the device's potential as a secondary peripheral device for established gaming systems. Our observationbased metrics for total kills, self-kills, and exaggerated head movements, combined with several unique verbalizations and use cases, provided the client with unique user interactions with the device, showing their initial frustrations, especially with input lag and limited movement, potential avenues for improvement, and even new uses, as in the comment about players with disabilities.

Discussion
In the larger ecology of studies focused on player experience (Caroux et al., 2015;Federoff, 2002;Kiili et al., 2014;Law & Sun, 2012;Nacke et al., 2012;Renshaw et al. 2009;Sanchez et al., 2012), our case adds to the existing literature that seeks to expand on, while making more precise, existing UX metrics with player-situated metrics that integrate fun and enjoyability into their evaluations of games-based technologies. Such technologies require focusing on the ways in which immersion enhances player engagement, leading, ultimately to the desired emotional reaction, such as fun in our case.
Fun Factor's prominent place as the final, holistic metric that concludes each GamePro review indicates that, despite the more easily measurable metrics like control or graphics, Fun Factor served as the most crucial element of a game. In terms of scholarship, fun can be represented as enjoyability-a heightened emotionally-satisfying response to the game (Caroux et al., 2015). Certainly, games may seek different kinds of emotional experiences, but the simplified Fun Factor metric stood in for that intended emotional response, whether that's feeling frightened during a horror game or excited during a first-person shooter. The ineffable nature of fun means that no matter what metric we adopt, it is going to be difficult to gauge how much a player is enjoying a particular game. Our mixed-methods approach integrates observations and user responses to measure fun from the context of the specific product we tested and the core player group. As mentioned earlier, a mixed-methods approach is common to all UX practitioners, but the contextually dependent nature of the player experience cannot be standardized entirely, though some have tried (Federoff, 2002;. Gaming software and hardware, and even applications based on complex, gamified systems, require flexible testing. The mix of observational data that covers items our participants could not report (like facial expressions), quantifiable metrics like self-kills, and participants' direct responses to questions about fun covers a broad spectrum of gaming experience. These various metrics best accounted for fun in our testing environment. Our metrics document engagement, and, indirectly, immersion. Like the Fun Factor described in the GamePro ratings, there is no simple way to tally up self-kills, facial expressions, and interview response questions to create a definitive, objective measurement of fun.
The indefinable nature of a Fun Factor makes it difficult to precisely measure, but our mixed methods approach places some of the most essential indicators of fun for our specific situationself-kills, facial expressions, and interview responses-in conversation with each other. Broadly speaking, self-kills serve as a stand-in for a player's ability to control an avatar, to avoid pitfalls. Avatar responsiveness is key to maintaining immersion and avoiding player dissonance with the game world-particularly for seasoned gamers not accustomed to falling off the map. Our second metric, facial expressions, indicates immediate player responses to the game. These mostly unplanned responses to in-game stimuli helped us gauge when participants were struggling, when they were mastering the game, and when they became fully engaged to the point of reacting to the in-game events rather than struggling to control their avatar. Immersion and even potential proficiency in a game are not substitutes for fun, but they are useful indicators that the gamer is engaged. Finally, the interview responses created space for gamers to discuss directly how much enjoyment they felt while playing the game. Interview responses must always be taken with a degree of skepticism, which is why we compared our users' self-reports to the observational metrics we created during the testing process. The sum total of these mixedmethods metrics provided our team with insights into our players' experiences with the peripheral device, allowing us to provide our client with evidence-based recommendations to improve their peripheral.
This discussion of enjoyment takes place within a larger conversation about the ways UX professionals and game developers are tasked with accounting for player experience, broadly speaking, including immersion and enjoyability. Current research (Caroux et al., 2015;Örtqvist & Liljedahl, 2010) demonstrate the ways in which immersion and enjoyability-also known as gamer experience-can be studied as separate constructs. Though certain UX methods can separate these constructs, it is important to keep in mind that immersion, along with flow, is a primary contributing factor to engagement (Caroux et al., 2015). And, engagement contributes to the enjoyability of the experience itself. This can be seen in our case, as the immersion of playing the game was often broken by self-kills and exaggerated head movements as players attempted to navigate the gameworld using the prototype. That said, players displayed excitement about the device's potential, a finding confirmed by research that shows movementbased controls result in positive experiences (Caroux et al., 2015). This claim certainly needs further evidence, but it could explain the types of responses our participants gave during the test.
Though it is often easy to identify areas for improvement after UX testing has been completed, our case provides an example of agilely pivoting from traditional systems-based satisfaction metrics to unique metrics that respond to the contextually dependent situation. Marcus (2011) highlights the ways UX practitioners, procedures, and processes can benefit by adopting common gaming qualities. Though Marcus (2011) argues that these two sets of approaches are not identical, he foresees a fruitful future. Ferrara (2012aFerrara ( , 2012bFerrara ( , & 2013 focuses squarely on the importance of the player's experience, an all-hands-on-deck approach that would welcome various hybrids of UX and gaming research. Our study certainly embraces such a hybrid approach, the same kind practiced across the field of HCI when designing video game experiences (Caroux et al., 2015), based on the context of our product and user, and we see the necessity of future hybrid studies that continue developing and exploring player experience. The self-kill metric was situational to our test, and our approach to discussing, observing, and quantifying (as much as possible) a fun metric can apply to UX testing conducted in various gaming environments and beyond to game developers, marketers, and even instructional designers developing educational applications.
Our case focuses specifically on FPS games and players, but the flexible concept of fun plays a significant role in the future of both gaming and UX. Each situational, fun metric targets a different kind of user experience, as additional variables like the game being tested and the target user group will produce different measurements for what fun-or whatever the desired emotional response-looks like to different audiences. Ultimately, we share Marcus' (2011) prediction of a bright future between the overlap of UX and gaming research, one that has gradually taken place over the last several years-the present case included. Fun, as we have studied it, represents a basic yet difficult-to-quantify concept; the clearest statement we can make is that fun is an essential factor for digital applications as creators strive to produce more engaging, appealing, and successful products within an increasingly gamified society. UX research may not be able to quantify the exact difference between what games will score a 4.5 versus a 5.0 on GamePro's Fun Factor metric, but it can provide insights into the basic elements that make for an engaging gaming experience. UX practitioners can provide research, metrics, and insights about ways to measure fun in both gaming and other interactive media. Demand for such novel UX approaches to study media applications will only grow.

Conclusions
Based on our experiences with this case, we provide some takeaways below that may be useful to individuals and teams who may design, develop, or test games and hardware.
Focus on the specific contextual relationships between the product being tested, the intended use, and the specific player group(s) to fill in traditional testing measurement gaps.
• We used performative metrics during usability observations to reimagine satisfaction as enjoyability, which allowed us to measure whether or not the product was fun for users. Ultimately, our client used this feedback to determine where changes could be made to the product to make it more enjoyable for users with the next iteration.
Embrace after-the-fact observational, player-influenced metrics for enjoyability when testing gaming hardware, peripherals, and software.
• Performative metrics can be created based on players' experiences during testing, and researchers may discover surprising information that, though not initially considered for inclusion in a test plan, may provide insights that ultimately lead to better products. We suggest observing target users interacting with the product in an appropriate setting to produce new performative data that can help account for seemingly unaccountable components like enjoyability, which might be difficult to discuss with traditional metrics.
Perform small-scale usability tests on gaming and game-like products that can help designers and developers contribute to a larger disciplinary ecology.
• We embraced situational and systematic complexity by acknowledging potential test constraints, including the uniqueness of the enjoyability metric. Successfully testing new, innovative technologies like the device in our case requires willingness on the part of product testers to revise their test designs and metrics, sometimes significantly, after observing player interaction during pilot tests in order to avoid larger problems associated with nontraditional products, unexpected shifts in hardware, and evolving client goals.
One way to leave room for situationally specific constraints is to keep tests small and specific to the product itself. Conducting small-scale tests throughout development reinforces several of Tran and Biddle's (2009) key findings from their ethnographic study of a game development team, including team situational awareness, knowledge sharing, and collaboration.
Gaming and gamified systems will continue to grow in use in the coming years. To study how best to address more subjective metrics, UX practitioners and game designers must be creative and use situated, context-specific metrics as well as an ecological understanding of the various ways this new medium will continue to mutate, to evolve.