• Research

thesis

Towards a Framework for User Experience in Online Games: A Mixed Methods Approach


Date:
May 2017 - July 2018
Organization:
Doshisha University

This is a three-part series. This part covers the introduction, literature review and qualitative data analysis. The second will cover the quantitative data analysis, and the final part will look at limitations and recommendations.

Kindly note that this is a long read that includes some technical jargon. All in all, 120+ pages were summarized into a relatively light post.

Destiny 2
Different types of player classes in Destiny 2 (Source - PCGamesN)

it starts

How did I get to talk about user experience in online video games?

Well, originally I wanted to tackle the lack of customer lifecycle management in Mozambican commercial banks. However, banks in Mozambique are not really open and there’s a fair bit of espionage concerning product development and pricing. I did NOT want to be frustrated trying to get data all the way from Japan.

My second choice was how big data influences user experience. First in general digital products, and then in video games. I got a solid reality check after a workshop. Essentially, I needed to work with big data to make any headway…and no company was gonna allow me to look into their data.

I finally settled on a single focus: attempting to build a framework for user experience in online games.

Controller
(Source - BrainSINS)

why?

Video games have grown into a multibillion dollar industry, and this development can be attributed to growing disposable income in Asian and other emerging markets, as well as the proliferation of e-sports.

The industry has made a substantial contribution to creating best user experience practices. However, no such framework exists for either purely online games or those that have online play as a focus. This is in part due to it being contained mainly within game development houses themselves, and partly due to a lack of game usability & playability experts in academia and industry.

A good user experience is fundamental in expanding the audience and retaining them throughout the lifetime of a video game. Any aspect of a game that detracts from the user experience is likely to affect the developer’s goodwill, the brand strength, and impact profitability.

Larger studios and publishers often have enough financial resources to absorb any negative repercussions resulting from a poorly implemented user experience. Smaller development outfits are far more negatively impacted in such circumstances and may be forced to enter liquidation. To combat this, a framework could be developed that would provide an academic foundation for further research, and a guide for developers regardless of size and resources.

Thus, the goal of this study is to fill that gap by providing a foundation to build a framework that could drive better user experience in online games, and discussion in the academic world. This was done by looking at aspects of both user experience and player experience in pre-established frameworks, and completing the triangulation with qualitative and quantitative insights.

What are desirable and undesirable factors in the user experience of online games, and how can these better inform game design and development?

Over the past decade, several researchers have built user experience and player experience frameworks that have relied on both purely academic work and usability tests. Player experience occurs before, during and after a playing session. The player’s expected value, however, is dependent on the type of game being played, and on what platform. Related to this is Csikszentmihalyi’s concept of flow [as a mental state].

The concept of flow in games was subsequently tackled by Jenova Chen in 2007 in the eponymous body of work, by establishing that players have different flow zones depending on their experience level. Developers should aim to keep players ‘in the flow’ through a variety of adaptive choices and dynamic difficulty levels, in order for their [players’] actions to become reflexive regardless of the complexity level.

Journey
Jenova Chen directed the award-winning game Journey. The game's focus is on flow, specifically kinetic flow.

key frameworks

Seven frameworks have proved key in addressing the academic gap concerning and UX and PX in games. Please click through each tab to find out more, or just skip to the summary right after the tabs.

Towards a Framework of Player Experience Research

Nacke & Drachen, 2011

The first PX model is composed of three layers: technical experience, individual experience, and processed experience. There are primary elements to consider such as engagement, fun, action, challenge, attitude, and flow. And the secondary elements to be accounted for are the environment or physical properties of where the game is being played, considering how time plays a role before, during, and after play.

The notion of time as it relates to PX operates at all abstraction layers. It can be broken down as follows: past, present, and future. While time is discussed, the paper does not consider playing time being constrained due to changes in technical experience. These can range from server downtime to patching, to network outages. Moreover, PX is not a sum of all experiences, but rather the result of direct and indirect influences.

Every play session is a different experience due to several factors often out of the game designer’s control. For example, Player X might not care about brand recognition or peer opinions and thus has different expectations for the game. Player Y might obtain a AAA game for a low price, or borrow it from someone else, and might not be as absorbed due to little financial outlay.

Playability: Analysing User Experience in Video Games

Sánchez, et al., 2012, p. 1033–1054

The playability factor as a measure of user experience was then expanded upon in the Six Facets of Playability, which sought to improve playability by observing people’s preferences and adapting the game to them. Understanding and evaluating this aspect of user experience of games would increase the likelihood of [the game’s] success.

This resulted in a weighted sum of six facets: intrinsic, mechanical, interactive, artistic, personal, and social. The importance of each would vary depending on the genre of the game, and other aspects (i.e., platform of choice). The facets can be expressed mathematically, or graphically: , with the innermost circle being the most important.

Playability Framework


Using this framework, developers would be able to focus their energies and resources on accomplishing tasks as they related to each facet of playability in the order of their importance. Throughout development and post-release support, they [the developers] can also prioritize tasks such as game fixing based on the same weights.

However, the framework discussed has several limitations. It was only tested on single-player Nintendo Wii title. This, therefore, does not account for interpersonal playability or multiplayer (be it local or online). Secondly, the Nintendo Wii console has an unconventional control scheme compared to most other platforms in that it uses a motion sensing remote as its primary device of input. In general, players use a keyboard and mouse combination, a touchscreen, or a game controller. Since being taken off the market, no other gaming or gaming-enabled device has adopted a similar control scheme as a primary means of interacting with the game environment or other players.

Last but not least, the framework was only tested on university students under laboratory conditions. This may not be an accurate reflection of the general gamer population and the environment in which they play games. Nevertheless, it is an example of how a theoretical model has practical applications that benefit both the game and the player.

Assessing the Player Interaction Experiences Based Playability

Sánchez & Vela, 2014, pp. 259-267

The lead researcher of the framework covered in the previous subsection further expanded on the playability factor as a measurement of user experience and studied how to best improve it [playability] by observing people’s preferences and adapting the game to them.

The researchers then analyzed the behaviour of players while they engaged with a single-player game, which was still under development, over the course of a playing session. Telemetry was recorded during the session, with particular care paid to flashpoints within the game. To reinforce the data collected, interviews and questionnaires were then used to obtain the players’ views on the game played. This proved invaluable in reaffirming whether the telemetric data was correct or not and whether it satisfied players on various levels.

The results showed that the artistic facet was ranked the highest (an average of 3.84 on a scale of 5) compared to other facets. The social facet was ranked the lowest with an average of 2.49 on the same scale.

While this framework has its merits, its focus on a single-player game may not be directly applicable on games with a mostly or exclusive multiplayer focus.

Gaming Taxonomy: An Overview of Concepts and Evaluation Methods for Computer Gaming QOE

Möller, et al., 2013

This taxonomy aimed at pinpointing factors which could be used to gauge the quality of experience in a game.

The authors surmised that there are three sets of influencing factors on Quality of Experience (QoE). The first of which is entirely user dependent such as playing style or intrinsic motivation. The second set, system, asserts that genre, in-game rules and their possible circumvention, as well as technical characteristics are likely to affect the expectations of playing experience. Context, the final set, considers the physical location of the playing session, the social context for the game, and extrinsic motivators such as financial rewards and service factors which might affect game availability (costs, access, etc.)

From these three sets, several metrics were devised but not covered in any sufficient detail, and this is likely to due to its taxonomical nature. This also justifies why primary qualitative or quantitative data was not collected. However, this paper lays a strong foundation for future research work by considering factors that other work has only covered superficially or not at all.

Empirical Analysis of Playability vs. Usability in a Computer Game

Novick, et al., 2014, pp. 720-731

As opposed to building a framework, the authors opted to develop a game and analyze the pitfalls that came with designing the user experience and how test respondents perceived specific design choices. To assess this, they considered whether any issue experienced by testers could be attributed to a usability issue (i.e., achieving a task such as quitting the game), or a playability issue (i.e., dying without knowing the cause).

Thorough post-mortem analysis revealed that games are likely to have issues in both areas, instead of exclusively UX and PX. The researchers switched to the aforementioned facets of playability in a bid to better measure these issues, with mixed success. Using this as a foundation resulted in most episodes being classified as usability issues, and some facets not being accounted for due to lack of entries.

They concluded that only one technique allows for assessing both sets of issues: by testing the user and collecting feedback. Additionally, the foundations the tests were built on were likely insufficient in producing desired results or that classifying issues in UX and PX categories may not improve the user experience of a game. Finally, this paper acknowledges that this test was conducted on a single-player example with only a sample size of 4.

PLEXQ: Towards a Playful Experiences Questionnaire

Boberg, et al., 2015, pp. 381-391

The authors subscribe to playfulness being an important part of the user experience in a game. This can be likened to playing experience. A four-factor structure for playfulness was eventually developed with the use of a large item questionnaire. The questionnaire was developed by collecting data from a number of different sources including but not limited to: a review of existing academic literature, formal interactions with developers and players (though not simultaneously), and market reports on user experience in products.

Firstly, the development of this data collection tool seems flawed due to only interviewing 13 players who represented two game genres (with only one having a highly optional online component). It is unclear whether the experts who vetted each of the questionnaire’s items were aware of this or not, and if they proceeded even if aware.

Secondly, only 15% of those who accessed the questionnaire via an online link filled it to completion. This may have been due to it taking 20 to 30 minutes, and without any incentive at the end. Thirdly, all respondents were Nokia employees who might have had a vested interest in seeing the research have positive results.

Furthermore, the analysis was conducted using principal components analysis which does not account for pre-existing correlation in the loaded factors. This is key, as user experience can also be considered a social science, where factors generally have correlations: some visible and some latent.

Contextual Influences on Mobile Player Experience – A Game User Experience Model

Engl & Nacke, 2013, pp. 83-91

The rise of mobile devices as a platform for gaming attracted the attention of the researchers, who looked at the contextual influences that may occur whilst playing a mobile game. These games faced their own unique challenges. The first of which is the rise of asynchronous online experiences.

The other major challenge is unlike personal computers or dedicated game consoles, mobile devices can be used anytime and anywhere, leading to a host of potential distractions. The researchers deployed the Game Experience Questionnaire (IJsselsteijn, et al., 2013) to assess what could affect PX during all three phases of the play session (pre-, during-, and post-). The results showed that each component of the user or player experience is not completely independent, and in fact, interlinked with others.

The authors also assumed that hardcore gamers are unlikely to be influenced by developments in mobile games, but this might have to be reconsidered with cross-platform play becoming a more popular feature in different games since the time of this article’s publishing.

--

The frameworks highlighted above have been useful in approaching PX in a more systematic manner and have their own individual strengths. However, they have limitations in how they address online games, mainly due to:

  • A predominant focus on games of the single player or local multiplayer variety
  • Not accounting for online infrastructure affecting gameplay
  • Survey and/or testing on a single platform
  • Testing in laboratory-only conditions
  • Small sample sizes
  • Not considering scalability in games on different platforms

The reviewed papers establish a baseline for what a user experience and player experience are. What could be withdrawn from the above is that a one-size-fits-all approach is non-existent; however, a framework can provide guidance as to what primary and secondary factors should be considered in developing a good user experience for online games. It is unlikely that a game will include elements that please everyone, but it might have the minimum necessary to generate engagement, and later, retention.

Although surprises and insights associated with player’s interaction with a particular game are difficult to predict and quantify, they have a significant implicit effect on the overall value and quality of the perceived association of the service. Since such effects are difficult to quantify a priori, especially at the early stages of the project, no standardized or “best practice” methods of their assessment could be established and implemented. Nevertheless, the importance and sometimes critical effect of such surprises should not be underestimated.

In this context, some approaches should be foreseen in order to account possible impact of various user/player experience factors.

Mixed Methods Podcast
(Source - Mixed Methods podcast)

methodology

sequential exploratory mixed method

A quantitative and qualitative approach were used in conjunction to provide a complete knowledge base, and thus stronger conclusions through triangulation, convergence, and corroboration of findings.

The rationale underpinning this is pragmatism, as it allowed me to identify the best methods for answering the research question or questions at his or her discretion. In other words, doing what is necessary (under ethical considerations) to achieve the best possible result.

interviews

Further building on the aforementioned triangulation, semi-structured interviews were conducted based on ground theory and interpretative phenomenological analysis. This was picked due to allowing insights into a specific phenomenon (user experience) from the perspective of different practitioners. This approach opts for flexibly exploring the aforementioned phenomenon, rather than testing a hypothesis. Its selection was also because it stems from it being suited to open-ended research questions that warrant exploration. The approach’s roots in psychology make it a good fit with its close cousin: human-computer interaction.

Due to its inherent idiographic properties, it is generally recommended that a small sample size is selected to allow for richer analysis. The sample size can range from three (3) to fifteen (15), with six (6) being the minimum ideal number. The interviews were geographically spread around the world and their location was considered due to user experience design varying from region to region, though they are built from the same fundamental principles. In some cases, the participants had moved from their country of origin, further contributing to unique insights. They were selected using the purposeful sampling method, namely, expert sampling.

interviewees
The six interviewees were selected using expert sampling (Icon Source - Flaticon)

The sampling strategy does not intend to provide an all-inclusive or representative sample that might be prevalent in quantitative research that is substantially larger in scope. In its place, only enough contributors were sought to provide relevant insights into the incipient emphasis of study.

The developers/designers approached either are freelancers or work in industries where user experience is highly emphasized. They [the participants] did not receive any compensation or incentive to be interviewed and expressed enthusiasm in being able to discuss a fundamental aspect of their professional and personal lives, as well as contribute to the research.

The respondents generally defined user experience as a whole experience that starts from the moment the customer first hears about that product, to the last time that product is used. A respondent dubbed this as the journey, which is, a measure of two parts; aesthetics and interactivity.”“ Producing the journey is a combination of tools, methods, designs, and principles aimed at creating a new user experience or augmenting an old one. It is up to the UX practitioner to devise the best way, which tends to come from a mix of existing frameworks and experience, of making that product approachable and that it meets the customer’s minimum expectations.

Sometimes, defining the journey is difficult due to business requirements. Is the game targerted at new players or just ones? How much of it relies on muscle memory from previous games in the genre or gaming as a whole? Some might even be based on real-life sports which makes the transition easier.

Super Mario
Super Mario Odyssey on Nintendo Switch (Source - GIPHY)
Gran Turismo Sport
Gran Turismo Sport on Sony PlayStation 4 (Source - GIPHY).

Usability in games is seen similarly to web design, but the vocabulary often means distinctive things. One respondent used the term “engageability” as an all-encompassing term that included fun, engagement, immersion, and retention: key factors in a game having a good user experience. Players fall into a good player experience, but they are not necessarily aware of it. Their perception of it varies from region to region, and many games or apps are not successful outside their home market.

One respondent cited Clash of Clans/Clash Royale as a game that transcended cognitive bias in user experience, “It's kind of nice to see that working quite well whereas Japanese is very-- the grid dimension is very heavy text based and lots of levels, lots of explosions and feedback.”

Text-heavy digital experiences are a common occurrence in Japanese applications and games and are rooted in chirashi. The Japanese usually require a significant amount of information before making decisions, though this contrasts the user benefits first message that pervades Western design. Concerning the Western or Western-influenced mindset, another respondent said that, “We've essentially raised an entire generation of people on minimalist design and they basically accepted it as the normality.” As such, designing a user experience that accounts for different preferences becomes a monumental task due to being resource intensive. Due to this, only mid-size and large developers can afford to conduct the user research necessary to generate global appeal.

Measuring can take many forms such traditional research tools (interviews and questionnaires) but typically hinges on usability tests, and in recent years: biometrics. The aim of these was to determine what is meaningful for players, and what are they trying to accomplish. Important tenets of the self-determination theory, were considered important factors for motivating players by one respondent: “So, this is what I have been developing with a lot of other researchers; it’s mainly around motivation and you know, SDT. But also around rewards.”

“We've essentially raised an entire generation of people on minimalist design and they basically accepted it as the normality.”

As for who is involved in measurement, one respondent asks if the stakeholders are pleased with the current user experience/journey or not. These include but are not limited to the financial investor, the project manager, the lead designer and the players. As a game’s scale grows so do other development concerns that can adversely affect the user experience. This is even more of a matter of interest with online games, due to stress testing the game under a variety of scenarios that simulate real-life conditions such as bandwidth, the number of concurrent players, etc.

League of Legends
Riot Games' immensely popular League of Legends is an example of an online game with significant measurement. (Source - Vortex Cloud Gaming)

themes



The coding of the interviews' (transcripts) led to the emergence of six major themes. I opted to leave the coding words I used off the page as it would cause havoc on those reading on mobile devices For those who are interested, please find the codes here. These central themes are hypothesized to have a substantial impact on the user experience of an online game. It is important to note that all of these themes (or factors) are interconnected to various degrees. The themes are:



Customization

Being able to customize one’s avatar added a sense of ownership and belonging to the player.

Exploration

Players will delve into worlds either for story reasons or to discover new gameplay mechanics.

Post-Release Support

Any content that is added after the first public release in order for the game to generate continuous user engagement.

Accessibility

Ensuring that players can adjust the game to suit their skill levels and/or preferences.

Competitive Play

Which is significant driver of online communities and e-sports.

Social

Where a digital-first or digital-preferred approach has created new ways of socializing that did not exist 20-30 years ago.

Similar to the key frameworks section, this section talks about each theme in some detail. Please click through each tab to find out more. There is no summary here.

Customization

As above, customization of the player character provides players with a sense of ownership and/or belonging. According to one respondent, this “plays a pretty big part in the way people feel about the products.” Developers caught wind of this and started locking customization options behind microtransactions as a way to complement main revenue streams. Other developers provide promotional customization options during the festive period or if they log in during a certain time of the month. Though the respondent admitted to not knowing the psychological effects of these promotional items on players.

Final Fantasy XIV
Customization in Final Fantasy XIV (Source - ThePinkProxy on YouTube)

Exploration

The effect of exploration on online games is twofold. First, it teaches the player about new skills or re-contextualizes old skills in new environments keeping the player engaged and likely to contribute to mastery of the available skills. Secondly, it allows developers to create content in which players can fully immerse themselves in. This plus sometimes reduces the likelihood of players having a similar experience to almost zero if not zero. The unique experiences feed into the social part in the real world, where players can trade different perspectives on events that use the same foundations.

Another respondent cited Minecraft’s influence in expanding the open-world or sandbox genre and popularizing crafting as being substantial. The game is also highly accessible to those with little to no game experience, allowing players to collaborate towards a variety of single or overlapping objectives. The removal of arbitrary limitations, by game standards, contributed to spurring creativity and has proved popular with critics and the general audience alike. Another respondent agreed and supplemented this by affirming that game design has become more systematic and procedural, and that everyone has different experiences in the same game. To classify it as merely as a mechanic, is a misnomer.

Minecraft
Minecraft is available on every modern gaming platform (Source - CTP Anima on YouTube)

Post-Release Support

More developers are conscious of the fact that all the factors that make up a user (or player) experience should harmonize and collectively deliver a good game experience. These points of contact, or touchpoints, range beyond the product or service delivered and might include a public developer roadmap, an active community manager, and timely patches that address issues regardless of size. An integrated user experience approach ensures that all of the moving parts are designed in a way that bests communicate the intent of the designer, and builds towards a positive and consistent user experience.

The community plays a part after games are released, in providing praise and criticism to several facets that make up a game’s user experience. But, in the case of competitive play, the developers have to be careful to not comprise on the balance of the entire game to a vocal minority. Sometimes, reacting to feedback is difficult as players are not able to vocalize their thoughts and feelings about a specific part of the game, due to players lacking “the language and introspection that is required to truly interpret”, one respondent said.

APEX Legends
Apex Legends' roadmap for the first year (Source - Apex Legends)

Accessibility

Accessibility was a recurring topic amongst all six interviewees, with several stating the need to embrace the potential to have a variety of users. One was adamant about the gaming environment, indicating that, “games that can cater for real time or asynchronous communities of remote players, battling, sharing, from anywhere to anywhere.” This ensures that games can contextually account for forgetfulness, interruptions, and impairment.

Both productivity applications and games tended to be tested in environments with laboratory-like conditions (i.e., no distractions, dividers). Due to this, developers, in general, do not account for disturbances, forgetfulness or poor connectivity. Many online games are not asynchronous, and thus require a stable internet connection. Though mobile games tend to be so, as they're mostly passive experiences.

Three interviewees used the example of Fortnite as games that have similar experiences across both sets of platforms or cross-platform play. The mobile variant is usually a stripped down (in terms of graphics and/or features), and more refined experience of PC/console title and also much more approachable. One respondent admitted that “it's more difficult to design a UI across multiple platforms.” Another oticed a trend towards scalability which is slowly entering mainstream consciousness. Customers are aware of the limitations of each platform and will adjust their expectations accordingly.

There is a lot more that was said about accessibility, and perhaps I'll dedicate one post to it in the future.

Fortnite
The smash hit Fornite (Source - The Verge)

Competitive Play

Competitive play was referenced, much more than collaborative play, but not directly. This covered mostly flow and motivation. One respondent, who conducts user research for game developers, noticed that players regularly try to break the rules in order to improve cognitive ability and dexterity.

Some gamers are capable of ignoring distractions, or at the very least, reduce the probability of being distracted regardless of environment. One respondent called called this game flow, another concept which was mentioned above. Though game flow does not always thrive in situations where there are not any distractions. During a competition, hearing a crowd cheer one on could spur them to perform better under a high-stress situation. A situation not dissimilar to those who play physical sports professionally.

Street Fighter V
Street Fighter V (Source - sensession on YouTube)

Social

The social elements of online games also play a role, with players opting to collaborate or compete. Mixed in-between are aspects such as trading, in-game mentorship, etc. One respondent shared an experience with World of Warcraft, in which they asked another player how they obtained a pet, “It's like it's almost like you see someone walking on the street with an ice cream, you're like where'd you get that ice cream, and you go into that store and check it out. It’s like word of mouth is in that online space, it's like living in another world, World of Warcraft.”

Red Dead Redemption 2
Red Dead Redemption's online mode allows players to form outlaw gangs (Source - Engadget)

--

All six interviewees expressed that even a small amount of UX knowledge can have a significant impact on the output of a game: and that can be the difference between a successful game or not (in monetary terms). However, there are still a number of obstacles to overcome:

  1. The discrepancy between business goals and user goals. Specifically, the financial metrics or scope creep becoming the most crucial factor in determining user experience.

  2. Developers struggle with going continuous user engagement and going against convention, especially in pursuit of new ideas for a genre or sub-genre of gaming. Since there is no pre-existing experience or knowledge, testing becomes tantamount to ensure that the developers’ ideas are transferred in the way that they are intended to. Engagement becomes a balancing act between attempting methods that will hook players in, and hopefully keep them there, as well as being socially responsible and not creating an addictive game loop.

  3. The future of these long-running games might be affected due to shorter attention spans, and the popularization of instant gratification by notifications in social networking services and the dopamine rush they provide.
Stats
(Source - Smile Software)

next time

I apologize for the text dump, but it could have been MUCH worse. I hope you enjoyed reading this. The next part will focus on the quantative part of the study.

Ed's Logo

© 2020 Edson Chilundo