As the software industry has matured, the demands that society puts on the quality of software systems have increased. It is no longer enough to focus only on the many functions that a piece of software should supply. To deliver a system that is consistent and of high quality there are a large number of characteristics that need to be considered (Chung et al., 2000). Some, such as testability, are internal or relate to the development process and mainly concern developers, while others such as performance and usability, are critical for users (International Organisation for Standardisation (ISO), 2011). At an even broader level, the actual experience of the end users as they interact with the software needs to be taken into account.
Recently, this widening scope of software quality characteristics has led to the introduction and study of the concept of user experience (UX). Even though different definitions of UX exist, they share the same essence: UX is a user’s holistic perception of functionality and quality characteristics of a piece of software (Hassenzahl, 2003; Wright & McCarthy, 2010; Jordan, 2002). In general, UX literature emphasizes that assuring efficiency and effectiveness during use of the software, i.e high usability, does not guarantee that the end users will have a positive experience (Hassenzahl, 2010). However, the perception of UX is generally different in academic and industrial contexts: whereas the former concentrates on hedonic aspects and emotions, the latter focuses more on functionality and usability issues (Väänänen-Vainio-Mattila, Roto & Hassenzahl, 2008).
Current UX models (e.g., Hassenzahl, 2003; Wright & McCarthy, 2010) differ in their view on how various underlying elements and processes contribute to forming the end user’s overall experience with products and services. One of the well-known UX models is developed by Hassenzahl (Hassenzahl, 2003). It breaks UX down into pragmatic and hedonic attributes. Pragmatic attributes concern usability and functionality of software (e.g., clear, supporting, useful and controllable). Hedonic attributes, on the other hand, concern communicating identity, provoking memories, and providing stimulation (e.g., outstanding, impressive, exciting, and interesting). These attributes emphasize individuals’ psychological well-being. An end user’s perception of these attributes leads to a judgment about the product’s appeal (e.g., “It is good/bad”), emotional consequences (e.g., pleasure, satisfaction) and behavioral consequences (e.g., increased time spent with the product). While pragmatic attributes concern achieving do-goals, hedonic attributes concern satisfying be-goals (Hassenzahl, 2003). Do-goals are the concrete outcome that the end user wishes to achieve whereas be-goals rest in essential human needs. To provide a better understanding of UX, Hassenzahl (Hassenzahl, 2010) emphasizes five characteristics of UX: subjective, holistic, dynamic, context-dependent and worthwhile.
All software systems deliver some UX, positive or not, whether the UX has explicitly been taken into account during development. Research has shown that certain practices can increase the likelihood of delivering a desirable UX (Hassenzahl, 2010) (hereafter, UX practices). However, simply applying these practices in isolation is not enough (Abrahão et al., 2010; Ferreira, Sharp & Robinson, 2012; Ovad & Larsen, 2015). Like methods and practices used to support other software quality characteristics (Chung et al., 2000), they need to be integrated into development processes and considered throughout projects. Nevertheless, UX is often neglected in software projects and the UX state-of-practice is immature (Abrahão et al., 2010; Isomursu et al., 2012; Law & Abrahão, 2014). In order to improve the state-of-practice, we must first understand and address challenges that practitioners face in their everyday work with UX (hereafter UX challenges). To achieve this, the distinctions and interrelations between UX, usability and other software quality characteristics must first be properly analyzed. A number of studies claim to have studied UX challenges, but their results need to be interpreted with caution because they treat UX and usability as interchangeable (e.g., Cajander, Lárusdóttir & Gulliksen, 2013; Ardito et al., 2014; Lanzilotti et al., 2015), or examine UX in isolation so its relation to usability and other software quality characteristics is not recognized (e.g., Isomursu et al., 2012; Vermeeren et al., 2010).
Our study aims to complement the current body of knowledge by investigating UX challenges while explicitly differentiating UX and usability. In doing so, we discuss these challenges in relation to the unique characteristics of UX. We also contribute to the current literature by providing an explanation for the industry’s lopsided focus on the pragmatic aspect of UX. Also, in our analysis, we examine how the handling of UX compares to the handling of general software quality characteristics, especially usability. Here, we report our findings and answer the following research questions: what challenges do practitioners face when integrating UX practices into software development processes and organizations?, and how do UX challenges relate to challenges of handling software quality characteristics, in particular usability? Our findings can be useful for researchers in identifying new and industry-relevant research areas, and for practitioners in learning from empirically investigated UX challenges, and basing their improvement efforts on such knowledge.
The structure of this paper is as follows: The second section provides background on the topic and summarizes the related literature. The third section describes our research methodology and presents the different research sites. The fourth section presents the results from our study: the identified themes of challenges. The fifth section discusses our findings and puts them in relation to the current literature. The last section concludes our study and suggests future research.
Background and Related Work
This section elaborates on the characteristics of UX and their implications for UX work. Understanding these characteristics is important when analyzing and discussing our findings on UX challenges because it provides a basis for comparing and contrasting UX with usability and other software quality characteristics. This section also summarizes related empirical studies on software quality characteristics, in particular, usability and UX.
UX is dominated by subjective aspects of human perception (Wright & McCarthy, 2010; Hassenzahl, 2010). For instance, one user may perceive particular software features as simple, novel, and admirable while another may perceive them as complicated and old. Other software quality characteristics can be treated more objectively, at least in theory (Kashfi et al., 2016). The term subjective has also been used in requirements literature to highlight that since quality characteristics are hard to measure, practitioners tend to judge them based on their personal opinion (Chung et al., 2000; Paech & Kerlow, 2004). Here, we use this term to refer to its meaning in the context of UX. Currently, the most efficient and feasible approach to measure perception and emotion of users is to directly gather their opinions, and let them express themselves (Law, Van Schaik & Roto, 2014). This is often performed through questionnaires or scales (e.g., AttrakDiff, Self-Assessment Manikin, Affect Gird Zimmermann, 2008). However, to gain reliable results, practitioners need to gather responses from statistically significant number of heterogeneous users (Law, Van Schaik & Roto, 2014).
UX is also holistic and not totally reducible to its complexly intertwined underlying elements (Hassenzahl, 2010). UX emerges from underlying functionality and quality characteristics of software, and the user’s perception of them in a given situation (Hassenzahl, 2003). The holistic nature of UX resembles the cross-cutting nature of other quality characteristics. In both cases, more than one functionality is affected by the related requirements. However, in the case of UX, this interrelation is more complex, especially considering that UX is dynamic (aka temporal) and thus a user’s perception of the UX underlying elements changes over time (Hassenzahl & Tractinsky, 2006).
Although practitioners may manipulate UX of a product through its underlying elements, they cannot guarantee a certain holistic UX (Wright & McCarthy, 2010; Hassenzahl, 2010). Practitioners can only to some extent increase the likelihood of delivering a certain UX (Hassenzahl & Tractinsky, 2006). Consequently, there is no consensus in the community on how UX should be designed or measured and whether the overall UX of a piece of software can be predicted by merely manipulating and measuring these elements (Law, Van Schaik & Roto, 2014). This is explained further below.
UX is dynamic and emerges and changes over time (Hassenzahl, 2010). For example, over time, the user may find a novel feature as old, or a complex feature as simple. Hence, in designing and evaluating UX, practitioners should pay certain attention to different episodes of experience (Hassenzahl, 2010); namely expected experience (before usage), momentary experience (during usage), remembered experience (shortly after usage) and accumulated experience (over longer period of use) (Hassenzahl, 2010). Practitioners need to decide which episodes are more important than others for the software being developed and why; for instance, for an e-marketing website first impression is more important than it is for a work application. This knowledge can then help them suggest more suitable design solutions.
UX research uses the term context-dependent (aka situated) to emphasize that any experience is unique, unrepeatable, and situated in its context (Wright & McCarthy, 2010). Nevertheless, experiences can be categorised because their essence is the same, i.e., they connect to essential human needs or be-goals (Hassenzahl, 2010). Implications of being context-dependent largely overlap the implications we discussed for the holistic nature of UX.
UX is worthwhile (aka positive), meaning that it focuses on value and creating desirable experiences than only preventing negative ones, i.e., the focus of usability (Hassenzahl, 2010). While usability concerns removing problem, frustration, and stress (i.e., negative), UX is based on the idea that removal of dissatisfaction does not necessarily lead to satisfaction and pleasure. Through being holistic, UX addresses both satisfiers (e.g., fulfilled needs, emerging emotions) and dissatisfiers (e.g., usability or technical problems) (Hassenzahl, 2010). Therefore to a large extent, the implications of being worthwhile overlap the implications of its holistic nature.
Software organizations often face various challenges when dealing with quality characteristics. Berntsson Svensson et al. (2012) found that if practitioners lack understanding and knowledge about software quality characteristics, they tend to undervalue, and ignore these characteristics during development. Berntsson Svensson et al. (2012) also report that practitioners are more likely to dismiss those characteristics that are considered less important, e.g., usability is more dismissed than performance. Karlsson et al. (2007) show that functional issues are often perceived to be more important than quality issues. Based on their findings, practitioners find it difficult to deal with the dependencies between quality requirements and the functionality or other characteristics of the system.
Limited knowledge and awareness has also been reported as one of the challenges to usability work (Rosenbaum, Rohn & Humburg, 2000). Bak et al. (2008) report that developers’ minds are set more on the programming aspect of the product than its usability, and they often misunderstand the term “usability evaluation” and often relate it to functionality. Gulliksen et al. (2004) report that limited awareness in different levels of organizations, in particular at the management level, can lead to down-prioritizing usability. They therefore suggest increasing knowledge and awareness about usability among different stakeholders and in various levels of organizations, in particular among management.
Other studies show that, in general, practitioners find it more difficult to perform requirements and testing activities for quality characteristics than for functionality (Paech & Kerlow, 2004; Berntsson Svensson et al., 2012; Chung & do Prado Leite, 2009). For instance, it is generally more difficult to document quality requirements in a measurable way, or handle their dependencies to functional requirements (Berntsson Svensson et al., 2012; Karlsson et al., 2007; Borg et al., 2003; Nuseibeh & Easterbrook, 2000). Borg et al. (2003) report that lack of competencies to handle quality characteristics in projects often leads to ignoring related requirement in projects. Even if these requirements are properly quantified (specified in a measurable manner), they require suitable competencies to be tested and verified. Similar problems have also been discussed in the usability literature. Practitioners find it challenging to document measurable usability requirements (Cajander, Lárusdóttir & Gulliksen, 2013; Gulliksen et al., 2004). Ardito et al. (2014) showed that if practitioners fail to include usability in requirements documents, these requirements might be ignored later in testing.
The limited access to competencies and unclear responsibilities are also reported as challenges to better usability work. For instance, Rosenbaum, Rohn & Humburg (2000) report that lack of usability professionals is one of the main obstacles organizations face concerning usability. Boivie, Gulliksen & Göransson (2006) show that even in cases when organizations have access to the right expertise, usability professionals are not sure about their responsibilities, and are uncertain as to how to contribute to the projects. Chamberlain, Sharp & Maiden (2006) report that power struggles rise as designers within a project defend their discipline in response to the decisions made by developers, and vice versa.
Despite the differentiation of UX and usability, only a limited number of empirical studies have so far analyzed the implications of these differences. One example is the study by Vermeeren et al. (2010) that compares the challenges of evaluating usability and UX. They argue that some of UX evaluation methods need to be further improved and developed for better use in practice. According to Vermeeren et al., there is still a lack of suitable methods for evaluating UX in earlier development phases or in the period before actual use (i.e., anticipated use). They also highlight that current methods are not often practical because they need special expertise, are time consuming, or their data analysis is hard. Similarly, Isomursu et al. (2012) discuss that practitioners face difficulties in UX work compared to usability work because they do not have access to tools and methods to objectively measure UX.
Law, Van Schaik & Roto (2014) explore the basic question of whether UX elements are measurable. They report that their interviewees expressed skepticism and ambivalence towards specific UX measures even if attitudes were more positive overall. They note that practitioners show opposing views on whether UX can or should be divided into composing elements, or whether it needs to be considered or measured as a whole. Results from their interviews show three categories of challenges concerning the interplay between UX evaluation and software development: (i) theoretical (measuring UX holistically or in elements, and conceptualizing long-lasting versus momentary experience), (ii) methodological (differing preferences for quantitative versus qualitative data by design- and engineering-oriented stakeholders), and (iii) practical (lack of knowledge and competencies for interpreting measurement outcomes). The participants in the Law et al.’s study emphasized they need to use a variety of media (e.g., video, TV, social media) to develop the required prototypes for measuring UX and that they often even need more than one such prototype to gather enough input for design. Their practitioners also argued that UX measures are essentially prone to fading and fabrication, or that there is a lack of means to measure the exact emotion of users at each moment (Law, Van Schaik & Roto, 2014).
The work of Law, Van Schaik & Roto (2014) is duplicated in the context of the Latin American software development industry by Gerea & Herskovic (2007). They conclude that practical aspects such as cost and time play a more important role in whether or not practitioners measure UX. Other challenges reported by Gerea et al. include: limited access to the end users, and lack of knowledge and experience in UX measurement similar to what Vermeeren et al. (2010) found.
Alves, Valente & Nunes (2014) investigated how UX evaluation is performed in practice (i.e., by whom, in what phases of software development, and using what tools and methods). According to their data, in around 50% of the cases UX evaluation is performed without involving the end users, and often evaluators ‘assume’ what the perception of users will be. Alves, Valente & Nunes (2014) also report that sometimes evaluations are performed by software developers that do not necessarily have the required competencies. Also, often tools and methods are selected based on cost rather than suitability for the project. Alves el al. used a list of evaluation tools and methods that are mainly usability-specific and lack UX-specific methods (e.g., Attrakdiff questionnaire (Hassenzahl, Burmester & Koller, 2003)1 ). This can introduce a risk to their data because practitioners might have preferred the use of a generic method such as a questionnaire for evaluating UX, without necessarily acknowledging that such generic methods may purely produce usability-related data if not used with specific attention to UX-related concepts, such as the non-tasks-related aspect of use (Hassenzahl, Diefenbach & Göritz, 2010).
Studies also show that the perception of UX is generally different in academic and industrial contexts. Väänänen-Vainio-Mattila, Roto & Hassenzahl (2008) report that practitioners still focus more on functionality and usability issues in their UX work. Similarly, in Kuusinen & Väänänen-Vainio-Mattila’s (2012) study in a large software organization, ease of use and efficiency were the most often reported sources of good UX. Hence, these studies conclude that while academia concentrates more on the hedonic aspect, the industry focuses more on the pragmatic aspect.
As we saw, those studies that explicitly take differences between UX and usability into account, mainly focus on evaluation activities and the role of UX measures in challenges practitioners face. They therefore do not sufficiently discuss other aspects of UX work, for instance requirements or communication and collaboration between UX and non-UX practitioners.2 In addition, those studies that report software organizations often focus more on usability than UX rarely provide an explanation for this lopsided focus. Hence, there is a need for further empirical studies that investigate the role of UX characteristics and differentiate UX from other software quality characteristics including usability. Our study is a response to this research gap.
We have conducted an explorative, qualitative study (Braun & Clarke, 2006) to investigate the challenges to integrating UX practices into software development processes. Below, we detail our research approach by describing the different companies where we conducted interviews, how the data was collected, and our approach to data analysis.
We selected a variety of companies with different characteristics for our study in order to improve the generalizability of our findings (Runeson & Höst, 2008). Table 1 shows an overview of these companies and their main characteristics and labels them (A–H) for easier reference later in the paper.
|Company type||Number of employees|
|(A) IT services, consultancy & outsourcing||>130,000|
|(B) Defense & security||∼14,000|
|(C) Wireless solutions & network testing||∼1,800|
|(D) Systems & software development company||∼120|
|(E) User experience consultancy & training||∼50|
|(F) IT & management consultancy||>1,300|
|(G) User experience & Usability consultancy||34|
The companies span various domains (company type) and vary in size (number of employees). We approached both consultancy and product development companies in order to cover both perspectives. Both A and E are well-known consultancy companies in Sweden, while B, C, D, and H are well-known product development companies in Sweden. Throughout the study, we were introduced to additional companies by our interviewees, and we also included a number of interviewees from those companies (F and G). Company F is an international consultancy company, also active in Sweden. Company G is an American company. Only one of the companies (E) was previously known to the authors based on previous research collaborations.
The aim of the interviews was presented to our main contacts in each company to assure selected interviewees were suitable for our research. The selected practitioners (see Table 2) represent technical (e.g., developers), design (e.g., interaction designers), and management roles. We had the option to ask for more interviewees, but since the study was explorative, after 17 interviews we were confident that we had covered the major challenges from a sufficiently broad range of perspectives.
|Code||Roles & responsibilities||Field of education||Industrial experience (since)|
|A-1||Business developer, UX lead & interaction designer||Cartoonist||2010|
|A-2||Interaction designer & UX designer||Computer science & interaction design||2005|
|A-3||Project leader & developer||Computer science||2006|
|A-4||Art director & graphical designer||Photography||1996|
|B-1||Software engineer & project manager||Psychology & electrical engineering||2007|
|B-2||Technical product manager||Computer science||2000|
|B-3||Senior project manager||Computer science||1995|
|B-4||Product manager & project manager||Computer science||2006|
|B-5||Technical project manager & requirements analyst||Computer science||2002|
|B-6||System developer & interaction designer||Computer science & interaction design||2011|
|C-1||UX lead & interaction designer||Cognitive science||1996|
|D-1||System developer & requirements analyst||Computer science||2000|
|E-1||Business developer, UX lead & interaction designer||Software engineering||1983|
|F-1||UX lead & human factors specialist||Cognitive science||1989|
|G-1||Manager, design strategist & UX lead||Design||1999|
|H-1||Manager, design strategist & UX designer||Multimedia & interaction design||1997|
|H-2||UX designer||Multimdeia & design||1996|
We conducted semi-structured interviews (Flick, 2009) to collect more of the interviewees’ viewpoints, which was important to the explorative nature of our study. The questions covered different phases of development processes, and also the concept of UX and how it differed from usability in the interviewee’s eye.
We prepared an interview guide with a set of pre-designed questions based on the knowledge gained from literature. In each interview, questions were rephrased, added, or skipped based on the interviewee’s background and responses. Each interview covered all phases of software development, the activities performed in each phase, and the tools and methods applied. Thirteen of the interviews were conducted face-to-face, and four via video or telephone conference. Each interview lasted between 30–60 min, and was recorded and transcribed. The interviews were all performed in the spring of 2012.
Figure 1 provides an overview of the data gathering and analysis process. We applied thematic analysis as described in Braun and Clarke (Braun & Clarke, 2006) to encode and analyze the interview data.
We segmented the interview transcriptions into meaningful paragraphs or sentences in a way that each of these segments presented one concept. We then coded these segments (Braun & Clarke, 2006). We used Microsoft Excel to document the coded data. Each interview transcript was recorded in a separate sheet. Segments of this transcript was recorded in separate rows, and different codes were assigned to each segment in separated columns, following that segment. After coding, categories of challenges, i.e., themes, emerged as we put together similar concepts presented in the coded segments. A mind-map of challenges and categories was created to better identify and visualize the relations.
We created a pilot list of codes (Braun & Clarke, 2006) that was iteratively refined. The initial codes captured challenges and their context and included: definition, challenges, solutions, tools and methods, evaluations, requirements, UX versus usability, and activities.
As the interviewees mentioned various challenges, we asked follow-up questions to better understand attempted solutions, or previous experiences with usability or other quality characteristics in relation to similar challenges. When the interviewees used different terminologies, or had limited knowledge concerning UX or usability, we mapped their statements to the relevant concepts based on the definitions of the concepts in literature.
For instance this statement: “There is a bit of confusion in the field and in the company as well, what’s the difference? design is design” (A-1) mapped to these key points: definition of UX is not clear, and practitioners do not know what UX exactly means. This statement was then coded as ‘challenge’, and ‘definition’. Segments that related to understanding and definition of UX, such as this example, resulted in the theme ‘lack of consensus on definition and construct of UX’.
Threats to validity
Threats to validity are outlined and discussed based on the classification by Runeson and Höst (Runeson & Höst, 2008): Selection process of subjects for interviews can cause a threat to construct validity. Selection bias is always present when subjects are not fully randomly sampled. However, our subjects were selected based on their role, experience and availability so there is little more we could do to alleviate this threat. The presence of a researcher may influence the behavior and response of the subjects. This threat was alleviated somewhat by the guarantee of confidentiality of the data but is an inherent limitation of the research method used.
In any empirical study, incorrect data is a threat to internal validity. Interviews were audio recorded to mitigate this threat. The authors also analyzed the material in several rounds of independent as well as joint sessions to gradually reach consensus on the intended meaning of the respondents. We also shared the results of our analysis with the interviewees to validate and confirm the findings.
External validity concerns the ability to generalize the results beyond the actual study. Since the interviews are just a sample from a potentially very large population, they should be interpreted with some caution. We sampled a number of different organizations in different industrial domains to decrease the effect of this threat. In addition, the majority of the organizations we studied are Swedish (exceptions are G (American), and F (Dutch)), and the culture of Swedish software industry can have an impact on how the studied organizations perceive and address UX or other quality characteristics. This distribution however is not sufficient to draw any conclusions based on the cultural differences of these companies, and in interpretation of findings this matter needs to be taken into consideration. Qualitative studies rarely attempt to generalize beyond the actual setting and are more concerned with explaining and understanding the phenomena under study.
Another concern is that our data gathering was performed in the spring of 2012. Therefore, our data may not reflect today’s UX state-of-practice in the studied organizations. However, the data is valid when interpreted in its own time frame. Also, to minimize the effect of the time frame on our analysis, we have included recent studies published since 2012 when discussing the results.
Another threat concerns reliability, the extent to which the data and analysis are dependent on the specific researchers. Although the coding process is performed by the first author, to improve reliability of the generated themes, the three authors individually and independently conducted a pilot coding of these segments using an initial coding guide as explained above. The outcomes of the pilot coding were discussed in several sessions with all three authors, and the differences in coding were analyzed and resolved. Also, we had carefully designed the interviews before running them. We also defined the coding process after the interviews and before analyzing the data. The initial codes were therefore identified mainly based on observed interview responses. We also ensured the themes are not imposed on the data rather emerged from it.
This section presents eight themes of challenges that emerged from the interview data (see Fig. 2). These themes are described and supported by the interviewees’ quotations. Each quotation is provided with the interviewee’s code, and labeled either ‘SE’ or ‘HCI’ to reflect the community to which the interviewee belongs. This helps better understanding of the current state of practice from these two perspectives.
Theme 1. Lack of consensus on definition and construct of UX
Our data shows that practitioners’ understanding of the concept of UX differs and is, in some cases, even inconsistent and contradictory (see Fig. 3). According to the respondents, UX is a new concept and there is a lack of general agreement on its meaning in the field in general, and among practitioners within organizations in particular. In some practitioners’ view, UX is the same as usability or Interaction Design (IxD) despite the fact that UX includes the user’s subjective perception and overall experience. The lopsided focus on usability and IxD can be explained by their relative simplicity: “I think discussions at large when it comes to UX design at common ground is still about IxD and usability. Usability is easy to talk about and everybody understands it.” (A-1/HCI).
One of the interviewees referred to UX as “just another buzz word” (E-1/HCI). In her view, UX contains the same concepts that have been around for a long time under other names such as usability and emotional design. On the other hand, some practitioners explicitly differentiated between UX and usability. One of the participants emphasized that usability is “a minor subset of UX” and added: “I’ve never, ever called myself a usability expert, and I would never do that.” (H-1/HCI). Similarly, another practitioner stressed that UX is not about IxD and has a much broader perspective: “[IxD] is just the end results, we do not call ourselves interaction designers, because that is only 10% of our work but that is an important element because that is the most visual part of our work .” (F-1/HCI). Similarly, another practitioner referred to UX as: “a wholeness with the emotional, social, economical, functional, and technical parts.” (A-4/HCI). Another practitioner described UX as: “pretty much everything that affects a user’s interaction with a product.” (H-2/HCI). He further emphasized UX is “the whole package” and usability is only one part of it. A number of practitioners stated that UX is a ‘broad’ and ‘holistic’ concept covering not only the user perspective, but also the business perspective. While the latter looks into how the design contributes to achieving business goals, the former assures satisfying the end users’ needs. The user perspective includes aspects such as ‘emotions’, ‘values’ and ‘preferences’.
Some practitioners related UX to the ‘why’ behind the functional requirements and the software in general. For instance, one practitioner described this through three main questions of ‘what’ to build, ‘how’ to build it, and ‘why’ to build it (A-4/HCI). Some practitioners related UX to the GUI by stating that it is about “the cool things, the new things, the flashy things.” (A-3/SE). In their view, UX is mainly about emotions and aesthetics therefore not applicable to all types of applications, for instance ‘productivity applications’.
In general, the practitioners with technical backgrounds showed limited knowledge about UX. Their knowledge was often limited to cognitive aspects of design, i.e., usability. One of the technical managers stated: “ [our customers] talk about increased workload. That is a negative thing. I don’t know if that qualifies as UX.” (B-2/SE). On the other hand, a UX practitioners emphasized that UX “goes beyond cognitive aspects of design” (E-1/HCI), the main focus of usability.
The participants also discussed that customers’ limited knowledge about UX is a challenge. Customers who have heard about UX can be too ambitious regarding emotional and non-task-related user needs. Customers’ limited knowledge also means that they often specify the related requirements vaguely and using inconsistent and subjective terminology. They often indicate a need for quality characteristics such as ‘cool’, ‘fun’, or ‘high-tech’ mostly because they are affected by such ‘buzz words’ while they do not necessarily know what these terms actually mean. Practitioners emphasized that to prevent misunderstandings UX-related requirements should be refined early on to concrete requirements and specified in a measurable way. Regarding this one of the interviewees said: “usually they say ‘we want something like that app’, ‘we want it to be cool and high tech’. Then you have to initiate a dialog to find out what that means for this particular customer.” (A-1/HCI).
Theme 2. Lack of consensus on the value of UX
Generally speaking, our data shows that various stakeholders have different views on whether UX is important or not (see Fig. 4). A group of the interviewees believed UX of software is increasingly important because of the growing general importance of software in recent decades. Interactions users had with earlier software were limited to command-line, and software was mainly built to support existing manual work. Software is now a large part of life, and users are exposed to a variety of interaction styles (mouse, touch, etc.), and experiences. This exposure to various software systems affects the experience and expectations of users. Regarding this, one practitioner said: “Users are meeting a lot of good things, and they are expecting good things all the time.” (E-1/HCI).
According to the practitioners, various businesses are learning from successful products in the market. This has inspired not only market-driven but also business-to-business software projects. This is supported by one of the interviewees saying: “A lot of business-to-business applications are being informed by business-to-consumer apps.” (G-1/HCI). In particular, in cases where a product has competitors, the motivation to improve UX increases. One of the practitioners argued that UX is now a ‘differentiator’: “Today I think that many companies do usable products, in order to distinguish a brand or a product we need to add an extra level to the product so that is really what I call UX. We need to take more things into account, e.g., emotions, and that it needs to look great, and it is not only about being usable.” (A-2/HCI). In market-driven products, branding, emotional concerns, and relations with end users are important. Therefore, in our participants’ views, these businesses are often more concerned about UX. Another interviewee argued that for market-driven software, in particular game development, UX is “part of the common practice” (A-1/HCI) while this is not the case for business-to-business software. He further emphasized this approach of market-driven projects should be “transferred to other projects” as well.
Some of the practitioners emphasized that UX is an important software quality characteristic that needs to be taken into account more in projects. However, in some functionality-focused organizations, UX is considered something “on the side” and not a core concept or value. According to these practitioners, the business units often focus on functionality and not UX. As one of the interviewees emphasized, in this case the business unit is not “that concerned with the look and feel” (A-4/HCI). The practitioners were generally positive that more and more organizations, even the technology-focused ones are learning about UX and the importance of taking it into account in their products. For instance, one practitioner said that in their company, business units now show less resistance towards UX and the importance of UX “starts to be visible” (H-2/HCI) to these units.
In addition, the interviewees emphasized managements’ preferences, values, and motivations also influence UX work in organizations. Regarding this, one practitioner stated: “I think it’s sometimes just the reason people go into business in the first place …the people who are in it because they think that their product or service solves a real problem, they generally care about UX more.” (G-1/HCI). This interviewee further discussed how individuals, in particular higher management, play an important role in whether UX is a priority in an organization or not. One of the interviewees stated that the inputs from their UX group in research and development are ignored by business units because some of the ideas are ahead of their time while these units tend to focus on near future: “business units are occupied with their very close, near time results so they look at what they can sell now.” (H-2/HCI). He further highlighted that some of these previously rejected UX ideas are now being incorporated in the products and are getting support from management because competitors now implement similar ideas.
Another challenge is that customers often resist taking on the costs of performing UX work. In practitioners’ view, often customers are not aware of UX and its value, and in particular how it differs from usability and IxD. Regarding difficulties in convincing customers, a practitioner said: “It can be hard …If you have to have three weeks extra to make the graphics work in a certain way, they might think it’s unnecessary. And maybe the software’s going to work fine, but if they made these extra efforts or put in this extra amount of money, they would have actually gone much, much further maybe.” (A-4/HCI). As a way to convince customers to agree to the cost of UX practices, one of the organizations (A) uses examples of successful products in the market that are known for their good UX (e.g., Apple). Another organization (F) uses fixed prices for their projects so that practitioners can freely spend part of the budget on UX practices. Some practitioners emphasized that they talk about UX ‘indirectly’. They connect UX to business goals, and argue for UX from the point of view of the business success, and not the end users. An interviewee motivated this by saying: “if you start babbling about usability and strange kind of things, they will say ‘oh! I don’t want to pay for this’.” (E-1/HCI).
According to some practitioners, UX practices should be sold to the customer as part of the contract to assure coverage of the associated costs. This requires showing how such practices will add value to the software and have a return on investment (ROI) for the customer. Nevertheless, often presenting a ROI is difficult. This is illustrated by a UX practitioner who said: “I don’t think you can put ROI into a proposal necessarily. I think that’s irresponsible, frankly. Because 95% of the time, we don’t understand the true and false nature of an issue when we’re writing a proposal. It’s only after working with a client for a little bit of time that we begin to see the nuance there. That usually undercuts any kind of understanding that’s used to generate proposed improvements in ROI, for example.” (G-1/HCI).
Theme 3. Low industrial impact of UX models, tools and methods
We identified a number of challenges concerning how practitioners apply UX theories, tools and methods (see Fig. 5). We observed that often practitioners’ knowledge regarding UX is based on experience and work with similar concepts, not on any specific UX models or theories. Some practitioners had formal training in usability and IxD and used that foundation to build UX skills over time in an ad hoc way. For instance, one of the practitioners stated: “When I first started in the 90s, there was no such thing as UX or IxD, actually … [then] it has been IxD and then sort of merged into UX design.” (H-1/HCI). In general, the interviewees were not familiar with currently popular approaches to UX and corresponding models, even those interviewees that demonstrated a relatively good understanding of UX. One interviewee (A-4/HCI) stated that she handles users’ emotions, values etc. ‘in a non-academic way’, another interviewee (A-2/HCI) said that the way she handles UX is not formal or ‘by the book’. The practitioners generally showed a positive attitude towards applying new models, tools, methods and techniques to their work: “we are lacking this, so this would be really nice to have more research results that we could apply.” (A-2/HCI). However, often organizations resist introducing new models, tools, methods, or techniques. Hence, in the studied organizations, practitioners often only rely on traditional interview and observation techniques in their work with UX. The interviewees referred to two models they use in their UX work: ‘emotional design’ by Donald Norman and Maslow’s hierarchy of human needs. The latter is used as an inspiration when eliciting user needs. The interviewees mentioned that such models can help create a methodology to work with UX and “build the right things in the right order” (F-1/HCI). Respondents with this type of experience were a clear minority.
Theme 4. Focus on objectively measurable aspects of software
Another identified theme of challenges concerns how the studied organizations handle subjective aspects of software (see Fig. 6). A group of practitioners emphasized that the software development and engineering community has traditionally had much greater focus on software functionality. One practitioner stressed the relation between functionality and UX as follows: “The quality of experience is really depending to some extent on how the functional requirements are met, but also actually on what the functional requirements are, also just the amount of them.” (H-1/HCI). This interviewee further emphasized: “For us, technology is the material that we can shape in order to create experiences.” The interviewees emphasized that satisfying functional requirements does not necessarily mean that the software includes correct or valuable functionality. This is evidenced by one interviewee saying: “what do you know when you have signed [the technical specification]? Do you know that it is a good solution? No! you only know that it meets the functional requirements and to me it is silly!” (E-1/HCI).
In addition, the participants believed that the software community often focuses more on ‘actual’ than ‘perceived’ quality characteristics of software. While the former concerns objectively measurable quality characteristics, the latter concerns how users subjectively perceive these qualities. For instance, users may perceive a response time of 50 ms (i.e., actual performance) as fast or slow (i.e., perceived performance). Regarding the role of perceived qualities in experience of users an interviewee stated: “sometimes the perception of time is more important than the actual time, and these are the things you should pinpoint [to the stakeholders].” (E-1/HCI).
Theme 5. Difficulties in engineering UX-related requirements
Another identified theme of challenges concerns handling UX in requirements work (see Fig. 7). According to the practitioners, in many cases, the non-task-related needs of users (i.e., their be-goals) or their emotions are either neglected or treated only informally: “We’d specify this more in the persona descriptions; for example that this persona needs to, or wants to experience some kind of things. In the wireframes, we might specify an animation for example that it should feel smooth or something like that.” (A-2/HCI). For instance, in one of the organizations, emotional design goals are often only documented and communicated in the form of a “post-it note on the wall”, as a reminder. Although the interviewee emphasized this approach is not optimal and there is a need for more formal approaches to deal with such requirements: “it should be good if we could formalize it a little bit more I think.” (A-2/HCI).
The practitioners highlighted that it is an open problem as to how to map these types of needs to measurable requirements. For instance, one of the interviewees stated: “I would say the emotional part of this is very very rarely formally put into words.” (A-1/HCI). In particular, when the software product is innovative, these problems are compounded. This is evidenced by an interviewee saying: “This is a kind of project where nobody really can tell how this should be, what it should be like, nobody has done it before, there are no standards to refer to …who can specify those [UX-related] requirements? You need to have a certain quality but what is the suitable level of that quality? We haven’t really found what that level is.” (B-3/SE).
The practitioners generally agreed that to communicate UX-related requirements, they require forms other than text (e.g., sketches, wireframes). They emphasized “concrete and tangible” forms can facilitate communicating these intangible and abstract requirements, and ensure that be-goals as well as do-goals are taken into consideration. Regarding this an interviewee told us: “we create ‘mood boards’ where you take an image-driven approach to the look and feel, and we use references of course, like ‘that app has a good flow in it’, and ‘that app has a good feeling to it’.” (A-1/HCI). Practitioners with technical backgrounds also seemed to have a similar approach to UX-related requirements: “If the customer said that they want it to ‘look nice’, then you have to make the graphical design first and then they can say ‘hey! this looks nice’ and then you have taken care of that requirement.” (A-3/SE).
It is important that these non-textual UX-related requirements (e.g., wireframes) are traceable to other requirements. Regarding this a practitioner stated: “at the end we show the wireframes with all kinds of numbers and those numbers are linked to the excel sheets of the requirements, their descriptions and how they are linked with the CPR and business case.” (F-1/HCI). Practitioners related these problems to a lack of competence in dealing with UX-related requirements within their organizations: “I think it is largely a competence thing. Doing emotional aspects of design is quite a new concept, I have only heard about it in the last year, or last two years maybe, so I do not think that knowledge has really reached the industry yet.” (A-3/SE). UX practitioners believed that handling functionality is comparatively easy and straightforward. Regarding this, one practitioner highlighted: “features are what most project managers, most managers can understand. You can count them, you can map them to customers, customer dialog for instance, and so forth, and you can compare your amount of features with the competitors...when it comes to both features and user experience requirements—features are a bit easier to define somehow, because you can say things like, ‘Right, we need an email component with this capability”’ (H-1/HCI). As another example, an interviewee stated: “Functional requirements are easy to create, to merge into a design; more emotional things are more difficult.” (F-1/HCI).
UX practitioners argued that UX-related requirements should be elicited before refining functional requirements. For instance, one interviewee stated: “First you have to define the business requirements, the user requirements, the IxD, then you can define the functional requirements.” (E-1/HCI). Another practitioner said: “I think that the functional requirements should come as a result of a dialog between different types of domains such as user experience, business, and technology.” (H-1/HCI). Nevertheless, according to these practitioners, such an approach is not common in practice. This is inline with the view of practitioners with technical backgrounds who believe the requirement process should start by first eliciting and refining functional requirements and then quality requirements.
Some practitioners highlighted the challenge of finding a balance between UX-related needs, business goals, and technological constraints. Regarding the importance of finding a balance between emotional and business needs, one of the interviewees stated: “you can spend a lot of time, thinking about people’s emotions and so on, but if you are going to succeed you have to look at the business perspective.” (E-1/HCI). In addition, according to the interviewees, while customers should be involved to assure alignment of projects with business perspectives, the end users should be involved to assure alignment with UX-related needs, and practitioners require to find a balance between these needs. As an example, an interviewee stated: “we can buy the best camera in the world, users will have a better image quality than they have in reality but it is too expensive today, ….that ruins the business case for that customer” (e.g., B-3/SE).
We observed some inconsistencies in practitioners’ views concerning user involvement. In favor of involving users an interviewee stated: “maybe you want to have the end user involved also with the developers so that developers understand what they are doing, instead of just following the specifications. I think that would be very very valuable.” (A-3/SE). On the other hand, a project manager stated that involving users in requirements discussions increases the costs of the projects. Therefore, it is better to negotiate requirements and sign a contract without involving the users. Regarding this an interviewee stated: “[they] think they can say anything during the [requirement] workshop and then get it. It is not the case... So it leads to lots of long long long discussions afterwards.” (B-2/SE). As another example: “it usually leads to features that you take on more than what you agreed from the beginning. So it’s possible that the customer gets a better system but they still don’t pay you more money for this.” (B-1/SE).
One developer stated that they often have less access to the end users, which can be problematic for their work mainly because this means developers do not often understand the rationale for the requirements: “developers don’t get the motivation behind the requirements because that gets lost during the way. So, the marketing people say that we must have this requirement, and interaction designers say we must do it this way, so you have the ‘what’ and the ‘how’ but you never get the ‘why’.” (A-3/SE). Some practitioners emphasized that relying too much on the end users’ opinions might lead to less creativity in the design work: “we have a quote sitting on the wall here, it’s from Henry Ford [that says] ‘If I had asked people what they wanted, they would have said a faster horse’.” (H-2/HCI).
Theme 6. Focus on evaluating functionality and usability, not UX
We also identified a number of challenges regarding evaluation of UX in the studied organizations (see Fig. 8). Our data shows that in general, our selected organizations mainly focus on testing functionality. In addition, similar to the previous theme on requirements, the interviewees highlighted that limited involvement of the end users in projects is a challenge to evaluating UX, especially since evaluating UX requires gathering users subjective perception of the software. We observed that practitioners with technical backgrounds are often unfamiliar with how their organizations handle UX or even usability testing. They also showed limited knowledge as to why such evaluations can contribute to the success of the software.
Generally speaking, UX evaluation is immature in the studied organizations. In projects with limited time or budget, UX evaluation is either non-existent or rare, especially when compared to other testing activities: “we have done much functional testing of course, system tests etc., but end user testing we have not performed much I’d say.” (A-2/HCI). In some organizations, UX evaluation is basically replaced by usability testing. Regarding this, one interviewee stated: “I think user tests tend to be more focused on pure usability. I guess it’s when you’re releasing the product into the wild, that’s when you start to get maybe the most valuable feedback or the most truthful ones.” (A-4/HCI). According to these practitioners, usability testing is not enough to evaluate the whole UX of the software. Regarding this, a UX practitioner stated: “when you evaluate usability, it’s when you go into the nitty gritty details, and try to look at efficiency within the user interface. My personal view is that that is not that relevant. I mean it’s relevant but not in what we do [i.e., UX].” (H-2/HCI). To compensate for limited formal UX evaluations, some practitioners gather informal qualitative user feedback after release, for instance through comments in the App Store or on social media.
According to the practitioners we spoke with, UX evaluation is limited because it is difficult compared to evaluating usability or other quality characteristics. Some practitioners related this difficulty to the fact that UX involves emotions, and non-task-related user needs (i.e., be-goals), and that limited tools and methods exist to support addressing these needs in evaluations. Emotion can be even impossible to measure using current quantitative approaches, as one of the practitioners stated: “it is difficult to sort out or really get the correct feeling that the user has because they will try to explain it but it perhaps is not the real emotion that we catch at the end. If we had a method or approach to really get these from the user that would be great.” (A-2/HCI). Similarly, one interviewee said: “some goals are more difficult to measure than others, e.g., if this is a feeling thing: ‘I should be very well informed’, but mostly you can measure [them] in the usage test through observations and interviews” (E-1/HCI). This practitioner further emphasized that they can specify quantitative measures for UX-related requirements (e.g., “10 out of 10 users should succeed, and they should be content”), but also need to observe users in order to gain better understanding of the experience: “can the users perform the tasks? how do they perform the tasks? how do they feel afterwards? are they content?” (E-1/HCI). This interviewee further emphasized that to measure feelings of users they need ‘a rather complete prototype’. Practitioners highlighted that despite importance of involving the end users in UX evaluations, it is difficult to access them: “I think we should involve the end users a lot more but it is a political issue, the project needs to convince the customer that this is necessary.” (A-2/HCI).
Some of the interviewees related the difficulty in UX evaluation to the holistic nature of UX. Discussing cases where practitioners take a holistic approach to UX, one interviewee stated: “how would you measure that sort of holistic experience throughout the process of designing it? Because, of course you cannot [implement or design] everything at the same time, and you know there are so many dependencies. How do you straighten those out and how do you understand what you’re measuring and not measuring?” (H-1/HCI). This interviewee further emphasized that the broader scope of UX negatively impacts evaluations: “Evaluation is a problem of course, cause user experience is much broader in scope than usability, it’s more difficult to evaluate also. Like the phone example that I gave you before, how do you pick up on the fact that someone has experienced the competition unless you ask for it? Can you count on what a person would actually say about their experience? What if the person doesn’t say anything? That’s a problem, of course, because UX is much broader in scope, and if you have a wider scope on it, then you have a much more difficult task to actually frame it in an evaluation phase.” (H-1/HCI).
Some practitioners related the difficulty in UX evaluation to the fact that users’ expectations and their perception of a product change over time and are affected by various factors; e.g., introduction of new technologies or appearance of a competing product. One of the interviewees highlighted: “It’s like when you try on new clothes. The shirt you were wearing going into the dressing room and looked fine, looks shabby when you’ve tried out the new shirt.” (H-1/HCI). The interviewee used this analogy to explain the subjective and dynamic nature of expectations, and that for each individual a new experience can affect the user’s perception of other products.
Another problem concerns difficulties in relating the result of laboratory evaluations to the real experience of users, and the role of context in forming experiences: “I think user tests tend to be more focused on pure usability. I guess it’s when you’re releasing the product into the wild, that’s when you start to get maybe the most valuable feedback or the most truthful ones. So it’s often good to make some sort of what we call a beta product and publish it. Let people interact with it. But that’s a luxury also because that takes time. Often, in many, many cases, you have to just make something work. In an ideal world you should have a testing phase, of course. To get the users’ perspective. That should be really nice to have.” (A-4/HCI). Another problem concerns the relation between different episodes of experience, for instance first impression of users and UX after using the software for a while: “It’s a fantastic feeling to sit in a brand-new car in a car shop. How do you feel about the same car if you’ve had it for a year, driving it around with two kids in the backseat, for instance? Experiencing all of the quirks from the interior, and all the things that don’t work, the stupid things that make your hands go dirty every time you’re supposed to close a door, for instance? That’s part of user experience design. So it’s a very floating scale.” (H-1/HCI).
Theme 7. Lack of consensus on UX-related competencies and responsibilities
Another group of challenges concerns the competencies and responsibilities required to perform UX practices (see Fig. 9). Our data shows that for better UX work, organizations need to have access to a variety of competencies including brand management, visual design, usability engineering, interaction design, and emotional design. In particular, to be able to address the hedonic aspect of UX, it is crucial to have access to competencies for eliciting, refining, communicating, and testing be-goals or non-task-related needs of users.
A group of interviewees believed that due to the wide spectrum of UX competencies, it is unlikely to find all of those competencies in any individual practitioner. This group therefore argued that all of the team members (with complementary competencies) should jointly take on the UX responsibilities. Regarding this, one of the practitioners said: “I’m not sure if that should be a specific role […] so everybody should have a UX focus now. I’m not sure if we can have some sort of UX guy [who takes the final decisions].” (A-4/HCI). This interviewee further emphasized that achieving a better UX requires a ‘UX-mindset’ that even the technical roles in the projects (e.g., programmers) should have. On the other hand, a group of interviewees believed there is a need for specific practitioners with these multidisciplinary competencies because these competencies are tightly connected and hard to separate. This group therefore argued for defining specific UX-related roles and responsibilities in projects. Regarding the importance of access to individuals with diverse competencies, an interviewee stated: “I, as an art director, have to have somewhat deep knowledge about UX, and also IxD …you can’t separate them.” (A-4/HCI). Another interviewee said: “You have to be knowledgeable in many areas. You have to be very holistic in the way you think about things, cause you have to speak with programmers in the language of programmers, to some extent, at least, to gain their respect, but also you have to be treated as an equal...You also have to understand business, and that side of software. You have to be a little bit of everything but you also have to be a specialist in your own domain so it’s kind of like a juggling act.” (H-1/HCI).
Being able to gain an overall view of UX design was one of the concerns highlighted in the interviews. According to the practitioners, this is often a difficult yet important prerequisite for creating a coherent UX. As the interviewees stated, to deal with the complexity of today’s systems, the common approach in the software community is to break down the whole system into various sub-systems and work on them independently. Such an approach can harm the UX of the software since often in these cases UX practitioners lose the overall view on the UX design, how these different sub-systems fit together as a whole, and how they individually and in combination contribute to the experience of the end users. Regarding this, one interviewee emphasized: “you have to tear it down, yeah! but what happens to the whole? who is going to define the whole?” (E-1/HCI). The interviewees also highlighted that in agile settings, the decision-making process is spread out both over time as well as among the team members. This further complicates the process of creating a unified and coherent UX design (e.g., H-1/HCI). Further, agile processes enforce a focus on a few piece(s) of the design at each iteration. Regarding this, one interviewee said: “you need to deliver wireframes for parts of the application but you still do not know how it all will fit together at the end.” (A-2/HCI).
Similar to what we saw for competencies, our data shows that responsibilities of UX practitioners are also very diverse. As some UX practitioners expressed, they have varying responsibilities in and contributions to different projects; this depends on factors such as management support, available resources, and timing. In one organization (C), the UX team is responsible for handling requirements and feeding them to the development teams. In another organization (G), the UX group is part of R&D where the group mainly focuses on future products, and long-term vision of the company. One UX practitioner from company G described her responsibilities as: “that can loosely be described as discovery, research, overall strategy, and then high-level design.” (G-1/HCI). In another organization (H), since the number of practitioners with UX knowledge is low, none of these practitioners are part of any particular project teams, and are instead shared resources among projects. UX practitioners are also often responsible for spreading knowledge and awareness about UX in the organization, or giving support to development teams concerning UX matters. Regarding this, a practitioner expressed: “I think on a very high level, our responsibility is to inform, influence and inspire.” (H-1/HCI). He further stated: “we contribute to the process by running workshops, by providing provocative questions, or providing examples, engaging in discussions in which people from other domains dig down really deep into their own layers of knowledge, and we can ask really simple questions to poke them with our perspective.” (H-1/HCI).
Theme 8. Communication and collaboration gap between UX and non-UX practitioners
This theme of challenges concerns how UX and non-UX practitioners communicate with each other and collaborate in projects (see Fig. 10). The interviewees generally agreed that better UX work requires early involvement of UX practitioners in projects. Early involvement helps UX practitioners get first-hand information about the customer and the end users. In addition, practitioners get a chance to negotiate trade-offs between the design solutions and technical constraints. If required, a design concept can then be updated based on the developers’ feedback. Nevertheless, UX practitioners often are involved in projects only in later stages. Regarding this a practitioner stated: “The worst case is when someone has met a client and talked a lot about the software, then I meet this guy who has met the client … then it’s secondhand information and everything gets distorted.” (A-4/HCI).
Early involvement is in particular important since in later stages it is difficult or even impossible to make effective changes to the UX design. As an interviewee stated: “on the personal level developers can be offended if someone thinks it’s not a good solution, they can have a hard time killing their darlings, they could have been thinking of it as a very beautiful solution they have made with 40,00 lines of code, so they don’t want to throw it away.” (H-2/HCI). Some stakeholders however believe that UX can be improved with just minor GUI changes in later stages of development, they therefore down-prioritize UX practices in earlier stages. The interviewees also highlighted that agile processes have a limited focus on strategic decisions such as the overall UX of the software. Often these strategic decisions are either ‘skipped’ or postponed since agile methodologies tend to prioritize immediate and current problem solving: “agile is a lot about problem solving and that’s what sort of gets priority.” (H-1/HCI).
According to the interviewees, even in cases when UX practitioners are involved in early phases, they may lose their connection to the project in later phases. To overcome this problem, the practitioners proposed various solutions. One interviewee, for instance, stated that they rely on UX advocates in projects: “We are not UX experts that you hire if you want to give a problem to somebody, have them go away for three months, and then have them come back with the solution... If something needs to go into production after our principal work is finished, you need somebody who’s going to be involved in that process and involved from the very beginning. So we have a very strong focus on collaboration...Oftentimes, at least when it comes to transitioning into a production environment, we’re looking for UX savvy development managers or developers.” (G-1/HCI). Another UX practitioner told us that throughout their projects, they continuously check the status of the UX design: “we try to have always at least one person who was part of the original dialog present during the weekly checkups, and basically just going by the desks and checking, informally.” (A-1/HCI).
UX practitioners need to be involved in projects early on, and need to have an active role throughout the projects which, in our interviewees’ view, can lead to power struggles between UX and non-UX practitioners: “sometime it can be perceived as we’re trying to take control of the situation.” (H-1/HCI). This interviewee further emphasized this often means that working with UX is more difficult than usability: “I think the reason why it was easier to work with usability to some extent was that you didn’t take up any space. It was like being a woman in the early 20th century. You were there, but you didn’t vote, you didn’t do anything.” (H-1/HCI). Similarly another interviewee said: “There are a lot of strong stakeholders that are really interested in doing those kind of things, programmers for instance, who like to be in control.” (A-2/HCI).
Power struggles between UX and non-UX practitioners were also attributed to different motivations of practitioners (e.g., a developer wanting to develop more efficient code vs. a designer wanting to create a better design). Regarding this, a UX practitioners emphasized: “Everyone wants to start with their own domain, as soon as possible, from day one. Then, of course, you have the problem of ownership of the direction, where to go and what to do and why. That’s something that we struggle with quite a bit.” (E-1/HCI). Similarly, another interviewee said: “I think there is an ongoing struggle, cause we all have different drivers; like my driver is always about making technology that is interesting, useful, and brings some sort of value to the users in the context in which it is being used.” (H-1/HCI). This interviewee further emphasized the situation has changed in their work as they moved from usability towards UX in recent years: “Compared to when we only worked with usability, I would argue that the situation is rather different, and part of it has to do with, being part of the early phases, but I think the biggest challenge and the biggest struggle nowadays is really the ownership of decision making, and who calls the shots …Because usability didn’t disturb the big process, the big decision-making, the prioritization of what to do, or what constituted a feature that was needed to be implemented. Usability wasn’t part of that discussion. Usability was always part of like, how you should make a font, or the colors, like lipstick on the pig!” (H-1/HCI).
Better UX work requires regular communication and collaboration among UX and non-UX practitioners which sometimes can be challenging since these two groups of practitioners often have different responsibilities, education, motivation, and constraints in their work. Regarding the importance of communicating with designers a developer stated: “I think we need to be working tighter with the design department to help them know what can be done.” (D-1/SE). Similarly, a UX practitioner said: “we really try to make this give and take …we have constant communication, and I will say that we get always input from developers that we need to consider.” (A-2/HCI).
According to the UX practitioners, if they are disconnected from the technical roles, e.g., developers, they may not be aware of technological constraints when choosing and developing a design concept: “honestly the further we are away from the people that actually build the stuff, we run the real risk of becoming hand waving idiots.” (G-1/HCI). Another UX practitioner stated: “I realized quite early in my career that I have to communicate with these guys who program or develop something, and I have to understand what they’re saying.” (A-4/HCI). The interviewees however emphasized that overcoming this communication gap should be a two-way effort: “I’m not saying that we should be the only ones with this kind of multidisciplinary approach. I think the other ones should also have that, that’s a big challenge, I would say.” (H-1/HCI). Similarly, another practitioner said: “So I have to have some sort of knowledge about the technology because I have to know what my limitations are …. I have to have some sort of technical know-how so I can communicate with developers. I expect the same from them. So they realize that the aesthetic choice has to be made and it can take time.” (A-4/HCI).
According to the practitioners, in order to facilitate a better communication, UX practitioners need to acquire basic knowledge about various technical topics., e.g., programming, testing, architecture etc. As emphasized by one respondent: “You have to be like knowledgeable in many areas…you have to be very holistic in the way you think about things, cause you have to speak with programmers in the language of programmers, to some extent…You also have to understand the business.” (H-1/HCI). Similarly, another practitioner stated: “you have to speak in an engineering language. That’s a real challenge for UX work, because you always have to translate it to terms that makes sense to an engineer or economist.” (A-1/HCI).
The respondents also emphasized that communication between UX and non-UX practitioners can be challenging because of lack of trust in UX practitioners. Regarding this, one interviewee emphasized: “we have had problems with some of the developers sometimes. It has been a bit of conflict. I think we have some work to do to really get a ‘we-feeling’ that we, together, are developing an awesome product.” (C-1/HCI). One practitioner attributed this lack of trust to the fact that the field of UX is a relatively new field and less established compared to the technical fields, e.g., SE. This also means that UX practitioners are often younger and less experienced than non-UX practitioners: “most UX practitioners are quite young still, I do not think they have been that long in the market for development of their competencies yet.” (A-3/SE). One UX practitioner emphasized that they can gain more trust over time and as they accomplish more in their work: “[over time] we are adding on the pile of what we would call successful things we have done, and of course that gives us a bit more ‘trust’ I could say.” (H-2/HCI).
In this section, we answer the research questions and describe the implications of our findings for research and practice. We use Hassenzahl’s model of UX and his proposed list of UX characteristics as an analytical lens to investigate the challenges in greater detail.
In answer to RQ1 (what challenges do practitioners face when integrating UX practices into software development processes and organizations?), we found 8 themes of challenges as depicted in Fig. 2. Some of these themes are more fundamental and concern the views, attitudes and knowledge of stakeholders (Theme 1, Theme 2, Theme 3, and Theme 4) while some others are more tactical (Theme 5, Theme 6, Theme 7, and Theme 8). There is clearly a multifaceted and complex set of relations between the themes. These interrelations are discussed here to the extent supported by our data. More empirical data should be gathered in the future to elaborate and validate these connections.
At a high level, the more fundamental themes can explain the tactical themes. A limited knowledge of UX (Theme 1) is likely also connected with the low impact of UX models in industry (Theme 3). In addition, a lack of common reference and knowledge base can lead to language gap, meaning that the concepts of UX and UX practices can mean different things to different people. This gap may contribute to communication problems within software development organizations (Theme 8).
Through a lopsided emphasis on functionality and actual quality characteristics (Theme 4), we can explain the identified challenges concerning UX-related requirements (Theme 5) and testing (Theme 6). Requirements and testing challenges can also be attributed to limited tools, methods and competencies, and communication issues (Theme 3, Theme 7, and Theme 8). Even if companies intend to focus more on UX, they do not always have the capability (e.g., competencies) to turn this intention into action. Moreover, even if UX practitioners with the right competencies are present in an organization, they often face power struggles and lack of trust therefore fail to effectively communicate and collaborate with non-UX practitioners.
In answer to RQ2 (how do UX challenges relate to challenges of handling software quality characteristics, in particular usability?), we found that there are in fact various overlaps between the identified themes and challenges reported for handling usability and/or software quality characteristics in general. However, despite similarities at a superficial level, we differentiate the challenges through the characteristics of UX: subjective, holistic, context-dependent, temporal and worthwhile. In particular, through these characteristics, we highlight the inherent differences between UX and other quality characteristics, including usability, and explain the lopsided focus of the software industry on usability and the pragmatic aspect of UX.
While previous empirical studies report that usability is not understood properly, especially among technical stakeholders (e.g., Bak et al., 2008) we observed that in the studied organizations practitioners have fair knowledge and awareness about usability. This can be an indication that industrial knowledge and awareness of usability is improving. In the case of UX, the same level of maturity however is not yet achieved in industry and organizations struggle with knowledge and awareness concerning (i) the concept of UX and how it differs from usability, (ii) the potential of UX, in particular its hedonic aspect to add value to software, (iii) existing UX theories, tools and methods, and how to apply them in current development processes, (iv) competencies required for UX work, (v) handling UX-related responsibilities, and (vi) handling communication and collaboration among UX and non-UX practitioners.
Previous research shows that usability and purpose of use often dominate over UX in the software industry (Väänänen-Vainio-Mattila, Roto & Hassenzahl, 2008; Kuusinen & Väänänen-Vainio-Mattila, 2012). However, an explanation for why this is the case is missing in the current body of knowledge. We observed a similar trend in the studied organizations. We additionally found explanations for this lopsided focus by investigating how the characteristics of UX impact day-to-day work of practitioners. These characteristics were in fact also highlighted by the interviewees when describing the challenges or discussing their work with UX mainly in comparison to usability. Some examples of the interviewees’ quotations in relation to these characteristics are shown in Table 3. It is not surprising that these characteristics were pointed out only by those interviewees who have knowledge and experience on UX. Below, we clarify how our practitioners experience the implications of these characteristics in their work with UX (see Table 4 for a summary). In total, we identified 20 such difficulties that mainly concern requirements and testing activities in software development. As shown in Table 4, 11 of these implications have been previously reported in literature. However, to the best of our knowledge, the remaining nine implications are unique to our study. Clearly, our study did not focus on identifying these implications. Further research is needed to provide a more comprehensive understanding of them, and their interconnections.
|Characteristic||Sample quotes regarding challenges|
|Subjective||“it is difficult to sort out or really get the correct feeling that the user has because they will try to explain it but it perhaps is not the real emotion that we catch at the end, if it would be a method or approach to really get these from the user that would be great.” (A-2/HCI)|
|Holistic||“Functional requirements are easy to create, to merge into a design; more emotional things are more difficult.” (F-1/HCI)|
|Dynamic||“It’s a fantastic feeling to sit in a brand-new car in a car shop. How do you feel about the same car if you’ve had it for a year, driving it around with two kids in the backseat, for instance? Experiencing all of the quirks from the interior, and all the things that don’t work, the stupid things that make your hands go dirty every time you’re supposed to close a door, for instance? That’s part of user experience design. So it’s a very floating scale.” (H-1/HCI)|
|Context-dependent||“I think user tests tend to be more focused on pure usability. I guess it’s when you’re releasing the product into the wild, that’s when you start to get maybe the most valuable feedback or the most truthful ones. So it’s often good to make some sort of what we call a beta product and publish it. Let people interact with it. But that’s a luxury also because that takes time. Often, in many, many cases, you have to just make something work. In an ideal world you should have a testing phase, of course. To get the users’ perspective. That should be really nice to have.” (A-4/HCI)|
|Worthwhile||“if I start with ‘how’, I will never get to the ‘why’. If I start with ‘what’, with just making things, I will totally miss every important point there is. For me it tends to be very, very useful to focus on the why. So if I can sort of see this why, even if it’s very, very unclear, I can sort of approach this ‘how’ and ‘what’ in a much better way. (G-1/HCI)|
|UX characteristics||Implications (i.e., extra difficulties) as perceived by practitioners|
|• UX relies on human subjective perception
• UX is temporal and there is a complex relation between various episodes of experience
• UX emerges from complexly intertwined underlying elements
• UX includes both hedonic (be-goals) and pragmatic (do-goals) aspects
• UX is unique and situated in the context
• UX is worthwhile, about adding value, and more than just preventing problems and frustrations
|• UX work is perceived to be morea visible in the development process than usability work (our finding)
• abstract UX-related needs (e.g., be-goals or emotional needs) are difficult to refine and translate into more
concrete ones (our finding)
• it is hard to translate abstract UX-related needs to concrete design solutions (our finding)
• it is hard to frame UX in evaluations because of all the complex underlying dependencies between its elements
• there is a lack of consensus among practitioners on whether abstract UX-related needs should be elicited
before, after or in parallel with other needs (our finding)
• translating abstract UX-related needs (in particular for the hedonic aspect) to measurable requirements is perceived to be more
difficult than for functionalities or other quality characteristics (our finding)
• compared to the case of usability, more power struggles, and disagreements are perceived to rise between UX and non-UX practitioners (our findings)
• it is perceived to be more difficult to decide between design alternatives and resolve power struggles and disagreements that can
arise between UX and non-UX practitioners (our findings)
• requirements related to the hedonic aspect of UX are either neglected or treated informally (our findings)
• UX-related needs of users are abstract concepts (literature, our finding)
• practitioners do not have access to standardized and agreed upon set of UX measures and metrics (literature, our finding)
• field studies are crucial in UX work, in particular to gather authentic user experiences but are perceived to be
more resource demanding (literature, our findings)
• creating and using more sophisticated prototypes are vital in UX evaluations but are perceived to
require more resources (literature, our finding)
• practitioners are skeptical about effectiveness and reliability of current UX evaluation methods (literature, our finding)
• practitioners do not always have access to enough users (literature, our finding)
• practitioners do not always have access to sophisticated prototypes in earlier phases (literature, our finding)
• practitioners do not often know how to measure the whole UX in relation to its underlying elements (literature, our finding)
• users’ memory of their experience is prone to fabrication and fading (literature, our finding)
• a deeper understanding of the relation between UX and time is missing (literature, our finding)
• a deeper understanding of the relation between UX and memory is missing (literature, our finding)
The holistic nature of UX requires addressing not only do-goals but also be-goals. Practitioners however believe that eliciting be-goals and translating them into requirements and design solutions is more difficult than for functionality or other quality characteristics. One explanation may be that practitioners disagree on whether they should identify and refine these be-goals in parallel, before or after other do-goals, including functionality. Another explanation may be that these practitioners have limited access to guidelines, tools and methods for treating be-goals in their word.
Previous research (Law, Van Schaik & Roto, 2014) emphasizes that still limited practical guidelines exist on how to choose suitable UX measures and metrics for UX underlying elements, and interpret their findings to improve the overall UX. We additionally found that this challenge also negatively impacts requirements and design. UX requirements, design and evaluation methods are still developing. Methods such as emocards (Desmet, Overbeeke & Tax, 2001) for gathering users’ momentary emotions about the interaction and UX curve (Kujala et al., 2011) for long-term UX evaluation have been introduced but they are not widely accepted in industrial software development. We therefore highlight a need for more research on developing industry-relevant guidelines, tools and methods for identifying and selecting underlying hedonic and pragmatic elements of UX not only for measurement purposes but also for requirements and design activities.
Practitioners were also generally skeptical about the ability to address the subjective nature of UX in evaluations. For instance, our interviewees were concerned that what a user remembers about his or her emotions and shares in interviews or observations may not reflect the reality. Our interviewee’s concern confirms the findings of Law, Van Schaik & Roto (2014). Practitioners believed that the most truthful data about experience and perception of users can only be gathered in the field, and through using a functional version of the product in a real situation. However, field studies are too resource demanding (Vermeeren et al., 2010) and not feasible in all projects. We additionally found that despite skepticism about interviews and observations, these methods are common in industry. Moreover, organizations often resist introducing more novel tools and methods for various reasons including the cost of these tools and methods, or lack of knowledge and awareness about their value.
In addition, our interviewees found it difficult to evaluate UX because of its subjectivity. They emphasize that while there is a great need to involve the end users in evaluations, accessing them in projects is often rare. Limited access to the end users is a known and persistent problem in the industry (Bak et al., 2008). However, we argue that compared to usability work, UX work is, to a larger extent, impacted by this limitation. According to the current studies, to reduce the impact of subjectivity on evaluations and to generate less biased and more reliable results (i) more users need to be involved in evaluations for statistical significance (Law, Van Schaik & Roto, 2014), and (ii) other stakeholders (e.g., developers) cannot act as substitutes for users in evaluations because they are biased towards the software being developed by their organizations (Locascio et al., 2016). On the other hand, the limitations of access to end users and competencies can be easier to overcome in the case of usability through involving limited number of users (e.g., involving five users in usability tests (Nielsen, 1993)) or using internal stakeholders as test subjects (Locascio et al., 2016).
Also, our interviewees found it difficult to evaluate subjective and holistic experiences of users because access to functioning or complete prototypes is often rare in earlier phases of projects. One solution is to apply evaluation tools and methods that do not require functioning or complete prototypes but rely on practitioners’ imagination of how users may perceive the design, which is however an open research topic (Vermeeren et al., 2010). We also saw practical difficulties concerning handling the relation between different episodes of experience. It is important to take this relation into account in evaluations because of the dynamic nature of UX. The practitioners lacked knowledge and expertise on finding a balance between short- and long-term experiences in both design and evaluation. Hence, we agree with Law, Van Schaik & Roto (2014) that the community still lacks enough understanding of the relation between UX, time and memory (the implications of the subjective and dynamic nature of UX), and suitable UX metrics and measures that can support this relation are lacking.
The UX practitioners brought up the worthwhile (i.e., positive) nature of UX mainly when discussing their approach to requirements and design. The UX practitioners believed this aspect of experience (i.e., the value) or the ‘why’ behind developing the software for the end users, should be identified before the ‘how’ and ‘what’ that is connected to usability and functionality. Nevertheless, this view is often in conflict with traditional and common approaches to software development that focus on identifying functionality or other quality characteristics (Chung et al., 2000). The practitioners also pointed to the worthwhile nature of UX when describing their way of working and priorities. While the UX practitioners emphasize the software value from both the user and business perspective, non-UX practitioners often tend to merely focus on the business perspective which can create tensions between UX and non-UX practitioners. In particular, since practitioners still do not have enough knowledge, or tools and methods to make informed trade-offs between these two perspectives. For instance, to deliver a certain experience, UX practitioners may suggest one design solution to support specific user needs, but business stakeholders may suggest another solution to support specific business goals.
While previous empirical studies report limited access to usability professionals (e.g., Bak et al., 2008), the organizations we studied seem to have enough access to usability competencies which may indicate a general improvement of usability work in the industry. However, these organizations suffer from limited access to UX competencies, in particular concerning the hedonic aspect of UX and addressing various be-goals. To overcome the lack of UX and usability competencies, current research suggests simplifying well-known testing methods and training developers to perform them (Oevad & Larsen, 2016). While this approach is shown to be useful in the case of usability and the pragmatic aspect of UX (Locascio et al., 2016; Oevad & Larsen, 2016), this is not necessarily the case for the hedonic and more subjective aspect of UX. For instance, as mentioned before, developers’ bias is shown to compromise the validity of testing results because developers are loyal to the products they develop and their organizations (Locascio et al., 2016).
In the studied organizations, the responsibilities of usability professionals are well understood and defined while there are uncertainties concerning the UX-related responsibilities. In addition, our interviewees perceive that power struggles appear more often and are more difficult to resolve for UX than for usability. Previous studies report power struggles between developers and designers as a challenge to UX and usability work (Chamberlain, Sharp & Maiden, 2006) but do not give explanations for why this is the case. Here, we give more empirical support for this challenge, and additionally provide explanations for it.
One explanation can be that UX responsibilities are diverse and can overlap the work of non-UX practitioners such as business and requirements analysts. Another explanation is that all stakeholders experience different products and services on a day-to-day basis. Therefore, non-UX practitioners can easily have opinions about what experiences the software should deliver, and how they should be delivered, i.e., through which design solutions. This can lead to power struggles even in cases when there is no overlap between UX and non-UX responsibilities.
Another explanation may be that UX practitioners should be involved early on and throughout projects, and their responsibilities involve strategic decision making hence the work of UX practitioners is more visible in projects as our interviewees emphasized. The practitioners believed they can partially overcome the power struggles by increasing UX knowledge and awareness in organizations, and informing other stakeholders about UX responsibilities (what, why and how of UX practices), especially in relation to the overall strategies of the organization.
In contrast to what we discussed above for UX, practitioners can test other software quality characteristics in labs, and even without necessarily using a sophisticated prototype of the software or even involving the end users. For example, usability can be tested on simple paper prototypes, using heuristic or other expert evaluations, or performance problems can be minimized through the use of software performance prediction approaches in earlier phases (Balsamo et al., 2004). In the case of other software quality characteristics, the focus is on actual measures and values not the users’ subjective perception of these values. Additionally, in contrast to UX, other quality characteristics are not dependent on time. For instance, performance or security measurements will have the same results even when repeated over time, provided that the software, and the test context (e.g., CPU load) have not changed. In other words, the difficulties we discussed above concerning requirements and testing of UX are either less severe or not even present for other quality characteristics, making their practice less challenging as perceived by our participants. Limited access to the end users and competencies are also less critical when addressing usability.
Power struggles can be resolved using objective evidence that shows why an alternative is better than another. This can for instance be achieved through measurements which is possible for other quality characteristics, including usability, at least in theory. However, as we saw, measuring UX is difficult and even impossible in earlier phases. In addition, UX metrics and measures are not agreed upon or standardized, yet for other quality characteristics, practitioners have access to standards, e.g., ISO/IEC 9126-4 (International Organisation for Standardisation (ISO), 2004). Therefore, in the case of UX, it is difficult to decide among various alternatives in order to resolve the power struggles.
Furthermore, the importance of the pragmatic and hedonic aspects of UX varies in different projects. For instance, maximizing efficiency is not always relevant for the software being designed, and not every software is required to support different be-goals such as curiosity, motivation, interest or fun. Similarly, different characteristics of UX have different importance in different projects. While the first impression (i.e., temporality) may be more important for an e-commerce software, an e-learning software may require more focus on motivation (i.e., situatedness). We however argue that practitioners need to have sufficient knowledge and awareness on the hedonic and pragmatic aspects, UX characteristics, and be-goals to be able to make an informed decision regarding their relevance and importance in various projects.
Our study contributes to the current body of knowledge through identifying the overlaps in challenges of UX and other quality characteristics, in particular usability. These overlaps underline the significance of these challenges and call for more research on how to overcome them in practice. By highlighting these overlaps, we can help identify research areas for improving the work of UX as well as software quality in general. We can also make it easier for practitioners to spot, better understand, as well as find mitigation strategies for UX challenges by learning from past experiences and developments in the area of software quality. In addition, our study contributes to the current body of knowledge through connecting the challenges to the UX characteristics and clarifying the implications of these characteristics in day-to-day work of practitioners, in particular in comparison to usability and other quality characteristics.
Based on our work, there are at least three future studies that could be particularly interesting. First, it would be relevant to conduct an in-depth study to investigate various approaches that the academics or practitioners propose or apply to address the identified challenges, most importantly in relation to the characteristics of UX, and different software development cultures. Second, it would also be valuable to examine the differences between UX and other quality characteristics, and their associated practices in more depth through a deeper case study. We have reported on the similarities and differences between UX and other quality characteristics to the extent supported by our data. Because of the explorative nature of this study our data does not support details about how the studied organizations handle various quality characteristics in their day-to-day work. Third, we did not study how the type of software being developed (e.g., leisure vs work) may impact the identified challenges in the studied organizations. Investigating the correlation between the types of software and practices and challenges is an interesting future research direction. Investigating this correlation can help organizations better understand the cause and effect of the challenges and better plan for addressing them.
Designing and developing for UX is a multidimensional and multidisciplinary activity which has not yet gained an established position in the software industry. Many development companies still fail to deliver good UX in their products and services, and practitioners in these companies face various challenges when dealing with UX throughout their projects. Although UX is different from usability, many of the present frameworks and guidelines for UX integration do not clearly separate these two concepts. Moreover, the perception of UX generally differs in academic and industrial contexts: whereas the former concentrates on the hedonic aspect and emotions, the latter focuses more on functionality and usability issues. Our study shows that approaches for integrating UX into software development require further development. In particular, we showed that the characteristics of UX (Hassenzahl, 2003), i.e., subjective, holistic, dynamic, context-dependent, and worthwhile play an important role in shaping UX challenges.
We provided a deeper analysis of UX challenges through the lens of the aforementioned UX characteristics. These characteristics have at least 20 implications for the day-to-day work of practitioners. To the best of our knowledge, nine of these implications have not been previously reported, and are unique to our study. Through these implications, we explained the extra difficulties practitioners face in their work with UX compared to usability and other quality characteristics. We could also explain the lopsided focus of industry on the pragmatic aspect of UX. The related work does not provide such an analysis nor does it often make such a differentiation between UX and usability except for the few studies we summarized the related work. We therefore bring depth to the identified challenges by presenting their relation to the UX characteristics. We hope to have helped the community to identify ways to systematically improve the current UX.
To make future progress with integrating UX practices into software development processes, the community needs to take into account these UX characteristics and their implications for the day-to-day work of practitioners when developing guidelines, tools and methods to address related challenges. Practitioners need to be aware of the UX characteristics not only in evaluations but also requirement and design activities. It is also of crucial importance to understand and address the impact of these characteristics on communication and collaboration between UX and non-UX practitioners. Admittedly, the importance of the pragmatic and hedonic aspects of UX is different in projects. Nevertheless, practitioners need to be aware of the differences between these aspects, and various UX characteristics to make an informed decision in each project regarding where to focus and why. The challenges and implications we present in this paper can guide future research on UX integration and creating more awareness concerning UX, and related challenges in the software industry. Our findings can also shed light on the more general problem of how practitioners should integrate new and less mature knowledge areas, of which software quality characteristics and UX are examples, into software development processes.