The initial process of creating a guide to evaluate the usability in Virtual Learning Environments

Abstract

The usability evaluation includes a set method to analyze the system quality used in different development life cycle stages, for which there is a wide variety of evaluation methods (EM), each method uses certain ways and techniques to measure several aspects. Its choice does not only depend on which answer you are looking for but on multiple factors. The existing EMs are appropriate to assess the Virtual Learning Environments (VLE) usability due to the lack of combination methods or specific evaluation methods for this type of software. This paper focuses on showing the initial process before the creation of the guide from the EM combination that allows the usability evaluation in VLE. The process was carried out, initially, with a bibliographic analysis about the EM usability of existing interactive systems and a comparison among them, where the first combination version was obtained to address the object of study and to select the useful methods in order to assess usability in this context. The result of the application of these metrics will be the combination of EM usability to form the VLE evaluation guide in upcoming research.

Resumen

La evaluación de usabilidad incluye un conjunto de métodos para analizar la calidad de uso del sistema en diferentes etapas del ciclo de vida del desarrollo, para lo cual existe una amplia variedad de métodos de evaluación (ME), cada método usa ciertas formas y técnicas para medir diferentes aspectos. Su elección no solo depende de qué respuesta se está buscando, sino de múltiples factores. El problema surge cuando se busca cuál de los ME existentes son apropiados para evaluar la usabilidad de los entornos virtuales de aprendizaje (EVA), considerando que no hay métodos combinados o métodos de evaluación específicos para este tipo de software para obtener una evaluación completa, consistente y considerar factores tales como efectividad, eficiencia, satisfacción, tiempos razonables, entre otros. Es por eso que surge la siguiente pregunta: ¿Qué combinación de ME de usabilidad son apropiadas para aplicar en EVAs? Este artículo se centra en mostrar el proceso inicial antes de la construcción de la guía obtenida de la combinación de ME que permitan la evaluación de usabilidad en EVAs. Proceso que se realizó, inicialmente, con un análisis bibliográfico sobre los ME de usabilidad de sistemas interactivos existentes y una comparación entre ellos, donde se obtuvo la primera versión de la combinación, con esta versión, se definió un conjunto de métricas que se aplicarán al EVA objeto de estudio y permitirá seleccionar los métodos útiles para evaluar la usabilidad en este contexto. El resultado de la aplicación de estas métricas será la combinación de ME de usabilidad para formar la guía de evaluación de EVA en próximas investigaciones.

1. Introduction

Due to the great Internet growth in recent decades, online education has become a great alternative to traditional education. In the same way, educational institutions use available technologies and advances to provide more information to a growing audience. While the online education system proposals and modalities are growing, the number of people who use them is growing, too. So, it is necessary to consider the diversity in people's needs and characteristics to design Virtual Learning Environments (VLE) [1]. In this way, it contributes to design and builds online education systems; so, people can use them in a simple, effective, and efficient way that might provide a positive user experience. The increasing number of publics using them is growing; for this reason, the User Experience (UX) is a fundamental part of the success of the VLE [2]. The UX refers to "how people feel about a product and its satisfaction when they use it, look at it, sustain it, open it or close it" [2] The UX covers different aspects related to the software product quality such as accessibility, emotionality, usability, among others [3]. In this sense, the current research focuses exclusively on the UX "usability" feature concerning the "ease of learning", (which is defined as the time that a user - who has never seen an interface- can learn to use it well and perform basic operations, how much does it take a typical community user to learn how to use relevant commands from a set of tasks? [4]), specifically, in the VLE usability study.

Also, the usability evaluation has been determined as the activity comprising a method set that analyzes the interactive system quality use in different development life cycle stages [5]. It is necessary to perform the usability evaluation to validate if the final product meets the requirements and is easy to use. The evaluation’s main objectives are; to evaluate the system functionality scope and accessibility in order to evaluate the user’s experience in his/her interaction and to identify specific problems [6], These are the objectives that will be sought when evaluating a VLE with a combination method guide. To perform the usability evaluation, there are different Usability Assessment Methods (UAM), which depend on variables such as costs, time availability, and human resources among others [7]. In this way, choosing methods to evaluate VLE usability is not an easy task [8]. A series of UAM can be applied on a VLE, but the concern is related to how precise the information is given and at any combination of it. Similarly, there is no standardization regarding what, how, and when to perform the usability evaluation, but methods have been developed and used in an isolation way and with specific criteria to evaluate a particular product [9]. The usability assessment methods have strengths and weaknesses, they are focused on evaluating certain aspects or usability requirements, too. So, it is advisable to combine them in evaluation to complement each other in terms of their strengths and to cover a greater number of evaluation aspects [10]. The selection and evaluation methods combination will depend on financial and time constraints, the development cycle phases, and the development nature system [11].

Based on that, the problem arises when deciding which of the existing evaluation methods or combination is appropriate to evaluate the VLE usability. Therefore, the evaluation is completely and consistently carried out, getting concrete results on its usability, considering factors such as effectiveness, efficiency, satisfaction, reasonable timing, among other factors [11]. For this reason, the following research question emerges: Which of the existing UAMs are appropriate to apply in Virtual Learning Environments? That is why this paper focuses on the study about a set of methods for evaluating usability on VLE. These methods, after being selected, characterized and analyzed, will constitute a new combination method to evaluate the VLE usability, which can provide more complete and integral usability information, regarding the performance of the evaluation methods indiscriminately and independently. This paper is structured as follows: Section 2 shows a theoretical context to contextualize relevant research topics, section 3 contains some related works to usability evaluation in VLE. In section 4, the process to make the first UAM combination version and it is shown, and in section 5 the conclusions and future work are described

2. Theoretical context

Important theoretical references about the guide development process for usability evaluation in Virtual Learning Environments are outlined below:

3. Related works

Below, some related works are presented to justify the need to do the research presented in this paper (also analyzed in the paper [17]). Otaiza in [18], presents a study in which the UAMs have been studied in transactional web applications, contrasting their characteristics, and generating a methodological evaluation proposal to obtain the largest amount of relevant information regarding the usability of these kinds of applications.

In the same way [19], they examine e-learning usability evaluation methods, compare them, and propose criteria set that should be consulted when choosing the appropriate method to evaluate the e-learning systems usability. The research shows that none of the examined methods has allowed integral e-learning platforms usability evaluation and none of them addresses all relevant specific topics for the learning systems and modules.

In [15], methodological criteria are presented to evaluate the Course Management Systems’ (CMS) usability. The evaluation was carried out by combining different methods and instruments with the potential platform users: a group of teachers and language students. Traditional usability evaluation methods were used, they were mixed and some new ones were originated to evaluate not only the elements that make up the usability but also the functionality and the pedagogical aspect of the CMS.

In [20], a model to evaluate the VLE quality is proposed, considering usability as the central axis. The model is called MUSA, because it is a model based on usability, and is oriented to evaluate products in use. The general ideas are based on a four-level strategy or evaluation layers, which start from the general to reach the particular, where the usability definitions among attributes and heuristics form the core model.

Similarly, [21] is a research that is focused on analyzing the virtual learning environment usability for undergraduate university students, emphasizing psycho-pedagogical aspects that allow evaluating both, the quality contents and the system that contains them. The students' frustration implications in their cognitive process are analyzed, establishing this emotion as an immediate bad interface designing consequence.

4. Guide on construction activities for the usability evaluation in virtual learning environments

4.1. Research Methodology

This work is developed following a research methodology based on multi-cycle action with bifurcation [22]. The strategy starts with an initial research cycle where three problems are identified: conceptual, methodological, and evaluation. This allows us to divide the work into three research cycles: conceptual cycle, methodological cycle, and evaluation cycle.

4.1.1. Conceptual cycle

In this cycle, a contextual analysis is carried out to find the problem to be studied, this cycle has three research phases: A literature study about the virtual learning environments, user experience, usability, usability evaluation, and characteristics, attributes, or elements that are related to the usability assessment methods are analyzed from an extensive literature review. The second phase is the appropriate UAMs for executing in VLE identification, where, from the literature study, a possible set of methods are established to make up the proposed combination in this research, and finally, the activities, resources, and the phase of assigning a person in charge for each of the selected methods.

Besides, the reviewed and analyzed methods in the literature and the possibly formed combination of UAM in VLE are made and analyzed regarding what is presented in the project “Usability integration in software development process framework [3]".

As evaluation methods and according to the research carried out by Muñoz et al. [23], the three main groups were analyzed: inspection, inquiry, and testing. The methods of each group were compared with each other, this in order to determine their advantages and disadvantages, analyzing the most relevant characteristics (development stage where should the method be used, place of performance, if the method generates quantitative data, if it can be done remotely, the time it takes to carry it out, the number of required evaluators, the number of users required to execute it, if it is possible to do it automatically, and if the method analyzes the usability characteristics such as intelligibility, learning, operability, errors, esthetic, accessibility), considering the following related works [24], [18], [19], [15], [20], [21].

From the comparison of the inspection methods shown in Table 1, it can be said, in certain aspects, that the heuristic evaluation takes some advantages over the other methods, mainly because of the ease of carrying it out, which is not accomplished in other methods. According to the methods: actions analysis and standard inspection, highest level experts are required, and for the routes (cognitive and pluralistic) there must be considered task definition methodologies and certain training, characteristics that increase the complexity of carrying them out. However, the inspection potential methods are not in doubt, because, if the time and necessary conditions to carry them out are available, good results can be obtained [23]. From the comparison, it can be seen that factors such as time, equipment, and experts’ level, heuristic evaluation is still the simplest method to perform. However, the cognitive journey characteristics are very similar to the heuristic evaluation ones, except for the experts’ experience level. Considering the aforementioned factor, the methods: inspection of standards and analysis of actions, become the most complex ones to perform, but other factors that benefit them from the other methods, such as the data type they obtain, and the evaluators need to carry it out.

Table 1 convention: Heuristic Evaluation: 1, Cognitive Walkthrough: 2, Action Analysis: 3, Pluralist path: 4, Formal Inspection: 5, Standard Inspection: 6.

Table 1

Comparative summary among inspection methods [25

Thus, it is possible to establish certain comparisons among the test methods, shown in Table 2. The interrogation methods (questionnaires and interviews) are the simplest test methods to perform. Its characteristics allow, with few economic resources and with a preparation that does not take too much time, to obtain satisfactory results regarding the system’s usability evaluation. The interrogation methods are aimed to obtain subjective information from the system under evaluation, obtaining, in many cases, information that cannot be collected through other evaluation methods.

Table 2 convention: Focus Group: 1, Thinking aloud: 2, Constructive Interaction: 3, Questionnaires: 4, Interviews: 5, Surveys: 6, Formative Experiments: 7, Recording of use: 8, Performances Measurement: 9, Driver's method: 10, Test retrospective: 11.

Table 2

Comparative summary between the Test methods [25

And finally, it is also possible to establish certain comparisons among the inquiry methods, shown in Table 3. We can say then that contextual inquiry offers a deep understanding of the user's work, but on the contrary, it can only be used in the early stages of development but it generates a lot of information that makes it difficult to assimilate and analyze it, too. The participatory inquiry has the advantages that it does not waste users' time, and that it can be carried out remotely and does not need experts to use it, although it can be time-consuming when dealing with complex system tasks.

Table 3

Comparative summary among the inquiry methods [25

4.1.2. UAM study object selection

UAMs evaluate specific usability aspects to obtain the best processes for evaluating usability [25]. Different factors influence this, such as time, simplicity, the type of results, phases within the development cycle, economic resources, users, and experts’ number, among others.

Due to the large number of UAMs, it is necessary to select a smaller set of them to be the study object in this research [3]. Hence, it has been taken as a reference due to the following criteria: training need-closeness to Software Engineering-user's presence-applicability-contribution vs effort and representativeness. Also, giving them the following values: a little useful, useful, and very useful. Thus, UAMs classification parameters were considered and determined as it appears in the information tables 1, 2, and 3.

Inspection of selection methods

The selected inspection methods are:

• Heuristic evaluation

• Cognitive walkthrough

The UAM: pluralistic path, standards inspection, and action analysis are not considered in this project.

As it is shown in Table 1, the pluralistic path is not considered by its impractical simulation, as well as the fact that it makes it difficult to analyze due to the number of participants per session. The standard inspection is not considered either due to the wide standard of knowledge in terms of the high level of training and the lack of considering actions to be evaluated. The action analysis was not selected, mainly, because it requires a higher-level expert, which is expensive for most organizations to achieve. The formal inspection was also discarded by the need for highly experienced evaluators and a numerous team, which makes the implementation cost higher.

Test methods selection

Below, the instruments of the selected inspection methods are mentioned:

• Formal experiments

• Questionnaires and interviews.

• Constructive interaction

• Driver's method

The UAM: thinking out loud, use recording, performance measure, and retrospective tests, are not considered in this research. Based on the information in Table 3, thinking aloud is not considered because it interferes with the normal user’s behavior, which influences the interaction with the system. The use of the recording is not considered because its performance requires a high training level by the evaluators, the effort to establish the equipment is high, and, in addition, it is especially indicated only to analyze websites (level of low applicability). The performance measurement was not selected because it does not ensure the usability target of the obtained measure. Subjective information was neither used such as opinions, attitudes, satisfaction, and the environment used is not natural for users, so it can distort the users’ performance. The retrospective test was not selected because it takes at least twice as much time as any other method.

Inquiry methods selection

The inquiry methods will not be considered for the VLE usability evaluation methods combination because they were discarded in the related works [24], [xref>16], [19], [15], [20], [21].

It should be highlighted that the UAMs initial proposed selection is here linked to fundamental usability aspects and to different variables that must be considered in the development. The chosen methods could be applied correctly, effectively, and simply. However, these methods do not take into account the characteristics of a VLE, and for this reason, they do not evaluate the usability of these features and are mainly focused on a general application. Thus, it is expected that the usability evaluation in VLEs is carried out subsequently under the selected usability metrics that will yield the most appropriate UAMs to evaluate the usability in VLEs, due to the fact that those evaluations consider VLEs usability characteristic-aspects.

5. Conclusions

After the literature analysis, it was found that there is no standardization regarding settings of usability evaluation in VLEs. Nowadays, methods have been used in isolation, and with specific criteria to evaluate a product, methods that are not designed to evaluate VLEs usability. Therefore, what is intended to improve with the defined guide in this proposal will contain an evaluation combination method to be applied in virtual learning environments, due to the current boom of these software types of systems and the need to think of a satisfied end-user.

The selected methods to form the initial combination have been chosen because they are the fittest to evaluate the usability specifically in VLE, this selection has had several evaluation criteria as previously shown and from which it was possible to select of inspection methods: Heuristic Evaluation and Cognitive walkthrough, of the Test methods: Formal experiments, Questionnaires and interviews, Constructive interaction and Driver's method.

The types of inspection and test methods selected will be applied at later stages of the investigation in a VLE and will be executed together with the previously defined and selected metrics: number of identified problems, number of critical problems, number of frequent problems, time used to complete the planning stage, time used to complete the execution stage, time used to complete the analysis of the results. This in order to make another filter to the methods and leave those that really contribute in the evaluation of a VLE and as a result, it is expected to obtain a final combination that will be the basic guide to apply in a VLE, generating the information of usability required for these contexts.

Based on the obtained results in this research, we consider that the UAM selection strategy for VLE is a possible way to choose them, without leaving aside other strategies that can contribute with a selection that may fit in a better way to the VLE needs.

• E. Mor, M. Garreta and M. Galofré, Diseño Centrado en el Usuario en Entornos Virtuales de Aprendizaje, de la Usabilidad a la Experiencia del Estudiante, 2007.
• H. Sharp, Y. Rogers and J. Preece, Interaction Design Beyond Human - Computer Interaction, 2 ed., Wiley, John & Sons, Incorporated, 2007.
• X. Ferré, Marco de Integración de la Usabilidad en el proceso de Desarrollo de Software, Madrid, 2005.
• W. Sanchez, "La usabilidad en ingeniería de Software: definición y caracteristicas," Ing-novación , 2011.
• J. Nielsen, "The usability engineering lifecycle," Computer, vol. 25, no. 3, pp. 12-22, 1992.
• A. Dix, J. E. Finlay, G. D. Abowd and R. Beale, Human-computer interaction: Prentice hall, Inc, England, 2004.
• T. Granollers, MPIu+a una metodología que integra la ingeniería del software, la interacción persona-ordenador y la accesibilidad en el contexto de equipos de desarrollo multidisciplinares, LLeida, 2004.
• A. Solano, Propuesta metodológica para la evaluación colaborativa de la usabilidad de aplicaciones de Televisión Digital Interactiva, Popayán, Cauca, 2012.
• M. E. Alva Obeso, Metodología de Medición y Evaluación de la Usabilidad en Sitios Web Educativos, Oviedo, 2005.
• N. Bevan and M. Macleod, "Usability measurement in context," Behaviour & information technology, vol. 13, no. 1, pp. 132-145, 1994.
• S. Thüer and A. Ferreira Szpiniak, "Entornos Virtuales de Aprendizaje: Diseño de experiencias de usuario para la web 2.0," in Conferencia Internacional ICDE – UNQ 2011 “Educación a distancia, TIC y universidad: calidad, equidad y acceso a la educación superior, Buenos Aires,Argentina, 2011.
• R. Cicala, M. Perazzo, F. Bordignon and C. J. De Salvo, Investigación sobre entornos virtuales de aprendizaje utilizados para la enseñanza en profesorados y universidades nacionales, Buenos Aires: Unipe, 2011, p. 96.
• ISO, International Software Quality Standard, ISO/IEC 25010, Systems and software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - Systems and software quality models, ed,, 2011.
• J. Offutt, "Quality Attributes of Web Software Applications," IEEE Software: Special Issue on Software Engineering of Internet Software, p. 25‐32, 2002.
• J. Reyes Vera, M. I. Berdugo and L. Machuca Villegas, "Evaluación de usablidad de un sistema de administración de cursos basado en la plataforma LingWeb," Ingenieare, vol. 24, no. 3, pp. 435-444, 2016.
• R. Otaiza, "Metodología de evaluación de usabilidad para aplicaciones web transaccionales," Tesis de Maestría, Escuela de Ingeniería Informática, Pontificia Universidad Católica de Valparaíso, Valparaíso, Chile, 2008.
• J. D. Pinto, V. Agredo-Delgado and C. Collazos, "Construyendo una guía para la evaluación de la usabilidad en EVAs," Campus Virtuales, vol. 7, no. 2, pp. 93-104, 2018.
• R. Otaiza, C. Rusu and S. Roncagliolo, Evaluating the usability of transactional Web Sites, Saint Maarten: Third International Conference on Advances in ComputerHuman Interactions (ACHI'10), 2010.
• D. Plantak Vukovac, V. Kirinic and B. Klicek, "A comparision of usability evaluation methods for E- learning systems," in DAAAM international scientific book, 2010, pp. 271-288.
• A. Ferreira Szpiniak and C. V. Sanz, Diseño de un modelo de evaluación de entornos virtuales de enseñanza y aprendizaje basado en la usabilidad, La plata, 2013.
• M. P. Ponce Martinez, "Usabilidad en un sistema E-Learning," in 7 - th Europe conference E-COMM-LINE, Bucharest, 2013.
• F. J. Pino, M. Piattini and G. Horta Travassos, "Managing and developing distributed research projects in software engineering by means of actionresearch," Revista Facultad de Ingeniería Universidad de Antioquia, pp. 61-74, 2013.
• J. Muñoz, Y. Hernández, V. Bustos, A. Aranda, M. Calderon, C. Collazos, Y. Mendez, A. Solano, J. Guzman, F. Alvarez, R. Mendoza, J. Guerrero, J. M. Gonzalez, L. Rodriguez, T. Granollers, R. Gil and D. Cespedes, Temas de diseño en interacción humano computadora, Guayaquil: Proyecto LATin-Iniciativa Latinoamericana de Libros de Texto Abierto, 2014.
• G. Cockton, A. Woolrych and D. Lavery, "Inspection-based evaluations," in The Human-Computer Interaction Handbook, 2nd ed., Lawrence Erlbaum Associates, 2008, pp. 1171-1190.
• M. F. Lopez, "Métodos de evaluación de Usabilidad para aplicaciones web transaccionales," Informe final de proyecto para optar por el titulo profesional de ingeniero civil en informática, Valparaiso, 2012.
• ISO, International Standard ISO/IEC 9241, Ergonomic requirements for office work with visual display terminals, 1998.
• W. Albert and T. Tullis, Measuring the user experience: collecting, analyzing, and presenting usability metrics, Newnes, 2013.
• C. Martinie, P. Palanque and M. Winckler, "Structuring and composition mechanisms to address scalability issues in task models," in FIP Conference on Human-Computer Interaction, Berlin, Heidelberg, 2011.
• J. Rubin and D. Chisnell, "How to plan, design, and conduct effective tests," in Handbook of usability testing, Wiley Publishing, Inc, 2008.
• M. G. Piattini Velthuis, F. Ó. García Rubio, J. Garzás Parra and M. F. Genero Bocco, Medición y estimación del software: Técnicas y métodos para mejorar la calidad y productividad del software, Alfaomega, 2008.
• ISO, International Standard ISO/IEC 9126, Software engineering-Product Quality, 2001.
• S. Riihiaho, "Experiences with usability evaluation," Helsinki University of Technology - Laboratory of Information Processing Science., 2000.