A technique for interrater reliability evaluation of a mobile game aimed for executive functions stimulation

Bernardo Cerqueira; Debora Nice Ferrari Barbosa; João Mossmann; Jorge Barbosa

doi:10.31686/ijier.vol7.iss5.1518

Authors

Bernardo Cerqueira

Universidade Feevale, Brazil
Debora Barbosa

Feevale University, Brazil
João Mossmann

Feevale University, Brazil
Jorge Barbosa

UNISINOS

DOI:

https://doi.org/10.31686/ijier.vol7.iss5.1518

Keywords:

Executive Functions, Inhibitory Control, Game Evaluation, Cognitive Stimulation, Digital Games

Abstract

This work presents an interrater reliability evaluation of a mobile game aimed for the executive function’s stimulation, specifically the inhibitory control. The educational Exergame “The Incredible Adventures of Apollo & Rosetta in the Space” was presented in a previous piece of work, which showed the development of the game and its application in a researching intervention with children, in the 6 to 10 age-group, in a school environment. Subsequently, the game had its code reworked for being able to be used across different platforms, hence culminating in the present work. The methodology in this paper consists in a mixed qualitative-quantitative evaluation through questionnaires with four domain professionals experienced in the executive functions field. The statistical measurement used was based on the Kappa coefficient and average percentage among the judges. As the results indicate, there was a substantial agreement (k=0,659; P-value=0,000) between the raters, as well as a high percentage of agreement in general on the mobile game’s capability of Executive Functions stimulation for children.

References

Baba, Y. & Tschang, F., Product development in japanese tv game software: The case of an innovative game. International Journal of Innovation Management, 05(04), 2001, 487–515. DOI: 10.1142/S1363919601000464

BARBOSA, D. et al, 2018. Adaptation of an educational Exergame to mobile platforms: A development process. Communications in Computer and Information Science (PRINT), v. 1, p. 287-298. DOI: 10.1007/978-3-319-95522-3_24

Bernier, A. et al., From external regulation to self-regulation: Early parenting precursors of young children’s executive functioning. Child development, 81(1), 2010, 326-339. DOI: 10.1111/j.1467-8624.2009.01397.x

Buelow, M.; et al., A. The influence of video games on executive functions in college students. Computers in Human Behavior, 45, 2015, 228-234. DOI: 10.1016/j.chb.2014.12.029

Bull, R. et al., A comparison of performance on the Towers of London and Hanoi in young children. Journal of Child Psychology and Psychiatry, 45(4), 2004, 743-754. DOI: 10.1111/j.1469-7610.2004.00268.x

Bull, R. et al., Short-term memory, working memory, and executive functioning in preschoolers: Longitudinal predictors of mathematical achievement at age 7 years. Developmental neuropsychology, 33(3), 2008, 205-228. DOI: 10.1080/87565640801982312

Carlson, S. et al., Individual differences in executive functioning and theory of mind: An investigation of inhibitory control and planning ability. Journal of experimental child psychology, 87(4), 2004, 299-319. DOI: 10.1016/j.jecp.2004.01.002

Chen, P., Krauss, A., Interrater agreement. The sage encyclopedia of social science research methods, 2, 2004, 511-513. DOI: 10.4135/9781412950589.n444

Christopher, M. et al., Predicting word reading and comprehension with executive function and speed measures across development: A latent variable analysis. Journal of Experimental Psychology: General, 141(3), 2012, 470. DOI: 10.1037/a0027375

Conklin, H. et al., Working memory performance in typically developing children and adolescents: Behavioral evidence of protracted frontal lobe development. Developmental neuropsychology, 31(1), 2007, 103-128. DOI: 10.1080/87565640709336889

de Jong., Effects of training working memory in adolescents with a below average IQ. Workshop on Enhancing Executive Functions in Education in Nijmegen, May 20, 2014 Nijmegen, NL.

Diamond, A., & Lee, K., Interventions shown to aid Executive Function development in children 4 to 12 years old. Science, 333(6045), 2011, 959-964. DOI: 10.1126/science.1204529

Diamond, A., Executive functions. Annual review of psychology, 64. 2013, 135-168. DOI: 10.1146/annurev-psych-113011-143750

Diamond, A; Ling, D., Conclusions about interventions, programs, and approaches for improving executive functions that appear justified and those that, despite much hype, do not. Developmental cognitive neuroscience, 18. 2016, 34-48. DOI: 10.1016/j.dcn.2015.11.005

Fleiss, J. L.; Cohen, J.M., The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and psychological measurement, 33(3), 1973, 613-619. DOI: 10.1177/001316447303300309

Holmes, J.; et al., Adaptive training leads to sustained enhancement of poor working memory in children. Developmental science, 12(4), 2009, 1-7. DOI: 10.1111/j.1467-7687.2009.00848.x

Hughes, C. & Ensor, R., Executive function and theory of mind: Predictive relations from ages 2 to 4. Developmental psychology, 43(6), 2007, 1447. DOI: 10.1037/0012-1649.43.6.1447

Baranowski et al., Games for health for children—Current status and needed research. Games for Health Journal, 5(1), 2016, 1-12. DOI: 10.1089/g4h.2015.0026

Jurado, M., Rosselli, M., The elusive nature of executive functions: a review of our current understanding. Neuropsychology review, 17(3), 2007, 213-233. DOI: 10.1007/s11065-007-9040-z

Landis, J.R., Koch, G.G., The measurement of observer agreement for categorical data. Biometrics, 2000, 159–174.

Likert, R., A technique for the measurement of attitudes. Archives of Psychology. New York: Columbia University Press, 1932.

Mansur-Alves, M; Saldanha-Silva, R., Does Working Memory Training Promote Changes in Fluid Intelligence? Trends in Psychology, 25 (2), 2017,787-807. DOI: 10.9788/TP2017.2-19En

Monette, S. et al. The role of the executive functions in school achievement at the end of Grade 1. Journal of experimental child psychology, 109(2), 2011, 158-173. DOI: 10.1016/j.jecp.2011.01.008

Mossmann, J. B. et al, 2017. Evaluation of the Usability and Playability of an Exergame for Executive Functions Stimulation and Its Development Process. Lecture Notes in Computer Science, v. 10275, p. 164. DOI: 10.1007/978-3-319-58472-0_14

Mossmann, J., in press. Exergames Como Mediadores Da Estimulação De Componentes Das Funções Executivas Em Crianças Do Ensino Fundamental I. Tese de Doutorado em Informática na Educação. Universidade Federal do Rio Grande do Sul. Programa de Pós-Graduação em Informática na Educação – PPGIE, UFRGS, Brasil.

Müller, U. et al., The effect of labeling on preschool children's performance in the Dimensional Change Card Sort Task. Cognitive Development, 23(3), 2008, 395-408. DOI: 10.1016/j.cogdev.2008.06.001

Posner, K. et al., Measuring interrater reliability among multiple raters: an example of methods for nominal data. Statistics in medicine, 9(9), 1990, 1103-1115.

R Core Team., R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. 2018, URL <https://www.R-project.org/>.

Toll, S. et al., Executive functions as predictors of math learning disabilities. Journal of Learning Disabilities, 44(6), 2011, 521-532. DOI: 10.1177/0022219410387302

Tschang, T., Videogames as interactive experiential products and their manner of development. International Journal of Innovation Management, 9(1), 2005, 103-131. DOI: 10.1142/S1363919605001198

Welsh et al., The development of cognitive skills and gains in academic school readiness for children from low-income families. Journal of Educational Psychology, 102(1), 2010, 43-53. DOI: 10.1037/a0016738