A technique for interrater reliability evaluation of a mobile game aimed for executive functions stimulation
DOI:
https://doi.org/10.31686/ijier.vol7.iss5.1518Keywords:
Executive Functions, Inhibitory Control, Game Evaluation, Cognitive Stimulation, Digital GamesAbstract
This work presents an interrater reliability evaluation of a mobile game aimed for the executive function’s stimulation, specifically the inhibitory control. The educational Exergame “The Incredible Adventures of Apollo & Rosetta in the Space” was presented in a previous piece of work, which showed the development of the game and its application in a researching intervention with children, in the 6 to 10 age-group, in a school environment. Subsequently, the game had its code reworked for being able to be used across different platforms, hence culminating in the present work. The methodology in this paper consists in a mixed qualitative-quantitative evaluation through questionnaires with four domain professionals experienced in the executive functions field. The statistical measurement used was based on the Kappa coefficient and average percentage among the judges. As the results indicate, there was a substantial agreement (k=0,659; P-value=0,000) between the raters, as well as a high percentage of agreement in general on the mobile game’s capability of Executive Functions stimulation for children.
Downloads
Download data is not yet available.
References
Baba, Y. & Tschang, F., Product development in japanese tv game software: The case of an innovative game. International Journal of Innovation Management, 05(04), 2001, 487–515. DOI: 10.1142/S1363919601000464
BARBOSA, D. et al, 2018. Adaptation of an educational Exergame to mobile platforms: A development process. Communications in Computer and Information Science (PRINT), v. 1, p. 287-298. DOI: 10.1007/978-3-319-95522-3_24
Bernier, A. et al., From external regulation to self-regulation: Early parenting precursors of young children’s executive functioning. Child development, 81(1), 2010, 326-339. DOI: 10.1111/j.1467-8624.2009.01397.x
Buelow, M.; et al., A. The influence of video games on executive functions in college students. Computers in Human Behavior, 45, 2015, 228-234. DOI: 10.1016/j.chb.2014.12.029
Bull, R. et al., A comparison of performance on the Towers of London and Hanoi in young children. Journal of Child Psychology and Psychiatry, 45(4), 2004, 743-754. DOI: 10.1111/j.1469-7610.2004.00268.x
Bull, R. et al., Short-term memory, working memory, and executive functioning in preschoolers: Longitudinal predictors of mathematical achievement at age 7 years. Developmental neuropsychology, 33(3), 2008, 205-228. DOI: 10.1080/87565640801982312
Carlson, S. et al., Individual differences in executive functioning and theory of mind: An investigation of inhibitory control and planning ability. Journal of experimental child psychology, 87(4), 2004, 299-319. DOI: 10.1016/j.jecp.2004.01.002
Chen, P., Krauss, A., Interrater agreement. The sage encyclopedia of social science research methods, 2, 2004, 511-513. DOI: 10.4135/9781412950589.n444
Christopher, M. et al., Predicting word reading and comprehension with executive function and speed measures across development: A latent variable analysis. Journal of Experimental Psychology: General, 141(3), 2012, 470. DOI: 10.1037/a0027375
Conklin, H. et al., Working memory performance in typically developing children and adolescents: Behavioral evidence of protracted frontal lobe development. Developmental neuropsychology, 31(1), 2007, 103-128. DOI: 10.1080/87565640709336889
de Jong., Effects of training working memory in adolescents with a below average IQ. Workshop on Enhancing Executive Functions in Education in Nijmegen, May 20, 2014 Nijmegen, NL.
Diamond, A., & Lee, K., Interventions shown to aid Executive Function development in children 4 to 12 years old. Science, 333(6045), 2011, 959-964. DOI: 10.1126/science.1204529
Diamond, A., Executive functions. Annual review of psychology, 64. 2013, 135-168. DOI: 10.1146/annurev-psych-113011-143750
Diamond, A; Ling, D., Conclusions about interventions, programs, and approaches for improving executive functions that appear justified and those that, despite much hype, do not. Developmental cognitive neuroscience, 18. 2016, 34-48. DOI: 10.1016/j.dcn.2015.11.005
Fleiss, J. L.; Cohen, J.M., The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and psychological measurement, 33(3), 1973, 613-619. DOI: 10.1177/001316447303300309
Holmes, J.; et al., Adaptive training leads to sustained enhancement of poor working memory in children. Developmental science, 12(4), 2009, 1-7. DOI: 10.1111/j.1467-7687.2009.00848.x
Hughes, C. & Ensor, R., Executive function and theory of mind: Predictive relations from ages 2 to 4. Developmental psychology, 43(6), 2007, 1447. DOI: 10.1037/0012-1649.43.6.1447
Baranowski et al., Games for health for children—Current status and needed research. Games for Health Journal, 5(1), 2016, 1-12. DOI: 10.1089/g4h.2015.0026
Jurado, M., Rosselli, M., The elusive nature of executive functions: a review of our current understanding. Neuropsychology review, 17(3), 2007, 213-233. DOI: 10.1007/s11065-007-9040-z
Landis, J.R., Koch, G.G., The measurement of observer agreement for categorical data. Biometrics, 2000, 159–174.
Likert, R., A technique for the measurement of attitudes. Archives of Psychology. New York: Columbia University Press, 1932.
Mansur-Alves, M; Saldanha-Silva, R., Does Working Memory Training Promote Changes in Fluid Intelligence? Trends in Psychology, 25 (2), 2017,787-807. DOI: 10.9788/TP2017.2-19En
Monette, S. et al. The role of the executive functions in school achievement at the end of Grade 1. Journal of experimental child psychology, 109(2), 2011, 158-173. DOI: 10.1016/j.jecp.2011.01.008
Mossmann, J. B. et al, 2017. Evaluation of the Usability and Playability of an Exergame for Executive Functions Stimulation and Its Development Process. Lecture Notes in Computer Science, v. 10275, p. 164. DOI: 10.1007/978-3-319-58472-0_14
Mossmann, J., in press. Exergames Como Mediadores Da Estimulação De Componentes Das Funções Executivas Em Crianças Do Ensino Fundamental I. Tese de Doutorado em Informática na Educação. Universidade Federal do Rio Grande do Sul. Programa de Pós-Graduação em Informática na Educação – PPGIE, UFRGS, Brasil.
Müller, U. et al., The effect of labeling on preschool children's performance in the Dimensional Change Card Sort Task. Cognitive Development, 23(3), 2008, 395-408. DOI: 10.1016/j.cogdev.2008.06.001
Posner, K. et al., Measuring interrater reliability among multiple raters: an example of methods for nominal data. Statistics in medicine, 9(9), 1990, 1103-1115.
R Core Team., R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. 2018, URL <https://www.R-project.org/>.
Toll, S. et al., Executive functions as predictors of math learning disabilities. Journal of Learning Disabilities, 44(6), 2011, 521-532. DOI: 10.1177/0022219410387302
Tschang, T., Videogames as interactive experiential products and their manner of development. International Journal of Innovation Management, 9(1), 2005, 103-131. DOI: 10.1142/S1363919605001198
Welsh et al., The development of cognitive skills and gains in academic school readiness for children from low-income families. Journal of Educational Psychology, 102(1), 2010, 43-53. DOI: 10.1037/a0016738
BARBOSA, D. et al, 2018. Adaptation of an educational Exergame to mobile platforms: A development process. Communications in Computer and Information Science (PRINT), v. 1, p. 287-298. DOI: 10.1007/978-3-319-95522-3_24
Bernier, A. et al., From external regulation to self-regulation: Early parenting precursors of young children’s executive functioning. Child development, 81(1), 2010, 326-339. DOI: 10.1111/j.1467-8624.2009.01397.x
Buelow, M.; et al., A. The influence of video games on executive functions in college students. Computers in Human Behavior, 45, 2015, 228-234. DOI: 10.1016/j.chb.2014.12.029
Bull, R. et al., A comparison of performance on the Towers of London and Hanoi in young children. Journal of Child Psychology and Psychiatry, 45(4), 2004, 743-754. DOI: 10.1111/j.1469-7610.2004.00268.x
Bull, R. et al., Short-term memory, working memory, and executive functioning in preschoolers: Longitudinal predictors of mathematical achievement at age 7 years. Developmental neuropsychology, 33(3), 2008, 205-228. DOI: 10.1080/87565640801982312
Carlson, S. et al., Individual differences in executive functioning and theory of mind: An investigation of inhibitory control and planning ability. Journal of experimental child psychology, 87(4), 2004, 299-319. DOI: 10.1016/j.jecp.2004.01.002
Chen, P., Krauss, A., Interrater agreement. The sage encyclopedia of social science research methods, 2, 2004, 511-513. DOI: 10.4135/9781412950589.n444
Christopher, M. et al., Predicting word reading and comprehension with executive function and speed measures across development: A latent variable analysis. Journal of Experimental Psychology: General, 141(3), 2012, 470. DOI: 10.1037/a0027375
Conklin, H. et al., Working memory performance in typically developing children and adolescents: Behavioral evidence of protracted frontal lobe development. Developmental neuropsychology, 31(1), 2007, 103-128. DOI: 10.1080/87565640709336889
de Jong., Effects of training working memory in adolescents with a below average IQ. Workshop on Enhancing Executive Functions in Education in Nijmegen, May 20, 2014 Nijmegen, NL.
Diamond, A., & Lee, K., Interventions shown to aid Executive Function development in children 4 to 12 years old. Science, 333(6045), 2011, 959-964. DOI: 10.1126/science.1204529
Diamond, A., Executive functions. Annual review of psychology, 64. 2013, 135-168. DOI: 10.1146/annurev-psych-113011-143750
Diamond, A; Ling, D., Conclusions about interventions, programs, and approaches for improving executive functions that appear justified and those that, despite much hype, do not. Developmental cognitive neuroscience, 18. 2016, 34-48. DOI: 10.1016/j.dcn.2015.11.005
Fleiss, J. L.; Cohen, J.M., The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and psychological measurement, 33(3), 1973, 613-619. DOI: 10.1177/001316447303300309
Holmes, J.; et al., Adaptive training leads to sustained enhancement of poor working memory in children. Developmental science, 12(4), 2009, 1-7. DOI: 10.1111/j.1467-7687.2009.00848.x
Hughes, C. & Ensor, R., Executive function and theory of mind: Predictive relations from ages 2 to 4. Developmental psychology, 43(6), 2007, 1447. DOI: 10.1037/0012-1649.43.6.1447
Baranowski et al., Games for health for children—Current status and needed research. Games for Health Journal, 5(1), 2016, 1-12. DOI: 10.1089/g4h.2015.0026
Jurado, M., Rosselli, M., The elusive nature of executive functions: a review of our current understanding. Neuropsychology review, 17(3), 2007, 213-233. DOI: 10.1007/s11065-007-9040-z
Landis, J.R., Koch, G.G., The measurement of observer agreement for categorical data. Biometrics, 2000, 159–174.
Likert, R., A technique for the measurement of attitudes. Archives of Psychology. New York: Columbia University Press, 1932.
Mansur-Alves, M; Saldanha-Silva, R., Does Working Memory Training Promote Changes in Fluid Intelligence? Trends in Psychology, 25 (2), 2017,787-807. DOI: 10.9788/TP2017.2-19En
Monette, S. et al. The role of the executive functions in school achievement at the end of Grade 1. Journal of experimental child psychology, 109(2), 2011, 158-173. DOI: 10.1016/j.jecp.2011.01.008
Mossmann, J. B. et al, 2017. Evaluation of the Usability and Playability of an Exergame for Executive Functions Stimulation and Its Development Process. Lecture Notes in Computer Science, v. 10275, p. 164. DOI: 10.1007/978-3-319-58472-0_14
Mossmann, J., in press. Exergames Como Mediadores Da Estimulação De Componentes Das Funções Executivas Em Crianças Do Ensino Fundamental I. Tese de Doutorado em Informática na Educação. Universidade Federal do Rio Grande do Sul. Programa de Pós-Graduação em Informática na Educação – PPGIE, UFRGS, Brasil.
Müller, U. et al., The effect of labeling on preschool children's performance in the Dimensional Change Card Sort Task. Cognitive Development, 23(3), 2008, 395-408. DOI: 10.1016/j.cogdev.2008.06.001
Posner, K. et al., Measuring interrater reliability among multiple raters: an example of methods for nominal data. Statistics in medicine, 9(9), 1990, 1103-1115.
R Core Team., R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. 2018, URL <https://www.R-project.org/>.
Toll, S. et al., Executive functions as predictors of math learning disabilities. Journal of Learning Disabilities, 44(6), 2011, 521-532. DOI: 10.1177/0022219410387302
Tschang, T., Videogames as interactive experiential products and their manner of development. International Journal of Innovation Management, 9(1), 2005, 103-131. DOI: 10.1142/S1363919605001198
Welsh et al., The development of cognitive skills and gains in academic school readiness for children from low-income families. Journal of Educational Psychology, 102(1), 2010, 43-53. DOI: 10.1037/a0016738
Downloads
Published
2019-05-01
Issue
Section
Journal Articles
License
Copyright (c) 2019 Bernardo Cerqueira, Debora Nice Ferrari Barbosa, João Mossmann, Jorge Barbosa
This work is licensed under a Creative Commons Attribution-NoDerivatives 4.0 International License.
Copyrights for articles published in IJIER journals are retained by the authors, with first publication rights granted to the journal. The journal/publisher is not responsible for subsequent uses of the work. It is the author's responsibility to bring an infringement action if so desired by the author for more visit Copyright & License.
How to Cite
Cerqueira, B., Nice Ferrari Barbosa, D., Mossmann, J., & Barbosa, J. (2019). A technique for interrater reliability evaluation of a mobile game aimed for executive functions stimulation. International Journal for Innovation Education and Research, 7(5), 338-354. https://doi.org/10.31686/ijier.vol7.iss5.1518
