From sensemaking to school improvement? Exploring educational professionals' use of school performance feedback

Purposeful and systematic data use has a positive impact on the quality of educational decisions  (Rossi et al., 2004). However, research has shown that Flemish schools make limited use of pupil outcome data, benchmarks, standardized assessment and feedback from scientific research in order to shape their policy and practice (Van Gasse et al., 2015; Vanlommel et al., 2017). In this doctoral project, we investigated to what extent and in what manner Flemish educational professionals use school performance feedback (SPF) for school development, with their use of SPF from the Flemish national assessments and parallel tests as a specific case. The project ran parallel in part to user research conducted at STEP – the Flemish policy research center for test development and assessments.

The dissertation has a theoretical component and an empirical component. By employing conceptual, quantitative and qualitative methods, we studied enablers and barriers to SPF use, both broadly and in depth. A central point of departure was the insight that, in line with other types of data use, SPF use is influenced by factors pertaining to the data users themselves, the data system, the school organization and the educational context. In order to zoom in on mechanisms that come into play when teachers and school leaders engage with SPF, we undertook an extensive conceptual exploration of the notion of ‘sensemaking’ (Bertrand & Marsh, 2015; Datnow et al., 2012; Klein et al., 2007; Schildkamp, 2019; Weick, 1995). This sensemaking perspective served as a lens to shed light on aspects of data use and data literacy. Based on interview data, we zoomed in on the user validity (Kane, 2013; MacIver et al., 2014) of authentic SPF. We propose that misconceptions pertaining to SPF reports can be explained in part by a disconnect between SPF providers’ and SPF users’ frames of reference. We also explored teachers’ and school leaders’ attributions (Wang & Hall, 2018; Weiner, 1985) of school performance. Wide-ranging attributions, and differences in attribution according to work role, illustrate that formulating a diagnosis is not straightforward. Furthermore, based on survey data and on the Theory of Planned Behavior (Ajzen, 1991), we looked at the interplay of user-level and school-level factors associated with SPF use. The perception of a shared goal orientation on school level emerges as an important driver.

By examining how teachers and school leaders engage with SPF from external standardized assessments, this dissertation is at the nexus of educational effectiveness and school improvement research. Central findings illustrate the importance of engaging users in SPF design, rethinking the concept of data literacy, and stimulating collective sensemaking in school teams.

References:

Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50(2), 179–211.

Bertrand, M., & Marsh, J. A. (2015). Teachers’ sensemaking of data and implications for equity. American Educational Research Journal, 52(5), 861–893.

Datnow, A., Park, V., & Kennedy-Lewis, B. (2012). High school teachers’ use of data to inform instruction. Journal of Education for Students Placed at Risk (JESPAR), 17(4), 247–265.

Kane, M. (2013). The argument-based approach to validation. School Psychology Review, 42(4), 448–457.

Klein, G., Phillips, J. K., Rall, E. L., & Peluso, D. a. (2007). A data-frame theory of sensemaking. In R. R. Hoffman (Ed.), Expertise Out of Context - Proceedings of the Sixth International Conference on Naturalistic Decision Making (pp. 113–155). Lawrence Erlbaum Associates.

MacIver, R., Anderson, N., Costa, A.-C., & Evers, A. (2014). Validity of Interpretation: A user validity perspective beyond the test score. International Journal of Selection and Assessment, 22(2), 149–164.

Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2004). Evaluation: A Systematic Approach. (7th ed.). Sage Publications.

Schildkamp, K. (2019). Data-based decision-making for school improvement: Research insights and gaps. Educational Research, 61(3), 257–273.

Van Gasse, R., Vanhoof, J., Mahieu, P., & Van Petegem, P. (2015). Informatiegebruik door schoolleiders en leerkrachten. Garant.

Vanlommel, K., Van Gasse, R., Vanhoof, J., & Van Petegem, P. (2017). Teachers’ decision-making: Data based or intuition driven? International Journal of Educational Research, 83, 75–83.

Wang, H., & Hall, N. C. (2018). A systematic review of teachers’ causal attributions: Prevalence, correlates, and consequences. Frontiers in Psychology, 9(DEC), 1–22.

Weick, K. E. (1995). Sensemaking in Organizations. Sage Publications.

Weiner, B. (1985). An attributional theory of achievement motivation and emotion. Psychological Review, 92(4), 548–573.