Designing and Evaluating Student-facing Learning Dashboards: Lessons Learnt

Through the rise of on-line education, an abundance of learner data is generated and gathered. While Educational Data Mining provides insights algorithmically to better understand students, Learning Analytics (LA) attempts to leverage these traces to empower learners by increasing motivation, autonomy, effectiveness, and efficiency. One method to achieve this empowerment is that of Learning Dashboards, a personal informatics or Quantified Self approach, helping learners self-reflect and gain self-knowledge through the visualisation of these traces. 

Learning Dashboards are a welcome and much needed topic that shifts the research focus towards, and actively involves, teachers, advisers, and students, providing them with insights into behaviour of learners at both individual (a student) and group level (peer activities in courses, study programs, institutions).  Investigating the potential impact of visualising these traces is important, but such research demands long-term deployments in realistic settings. Such an endeavour is challenging, as it requires commitment from institutions, teachers, and learners of (usually) unproven technology during sensitive, life-impacting situations, as well as running the risk of discovering problems with the dashboard designs after commitment, which raises ethical issues. Our research attempts to provide some leverage for such deployments by i) providing evidence of the perceived benefits and ii) providing design guidelines required to create useful and meaningful dashboards. To explore the required design choices, we take an iterative, design-based research approach, in close collaboration with experts, teachers, and students.

The work starts by tackling following research questions: "How should we visualise learner data to support students to explore the path from effort to outcomes? (RQ1)", and "How can we promote students, inside and outside the classroom, to actively explore this effort to outcomes path? (RQ2)." To explore these questions, we have designed, deployed, and evaluated five learning dashboards in blended learning environments. This research resulted in several guidelines on how to visualise the LA data and how to promote exploration of students' efforts to outcomes. These lessons cover topics such as abstraction to deal with the abundance of data, facilitating easy access to learner artefacts and feedback, and integrating Learning Dashboards into the work-flow for better user acceptance.

From our research, we noticed potential in the design of collaborative Learning Dashboards, and further explored possible scenarios that could benefit from this approach in two case studies: live dashboards to orchestrate feedback activities in the classroom, and support of the dialogue during advising sessions with students. 

The first case study focuses on the following questions: "What are the design challenges for ambient Learning Dashboards to promote balanced group participation in classrooms, and how can they be met? (RQ3)", and "Are ambient Learning Dashboards effective means for creating balanced group participation in classroom settings? (RQ4)". Exploring these questions resulted in a Learning Dashboard that raises activity awareness, activates students, and assists with classroom orchestration. We learn that it is important to visualise the data in a neutral way, toning down over-participators to leave room for under-participators. 

The second case study explores how a collaborative Learning Dashboard can assist both student and study adviser during advising session, and addresses the following research questions: "What are the design challenges for creating a Learning Dashboard to support study advice sessions, and how can they be met? (RQ5)", and "How does such a Learning Dashboard contribute to the role of the adviser, student, and dialogue? (RQ6)". Our design and evaluation process reveal that a passive, supportive dashboard can assist in guiding the student-study adviser conversation, provide further insights and new perspectives, and help convince students of taking specific actions.