By Andrew Jaffrey (Head of the Office for Digital Learning, Ulster University)
Ulster University, Northern Ireland’s civic University, is the first institution in the European Union to implement Blackboard Predict (Predict), a predictive analytics solution which uses historical student data, and outcomes, to identify and compare characteristics with the current cohort of students.
Historically Ulster has been involved in a UK-wide higher education project published in 2013 called What works? Supported by the Paul Hamlyn Foundation, the initiative examined how higher education providers can improve student retention and success. Ulster had a number of case studies aligned to this work, and some of the strategies that emerged from these were building engagement through partnerships between staff and students, promoting peer support and creating a sense of belonging, among others.
As a result, the university has taken a strategic, long-term, longitudinal approach to student retention, and Predict is seen as an enhancement to these strategies providing more timely data-based decision making to support interventions.
Ulster’s predictive model helps answer one simple question “"Is this student at-risk of failing this module or leaving the course at this point in time?". It is hoped that the predictive aspect will enhance existing descriptive data analysis at Ulster which asks the question "How many students have failed or left the course?" after the event has occurred. This difference in the implementation of a learning analytics project is significant and is reflective in a shift in thinking within the project board that is supporting the pilot. Originally Ulster’s work in learning analytics was to be focussed on descriptive and diagnostic analytics specifically interpreting the use of learning technologies within modules and programmes. The board felt that an investment in a student focussed project would be more beneficial and the project board began to explore predictive solution which are based around individual students rather than specific IT systems.
Predict uses data from the Student Records Information System (SRIS) and combines it with interactions in Blackboard Learn to generate a predictive model based on four years of historical data including demographic information, tariff point entry scores and results of formative and summative assessment. There are approximately 200 data fields describing each student. 20% of the historical data used to create the model was held back for testing. As the academic results and outcomes of the remaining 20% were known, this was an accurate test of the model's accuracy.
The dashboards in Predict use a percentage score which relates to the likelihood of a student achieving a grade of 50% or above. Historical analysis shows that less than 3.1% of students scored 40 or below, rising to 14.7% for 50% or below. The 50% grade cut off was therefore more significant as an 'at-risk' indicator and can support targeted interventions with students close to fail boundaries.
The Predict solution was specified and delivered from within the Education portfolio at the University and a primary motivation for the project was to better understand our data, within the Education portfolio, and to make sure that assumptions about our data were accurate. Our project sponsor, the PVC Education, identified the project as an opportunity to encourage discussion about data based decision making and to explore tacit assumptions and barriers to using data in different ways within the portfolio.
A large part of the project has been understanding our data better and data cleansing, making sure that the data we were sending to the predictive model was accurate. The discussions that have resulted from learning more about our data have changed the way we think within the Education portfolio and have accelerated other data based projects within specific Faculties and Schools.
The project was established in Summer 2017 and has been institutionally available since Spring 2018, it is unlikely that we will see measurable benefits until academic year 2018/19 but the project has supported new intervention discussions, and designs, within areas that have high retention rates.
Predict is very much seen as something that starts a conversation with a student. So academic colleagues are intervening in many different ways across the institution. Data is not seen as a definitive answer, but as a starting point. We want to learn more about our data and have more conversations about it.
People have many different views about the ethics of learning analytics and privacy. Learning analytics sits on top of deeper layers of assumptions and beliefs that academic colleagues have. With any data based project, we should recognise that data is imperfect and we must treat that data with humility, because it is only giving us a narrow picture and a narrow view of what's going on with a human being. There are lots of external factors going on with a student, well beyond anything that we can hope to measure within an institution. Recognising this, having the discussions and having a realistic expectation for the benefits of a predictive analytics solution has helped the project become established at Ulster.