Testing Educator Beliefs About At-Risk Students: Our Journey in Using Machine Learning and Big Data
DEBORAH LAU & JOANNE ROSS | CHARTERED ACCOUNTANTS AUSTRALIA AND NEW ZEALAND
Over 10,000 students enrol in the Chartered Accountants Australia and New Zealand (CAANZ) Graduate Diploma of Chartered Accounting program every term. These students come from all walks and stages of life, via multiple entry pathways, and undertake the program whilst working full time.Passing the exams is an important career milestone for our students; becoming a Chartered Accountant means better career, business and salary opportunities.In our search for a better, more efficient way to help our students succeed, we decided to experiment with ‘machine learning’ by using an artificial neural network to predict whether students are likely to pass their current subject.We’ll share what we learned in our journey to tackle the following challenges:
- Testing closely held beliefs about what contributes to student success against actual data and trying to move away from the “rule of thumb” to the “rule of data”
- A manual approach to identifying at-risk students
- Capturing a large amount of data across different systems but lacking a systematic way to bring everything together
- Working within the constraints of IT policies and the new privacy requirements
Some of the themes and questions we’ll explore are:
- Which learning materials really make a difference in student success?
- Is it true that a diverse cohort requires a diverse range of learning materials?
- How can we allocate resources to maximise our “return on investment” as measured by student success?
- Can a computer be better at predicting at-risk candidates compared to a human?
- Can machine learning help us effectively evaluate and account for individual differences across a large cohort?
- What knowledge and skills do you really need to attempt something like this?
- How can we achieve this using the “out of the box” version of Blackboard?
- How can we make sure results are interpreted – and decisions implemented – in a methodical and rigorous manner?
Data and model considerations
- How do we design an algorithm that is valid, robust and practical to implement?
- How do you sift through the ever-increasing volume and types of data available to find the variables that are truly associated with student success?
- How long does it really take to identify, capture, and verify your data is valid, complete and accurate?
- Does more data give you predictions that are more accurate?
- How do we recalibrate our models through machine learning to incorporate cohort specific differences?
Session takeaways will include:
- A practical roadmap for implementing similar projects at your institution
- Access to the actual machine learning algorithm code we used, that can be adapted for your project
- A list of additional resources/reading materials