Log in to create and rate content, and to follow, bookmark, and share content with other members. Not a member? Join Now!

Measuring impact of course design

Discussion created by djagger on Jan 17, 2019
Latest reply on Aug 31, 2019 by smachaje

Like • Show 1 Like1
Comment • 10

Hi,

Within our faculty we've been redesigning course units based on Blackboard Exemplar Course Design Rubric and now I'm trying to measure the impact of that work.

I'm essentially measuring a poorly designed course ("a file dump") with a well designed course (descriptive elements, signposting etc)...With minimal changes to the actual learning content.

Just looking for any ideas and suggestions from anyone who has attempted to use BB statistics/Course Reports to measure student engagement and/or Course Design Effectiveness.

Thanks in advance!
dan.

Outcomes

10 Replies

gw0046349 Jan 17, 2019 10:47 AM
I believe there is a rubric. Wouldn't that help?
Blackboard Exemplary Course Program Rubric
1 person found this helpful
Like • Show 1 Like1
Actions
- djagger @ gw0046349 on Jan 17, 2019 10:52 AM
  Hi Gail,
  
  Thanks, we've already built the courses using the BB Exemplar Rubrics and other similar tools. We're now trying to measure the imapct of that work.
  We're using Unit surveys as one method of measurement, we plan to run some focus groups, but I'm interested if anyone has tried to measure engagement/impact of course design by using Blackboard Stats.
  1 person found this helpful
  Like • Show 1 Like1
  Actions
sm0077250 Jan 23, 2019 5:29 PM
G'day Dan ...
Just clarify what you mean by 'Blackboard Stats'.
And Course Reports - these are the set of reports within a Learn site, located within the Evaluation link? If so..... we, along with other uni's have found these reports to be suspect in their data integrity. And in actually trying to work out what the data means.

Happy to discuss further, for what you're trying to do - establish an evidence based way for gauging learning design changes - is a central matter in assessing program quality.

Stewart McKinney
Adaptive Learning and Teaching Project Officer, Learning Technologies | Division of Learning and Teaching
Charles Sturt University, Dubbo Campus
Tel: +61 2 6885 7304 Email: smckinney@csu.edu.au
1. www.csu.edu.au/division/learning-and-teaching
1 person found this helpful
Like • Show 1 Like1
Actions
- djagger @ sm0077250 on Jan 24, 2019 7:07 AM
  Hi Stewart,
  
  Yes that's what I mean. Also...we're aware of the suspect nature of the data, but we're hoping that we could still compare datasets on a yearly comparison.
  
  We're working on the assumption that a well designed Blackboard space enhances the student experience, but struggling to identify how this enhancement would manifest itself in measurable data.
  
  Well Designed site = Students use Blackboard more = more user "hits"
  but...
  Poorly Designed site = Students can't find documents (students click around) = more user "hits"
  
  Does that make sense? Just curious if anyone has struggled with this and what (if any) uses/conclusions they came to.
  
  FYI... BB Data is only a subset of the info we're using to measure the impact of the redesign.
  
  Regards,
  dan.
  1 person found this helpful
  Like • Show 1 Like1
  Actions
  - sm0077250 @ djagger on Jan 29, 2019 6:37 PM
    G'day Dan ...
    
    “…working on the assumption that a well designed Blackboard space enhances the student experience …”
    This idea seems to follow the “Build it and they will come” mantra, which is flawed at several points. This is not to say we should not invest time and creativity in building course spaces. We definitely need to, of course.
    What data to use to measure ‘enhancement’ is, as you say, more than a conundrum. More (or less) accesses and more (or less) interaction do not necessarily translate into anything meaningful. Using surveys and focus groups to gain user experience and personalised feedback will prove invaluable towards building a picture of end-user satisfaction and preferences, which you will need to move forward with anything. BTW… We framed enhancing student experience holistically, to include all aspects of student life at the uni, not just Blackboard course sites. Student perceptions regarding Bb course sites was just one touchpoint for assessing student experience surveying – an on-going aspect of assuring quality of student experience at the uni.
    
    I think there is another priority issue here, in all of this. We have found that, despite our course spaces/sites being everything they need to be – quality content, use of multimedia, intuitive navigation, creative design, user informed – we still have students who do not engage in our courses. In essence, we have separated the two things: enhancing student experience and assessing student engagement. The latter, we feel, is more dependent upon the learning design in the unit of learning. This is currently where we are focussing attention, with determining what measureable data will most provide the insights we need.
    Trust something might gel for you
    Ciao …. Stewart
    2 people found this helpful
    Like • Show 1 Like1
    Actions
    - djagger @ sm0077250 on Jan 30, 2019 4:46 AM
      Hi Stewart thanks for this... Many echos of our own thinking here!
      1 person found this helpful
      Like • Show 1 Like1
      Actions
dmartin Jan 30, 2019 9:17 PM
Our university has run a few improvement projects that sound similar to yours. I don't create the analytics but do have access to a dashboard that compared file sizes, folder depth, items accessed etc. Might give you some ideas.
2 people found this helpful
Like • Show 1 Like1
Actions
ian.craine Jan 31, 2019 2:03 PM
Hi Dan,

You have come up against a hard problem most of us are facing.
I agree with Stewart that the analytic metrics provided by Blackboard regarding student interactions, minutes, downloads, etc. can be relatively meaningless when it comes to measuring the quality of an online course.
We have found a lot of variation in course metrics within courses, regardless of their quality. For example, some of the best and most engaged students actually spend very little time or activity in their Blackboard course, often because they download course materials and interact with them offline (or are they just very fast at absorbing the material?). Ironically, some of the least successful students spend more time in Blackboard, perhaps because they are struggling more with poorly arranged material (in a bad course) or perhaps because they need more time and practice to absorb it (in a good course). Of course, there are many more potential explanations for each scenario. As a result we have found it nearly impossible to use Blackboard's course metrics to give us any meaningful insights into the quality of an online course beyond the obvious.

Rather than measuring aspects of student activity within a course, we find that measuring student outcomes (e.g. grades) and levels of satisfaction (via student surveys) gives us a better picture of online course quality. We then compare these results with formal course reviews of online course quality using a Quality Matters tool. They generally are in agreement, but the quality of the professor in the online course can really affect the results for students (i.e. even in a badly designed course, an enthusiastic engaged prof can make up for a lot of problems).

And finally, like all comparison studies, you need quite large sample sizes to detect real differences. In the case of online course comparisons, the sample unit is the course, not the number of students, so you will need many replicates of the same course offering in your study which is hard to do at most schools, including ours.
Cheers,
Ian

Ian Craine
Professor and Instructional Designer
George Brown College, Toronto, Canada
3 people found this helpful
Like • Show 2 Likes2
Actions
ah0040220 Feb 6, 2019 9:13 PM
Hi Dan,
Check out this on Sys Admin community Using Blackboard Goals to Monitor Curriculum and for Accreditation .
It works retrospectively so if you map the goals in the courses you can get a student progress with and without your design changes and attribute change to these changes.
In a similar way with Analytics you could look at the formative and summative assessment grade as measure of student success comparing previous (non-redesign) cohorts to the re-designed looking at the normal curve and looking for significant changes (move in the standard deviation).
2 people found this helpful
Like • Show 2 Likes2
Actions
smachaje Aug 31, 2019 10:02 PM
It's important to remember that good pedagogy does not necessarily require full use of the LMS. However, we can compare courses and group them based on course design:
Evaluate adoption of your LMS with course design groups

I would recommend the 21 Days of Blackboard Data course, where a number of SQL assignments deal with student performance, activity, etc. Blackboard Data holds a promise that we can create SQL to evaluate student activity or instructor activity in a systematic way. Within the same course this tool will help us to compare student activity in an objective way. Within the same course we can identify student engagement, predict failure, and do a lot more.
Like • Show 0 Likes0
Actions

Measuring impact of course design

Outcomes

Related Content

Recommended Content