Skip navigation
All People > tjatkin > CCO Blog > 2019 > January

Quality Matters - Part II

Posted by tjatkin Jan 8, 2019

Happy New Year everyone! I hope you had a nice holiday break with family and friends. My husband and I were lucky to visit Quebec City for the first time over the holidays – it was cold, and awesome, and we can’t wait to go back! As our paths cross this year, remind me to share stories of our adventures.


As many of you know, to close out 2018, I used my blog to introduce you to Ashley Hunsberger, Blackboard’s Quality Architect. As promised, here is Part 2 to that blog post, where she provides greater insight into our testing process, the programs to gather client feedback and how Blackboard strives diligently to deliver quality software.


1. What are we doing at Blackboard to deliver quality products?

Our work begins with having a rigorous and informed approach. There are several tenets that I think are critical to our approach for driving improvements in quality – and by quality, I mean that our products work well, and that they work in ways our clients want them to work. We’ve been working hard this past year to make process improvements to our approach to quality. We know we need to understand what’s important to you and then test our work to make sure we get there!


We put the user first. We ask ourselves, how will they use the product? What will that experience be like for them? We know that automation cannot tell us how our product makes the user feel. What we can do is learn from you, understand how you expect what we build to behave, and explore the system accordingly to make sure it reflects your desires and incorporates your feedback.


We can’t test quality in. This means that even if all testing is complete, you won't necessarily have a quality product at the end of the day. Testing is merely feedback, telling us what the impact of a code change is on the system. What we CAN do is get that feedback as fast as possible and prevent known errors from traveling any further downstream. To do that though, we need the right feedback at the right time. We do that by understanding our business goals as well as the corresponding questions we must answer and taking a risk-based approach to testing since, as unfair as it may seem, we can’t test everything. We identify the areas where feedback is critical after every single change and then automate those areas as much as possible. Doing this enables us to have better results faster AND it frees us to explore our product in other ways – such as human-based testing.


We believe in small, incremental changes. It is much easier to get the feedback we need on very small changes than it is to leave testing to the end of massive, sweeping changes that could take months. This is a shift in software development over the last several years that, frankly, makes our lives easier in development. We must continue to view quality and testing as an activity, not a role, and take a whole team approach. We are all in this together and want the same thing - a wonderful experience for our users.


2. What are some of the results or process improvements that have been made over the past year?

When we started our risk-based testing approach, we were able to invest in automation in high-risk areas by evaluating the likelihood we thought something could occur and the impact if it did. Once we were able to get testing in place for those higher risk areas, we introduced something called gates. What this means is, if a test fails, it will automatically fail the build and won’t go to production. This is really important, driving higher quality.


Of course, since we can’t test everything, some things may unfortunately make it through. When we don’t capture something during multiple facets of testing, we do root-cause analysis to understand why it happened, and then we look at our testing to ensure that we could find the issue with our testing to avoid it happening again.



3. What are some ways we gather client input into this process?

For Learn Original, we offer a Tech Preview program twice a year that allows self-hosted and managed-hosted clients to test the release right before launch. We had 28 clients contribute in providing feedback in the Q4 2018 Tech Preview. The Q2 2019 Tech Preview will start in late March 2019. (CCO note: stay tuned for the details of how to sign up).


For SaaS clients, there is a client-led weekly user group meeting called the SaaS Virtual Group with support from Blackboard's product management team. For clients on Ultra, there is a monthly Learn Ultra Office Hours program, where clients can give feedback directly to our product management team.


There is a Blackboard Idea Exchange section on the Community site where clients can submit new ideas directly to Product Management. We are focused on finding ways to provide better and more timely feedback and show how your ideas are informing our roadmap and prioritization.


4. What are some other areas you are excited to think about for the future of driving quality at Blackboard?

In the coming year, we’ll be working even more closely on how teams go from idea to production as fast as possible. I’ll be increasing my focus on Release Engineering. We’ll be concentrating on improving flow and feedback - establishing the processes, tooling and infrastructure to enable teams to do so. Our team will also work very closely with our Site Reliability Engineering team, all to build towards better predictability, observability, and reliability (and recoverability) of our systems. It’s an incredibly exciting time in our field!


I also hope to look at other areas where we may enhance testing to drive better quality, like testing in production; improving our cross-browser testing (how we test across all the browsers); and introducing a type of testing that’s been around awhile but really catching attention in the testing field lately – Visual Regression Testing (pixel by pixel comparison on each change). All experimental, all ways to deepen our learning of the impact of our changes on the system.


I want to thank Ashley for sharing her vision and work via my blog.


At the heart of all this work that Ashley and our teams do, is you - our partners and users. How you experience our products is what matters. We will never be done with improving quality, and we improve every day thanks to your feedback and inputs!