Hello, I would like to share with you some of my thoughts on creating CLIs that work with Blackboard Learn’s REST APIs. In this post, I will be covering: My initial thoughts on BbRest API’s, Why I decided to get started on working with them for our institution, What I have ‘Learned’, Ports, and the future of these CLIs along side with Blackboard’s road map.
My initial thoughts.
When I first saw Mark O'Neil and Scott Hurrey's presentation at DevCon16 in Las Vegas on creating simple CLI’s for Blackboard Learn’s REST APIs, I was hooked. That fact the Bb was actually listening to the needs their clients and the problems they are facing had installed great promise. When they announced that they were implementing REST and that you could start testing/using them right away was a great boost in my current thoughts of Learn at the time. I admit, as an LMS administrator at the institution I work for, I wasn’t impressed of the current work flow in which my colleagues and I have to endure….. DevCon16 was also my first year in presenting as well: I presented on creating a B2 for my institution on creating a way for students to submit and application for off campus testing. After completing that presentation and project, I decided that B2s are not the best solution for forward thinking ideas and needs of today’s end users. Personal opinion: I believe that B2s limit the overall flexibility of Learn. (B2: Building Block).
When sitting through the presentation, I realized that Blackboard is changing and that Learn will be changing as well. The overall product design was (is) moving forward and that if we do not start planning along side this road map, we are going to have a hard time catching up. So when @Mark O’Neil walked up and started presenting his Python Demo, I was impressed in what he had done; with only just learning Python and creating a CLI in approximately eight hours.
When Mark was finished, Scott walked up...Boom, floored again as he did his presentation. Although the demo was slightly shot due to internet issues (live demos ftw!), the work he did was amazing, and almost made me want to jump back in to JAVA...(sorry just kidding; No JAVA for me ;p ). Now there is nothing wrong with JAVA if you love it, but as for me it’s not my cup of tea (facepalm for the pun, sorry). The overall idea came in when they both said that we could take these examples and make them our own, run with them, you can do it! I was highly impressed as most companies are trying to get you to pay for consulting. The notion of being able to get free code created by the developers themselves was very forwarded thinking and that’s where I said sign me up!
Deciding to redo the original concept.
I followed the GitHib repo for Mark’s CLI and wanted to test it, so naturally I cloned it down and started using it. I noticed that there was some issues and made some tickets. I ended up talking to Mark later and we ended up showing some ideas we both were working on and I asked if it would be ok if I took on the project and run with it. Mark was ok with it and said go forth and code!
Now being a somewhat experienced Python programmer (at least I like to think I am ), I jumped on it as fast as I could. No, really, I was leaving the next day and started working on it at the airport. I had a four hour flight back home and by the time I arrived I had a new base model.
I decided to use Docopt, which is a python module created by Vladimir Keleshev, for creating the help screen of the CLI in which in turn, turns into the CLI. It is a beautiful package and the beauty of it is that it has ports over to other languages. More on that later. After a year on working with my CLI and learning the ins and outs of Learn’s Rest API’s, I have found that there was indeed a need for this sort of thing.
How we use this at our institution.
It first started with just doing tests and after a few successful use case tests, I decided to share this CLI with a colleague. We decided to test this further as we were at the time just in the process of changing our SIS/ERP system. We found out that it will have the capability to do integrations and make web calls. With our current legacy system this is just not possible without major re-programming and serious debugging. So we started running through some of the scenarios as if it couldn’t actually make a web call and it was just dumping data files to a directory on a system. I looked over the Bb REST APIs and started gathered some examples of the json models and we used it to mock up some test data. After a few tests we knew that this was a very good go to in the event that we wouldn’t be able to get the web services running on the new SIS/ERP system right away.
We also found out, that we do a lot of manual data manipulation that our current system cannot process due to business rules and system limitations. We have found that using this CLI made our current daily tasks easier as we did not have to try to remember the ins and outs of integration files and send them up to Bb. Knowing that we have to make sure to use the right integration as that feature in Bb has its own limitations based flaws. If you don’t know the main issue: Bb SIS Integrations own the record in which creates a learn object (User, Course, Enrollment, etc). This means that you cannot modify records that are owned by one integration from another. Now, in a perfect world this is ok, but this isn’t a perfect world and we can’t always use the same integration with other processes; for the institution I work for anyway. This is because we do not like to skew the data coming from one system that belongs to another. Doing this has helped us tremendously in making sure we keep data integrity intact.
Using the REST API method we have found that a lot of the data that we need to change isn’t system dependent: Names, Title Changes, Roles for non SIS records; even some SIS records. Now, we have found that somethings we cannot change still with this method: Some data is still protected via the Integration. So until we have successfully moved over to our new SIS/ERP System, there is still a need for the older SIS integration methods. But it’s a start!
What I ‘Learned’ from doing this.
This particular project has given me great insight on programming methodologies, why programming in one particular language is limits your overall developer growth, and a few issues with Bb’s REST. Yep, like all great products, there will be bugs.
One of the main issues I noticed is that there is no result population. Meaning that when you make a call to the membership routes, only the internal/external ids are sent back; no friendly readable structure. You will have to make up your own work flow in delivering the readable end result. There are some issues with certain http methods on certain routes: Course Membership (at the time of this article a patch is/was being implemented). You cannot change a users course role; even if it was created via rest. You can only do limited searching upon one major criteria group: /courses?dataSourceId=<id> vs. /courses?dataSourceId=<id>&termId=<id>. This results in a 400 status error and states you cannot load data based on multiple search terms. This in my opinion is a serious limitation, but I am sure that there is some very good and reasonable reasons on this particular issue. There are some inconsistencies in the HTTP Methods: Bb stated that they would favor POST for create and PATCH for updating. All well and good but please be aware of these routes:
These routes break the work flow and when creating a CLI to auto-detect the correct HTTP method to use. Personal thought: PUT should be for creating, POST should be for updating partial information and PATCH should be used to update a record due to a previous error in the information (or error preventing); kind of like a reset method. Not trying to raise and old debate, but to me, that makes more sense.
These are just a few things I have noticed that caused some major rethinking on my end when creating a CLI for Bb. There are many other small issues but they are manageable and/or have been corrected (or being looked into) since the first initial release of the APIs.
Deciding to port the CLI over to PHP.
I just recently did a presentation at DevCon17 on two projects that I am working on for my institution: Course Role Management Web Application and my Python CLI. During my visit, I had the pleasure of meeting some fellow developers and during our talks it came to me about looking at this with a different lens. I was able to sit down with Scott Hurrey and Mark O'Neil and had a discussion about Blackboard's just announced Open Source Initiative. I started bringing the thought of an SDK to the table. Of course, this was something they have been working on for quite some time. I expressed my opinions on the matter and after one thing after another, I thought I should put this to a concept.
I tasked myself on porting my Python CLI over to PHP and still keeping the same usage. This is where docopt comes into play. Docopt has been ported over to several other languages and the creator has supplied a spec in which to write a port of the package. I currently at this time have ported over about 90% of this CLI to PHP while still using the same usage from the Python version; with only working on it a few hours a night spread over a week. This brings me back to SDK idea. I expressed my interest in Bb’s open source solution and that there needs to be an official spec in writing Bb applications from a outside source. This overall idea isn’t new, but new to this particular world of educational products. If we (The Bb Community) can come up with a official spec to create applications that can be used in language agnostic manner, we can offer a beautiful solution for those will come behind us.
The idea of an official spec would include but not limited to: Overall protocol flow, Naming Conventions, Data Conventions, Expectancy, Security, Delivery, etc, etc. Having this higher abstracted overview on how to develop a package for the end users choice is a huge help in future productivity and the community. Imagine having an idea and then to be able to quickly try out and based off of programming language/package that you, the developer, are comfortable using. To be able to do it quickly without all the hassle that others have had the pleasure of finding out. Wouldn’t that be great?
Plans on other ports?
Do I plan on porting this over to other languages? Yep, Why wouldn’t I? To me, it’s a great learning tool. Now I am not planing on doing all languages as that would be a major commitment and as one person, I cannot commit to that. I do however plan on porting it over to NodeJS. I welcome and challenge the Bb community to join the fun! I also welcome all to challenge this concept in general: Is this going to be beneficial? How will this shape not only our personal development growth but for Bb in terms of community and product?
Conclusion. What will come out of these examples?
Well that is up to everyone here! I will continue to support the applications I create and share them as long as there is a need. I hope to see others use these ideas and concepts and make them their own and share back. I hope that I can continue with the great relationships I have formed with some of the Bb team and community and in turn be able to produce something that is only going to help the future of the educational field.
My repos for you to clone, fork, hack, destroy, use:
https://github.com/elmiguel/BBDN-REST-PHP Depends on bbdn-core-php.
https://github.com/elmiguel/bbdn-core-php The PHP port of the Python CLI code. Work in progress.
https://packagist.org/packages/bbdn/core same as the github repo just a package to install via composer
https://github.com/elmiguel/aMEBA A Mongo DB, Express JS, Node JS, Blackboard REST Application.
https://github.com/elmiguel/CM-Role-Management A MEAN Stack Application PoC using Bb REST APIS.
https://github.com/elmiguel/BBDN-REST-DEMO_Python Planning on porting into a pip package eventually.