I am now in the midst of performing the fifth upgrade to Q4 2016 - all performed during the past month.
Due to the situation we currently are in, and due to the many issues encountered, I am writing this "review" of the Q4 2016 (and Q2 2016) new installer scripts.
As I am trying to be objective about this, and trying to avoid the new trend of "alternative truths", I have written up a number of "facts" - intended objective depictions of the issues we have encountered, but first some data about the types of environments, that we have:
- Linux RHEL 6.4
- Oracle DB 18.104.22.168 and in 1 case Oracle two-node RAC
- 3 systems with locally attached storage (/content) and 1 with NFS mounted storage
- VMware systems - all servers very well provisioned (typically app RAM 16 GB - 4 vCPU and db server RAM 32 GB - 16 vCPU)
- current release of Blackboard Q2 2016 CU1
I can state with 100% objectivity the following:
1. The installer scripts has in the previous 4 upgrades always failed (and the fifth concurrent upgrade hasnt failed to follow that pattern - it has also failed...)
2. The installer scripts has in all cases produced new and unique errors - plus off course also a host of the same errors. Recommendation - keep a list of all issues encountered on your pathway to upgrading your Production environment...
3. A wrong parameter in the installer scripts (executed by a toplevel BtB supporter) resulted errouneously in the uninstallation of the systems database. Recommendation - be very careful in adding parameters to the installation script (especially adding parameter "--DEBUG" - just dont do it :-) - and another recommendation - take the Blackboard advise about making sure that you have a current and working backup of the DB and /content seriously...
4. The upgrade process takes much longer - and here I am not talking just about the "installers running time" - but the whole process of preparing environments for installation to finally getting the system to start after a seemingly successful upgrade. Recommendation - leave yourself with a lot of time to work on upgrades, and have (many) environments to train yourself in performing upgrades on with some kind of Production data
5. Every upgrade performed has demanded extensive time spent with Blackboard Support in getting the errors fixed - ranging from 3 to 14 days. Recommendation - same as 4 - plan for spending a lot of time on upgrades
6. Every upgrade has demanded long Collab sessions with level 2 Blackboard supporters (and thanks very much for your help and expertise) in getting these issues resolved. Recommendation - buy a comfortable headset
7. The preupgrade Help pages are far from being specific enough about what should be checked beforehand in order to minimize the time needed for an upgrade. Recommendation - this for Blackboard - It is off course hard to accomplish a full preupgrade list, but from my perspective - Blackboard should just lay the issues on the table. Make a list or rather a Help page that contains all the known issues that clients/Blackboard Support has encountered in regards to upgrades...
8. The installer scripts when used on upgrades very often leaves the Blackboard system in a state, where files have gone missing (ex. Xythos.properties and Tomcat files). And then when either running the installer scripts again in order to get the upgrade to complete, the "halfinstalled/upgraded" state that the system is in, just opens up a new host of issues, to be dealt with. I have twice had to move the existing /usr/local/blackboard files to a new location, in order for the installer to be able to run. Also after having run an install, it has twice been necessary to create a new installer directory, and then unzip the original installer into there. This definitely leaves the impression of not having complete knowledge about exactly what happens to both the existing Blackboard environment as well as the installer directories after having run the installer scripts once.
9. I have had issues with a system not starting after an upgrade ran through successfully. The issues pertained to the state of building blocks after an upgrade. Part of it was my own fault, as I had left the "Admin Tools" in an active state before upgrading. This B2 should according to BB support always be put in an "inactive" state, when not being used - didnt know/remember this but ok... However the "bb-stream" and the "bb-blackboard-core" was also "inactive" after the upgrade, and this off course made it impossible for Blackboard to start - so check all your building blocks after an upgrade. Recommendation - only celebrate a successful upgrade after the system has actually started - dont pop the champagne when reading "Build Succesfully".
And now for impressions and the actual review of the new installer scripts based on above and 2 months of working with the installer:
- the installer works flawlessly when performing new installations. After you have grasped how the preinstallation routines are done, you can perform installations ad nauseam. I havent had any issues and the documentation is ok in helping you to perform your first installation.
- the installer is probably a heaven-sent new tool for Blackboard Managed Hosting - it seems like a brilliant tool for mass deployment of uniform Blackboard environments.
- The installerscripts seems to be not very well tested, when using them for upgrades. Upgrades are off course also the hardest challenge for installerscripts to handle - an automated response to the unknown. However that just makes long and hard testing before release of new installerscripts even the more neccessary.
- If the installerscripts are proving hard to use (and completely understand) for the experts in Blackboard Support, it just expands the problem at the local selfhosted system admins end. As usual the Blackboard Support team is doing there best with what is given them, but it just seems exhausting for a Blackboard customer to yet again witness how knowledge and technological changes are so poorly handed over to the operational side of Blackboard Inc.
- The new installerscripts has been created by developers - its got the distinct touch of developers fascinated in nice new technologies. Its also got the sense of trying something completely new, and thereby breaking the bond with the old. Not surprising maybe, but from where I stand installerscripts has to be developed by people who has vast experience in dealing with upgrades. It has to be at least "DevOps", and it has to be people who will actually face the music in reg. to all the issues, that comes with introducing new methods for performing something as basic as installation and in this case upgrades.
- It leaves the technical staff at selfhosted clients with a very big problem of predicting how long time they need for an upgrade - this off course even the more important, when planning for service windows when upgrading a Production environment. I can honestly say, even with the experience of having performed 4½ upgrades up to now, I am struggling to come up with a prediction of how long it will take, and also struggling with the thought of how the business/faculty will respond to the probably lenghy service window needed. I have however (and this is my advise) kept the business/faculty updated with statusmails and also held meetings with them in order to keep their expectations in check.
Above enforces the impression of Blackboard Inc. saying that they care for their selfhosted customers, but actually leaving them in a state, where they have an exceedingly difficult task in managing the Blackboard software.
Recent changes also in reg. to removing the patching functionality, and thereby converting the appliance of Cumulative Updates to the equivalent of a "new installation" (or even worse - an upgrade) has also left the technical personel at selfhosted clients with the tiresome task of having to request very long "service windows" from their faculty. I acknowledge all the issues that Blackboard had created for themselves in reg. to maintaining a web of dependencies between Building Blocks with the old patch/update tool also when applying CU's, but maybe more time could have been spent on providing a solution, that could resolve this.
And this I probably shouldnt say - but I will do it anyway - a business strategy that seems to be focused on Managed Hosting (and SAAS), when leaving selfhosted clients technical personnel with software, that is complex to update and upgrade just seems to be a self fulfilling prophecy in regards to making the "business/faculty side" of the client grow impatient with IT and push for handing over the hosting of Blackboard to Managed Hosting. The "business/faculty" side of the Blackboard clients are left with an impression, that their own technicians cant deliver in reg. to short downtime when updating/upgrading or an assurednes in that problems can be handled easily and transparently for the users.
Again - I am greatly appreciative of all the help received by Blackboard Support - they are the soldiers, that on a daily basis are keeping Blackboard afloat amongst the dying breed of the selfhosted Blackboard customer (the last bit of the sentence said with a twinkle in the eye ;-)
System Coordinator/System Admin