mdeeprose

Updates to our upgrade process

Blog Post created by mdeeprose on Jan 24, 2018

This winter like many other institutions we applied the latest CU to enable the crocodoc to box transition.

 

The upgrades worked well as we were able to update and adjust the implementation plans we had spent a lot of time on last year.

 

Changes undone by new CUs.

Cumulative Updates use the full installation procedure and will usually undo changes to configuration files that have been made previously to resolve known issues.  This means that when applying a CU or performing a new upgrade that one should verify whether the installer has undone changes and redo them. This became immediately clear to us this winter.  This blog post covers what we did to make the experience less painful.

 

Last summer we had already started documenting the fixes we had made separately so that they were not lost and forgotten in old implementation plans, so we had a nice list of fixes to check.  Now when applying a new CU we have to check whether those fixes have been undone.  Here are the changes we made to our procedures to deal with this.

 

 

1. Analyse whether changes have been made to key files.

 

Before the upgrade we copy all the files to which we make changes to a separate directory outside of Blackboard.

mkdir /home/bbuser/original_files

cp /usr/local/blackboard/config/bb-config.properties /home/bbuser/original_files/
cp /usr/local/blackboard/config/bb-config.properties.template /home/bbuser/original_files/
cp /usr/local/blackboard/config/tomcat/conf/wrapper.conf /home/bbuser/original_files/
cp /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb /home/bbuser/original_files/
cp /usr/local/blackboard/config/tomcat/conf/server.xml.bb /home/bbuser/original_files/
cp /usr/local/blackboard/config/internal/http-cacheable-extensions.txt /home/bbuser/original_files/
cp /usr/local/blackboard/config/internal/xythos-indexing-filter.txt /home/bbuser/original_files/
cp /usr/local/blackboard/config/tomcat/classes/Search.properties.bb /home/bbuser/original_files/
cp /usr/local/blackboard/config/tomcat/classes/Search.properties /home/bbuser/original_files/
cp /usr/local/blackboard/config/service-config.properties /home/bbuser/original_files/
cp /usr/local/blackboard/config/internal/nautilus-config.properties /home/bbuser/original_files/

Then after the upgrade we make copy those files again to a different directory.

mkdir /home/bbuser/post_upgrade_files
cp /usr/local/blackboard/config/internal/http-cacheable-extensions.txt /home/bbuser/post_upgrade_files/
cp /usr/local/blackboard/config/internal/xythos-indexing-filter.txt /home/bbuser/post_upgrade_files/
cp /usr/local/blackboard/config/tomcat/conf/server.xml.bb /home/bbuser/post_upgrade_files/
cp /usr/local/blackboard/config/internal/nautilus-config.properties /home/bbuser/post_upgrade_files/
cp /usr/local/blackboard/config/service-config.properties /home/bbuser/post_upgrade_files/
cp /usr/local/blackboard/config/tomcat/conf/wrapper.conf /home/bbuser/post_upgrade_files/
cp /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb /home/bbuser/post_upgrade_files/
cp /usr/local/blackboard/config/bb-config.properties /home/bbuser/post_upgrade_files/
cp /usr/local/blackboard/config/bb-config.properties.template /home/bbuser/post_upgrade_files/
cp /usr/local/blackboard/config/tomcat/classes/Search.properties.bb /home/bbuser/post_upgrade_files/
cp /usr/local/blackboard/config/tomcat/classes/Search.properties /home/bbuser/post_upgrade_files/

We then use diff to analyse whether the files have changed.

diff /home/bbuser/original_files/bb-config.properties /home/bbuser/post_upgrade_files/bb-config.properties

diff /home/bbuser/original_files/bb-config.properties.template /home/bbuser/post_upgrade_files/bb-config.properties.template

diff /home/bbuser/original_files/http-cacheable-extensions.txt /home/bbuser/post_upgrade_files/http-cacheable-extensions.txt

diff /home/bbuser/original_files/nautilus-config.properties /home/bbuser/post_upgrade_files/nautilus-config.properties

diff /home/bbuser/original_files/Search.properties /home/bbuser/post_upgrade_files/Search.properties

diff /home/bbuser/original_files/Search.properties.bb /home/bbuser/post_upgrade_files/Search.properties.bb

diff /home/bbuser/original_files/server.xml.bb /home/bbuser/post_upgrade_files/server.xml.bb

diff /home/bbuser/original_files/service-config.properties /home/bbuser/post_upgrade_files/service-config.properties

diff /home/bbuser/original_files/wrapper.conf /home/bbuser/post_upgrade_files/wrapper.conf

diff /home/bbuser/original_files/wrapper.conf.bb /home/bbuser/post_upgrade_files/wrapper.conf.bb

diff /home/bbuser/original_files/xythos-indexing-filter.txt /home/bbuser/post_upgrade_files/xythos-indexing-filter.txt

 

This tells us not only if our fixes have been undone by the installer, but also whether additional changes have been made to the file concerned.  If no other changes have been made we can copy the old file back rather than edit it within the Blackboard directory, saving typing time.

 

2. Produce verification scripts to verify whether the changes we have made have been done successfully.

Previously, we made individual test to verify whether a change had been correctly made.  For example:

grep 'eot\|ttf\|woff' /usr/local/blackboard/config/internal/http-cacheable-extensions.txt | wc -l  
>> Result must equal 3.

The number of these grew during the project and so to save time and make it simpler we developed a pair of scripts that would perform the checks and report back in red or green text whether the verifications had passed or failed and what the result of the test was.  We broke these down into two separate scripts, one checking the bb-config.properties file where most of our changes are, and the other checking changes to any other files.

 

The scripts also check the hostname to be sure whether settings that should be only set on the tasks node are set on the tasks node, e.g. the “backend.processor” setting.

 

Here are example screenshots showing the output of the scripts.

script1.png

 

script2.png

And here is the code for one of the scripts as an example.

 

#!/bin/bash

# Checking script for Blackboard 2016 Q4 CU6 Upgrade - bb.config.properties checks

 

 

# Set colours

red=$'\e[1;31m'

grn=$'\e[1;32m'

end=$'\e[0m'

 

 

# Set variables

# Hostname

hostname=$(hostname)

# Details for SCORM fix

ScormFixinbbcptemplate=$(cat /usr/local/blackboard/config/bb-config.properties.template | grep "ancestors=self'")

ScormFixinbbcp=$(cat /usr/local/blackboard/config/bb-config.properties | grep "ancestors=self'")

# Details for tomcat settings

bbconfigjvmoptionsextratomcat=$(cat /usr/local/blackboard/config/bb-config.properties | grep "bbconfig.jvm.options.extra.tomcat")

bbconfigjvmoptionsgc=$(cat /usr/local/blackboard/config/bb-config.properties | grep "bbconfig.jvm.options.gc")

# DMARC setting

bbconfigemaildmarc=$(cat /usr/local/blackboard/config/bb-config.properties | grep "bbconfig.email.use.dmarc.from.override=")

# Backend processor setting (note different on tasks/web nodes)

backendproc=$(cat /usr/local/blackboard/config/bb-config.properties | grep "bbconfig.server.backend.processor")

# HTTP compression setting

httpcomp=$(cat /usr/local/blackboard/config/bb-config.properties |grep bbconfig.appserver.http.compression)

# Clear the screen

clear

# Report results

echo "bb.config.properties checks on $hostname"

echo ""

echo ""

echo "Test 1 - Scorm Fix in bbcp-template"

echo ""

if [ "$ScormFixinbbcptemplate" = "bbconfig.headers.CSP.frame-ancestors=self' https://*.soton.ac.uk 'soton.ac.uk" ]; then

   printf "%s\n" "${grn}TEST PASSED${end}: value returned is $ScormFixinbbcptemplate"

else

   printf "%s\n" "${red}TEST FAILED${end}: value returned is $ScormFixinbbcptemplate"

fi

echo ""

echo ""

echo ""

echo "Test 2 - Scorm Fix in bbcp"

echo ""

if [ "$ScormFixinbbcp" = "bbconfig.headers.CSP.frame-ancestors=self' https://*.soton.ac.uk 'soton.ac.uk" ]; then

   printf "%s\n" "${grn}TEST PASSED${end}: value returned is $ScormFixinbbcp"

else

   printf "%s\n" "${red}TEST FAILED${end}: value returned is $ScormFixinbbcp"

fi

echo ""

echo ""

echo""

echo "Test 3 - bbconfig.jvm.options.extra.tomcat"

echo ""

if [ "$bbconfigjvmoptionsextratomcat" = "bbconfig.jvm.options.extra.tomcat=-XX:NewSize=2048m -XX:MaxNewSize=2048m -XX:+UseTLAB -XX:SurvivorRatio=4 -XX:+UseCompressedOops -Xss512k -XX:+PrintVMOptions -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCTaskTimeStamps -XX:+PrintCommandLineFlags -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+AlwaysPreTouch -XX:+DoEscapeAnalysis -Xverify:none -Djava.util.prefs.systemRoot=/home/bbuser/.java/" ]; then

   printf "%s\n" "${grn}TEST PASSED${end}: value returned is $bbconfigjvmoptionsextratomcat"

else

   printf "%s\n" "${red}TEST FAILED${end}: value returned is $bbconfigjvmoptionsextratomcat"

fi

echo ""

echo ""

echo ""

echo "Test 4 - bbconfig.jvm.options.gc"

echo ""

if [ "$bbconfigjvmoptionsgc" = "bbconfig.jvm.options.gc=-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=45 -XX:+UseStringDeduplication" ]; then

   printf "%s\n" "${grn}TEST PASSED${end}: value returned is $bbconfigjvmoptionsgc"

else

   printf "%s\n" "${red}TEST FAILED${end}: value returned is $bbconfigjvmoptionsgc"

fi

echo ""

echo ""

echo ""

echo "Test 5 - bbconfig.email.use.dmarc.from.override"

echo ""

if [ "$bbconfigemaildmarc" = "bbconfig.email.use.dmarc.from.override=false" ]; then

   printf "%s\n" "${grn}TEST PASSED${end}: value returned is $bbconfigemaildmarc"

else

   printf "%s\n" "${red}TEST FAILED${end}: value returned is $bbconfigemaildmarc"

fi

echo ""

echo ""

echo ""

# Note that a different result is expected depending on whether we are on the tasks node or a web node

echo "Test 6 - backend processor setting"

echo ""

if [[ "$backendproc" = "bbconfig.server.backend.processor=true" && "$hostname" = "removed" ]] || \

[[ "$backendproc" = "bbconfig.server.backend.processor=false" && "$hostname" != "removed" ]]; then

   printf "%s\n" "${grn}TEST PASSED${end}: Hostname is $hostname and value returned is $backendproc"

else

   printf "%s\n" "${red}TEST FAILED${end}: Hostname is $hostname and value returned is $backendproc"

fi

echo ""

echo ""

echo ""

echo "Test 7 - http.compression"

echo ""

if [ "$httpcomp" = "bbconfig.appserver.http.compression=off" ]; then

   printf "%s\n" "${grn}TEST PASSED${end}: value returned is $httpcomp"

else

   printf "%s\n" "${red}TEST FAILED${end}: value returned is $httpcomp"

fi

echo ""

echo ""

Final thoughts

Although it is really annoying to have to spend time to reimplement config changes and fixes just to install a CU we are slowly developing our practice to streamline the process as much as we can.  I would be very interested to hear from other Blackboard admins about how they deal with this aspect of upgrades.

Outcomes