mdeeprose

Our live upgrade plan from 2016 Q4 CU6 to 2017 Q4 CU3

Blog Post created by mdeeprose on Nov 8, 2018

Here is a slightly redacted version of our live upgrade plan.  I'm sharing this in case it is useful for others...  I will also post a summary of our current configuration changes and tweaks in a separate blog post.

Live Upgrade to from 9.1 Q4 2016 CU6 to 9.1 Q4 2017 CUx

Overview

We will upgrade Blackboard Live to 9.1 Q4 2017 CU3.

Server Address

Blackboard preproduction is made up of four applications nodes, a tasks node, an Oracle 12C database, and a NFS server.

hostname_removed – tasks node

hostname_removed – web node

hostname_removed – web node

hostname_removed – web node

hostname_removed – web node

hostname_removed – Oracle database

hostname_removed – NFS server

Note

Where "on each node" is stated, this refers to each node on which the Blackboard application runs, i.e. the tasks node and the web nodes.

Summary of stages

  1. Stage 1 covers work that can be carried out before the upgrade work starts. This work is under CHG_______.
  2. Stage 2 covers making Blackboard inaccessible to users and taking our baseline snapshot. This stage onwards are under CHG_______.
  3. Stage 3 covers the Oracle update.
  4. Stage 4 covers our upgrade of Blackboard to the new version.
  5. Stage 5 is where we check what changes have been made by the installer.
  6. Stage 6 is where we re-apply configuration changes that have been reset by the installer.
  7. Stage 7 is where configuration changes are applied to the application using pushconfigupdates.
  8. Stage 8 is where we check the log files for any possible issues before the finish for the day.
  9. Stage 9 is where we start the day by checking the logs after the application has been running overnight. We will also check that data integration has run ok.
  10. Stage 10 encompasses GUI based configuration changes if required, and acceptance testing.
  11. Stage 11 is a restart test.
  12. Stage 12 is where we make Blackboard available to users
  13. Stage 13 involves tidying up after successful testing.

Stage 1 - Prologue

Stage 1 covers work that can be carried out in preprod and dev environments before the upgrade work starts. This is under CHG_______

  1. Bb_Team and DBA: confirm and agree password to be used.
  2. DBA: consider whether Oracle backups should be stopped from running on the night of Saturday 23 June.
  3. DBA: Prepare Oracle patching files.

Prepare the installer

  1. Bb_Team: Prepare installation files:
    # as bbuser
    # on each node
    unzip /home/bbuser/learn-installer-[VERSION_NUMBER].zip -d /usr/local/bbinstaller/
    1. Download the installer
      1. Go to https://support.blackboard.com and log in.
      2. Go to page https://behind.blackboard.com/downloads/details.aspx?d=1783
      3. Click on Learn 9.1 Q4 2017 CU3 Installer.
      4. Click on I agree to the terms and conditions above.
      5. Click Get File.
      6. Download will start. Download to /home/bbuser/ on each node.
    2. Unzip the installer to /usr/local/bbinstaller/ on each node.
  2. Bb_Team: Set files under /usr/local/bbinstaller/ to be owned by bbuser:bbuser on each node.
    # as bbuser
    # on each node
    cd /usr/local ;
    chown -R bbuser\: bbinstaller

Prepare the licence file

  1. Bb_Team: Copy /usr/local/blackboard/config/licence /blackboard-licence .xml to /home/bbuser/ on each node.
    # as bbuser
    # on each node
    cp /usr/local/blackboard/config/licence /blackboard-licence .xml /home/bbuser/

Prepare the installer properties files

Prepare the installer properties for tasks node (hostname_removed)

  1. Bb_Team: create file at /usr/local/bbinstaller/installer.properties with following draft contents on tasks node:
    # Installer Properties for Tasks node
    antargs.default.vi.db.password=[ ]
    antargs.default.users.administrator.password=xxxx
    antargs.default.users.integration.password=xxxx
    antargs.default.users.rootadmin.password=xxxx
    bbconfig.file.licence =/home/bbuser/blackboard-licence .xml
    bbconfig.java.home=/usr/lib/jvm/java-1.8.0-oracle.x86_64
    bbconfig.jvm.options.extra.tomcat=-XX:NewSize=2048m -XX:MaxNewSize=2048m -XX:+UseTLAB -XX:SurvivorRatio=4 -XX:+UseCompressedOops -Xss512k -XX:+PrintVMOptions -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCTaskTimeStamps -XX:+PrintCommandLineFlags -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+AlwaysPreTouch -verbose:gc -XX:+DoEscapeAnalysis -Xverify:none -XX:+UseMontgomeryMultiplyIntrinsic -XX:+UseMontgomerySquareIntrinsic -XX:MetaspaceSize=768m -XX:MaxMetaspaceSize=1536m -Djava.util.prefs.systemRoot=/home/bbuser/.java/
    bbconfig.jvm.options.gc=-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=45 -XX:+UseStringDeduplication
    bbconfig.email.use.dmarc.from.override=false
    bbconfig.max.stacksize.tomcat=512k
    bbconfig.appserver.http.compression=off
    bbconfig.jvm.options.codecache.reserved=256m
    bbconfig.jvm.options.codecache.initial=256m
    bbconfig.cs.database.maxpoolsize=50
    bbconfig.peer.discovery.timeout.inactive=20
    bbconfig.peer.discovery.timeout.dead=300
    bbconfig.server.backend.processor=true
    bbconfig.gradecenter.cache.grade_threshold=0
  2. Bb_Team: note blank lines can be deleted with sed /^$/d

Prepare the installer properties for web nodes (hostname_removed - hostname_removed)

  1. Bb_Team: create file at /usr/local/bbinstaller/installer.properties with following draft contents on web node:
    # Installer Properties for Web nodes
    antargs.default.vi.db.password=[ ]
    antargs.default.users.administrator.password=xxxx
    antargs.default.users.integration.password=xxxx
    antargs.default.users.rootadmin.password=xxxx
    bbconfig.file.licence =/home/bbuser/blackboard-licence .xml
    bbconfig.java.home=/usr/lib/jvm/java-1.8.0-oracle.x86_64
    bbconfig.jvm.options.extra.tomcat=-XX:NewSize=2048m -XX:MaxNewSize=2048m -XX:+UseTLAB -XX:SurvivorRatio=4 -XX:+UseCompressedOops -Xss512k -XX:+PrintVMOptions -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCTaskTimeStamps -XX:+PrintCommandLineFlags -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+AlwaysPreTouch -verbose:gc -XX:+DoEscapeAnalysis -Xverify:none -XX:+UseMontgomeryMultiplyIntrinsic -XX:+UseMontgomerySquareIntrinsic -XX:MetaspaceSize=768m -XX:MaxMetaspaceSize=1536m -Djava.util.prefs.systemRoot=/home/bbuser/.java/
    bbconfig.jvm.options.gc=-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=45 -XX:+UseStringDeduplication
    bbconfig.email.use.dmarc.from.override=false
    bbconfig.max.stacksize.tomcat=512k
    bbconfig.appserver.http.compression=off
    bbconfig.jvm.options.codecache.reserved=256m
    bbconfig.jvm.options.codecache.initial=256m
    bbconfig.cs.database.maxpoolsize=50
    bbconfig.peer.discovery.timeout.inactive=20
    bbconfig.peer.discovery.timeout.dead=300
    bbconfig.server.backend.processor=false
    bbconfig.gradecenter.cache.grade_threshold=0
  2. Bb_Team: note blank lines can be deleted with sed /^$/d

Back up files

Back up config files

  1. Bb_Team: Backup the various configuration files on each node.
    # as bbuser
    # on each node
    mkdir /home/bbuser/original_files ;
    cp /usr/local/blackboard/config/bb-config.properties /home/bbuser/original_files/ ;
    cp /usr/local/blackboard/config/bb-config.properties.template /home/bbuser/original_files/ ;
    cp /usr/local/blackboard/config/tomcat/conf/wrapper.conf /home/bbuser/original_files/ ;
    cp /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb /home/bbuser/original_files/ ;
    cp /usr/lib/jvm/java-1.8.0-oracle.x86_64/jre/lib/security/java.security /home/bbuser/original_files/ ;
    cp /usr/local/blackboard/config/tomcat/conf/server.xml.bb /home/bbuser/original_files/ ;
    cp /usr/local/blackboard/config/internal/http-cacheable-extensions.txt /home/bbuser/original_files/ ;
    cp /usr/local/blackboard/config/internal/xythos-indexing-filter.txt /home/bbuser/original_files/ ;
    cp /usr/local/blackboard/config/tomcat/classes/Search.properties.bb /home/bbuser/original_files/;
    cp /usr/local/blackboard/config/tomcat/classes/Search.properties /home/bbuser/original_files/;
    cp /usr/local/blackboard/config/service-config.properties /home/bbuser/original_files/ ;
    cp /usr/local/blackboard/config/internal/nautilus-config.properties /home/bbuser/original_files/ ;
    cp /usr/local/blackboard/config/cache-settings.properties /home/bbuser/original_files/ ;
    cp /usr/local/blackboard/config/message-queue-service-config.xml /home/bbuser/original_files/ ;
    cp /usr/local/blackboard/config/message-queue-service-config.xml.bb /home/bbuser/original_files/ ;
    cp /usr/local/blackboard/content/vi/bb_bb60/plugins/bb-xss-filter/webapp/WEB-INF/classes/blackboard/xss/regex/bb-xss-regex-filter.properties /home/bbuser/original_files/;
    cp /usr/local/blackboard/config/internal/bb-file-filter-configuration.properties /home/bbuser/original_files/
  2. Bb_Team: Verify that the correct number of files have been copied.
    # as bbuser
    # on each node
    ls -1 /home/bbuser/original_files/ |wc -l
    #result should equal 17

Back up branding files

  1. Bb_Team: Backup files under /usr/local/blackboard/docs/login/* to /home/bbuser/login.zip on each node.
    # as bbuser
    # on each node
    cd /home/bbuser ;
    zip -r login.zip /usr/local/blackboard/docs/login/* ;
    # set permissions
    chown bbuser:bbuser login.zip

Back up data integration files

  1. Bb_Team: Back up the data_integration directory on the tasks node only (hostname_removed).
    # on tasks node (hostname_removed)
    cd /home/bbuser ;
    zip -r data_integration.zip /usr/local/blackboard/data_integration/* ;
    # Set permissions
    chown bbuser:bbuser data_integration.zip
  2. Bb_Team: Back up the snapshot* files in /blackboard/apps/snapshot/data/ directory on the tasks node only.
    # on tasks node (hostname_removed)
    cd /home/bbuser ;
    zip -r snapshotfiles.zip /usr/local/blackboard/apps/snapshot/data/snapshot.* ;
    # set permissions
    chown bbuser:bbuser snapshotfiles.zip

Back up photo files

  1. Bb_Team: Back up the Photos directory /usr/local/blackboard/content/PHOTOS/ on tasks node.
    # on tasks node (hostname_removed)
    cd /home/bbuser
    zip -r PHOTOS.zip /usr/local/blackboard/content/PHOTOS/*
    # set permissions
    chown bbuser:bbuser PHOTOS.zip

Create Verification Scripts

  1. Bb_Team: Create first verification script "bb-config-tests-1" to be used for bb.config.properties on each node.
    # as bbuser
    # on each node
    nano /home/bbuser/bb-config-tests-1.sh
    #paste following script from hostname_removed/wiki/Blackboard/Live_Verification_Script_1
  2. Bb_Team: Make the script executable on each node.
    # as bbuser
    # on each node
    chmod +x /home/bbuser/bb-config-tests-1.sh
  3. Bb_Team: Create second verification script "bb-config-tests-2" to be used for non bb.config.properties checks on each node.
    # as bbuser
    # on each node
    nano /home/bbuser/bb-config-tests-2.sh
    # paste following script from hostname_removed/wiki/Blackboard/Live_Verification_Script_2
  4. Bb_Team: Make the script executable on each node.
    # as bbuser
    # on each node
    chmod +x /home/bbuser/bb-config-tests-2.sh

Prepare Blackboard for upgrade

  1. Bb_Team: Get the updated "forum_is_enabled.sql" file (see https://blackboard.secure.force.com/btbb_caseview?id=5003900001zkPvwAAE) and place in bbuser's home folder on each node.
    # on each node
    # as bbuser
    cd ;
    wget hostname_removed.ac.uk/bb/bb2018-fixes/forum_is_enabled.sql
  2. Bb_Team: Get the updated "widget.js" file (see https://blackboard.secure.force.com/btbb_articleview?id=kA3390000008bOi) and place in bbuser's home folder on each node.
    # on each node
    # as bbuser
    cd ;
    wget hostname_removed.ac.uk/bb/bb2018-fixes/widget.js

Stage 2 - Day 1 - Live Upgrade begins

In this stage we will disable Blackboard access to users and take our baseline snapshot back up.
CHG_______
Expected start time: 09:00 23 June 2018
N.B. Highfield team to be in place and logged in ready to start by 08:55

  1. Bb_Team: Note start time on whiteboard. Note in RFC that stage has started.
  2. Bb_Team: Tweet that upgrade has started.
  3. Bb_Team: Update upgrade status page.
  4. SYS_TEAM: Remove Blackboard from NLB Pool, thus preventing user access.
    Locate the node objects for the front end servers and 'force offline'
  5. APPS_TEAM: Stop Blackboard on each node, web nodes (hostname_removed – hostname_removed) first, then tasks node (hostname_removed).
    # on each node, web first, then tasks
    # web nodes may be stopped concurrently
    service blackboard stop
  6. SYS_TEAM: VMWare Snapshot #1.
  7. APPS_TEAM: Stop Blackboard on each node, web nodes (hostname_removed – hostname_removed) first, then tasks node (hostname_removed).
    # on each node, web first, then tasks
    # web nodes may be stopped concurrently
    service blackboard stop
  8. Bb_Team: Note stage completion time on whiteboard.

Stage 3 - Day 1 - Oracle update

In this stage we apply the Oracle April Update and remove a superfluous view.

Expected start time: 09:35 23 June 2018


On Dev 1, this stage took 20 minutes.
On Dev 2, this stage took 20 minutes.
On Preprod, this stage took 30 minutes.
On Dev 1 in May, this stage took 30 minutes.
On Dev 2 in May, this stage took 35 minutes.
On Preprod in May, this stage took 45 minutes.

  1. Bb_Team: Note start time on whiteboard. Note in RFC that stage has started.
  2. Bb_Team: Tweet that stage has started.
  3. Bb_Team: Update upgrade status page.
  4. ALL: Check Blackboard is stopped.
  5. DBA: Apply Oracle patch.
    See hostname_removed/wiki/Applying_Patch_PSUAPR2018
  6. DBA: Remove the superfluous view by doing the following.
    backup Table:
    Create Table BB_BB60_STATS.ETL_MATERIALIZED_VIEW_20180226
    As
    Select * from BB_BB60_STATS.ETL_MATERIALIZED_VIEW;
    Execute delete SQL:
    delete from BB_BB60_STATS.ETL_MATERIALIZED_VIEW
    where view_name in ('ods_sog_act_related_ob_vw','ods_ipo_related_obj_vw','ods_instr_measure_ipo_vw');
    commit;
  7. SYS_TEAM: VMWare Snapshot #2.
  8. Bb_Team: Sanity Check Blackboard.
  9. Bb_Team: For preprod and live, reset default language pack to UK English.
  10. APPS_TEAM: Stop Blackboard on each node, web nodes (hostname_removed – hostname_removed) first, then tasks node (hostname_removed).
    # on each node, web first, then tasks
    # web nodes may be stopped concurrently
    service blackboard stop
  11. Bb_Team: Note finish time on board.

Stage 4 - Day 1 - The upgrade

Stage 4 covers our upgrade of Blackboard to the new version.

Expected start time: 10:30 23 June 2018


On Dev 1, this stage took 25 minutes, with the upgrade itself taking 9 minutes 46 seconds.
On Dev 2, this stage took 30 minutes, with the upgrade itself taking 19 minutes 27 seconds.
On Preprod, this stage took 1 hour and 5 minutes, with the first full upgrade itself taking 19 minutes 15 seconds, and subsequent app-server only upgrades taking between 4 and 5 minutes each.
On Dev 1 in May, this stage took 37 minutes, with the upgrade itself taking 20 minutes 4 seconds.
On Dev 2 in May, this stage took 40 minutes, with the upgrade itself taking 21 minutes 4 seconds.
On Preprod in May, this stage took 1 hour and 30 minutes, with the first full upgrade itself taking 20 minutes 23 seconds, and subsequent app-server only upgrades taking between 4 and 5 minutes each.

  1. Bb_Team: Note start time on whiteboard. Note in RFC that stage has started.
  2. Bb_Team: Tweet that stage has started.
  3. Bb_Team: Update upgrade status page.
  4. ALL: Verify that Blackboard is not running.
  5. DBA: Reset password for "sys" and "system"

Java Security Randomness Fix

  1. APPS_TEAM: Edit the random source file attribute of the $JAVA_HOME/jre/lib/security/java.security file on each node.
    # with staff account, e.g. xxx, xxx. NOT bbuser
    # on each node
    # confirm it's not bbuser
    whoami
    export JAVA_HOME=/usr/lib/jvm/java-1.8.0-oracle.x86_64/ ;
    adm sed -i 's/securerandom.source=file:\/dev\/random/securerandom.source=file:\/dev\/urandom/' $JAVA_HOME/jre/lib/security/java.security;

Log Spam fix

  1. NOTE: NEXT STEPS SHOULD BE COMPLETED IN SAME SHELL USING bbuser
  2. APPS_TEAM: change user to bbuser.
  3. APPS_TEAM: Remove the data_integration_log_b directory, to prevent log spam. On each node.
    adm su bbuser
    >> Dev 1+2
    cd /data/bb/blackboard/content/vi/BBLEARN1/lucene ;
    rm -rf data_integration_log_b/
    >> Preprod and Live
    cd /data/bb/blackboard/content/vi/bb_bb60/lucene/ ;
    rm -rf data_integration_log_b/

First full upgrade – tasks node hostname_removed

  1. NOTE: NEXT STEPS SHOULD BE COMPLETED IN SAME SHELL USING bbuser
  2. APPS_TEAM: using shell on tasks node hostname_removed change user to bbuser
    adm su - bbuser
  3. APPS_TEAM: Set JAVA_HOME to Java 8 JDK location; confirm JAVA_HOME set correctly; and change to the installer directory.
    export JAVA_HOME=/usr/lib/jvm/java-1.8.0-oracle.x86_64/ ;
    echo $JAVA_HOME ;
    cd /usr/local/bbinstaller
  4. APPS_TEAM: Run upgrade [as bbuser]
    ./installer.sh -c /usr/local/bbinstaller/installer.properties | tee -a bb-installer-log.txt 2>&1
  5. ALL: During upgrade monitor two logs – could do this in two putty windows on big screen.
    tail -f /usr/local/bbinstaller/bb-installer-log.txt
    tail -f /usr/local/blackboard/logs/update-tools/update-tool-log.txt

Four "app server only" upgrades – web node hostname_removed to hostname_removed

  1. NOTE: NEXT STEPS SHOULD BE COMPLETED IN SAME SHELL USING bbuser
  2. APPS_TEAM: on web nodes, starting with hostname_removed, doing one node at a time, change user to bbuser
    adm su - bbuser
  3. APPS_TEAM: Set JAVA_HOME to Java 8 JDK location; confirm JAVA_HOME set correctly; and change to the installer directory.
    export JAVA_HOME=/usr/lib/jvm/java-1.8.0-oracle.x86_64/ ;
    echo $JAVA_HOME ;
    cd /usr/local/bbinstaller
  4. APPS_TEAM: Run app-server only upgrade [as bbuser]
    ./installer.sh -c /usr/local/bbinstaller/installer.properties -a| tee -a bb-installer-log.txt 2>&1
  5. ALL: During upgrade monitor two logs – could do this in two putty windows on big screen.
    tail -f /usr/local/bbinstaller/bb-installer-log.txt
    tail -f /usr/local/blackboard/logs/update-tools/update-tool-log.txt
  6. APPS_TEAM, repeat above until all four web nodes are upgraded.

Post-upgrade activities

  1. SYS_TEAM: If upgrades are successful, take VMWare Snapshot #3.
  2. Bb_Team: Check you can login.
  3. APPS_TEAM: Stop Blackboard on each node, web nodes (hostname_removed – hostname_removed) first, then tasks node (hostname_removed).
    # on each node, web first, then tasks
    # web nodes may be stopped concurrently
    service blackboard stop
  4. Bb_Team: Note finish time on board.

Stage 5 - Day 1 - Verifying whether previous configuration changes have been removed

Having successfully upgraded Blackboard we will verify that previous configuration changes have been retained.

Expected start time: 12:35 23 June 2018

On Dev 1, this stage took 17 minutes.
On Dev 2, this stage took 5 minutes.
On Preprod, this stage took 8 minutes.
On Dev 1 in May, this stage took 4 minutes.
On Dev 2 in May, this stage took 5 minutes.
On Preprod in May, this stage took 10 minutes.

  1. Bb_Team: Note start time on whiteboard. Note in RFC that stage has started.
  2. Bb_Team: Tweet that stage has started.
  3. Bb_Team: Update upgrade status page.

Back up key files to monitor if there have been any changes

  1. APPS_TEAM: Backup the various configuration files on each node to record any changes following the upgrade.
    # as bbuser
    # on each node
    mkdir /home/bbuser/post_upgrade_files ;
    cp /usr/local/blackboard/config/internal/http-cacheable-extensions.txt /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/config/internal/xythos-indexing-filter.txt /home/bbuser/post_upgrade_files/ ;
    cp /usr/lib/jvm/java-1.8.0-oracle.x86_64/jre/lib/security/java.security /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/config/tomcat/conf/server.xml.bb /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/config/internal/nautilus-config.properties /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/config/service-config.properties /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/config/tomcat/conf/wrapper.conf /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/config/bb-config.properties /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/config/bb-config.properties.template /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/config/tomcat/classes/Search.properties.bb /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/config/tomcat/classes/Search.properties /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/config/cache-settings.properties /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/config/message-queue-service-config.xml /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/config/message-queue-service-config.xml.bb /home/bbuser/post_upgrade_files/
    cp /usr/local/blackboard/system/database/vi/discussionboard/functions/forum_is_enabled.sql /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/content/vi/bb_bb60/plugins/bb-xss-filter/webapp/WEB-INF/classes/blackboard/xss/regex/bb-xss-regex-filter.properties /home/bbuser/post_upgrade_files/;
    cp /usr/local/blackboard/config/internal/bb-file-filter-configuration.properties /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/docs/javascript/ngui/widget.js /home/bbuser/post_upgrade_files/ ;
  2. APPS_TEAM: On each node verify that the correct number of files have been copied.
    # as bbuser
    # on each node
    ls -1 /home/bbuser/post_upgrade_files/ |wc -l
    # result must equal 19

Verification 1 - bb-config.properties

  1. ALL: Verify that the following settings held in bb-config.properties files have been preserved on each node.
    # on each node
    # as bbuser
    ~/bb-config-tests-1.sh
  2. ALL: List which tests failed.

Verification 2 - Other config changes

  1. Bb_Team: Verify that the settings held in other files have been preserved on each node.
    # on each node
    # as bbuser
    ~/bb-config-tests-2.sh
  2. ALL: List which tests failed.
  3. Bb_Team: Note finish time on board.

Stage 6 - Day 1 - Configuration changes

Here we will restore files that get deleted by the installer and re-apply any configuration changes that failed our tests.

Expected start time: 13:50 23 June 2018


On Dev 1, this stage took 10 minutes.
On Dev 2, this stage took 7 minutes.
On Preprod, this stage took 23 minutes.
On Dev 1 in May, this stage took 10 minutes.
On Dev 2 in May, this stage took 15 minutes.
On Preprod in May, this stage took 35 minutes.

  1. Bb_Team: Note start time on whiteboard. Note in RFC that stage has started.
  2. Bb_Team: Tweet that stage has started.
  3. Bb_Team: Update upgrade status page.

Restoring files

The installer will often delete files that are not part of a standard Blackboard installation, we will restore those files as required.

Restore Branding

  1. APPS_TEAM: On each node restore branding from /home/bbuser/login.zip to /usr/local/blackboard/docs/login/.
    # on each node
    # as bbuser
    cd /home/bbuser/ ;
    mkdir login-extracted ;
    unzip login.zip -d login-extracted ;
    cd login-extracted/usr/local/blackboard/docs/login ;
    cp -n * /usr/local/blackboard/docs/login
    # ignore message "cp: omitting directory `WEB-INF'"

Restore data integration files

  1. APPS_TEAM: on tasks node, restore data_integration directory.
    # as bbuser
    # on tasks node hostname_removed
    # verify whether
    # /usr/local/blackboard/data_integration/ folder has been wiped.
    # if not ask Bb_Team to double check contents
    # if Bb_Team agree it is ok, then skip the following.
    cd /home/bbuser/ ;
    mkdir data_integration-extracted ;
    unzip data_integration.zip -d data_integration-extracted ;
    cd data_integration-extracted/usr/local/blackboard/data_integration/ ;
    cp -n /home/bbuser/data_integration-extracted/* /usr/local/blackboard/data_integration

Restore photos

  1. APPS_TEAM: on tasks node, restore PHOTOS directory.
    # as bbuser
    # on tasks node hostname_removed
    # verify whether
    # /usr/local/blackboard/content/PHOTOS/ folder has been wiped.
    # if not ask Bb_Team to double check contents
    # if Bb_Team agree it is ok, then skip the following.
    cd /home/bbuser/
    mkdir PHOTOS-extracted
    unzip PHOTOS.zip -d PHOTOS-extracted
    cd /usr/local/blackboard/content/PHOTOS/
    # if PHOTOS directory has been removed, create a new one
    mkdir /usr/local/blackboard/content/PHOTOS/
    cp -n /home/bbuser/PHOTOS-extracted/* /usr/local/blackboard/content/PHOTOS/

Copy the new forum_is_enabled.sql file

  1. APPS_TEAM: On each node copy the new forum_is_enabled.sql file (see https://blackboard.secure.force.com/btbb_caseview?id=5003900001zkPvwAAE for background). Note that we took a back up of the file earlier and placed it in the /home/bbuser/post_upgrade_files/ directory.
    # on each node
    # as bbuser
    cp /home/bbuser/forum_is_enabled.sql /usr/local/blackboard/system/database/vi/discussionboard/functions/forum_is_enabled.sql

Copy the new widget.js file

  1. APPS_TEAM: On each node copy the new widget.js file (see https://blackboard.secure.force.com/btbb_caseview?id=50039000023VpUqAAK for background). Note that we took a back up of the file earlier and placed it in the /home/bbuser/post_upgrade_files/ directory.
    # on each node
    # as bbuser
    cp /home/bbuser/widget.js /usr/local/blackboard/docs/javascript/ngui/widget.js

Changes in bb-config.properties

The bb-config.properties file contains a number of key settings we use that can be overwritten in the installation/upgrade process. A list of all changes is held at hostname_removed.

Apply Scorm fix to bb-config-properties.template

  1. APPS_TEAM: If earlier bb-config.properties Test 1 failed, apply the SCORM fix to bb-config-properties.template on each node.
    # as bbuser
    # on each node
    sed -i "s/bbconfig.headers.CSP.frame-ancestors=@bbconfig.headers.CSP.frame-ancestors@/bbconfig.headers.CSP.frame-ancestors=self\' https:\/\/\*.soton.ac.uk \'soton.ac.uk/g" /usr/local/blackboard/config/bb-config.properties.template

Apply Scorm fix to bb-config-properties

 

Other changes to bb-config.properties

Any other required changes to bb-config.properties to be agreed and inserted here if required.

Verification script

  1. ALL: Verify that configuration changes to bb-config.properties files have made successfully on each node.
    # on each node
    # as bbuser
    ~/bb-config-tests-1.sh
  2. ALL: Check that all tests have passed.

Changes in other files

We have applied a number of changes and optimisations to other configuration files. Where required we re-implement those changes here. A list of all changes is held at hostname_removed.

Apply Math Editor fix

  1. APPS_TEAM: Assuming that other tests Test E failed, and provided no changes identified in http-cacheable-extensions.txt file, copy it back from the original files folder.
    # as bbuser
    # on each node
    cp /home/bbuser/original_files/http-cacheable-extensions.txt /usr/local/blackboard/config/internal/http-cacheable-extensions.txt

Apply Notification Distribution Fix

  1. APPS_TEAM: Assuming that other tests Test H failed, change notification setting from default 10000 to 20000. N.B. File has already been backed up in earlier step.
    # as bbuser
    # on each node
    sed -i 's/nautilus.distribution.notificationsPerDistribution=10000/nautilus.distribution.notificationsPerDistribution=20000/' /usr/local/blackboard/config/internal/nautilus-config.properties

Wrapper.conf and wrapper.conf.bb fixes

  1. APPS_TEAM: Assuming that other tests Tests A, B, C, D, X7, and X8 failed, restore change to wrapper.conf and wrapper.conf.bb on each node.
    # as bbuser
    # on each node
    sed -i 's/wrapper.shutdown.timeout=300/wrapper.shutdown.timeout=30/' /usr/local/blackboard/config/tomcat/conf/wrapper.conf ;
    sed -i 's/wrapper.jvm_exit.timeout=15/wrapper.jvm_exit.timeout=5/' /usr/local/blackboard/config/tomcat/conf/wrapper.conf ;
    sed -i 's/wrapper.ping.timeout=60/wrapper.ping.timeout=90/' /usr/local/blackboard/config/tomcat/conf/wrapper.conf ;
    sed -i 's/wrapper.shutdown.timeout=300/wrapper.shutdown.timeout=30/' /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb ;
    sed -i 's/wrapper.jvm_exit.timeout=15/wrapper.jvm_exit.timeout=5/' /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb ;
    sed -i 's/wrapper.ping.timeout=60/wrapper.ping.timeout=90/' /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb

Resolve LRN-107071: High CPU and High Load Caused by Script MicrosoftDocumentParser.sh

APPS_TEAM: Assuming that other tests Tests J and K failed, apply fix to Search.properties files on each node.

  1. # as bbuser
    # on each node
    sed -i 's/ExtensionsNotToIndex=exe,dll,zip,jpg,gif,tif,tiff,dmp/ExtensionsNotToIndex=exe,dll,zip,jpg,gif,tif,tiff,dmp,pdf,doc,docx,ppt,pps,pptx,ppsx,ppts,xlsx,xls,xlsm/' /usr/local/blackboard/config/tomcat/classes/Search.properties ;
    sed -i 's/ExtensionsNotToIndex=exe,dll,zip,jpg,gif,tif,tiff,dmp/ExtensionsNotToIndex=exe,dll,zip,jpg,gif,tif,tiff,dmp,pdf,doc,docx,ppt,pps,pptx,ppsx,ppts,xlsx,xls,xlsm/' /usr/local/blackboard/config/tomcat/classes/Search.properties.bb
  2. APPS_TEAM: Restore the /usr/local/blackboard/config/internal/xythos-indexing-filter.txt file that we backed up earlier.
    # as bbuser
    # on each node
    cp /home/bbuser/original_files/xythos-indexing-filter.txt /usr/local/blackboard/config/internal/xythos-indexing-filter.txt

Adjust logging and task management

  1. APPS_TEAM: Assuming that other tests Test F failed, edit the /usr/local/blackboard/config/service-config.properties on web nodes only to ensure that the tasks node is the only node set to carry out background tasks, thus optimising performance of web facing end user nodes.
    # as bbuser
    # on web nodes hostname_removed - hostname_removed
    sed -i 's/blackboard.service.queuedtaskmanager.param.num-concurrent-tasks=2/blackboard.service.queuedtaskmanager.param.num-concurrent-tasks=0/' /usr/local/blackboard/config/service-config.properties
  2. APPS_TEAM: Assuming that other tests Test G failed, change logging level to fatal on web nodes only
    # as bbuser
    # on web nodes hostname_removed - hostname_removed
    sed -i 's/blackboard.service.log.param.logdef.services.verbosity=warning/blackboard.service.log.param.logdef.services.verbosity=fatal/' /usr/local/blackboard/config/service-config.properties

Apply Config Cache fix

  1. APPS_TEAM: Assuming that other tests Test X1 and X2 failed, on each node, edit /usr/local/blackboard/config/cache-settings.properties and reduce bbconfig.cache.contextualizedNodeAffiliates.timetoidle=345600 to 1800, and reduce bbconfig.cache.default.timetoidle=172800 to 21600. All nodes.
    # as bbuser
    # on each node
    sed -i 's/bbconfig.cache.contextualizedNodeAffiliates.timetoidle=345600/bbconfig.cache.contextualizedNodeAffiliates.timetoidle=1800/' /usr/local/blackboard/config/cache-settings.properties ;
    sed -i 's/bbconfig.cache.default.timetoidle=172800/bbconfig.cache.default.timetoidle=21600/' /usr/local/blackboard/config/cache-settings.properties

Apply Message Queue fix

  1. APPS_TEAM: Assuming that other tests Test X3, X4, X5, and X6 failed, on each node, edit /usr/local/blackboard/config/message-queue-service-config.xml and /usr/local/blackboard/config/message-queue-service-config.xml.bb. Increase memoryUsage limit from 256 MB to 512 MB and increase storeUsage limit from 1g to 2g. On all nodes.
    # as bbuser
    # on each node
    sed -i 's/memoryUsage limit="256 MB"/memoryUsage limit="512 MB"/' /usr/local/blackboard/config/message-queue-service-config.xml ;
    sed -i 's/storeUsage limit="1g"/storeUsage limit="2g"/' /usr/local/blackboard/config/message-queue-service-config.xml ;
    sed -i 's/memoryUsage limit="256 MB"/memoryUsage limit="512 MB"/' /usr/local/blackboard/config/message-queue-service-config.xml.bb ;
    sed -i 's/storeUsage limit="1g"/storeUsage limit="2g"/' /usr/local/blackboard/config/message-queue-service-config.xml.bb

Apply XSS security fix

Apply fix to bb-xss-regex-filter.properties
  1. APPS_TEAM: edit the bb-xss-regex-filter.properties file on each node. Note that doing on one node should do on all, but verify on at least two nodes to be sure.
    # On each node (note above about nodes)
    # As bbuser
    nano /usr/local/blackboard/content/vi/bb_bb60/plugins/bb-xss-filter/webapp/WEB-INF/classes/blackboard/xss/regex/bb-xss-regex-filter.properties
    Change:
    disabled_tags.template=(?<=<)\\s*@X@tag@X@(?=[\\s/|>])|(?<=</)\\s*@X@tag@X@(?=\\s*>?)
    to:
    disabled_tags.template=(?<=<)\\s*(?>[^:<>]+:)?\\s*@X@tag@X@(?=[\\s/|>])|(?<=</)\\s*(?>[^:<>]+:)?\\s*@X@tag@X@(?=\\s*>?)
  2. APPS_TEAM: display file contents to confirm the change was made
    # On each node
    # As bbuser
    cat /usr/local/blackboard/content/vi/bb_bb60/plugins/bb-xss-filter/webapp/WEB-INF/classes/blackboard/xss/regex/bb-xss-regex-filter.properties | grep disabled_tags.template ;
Apply fix to bb-file-filter-configuration.properties
  1. APPS_TEAM: edit the bb-file-filter-configuration.properties
    ## On each node
    ## As bbuser
    nano /usr/local/blackboard/config/internal/bb-file-filter-configuration.properties
  2. APPS_TEAM: Change the section:
    # Filtered file types
    .htm=filter
    .html=filter
    .css=filter
    .js=filter
    .txt=filter
    .xml=filter
    .svg=filter
    to
    # Filtered file types
    .htm=filter
    .html=filter
    .xhtml=filter
    .xhm=filter
    .css=filter
    .js=filter
    .txt=filter
    .xml=filter
    .svg=filter
    .svgz=filter
    .xsl=filter
  3. APPS_TEAM: display file contents to confirm the change was made correctly
    cat /usr/local/blackboard/config/internal/bb-file-filter-configuration.properties

 

Verification script

  1. ALL: Verify that configuration changes to other files files have been made successfully on each node.
    # on each node
    # as bbuser
    ~/bb-config-tests-2.sh
  2. ALL: Check that all tests have passed.
  3. Bb_Team: Note finish time on board.

Stage 7 - Day 1 - Pushconfigupdates

Stage 7 is where configuration changes are applied to the application using pushconfigupdates.

Expected start time: 14:55 23 June 2018


On Dev 1, this stage took 20 minutes.
On Dev 2, this stage took 20 minutes.
On Preprod, this stage took 1 hour and 26 minutes.
On Dev 1 in May, this stage took 10 minutes.
On Dev 2 in May, this stage took 20 minutes.
On Preprod in May, this stage took 45 minutes.

  1. Bb_Team: Note start time on whiteboard. Note in RFC that stage has started.
  2. Bb_Team: Tweet that stage has started.
  3. Bb_Team: Update upgrade status page.
  4. ALL: Verify that Blackboard is stopped.
  5. APPS_TEAM: Run pushconfigupdates to propagate the configuration changes on each node, starting with the tasks node (hostname_removed).
    # as bbuser
    # on each node, tasks node first
    cd /usr/local/blackboard/tools/admin/ ;
    ./PushConfigUpdates.sh | tee -a /home/bbuser/pushconfig-stage6.txt.$(date +"%Y-%m-%d") 2>&1
  6. ALL: Monitor progress on tv
    # on big tv
    tail -f /home/bbuser/pushconfig-stage6.txt.$(date +"%Y-%m-%d")
    # during the process
  7. Bb_Team: Check you can log in.
  8. APPS_TEAM: Repeat the verification scripts on the node on which pushconfigupdates has run. This is to check that pushconfigupdates has not undone any of our earlier changes.
    # as bbuser
    # on each node
    ~/bb-config-tests-1.sh
    ~/bb-config-tests-2.sh
  9. APPS_TEAM: Repeat above process until all nodes have had pushconfigupdates run.
  10. APPS_TEAM: Having confirmed that Bb_Team were able to login to each node, stop Blackboard on each node, web nodes (hostname_removed – hostname_removed) first, then tasks node (hostname_removed).
    # on each node, web first, then tasks
    # web nodes may be stopped concurrently
    service blackboard stop
  11. SYS_TEAM: Take VMWare Snapshot #4.
  12. APPS_TEAM: Stop Blackboard on each node, web nodes (hostname_removed – hostname_removed) first, then tasks node (hostname_removed).
    # on each node, web first, then tasks
    # web nodes may be stopped concurrently
    service blackboard stop
  13. Bb_Team: Note finish time on board.

Stage 8 - Day 1 – Log files check



Expected start time: 15:50 23 June 2018

  1. ALL: Review key log files to identify any potential issues. See hostname_removed/wiki/Blackboard/Troubleshooting#Blackboard_Logs for a list of log files to check.
  2. ALL: Research any issues on the Blackboard support site, and raise a ticket with Blackboard support if required.

Stage 9 - Day 2 – Log files check



Expected start time: 09:00 24 June 2018
N.B. Highfield team to be in place and ready to start by 08:55

  1. ALL: Review key log files to identify any potential issues. See hostname_removed/wiki/Blackboard/Troubleshooting#Blackboard_Logs for a list of log files to check.
  2. ALL: Research any issues on the Blackboard support site, and raise a ticket with Blackboard support if required.
  3. Bb_Team: Check that data integration snapshots ran ok overnight.

Stage 10 - Day 2 – GUI based configuration and UAT

This stage encompasses Bb_Team team GUI based configuration and testing.

Expected start time: 09:35 24 June 2018


On Dev 1, this stage took 1 hour and 21 minutes
On Dev 2, this stage took 40 minutes
On Preprod, this stage took 1 hour and 6 minutes
On Dev 1 in May, this stage took 30 minutes
On Dev 2 in May, this stage took 35 minutes
On Preprod in May, this stage took 1 hour and 20 minutes

  1. Bb_Team: Note start time on whiteboard. Note in RFC that stage has started.
  2. Bb_Team: Tweet that stage has started.
  3. Bb_Team: Update upgrade status page.

Permissions fix

  1. Bb_Team: Use the support tool to generate the output to these queries:
    select *
    from ENTITLEMENT
    order by pk1 DESC
    select *
    from SYSTEM_ROLES_ENTITLEMENT
    order by pk1 DESC
  2. Bb_Team: Using the methodology described in the document hostname_removed create an SQL query to restore access to the Spell Check and Math Editor settings on the Sys Admin panel. Send the query to DBA.
  3. DBA: Back up the SYSTEM_ROLES_ENTITLEMENT and ENTITLEMENT tables and commit the query.

Branding

Bb_Team: Dev/Preprod - adjust version number in Gateway file.

  1. # as bbuser
    # on each node
    nano /usr/local/blackboard/webapps/content/branding/_1_1/gateway.bb
    nano /usr/local/blackboard/content/vi/BBLEARN1/branding/gateway.bb
    nano /usr/local/blackboard/content/vi/bb_bb60/branding/gateway.bb

Building Blocks

Portfolio

  1. Bb_Team: Check whether the Portfolio Building Block is currently installed.
  2. Bb_Team: Stop Blackboard on each node, web nodes (hostname_removed – hostname_removed) first, then tasks node (hostname_removed).
    # on each node, web first, then tasks
    # web nodes may be stopped concurrently
    service blackboard stop
  3. Bb_Team: If the Portfolio building block is installed, remove it.
    # as bbuser
    # on tasks node (hostname_removed)
    cd /usr/local/blackboard/tools/admin/ ;
    ./B2Manager.sh -dor bb-portfolio
  4. Bb_Team: Delete the portfolio directory.
    # as bbuser
    # on tasks node (hostname_removed)
    DEVs: cd /usr/local/blackboard/content/vi/BBLEARN1/plugins
    PPRD/LIVE cd /usr/local/blackboard/content/vi/bb_bb60/plugins
    rm -r bb-portfolio/
  5. Bb_Team: Check whether the portfolio building block exists at this location.
    /usr/local/blackboard/system/autoinstall/enterprise.contentsystem/allavailable/portfolio.war
  6. Bb_Team: If not then download the portfolio b2 to bbuser's home directory.
    # as bbuser
    # on tasks node (hostname_removed)
    cd
    wget hostname_removed.ac.uk/bb/bb2018-fixes/portfolio.war
  7. Bb_Team: Load the portfolio building block.
    # as bbuser
    # on tasks node (hostname_removed)
    cd /usr/local/blackboard/tools/admin/
    # if using local version
    ./B2Manager.sh -doi /usr/local/blackboard/system/autoinstall/enterprise.contentsystem/allavailable/portfolio.war
    # if using bbuser version
    ./B2Manager.sh -doi /home/bbuser/bb-portfolio.war
  8. Bb_Team: Verify the building block
    # as bbuser
    # on tasks node (hostname_removed)
    ./B2Manager.sh -v bb-portfolio
    # should result with Version: 3300.0.1-rel.60+a05276c
  9. Bb_Team: Make the Portfolio building block available.
    # as bbuser
    # on tasks node (hostname_removed)
    ./B2Manager.sh -s AVAILABLE bb-portfolio
  10. Bb_Team: Start Blackboard on tasks node (hostname_removed).
    # as bbuser
    # on tasks node only
    service blackboard start

Other Building Blocks

  1. Bb_Team: If required, update Assignments Building Block on tasks node GUI.
    location: hostname_removed\Assignments\Version 3300.0.1-rel.61+f9847e0\bb-assignment.war
  2. Bb_Team: Restart Blackboard on tasks node (hostname_removed).
    # on tasks node only
    service blackboard restart
  3. Bb_Team: If required, update Partner Cloud building block on tasks node GUI.
    Currently held at hostname_removed\bbgs-partner-cloud.war
  4. Bb_Team: Restart Blackboard on tasks node (hostname_removed).
    service blackboard restart
  5. Bb_Team: If required, update Telemetry building block on tasks node GUI.
    Currently held at hostname_removed\bb-telemetry.war
  6. Bb_Team: Restart Blackboard on tasks node (hostname_removed).
    service blackboard restart
  7. Bb_Team: Update Qwickly building block on tasks node GUI.
    Currently held at hostname_removed\qwickly\4.4.2.1 for 2017 Q4\Qwickly4.4.2.1.war
  8. Bb_Team: Restart Blackboard on tasks node (hostname_removed).
    service blackboard restart
  9. Bb_Team: Update Turnitin building block on tasks node GUI.
    Currently held at hostname_removed\Turnitin\2.7.5\bb-basic-2.7.5.war
  10. Bb_Team: Restart Blackboard on tasks node (hostname_removed).
    service blackboard restart
  11. Bb_Team: Having checked you can log in, start Blackboard on web nodes hostname_removed - hostname_removed.
    # as bbuser
    # on web nodes
    service blackboard start

Disable certain building block tools

  1. Bb_Team: On tasks node GUI, disable Portfolio and Ally Tools via
    1. Administrator Panel
    2. Tools
    3. Ally -> All OFF
    4. API Gateway -> OFF
    5. Portfolio -> OFF
    6. Submit
  2. Bb_Team: Check that Video Integration tool (currently unsupported) is not enabled.
    1. Administrator Panel
    2. Building Blocks
    3. Video Integration
    4. Settings
    5. Availability -> Off
    6. Submit

Other GUI based configuration

  1. Bb_Team: Check that language packs have not been disabled.
  2. Bb_Team: Check whether UK English UoS version needs to be updated. Update if necessary by following the guidance at hostname_removed/wiki/Blackboard/Language_Pack_Changes
  3. Bb_Team: Disable "Application Authorisation".
    1. System Admin
    2. Tabs and Modules
    3. Tool Panel
    4. Application Authorisation - Delete
    5. Goals - Delete
    6. Goal Performance - Delete
    7. Enterprise Surveys - Delete
  4. Bb_Team: Set the default course theme.
    1. System Admin
    2. Brands and Themes
    3. Default Course Design
    4. Set "Use Theme Colours" to "BB2012 Theme"
    5. Click Preview
    6. Click Submit
  5. Bb_Team: Set Google Oauth code
    1. System Admin
    2. Building Blocks
    3. Installed Tools
    4. Page 2
    5. Content Editor -> Settings
    6. Set Oauth address to that stated in hostname_removedGoogleAPI codes-for-WYSIWYG_editor\
  6. Bb_Team: Check that spell check is set to UK English and other dictionaries are disabled.
  7. Bb_Team: Set students to not be able to email all users on a course:
    1. Course/Organisation Roles > Manage Privileges: Student/Participant
    2. All users -> Untick.
    3. All students -> Untick.
  8. Bb_Team: Check Mobile Building Block settings via Building Blocks > Installed Tools > Blackboard Mobile Web Services Building Block > Server Registration Settings and ensure that Environment is set correctly.
  9. Bb_Team: Set icon set to 2016 theme.

Testing

  1. Bb_Team: UAT Blackboard via GUI on each node.
  2. Bb_Team: Scan testing results and save in J drive upgrade folder.
  3. Bb_Team: Note finish time on board.

Stage 11 - Day 2 – Restart Test

In this stage we check that Blackboard restarts ok.


Expected start time: 13:10 24 June 2018

  1. Bb_Team: Note start time on whiteboard. Note in RFC that stage has started.
  2. Bb_Team: Tweet that stage has started.
  3. Bb_Team: Update upgrade status page.
  4. SYS_TEAM: Perform vApp stop of Blackboard nodes in line with standard method used for patching.
  5. SYS_TEAM: Perform vApp start of Blackboard nodes in line with standard method used for patching.
  6. APPS_TEAM: Once Blackoard available again, restart ActiveMQ on tasks node to ensure that it is running correctly.
    >> as bbuser
    >> on tasks node
    cd /usr/local/blackboard/tools/admin
    ./ServiceController.sh services.activemqbroker.restart
  7. Bb_Team: Check you can log into web nodes.
  8. Bb_Team: Make a non-immediate announcement on each node, to test later that ActiveMQ is working. Emails should appear within an hour.
  9. Bb_Team: On each node, add a new item to a test course and check it appears ok.
  10. Bb_Team: Note stage completion time on whiteboard.

Stage 12 - Day 2 - Go Live



Expected start time: 14:45 24 June 2018

  1. Bb_Team: Note start time on whiteboard. Note in RFC that stage has started.
  2. Bb_Team: Tweet that stage has started.
  3. SYS_TEAM: Following confirmation by Bb_Team, make Blackboard live via NLB.
  4. Bb_Team: Announce service availability via twitter @uos_Bb_Team and blackboard-users@lists mailing list.
  5. Bb_Team: Note stage completion time on whiteboard.
  6. Bb_Team: Photograph whiteboard.
  7. Bb_Team: Scan testing sheets and attach PDFs to RFC
  8. Bb_Team: Close RFC.

Stage 13 - Day 2 – Epilogue

This stage involves tidying up after successful testing.

Expected start time: 15:10 24 June 2018

  1. Bb_Team: Note start time on whiteboard. Note in RFC that stage has started.
  1. APPS_TEAM: Remove backups of zip files and uncompressed directories.
    # as bbuser
    # on all nodes
    rm /home/bbuser/login.zip
    rm /home/bbuser/photos.zip
    rm -rf /home/bbuser/login-extracted
    # on tasks node
    rm /home/bbuser/snapshotfiles.zip
    rm /home/bbuser/data_integration.zip
    rm -rf /home/bbuser/data_integration-extracted
    rm -rf /home/bbuser/snapshotfiles-extracted
  2. APPS_TEAM: Remove copied licence file.
    # as bbuser
    # on all nodes
    rm /home/bbuser/blackboard-licence .xml
  3. APPS_TEAM: Clear installer directory.
    # bbuser
    rm -rf /usr/local/bbinstaller
  4. DBA: Reset DB passwords for sys, system.
  5. SYS_TEAM: Following specific confirmation from Bb_Team delete snapshots.
  6. Bb_Team: Note finish time on board.
  7. Bb_Team: Close RFC.

Summary of VMWare Snapshots

Snapshot #1 – Installer in place, licence file copied, branding backed up, lucene log fix, permissions set, no application changes.

Snapshot #2 - Oracle patch applied

Snapshot #3 - Application upgraded

Snapshot #4 – Configuration changes applied

Outcomes