Skip navigation
All People > mdeeprose > Matt's Upgrade Blog > 2018 > November > 08

This is a follow up to last years post Highlights of fixes from our upgrade implementation plan where I've tried to summarise all the little fixes that we have had to implement due to issues with the Blackboard application, and which the installer has a nasty habit of removing each time it is used to perform an upgrade.

 

This is purely for information purposes and Blackboard support should be consulted before making changes to your installation.

 

Apologies for the occasional formatting issues.

Fixes in bb-config.properties

Files affected:
/usr/local/blackboard/config/bb-config.properties
/usr/local/blackboard/config/bb-config.properties.template

DEV/PREPROD/LIVE Java Log Spam fix

Files affected:
/usr/local/blackboard/config/bb-config.properties

 

To prevent log spam in the stdout-stderr log perform the following:

#on each node
mkdir -p /home/bbuser/.java/.userPrefs mkdir /home/bbuser/.java/.systemPrefs chown -R bbuser:bbuser /home/bbuser/.java/ chmod 755 -Rv /home/bbuser/.java/ #on each node #within /usr/local/blackboard/config/bb-config.properties #add to bbconfig.jvm.options.extra.tomcat= -Djava.util.prefs.systemRoot=/home/bbuser/.java/

Verification

cat /usr/local/blackboard/config/bb-config.properties |grep Djava
result should contain: bbconfig.jvm.options.extra.tomcat=-Djava.util.prefs.systemRoot=/home/bbuser/.java/
Depending on environment additional entries will be included within this stanza.

DEV/PREPROD/LIVE SCORM connection loss fix

Files affected:
/usr/local/blackboard/config/bb-config.properties
/usr/local/blackboard/config/bb-config.properties.template

 

As recommended by Blackboard support in case #02455370 apply a fix in two files to prevent SCORM learning objects losing connection to Blackboard.

As bbuser edit bb-config.properties.template on each node and make the following change.

#as bbuser
#on each node
sed -i "s/bbconfig.headers.CSP.frame-ancestors=self/bbconfig.headers.CSP.frame-ancestors=self\' https:\/\/\*.soton.ac.uk \'soton.ac.uk/g" /usr/local/blackboard/config/bb-config.properties.template ;
sed -i "s/bbconfig.headers.CSP.frame-ancestors=self/bbconfig.headers.CSP.frame-ancestors=self\' https:\/\/\*.soton.ac.uk \'soton.ac.uk/g" /usr/local/blackboard/config/bb-config.properties

Verification

#on each node
cat /usr/local/blackboard/config/bb-config.properties.template | grep "ancestors=self'"
#Result should be bbconfig.headers.CSP.frame-ancestors=self' https://*.soton.ac.uk 'soton.ac.uk
#on each node
cat /usr/local/blackboard/config/bb-config.properties.| grep "ancestors=self'"
#Result should be bbconfig.headers.CSP.frame-ancestors=self' https://*.soton.ac.uk 'soton.ac.uk

DEV/PREPROD/LIVE Resolve "JDK 1.8.0_151 and 1.8.0_152 Break Grade Center" issue

Files affected:
/usr/local/blackboard/config/bb-config.properties

 

According to https://blackboard.secure.force.com/apex/btbb_articleview?id=kA3390000004HdwCAE the Java Development Kits (jdk1.8.0_152 and jdk1.8.0_151) break the Grade Center when compression is turned on, displaying only JSON errors.

 

To avoid this happening due to an automatic java update we will follow the recommended steps to turn off http compression.

#As bbuser
#On each node
sed -i 's/bbconfig.appserver.http.compression=on/bbconfig.appserver.http.compression=off/' /usr/local/blackboard/config/bb-config.properties

Verification:

#As bbuser
#On each node
cat /usr/local/blackboard/config/bb-config.properties |grep bbconfig.appserver.http.compression
result should be bbconfig.appserver.http.compression=off

Summary of all changes made to bb-config.properties

Files affected:
/usr/local/blackboard/config/bb-config.properties
/usr/local/blackboard/config/bb-config.properties.template

 

The bb-config.properties file is one of the primary locations for Blackboard settings. With the 2016 Q4 upgrade we made the following adjustments. Note that this includes the Java log spam, http compression and SCORM connection loss fix already detailed above.

As bbuser edit the bb-config.properties file on each node and make the following changes.

#as bbuser
#on each node
nano /usr/local/blackboard/config/bb-config.properties

 

DEV/PREPROD/LIVE Change 1

#Change bbconfig.max.stacksize.tomcat=
#from 400k to 512k
i.e. bbconfig.max.stacksize.tomcat=512k

Verification

#On each node
#Change 1
cat /usr/local/blackboard/config/bb-config.properties | grep "bbconfig.max.stacksize.tomcat"
#Result should be bbconfig.max.stacksize.tomcat=512k

PREPROD/LIVE Change 2

#Change bbconfig.jvm.options.extra.tomcat=
#So that whole section is replaced with
bbconfig.jvm.options.extra.tomcat=-XX:NewSize=2048m -XX:MaxNewSize=2048m -XX:+UseTLAB -XX:SurvivorRatio=4 -XX:+UseCompressedOops -Xss512k -XX:+PrintVMOptions -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCTaskTimeStamps -XX:+PrintCommandLineFlags -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+AlwaysPreTouch -verbose:gc -XX:+DoEscapeAnalysis -Xverify:none -XX:+UseMontgomeryMultiplyIntrinsic -XX:+UseMontgomerySquareIntrinsic -XX:MetaspaceSize=768m -XX:MaxMetaspaceSize=1536m -Djava.util.prefs.systemRoot=/home/bbuser/.java/

Verification

cat /usr/local/blackboard/config/bb-config.properties |grep "Djava"
#Result should be bbconfig.jvm.options.extra.tomcat=--XX:NewSize=2048m -XX:MaxNewSize=2048m -XX:+UseTLAB -XX:SurvivorRatio=4 -XX:+UseCompressedOops -Xss512k -XX:+PrintVMOptions -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCTaskTimeStamps -XX:+PrintCommandLineFlags -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+AlwaysPreTouch -verbose:gc -XX:+DoEscapeAnalysis -Xverify:none -XX:+UseMontgomeryMultiplyIntrinsic -XX:+UseMontgomerySquareIntrinsic -XX:MetaspaceSize=768m -XX:MaxMetaspaceSize=1536m -Djava.util.prefs.systemRoot=/home/bbuser/.java/

PREPROD/LIVE Change 3

#Change bbconfig.jvm.options.gc=
#So that whole section is replaced with
bbconfig.jvm.options.gc=-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=45 -XX:+UseStringDeduplication

Verification

cat /usr/local/blackboard/config/bb-config.properties |grep UseG1GC
#Result should be
bbconfig.jvm.options.gc=-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=45 -XX:+UseStringDeduplication

 

DEV/PREPROD/LIVE Change 4

#Check dmarc setting, ensure that: bbconfig.email.use.dmarc.from.override=false

Verification

cat /usr/local/blackboard/config/bb-config.properties | grep "dmarc.from.override"
#Result should be bbconfig.email.use.dmarc.from.override=false

 

DEV/PREPROD/LIVE Change 5

#Update click jacking setting to resolve issue where
#SCORM learning objects suffer disconnections #Blackboard case #02455370
#Change
#bbconfig.headers.CSP.frame-ancestors=[none OR self OR <comma-delimited URL list>]
bbconfig.headers.CSP.frame-ancestors=self
#to
#bbconfig.headers.CSP.frame-ancestors=[none OR self OR <comma-delimited URL list>]
bbconfig.headers.CSP.frame-ancestors=self' https://*.soton.ac.uk 'soton.ac.uk
#Note that this is the last line the file.

Verification

cat /usr/local/blackboard/config/bb-config.properties | grep "ancestors=self'"
#Result should be bbconfig.headers.CSP.frame-ancestors=self' https://*.soton.ac.uk 'soton.ac.uk

 

DEV/PREPROD/LIVE Change 6

According to https://blackboard.secure.force.com/apex/btbb_articleview?id=kA3390000004HdwCAE the Java Development Kits (jdk1.8.0_152 and jdk1.8.0_151) break the Grade Center when compression is turned on, displaying only JSON errors.

This is apparently now fixed.

To avoid this happening due to an automatic java update we will follow the recommended steps to turn off http compression.

#As bbuser
#On each node
sed -i 's/bbconfig.appserver.http.compression=on/bbconfig.appserver.http.compression=off/' /usr/local/blackboard/config/bb-config.properties

Verification:

#As bbuser
#On each node
cat /usr/local/blackboard/config/bb-config.properties |grep bbconfig.appserver.http.compression
#result should be bbconfig.appserver.http.compression=off

DEV/PREPROD/LIVE Change 7


Raise bbconfig.cs.database.maxpoolsize from 25 to 50.

#As bbuser
#On each node
sed -i 's/bbconfig.cs.database.maxpoolsize=25/bbconfig.cs.database.maxpoolsize=50/' /usr/local/blackboard/config/bb-config.properties

Verification:

#As bbuser
#On each node
cat /usr/local/blackboard/config/bb-config.properties |grep bbconfig.cs.database.maxpoolsize
#result should be bbconfig.cs.database.maxpoolsize=50

DEV/PREPROD/LIVE Change 8


Set Grade Center caching to zero.
See https://blackboard.secure.force.com/btbb_articleview?id=kA339000000CevB for background.

# as bbuser
# on each node
nano /usr/local/blackboard/config/bb-config.properties
# edit
bbconfig.gradecenter.cache.grade_threshold
# to
bbconfig.gradecenter.cache.grade_threshold=0

Verification

cat /usr/local/blackboard/config/bb-config.properties| grep "bbconfig.gradecenter.cache.grade_threshold=0" |wc -l

Fixes in other files

DEV/PREPROD/LIVE XSS security fixes

This is not needed from 2018 Q2 onwards
Blackboard have recommended adjusting two files in order to resolve XSS issues.
See:

Blackboard files affected:

Dev:/usr/local/blackboard/content/vi/BBLEARN1/plugins/bb-xss-filter/webapp/WEB-INF/classes/blackboard/xss/regex/bb-xss-regex-filter.properties

Preprod/Live: /usr/local/blackboard/content/vi/bb_bb60/plugins/bb-xss-filter/webapp/WEB-INF/classes/blackboard/xss/regex/bb-xss-regex-filter.properties

/usr/local/blackboard/config/internal/bb-file-filter-configuration.properties

Apply fix to bb-xss-regex-filter.properties

All commands are run as bbuser.

For dev environments use /usr/local/blackboard/content/vi/BBLEARN1/plugins/bb-xss-filter/webapp/WEB-INF/classes/blackboard/xss/regex/bb-xss-regex-filter.properties

This is performed on any one node because the file is in a shared folder.

Edit the bb-xss-regex-filter.properties file

nano /usr/local/blackboard/content/vi/bb_bb60/plugins/bb-xss-filter/webapp/WEB-INF/classes/blackboard/xss/regex/bb-xss-regex-filter.properties

Change

disabled_tags.template=(?<=<)\\s*@X@tag@X@(?=[\\s/|>])|(?<=</)\\s*@X@tag@X@(?=\\s*>?)

to

disabled_tags.template=(?<=<)\\s*(?>[^:<>]+:)?\\s*@X@tag@X@(?=[\\s/|>])|(?<=</)\\s*(?>[^:<>]+:)?\\s*@X@tag@X@(?=\\s*>?)

Test change to to bb-xss-regex-filter.properties

Performed by Apps Man. All commands are run as bbuser.

This is performed on on any one node.

Run the command:

cat /usr/local/blackboard/content/vi/bb_bb60/plugins/bb-xss-filter/webapp/WEB-INF/classes/blackboard/xss/regex/bb-xss-regex-filter.properties | grep disabled.tags.template;

Confirm the following is displayed:

# The disabled_tags.template pattern will be applied to each tag in the comma-delimited list disabled_tags.template=(?<=<)\\s*(?>[^:<>]+:)?\\s*@X@tag@X@(?=[\\s/|>])|(?<=</)\\s*(?>[^:<>]+:)?\\s*@X@tag@X@(?=\\s*>?)

Apply fix to bb-file-filter-configuration.properties

All commands are run as bbuser.

This is performed on every node

# On each node
# As bbuser
nano /usr/local/blackboard/config/internal/bb-file-filter-configuration.properties

Change the section:

# Filtered file types
.htm=filter
.html=filter
.css=filter
.js=filter
.txt=filter
.xml=filter
.svg=filter

to

# Filtered file types
.htm=filter
.html=filter
.xhtml=filter
.xhm=filter
.css=filter
.js=filter
.txt=filter
.xml=filter
.svg=filter
.svgz=filter
.xsl=filter

Test change to bb-file-filter-configuration.properties

All commands are run as bbuser.

This is performed on every node

Test 1: File contents

Display file contents

cat /usr/local/blackboard/config/internal/bb-file-filter-configuration.properties;

Confirm the below text appears:

# Filtered file types
.htm=filter
.html=filter
.xhtml=filter
.xhm=filter
.css=filter
.js=filter
.txt=filter
.xml=filter
.svg=filter
.svgz=filter
.xsl=filter
Test 2: Size and modified date

Run the following command on every node.

ls -l /usr/local/blackboard/config/internal/bb-file-filter-configuration.properties;

Confirm:

  1. This file is the same size and date across all nodes

DEV/PREPROD/LIVE Randomness fix

Edit the random source file attribute of the $JAVA_HOME/jre/lib/security/java.security file on each node.

# with staff account, NOT bbuser
# on each node
# confirm it's not bbuser
whoami
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-oracle.x86_64/ ;
adm sed -i 's/securerandom.source=file:\/dev\/random/securerandom.source=file:\/dev\/urandom/' $JAVA_HOME/jre/lib/security/java.security;

DEV/PREPROD/LIVE File ownership

Blackboard files should be owned by bbuser, otherwise upgrades will fail.

#On each node
find /data/bb/blackboard/ -maxdepth 2 -mindepth 2 -type d -not -name content | xargs -I {} chown -R bbuser\: {}

DEV/PREPROD/LIVE Ulimit

The upgrade produced errors due to ulimit being too low. Therefore we should raise the ulimit as follows:

#On each node
#Login as root
- vim /etc/security/limits.conf
- add following lines:
* soft nofile 5000
* hard nofile 65536
#Pay attention that
# does not appear in front of either line - save - logout - login
#as root check: - ulimit -n - ulimit -Hn
[root@~]# ulimit -n 5000 [root@~]
# ulimit -Hn 65536
#switch to bbuser
and check is the same result:
- su bbuser -
ulimit -n -
ulimit -Hn
[bbuser@]$ ulimit -n 5000
[bbuser@]$ ulimit -Hn 65536

Verification

[bbuser@]$ ulimit -n 5000 [bbuser@]$ ulimit -Hn 65536

PREPROD/LIVE Speed up Blackboard start stop

Blackboard files affected:
/usr/local/blackboard/config/tomcat/conf/wrapper.conf
/usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb

  1. In order to reduce time that it takes for Blackboard to services to stop as recommended at https://blackboard.secure.force.com/btbb_articleview?id=kAC7000000000FS, make following change to /usr/local/blackboard/apps/tomcat/conf/wrapper.conf on each node:

#As bbuser

#On each node

sed -i 's/wrapper.shutdown.timeout=300/wrapper.shutdown.timeout=30/' /usr/local/blackboard/config/tomcat/conf/wrapper.conf ;

sed -i 's/wrapper.jvm_exit.timeout=15/wrapper.jvm_exit.timeout=5/' /usr/local/blackboard/config/tomcat/conf/wrapper.conf ;

sed -i 's/wrapper.ping.timeout=60/wrapper.ping.timeout=90/' /usr/local/blackboard/config/tomcat/conf/wrapper.conf ; sed -i 's/wrapper.shutdown.timeout=300/wrapper.shutdown.timeout=30/' /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb ;

sed -i 's/wrapper.jvm_exit.timeout=15/wrapper.jvm_exit.timeout=5/' /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb ;

sed -i 's/wrapper.ping.timeout=60/wrapper.ping.timeout=90/' /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb

Verification

#On each node
cat /usr/local/blackboard/config/tomcat/conf/wrapper.conf |grep wrapper.shutdown.timeout ;
cat /usr/local/blackboard/config/tomcat/conf/wrapper.conf |grep wrapper.jvm_exit.timeout ;
cat /usr/local/blackboard/config/tomcat/conf/wrapper.conf |grep wrapper.ping.timeout
#result should be
wrapper.shutdown.timeout=30
wrapper.jvm_exit.timeout=5
wrapper.ping.timeout=90
#On each node
cat /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb |grep wrapper.shutdown.timeout ;
cat /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb |grep wrapper.jvm_exit.timeout ;
cat /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb |grep wrapper.ping.timeout
#result should be wrapper.shutdown.timeout=30 wrapper.jvm_exit.timeout=5 wrapper.ping.timeout=90

PREPROD/LIVE - Xythos Cross Server Communication Fix

This issue is fixed in 2017 Q4 and should no longer need to be applied.
Blackboard files affected:
/usr/local/blackboard/config/tomcat/conf/server.xml.bb

  1. Apply xythos cross server communication fix on each node.
    1. On each node
    2. Backup file
    cp /usr/local/blackboard/config/tomcat/conf/server.xml.bb /home/bbuser/server.xml.bb.$(date +"%Y-%m-%d")
    1. On each node
    2. Edit /usr/local/blackboard/config/tomcat/conf/server.xml.bb
    nano /usr/local/blackboard/config/tomcat/conf/server.xml.bb
    1. find the stanza (near end of file around lines 203 - 207) defining the xythos admin connector and add secure="true" i.e.
    2. Currently it is:
    <Connector port="@bbconfig.xythosremoteadmin.xythos.servergroup.portnumber@"
    protocol="org.apache.coyote.http11.Http11NioProtocol"
    executor="crossServerExecutor"
    backlog="@@bbconfig.xythosremoteadmin.connector.backlog@@"
    enableLookups="false"
    keepAliveTimeout="@@bbconfig.appserver.http.keepalive.timeout@@"
    maxKeepAliveRequests="@@bbconfig.appserver.http.keepalive.requests@@"
    maxPostSize="@@bbconfig.appserver.max.post.size@@"
    maxParameterCount="@@bbconfig.appserver.max.parameter.count@@"
    soTimeout="@@bbconfig.appserver.http.so.timeout@@"
    URIEncoding="UTF-8"/>
    1. Change to:
    <Connector port="@bbconfig.xythosremoteadmin.xythos.servergroup.portnumber@"
    protocol="org.apache.coyote.http11.Http11NioProtocol"
    executor="crossServerExecutor"
    backlog="@@bbconfig.xythosremoteadmin.connector.backlog@@"
    enableLookups="false"
    keepAliveTimeout="@@bbconfig.appserver.http.keepalive.timeout@@"
    maxKeepAliveRequests="@@bbconfig.appserver.http.keepalive.requests@@"
    maxPostSize="@@bbconfig.appserver.max.post.size@@"
    maxParameterCount="@@bbconfig.appserver.max.parameter.count@@"
    soTimeout="@@bbconfig.appserver.http.so.timeout@@"
    URIEncoding="UTF-8"
    secure="true"/> <----- Part to add
  2. New method using sed
    # as bbuser
    # on each node
    sed -i 's;URIEncoding="UTF-8"/>;URIEncoding="UTF-8" \n secure="true"/>;g' /usr/local/blackboard/config/tomcat/conf/server.xml.bb
  3. Verification step of above on each node.
    #on each node
    cat /usr/local/blackboard/config/tomcat/conf/server.xml.bb |grep secure
    secure="true"
    secure="true"
    secure="true"/>

DEV/PREPROD/LIVE Resolve Math Editor not loading issue

Blackboard files affected:
/usr/local/blackboard/config/internal/http-cacheable-extensions.txt

Issue described at https://blackboard.secure.force.com/btbb_articleview?id=kA370000000H4uu can be resolved by:

  1. On each node update the http-cacheable-extensions.txt to include reference to resolve issues with Equation editor.

#on each node

#as bbuser

cd /usr/local/blackboard/config/internal/ nano http-cacheable-extensions.txt

#If not already present, add the following to the list in their alphabetical place: eot ttf woff

#and save it.

Verification

#As bbuser
#On each node
grep 'eot\|ttf\|woff' /usr/local/blackboard/config/internal/http-cacheable-extensions.txt | wc -l
#Result must equal 3.

PREPROD/LIVE Workaround to issue LRN-107071: High CPU and High Load Caused by Script MicrosoftDocumentParser.sh

Blackboard files affected:
/usr/local/blackboard/config/internal/xythos-indexing-filter.txt
/usr/local/blackboard/config/tomcat/classes/Search.properties
/usr/local/blackboard/config/tomcat/classes/Search.properties.bb

LRN-107071: High CPU and High Load Caused by Script MicrosoftDocumentParser.sh is an issue that should be resolved in 2017 Q2, so this workaround should not be needed in future upgrades.

  1. Stop Blackboard on all nodes, web first, then tasks node
  2. Apply database fix to CMS schema
    UPDATE XY_SERVER_GROUP_PARAMETERS
    SET PARAMETER_VALUE = 'exe,dll,zip,jpg,gif,tif,tiff,dmp,pdf,doc,docx,ppt,pps,pptx,ppsx,ppts,xlsx,xls'
    WHERE PARAMETER_NAME = 'Xythos.Search.ExtensionsNotToIndex';
    UPDATE XY_SERVER_GROUP_PARAMETERS
    SET PARAMETER_VALUE = 'application/pdf,application/x-pdf,application/vnd.ms-excel,
    application/msexcel,
    application/x-msexcel,application/x-ms-excel,
    application/x-excel,
    application/x-dos_ms_excel
    ,application/xls,
    application/x-xls
    ,application/doc,
    application/x-doc,
    application/msword
    ,application/ppt,
    application/x-ppt,
    application/vnd.ms-powerpoint,
    application/pps
    ,application/x-pps
    ,application/vnd.openxmlformats-officedocument.spreadsheetml.sheet,
    application/vnd.openxmlformats-officedocument.wordprocessingml.document
    ,application/vnd.openxmlformats-officedocument.presentationml.presentation,
    application/vnd.openxmlformats-officedocument.presentationml.slideshow'
    WHERE PARAMETER_NAME = 'Xythos.Search.MIMETypesNotToIndex';
  3. Apply workaround to issue LRN-107071: High CPU and High Load Caused by Script MicrosoftDocumentParser.sh by adding lines to /usr/local/blackboard/config/internal/xythos-indexing-filter.txt (note that Blackboard claim this issue is resolved in 9.1 2017 Q2)
    1. On each node
    2. Backup file
    cp /usr/local/blackboard/config/internal/xythos-indexing-filter.txt /home/bbuser/xythos-indexing-filter.txt.$(date +"%Y-%m-%d")
    1. On each node
    2. Edit /usr/local/blackboard/config/internal/xythos-indexing-filter.txt
    nano /usr/local/blackboard/config/internal/xythos-indexing-filter.txt
    1. Set the whole file to contain
    # This file contains mime types that should be filtered from Xythos indexing
    # This file contains mime types that should be filtered from Xythos indexing
    application/doc
    application/msword
    application/msexcel
    application/pps
    application/ppt
    application/vnd.ms-excel
    application/vnd.ms-powerpoint
    application/vnd.openxmlformats-officedocument.presentationml.presentation
    application/vnd.openxmlformats-officedocument.presentationml.slideshow
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
    application/vnd.openxmlformats-officedocument.wordprocessingml.document
    application/x-doc
    application/x-dos_ms_excel
    application/x-excel
    application/xls
    application/x-msexcel
    application/x-ms-excel
    application/x-pps
    application/x-ppt
    application/x-xls

Verification

#as bbuser
cat /usr/local/blackboard/config/internal/xythos-indexing-filter.txt
#output that should be returned:
# This file contains mime types that should be filtered from Xythos indexing
application/doc
application/msword
application/msexcel
application/pps
application/ppt
application/vnd.ms-excel
application/vnd.ms-powerpoint
application/vnd.openxmlformats-officedocument.presentationml.presentation
application/vnd.openxmlformats-officedocument.presentationml.slideshow
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
application/vnd.openxmlformats-officedocument.wordprocessingml.document
application/x-doc application/x-dos_ms_excel
application/x-excel
application/xls
application/x-msexcel
application/x-ms-excel
application/x-pps
application/x-ppt
application/x-xls
  1. Adjust Search.properties files accordingly on each node.
#as bbuser
#on each node
sed -i 's/ExtensionsNotToIndex=exe,dll,zip,jpg,gif,tif,tiff,dmp/ExtensionsNotToIndex=exe,dll,zip,jpg,gif,tif,tiff,dmp,pdf,doc,docx,ppt,pps,pptx,ppsx,ppts,xlsx,xls,xlsm/' /usr/local/blackboard/config/tomcat/classes/Search.properties ;
sed -i 's/ExtensionsNotToIndex=exe,dll,zip,jpg,gif,tif,tiff,dmp/ExtensionsNotToIndex=exe,dll,zip,jpg,gif,tif,tiff,dmp,pdf,doc,docx,ppt,pps,pptx,ppsx,ppts,xlsx,xls,xlsm/' /usr/local/blackboard/config/tomcat/classes/Search.properties.bb

PREPROD/LIVE Adjust logging and task management

Blackboard files affected:
/usr/local/blackboard/config/service-config.properties

 

Our Blackboard environment has four web nodes and a tasks node. We will set the tasks node to run tasks and the web nodes to not run tasks. We will set the logging on the web nodes to show fatal errors only, whilst the tasks node will log warnings as well (default setting).

N.B. To set a node to run tasks it is also necessary to set bbconfig.frontend.fullhostname=true in the bb-config.properties file on that node. This setting is honoured by the installer, but the following additional necessary setting is wiped by the installer.

Edit the /usr/local/blackboard/config/service-config.properties on all nodes EXCEPT tasks node to ensure that tasks node is the only node set to carry out background tasks, thus optimising performance of web facing end user nodes.

#As bbuser on all nodes except tasks node
#Edit the file /usr/local/blackboard/config/service-config.properties
nano /usr/local/blackboard/config/service-config.properties
#and alter the line
#blackboard.service.queuedtaskmanager.param.num-concurrent-tasks=2
#to read
#blackboard.service.queuedtaskmanager.param.num-concurrent-tasks=0
# using sed:
# on web nodes only:
sed -i 's/blackboard.service.queuedtaskmanager.param.num-concurrent-tasks=2/blackboard.service.queuedtaskmanager.param.num-concurrent-tasks=0/' /usr/local/blackboard/config/service-config.properties
#and change logging level to fatal on all nodes but tasks node
blackboard.service.log.param.logdef.services.verbosity=fatal
# using sed:
# on web nodes only:
sed -i 's/blackboard.service.log.param.logdef.services.verbosity=warning/blackboard.service.log.param.logdef.services.verbosity=fatal/' /usr/local/blackboard/config/service-config.properties

Verification

On web nodes
cat /usr/local/blackboard/config/service-config.properties |grep blackboard.service.queuedtaskmanager.param.num-concurrent-tasks ;
cat /usr/local/blackboard/config/service-config.properties |grep blackboard.service.log.param.logdef.services.verbosity
#should return
blackboard.service.queuedtaskmanager.param.num-concurrent-tasks=0
blackboard.service.log.param.logdef.services.verbosity=fatal
On tasks node
cat /usr/local/blackboard/config/service-config.properties |grep blackboard.service.queuedtaskmanager.param.num-concurrent-tasks ;
cat /usr/local/blackboard/config/service-config.properties |grep blackboard.service.log.param.logdef.services.verbosity
#should return
blackboard.service.queuedtaskmanager.param.num-concurrent-tasks=2
blackboard.service.log.param.logdef.services.verbosity=warning

DEV/PREPROD/LIVE Raise Notification Distribution setting

Blackboard files affected:
/usr/local/blackboard/config/internal/nautilus-config.properties

 

Raise the Notification Distribution setting to ensure more timely notification delivery.

On each node, edit /usr/local/blackboard/config/internal/nautilus-config.properties and change nautilus.distribution.notificationsPerDistribution=10000 to 20000

#as bbuser
#on each node
sed -i 's/nautilus.distribution.notificationsPerDistribution=10000/nautilus.distribution.notificationsPerDistribution=20000/' /usr/local/blackboard/config/internal/nautilus-config.properties

Verification

cat /usr/local/blackboard/config/internal/nautilus-config.properties | grep "nautilus.distribution.notificationsPerDistribution=20000" | wc -l
#answer should be 1

DEV/PREPROD/LIVE Improve Tool Settings Changes Time To Take Effect

This resolves an issue where changes to Tool Settings can take a long time do take effect across all nodes, see https://blackboard.secure.force.com/btbb_articleview?id=kA370000000H0jw. Blackboard files affected:
/usr/local/blackboard/config/cache-settings.properties

 

Reduce time for tool changes to take effect.

 

 

On each node, edit /usr/local/blackboard/config/cache-settings.properties and reduce bbconfig.cache.contextualizedNodeAffiliates.timetoidle=345600 to 1800 and reduce bbconfig.cache.default.timetoidle=172800 to 21600

#as bbuser
#on each node
sed -i 's/bbconfig.cache.contextualizedNodeAffiliates.timetoidle=345600/bbconfig.cache.contextualizedNodeAffiliates.timetoidle=1800/' /usr/local/blackboard/config/cache-settings.properties ;
sed -i 's/bbconfig.cache.default.timetoidle=172800/bbconfig.cache.default.timetoidle=21600/' /usr/local/blackboard/config/cache-settings.properties

Verification

cat /usr/local/blackboard/config/cache-settings.properties | grep "bbconfig.cache.contextualizedNodeAffiliates.timetoidle=1800" | wc -l #answer should be 1 cat /usr/local/blackboard/config/cache-settings.properties | grep "bbconfig.cache.default.timetoidle=21600" | wc -l
#answer should be 1

DEV/PREPROD/LIVE Resolve "ActiveMQ Memory Limit Causing SIS Failures" Known Issue

The ActiveMQ message queue can grow past the amount of memory allocated to it. This causes a back up of messages, and will prevent SIS integrations from being processed. This can be resolved by raising a couple of values in the message queue configuration files. See https://blackboard.secure.force.com/btbb_articleview?id=kA339000000CfVn

 

Blackboard files affected:
/usr/local/blackboard/config/message-queue-service-config.xml
/usr/local/blackboard/config/message-queue-service-config.xml.bb

 

On each node, edit /usr/local/blackboard/config/message-queue-service-config.xml and /usr/local/blackboard/config/message-queue-service-config.xml.bb and increase memoryUsage limit from 256 MB to 512 MB and increase storeUsage limit from 1g to 2g

#as bbuser
#on each node
sed -i 's/memoryUsage limit="256 MB"/memoryUsage limit="512 MB"/' /usr/local/blackboard/config/message-queue-service-config.xml ;
sed -i 's/storeUsage limit="1g"/storeUsage limit="2g"/' /usr/local/blackboard/config/message-queue-service-config.xml ;
sed -i 's/memoryUsage limit="256 MB"/memoryUsage limit="512 MB"/' /usr/local/blackboard/config/message-queue-service-config.xml.bb ;
sed -i 's/storeUsage limit="1g"/storeUsage limit="2g"/' /usr/local/blackboard/config/message-queue-service-config.xml.bb

Verification

cat /usr/local/blackboard/config/message-queue-service-config.xml | grep "memoryUsage limit=\"512 MB\"" | wc -l
#answer should be 1
cat /usr/local/blackboard/config/message-queue-service-config.xml | grep "storeUsage limit=\"2g\"" | wc -l
#answer should be 1
cat /usr/local/blackboard/config/message-queue-service-config.xml.bb | grep "memoryUsage limit=\"512 MB\"" | wc -l
#answer should be 1
cat /usr/local/blackboard/config/message-queue-service-config.xml.bb | grep "storeUsage limit=\"2g\"" | wc -l
#answer should be 1

DEV/PREPROD/LIVE Forum is enabled issue

According to https://blackboard.secure.force.com/btbb_articleview?id=kA3390000008aEg usage of the discussion board can caused high DB load. Whilst we await a fix Blackboard have provided a replacement file that resolves the issue.

Blackboard files affected:
/usr/local/blackboard/system/database/vi/discussionboard/functions/forum_is_enabled.sql

Current File location: https://mle.southampton.ac.uk/bb/bb2018-fixes/forum_is_enabled.sql

Process:

  1. MLE: Get the updated "forum_is_enabled.sql" file (see https://blackboard.secure.force.com/btbb_caseview?id=5003900001zkPvwAAE) and place in bbuser's home folder on each node.
    # on each node
    # as bbuser
    cd ;
    wget https://mle.southampton.ac.uk/bb/bb2018-fixes/forum_is_enabled.sql
  2. APPSMAN: After upgrading Blackboard, back up the forum_is_enabled.sql file.
    cp /usr/local/blackboard/system/database/vi/discussionboard/functions/forum_is_enabled.sql /home/bbuser/post_upgrade_files/
  3. APPSMAN: On each node copy the new forum_is_enabled.sql file.
    # on each node
    # as bbuser
    cp /home/bbuser/forum_is_enabled.sql /usr/local/blackboard/system/database/vi/discussionboard/functions/forum_is_enabled.sql

DEV/PREPROD/LIVE Safari upload issue

This is not needed from 2018 Q2 onwards
According to https://blackboard.secure.force.com/btbb_articleview?id=kA3390000008bOi users of Safari 11.1 are unable to upload files to Blackboard. Blackboard have provided a fixed widget.js file that resolves the problem.

Blackboard files affected:
/usr/local/blackboard/docs/javascript/ngui/widget.js

Current File location: https://mle.southampton.ac.uk/bb/bb2018-fixes/widget.js

Process:

  1. MLE: Get the updated "widget.js" file (see https://blackboard.secure.force.com/btbb_articleview?id=kA3390000008bOi) and place in bbuser's home folder on each node.
    # on each node
    # as bbuser
    cd ;
    wget https://mle.southampton.ac.uk/bb/bb2018-fixes/widget.js
  2. APPSMAN: After upgrading Blackboard, back up the forum_is_enabled.sql file.
    cp /usr/local/blackboard/docs/javascript/ngui/widget.js /home/bbuser/post_upgrade_files/
  3. APPSMAN: On each node copy the new forum_is_enabled.sql file.
    # on each node
    # as bbuser
    cp /home/bbuser/widget.js /usr/local/blackboard/docs/javascript/ngui/widget.js

List of affected files

Files where changes are made in DEV environments

/usr/local/blackboard/config/bb-config.properties
/usr/local/blackboard/config/bb-config.properties.template
/usr/local/blackboard/config/internal/http-cacheable-extensions.txt
/usr/local/blackboard/config/internal/nautilus-config.properties
/usr/local/blackboard/config/cache-settings.properties
/usr/local/blackboard/config/message-queue-service-config.xml
/usr/local/blackboard/config/message-queue-service-config.xml.bb
/usr/local/blackboard/content/vi/BBLEARN1/plugins/bb-xss-filter/webapp/WEB-INF/classes/blackboard/xss/regex/bb-xss-regex-filter.properties
/usr/local/blackboard/config/internal/bb-file-filter-configuration.properties
/usr/local/blackboard/system/database/vi/discussionboard/functions/forum_is_enabled.sql
/usr/local/blackboard/docs/javascript/ngui/widget.js

Files where changes are made in PREPROD/LIVE environments

/usr/local/blackboard/config/bb-config.properties
/usr/local/blackboard/config/bb-config.properties.template
/usr/local/blackboard/config/tomcat/conf/wrapper.conf
/usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb
/usr/local/blackboard/config/tomcat/conf/server.xml.bb
/usr/local/blackboard/config/internal/http-cacheable-extensions.txt
/usr/local/blackboard/config/internal/xythos-indexing-filter.txt
/usr/local/blackboard/config/tomcat/classes/Search.properties
/usr/local/blackboard/config/tomcat/classes/Search.properties.bb
/usr/local/blackboard/config/service-config.properties
/usr/local/blackboard/config/internal/nautilus-config.properties
/usr/local/blackboard/config/cache-settings.properties
/usr/local/blackboard/config/message-queue-service-config.xml
/usr/local/blackboard/config/message-queue-service-config.xml.bb
/usr/local/blackboard/content/vi/bb_bb60/plugins/bb-xss-filter/webapp/WEB-INF/classes/blackboard/xss/regex/bb-xss-regex-filter.properties
/usr/local/blackboard/config/internal/bb-file-filter-configuration.properties
/usr/local/blackboard/system/database/vi/discussionboard/functions/forum_is_enabled.sql
/usr/local/blackboard/docs/javascript/ngui/widget.js

Here is a slightly redacted version of our live upgrade plan.  I'm sharing this in case it is useful for others...  I will also post a summary of our current configuration changes and tweaks in a separate blog post.

Live Upgrade to from 9.1 Q4 2016 CU6 to 9.1 Q4 2017 CUx

Overview

We will upgrade Blackboard Live to 9.1 Q4 2017 CU3.

Server Address

Blackboard preproduction is made up of four applications nodes, a tasks node, an Oracle 12C database, and a NFS server.

hostname_removed – tasks node

hostname_removed – web node

hostname_removed – web node

hostname_removed – web node

hostname_removed – web node

hostname_removed – Oracle database

hostname_removed – NFS server

Note

Where "on each node" is stated, this refers to each node on which the Blackboard application runs, i.e. the tasks node and the web nodes.

Summary of stages

  1. Stage 1 covers work that can be carried out before the upgrade work starts. This work is under CHG_______.
  2. Stage 2 covers making Blackboard inaccessible to users and taking our baseline snapshot. This stage onwards are under CHG_______.
  3. Stage 3 covers the Oracle update.
  4. Stage 4 covers our upgrade of Blackboard to the new version.
  5. Stage 5 is where we check what changes have been made by the installer.
  6. Stage 6 is where we re-apply configuration changes that have been reset by the installer.
  7. Stage 7 is where configuration changes are applied to the application using pushconfigupdates.
  8. Stage 8 is where we check the log files for any possible issues before the finish for the day.
  9. Stage 9 is where we start the day by checking the logs after the application has been running overnight. We will also check that data integration has run ok.
  10. Stage 10 encompasses GUI based configuration changes if required, and acceptance testing.
  11. Stage 11 is a restart test.
  12. Stage 12 is where we make Blackboard available to users
  13. Stage 13 involves tidying up after successful testing.

Stage 1 - Prologue

Stage 1 covers work that can be carried out in preprod and dev environments before the upgrade work starts. This is under CHG_______

  1. Bb_Team and DBA: confirm and agree password to be used.
  2. DBA: consider whether Oracle backups should be stopped from running on the night of Saturday 23 June.
  3. DBA: Prepare Oracle patching files.

Prepare the installer

  1. Bb_Team: Prepare installation files:
    # as bbuser
    # on each node
    unzip /home/bbuser/learn-installer-[VERSION_NUMBER].zip -d /usr/local/bbinstaller/
    1. Download the installer
      1. Go to https://support.blackboard.com and log in.
      2. Go to page https://behind.blackboard.com/downloads/details.aspx?d=1783
      3. Click on Learn 9.1 Q4 2017 CU3 Installer.
      4. Click on I agree to the terms and conditions above.
      5. Click Get File.
      6. Download will start. Download to /home/bbuser/ on each node.
    2. Unzip the installer to /usr/local/bbinstaller/ on each node.
  2. Bb_Team: Set files under /usr/local/bbinstaller/ to be owned by bbuser:bbuser on each node.
    # as bbuser
    # on each node
    cd /usr/local ;
    chown -R bbuser\: bbinstaller

Prepare the licence file

  1. Bb_Team: Copy /usr/local/blackboard/config/licence /blackboard-licence .xml to /home/bbuser/ on each node.
    # as bbuser
    # on each node
    cp /usr/local/blackboard/config/licence /blackboard-licence .xml /home/bbuser/

Prepare the installer properties files

Prepare the installer properties for tasks node (hostname_removed)

  1. Bb_Team: create file at /usr/local/bbinstaller/installer.properties with following draft contents on tasks node:
    # Installer Properties for Tasks node
    antargs.default.vi.db.password=[ ]
    antargs.default.users.administrator.password=xxxx
    antargs.default.users.integration.password=xxxx
    antargs.default.users.rootadmin.password=xxxx
    bbconfig.file.licence =/home/bbuser/blackboard-licence .xml
    bbconfig.java.home=/usr/lib/jvm/java-1.8.0-oracle.x86_64
    bbconfig.jvm.options.extra.tomcat=-XX:NewSize=2048m -XX:MaxNewSize=2048m -XX:+UseTLAB -XX:SurvivorRatio=4 -XX:+UseCompressedOops -Xss512k -XX:+PrintVMOptions -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCTaskTimeStamps -XX:+PrintCommandLineFlags -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+AlwaysPreTouch -verbose:gc -XX:+DoEscapeAnalysis -Xverify:none -XX:+UseMontgomeryMultiplyIntrinsic -XX:+UseMontgomerySquareIntrinsic -XX:MetaspaceSize=768m -XX:MaxMetaspaceSize=1536m -Djava.util.prefs.systemRoot=/home/bbuser/.java/
    bbconfig.jvm.options.gc=-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=45 -XX:+UseStringDeduplication
    bbconfig.email.use.dmarc.from.override=false
    bbconfig.max.stacksize.tomcat=512k
    bbconfig.appserver.http.compression=off
    bbconfig.jvm.options.codecache.reserved=256m
    bbconfig.jvm.options.codecache.initial=256m
    bbconfig.cs.database.maxpoolsize=50
    bbconfig.peer.discovery.timeout.inactive=20
    bbconfig.peer.discovery.timeout.dead=300
    bbconfig.server.backend.processor=true
    bbconfig.gradecenter.cache.grade_threshold=0
  2. Bb_Team: note blank lines can be deleted with sed /^$/d

Prepare the installer properties for web nodes (hostname_removed - hostname_removed)

  1. Bb_Team: create file at /usr/local/bbinstaller/installer.properties with following draft contents on web node:
    # Installer Properties for Web nodes
    antargs.default.vi.db.password=[ ]
    antargs.default.users.administrator.password=xxxx
    antargs.default.users.integration.password=xxxx
    antargs.default.users.rootadmin.password=xxxx
    bbconfig.file.licence =/home/bbuser/blackboard-licence .xml
    bbconfig.java.home=/usr/lib/jvm/java-1.8.0-oracle.x86_64
    bbconfig.jvm.options.extra.tomcat=-XX:NewSize=2048m -XX:MaxNewSize=2048m -XX:+UseTLAB -XX:SurvivorRatio=4 -XX:+UseCompressedOops -Xss512k -XX:+PrintVMOptions -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCTaskTimeStamps -XX:+PrintCommandLineFlags -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+AlwaysPreTouch -verbose:gc -XX:+DoEscapeAnalysis -Xverify:none -XX:+UseMontgomeryMultiplyIntrinsic -XX:+UseMontgomerySquareIntrinsic -XX:MetaspaceSize=768m -XX:MaxMetaspaceSize=1536m -Djava.util.prefs.systemRoot=/home/bbuser/.java/
    bbconfig.jvm.options.gc=-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=45 -XX:+UseStringDeduplication
    bbconfig.email.use.dmarc.from.override=false
    bbconfig.max.stacksize.tomcat=512k
    bbconfig.appserver.http.compression=off
    bbconfig.jvm.options.codecache.reserved=256m
    bbconfig.jvm.options.codecache.initial=256m
    bbconfig.cs.database.maxpoolsize=50
    bbconfig.peer.discovery.timeout.inactive=20
    bbconfig.peer.discovery.timeout.dead=300
    bbconfig.server.backend.processor=false
    bbconfig.gradecenter.cache.grade_threshold=0
  2. Bb_Team: note blank lines can be deleted with sed /^$/d

Back up files

Back up config files

  1. Bb_Team: Backup the various configuration files on each node.
    # as bbuser
    # on each node
    mkdir /home/bbuser/original_files ;
    cp /usr/local/blackboard/config/bb-config.properties /home/bbuser/original_files/ ;
    cp /usr/local/blackboard/config/bb-config.properties.template /home/bbuser/original_files/ ;
    cp /usr/local/blackboard/config/tomcat/conf/wrapper.conf /home/bbuser/original_files/ ;
    cp /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb /home/bbuser/original_files/ ;
    cp /usr/lib/jvm/java-1.8.0-oracle.x86_64/jre/lib/security/java.security /home/bbuser/original_files/ ;
    cp /usr/local/blackboard/config/tomcat/conf/server.xml.bb /home/bbuser/original_files/ ;
    cp /usr/local/blackboard/config/internal/http-cacheable-extensions.txt /home/bbuser/original_files/ ;
    cp /usr/local/blackboard/config/internal/xythos-indexing-filter.txt /home/bbuser/original_files/ ;
    cp /usr/local/blackboard/config/tomcat/classes/Search.properties.bb /home/bbuser/original_files/;
    cp /usr/local/blackboard/config/tomcat/classes/Search.properties /home/bbuser/original_files/;
    cp /usr/local/blackboard/config/service-config.properties /home/bbuser/original_files/ ;
    cp /usr/local/blackboard/config/internal/nautilus-config.properties /home/bbuser/original_files/ ;
    cp /usr/local/blackboard/config/cache-settings.properties /home/bbuser/original_files/ ;
    cp /usr/local/blackboard/config/message-queue-service-config.xml /home/bbuser/original_files/ ;
    cp /usr/local/blackboard/config/message-queue-service-config.xml.bb /home/bbuser/original_files/ ;
    cp /usr/local/blackboard/content/vi/bb_bb60/plugins/bb-xss-filter/webapp/WEB-INF/classes/blackboard/xss/regex/bb-xss-regex-filter.properties /home/bbuser/original_files/;
    cp /usr/local/blackboard/config/internal/bb-file-filter-configuration.properties /home/bbuser/original_files/
  2. Bb_Team: Verify that the correct number of files have been copied.
    # as bbuser
    # on each node
    ls -1 /home/bbuser/original_files/ |wc -l
    #result should equal 17

Back up branding files

  1. Bb_Team: Backup files under /usr/local/blackboard/docs/login/* to /home/bbuser/login.zip on each node.
    # as bbuser
    # on each node
    cd /home/bbuser ;
    zip -r login.zip /usr/local/blackboard/docs/login/* ;
    # set permissions
    chown bbuser:bbuser login.zip

Back up data integration files

  1. Bb_Team: Back up the data_integration directory on the tasks node only (hostname_removed).
    # on tasks node (hostname_removed)
    cd /home/bbuser ;
    zip -r data_integration.zip /usr/local/blackboard/data_integration/* ;
    # Set permissions
    chown bbuser:bbuser data_integration.zip
  2. Bb_Team: Back up the snapshot* files in /blackboard/apps/snapshot/data/ directory on the tasks node only.
    # on tasks node (hostname_removed)
    cd /home/bbuser ;
    zip -r snapshotfiles.zip /usr/local/blackboard/apps/snapshot/data/snapshot.* ;
    # set permissions
    chown bbuser:bbuser snapshotfiles.zip

Back up photo files

  1. Bb_Team: Back up the Photos directory /usr/local/blackboard/content/PHOTOS/ on tasks node.
    # on tasks node (hostname_removed)
    cd /home/bbuser
    zip -r PHOTOS.zip /usr/local/blackboard/content/PHOTOS/*
    # set permissions
    chown bbuser:bbuser PHOTOS.zip

Create Verification Scripts

  1. Bb_Team: Create first verification script "bb-config-tests-1" to be used for bb.config.properties on each node.
    # as bbuser
    # on each node
    nano /home/bbuser/bb-config-tests-1.sh
    #paste following script from hostname_removed/wiki/Blackboard/Live_Verification_Script_1
  2. Bb_Team: Make the script executable on each node.
    # as bbuser
    # on each node
    chmod +x /home/bbuser/bb-config-tests-1.sh
  3. Bb_Team: Create second verification script "bb-config-tests-2" to be used for non bb.config.properties checks on each node.
    # as bbuser
    # on each node
    nano /home/bbuser/bb-config-tests-2.sh
    # paste following script from hostname_removed/wiki/Blackboard/Live_Verification_Script_2
  4. Bb_Team: Make the script executable on each node.
    # as bbuser
    # on each node
    chmod +x /home/bbuser/bb-config-tests-2.sh

Prepare Blackboard for upgrade

  1. Bb_Team: Get the updated "forum_is_enabled.sql" file (see https://blackboard.secure.force.com/btbb_caseview?id=5003900001zkPvwAAE) and place in bbuser's home folder on each node.
    # on each node
    # as bbuser
    cd ;
    wget hostname_removed.ac.uk/bb/bb2018-fixes/forum_is_enabled.sql
  2. Bb_Team: Get the updated "widget.js" file (see https://blackboard.secure.force.com/btbb_articleview?id=kA3390000008bOi) and place in bbuser's home folder on each node.
    # on each node
    # as bbuser
    cd ;
    wget hostname_removed.ac.uk/bb/bb2018-fixes/widget.js

Stage 2 - Day 1 - Live Upgrade begins

In this stage we will disable Blackboard access to users and take our baseline snapshot back up.
CHG_______
Expected start time: 09:00 23 June 2018
N.B. Highfield team to be in place and logged in ready to start by 08:55

  1. Bb_Team: Note start time on whiteboard. Note in RFC that stage has started.
  2. Bb_Team: Tweet that upgrade has started.
  3. Bb_Team: Update upgrade status page.
  4. SYS_TEAM: Remove Blackboard from NLB Pool, thus preventing user access.
    Locate the node objects for the front end servers and 'force offline'
  5. APPS_TEAM: Stop Blackboard on each node, web nodes (hostname_removed – hostname_removed) first, then tasks node (hostname_removed).
    # on each node, web first, then tasks
    # web nodes may be stopped concurrently
    service blackboard stop
  6. SYS_TEAM: VMWare Snapshot #1.
  7. APPS_TEAM: Stop Blackboard on each node, web nodes (hostname_removed – hostname_removed) first, then tasks node (hostname_removed).
    # on each node, web first, then tasks
    # web nodes may be stopped concurrently
    service blackboard stop
  8. Bb_Team: Note stage completion time on whiteboard.

Stage 3 - Day 1 - Oracle update

In this stage we apply the Oracle April Update and remove a superfluous view.

Expected start time: 09:35 23 June 2018


On Dev 1, this stage took 20 minutes.
On Dev 2, this stage took 20 minutes.
On Preprod, this stage took 30 minutes.
On Dev 1 in May, this stage took 30 minutes.
On Dev 2 in May, this stage took 35 minutes.
On Preprod in May, this stage took 45 minutes.

  1. Bb_Team: Note start time on whiteboard. Note in RFC that stage has started.
  2. Bb_Team: Tweet that stage has started.
  3. Bb_Team: Update upgrade status page.
  4. ALL: Check Blackboard is stopped.
  5. DBA: Apply Oracle patch.
    See hostname_removed/wiki/Applying_Patch_PSUAPR2018
  6. DBA: Remove the superfluous view by doing the following.
    backup Table:
    Create Table BB_BB60_STATS.ETL_MATERIALIZED_VIEW_20180226
    As
    Select * from BB_BB60_STATS.ETL_MATERIALIZED_VIEW;
    Execute delete SQL:
    delete from BB_BB60_STATS.ETL_MATERIALIZED_VIEW
    where view_name in ('ods_sog_act_related_ob_vw','ods_ipo_related_obj_vw','ods_instr_measure_ipo_vw');
    commit;
  7. SYS_TEAM: VMWare Snapshot #2.
  8. Bb_Team: Sanity Check Blackboard.
  9. Bb_Team: For preprod and live, reset default language pack to UK English.
  10. APPS_TEAM: Stop Blackboard on each node, web nodes (hostname_removed – hostname_removed) first, then tasks node (hostname_removed).
    # on each node, web first, then tasks
    # web nodes may be stopped concurrently
    service blackboard stop
  11. Bb_Team: Note finish time on board.

Stage 4 - Day 1 - The upgrade

Stage 4 covers our upgrade of Blackboard to the new version.

Expected start time: 10:30 23 June 2018


On Dev 1, this stage took 25 minutes, with the upgrade itself taking 9 minutes 46 seconds.
On Dev 2, this stage took 30 minutes, with the upgrade itself taking 19 minutes 27 seconds.
On Preprod, this stage took 1 hour and 5 minutes, with the first full upgrade itself taking 19 minutes 15 seconds, and subsequent app-server only upgrades taking between 4 and 5 minutes each.
On Dev 1 in May, this stage took 37 minutes, with the upgrade itself taking 20 minutes 4 seconds.
On Dev 2 in May, this stage took 40 minutes, with the upgrade itself taking 21 minutes 4 seconds.
On Preprod in May, this stage took 1 hour and 30 minutes, with the first full upgrade itself taking 20 minutes 23 seconds, and subsequent app-server only upgrades taking between 4 and 5 minutes each.

  1. Bb_Team: Note start time on whiteboard. Note in RFC that stage has started.
  2. Bb_Team: Tweet that stage has started.
  3. Bb_Team: Update upgrade status page.
  4. ALL: Verify that Blackboard is not running.
  5. DBA: Reset password for "sys" and "system"

Java Security Randomness Fix

  1. APPS_TEAM: Edit the random source file attribute of the $JAVA_HOME/jre/lib/security/java.security file on each node.
    # with staff account, e.g. xxx, xxx. NOT bbuser
    # on each node
    # confirm it's not bbuser
    whoami
    export JAVA_HOME=/usr/lib/jvm/java-1.8.0-oracle.x86_64/ ;
    adm sed -i 's/securerandom.source=file:\/dev\/random/securerandom.source=file:\/dev\/urandom/' $JAVA_HOME/jre/lib/security/java.security;

Log Spam fix

  1. NOTE: NEXT STEPS SHOULD BE COMPLETED IN SAME SHELL USING bbuser
  2. APPS_TEAM: change user to bbuser.
  3. APPS_TEAM: Remove the data_integration_log_b directory, to prevent log spam. On each node.
    adm su bbuser
    >> Dev 1+2
    cd /data/bb/blackboard/content/vi/BBLEARN1/lucene ;
    rm -rf data_integration_log_b/
    >> Preprod and Live
    cd /data/bb/blackboard/content/vi/bb_bb60/lucene/ ;
    rm -rf data_integration_log_b/

First full upgrade – tasks node hostname_removed

  1. NOTE: NEXT STEPS SHOULD BE COMPLETED IN SAME SHELL USING bbuser
  2. APPS_TEAM: using shell on tasks node hostname_removed change user to bbuser
    adm su - bbuser
  3. APPS_TEAM: Set JAVA_HOME to Java 8 JDK location; confirm JAVA_HOME set correctly; and change to the installer directory.
    export JAVA_HOME=/usr/lib/jvm/java-1.8.0-oracle.x86_64/ ;
    echo $JAVA_HOME ;
    cd /usr/local/bbinstaller
  4. APPS_TEAM: Run upgrade [as bbuser]
    ./installer.sh -c /usr/local/bbinstaller/installer.properties | tee -a bb-installer-log.txt 2>&1
  5. ALL: During upgrade monitor two logs – could do this in two putty windows on big screen.
    tail -f /usr/local/bbinstaller/bb-installer-log.txt
    tail -f /usr/local/blackboard/logs/update-tools/update-tool-log.txt

Four "app server only" upgrades – web node hostname_removed to hostname_removed

  1. NOTE: NEXT STEPS SHOULD BE COMPLETED IN SAME SHELL USING bbuser
  2. APPS_TEAM: on web nodes, starting with hostname_removed, doing one node at a time, change user to bbuser
    adm su - bbuser
  3. APPS_TEAM: Set JAVA_HOME to Java 8 JDK location; confirm JAVA_HOME set correctly; and change to the installer directory.
    export JAVA_HOME=/usr/lib/jvm/java-1.8.0-oracle.x86_64/ ;
    echo $JAVA_HOME ;
    cd /usr/local/bbinstaller
  4. APPS_TEAM: Run app-server only upgrade [as bbuser]
    ./installer.sh -c /usr/local/bbinstaller/installer.properties -a| tee -a bb-installer-log.txt 2>&1
  5. ALL: During upgrade monitor two logs – could do this in two putty windows on big screen.
    tail -f /usr/local/bbinstaller/bb-installer-log.txt
    tail -f /usr/local/blackboard/logs/update-tools/update-tool-log.txt
  6. APPS_TEAM, repeat above until all four web nodes are upgraded.

Post-upgrade activities

  1. SYS_TEAM: If upgrades are successful, take VMWare Snapshot #3.
  2. Bb_Team: Check you can login.
  3. APPS_TEAM: Stop Blackboard on each node, web nodes (hostname_removed – hostname_removed) first, then tasks node (hostname_removed).
    # on each node, web first, then tasks
    # web nodes may be stopped concurrently
    service blackboard stop
  4. Bb_Team: Note finish time on board.

Stage 5 - Day 1 - Verifying whether previous configuration changes have been removed

Having successfully upgraded Blackboard we will verify that previous configuration changes have been retained.

Expected start time: 12:35 23 June 2018

On Dev 1, this stage took 17 minutes.
On Dev 2, this stage took 5 minutes.
On Preprod, this stage took 8 minutes.
On Dev 1 in May, this stage took 4 minutes.
On Dev 2 in May, this stage took 5 minutes.
On Preprod in May, this stage took 10 minutes.

  1. Bb_Team: Note start time on whiteboard. Note in RFC that stage has started.
  2. Bb_Team: Tweet that stage has started.
  3. Bb_Team: Update upgrade status page.

Back up key files to monitor if there have been any changes

  1. APPS_TEAM: Backup the various configuration files on each node to record any changes following the upgrade.
    # as bbuser
    # on each node
    mkdir /home/bbuser/post_upgrade_files ;
    cp /usr/local/blackboard/config/internal/http-cacheable-extensions.txt /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/config/internal/xythos-indexing-filter.txt /home/bbuser/post_upgrade_files/ ;
    cp /usr/lib/jvm/java-1.8.0-oracle.x86_64/jre/lib/security/java.security /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/config/tomcat/conf/server.xml.bb /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/config/internal/nautilus-config.properties /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/config/service-config.properties /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/config/tomcat/conf/wrapper.conf /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/config/bb-config.properties /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/config/bb-config.properties.template /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/config/tomcat/classes/Search.properties.bb /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/config/tomcat/classes/Search.properties /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/config/cache-settings.properties /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/config/message-queue-service-config.xml /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/config/message-queue-service-config.xml.bb /home/bbuser/post_upgrade_files/
    cp /usr/local/blackboard/system/database/vi/discussionboard/functions/forum_is_enabled.sql /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/content/vi/bb_bb60/plugins/bb-xss-filter/webapp/WEB-INF/classes/blackboard/xss/regex/bb-xss-regex-filter.properties /home/bbuser/post_upgrade_files/;
    cp /usr/local/blackboard/config/internal/bb-file-filter-configuration.properties /home/bbuser/post_upgrade_files/ ;
    cp /usr/local/blackboard/docs/javascript/ngui/widget.js /home/bbuser/post_upgrade_files/ ;
  2. APPS_TEAM: On each node verify that the correct number of files have been copied.
    # as bbuser
    # on each node
    ls -1 /home/bbuser/post_upgrade_files/ |wc -l
    # result must equal 19

Verification 1 - bb-config.properties

  1. ALL: Verify that the following settings held in bb-config.properties files have been preserved on each node.
    # on each node
    # as bbuser
    ~/bb-config-tests-1.sh
  2. ALL: List which tests failed.

Verification 2 - Other config changes

  1. Bb_Team: Verify that the settings held in other files have been preserved on each node.
    # on each node
    # as bbuser
    ~/bb-config-tests-2.sh
  2. ALL: List which tests failed.
  3. Bb_Team: Note finish time on board.

Stage 6 - Day 1 - Configuration changes

Here we will restore files that get deleted by the installer and re-apply any configuration changes that failed our tests.

Expected start time: 13:50 23 June 2018


On Dev 1, this stage took 10 minutes.
On Dev 2, this stage took 7 minutes.
On Preprod, this stage took 23 minutes.
On Dev 1 in May, this stage took 10 minutes.
On Dev 2 in May, this stage took 15 minutes.
On Preprod in May, this stage took 35 minutes.

  1. Bb_Team: Note start time on whiteboard. Note in RFC that stage has started.
  2. Bb_Team: Tweet that stage has started.
  3. Bb_Team: Update upgrade status page.

Restoring files

The installer will often delete files that are not part of a standard Blackboard installation, we will restore those files as required.

Restore Branding

  1. APPS_TEAM: On each node restore branding from /home/bbuser/login.zip to /usr/local/blackboard/docs/login/.
    # on each node
    # as bbuser
    cd /home/bbuser/ ;
    mkdir login-extracted ;
    unzip login.zip -d login-extracted ;
    cd login-extracted/usr/local/blackboard/docs/login ;
    cp -n * /usr/local/blackboard/docs/login
    # ignore message "cp: omitting directory `WEB-INF'"

Restore data integration files

  1. APPS_TEAM: on tasks node, restore data_integration directory.
    # as bbuser
    # on tasks node hostname_removed
    # verify whether
    # /usr/local/blackboard/data_integration/ folder has been wiped.
    # if not ask Bb_Team to double check contents
    # if Bb_Team agree it is ok, then skip the following.
    cd /home/bbuser/ ;
    mkdir data_integration-extracted ;
    unzip data_integration.zip -d data_integration-extracted ;
    cd data_integration-extracted/usr/local/blackboard/data_integration/ ;
    cp -n /home/bbuser/data_integration-extracted/* /usr/local/blackboard/data_integration

Restore photos

  1. APPS_TEAM: on tasks node, restore PHOTOS directory.
    # as bbuser
    # on tasks node hostname_removed
    # verify whether
    # /usr/local/blackboard/content/PHOTOS/ folder has been wiped.
    # if not ask Bb_Team to double check contents
    # if Bb_Team agree it is ok, then skip the following.
    cd /home/bbuser/
    mkdir PHOTOS-extracted
    unzip PHOTOS.zip -d PHOTOS-extracted
    cd /usr/local/blackboard/content/PHOTOS/
    # if PHOTOS directory has been removed, create a new one
    mkdir /usr/local/blackboard/content/PHOTOS/
    cp -n /home/bbuser/PHOTOS-extracted/* /usr/local/blackboard/content/PHOTOS/

Copy the new forum_is_enabled.sql file

  1. APPS_TEAM: On each node copy the new forum_is_enabled.sql file (see https://blackboard.secure.force.com/btbb_caseview?id=5003900001zkPvwAAE for background). Note that we took a back up of the file earlier and placed it in the /home/bbuser/post_upgrade_files/ directory.
    # on each node
    # as bbuser
    cp /home/bbuser/forum_is_enabled.sql /usr/local/blackboard/system/database/vi/discussionboard/functions/forum_is_enabled.sql

Copy the new widget.js file

  1. APPS_TEAM: On each node copy the new widget.js file (see https://blackboard.secure.force.com/btbb_caseview?id=50039000023VpUqAAK for background). Note that we took a back up of the file earlier and placed it in the /home/bbuser/post_upgrade_files/ directory.
    # on each node
    # as bbuser
    cp /home/bbuser/widget.js /usr/local/blackboard/docs/javascript/ngui/widget.js

Changes in bb-config.properties

The bb-config.properties file contains a number of key settings we use that can be overwritten in the installation/upgrade process. A list of all changes is held at hostname_removed.

Apply Scorm fix to bb-config-properties.template

  1. APPS_TEAM: If earlier bb-config.properties Test 1 failed, apply the SCORM fix to bb-config-properties.template on each node.
    # as bbuser
    # on each node
    sed -i "s/bbconfig.headers.CSP.frame-ancestors=@bbconfig.headers.CSP.frame-ancestors@/bbconfig.headers.CSP.frame-ancestors=self\' https:\/\/\*.soton.ac.uk \'soton.ac.uk/g" /usr/local/blackboard/config/bb-config.properties.template

Apply Scorm fix to bb-config-properties

 

Other changes to bb-config.properties

Any other required changes to bb-config.properties to be agreed and inserted here if required.

Verification script

  1. ALL: Verify that configuration changes to bb-config.properties files have made successfully on each node.
    # on each node
    # as bbuser
    ~/bb-config-tests-1.sh
  2. ALL: Check that all tests have passed.

Changes in other files

We have applied a number of changes and optimisations to other configuration files. Where required we re-implement those changes here. A list of all changes is held at hostname_removed.

Apply Math Editor fix

  1. APPS_TEAM: Assuming that other tests Test E failed, and provided no changes identified in http-cacheable-extensions.txt file, copy it back from the original files folder.
    # as bbuser
    # on each node
    cp /home/bbuser/original_files/http-cacheable-extensions.txt /usr/local/blackboard/config/internal/http-cacheable-extensions.txt

Apply Notification Distribution Fix

  1. APPS_TEAM: Assuming that other tests Test H failed, change notification setting from default 10000 to 20000. N.B. File has already been backed up in earlier step.
    # as bbuser
    # on each node
    sed -i 's/nautilus.distribution.notificationsPerDistribution=10000/nautilus.distribution.notificationsPerDistribution=20000/' /usr/local/blackboard/config/internal/nautilus-config.properties

Wrapper.conf and wrapper.conf.bb fixes

  1. APPS_TEAM: Assuming that other tests Tests A, B, C, D, X7, and X8 failed, restore change to wrapper.conf and wrapper.conf.bb on each node.
    # as bbuser
    # on each node
    sed -i 's/wrapper.shutdown.timeout=300/wrapper.shutdown.timeout=30/' /usr/local/blackboard/config/tomcat/conf/wrapper.conf ;
    sed -i 's/wrapper.jvm_exit.timeout=15/wrapper.jvm_exit.timeout=5/' /usr/local/blackboard/config/tomcat/conf/wrapper.conf ;
    sed -i 's/wrapper.ping.timeout=60/wrapper.ping.timeout=90/' /usr/local/blackboard/config/tomcat/conf/wrapper.conf ;
    sed -i 's/wrapper.shutdown.timeout=300/wrapper.shutdown.timeout=30/' /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb ;
    sed -i 's/wrapper.jvm_exit.timeout=15/wrapper.jvm_exit.timeout=5/' /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb ;
    sed -i 's/wrapper.ping.timeout=60/wrapper.ping.timeout=90/' /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb

Resolve LRN-107071: High CPU and High Load Caused by Script MicrosoftDocumentParser.sh

APPS_TEAM: Assuming that other tests Tests J and K failed, apply fix to Search.properties files on each node.

  1. # as bbuser
    # on each node
    sed -i 's/ExtensionsNotToIndex=exe,dll,zip,jpg,gif,tif,tiff,dmp/ExtensionsNotToIndex=exe,dll,zip,jpg,gif,tif,tiff,dmp,pdf,doc,docx,ppt,pps,pptx,ppsx,ppts,xlsx,xls,xlsm/' /usr/local/blackboard/config/tomcat/classes/Search.properties ;
    sed -i 's/ExtensionsNotToIndex=exe,dll,zip,jpg,gif,tif,tiff,dmp/ExtensionsNotToIndex=exe,dll,zip,jpg,gif,tif,tiff,dmp,pdf,doc,docx,ppt,pps,pptx,ppsx,ppts,xlsx,xls,xlsm/' /usr/local/blackboard/config/tomcat/classes/Search.properties.bb
  2. APPS_TEAM: Restore the /usr/local/blackboard/config/internal/xythos-indexing-filter.txt file that we backed up earlier.
    # as bbuser
    # on each node
    cp /home/bbuser/original_files/xythos-indexing-filter.txt /usr/local/blackboard/config/internal/xythos-indexing-filter.txt

Adjust logging and task management

  1. APPS_TEAM: Assuming that other tests Test F failed, edit the /usr/local/blackboard/config/service-config.properties on web nodes only to ensure that the tasks node is the only node set to carry out background tasks, thus optimising performance of web facing end user nodes.
    # as bbuser
    # on web nodes hostname_removed - hostname_removed
    sed -i 's/blackboard.service.queuedtaskmanager.param.num-concurrent-tasks=2/blackboard.service.queuedtaskmanager.param.num-concurrent-tasks=0/' /usr/local/blackboard/config/service-config.properties
  2. APPS_TEAM: Assuming that other tests Test G failed, change logging level to fatal on web nodes only
    # as bbuser
    # on web nodes hostname_removed - hostname_removed
    sed -i 's/blackboard.service.log.param.logdef.services.verbosity=warning/blackboard.service.log.param.logdef.services.verbosity=fatal/' /usr/local/blackboard/config/service-config.properties

Apply Config Cache fix

  1. APPS_TEAM: Assuming that other tests Test X1 and X2 failed, on each node, edit /usr/local/blackboard/config/cache-settings.properties and reduce bbconfig.cache.contextualizedNodeAffiliates.timetoidle=345600 to 1800, and reduce bbconfig.cache.default.timetoidle=172800 to 21600. All nodes.
    # as bbuser
    # on each node
    sed -i 's/bbconfig.cache.contextualizedNodeAffiliates.timetoidle=345600/bbconfig.cache.contextualizedNodeAffiliates.timetoidle=1800/' /usr/local/blackboard/config/cache-settings.properties ;
    sed -i 's/bbconfig.cache.default.timetoidle=172800/bbconfig.cache.default.timetoidle=21600/' /usr/local/blackboard/config/cache-settings.properties

Apply Message Queue fix

  1. APPS_TEAM: Assuming that other tests Test X3, X4, X5, and X6 failed, on each node, edit /usr/local/blackboard/config/message-queue-service-config.xml and /usr/local/blackboard/config/message-queue-service-config.xml.bb. Increase memoryUsage limit from 256 MB to 512 MB and increase storeUsage limit from 1g to 2g. On all nodes.
    # as bbuser
    # on each node
    sed -i 's/memoryUsage limit="256 MB"/memoryUsage limit="512 MB"/' /usr/local/blackboard/config/message-queue-service-config.xml ;
    sed -i 's/storeUsage limit="1g"/storeUsage limit="2g"/' /usr/local/blackboard/config/message-queue-service-config.xml ;
    sed -i 's/memoryUsage limit="256 MB"/memoryUsage limit="512 MB"/' /usr/local/blackboard/config/message-queue-service-config.xml.bb ;
    sed -i 's/storeUsage limit="1g"/storeUsage limit="2g"/' /usr/local/blackboard/config/message-queue-service-config.xml.bb

Apply XSS security fix

Apply fix to bb-xss-regex-filter.properties
  1. APPS_TEAM: edit the bb-xss-regex-filter.properties file on each node. Note that doing on one node should do on all, but verify on at least two nodes to be sure.
    # On each node (note above about nodes)
    # As bbuser
    nano /usr/local/blackboard/content/vi/bb_bb60/plugins/bb-xss-filter/webapp/WEB-INF/classes/blackboard/xss/regex/bb-xss-regex-filter.properties
    Change:
    disabled_tags.template=(?<=<)\\s*@X@tag@X@(?=[\\s/|>])|(?<=</)\\s*@X@tag@X@(?=\\s*>?)
    to:
    disabled_tags.template=(?<=<)\\s*(?>[^:<>]+:)?\\s*@X@tag@X@(?=[\\s/|>])|(?<=</)\\s*(?>[^:<>]+:)?\\s*@X@tag@X@(?=\\s*>?)
  2. APPS_TEAM: display file contents to confirm the change was made
    # On each node
    # As bbuser
    cat /usr/local/blackboard/content/vi/bb_bb60/plugins/bb-xss-filter/webapp/WEB-INF/classes/blackboard/xss/regex/bb-xss-regex-filter.properties | grep disabled_tags.template ;
Apply fix to bb-file-filter-configuration.properties
  1. APPS_TEAM: edit the bb-file-filter-configuration.properties
    ## On each node
    ## As bbuser
    nano /usr/local/blackboard/config/internal/bb-file-filter-configuration.properties
  2. APPS_TEAM: Change the section:
    # Filtered file types
    .htm=filter
    .html=filter
    .css=filter
    .js=filter
    .txt=filter
    .xml=filter
    .svg=filter
    to
    # Filtered file types
    .htm=filter
    .html=filter
    .xhtml=filter
    .xhm=filter
    .css=filter
    .js=filter
    .txt=filter
    .xml=filter
    .svg=filter
    .svgz=filter
    .xsl=filter
  3. APPS_TEAM: display file contents to confirm the change was made correctly
    cat /usr/local/blackboard/config/internal/bb-file-filter-configuration.properties

 

Verification script

  1. ALL: Verify that configuration changes to other files files have been made successfully on each node.
    # on each node
    # as bbuser
    ~/bb-config-tests-2.sh
  2. ALL: Check that all tests have passed.
  3. Bb_Team: Note finish time on board.

Stage 7 - Day 1 - Pushconfigupdates

Stage 7 is where configuration changes are applied to the application using pushconfigupdates.

Expected start time: 14:55 23 June 2018


On Dev 1, this stage took 20 minutes.
On Dev 2, this stage took 20 minutes.
On Preprod, this stage took 1 hour and 26 minutes.
On Dev 1 in May, this stage took 10 minutes.
On Dev 2 in May, this stage took 20 minutes.
On Preprod in May, this stage took 45 minutes.

  1. Bb_Team: Note start time on whiteboard. Note in RFC that stage has started.
  2. Bb_Team: Tweet that stage has started.
  3. Bb_Team: Update upgrade status page.
  4. ALL: Verify that Blackboard is stopped.
  5. APPS_TEAM: Run pushconfigupdates to propagate the configuration changes on each node, starting with the tasks node (hostname_removed).
    # as bbuser
    # on each node, tasks node first
    cd /usr/local/blackboard/tools/admin/ ;
    ./PushConfigUpdates.sh | tee -a /home/bbuser/pushconfig-stage6.txt.$(date +"%Y-%m-%d") 2>&1
  6. ALL: Monitor progress on tv
    # on big tv
    tail -f /home/bbuser/pushconfig-stage6.txt.$(date +"%Y-%m-%d")
    # during the process
  7. Bb_Team: Check you can log in.
  8. APPS_TEAM: Repeat the verification scripts on the node on which pushconfigupdates has run. This is to check that pushconfigupdates has not undone any of our earlier changes.
    # as bbuser
    # on each node
    ~/bb-config-tests-1.sh
    ~/bb-config-tests-2.sh
  9. APPS_TEAM: Repeat above process until all nodes have had pushconfigupdates run.
  10. APPS_TEAM: Having confirmed that Bb_Team were able to login to each node, stop Blackboard on each node, web nodes (hostname_removed – hostname_removed) first, then tasks node (hostname_removed).
    # on each node, web first, then tasks
    # web nodes may be stopped concurrently
    service blackboard stop
  11. SYS_TEAM: Take VMWare Snapshot #4.
  12. APPS_TEAM: Stop Blackboard on each node, web nodes (hostname_removed – hostname_removed) first, then tasks node (hostname_removed).
    # on each node, web first, then tasks
    # web nodes may be stopped concurrently
    service blackboard stop
  13. Bb_Team: Note finish time on board.

Stage 8 - Day 1 – Log files check



Expected start time: 15:50 23 June 2018

  1. ALL: Review key log files to identify any potential issues. See hostname_removed/wiki/Blackboard/Troubleshooting#Blackboard_Logs for a list of log files to check.
  2. ALL: Research any issues on the Blackboard support site, and raise a ticket with Blackboard support if required.

Stage 9 - Day 2 – Log files check



Expected start time: 09:00 24 June 2018
N.B. Highfield team to be in place and ready to start by 08:55

  1. ALL: Review key log files to identify any potential issues. See hostname_removed/wiki/Blackboard/Troubleshooting#Blackboard_Logs for a list of log files to check.
  2. ALL: Research any issues on the Blackboard support site, and raise a ticket with Blackboard support if required.
  3. Bb_Team: Check that data integration snapshots ran ok overnight.

Stage 10 - Day 2 – GUI based configuration and UAT

This stage encompasses Bb_Team team GUI based configuration and testing.

Expected start time: 09:35 24 June 2018


On Dev 1, this stage took 1 hour and 21 minutes
On Dev 2, this stage took 40 minutes
On Preprod, this stage took 1 hour and 6 minutes
On Dev 1 in May, this stage took 30 minutes
On Dev 2 in May, this stage took 35 minutes
On Preprod in May, this stage took 1 hour and 20 minutes

  1. Bb_Team: Note start time on whiteboard. Note in RFC that stage has started.
  2. Bb_Team: Tweet that stage has started.
  3. Bb_Team: Update upgrade status page.

Permissions fix

  1. Bb_Team: Use the support tool to generate the output to these queries:
    select *
    from ENTITLEMENT
    order by pk1 DESC
    select *
    from SYSTEM_ROLES_ENTITLEMENT
    order by pk1 DESC
  2. Bb_Team: Using the methodology described in the document hostname_removed create an SQL query to restore access to the Spell Check and Math Editor settings on the Sys Admin panel. Send the query to DBA.
  3. DBA: Back up the SYSTEM_ROLES_ENTITLEMENT and ENTITLEMENT tables and commit the query.

Branding

Bb_Team: Dev/Preprod - adjust version number in Gateway file.

  1. # as bbuser
    # on each node
    nano /usr/local/blackboard/webapps/content/branding/_1_1/gateway.bb
    nano /usr/local/blackboard/content/vi/BBLEARN1/branding/gateway.bb
    nano /usr/local/blackboard/content/vi/bb_bb60/branding/gateway.bb

Building Blocks

Portfolio

  1. Bb_Team: Check whether the Portfolio Building Block is currently installed.
  2. Bb_Team: Stop Blackboard on each node, web nodes (hostname_removed – hostname_removed) first, then tasks node (hostname_removed).
    # on each node, web first, then tasks
    # web nodes may be stopped concurrently
    service blackboard stop
  3. Bb_Team: If the Portfolio building block is installed, remove it.
    # as bbuser
    # on tasks node (hostname_removed)
    cd /usr/local/blackboard/tools/admin/ ;
    ./B2Manager.sh -dor bb-portfolio
  4. Bb_Team: Delete the portfolio directory.
    # as bbuser
    # on tasks node (hostname_removed)
    DEVs: cd /usr/local/blackboard/content/vi/BBLEARN1/plugins
    PPRD/LIVE cd /usr/local/blackboard/content/vi/bb_bb60/plugins
    rm -r bb-portfolio/
  5. Bb_Team: Check whether the portfolio building block exists at this location.
    /usr/local/blackboard/system/autoinstall/enterprise.contentsystem/allavailable/portfolio.war
  6. Bb_Team: If not then download the portfolio b2 to bbuser's home directory.
    # as bbuser
    # on tasks node (hostname_removed)
    cd
    wget hostname_removed.ac.uk/bb/bb2018-fixes/portfolio.war
  7. Bb_Team: Load the portfolio building block.
    # as bbuser
    # on tasks node (hostname_removed)
    cd /usr/local/blackboard/tools/admin/
    # if using local version
    ./B2Manager.sh -doi /usr/local/blackboard/system/autoinstall/enterprise.contentsystem/allavailable/portfolio.war
    # if using bbuser version
    ./B2Manager.sh -doi /home/bbuser/bb-portfolio.war
  8. Bb_Team: Verify the building block
    # as bbuser
    # on tasks node (hostname_removed)
    ./B2Manager.sh -v bb-portfolio
    # should result with Version: 3300.0.1-rel.60+a05276c
  9. Bb_Team: Make the Portfolio building block available.
    # as bbuser
    # on tasks node (hostname_removed)
    ./B2Manager.sh -s AVAILABLE bb-portfolio
  10. Bb_Team: Start Blackboard on tasks node (hostname_removed).
    # as bbuser
    # on tasks node only
    service blackboard start

Other Building Blocks

  1. Bb_Team: If required, update Assignments Building Block on tasks node GUI.
    location: hostname_removed\Assignments\Version 3300.0.1-rel.61+f9847e0\bb-assignment.war
  2. Bb_Team: Restart Blackboard on tasks node (hostname_removed).
    # on tasks node only
    service blackboard restart
  3. Bb_Team: If required, update Partner Cloud building block on tasks node GUI.
    Currently held at hostname_removed\bbgs-partner-cloud.war
  4. Bb_Team: Restart Blackboard on tasks node (hostname_removed).
    service blackboard restart
  5. Bb_Team: If required, update Telemetry building block on tasks node GUI.
    Currently held at hostname_removed\bb-telemetry.war
  6. Bb_Team: Restart Blackboard on tasks node (hostname_removed).
    service blackboard restart
  7. Bb_Team: Update Qwickly building block on tasks node GUI.
    Currently held at hostname_removed\qwickly\4.4.2.1 for 2017 Q4\Qwickly4.4.2.1.war
  8. Bb_Team: Restart Blackboard on tasks node (hostname_removed).
    service blackboard restart
  9. Bb_Team: Update Turnitin building block on tasks node GUI.
    Currently held at hostname_removed\Turnitin\2.7.5\bb-basic-2.7.5.war
  10. Bb_Team: Restart Blackboard on tasks node (hostname_removed).
    service blackboard restart
  11. Bb_Team: Having checked you can log in, start Blackboard on web nodes hostname_removed - hostname_removed.
    # as bbuser
    # on web nodes
    service blackboard start

Disable certain building block tools

  1. Bb_Team: On tasks node GUI, disable Portfolio and Ally Tools via
    1. Administrator Panel
    2. Tools
    3. Ally -> All OFF
    4. API Gateway -> OFF
    5. Portfolio -> OFF
    6. Submit
  2. Bb_Team: Check that Video Integration tool (currently unsupported) is not enabled.
    1. Administrator Panel
    2. Building Blocks
    3. Video Integration
    4. Settings
    5. Availability -> Off
    6. Submit

Other GUI based configuration

  1. Bb_Team: Check that language packs have not been disabled.
  2. Bb_Team: Check whether UK English UoS version needs to be updated. Update if necessary by following the guidance at hostname_removed/wiki/Blackboard/Language_Pack_Changes
  3. Bb_Team: Disable "Application Authorisation".
    1. System Admin
    2. Tabs and Modules
    3. Tool Panel
    4. Application Authorisation - Delete
    5. Goals - Delete
    6. Goal Performance - Delete
    7. Enterprise Surveys - Delete
  4. Bb_Team: Set the default course theme.
    1. System Admin
    2. Brands and Themes
    3. Default Course Design
    4. Set "Use Theme Colours" to "BB2012 Theme"
    5. Click Preview
    6. Click Submit
  5. Bb_Team: Set Google Oauth code
    1. System Admin
    2. Building Blocks
    3. Installed Tools
    4. Page 2
    5. Content Editor -> Settings
    6. Set Oauth address to that stated in hostname_removedGoogleAPI codes-for-WYSIWYG_editor\
  6. Bb_Team: Check that spell check is set to UK English and other dictionaries are disabled.
  7. Bb_Team: Set students to not be able to email all users on a course:
    1. Course/Organisation Roles > Manage Privileges: Student/Participant
    2. All users -> Untick.
    3. All students -> Untick.
  8. Bb_Team: Check Mobile Building Block settings via Building Blocks > Installed Tools > Blackboard Mobile Web Services Building Block > Server Registration Settings and ensure that Environment is set correctly.
  9. Bb_Team: Set icon set to 2016 theme.

Testing

  1. Bb_Team: UAT Blackboard via GUI on each node.
  2. Bb_Team: Scan testing results and save in J drive upgrade folder.
  3. Bb_Team: Note finish time on board.

Stage 11 - Day 2 – Restart Test

In this stage we check that Blackboard restarts ok.


Expected start time: 13:10 24 June 2018

  1. Bb_Team: Note start time on whiteboard. Note in RFC that stage has started.
  2. Bb_Team: Tweet that stage has started.
  3. Bb_Team: Update upgrade status page.
  4. SYS_TEAM: Perform vApp stop of Blackboard nodes in line with standard method used for patching.
  5. SYS_TEAM: Perform vApp start of Blackboard nodes in line with standard method used for patching.
  6. APPS_TEAM: Once Blackoard available again, restart ActiveMQ on tasks node to ensure that it is running correctly.
    >> as bbuser
    >> on tasks node
    cd /usr/local/blackboard/tools/admin
    ./ServiceController.sh services.activemqbroker.restart
  7. Bb_Team: Check you can log into web nodes.
  8. Bb_Team: Make a non-immediate announcement on each node, to test later that ActiveMQ is working. Emails should appear within an hour.
  9. Bb_Team: On each node, add a new item to a test course and check it appears ok.
  10. Bb_Team: Note stage completion time on whiteboard.

Stage 12 - Day 2 - Go Live



Expected start time: 14:45 24 June 2018

  1. Bb_Team: Note start time on whiteboard. Note in RFC that stage has started.
  2. Bb_Team: Tweet that stage has started.
  3. SYS_TEAM: Following confirmation by Bb_Team, make Blackboard live via NLB.
  4. Bb_Team: Announce service availability via twitter @uos_Bb_Team and blackboard-users@lists mailing list.
  5. Bb_Team: Note stage completion time on whiteboard.
  6. Bb_Team: Photograph whiteboard.
  7. Bb_Team: Scan testing sheets and attach PDFs to RFC
  8. Bb_Team: Close RFC.

Stage 13 - Day 2 – Epilogue

This stage involves tidying up after successful testing.

Expected start time: 15:10 24 June 2018

  1. Bb_Team: Note start time on whiteboard. Note in RFC that stage has started.
  1. APPS_TEAM: Remove backups of zip files and uncompressed directories.
    # as bbuser
    # on all nodes
    rm /home/bbuser/login.zip
    rm /home/bbuser/photos.zip
    rm -rf /home/bbuser/login-extracted
    # on tasks node
    rm /home/bbuser/snapshotfiles.zip
    rm /home/bbuser/data_integration.zip
    rm -rf /home/bbuser/data_integration-extracted
    rm -rf /home/bbuser/snapshotfiles-extracted
  2. APPS_TEAM: Remove copied licence file.
    # as bbuser
    # on all nodes
    rm /home/bbuser/blackboard-licence .xml
  3. APPS_TEAM: Clear installer directory.
    # bbuser
    rm -rf /usr/local/bbinstaller
  4. DBA: Reset DB passwords for sys, system.
  5. SYS_TEAM: Following specific confirmation from Bb_Team delete snapshots.
  6. Bb_Team: Note finish time on board.
  7. Bb_Team: Close RFC.

Summary of VMWare Snapshots

Snapshot #1 – Installer in place, licence file copied, branding backed up, lucene log fix, permissions set, no application changes.

Snapshot #2 - Oracle patch applied

Snapshot #3 - Application upgraded

Snapshot #4 – Configuration changes applied