mdeeprose

Highlights of fixes from our upgrade implementation plan

Blog Post created by mdeeprose on Jun 23, 2017

Our implementation plan is quite long and hard to skim over.  Here I've tried to highlight some of the interesting fixes that we have included that may be of particular interest to others.

Note that some additional fixes are listed here: A successful upgrade

File ownership

Blackboard files should be owned by bbuser, otherwise upgrades will fail.

>> On each node 
find /data/bb/blackboard/ -maxdepth 1 -mindepth 1 -type d -not -name content | xargs -I {} chown -R bbuser\: {}

Ulimit

The upgrade produced errors due to ulimit being too low. Therefore we should raise the ulimit as follows:

>> On each node 
>> Login as root 
-     vim /etc/security/limits.conf
-     add following lines:
*               soft    nofile          5000
*               hard    nofile          65536
  >> Pay attention that # does not appear in front of either line
-     save
-     logout
-     login 
>> as root check:
-     ulimit -n
-     ulimit -Hn 
[root@~]# ulimit -n 5000
[root@~]# ulimit -Hn 65536 
>> switch to bbuser and check is the same result:
-     su bbuser
-     ulimit -n
-     ulimit -Hn
[bbuser@]$ ulimit -n
5000
[bbuser@]$ ulimit -Hn
65536

Java Log Spam fix

To prevent log spam in the stdout-stderr log perform the following:

 >> on each node 
mkdir -p /home/bbuser/.java/.userPrefs
mkdir /home/bbuser/.java/.systemPrefs
chown -R bbuser:bbuser /home/bbuser/.java/
chmod 755 -Rv /home/bbuser/.java/
>> on each node
>> within /usr/local/blackboard/config/bb-config.properties
>> add to bbconfig.jvm.options.extra.tomcat=
-Djava.util.prefs.systemRoot=/home/bbuser/.java/

Speed up Blackboard start and stop

In order to reduce time that it takes for Blackboard to services to stop as recommended at https://blackboard.secure.force.com/btbb_articleview?id=kAC7000000000FSmake following change to /usr/local/blackboard/apps/tomcat/conf/wrapper.conf on each node:

>> As bbuser 
>> On each node
cd /usr/local/blackboard/config/tomcat/conf/
vim /usr/local/blackboard/config/tomcat/conf/wrapper.conf
>> find the line
wrapper.shutdown.timeout=300
>> and change it to
wrapper.shutdown.timeout=30
>> find the line
wrapper.jvm_exit.timeout=15
>> and change it to
wrapper.jvm_exit.timeout=5

 

And make the following change to /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb on each node:

>> As bbuser 
>> On each node
cd /usr/local/blackboard/config/tomcat/conf/
vim /usr/local/blackboard/config/tomcat/conf/wrapper.conf.bb
>> find the line
wrapper.shutdown.timeout=300
>> and change it to
wrapper.shutdown.timeout=30
>> find the line
wrapper.jvm_exit.timeout=15
>> and change it to
wrapper.jvm_exit.timeout=5

Resolve Math Editor not loading issue

Issue described at https://blackboard.secure.force.com/btbb_articleview?id=kA370000000H4uu can be resolved by:

On each node update the http-cacheable-extensions.txt to include reference to resolve issues with Equation editor.

>> on each node 
>> as bbuser
cd /usr/local/blackboard/config/internal/
nano http-cacheable-extensions.txt
>> If not already present, add the following to the list in their alphabetical place:
eot
ttf
woff
>> and save it.

SCORM connection loss fix

As recommended by Blackboard support in case #02455370 apply a fix in two files to prevent SCORM learning objects losing connection to Blackboard.

As bbuser edit bb-config.properties.template on each node and make the following change.

>> as bbuser 
>> on each node
nano /usr/local/blackboard/config/bb-config.properties.template
>> Change
#bbconfig.headers.CSP.frame-ancestors=[none OR self OR <comma-delimited URL list>]
bbconfig.headers.CSP.frame-ancestors=self
>> to
#bbconfig.headers.CSP.frame-ancestors=[none OR self OR <comma-delimited URL list>]
bbconfig.headers.CSP.frame-ancestors=self' https://*.soton.ac.uk 'soton.ac.uk
>>Note that this is the last line in the file

As bbuser edit the bb-config.properties on each node

>> as bbuser 
>> on each node nano /usr/local/blackboard/config/bb-config.properties
>> Change
#bbconfig.headers.CSP.frame-ancestors=[none OR self OR <comma-delimited URL list>]
bbconfig.headers.CSP.frame-ancestors=self
>> to
#bbconfig.headers.CSP.frame-ancestors=[none OR self OR <comma-delimited URL list>]
bbconfig.headers.CSP.frame-ancestors=self' https://*.soton.ac.uk 'soton.ac.uk

Summary of all changes made to bb-config.properties

The bb-config.properties file is one of the primary locations for Blackboard settings. With the 2016 Q4 upgrade we made the following adjustments. Note that this includes the Java log spam and SCORM connection loss fix already detailed above.

As bbuser edit the bb-config.properties file on each node and make the following five changes.

>> as bbuser 
>> on each node
nano /usr/local/blackboard/config/bb-config.properties
_______________
>> Change 1
>> Change bbconfig.max.stacksize.tomcat=
>> from 400k to 512k i.e.
bbconfig.max.stacksize.tomcat=512k
_______________
>> Change 2
>> Change bbconfig.jvm.options.extra.tomcat=
>> So that whole section is replaced with
bbconfig.jvm.options.extra.tomcat=-XX:NewSize=2048m -XX:MaxNewSize=2048m -XX:+UseTLAB -XX:SurvivorRatio=4 -XX:+UseCompressedOops -Xss512k -XX:+PrintVMOptions
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCTaskTimeStamps -XX:+PrintCommandLineFlags -XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCApplicationConcurrentTime -XX:+AlwaysPreTouch -XX:+DoEscapeAnalysis -Xverify:none -Djava.util.prefs.systemRoot=/home/bbuser/.java/
_______________
>> Change 3
>> Change bbconfig.jvm.options.gc= >> So that whole section is replaced with
bbconfig.jvm.options.gc=-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=45 -XX:+UseStringDeduplication
_______________
>> Change 4
>> Check dmarc setting, ensure that:
bbconfig.email.use.dmarc.from.override=false
_______________
>> Change 5
>> Update click jacking setting to resolve issue where
>> SCORM learning objects suffer disconnections
>> Blackboard case #02455370
>> Change
#bbconfig.headers.CSP.frame-ancestors=[none OR self OR <comma-delimited URL list>]
bbconfig.headers.CSP.frame-ancestors=self
>> to
#bbconfig.headers.CSP.frame-ancestors=[none OR self OR <comma-delimited URL list>]
bbconfig.headers.CSP.frame-ancestors=self' https://*.soton.ac.uk 'soton.ac.uk
>>Note that this is the last line the file.

 

Workaround to issue LRN-107071: High CPU and High Load Caused by Script MicrosoftDocumentParser.sh

LRN-107071: High CPU and High Load Caused by Script MicrosoftDocumentParser.sh is an issue that should be resolved in 2017 Q2, so this workaround should not be needed in future upgrades.

add lines to /usr/local/blackboard/config/internal/xythos-indexing-filter.txt

>> On each node 
>> Edit /usr/local/blackboard/config/internal/xythos-indexing-filter.txt
nano /usr/local/blackboard/config/internal/xythos-indexing-filter.txt
>>Ensure the four lines below are included in the file:
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet  <---this line may already be in the file
application/vnd.openxmlformats-officedocument.wordprocessingml.document
application/vnd.openxmlformats-officedocument.presentationml.presentation
application/vnd.openxmlformats-officedocument.presentationml.slideshow

Adjust logging and task management

Our Blackboard environment has four web nodes and a tasks node. We will set the tasks node to run tasks and the web nodes to not run tasks. We will set the logging on the web nodes to show fatal errors only, whilst the tasks node will log warnings as well (default setting).

N.B. To set a node to run tasks it is also necessary to set

bbconfig.frontend.fullhostname=true

in the bb-config.properties file on that node.  This setting is honoured by the installer, but the following additional necessary setting is wiped by the installer.

Edit the /usr/local/blackboard/config/service-config.properties on all nodes EXCEPT tasks node to ensure that tasks node is the only node set to carry out background tasks, thus optimising performance of web facing end user nodes.

As bbuser on all nodes except tasks node 
Edit the file /usr/local/blackboard/config/service-config.properties 
nano /usr/local/blackboard/config/service-config.properties 
and alter the line
blackboard.service.queuedtaskmanager.param.num-concurrent-tasks=2
to read
blackboard.service.queuedtaskmanager.param.num-concurrent-tasks=0
>> and change logging level to fatal on all nodes but tasks node 
blackboard.service.log.param.logdef.services.verbosity=fatal

Outcomes