6 Replies Latest reply on May 1, 2017 5:28 AM by af0055432

    BlackBoard App server monitoring

    gk30746

      Hi All,

       

      Can anyone suggest a better tool for monitoring BlackBoard App server? Can Cacti monitor JMX? Or is there any other better option for the same?

        • Re: BlackBoard App server monitoring
          af0055432

          I want to preface this comment with two things:

          1. Monitoring is a very tricky thing, you need to balance between collecting data too frequently or not frequent enough. You need to ensure your triggers aren't too light, if people start getting too many messages, eventually people see the system as a "boy who cried wolf". If you or your institution are looking to begin monitoring your applications, let them know that time invested into planning will help tremendously and help you realise the RoI.
          2. I am a Zabbix Certified Professional. I'm not trying to sell anything, I have a lot of experience with this that I just don't have for any other monitoring systems.

           

          I've worked with an open source piece of software named Zabbix for approximately 7 years, at least 5 of those years I've been using it to monitor my Blackboard application servers.

          Without being sure on your environment or your organisation, the following pieces of Zabbix functionality are likely to help you out:

          • Simple Checks (Agentless. Including: Ping, port status, page scraping, etc)
          • Zabbix Agent (Includes: CPU % Used, % Memory Used, % Disk Full, etc)
          • JMX Monitoring (Agentless. Includes: Number of user sessions, number of abandoned sessions, number of errors, B2 metrics)
          • ODBC Monitoring (Agentless. Can query underlying databases for data/metrics)
          • Log File Monitoring (Needs an agent. Can read log files and raise alarms when regular expression conditions are met)

           


           

          Because you've mentioned by name that you're looking for JMX monitoring, these are my recommendations/pointers:

          1. You need arguments in your bb-config.properties. (See below)
          2. Try your connection from JVisualVM or JConsole before you invest time in Zabbix or another tool. Firewall or other networking issues are best worked out as early as possible.
          3. The unpredictable Java RMI port behind a firewall can make you very sad.
          4. If you're installing Zabbix, make sure you include the Java Gateway if compiling from source.
            Note: You can run every component of Zabbix on a single host, MySQL (or PostgresSQL), Zabbix and the Java Gateway.
          5. Read this as a general rule if you're using Zabbix, use discovery wherever possible.
            Edit: Strikethrough: The following link will save you hours in creation and maintenance of your templates for JMX monitoring: JMX Discovery
            Edit: Appended: Since Zabbix v3.4, JMX discovery and configurable endpoints are built-in!

           

          Arguments in Bb Configuration

          The following entries are required in the 'bbconfig.jvm.options.extra.tomcat' options list if you're presenting unauthenticated JMX monitoring:

          • -Dcom.sun.management.jmxremote
          • -Dcom.sun.management.jmxremote.port={jmxport}
          • -Dcom.sun.management.jmxremote.ssl=false
          • -Dcom.sun.management.jmxremote.authenticate=false
          • -Djava.rmi.server.hostname={yourhost.yourdomain}

           

          The FQDN entered under Djava.rmi.server.hostname must exactly match the hostname you're trying to connect to with your JMX tools.

          If your organisation/security policy dictates that you need to have a username + password. You will need to read up on the extra requirements.

           

           


           

          Zabbix is an amazing platform that fits my needs really well, I gave a very simplified presentation in 2015 at the ANZTLC about the benefits of monitoring your Blackboard application however now that I'm employed by another institution, I do not have the rights to distribute that presentation.

           

          I also found a presentation given in 2012 at Blackboard DevCon on Zabbix & Blackboard: Here.
          The links in that presentation Blackboard templates provided by Blackboard will be pointing to an old link that existed on EduGarage many years ago, I haven't been able to find a mirror of that article in a while though.

          Edit: Link to Blackboard provided templates (years old)

           

           

          Best of luck and have fun!

           

          Ashley.

          4 of 4 people found this helpful
          • Re: BlackBoard App server monitoring
            jefflong

            We use Zenoss and it includes JMX monitoring so it works fairly well for us.

            • Re: BlackBoard App server monitoring
              gk30746

              Hi Ashley,

               

              I have successfully installed zabbix and now I can see JMX gateway able to fetch details from our blackboard servers. Now I have a few questions which im trying to figure out:

               

              1. Which JMX paramer we need to configure to get the total number of concurrent assessments happening across all blackboard servers. Is there any single parameter for this or do we need to take the aggregate of a particular parameter across all the servers?

               

              2. Similarly which JMX paramer we need to monitor to get a count of total number of students currently logged into all blackboard servers?

              3. Also which all java/backboard parameters do we need to monitor to analyze the smooth working of blackboard application.

               

              Apologies if these questions are too basic. Im trying to understand how blackboard can be monitored efficiently.

               

              Thanks

              Gopihere

                • Re: BlackBoard App server monitoring
                  af0055432

                  Not a problem Gopi, I'm glad you were able to get it all configured nicely.

                   

                  I have just found a mirror of the templates Blackboard built for monitoring of the application, but just a warning, these are very old: https://github.com/blackboard/zabbix-ext

                  GitHub - blackboard/zabbix-ext: More metrics, elasticity, scalability, and automation for Zabbix.

                   

                  The reason that I dug up the above repository from Blackboard is that there is a building block available under the zabbix-ext/templates/blackboard_learn/b2/201404/ folder named 'bb-extended-monitoring'. This building block and the metrics presented through that were not required for me to report on, so I haven't used it in a production sense for the last 3 or so years.

                   

                  It may be perfect for you however, the items included in the 'application_single_agent.xml' template are overview metrics of your environment (you can run the template on just one of your hosts (I find this prospect scary though, if you lost this host for a period of time)). The items that appear as though they fit your needs:

                  1. <name>Application: Assessment Attempts in Past Hour</name>
                  2. <name>Application: Active Sessions (Last 60min)</name>

                   

                  Unfortunately I believe that the above building block needs to be deployed for these items to be exposed for you to use. If you cannot deploy the building block, you may be able to get indicative numbers by aggregating the number of tomcat sessions running across your hosts, and the number of users currently using the assessment/assignment building blocks.

                   

                  A very quick overview of what I would suggest to look at in terms of performance are the following:

                  • [JMX] Garbage Collections (frequency mainly, particularly the concurrent mark sweep GCs)
                  • [JMX] Heap Memory usage, this should be a saw-tooth pattern in your graphs, and the drops in used heap space should coincide with garbage collections, there should be a very apparent drop in the graph when a CMS GC runs.
                  • [JMX] Number of Tomcat session errors should be very low
                  • [Agent] If you've deployed agents on your hosts, watch the status of the Blackboard services (running or not, % CPU used for each). I do watch the % memory used for the services, but this is only because of how I've built my templates. In Blackboard, heap memory is what you need to be keeping an eye on.
                  • [Log Files] In my last position I had Splunk (not open source) to ingest and index all of my log files, but you can do log file monitoring with Zabbix I'm just not savvy to it. I would really urge you to look at your access logs for client responses, what you're interested in is how many 40# and 50# responses your users are receiving. These typically will tell you whether your clients are having a bad time using Blackboard.

                   

                  There's no such thing as a question that's too basic .

                   

                  I realise that you've probably got a lot of stuff to do, but if you find yourself with a little bit of time, I would recommend listening to presentations given at Zabbix conferences over the last few years. This one is my personal recommendation: Raymond Kuiper - Zen and The Art of Zabbix Template Design | Zabbix Conference 2016 - YouTube.

                   

                  I'm going to keep adding to this post over next few days, I've got a lot of meetings today and a strict deadline for the end of the week.