[Upgrade] High CPU Load after Upgrade - Release 2016 Qx

Document created by soomyoung.yi on Oct 15, 2017Last modified by soomyoung.yi on Oct 15, 2017
Version 2Show Document
  • View in full screen mode

Self Hosting 고객 중, 2016 버전으로 업그레이드 실시 직후 서비스는 접근 가능하나, Application Server 들의 CPU 사용률이 90% 이상으로 상승하는 이슈가 발생할 수 있습니다.

따라서, 이러한 현상 발생 시 Behind Team 에 Case 를 열어 해당 로그 및 쿼리를 조사하여 보내주시면 Patch를 전달받을 수 있습니다.

  • Target - Release 3100.x (2016 Qx)
  • 증상 - Blackboard Service 는 정상적으로 동작하나 Application Server 의 CPU 가 90% 이상 증가, 사용자가 늘어나면 서비스가 정상적으로 이루어지지 않음
  • 원인 - Tomcat 의 Session 관리 Bug
  • 체크사항 - 각 Application Server 의 CPU usage
  • 조치사항 - 아래의 항목에 대한 데이터를 수집하여 Behind 에 Case Open 및 패치를 전달 받아 각 Application Server 에 적용
  • 아래의 자료를 보내 주세요. (Performance Query의 결과는 Excel로 보내 주세요.)

    • 1. 쓰레드 텀프 생성 ( 30초 간격으로 3개)

      • For performance issues such as this we will need some performance logs as per the attached article.

      • The next time that this occurs can you please take 3 thread dumps, taken 30 seconds apart, and attach them to this case?

      • ServiceController.bat services.appserver.threaddump

    • 2. 로그 파일들

      • In addition, from the "Collect" section of the article, we will require the following log files:

      • Log Files

      • The following log files should be collected for all performance issues.  Note that many of these logs are automatically rotated at midnight.

      • Filename    Description

      • bb-services-log.txt    Primary Blackboard application log. While performance issues typically don't cause exceptions to be thrown, there is sometimes evidence of problems in this log

      • bb-sqlerror-log.txt    Exceptions from database statement executions, may indicate the cause of database related problems

      • tomcat/bb-access-log.<date>.txt    Tomcat HTTP access log

      • tomcat/catalina-log.<date>.txt    Tomcat error log

      • tomcat/gc.log    Java Garbage Collection diagnostic logging - enabled automatically as on 9.1, for other versions seeEnabling JVM Garbage Collection logging for tomcat

      • tomcat/stdout-stderr-<date>.log    Tomcat standard error/standard out logging

      • And lastly please also send a copy of your bb-config.properties file (you can remove any passwords) and a list of all patches installed on your server.

    • 3. DB 관련 performance 조사 자료 (6개의 결과)

      • Database:

      • The following queries will provide us with details on Database performance.  Can you please run these queries in MSSQL and send the results?

-- CPU stats

Select signal_wait_time_ms=sum(signal_wait_time_ms)

,'%signal (cpu) waits' = cast(100.0 * sum(signal_wait_time_ms) / sum (wait_time_ms) as numeric(20,2))

,resource_wait_time_ms=sum(wait_time_ms - signal_wait_time_ms)

,'%resource waits'= cast(100.0 * sum(wait_time_ms - signal_wait_time_ms) / sum (wait_time_ms) as numeric(20,2))

From sys.dm_os_wait_stats;

-- Top 5 SQL statements


total_worker_time/execution_count AS [Avg CPU Time],

(SELECT SUBSTRING(text,statement_start_offset/2,(CASE WHEN statement_end_offset = -1 then LEN(CONVERT(nvarchar(max), text)) * 2 ELSE statement_end_offset end -statement_start_offset)/2) FROM sys.dm_exec_sql_text(sql_handle)) AS query_text

FROM sys.dm_exec_query_stats


-- SQLServer:Databases stats

select 'SQLServer:Databases '+counter_name,sum(cntr_value) from sys.dm_os_performance_counters

where object_name='SQLServer:Databases'

and counter_name in ('Active Transactions','Log Flushes/sec', 'Percent Log Used', 'Transactions/sec')

group by counter_name ;

-- SQLServer:General Statistics

select 'SQLServer:General Statistics '+counter_name,sum(cntr_value) from sys.dm_os_performance_counters

where object_name='SQLServer:General Statistics'

and counter_name in ('Processes blocked','User Connections')

group by counter_name ;

-- SQLServer:Latches

select 'SQLServer:Latches '+counter_name,sum(cntr_value) from sys.dm_os_performance_counters

where object_name='SQLServer:Latches'

and counter_name in ('Latch Waits/sec','Total Latch Wait Time (ms)')

group by counter_name ;

-- SQLServer:Locks

select 'SQLServer:Locks '+counter_name,sum(cntr_value) from sys.dm_os_performance_counters

where object_name='SQLServer:Locks'

and counter_name in ('Lock Timeouts/sec','Lock Wait Time (ms)', 'Lock Waits/sec','Number of Deadlocks/sec')

group by counter_name ;