Javascript required
Skip to content Skip to sidebar Skip to footer
Home / Ca Application Performance Management Solutions for Sap

Ca Application Performance Management Solutions for Sap

General

Information to be Provided for Support Messages

The following items are required for support messages:

  • Version of software component LM-SERVICE on SAP Solution Manager Java stack
  • Version of component ST on SAP Solution Manager ABAP stack
  • http Remote connection (if not conflicting with company policy) to
    • SAP Solution Manager Java stack
    • SAP Solution Manager ABAP stack
    • Introscope Webview
  • Credentials (in Secure Area) for a user with admin permission
  • Log files from component with problems (EM, agent, ...)
  • Screenshots of the problem (if this makes sense)

Enterprise Manager (EM)

Additional Information to be Provided for EM Issues

  1. Description of the problem
    • What kind of problem (slow Workstation / Webview, gaps in line chart, MoM losing connection to collectors, ...)?
    • Screenshot of the problem (e.g. from Workstation / Webview screens)
    • Re-occurrence: periodically, occasionally,
    • Reproducible? If yes, how to reproduce?
  2. Time range
    • When did the problem start occurring?
    • Did the problem disappear without any action? If yes, when did it disappear?
  3. Thread dumps
    • If you observe any kind of hang / crash situation trigger a series of thread dumps for the MoM at least, if possible also for the collectors. See below for details on triggering thread dumps.
  4. Log files
    • Provide a complete set of log files (full contents of /logs folder) for MoM and all collectors. The log files must cover the time range when the problem occurred. Thread dumps will also arrive in the logs folder (em.log / EMService.log)
  5. Heap dumps
    • In case of out of memory situations heap dumps may have been generated (java_*.hprof)  in the EM installation folder. They might be needed as well
  6. Contents of the /data folder of the MoM installation
    • This contains important self-monitoring metrics for post-mortem analysis

EM Additional Diagnostics Options

To enhance the diagnostics options the following configuration changes are recommended (at least in case problems occur)

  • Run the Enterprise Manager on SAP JVM instead of the bundled Oracle / IBM JVM. This will provide more details in thread dumps, address some heap dump problems of Oracle JVM and enable a bunch of other diagnostics facilities. Make sure to use the same major Java version as present in the bundled JVM:
    • Java 8 for EM 10.1 and higher
  • Oracle / SAP JVM: Set the JVM parameter -XX:+HeapDumpOnOutOfMemoryError

Enterprise Manager Log Files

Most relevant log files for problems with EM in sub directorylogs are:

  • IntroscopeEnterpriseManager.log
  • IntroscopeEnterpriseManagerSupport.log

To get log out put with log level DEBUG in file IntroscopeEnterpriseManager.log replace line

log4j.logger.Manager=INFO, logfile

with

log4j.logger.Manager=DEBUG, logfile

in file file <EM_Home>/config/IntroscopeEnterpriseManager.properties.

On Unix platforms the file em.log in the EM home directory contains helpful information. All console output of the Java VM, including full thread dumps, goes here.

On Windows platforms the logs folder contains an additional file EMService.log. This is written by the Windows service wrapper and may contain important information in case the EM fails to start as Windows service.

Enterprise Manager Version

There are multiple ways to determine the EM version:

  1. The log file IntroscopeEnterpriseManager .log contains the full version string at each startup of the EM. Example:

    8/23/12 09:49:47.289 AM MESZ [ INFO ] [ WrapperSimpleAppMain ] [Manager] Introscope Enterprise Manager Release 9.1.0.2(Build 581100)


    SAP Solution Manager 7.1 discovers the version and displays it in Basic Configuration.

Adapt Enterprise Manager Java VM Parameters

Depending on the OS there are different locations where you adapt the Java VM parameters.

Windows Service
Adapt the file <EM_HOME>/bin/EMService.conf. The propertieswrapper.java.additional.<number> contain the Java parameters. Java heap size is specified in parameterswrapper.java.maxmemory andwrapper.java.initmemory.

Unix
Adapt the file Introscope_Enterprise_Manager.lax in the EM home directory. Property lax.nl.java.option.additional contains all JVM parameters. For Java heap size see Java parameter -Xmx and -Xms.

Recommended Java VM parameters for Enterprise Manager

In case the EM crashes with OutOfMemory (OOM) it makes sense to analyze the Garbage Collections (GC) in more detail. So setting additional java parameters
-verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails is recommended.
With java parameter -Xloggc:gc.log the GC output is redirected from file em.log to a dedicated file gc.log.

The parameter -XX:+HeapDumpOnOutOfMemoryError is useful to trigger automatically a heap dump in case of OOM.
NOTE: heap dumps get a big as the Java heap size is configured. So make sure you have enough disk space available.

The bigger the Java heap size is configured the more it makes sense to work with parallel GCs by
setting additional java parameters -XX:+UseParNewGC -XX:+UseConcMarkSweepGCor   -XX:+UseG1GC.

Start/Stop the Enterprise Manager

  1. Windows Service: Default Windows Service display name is "Introscope Enterprise Manager"
  2. Unix: use the script EMCtrl.sh in folder bin:
    • EMCtrl.sh start
    • EMCtrl.sh
    • stop

Register/Unregister Windows Service

The Windows Service to start/stop the Enterprise Manager usually gets registered by default during EM installation. In case you want to register or unregister this Windows Service manually you can do that with the script <EM_HOME>/bin/EMCtrl64.bat. The command to register the service isEMCtrl64.bat registerandEMCtrl64.bat unregisterto remove the Windows Service.

The Windows Service display name and other details can be set in file  <EM_HOME>\bin\EMService.conf. Changes to e.g. propertywrapper.displayname modifying the default service display name "Introscope Enterprise Manager" of an already registered Windows Service only take effect after unregistering and registering the Windows Service.

Other properties you may need to change e.g. if you run several EMs on the same Windows host arewrapper.name andwrapper.description.

Collect Thread Dumps from the Enterprise Manager

If the Enterprise Manager appears stuck it may make sense to trigger a thread dump. The procedure differs depending on the operating system and the mode it is started.

Windows
If the EM is running as service (which is normal): Open a command prompt as administrator and issue the command sc control IScopeEM 255 (IScopeEM is the default name of the service). The thread dump will be appended to the service log file logs/EMService.log. For processing in any thread dump analysis tools you will have to remove the wrapper prefix added to every line.

Example output of the command:

C:\usr\sap\ccms\apmintroscope>sc control IScopeEM 255

SERVICE_NAME: IScopeEM

         TYPE               : 10  WIN32_OWN_PROCESS

         STATE              : 4  RUNNING

                                 (STOPPABLE, NOT_PAUSABLE, ACCEPTS_SHUTDOWN)

         WIN32_EXIT_CODE    : 0  (0x0)

         SERVICE_EXIT_CODE  : 0  (0x0)

         CHECKPOINT         : 0x0

         WAIT_HINT          : 0x0

Using jmxremote or SAP JVM is an alternative, but requires extra configuration.

If the EM is launched interactively (Introscope_Enterprise_Manager.exe) you can hit Ctrl-Break to trigger a dump. Unfortunately you get it on the console, difficult to capture.

Unix
On OS level identify the process ID of the Enterprise Manager, e.g. via the command ps -ef | grep java.
Send signal 3 to this process: kill -3 pid.
The thread dump will be written to the file em.log in the EM installation directory

Collect Heap Dumps from the Enterprise Manager

If the EM crashes and you find errors like "JAVA heap space out of memory" in the IntroscopeEnterpriseManager.log may have either configured the EM with low Java heap size (then see "Adapt Enterprise Manager Java VM parameters" above to increase the heap size) or there could be a memory leak. For the latter heap dumps are needed for analysis. To get heap dumps in case of an OutOfMemory set the additional java parameter-XX:+HeapDumpOnOutOfMemoryError (see "Adapt Enterprise Manager Java VM parameters" above; a restart of the EM is needed). Setting this parameter automatically a heap dump is triggered if an OutOfMemory occurs. The heap dump could consist of more than one file and is usually named java_pid<pid>.*.

In case you want to trigger heap dumps manually set java parameter-XX:+HeapDumpOnCtrlBreak. With every triggered thread dump (see "Collect Thread Dumps from the Enterprise Manager") you then also get a heap dump.
NOTE: heap dumps get a big as the Java heap size is configured. So make sure you have enough disk space available.

Performance / Scalability / Capacity

The EM provides several self-monitoring metrics about its internal status. The metric Overall Capacity (%) is also reported for each EM in theSAP Solution Manager Administration work center. A value of 100% means that it has reached its capacity limit. Short peaks beyond 100% are not critical. Only if the capacity is permanently substantially beyond 100% you will experience problems like missing data points, poor performance, etc. You can find this Overall Capacity (%) metric in the Introscope Investigator via this path: Superdomain -> Custom Metric Host -> Custom Metric Process -> Custom Metric Agent -> Enterprise Manager -> Overall Capacity (%).

A more detailed view is available in the Introscope Investigator tree via the following path: Superdomain → Custom Metric Host → Custom Metric Process → Custom Metric Agent → Enterprise Manager. If you select the nodeEnterprise Manager in the tree, the tabOverview on the right side displays eight charts on EM sanity, as shown in the screenshot. It reveals heap utilization, number of metrics and agents.


Perflog Analysis

The EM collects Metrics about itself and reports them to a file called perflog.txt under 15 second time periods. By analyzing the entries in this file, it is possible identify beforehand if the EM is becoming overloaded and avoid future issues. The EM Perflog Analyzer performs this analysis automatically and provides the user with recommendations on each point checked.

EM Perflog Analyzer Tool

Java Bytecode Adapter (BCA)

Introscope Host Adapter (wilyhost)

Check if Introscope Host Adapter is connected to the Enterprise Manager

Check Introscope Host Adapter connection to EM

Missing ABAP metrics

Missing ABAP metrics

Log Files and Log Levels

Relevant logfiles for problems with Introscope Host Adapter are:

  • SMDAgentApplication*.log (most content here)
  • SMDSystem*.log (in case wilyhost does not start up at all)
    They are persisted in directory /usr/sap/<SMDAgentID>/<InstanceNo>/SMDAgent/log

The log level for SMDAgentApplication.log might have to be increased to DEBUG for troubleshooting. This is done via diagnostics agent administration.

If the Introscope host adapter does not show up in the Investigator the filejvm_SMDAgent.out may also be relevant. It corresponds to IntroscopeAgent*.log of the BCA. To increase the log level for the Introscope runtime of the host adapter (injvm_SMDAgent.out) edit the application resource "IntroscopeSapAgent.profile" and change the property

log4j.logger.IntroscopeAgent=DEBUG, console

Thread Dumps

Thread dumps for the Introscope host adapter means thread dumps for the whole diagnostics agent. They can be triggered in various ways, e.g.:

  1. Via the SAP Management Console / MMC: Right-Click on the java process for the agent and clickDump Stack Trace
  2. On OS level (Unix): find out the process ID of the agent process and issue the command kill -3 <pid>

In any case the thread dump is written to the file std_smdagent.out in the work directory of the diagnostics agent.

Ca Application Performance Management Solutions for Sap

Source: https://support.sap.com/en/alm/solution-manager/expert-portal/introscope-enterprise-manager.html