FAQ: RTI QUESTIONS ANSWERED


1 Features, Requirements and Compatibility

1.1 Does RTI require any additional software to work?

1.2 What platforms does RTI support?

See the System Requirements page for the most current information.

1.3 Does RTI run on OpenShift?

Yes! There is an RTI cartridge for JBoss running in OpenShift. Contact OC Systems for the download details.

1.4 Does RTI run in the Amazon cloud?

Yes! We do most of our testing there. There are no special considerations necessary, so if you're having trouble please ask.

1.5 What applications can RTI monitor?

See the System Requirements page for the most current information.

We’re constantly expanding our capabilities so contact us if you need RTI for additional applications and/or middleware such as WebLogic and WebSphere. We're all software people at heart and occasionally the documentation lags behind the development. Your best bet is to ask if you don’t see what you need: rti_support@ocsystems.com.

1.6 Does 'Enable Deep Dive Trace' mean, 'Start tracing every method now'?

No. It means, "Start logging detailed per-call information on the methods already being monitored."

1.7 Does RTI support 64-bit applications on Windows?

Yes. The RTI Enterprise collector automatically handles either 32- or 64-bit Java applications. The RTI Browser collector only supports 32-bit applications, but the default Windows Chrome, Firefox and Internet Explorer programs are 32-bit, so they should just work. However, the RTI IE collector won't work on 64-bit Internet Explorer. Contact us if you need that.

1.8 Can I use RTI Enterprise on both 32- and 64-bit applications on Linux?

Yes. RTI automatically recognizes whether 32- or 64-bit Java is starting up and uses the appropriate libraries.

1.9 Does RTI support WebSphere or WebLogic?

Not at this time (Version 3.4.4, May 2014); we're focusing on JBoss Enterprise middleware for now, including JBoss, Fuse, Wildfly and Tomcat.

1.10 How does RTI address the following use cases?

1.10.1 Use Case 1 - 503 errors

Application is noticing 503 errors. We need to collect the appropriate data to trouble shoot the issue and identify whether Application Issue or Infrastructure Issue or both.

RTI deep dive will mark failed transactions (503) and trace the transactions through the servlet infrastructure and application code with appropriate custom EDFs. The trace will indicate the call path of the failure. Resource shortages are indicated by collected JMX metrics. DB issues may be determined from JDBC results.

1.10.2 Use Case 2 - Not meeting response-time SLA

Application is seeing slow performance and we are not able to meet the SLA from a performance point of view.

RTI deep dive will trace the transactions through the server infrastructure and application code with appropriate custom EDFs. The trace will indicate the call path of the transactions and the elapsed time in each segment. Resource shortages are indicated by collected JMX metrics. DB issues may be determined from JDBC results.

1.10.3 Use Case 3 - Very high resource utilization

Availiability of the Infrastructure is dropping for example there was high threads or low memory reported by our monitoring tools.

RTI lightweight profile gives an overview of the performance of all transactions/requests in an application. RTI deep dive traces transactions through application code and indicates where time is spent. RTI deep dive collects JMX metrics about thread count, heap usage, garbage collection cycles. RTI takes periodic thread dumps and provides a complete thread analysis perspective.

1.10.4 Use Case 4 - Identify Problem Transaction

We need to identify which flow with an transaction is slow (for example if we have ecommerce application where we do have search, add to cart) and what is the root cause.

RTI lightweight profile gives an overview of the performance of all transactions/requests in an application and shows which transactiosns are slow. From the lightweight profile you can launch deep dive trace to find out where transactions are slow.

1.11 What monitoring is provided by RTI out of the box?

RTI monitors most frameworks used with JBoss, Tomcat and Fuse. A summary is provided in the user's guide section What Does RTI Monitor?.

The complete list of Event Definition Files (EDFs), which are organized by framework or functional area, is shown in the RTI Console's Event Editor view as described in Configuring which Java EE Frameworks are Monitored. When you select a given EDF from the list, the classes and methods covered are shown on the right.

Independent of the these frameworks, RTI also collects basic transaction metrics, detailed JMX metrics, and thread dumps. Lastly, users can create their own EDFs and custom probes for their application or other frameworks, or commission these from OC Systems.

1.12 Is there a probe to count open socket file descriptors?

Not built-in, but we can (and did) write a custom one that you can inject yourself. Download the SocketCount probe for RTI 3.4.2 and above, unpack it where you run your RTI Console, and follow the directions in the README.txt. The source is included: check it out and you might get ideas of other probes you or OC Systems could write.

1.13 What is the best method to find memory leaks using RTI?

RTI can help you tell if you have a memory leak, but not where it is. RTI collects Java heap and garbage collection information continuously and that information can be retrieved from the host with the RTI console by transferring deep-dive performance data (with or without an active deep-dive trace). You can then examine the Java heap and GC information in the GC chart. If the Heap Used metric is steadily increasing and/or there are a large number of major GC cycles (lots of long vertical bars), you may have a memory leak.

At this point you need to switch to a tool specific to Java heap tracking, like Eclipse Memory Analyzer (MAT). With MAT you can analyze Java heap dumps and determine which objects/classes are using heap memory, and which are potentially leaked.

1.14 What are the version compatibility dependencies between the different RTI components?

When stepping up from an older to a newer version, you should always update the components in the following order:

  1. RTI Enterprise
  2. RTI JON/RHQ Agent Plugin,
  3. RTI Console
  4. RTI Browser Collector
That is, the version of RTI Enterprise should always be the same or newer than the (optional) JON/RHQ Agent plugin, which in turn should be the same or newer than the RTI Console you're using to connect to your servers. (The RTI Browser Collector version is generally not so important, but it shouldn't be newer than the others unless explicitly directed by RTI Support.)

If you're having problems and you think it might be due to version mismatches, don't be shy: please ask us!


2 Installation

2.1 Where do I download from?

You can request a free trial on the download page.

2.2 What do I install?

These might all be the same machine. You should have gotten an e-mail with a link to a web site describing all this about the currently available version. If not, go to our Download page and fill out a form, or contact us.

2.3 How do I install?

It's all explained in the User's Guide.

2.4 How do I install the license key you just e-mailed me?

A license key is a single line with no trailing whitespace ending in a newline. Put that single line into the file $RTI_HOME/cfg/cookie on Linux or %RTI_HOME%\cfg\cookie on Windows. You can verify it with the RTI command rti check_license

2.5 How do I Uninstall?

There are four aspects of RTI to consider:

Each of these is described in the user's guide.

2.6 How do I run RTI command-line commands on Windows?

Run "Start→All Programs→RTI→RTI Enterprise Cmd" which opens a 'cmd' window that is set up to run rti commands. You generally won't be needing this beyond the "rti edit" command described in the user guide, but the command rti 2>&1|more gives usage for all the commands.

2.7 How do I run RTI command-line commands on Linux?

In a bash-compatible shell window, do
. /opt/ocsystems/rti/ee64/setup where "/opt/ocsystems/rti/ee64" would be replaced by whatever your RTI installation directory ($RTI_HOME). You generally won't be needing this beyond the "rti edit" command described in the user guide, but the command rti help gives usage for all the commands.

2.8 Can I have multiple JBoss installations on the same machine instrumented with RTI? How?

Yes you can. Simply run the "rti edit" command on each one, naming its JBOSS_HOME value, for example:
rti edit -t jboss /opt/jboss-soa-p.4.3.0/jboss-as
or rti edit -t jboss /opt/jboss-6.0.0.Final

2.9 Can I share the same RTI installation across multiple Linux machines?

No, this is currently not supported. The architecture of RTI assumes one or more collectors unique to each host machine. If this presents a problem, it may be possible to hand-edit some configuration files to get around this limitation: please contact rti_support@ocsystems.com.

2.10 Why do I get "command not found" when I do run "rti-enterprise-3.4.1-linux-x86.installer"?

If you are logged in as root, "." (the current directory) isn't generally in your PATH. Precede the installer name with "./" or "sh ".

2.11 When I run ./rti-enterprise-3.4-linux-x86.installer I get 'Permission denied.', but I'm signed in as 'root'?

When you downloaded the installer it may have lost its execute permission. Run it as: sh rti-enterprise-3.4.1-linux-x86.installer.

2.12 How do I install the rshd service on Windows if I skipped it during installation?

The easiest way is to re-install just the RTI Enterprise component on that computer. However, if that seems like overkill or you don't have the installer any more, you can do it manually as follows:

  1. Open Start→All Programs→RTI
  2. Right-click on 'RTI Enterprise Cmd' and select 'Run As Administrator'
  3. In the cmd window that opens, execute this exact command (that's an upper-case P at the end).
  4. The last command should show the service running. You may have to add rshd program to your Windows or other firewall or open up port 514 in anti-virus software.

2.13 How to fix the RTI error on Windows startup after upgrading Java?

The program that shows the "RTI" icon is an Eclipse RCP program like the RTI Console, and requires a 32-bit Java installation. It selects the path to this at installation. If you delete that Java installation it will cause the "RTISystray" application to fail. The easiest way to fix this is to download and install a new RTI Browser -- the installation process will find a new Java if one is available. However, this requires that you close your browser windows. Alternatively, you can change the Java path being used by editing the file C:\Program Files (x86)\OC Systems\RTI\Browser\RTISysTray\RTISysTray.ini as administrator, and changing the path shown there to a similar path to a 32-bit Java executeable or jvm.dll that exists.

2.14 How do I add new RTI EDF files (provided by OC Systems) to my installation?

For simplity, say RTI_HOME=/opt/ocsystems/rti/ee64/ -- replace $RTI_HOME with your actual value below.
Then:

  1. Log into your enterprise machine as a user with write-access to the RTI installation (e.g., root or Administrator)
  2. Copy the new or updated RTI*.edf files here to $RTI_HOME/probes/, overwriting the existing files there.
  3. Execute the following RTI command-line command to push those out to your application collectors:
    $RTI_HOME/bin/rti server update_collectors

The procedure is the same for Windows except with the obvious path changes.

2.15 MacOSX says RTIAnalyzer is damaged and can't be opened. Help?

This is a "feature" of OSX 10.8 and newer to fight malware. See MacOSX Gatekeeper and "Damaged" application in the user's guide for the workaround.

2.16 How do I do a "silent" Linux RTI Enterprise installation?

On Linux you can do silent installs using the -S option, explicitly defining the three values RTI_HOME, RTI_DATA, and JAVA_HOME on the command line.

For example:

$ sh rti-enterprise-3.3-linux-x86.installer -S \
  RTI_HOME=/opt/ocsystems/rti RTI_DATA=/var/opt/ocsystems/rti JAVA_HOME=/opt/jdk1.5.0

You will also need to enable RTI in the JBoss servers for each JBoss installation with:

$ . /opt/ocsystems/rti/ee64/setup
$ rti edit -t jboss /opt/jboss-as-7.1.1.Final 

Then the JBoss server will have to be restarted to pick up RTI.

If/when you have an RTI subscription license you should add that to the RTI installation and restart the JBoss servers to pick up the license.

3 Enabling and Connecting

3.1 Why does this command fail: rti edit -t jboss /opt/jboss-soa-p.4.3.0?

The given path must be the parent of the bin directory containing the "run.sh" script, for example:
   rti edit -t jboss /opt/jboss-soa-p.4.3.0/jboss-as

3.2 I installed RTI Enterprise on a host, and I can connect from the RTI Console through JBoss ON to that host--why can't it find the RTI Enterprise installation?

It may be unable to locate the installation's RTI_HOME. You can explicitly tell it where RTI_HOME is as follows:

3.3 When starting our app server with RTI using Java7, we get an error like, ERROR ... java.lang.VerifyError: Expecting a stackmap frame at branch target 60. Is there a fix?

This is a problem introduced by the Java7 bytecode verifier. If you are seeing this it is probably because you are running in domain mode rather than launching with standalone.sh. In this case you have to add the Java option -XX:-UseSplitVerifier to those already added to enable RTI, and restart the application server.

3.4 How can I keep RTI from being applied to my host controller and domain controller processes when running JBoss in domain mode?

This is apparently a side-effect of specifying the RTI-enabling Java options within the server-group definition. As of version 3.4.3 we recommend making the domain-mode changes to host.xml file as described in the current user's guide.

3.5 We launch multiple instances of JBoss from the same installation: how can we force each instance to have a different RTI Collector?

The easiest way is to define the RTI_COLLECTOR_ID environment variable to be the desired collector name, for example, for JBoss standalone mode:

# RTI_COLLECTOR_ID=jboss_users $JBOSS_HOME/bin/standalone.sh -Djboss.server.base.dir=$JBOSS_HOME/users \
    > jboss_users.log 2>&1 &
# RTI_COLLECTOR_ID=jboss_warehouse $JBOSS_HOME/bin/standalone.sh -Djboss.server.base.dir=$JBOSS_HOME/warehouse \
    > jboss_warehouse.log 2>&1 &

For JBoss domain mode, specify a different rti.collector.id property value in the jvm-options for each server definition, as shown in the current user's guide.

3.6 How do I enable RTI for Fuse Service wrapper?

This is described in the user's guide, and requires manually editing a properties file as for JBoss domain mode.

4 JBoss ON and RHQ

4.1 When I connect from the RTI Console to a host through JBoss ON, why does it gives a message that the RTI license expired?

This probably means it can't find the rti installation at all. See the next question.

4.2 How can I get more information about why JBoss ON/RHQ isn't finding RTI resources?

Enable JBoss ON agent debugging, force manual auto-discovery on the platform root node, then look in the agent log file for more information. See the next two questions for how to do this.

4.3 How do I enable JBoss ON/RHQ Agent debugging?

You can enable agent debugging by selecting the Inventory view, Resources→Platforms>→your platform)→RHQ Agent. Then right-click, choose Operations→Set Debug Mode, then choose "Yes" for both Enabled and Trace Messaging Parameters, then click Schedule button at the bottom of the Create New Operation Schedule view.

4.4 How do I force manual auto-discovery on a JBoss ON platform?

This is described in the user's guide.

4.5 OK, it's still not finding RTI, now what?

Perhaps the RTI plugin in in your remote JON/RHQ agent can't find the local RTI Enterprise installation, because the magic files it looks for are missing or have incorrect contents. The quick fix is to
  sudo $RTI_HOME/bin/update_info_file.sh
on each machine to create the correct file(s). Here is the explanation:

There are two magic files that point to the current RTI installation for a given host: There is a user-specific one and a global one, $HOME/.ocs-rti-info and /var/opt/.ocs-rti-info respectively. In this case, the "$HOME" value is the home directory for the user who is running the JON/RHQ agent, perhaps root or jonuser. So if you installed RTI Enterprise on that machine as some other user, with a different home directory, the "wrong" $HOME/.ocs-rti-info file would have been created/updated. But there's still the global one. But again, if the user you installed as doesn't have write access to /var/opt/, then that file wouldn't have been created/updated either. Hence the command above, to cause the global one to be created.

If you think these files are right, or our explanation wasn't clear, contact us.

4.6 Are there any unified views in the JBoss ON Dashboard that show RTI information from all the platforms?

No. For now you have to look at each platform separately, by clicking Inventory, Resources, Platforms, then choosing the platform (host) of interest. However, the dashboard is user-customizable as described in the JBoss ON documentation.

4.7 Can I create alerts from the RTI Console?

No, the rti console is read-only with respect to JBoss ON configuration. See the user's guide for how to create alerts in JBoss ON.

4.8 When defining an Alert Notification, I choose RTI Alert and get a small dialog. I click OK, but apparently no notification was defined?

It takes a few seconds to construct the RTI Alerts dialog. Unfortunately, it presents its OK button before it other controls appear. Wait until you see a big dialog with a number of Properties before proceeding.

4.9 In JBoss ON I see RTI Enterprise and 28 days left on license but when I click on the RTI Enterprise link to expand no RTI Collector resources show up. What am I missing?

You have RTI Enterprise and the accompanying JBoss ON plugins successfully installed. Now you must:

See the user's guide.

4.10 How can provide RTI console access to the JON/RHQ server but restrict permissions as much as possible?

See the user's guide section Configuring JBoss ON or RHQ to Allow RTI Console Access.

4.11 Is it possible to change the Platform names assigned by JON so they have a meaningful name in the RTI Console?

Yes! In the JON/RHQ Platform view, there's a ">" (tree expander) decoration next to the platform name/Title, right above the row of tabs. If you click that, it expands to show many properties of the platform. (This actually works for any resource you're currently viewing.). The 'Name:' property is editable -- if you hover over it an Edit operation will appear. Click that, change the name, click the checkmark to save your change. That's it.

Now, if you refresh the changed platform in your RTI Console, it will get the new name.

4.12 How can I do periodic or scheduled RTI Alert profiles/traces in JON?

RTI profiles and traces can be initiated and collected from RTI Alerts. Normally, these RTI Alerts are triggered by metric values, but RTI Alerts can be triggered when JON operations execute. JON operations can be scheduled to execute at specific times and to execute periodically. Combining these two features you can schedule RTI Alert profiles and traces.

First, schedule an operation for a specific time or a periodic repeat. You can use one of the RTI resource NOOP operations, which do nothing, as a the triggering operation, as shown here:

Then create an RTI Alert with the condition set to Operation Execution, the Operations set to the schedule operation from earlier, and results status SUCCESS, as shown here:

Then define the RTI Alert to perform the desired action: a timed trace and snapshot, a timed profile and snapshot, a profile snapshot only, or a trace snapshot only.

The RTI resources have three NOOP operations defined by default. If you need more NOOP operations you can extend the RTI JON agent plugin XML to add more. The operation names must start with the string "noop" to be considered a NOOP.

      <operation name="noopRti3"
                 description="No operation 3.">
         <results>
             <c:simple-property name="stderr" description="The stderr output of the command execution." />
         </results>
      </operation>
Contact RTI support if you need help with this.

Since RTI version 3.4.3.

4.13 Some RTI Console operations like Inject take a long time, and then time out. What happened?

If you're connected through JBoss Operations Network or RHQ, long-running commands are started by the Agent, which then waits for a result. If the result doesn't come with about 1 minutes, the agent will "time out" so as not to hold up other operations. This does not cause the server-side operation to abort, but means that the RTI Console won't get a result or error messages. Nearly always the operation will succeed if given another minute or so before assuming the result. If you're impatient you can edit the RTI_DATA/log/rtilog.txt file on the server to see if the operation (e.g., 'restore_collector' for an Inject) has succeeded.

4.14 How can I reduce the CPU load of RTI metrics on the JON agent?

Simply put, you need to reduce the quantity of RTI metrics flowing through the Jon agent, which can done in several ways:

Reducing the frequency of metric collection reduces the number of metric values passing through the JON agent, but impacts the delay between metrics showing a performance problem and the initiation of an RTI alert to collect performance information. This is the least desirable option. This method requires an adjustment to both the metrics collection schedule on the JON server, and the metric production schedule in the RTI collector configuration using collector→Edit Configuration→BTO Commands→METRICS.

Reducing the number of different transaction types will also reduce the number of metric values. By default, RTI will filter out requests for typical content types (JPEGs, GIFs, XML, Javascript) because they are generally not the source of performance problems. If you can filter out "uninteresting" requests and focus on key application requests (like Grails/Groovy dispatch) this could reduce the load on the JON agent significantly (using the Edit Configuration -> BTO Commands -> EXCLUDES).

Reducing the RTI metrics collected can reduce the load on the JON agent without affecting alerts. Several RTI metrics can be disabled and some others can have their collection frequency reduced. The greatest impact will be from adjusting the RTI metrics for RTI Transaction resources, because there are many more RTI Transactions than RTI Collectors.

For RTI Transaction metrics, the following adjustments can be made:

Metric Adjustment
responseTime Disable this calltime metric and use averageResponseTime instead.
type Reduce the collection interval to once a day or less frequently.
total No change.
errors No change.
averageResponseTime No change.

For RTI Collector metrics, the following adjustments can be made:

Metric Adjustment
transactionsResponseTime Disable this calltime metric and use averageResponseTime instead.
collectorType Reduce the collection interval to once a day or less frequently.
transactionsTotal No change.
transactionsError No change.
averageResponseTime No change.
events Change to once per hour.

The RTI Transaction metrics changes should reduce metric load to one third (3 metric values per minute instead of 9 values), and a similar reduction for the RTI Collector metrics.

Local tests of JON agent metric performance show the following performance:

The test collected all 4 RTI metrics (including the calltime) for each RTI transaction, every minute which is 20,000 metrics per minute (which equates to 45,000 metric values). The Agent Measurement Subsystem reported < 5 seconds per minute spent collecting metrics (which corresponds to CPU monitoring) with no late collections. The average CPU load for the JON agent was < 2% with one or two spikes at 90-100% CPU for 1 second each (or spread over two seconds at 50-60% each). There were about 3 intervals where the CPU usage is about 20-30% over the 30 minute test period.

4.15 What RTI Versions support what JON and RHQ versions?

As of July 2014:

RTI Version Release Date RHQ Version(s) JON Version(s) Notes
3.3.2 Apr 2012 4.1,4.2 3.0.0, 3.0.1 initial RHQ/JON support
3.3.2C Jun 2012 4.1-4.4 3.0.0/1,3.1.0 added RHQ 4.3,4.4, JON 3.1.0
3.4 Jan 2013 4.1-4.4 3.0.0/1,3.1.0 same as 3.3.2C
3.4.1 Jul 2013 4.1-4.8 JON 3.0.0/1,-3.1.0/2 added RHQ 4.5-4.8,JON 3.1.2
3.4.2 Sep 2013 4.1-4.8,4.9 JON 3.0.0/1,3.1.0/2 added RHQ 4.9
3.4.3 Feb 2014 4.4-4.8,4.9 JON 3.1.1-3.2.0GA Note 1 below
3.4.4 Apr 2014 4.4-4.8,4.9 JON 3.1.1-3.2.0GA no additional versions
3.4.5 Aug 2014 RHQ 4.4-4.8,4.9,4.12 JON 3.1.1-3.2.0GA added RHQ 4.12

Note 1: Version 3.4.3 added JON 3.2.0ER, 3.2.0GA. From this point on each JON release is explicitly incompatible from the RHQ release it is based on. To reduce product size, we removed support for RHQ 4.1 and JON 3.0.0, 3.0.1 but those are available upon request as plug-ins to the RTI Console.

5 Collectors

5.1 How do Enterprise Collectors get created?

The Java-based enterprise collectors that are started by scripts--currently JBoss, JMeter, SoapUI?, and Tomcat--are "enabled" for RTI by edit these scripts, after which collectors are automatically created or activated as-needed whenever the script is run. Thus, the way to create a script for these is to choose a host and collector type, "edit" the script, and then (re)start the application.

So you need to:

That will edit the catalina.sh script to call rti_enable_tomcat.sh. Then you need to restart Tomcat and it should create a collector (or use the one you had previously) and you should then see processes and data:
rti get_collector_processes -t tomcat
rti list_collector_data -t tomcat
or you can run the console and it will issue the same kind of commands to update the Tomcat collector node.

Just to continue some more, for all the collector kinds except java and rcp, we have to do this "enable" step that consists of editing the start-up script for the tool to call the appropriate $RTI_HOME/bin/rti_enable_*.sh/bat script. That can be done: automatically at installation on Linux (and in the future maybe Windows) if the env variables are set up and the user opts to, from the console using the Enable page in the New Collector wizard. manually from an RTI shell with the "rti edit" command,

It's spread out like this because the user must have privilege to edit the tool scripts and that may not be possible for the console user. In those cases they will have to get privilege or do it on the server manually or during installation.

After doing the enable/edit the user will have to restart any running tool processes to get data. This is another reason they may need to do the operation on the server with privilege.

5.2 Why do I get a permissions error when trying to create a JBoss collector from the RTI Console?

There are two operations done when you create a JBoss (or other Java-based) collector: creating the collector objects within RTI, and enabling the JBoss application. This second step, enabling JBoss, requires write access to JBoss application files. That's why we recommend you enable JBoss as a separate step, from the command line, as a JBoss administrative user, as described in the user's guide.

5.3 How do I apply RTI to my Apache httpd web server?

This capability is provided for Linux by the "mod_arm4" module and is described in the text file $RTI_HOME/mod_arm4/README.

5.4 Is it possible to restrict a tracing to a "white list" of incoming URLs?

The mod_arm4 plugin for Apache httpd can be used to provide a white list capability:

That will allow tracing for listed IP addresses and not trace all others. Contact rti_support@ocsystems.com for help in setting this up.

5.5 Where does RTI Enterprise create temporary files?

Version 3.4.1 of RTI manages temporary files differently from previous versions. Older versions of RTI stored temporary working files in the default temporary directory for that platform ($TMP for Linux and %TEMP% for Windows). RTI now places (almost) all the temporary files in $RTI_DATA/tmp. A few small temporary files may still be placed in the default platform temporary directory, but the space requirements are very small.

5.6 What metrics can we collect if we enable Apache (httpd) collecting in RTI?

If you want metrics about the Apache web server itself, you may be able to get these from JBoss operations network. Use RTI plus the modarm4 module for Apache httpd to provide deep-dive tracing (not lightweight-profiling) of all transactions that go through the web server.

5.7 How do I use RTI with JMeter?

If your JMeter is on your desktop where you run your RTI Console, then you can enable it directly from the RTI Console as described in 1.3.3.4 Enabling RTI Using the Console Wizard (just choose 'jmeter' instead of 'tomcat' as the collector type in section 1.3.3.4.2).

If your JMeter is remote relative to your RTI Console, then you probably need to use putty to log in directly to the machine where JMeter is running, as a user with write-access to $JMETER_HOME/bin/jmeter.sh, and use rti edit -t jmeter .... as described in the user's guide section 1.3.3.2, "Enabling RTI Through the Command Line".

Either way, once JMeter is enabled for RTI, it will trace RTI transactions whenever you run it.

NOTE: Most users do notapply RTI to JMeter: they just enable their web- and application servers for RTI, then drive their tests with JMeter (or a similar) tool as before. The reason to enable JMeter with RTI is to force every transaction sent by JMeter to be deep-dive traced. Though there are ways to fine-tune the tracing, for load-testing and longer-running test cases this generally collects a lot more data than you want.

6 RTI Console

6.1 How do I copy a file to or from the RTI Console's Navigator view to a Windows folder?

When the RTI Console is first started, there is a shortcut created on the desktop label "RTI Drop Box" that points to the RTI Drop Box/RTI Data folder in the Workspace View at the lower left of the console window. This is an easy way to access files from both within the RTI Console and your Windows Explorer. In the absence of the RTI Drop Box link.

You can Copy out of Workspace Drop Box by copy and paste:

Copy into the Workspace Drop Box by drag and drop: Copy/paste as above will work, but this is easier. Simply select the file in the Windows Explorer, and holding the mouse down, drag it onto the folder (e.g., RTI Data) shown in the Workspace view.

6.2 Why does Linux RTI Console crash with a JVM core dump?

Check that your DISPLAY environment variable valid is valid. If Firefox can work, then RTI Console should also.

6.3 Why does Linux RTI Console context help (opened by ? buttons) launch a browser window instead of appearing within the app like it's supposed to?

Probably because the installed version of xulrunner is newer than that supported by the version of Eclipse SWT that RTI is built from (3.7/Indigo) See http://www.eclipse.org/swt/faq.php#browserlinux

6.4 How do I fix Linux RTI Console Help not displaying due to error "Firefox is already running but is not responding."

The RTI Console is Eclipse-based, and uses the Eclipse help system. On Linux this requires a Firefox browser. If Firefox is running you will need to stop and restart it. If it's not running, delete ~/.mozilla/firefox/*.default/.parentlock file.

6.5 How can I improve the performance of the RTI console?

The performance and capacity of the RTI console is directly proportionally to the size of the Java heap the console uses. The 32-bit RTI console defaults to a 1GB heap and the 64-bit version defaults to a 2GB heap. These default sizes can be adjusted locally as described below. However, making it larger doesn't necessary help: It is very important that the Java heap fit in system memory and not be paged, or performance will be severely impacted.

The following table summarizes Java heap size, required free memory, and trace capacity for the RTI console:

Java Heap Required Memory RTI Events RTI Transactions
1GB 1.25GB 750,000 7,500
2GB 2.25GB 1,500,000 15,000
3GB 3.25GB 2,250,000 22,500

Note that the required memory is about 250MB above the heap size. RTI events correspond to method calls traced, and RTI transactions composed of RTI events. RTI transactions vary in complexity based on the system traced, and so the number of RTI events per transactions can vary between 10 to 200, on average--a value of 100 events per transaction was used in the table above.

The performance of the RTI console can be improved by closing tabs that are needed, particularly deep-dive trace tabs. In general you can increase performance by closing unneeded deep-dive editors which tax the Java heap. Closing all views (other than the Welcome view) works best, as it also disposes of data shared by multiple views.

You can track the Java heap usage by the RTI console by enabling the heap status as follows:

Windows→Preferences→General→Show heap status

This will display a heap status widget in the status line of the RTI console (along the bottom). The heap status shows the total Java heap used and the current size of the heap. The Java heap may grow in size up to the maximum specified (1GB for Windows and 32-bit Linux, 2GB for others).

You can increase (or decrease) the available heap by closing the RTI Console and editing the rti-console-install-path/RTIEEConsole/RTIAnalyzer.ini file to change the size specified in the -Xmx option there. For example, if you have 8MB of RAM on your machine and it says -Xmx2048M
you can change that to -Xmx4096M
to double the available memory.

6.6 Why does "Open Deep Dive View" open the wrong view sometimes?

This is a glitch in the Eclipse Windows UI with selecting items in the navigator. I open dataset A by selecting it under Saved Trace Data. Then I select on dataset B, choose 'Open'. and it re-opens A. If I double-click on B it opens correctly. We can't figure it out either, so recommend using double-click here.

6.7 In the Linux RTI Console, 'Summary' seems to do nothing, and Context help (? button) operations send the HTML output to a separate browser rather than in the help panel. Is there a fix for this?

Yes, this is a known bug in eclipse on newer Linuxes. The workaround is to install WebKitGTK and then tell Eclipse's SWT to use that. If you're using 64-bit RTI Console (rti-console-*-lx86_64-*) then:

  yum install webkitgtk.x86_64
  SWT_GTK3=1 /RTIEEConsole64/RTIAnalyzer
or else if you're using 32-bit RTI Console (rti-console-*-lx86-*) then:
  yum install webkitgtk.i686
  SWT_GTK3=1 /RTIEEConsole/RTIAnalyzer

6.8 Is there a way to give a better name than the raw hostname or IP address to a host in the RTI Console?

Yes: Select a host, right-click, choose Properties. In the Properties dialog change the "Description" to what is wanted, then click OK. Then right-click and Refresh the collector: The new Description will then appear instead of the original hostname for as long as that host is defined in your local RTI Console workspace.

6.9 Why can't the RTI Console (or JON) find RTI on a host after it successfully connects there?

See OK, it's still not finding RTI, now what?.

6.10 Can an RTI Console user connect directly to a linux server without having a login account there?

No. The two modes are ssh login or through JBoss Operations Network. You could have multiple users connect using the same login account dedicated just for that, if your security procedures allow this.

6.11 What's the best way to verify that the RTI Console is working in my environment?

These are covered in the Quick Start Guide and the RTI User Guide. If these operations work, then you should be in pretty good shape.

6.12 On Windows, the RTI Console sometimes becomes unresponsive. What do I do?

Use the X or context menu in the RTIAnalyzer window frame to close it. When prompted, click 'Close the Program' and when Windows says, "Searching for a solution..." click "Cancel". This can happen if RTI runs out of memory (see How can I improve the performance of the RTI Console. However, there also is an intermittent deadlock in the Windows RTI Console related to 'Edit Configuration' which we've as yet been unable to isolate.

6.13 How can I change the JVM used by the Windows RTI Console?

If you already have a 64-bit Windows JRE in your PATH but for some reason the RTI Console isn't finding it, OR you want to try a different JVM for some reason:

  1. Using "Run As Administrator" edit the file C:\Program Files (x86)\OC Systems\RTI\Console\RTIEEConsole\RTIAnalyzer.ini (the path above might be different if you changed it when installing)
  2. In that RTIAnalyzer.ini file, insert two lines BEFORE the "-vmargs" line:
    -vm
    C:\Path\to\your\64bit\javaw.exe

6.14 I can connect to my server in the cloud successfully, but when I choose a connected host, the resulting entry is deactivated. How do I fix that?

The $HOME/.ocs-rti-hosts is likely wrong, namingg one or more host(s) that don't really exist. This can happen if the IP address/hostname changed but an old .ocs-rti-info file was preserved. To fix this:

7 Events and Properties

7.1 Why does the order look wrong in the tree table when I sort by Elapsed Time

Sorting is done independently at each level of the transaction tree. So all the Top-level transactions are sorted, then under each of those, it's direct children are sorted, and so forth. So if you look at an expanded tree it can look out of order. You can flatten the tree into a single linear list by doing a filter on "/".

7.2 Why is the Elapsed Time for some erroneous transactions so long?

Some "failed" (light red color) transactions are so marked because no stop event was ever seen. The stop time associated with them is the time at which the tab or window is closed, which could be a long time after it was started. The statistics associated with failed transactions are not generally reliable.

7.3 What is the meaning of the Elapsed Time and Data Length values I see associated with a shockwave-flash file corresponding to a YouTube video?

This are the load time and size of the actual .swf file. The file may then display an animation or streaming video which will not be measured by RTI, since what you see is not interacting with the browser itself in any way.

7.4 What does 304 means as a value for the "Cached" property?

For cached, 0 means "page requested, not cached"; 1 means "page retrieved from cached, not requested"; and 304 means "request made with 'If-Modified-Since' header in request"; and "304 Not Modified" was returned, so the page was then retrieved from cache.

7.5 How can I obfuscate IP addresses and names in RTI-collected data?

Use the RTI Console to:

innocuous characters, and

7.6 How do I get a complete date/time to show in Excel from RTI-exported CSV file?

RTI allows you to export selected transactions from a transaction tree view in CSV format. When you open the exported .csv file with Microsoft Excel, you see only hours and minutes -- the date isn't visible.

Amazingly enough, Excel does not support automatically formatting a date/time with millisecond accuracy from a CSV file! You need to do a custom format in Excel to properly use the millisecond date/time exported by the Console in the CSV file, as follows:

Click here for a larger view

Now column A gives the date/time with millisecond precision.

7.7 Is there a limit on the length of string values logged in deep-dive?

Yes. Before version 3.4.5, string data such as SQL queries is truncated to 255 characters. Newer versions truncate at 1023.

7.8 How can I see long strings in the deep dive views?

The list views in the deep dive editor show only part of the string. To show the full text of the string (up to the length limit above) you can:

8 Events Editor

8.1 How can I see methods from my application in the RTI call profile and deep-dive trace?

Use the Collector Configuration Editor as described in the user's guide.

8.2 Does RTI support the blotz framework/API?

If you don't see RTI-blotz.edf in the Collector Configuration Editor's "Events" tab, then the answer is probably no. But we might be able to whip something up for you, or provide guidance on how you could add it yourself. Please contact rti_support@ocsystems.com and ask.

8.3 Are there any EDFs that I absolutely should not uncheck?

No: If you really want RTI to do nothing at all in your enterprise application, you can uncheck everything and your application will still work. This is basically what "Disable RTI" does when applied to a host.

But: Do not uncheck RTI-Monitoring.edf unless you want to disable all the lightweight profiling, JMX metrics, JON metrics, and statprof.

In general, you should only disable EDFs if you're getting data about parts of the application that you don't care about.

8.4 When I 'Inject' changes, when do they take effect?

Changes may take effect immediately, periodically, or only on restart, as follows:

8.5 What is the correct way of adding methods while creating a custom EDF file? The goal is to be able to add by package name and NOT by individual methods.

To add by package, you:

If you trace too many methods you will introduce too much overhead to give you very accurate timings. Avoid low-level methods and favor API-level methods.

8.5.1 What happens if I have two wildcards whose expansion overlaps?

If you have, for example:
MONITOR EDGE METHOD OCSTX com.ocsystems.example.ecommerce.*::* and
CHILD METHOD OCSOTHER com.ocsystems.example.*::*
You will get duplicate entries for those methods that match both wildcards, with one appearing as the caller of the other.

8.6 How can I further configure RTI to collect more profile and trace data?

The general mechanism for specifying where RTI collects profile and trace information is to create a custom EDF. This is generally done using the RTI console Collector Editor where you can select specific methods from your application and specify whether the method should be a trace/profile point, and whether it should be a monitoring point. For most situations this process will be quite sufficient, but the RTI EDF file support a number of features which are not accessible from the RTI Collector Editor (see the next question).

To use these features you will need to edit the EDF file manually, outside of the RTI Collector Editor. Actually, you can edit the EDF file as an ordinary text file in the RTI console by saving and opening it as such.

If you want to edit an existing EDF file, first save it to the RTI console workspace:

If you want to create a new EDF file, create it in the RTI console workspace:

The file may be opened in the default system editor which is fine, or you can choose to edit it in the embedded text editor in the RTI console:

The file will be available to edit.

8.7 What advanced directives are available in the EDF files?

The EDF file may contain blank lines, comment lines, and directives:

The most common and useful forms of directive lines are:

The first of these is for selecting application methods for monitoring, tracing, and profiling. Use the MONITOR keyword to specify that the method is a BTM: a business transaction that can produce metrics, and start profile summary trees and deep-dive traces. Use the EDGE keyword to specify that this method should start deep-dive traces unconditionally--this is not very common. Use the CHILD keyword to indicate that the method should be traced only if a caller starts the trace--this is the most common form. The quoted-id specifies the event class that appears in traces. The quoted-method-target specifies to which application methods these actions should be applied--these are generally method signatures but can be wildcard patterns.

The second form above is for specifying that custom probes should be applied when the target methods are executed. These are primarily used for built-in RTI probes for common frameworks. There are several useful RTI probes that you might want to use when investigating application structure:

An example of using the traceback probe is:

CUSTOM "traceback" "package.Class::method()" RtiLogTracebackProbe

The third form is for specifying custom probe beans, which contain sets of cooperating probes. These are primarily used for built-in RTI features.

The quoted-method-target specifications have the following form:

[package[.package...].]Class::method([parameter-type[,...]])

Note that the separator between class and method names is a double colon (::). Wildcards (*) can be used once in the class specification and once in the method signature specification and should be the rightmost character (on each side to the double colon). Be careful using wildcards because they may match many methods, and the RTI runtime limits the number that can be matched in one directive.

8.8 How do I add my own hand-written probe to my collector(s)?

Add a user EDF file in your RTI Console as described in the user's guide, 4.2. Configuring a Collector:

  1. Open the RTI Console application on your desktop
  2. In the Workspace view in the lower left open "Drop Box" folder
  3. Drag your EDF to Workspace>Drop Box>RTI Data OR select RTI Data, right-click, select Import... to use eclipse import wizard
  4. Select an appropriate collector in the Collectors view on the upper left
  5. Right-click and and select Edit Configuration in the context menu
  6. Load your EclipseLink.EDF as described in 4.2.1.2.3. Loading an External EDF
  7. Make sure the EDFs you want to use are checked.
  8. Click Inject at the top of the Event editor window
  9. Select collector(s) to inject the change into, and click OK.
Once this operation completes, the collector(s) should start collecting additional information the next time you run your test.

9 Browser (Chrome, Firefox, IE) Collector Plugins

9.1 Can I install the Browser Collector(S) without the Console?

Yes, and it will collect data from your browsers that can be viewed by other RTI Consoles.

9.2 Can I look at the collected data without using the RTI Console?

The .rti files are XML text, but they're not very readable. Contact us for an rti2csv tool which might help.

9.3 Why didn't my browser collect any RTI Events?

There are a few possible reasons:

RTI Firefox and Internet Explorer Collectors write under a sub-folder in %USERPROFILE%. You can change this behavior by editing %PROGRAMFILES%\OC Systems\RTI\Browser\conf\config.txt and modifying the parameter RING_LOG_DIR to a safe location; for example RING_LOG_DIR=C:\tmp\RTI. Restart Firefox and verify there are now .rti files contained in C:\tmp\RTI.

9.4 Does a top-level browser navigation event such as IE-Navigation always correspond to an interactive user action?

Not always. On pages with sophisticated dynamic content, transactions can be initiated that are impossible to relate back to the last user-initiated transaction that caused them. However, if you have reproducible output which doesn't look right, please report a bug.

9.5 Where did my RTI data from last week go?

In its default configuration, the Browser collector only keeps data for seven days or up to a limit of 32MB. See the next question.

9.6 How do I change how much or how long RTI data is saved?

If you want to keep more data, change the parameters. Here you can change the values of TOTAL_RING_SIZE, RING_SIZE, and KEEP_DAYS, respectively.

NOTE You must restart your browser(s) to pick up the changed configuration.

9.7 What is an "ARM Ring?"

By a "ring" of files, we mean a fixed-size group of files, such that each file is approximately the same size, and the oldest is deleted when creating the newest. There is always one more file than the stated ring size. For example, if ARM Ring Size is listed as 1, there's never more than 1 file in addition to the one being written to. If the Arm Ring File Size is 8388607, then when a transaction is recorded which brings the current file size at or above 8MB, that file is closed. If there's already an existing data file for the current process it is deleted, and then new file with the next sequence number is opened.

9.8 How do I change RTI Browser Collector configuration?

Using the RTI Console, right-click on the Localhost→Chrome, Firefox or IE collector, and choose Edit Configuration... to see the Properties page. Note that the same RTI browser configuration is shared by all three browser "brands" so changing the properties for Chrome changes them for IE and Firefox also.

The properties are are as described here.

9.9 How much data does RTI keep by default?

When you re-start your browser, it deletes data files changed more than 7 days before. For each IE process, the default is to keep 2 8MB data files, or about 10,000 transactions. These settings can be changed in the Properties page for the collector.

10 Reporting Bugs

We've built lots of support into RTI for collecting internal data to help us figure things out, so here's what to do...

10.1 Where do I send bug reports?

Attach all data in any compressed format to an e-mail describing the problem:

10.2 How do I gather data about a bug seen using the RTI Console?

Under the RTI Console, there is Help→Problem Report. Try that first, and send the file it tells you it creates.

10.3 How do I gather data about a bug against the Enterprise collectors?

The RTI Console's Help->Problem Report will gather data about all connected hosts and collectors, but if the problem isnt' with something you're currently connected to, then: