1 Features, Requirements and Compatibility
1.1 Does RTI require any additional software to work?
- RTI Browser tray control feature requires 32-bit Java 5, 6, or 7 to be installed.
- RTI Enterprise requires 32- or 64-bit Java 5, 6, or 7 to be installed.
- RTI Console on Mac requires 32- or 64-bit Java 6 or 7 to be installed.
- You can download Java for any platform from http://java.com/
1.2 What platforms does RTI support?
See the System Requirements page for the most current information.
1.3 Does RTI run on OpenShift?
Yes! There is an RTI cartridge for JBoss running in OpenShift.
Contact OC Systems for the download details.
1.4 Does RTI run in the Amazon cloud?
Yes! We do most of our testing there. There are no special considerations necessary,
so if you're having trouble please ask.
1.5 What applications can RTI monitor?
See the System Requirements page for the most current information.
We’re constantly expanding our capabilities so contact us if you need
RTI for additional applications and/or middleware such as WebLogic and
WebSphere. We're all software people at heart and occasionally the
documentation lags behind the development. Your best bet is to ask if
you don’t see what you need: email@example.com.
1.6 Does 'Enable Deep Dive Trace' mean, 'Start tracing every method now'?
No. It means, "Start logging detailed per-call information on the methods
already being monitored."
1.7 Does RTI support 64-bit applications on Windows?
Yes. The RTI Enterprise collector automatically handles either 32- or
64-bit Java applications. The RTI Browser collector only supports
32-bit applications, but the default Windows Chrome, Firefox and
Internet Explorer programs are 32-bit, so they should just work. However, the
RTI IE collector won't work on 64-bit Internet Explorer.
Contact us if you need that.
1.8 Can I use RTI Enterprise on both 32- and 64-bit applications on Linux?
Yes. RTI automatically recognizes whether 32- or 64-bit Java is
starting up and uses the appropriate libraries.
1.9 Does RTI support WebSphere or WebLogic?
Not at this time (Version 3.4.4, May 2014); we're focusing on JBoss
Enterprise middleware for now, including JBoss, Fuse, Wildfly and Tomcat.
1.10 How does RTI address the following use cases?
1.10.1 Use Case 1 - 503 errors
Application is noticing 503 errors. We need to collect the appropriate
data to trouble shoot the issue and identify whether Application Issue
or Infrastructure Issue or both.
RTI deep dive will mark failed transactions (503) and trace the
transactions through the servlet infrastructure and application code
with appropriate custom EDFs. The trace will indicate the call path
of the failure. Resource shortages are indicated by collected JMX
metrics. DB issues may be determined from JDBC results.
1.10.2 Use Case 2 - Not meeting response-time SLA
Application is seeing slow performance and we are not able to meet the SLA from a performance point of view.
RTI deep dive will trace the transactions through the server
infrastructure and application code with appropriate custom EDFs. The
trace will indicate the call path of the transactions and the elapsed
time in each segment. Resource shortages are indicated by collected
JMX metrics. DB issues may be determined from JDBC results.
1.10.3 Use Case 3 - Very high resource utilization
Availiability of the Infrastructure is dropping for example there
was high threads or low memory reported by our monitoring tools.
RTI lightweight profile gives an overview of the performance of all
transactions/requests in an application. RTI deep dive traces
transactions through application code and indicates where time is
spent. RTI deep dive collects JMX metrics about thread count, heap
usage, garbage collection cycles. RTI takes periodic thread dumps
and provides a complete thread analysis perspective.
1.10.4 Use Case 4 - Identify Problem Transaction
We need to identify which flow with an transaction is slow (for
example if we have ecommerce application where we do have search, add
to cart) and what is the root cause.
RTI lightweight profile gives an overview of the performance of all
transactions/requests in an application and shows which transactiosns
are slow. From the lightweight profile you can launch deep dive trace
to find out where transactions are slow.
1.11 What monitoring is provided by RTI out of the box?
RTI monitors most frameworks used with JBoss, Tomcat and Fuse. A summary is provided in
the user's guide section What Does RTI Monitor?.
The complete list of Event Definition Files (EDFs), which are organized by framework or
functional area, is shown in the RTI Console's Event Editor view as
Configuring which Java EE Frameworks are Monitored.
When you select a given EDF from
the list, the classes and methods covered are shown on the right.
Independent of the these frameworks, RTI also collects basic transaction metrics, detailed JMX metrics,
and thread dumps. Lastly, users can create their own EDFs and custom probes for their application
or other frameworks, or commission these from OC Systems.
1.12 Is there a probe to count open socket file descriptors?
Not built-in, but we can (and did) write a custom one that you can inject yourself.
for RTI 3.4.2 and above, unpack it where you run your RTI Console, and follow the directions in the
README.txt. The source is included: check it out and you might get ideas of other probes
you or OC Systems could write.
1.13 What is the best method to find memory leaks using RTI?
RTI can help you tell if you have a memory leak, but not where it
is. RTI collects Java heap and garbage collection information
continuously and that information can be retrieved from the host with
the RTI console by transferring deep-dive performance data (with or
without an active deep-dive trace). You can then examine the Java
heap and GC information in the GC chart. If the Heap Used metric is
steadily increasing and/or there are a large number of major GC cycles
(lots of long vertical bars), you may have a memory leak.
At this point you need to switch to a tool specific to Java heap tracking, like
Eclipse Memory Analyzer (MAT).
With MAT you can analyze Java heap dumps and determine which
objects/classes are using heap memory, and which are potentially
1.14 What are the version compatibility dependencies between the different RTI components?
When stepping up from an older to a newer version, you should always
update the components in the following order:
That is, the version of RTI Enterprise should always be the same or newer than the
(optional) JON/RHQ Agent plugin, which in turn should be the same or newer than the RTI Console you're
using to connect to your servers. (The RTI Browser Collector version is generally not so important, but
it shouldn't be newer than the others unless explicitly directed by RTI Support.)
If you're having problems and you think it might be due to version
mismatches, don't be shy: please ask us!
- RTI Enterprise
- RTI JON/RHQ Agent Plugin,
- RTI Console
- RTI Browser Collector
2.1 Where do I download from?
You can request a free trial on the download page.
2.2 What do I install?
These might all be the same machine. You should have gotten an e-mail
with a link to a web site describing all this about the currently
available version. If not, go to our Download page
and fill out a form, or contact us.
- RTI Enterprise on each Windows or Linux server where they will be running their Java-based applications.
- RTI Console on their Windows, Mac or Linux desktop machine, and
- RTI Browser (optional) on Windows where you run your Chrome, Firefox or IE browser
(If you run Firefox on your Linux Desktop, RTI Enterprise includes the RTI browser plugin for this.)
2.3 How do I install?
It's all explained in the User's Guide.
2.4 How do I install the license key you just e-mailed me?
A license key is a single line with no trailing whitespace ending in a
newline. Put that single line into the file
$RTI_HOME/cfg/cookie on Linux or
%RTI_HOME%\cfg\cookie on Windows.
You can verify it with the RTI command
2.5 How do I Uninstall?
There are four aspects of RTI to consider:
Each of these is described in the user's guide.
- the instrumentation in your applications by RTI Enterprise;
- the RTI programs themselves: Console, Enterprise and maybe Browser;
- the data that RTI collected; and
- the JBoss ON or RHQ plugins, if applicable.
2.6 How do I run RTI command-line commands on Windows?
Run "Start→All Programs→RTI→RTI Enterprise Cmd" which opens a 'cmd' window
that is set up to run rti commands. You generally won't be needing this beyond
the "rti edit" command described in the user guide, but the command
rti 2>&1|more gives usage for all the commands.
2.7 How do I run RTI command-line commands on Linux?
In a bash-compatible shell window, do
where "/opt/ocsystems/rti/ee64" would be replaced by whatever your RTI
installation directory ($RTI_HOME). You generally won't be needing this beyond the "rti
edit" command described in the user guide, but the command
rti help gives
usage for all the commands.
2.8 Can I have multiple JBoss installations on the same machine instrumented with RTI? How?
Yes you can. Simply run the "rti edit" command on each one, naming its JBOSS_HOME value, for example:
rti edit -t jboss /opt/jboss-soa-p.4.3.0/jboss-as
rti edit -t jboss /opt/jboss-6.0.0.Final
2.9 Can I share the same RTI installation across multiple Linux machines?
No, this is currently not supported. The architecture of RTI assumes one or more
collectors unique to each host machine. If this presents a problem,
it may be possible to hand-edit some configuration files to get around this limitation:
please contact firstname.lastname@example.org.
2.10 Why do I get "command not found" when I do run "rti-enterprise-3.4.1-linux-x86.installer"?
If you are logged in as root, "." (the current directory) isn't generally in your PATH. Precede the installer name with "./" or "sh ".
2.11 When I run ./rti-enterprise-3.4-linux-x86.installer I get 'Permission denied.', but I'm signed in as 'root'?
When you downloaded the installer it may have lost its execute permission. Run it as:
2.12 How do I install the rshd service on Windows if I skipped it during installation?
The easiest way is to re-install just the RTI Enterprise component on
that computer. However, if that seems like overkill or you don't have
the installer any more, you can do it manually as follows:
- Open Start→All Programs→RTI
- Right-click on 'RTI Enterprise Cmd' and select 'Run As Administrator'
- In the cmd window that opens, execute this exact command (that's an upper-case P at the end).
- "%RTI_HOME%\rshd-1.7\src\rshd.exe" -install -r -P
- sc start rshd
- sc query rshd
- The last command should show the service running. You may have to add rshd program to your Windows or other firewall or open up port 514 in anti-virus software.
2.13 How to fix the RTI error on Windows startup after upgrading Java?
The program that shows the "RTI" icon is an Eclipse RCP program like
the RTI Console, and requires a 32-bit Java installation. It selects
the path to this at installation. If you delete that Java installation
it will cause the "RTISystray" application to fail. The easiest way to
fix this is to download and install a new RTI Browser -- the
installation process will find a new Java if one is
available. However, this requires that you close your browser
windows. Alternatively, you can change the Java path being used by
editing the file
C:\Program Files (x86)\OC Systems\RTI\Browser\RTISysTray\RTISysTray.ini as administrator, and
changing the path shown there to a similar path to a 32-bit Java executeable or jvm.dll that exists.
2.14 How do I add new RTI EDF files (provided by OC Systems) to my installation?
For simplity, say RTI_HOME=/opt/ocsystems/rti/ee64/ -- replace $RTI_HOME with your actual value below.
The procedure is the same for Windows except with the obvious path changes.
- Log into your enterprise machine as a user with write-access to the RTI installation (e.g., root or Administrator)
- Copy the new or updated
RTI*.edf files here to
$RTI_HOME/probes/, overwriting the existing files there.
- Execute the following RTI command-line command to push those out to your application collectors:
$RTI_HOME/bin/rti server update_collectors
2.15 MacOSX says RTIAnalyzer is damaged and can't be opened. Help?
This is a "feature" of OSX 10.8 and newer to fight malware. See
MacOSX Gatekeeper and "Damaged" application
in the user's guide for the workaround.
2.16 How do I do a "silent" Linux RTI Enterprise installation?
On Linux you can do silent installs using the -S option, explicitly
defining the three values RTI_HOME, RTI_DATA, and JAVA_HOME
on the command line.
$ sh rti-enterprise-3.3-linux-x86.installer -S \
RTI_HOME=/opt/ocsystems/rti RTI_DATA=/var/opt/ocsystems/rti JAVA_HOME=/opt/jdk1.5.0
You will also need to enable RTI in the JBoss servers for each JBoss installation with:
$ . /opt/ocsystems/rti/ee64/setup
$ rti edit -t jboss /opt/jboss-as-7.1.1.Final
Then the JBoss server will have to be restarted to pick up RTI.
If/when you have an RTI subscription license you should add that to
the RTI installation and restart the JBoss servers to pick up the
3 Enabling and Connecting
3.1 Why does this command fail:
The given path must be the parent of the bin directory containing the "run.sh" script, for example:
rti edit -t jboss /opt/jboss-soa-p.4.3.0?
rti edit -t jboss /opt/jboss-soa-p.4.3.0/jboss-as
3.2 I installed RTI Enterprise on a host, and I can connect from the RTI Console through JBoss ON to that host--why can't it find the RTI Enterprise installation?
It may be unable to locate the installation's RTI_HOME. You can
explicitly tell it where RTI_HOME is as follows:
- Select the host name in the collectors view,
- right-click and choose Properties,
- fill in the RTI_HOME value for that host at the bottom, for example:
- click OK.
- Now then right-click on that host again and choose Activate/Deactivate Host; then
- right-click again and choose Refresh.
3.3 When starting our app server with RTI using Java7, we get an error like,
This is a problem introduced by the Java7 bytecode verifier. If you
are seeing this it is probably because you are running in domain mode
rather than launching with standalone.sh. In this case you have to
add the Java option
ERROR ... java.lang.VerifyError: Expecting a stackmap frame at branch target 60. Is there a fix?
-XX:-UseSplitVerifier to those
already added to enable RTI, and restart the application server.
3.4 How can I keep RTI from being applied to my host controller and domain controller processes when running JBoss in domain mode?
This is apparently a side-effect of specifying the RTI-enabling Java options within the server-group definition. As of version 3.4.3 we recommend making the domain-mode changes to host.xml file as described in the current user's guide.
3.5 We launch multiple instances of JBoss from the same installation: how can we force each instance to have a different RTI Collector?
The easiest way is to define the RTI_COLLECTOR_ID environment variable to be the desired collector name, for example, for JBoss standalone mode:
# RTI_COLLECTOR_ID=jboss_users $JBOSS_HOME/bin/standalone.sh -Djboss.server.base.dir=$JBOSS_HOME/users \
> jboss_users.log 2>&1 &
# RTI_COLLECTOR_ID=jboss_warehouse $JBOSS_HOME/bin/standalone.sh -Djboss.server.base.dir=$JBOSS_HOME/warehouse \
> jboss_warehouse.log 2>&1 &
For JBoss domain mode, specify a different rti.collector.id property value in the jvm-options for each server definition, as shown in the current user's guide.
3.6 How do I enable RTI for Fuse Service wrapper?
This is described in the user's guide, and requires manually editing a properties file as for JBoss domain mode.
4 JBoss ON and RHQ
4.1 When I connect from the RTI Console to a host through JBoss ON, why does it gives a message that the RTI license expired?
This probably means it can't find the rti installation at all. See the next question.
4.2 How can I get more information about why JBoss ON/RHQ isn't finding RTI resources?
Enable JBoss ON agent debugging, force manual auto-discovery on the
platform root node, then look in the agent log file for more
information. See the next two questions for how to do this.
4.3 How do I enable JBoss ON/RHQ Agent debugging?
You can enable agent debugging by selecting the Inventory view,
Resources→Platforms>→your platform)→RHQ Agent. Then right-click,
choose Operations→Set Debug Mode, then choose "Yes" for both Enabled
and Trace Messaging Parameters, then click Schedule button at the
bottom of the Create New Operation Schedule view.
4.4 How do I force manual auto-discovery on a JBoss ON platform?
This is described in the user's guide.
4.5 OK, it's still not finding RTI, now what?
Perhaps the RTI plugin in in your remote JON/RHQ agent can't find the local RTI
Enterprise installation, because the magic files it looks for are missing or
have incorrect contents. The quick fix is to
on each machine to create the correct file(s). Here is the
There are two magic files that point to the current RTI installation
for a given host: There is a user-specific one and a global one,
/var/opt/.ocs-rti-info respectively. In this case, the
"$HOME" value is the home directory for the user who is running the
JON/RHQ agent, perhaps root or jonuser. So if you installed RTI
Enterprise on that machine as some other user, with a different home
directory, the "wrong" $HOME/.ocs-rti-info file would have been
created/updated. But there's still the global one. But again, if the
user you installed as doesn't have write access to /var/opt/, then
that file wouldn't have been created/updated either. Hence the command
above, to cause the global one to be created.
If you think these files are right, or our explanation wasn't clear,
4.6 Are there any unified views in the JBoss ON Dashboard that show RTI information from all the platforms?
No. For now you have to look at each platform separately, by clicking
Inventory, Resources, Platforms, then choosing the platform (host) of
interest. However, the dashboard is user-customizable as described in
the JBoss ON documentation.
4.7 Can I create alerts from the RTI Console?
No, the rti console is read-only with respect to JBoss ON
configuration. See the user's guide for how to create alerts in JBoss
4.8 When defining an Alert Notification, I choose RTI Alert and get a small dialog. I click OK, but apparently no notification was defined?
It takes a few seconds to construct the RTI Alerts
dialog. Unfortunately, it presents its OK button before it other
controls appear. Wait until you see a big dialog with a number of
Properties before proceeding.
4.9 In JBoss ON I see RTI Enterprise and 28 days left on license but when I click on the RTI Enterprise link to expand no RTI Collector resources show up. What am I missing?
You have RTI Enterprise and the accompanying JBoss ON plugins successfully installed. Now you must:
See the user's guide.
- enable RTI for your application(s) on that platform and
- restart the application(s) once after enabling to create the collector.
4.10 How can provide RTI console access to the JON/RHQ server but restrict permissions as much as possible?
See the user's guide section Configuring JBoss ON or RHQ to Allow RTI Console Access.
4.11 Is it possible to change the Platform names assigned by JON so they have a meaningful name in the RTI Console?
Yes! In the JON/RHQ Platform view, there's a ">" (tree expander) decoration next to the platform name/Title, right above the row of tabs. If you click that, it expands to show many properties of the platform. (This actually works for any resource you're currently viewing.). The 'Name:' property is editable -- if you hover over it an Edit operation will appear. Click that, change the name, click the checkmark to save your change. That's it.
Now, if you refresh the changed platform in your RTI Console, it will get the new name.
4.12 How can I do periodic or scheduled RTI Alert profiles/traces in JON?
RTI profiles and traces can be initiated and collected from RTI
Alerts. Normally, these RTI Alerts are triggered by metric values,
but RTI Alerts can be triggered when JON operations execute. JON
operations can be scheduled to execute at specific times and to
execute periodically. Combining these two features you can schedule
RTI Alert profiles and traces.
First, schedule an operation for a specific time or a periodic repeat.
You can use one of the RTI resource NOOP operations, which do nothing,
as a the triggering operation, as shown here:
Then create an RTI Alert with the condition set to Operation Execution,
the Operations set to the schedule operation from earlier, and results
status SUCCESS, as shown here:
Then define the RTI Alert to perform the desired action: a timed trace and snapshot, a timed profile and snapshot, a profile snapshot only, or a trace snapshot only.
The RTI resources have three NOOP operations defined by default. If
you need more NOOP operations you can extend the RTI JON agent plugin
XML to add more. The operation names must start with the string
"noop" to be considered a NOOP.
description="No operation 3.">
<c:simple-property name="stderr" description="The stderr output of the command execution." />
Contact RTI support if you need help with this.
Since RTI version 3.4.3.
4.13 Some RTI Console operations like Inject take a long time, and then time out. What happened?
If you're connected through JBoss Operations Network or RHQ,
long-running commands are started by the Agent, which then waits for a
result. If the result doesn't come with about 1 minutes, the agent
will "time out" so as not to hold up other operations. This does not
cause the server-side operation to abort, but means that the RTI
Console won't get a result or error messages. Nearly always the
operation will succeed if given another minute or so before assuming
the result. If you're impatient you can edit the
RTI_DATA/log/rtilog.txt file on the server to see if the operation
(e.g., 'restore_collector' for an Inject) has succeeded.
4.14 How can I reduce the CPU load of RTI metrics on the JON agent?
Simply put, you need to reduce the quantity of RTI metrics flowing
through the Jon agent, which can done in several ways:
Reducing the frequency of metric collection reduces the number of
metric values passing through the JON agent, but impacts the delay
between metrics showing a performance problem and the initiation of an
RTI alert to collect performance information. This is the least
desirable option. This method requires an adjustment to both the
metrics collection schedule on the JON server, and the metric
production schedule in the RTI collector configuration using
collector→Edit Configuration→BTO Commands→METRICS.
Reducing the number of different transaction types will also reduce
the number of metric values. By default, RTI will filter out requests
are generally not the source of performance problems. If you can
filter out "uninteresting" requests and focus on key application
requests (like Grails/Groovy dispatch) this could reduce the load on
the JON agent significantly (using the Edit Configuration -> BTO
Commands -> EXCLUDES).
Reducing the RTI metrics collected can reduce the load on the JON
agent without affecting alerts. Several RTI metrics can be disabled
and some others can have their collection frequency reduced. The
greatest impact will be from adjusting the RTI metrics for RTI
Transaction resources, because there are many more RTI Transactions
than RTI Collectors.
For RTI Transaction metrics, the following adjustments can be made:
- reduce the number of different transaction types that are monitored (using EXCLUDES)
- reduce the particular RTI metrics being collected, and
- reduce the frequency of the RTI metric collection.
For RTI Collector metrics, the following adjustments can be made:
|| Disable this calltime metric and use averageResponseTime instead.
|| Reduce the collection interval to once a day or less frequently.
|| No change.
|| No change.
|| No change.
The RTI Transaction metrics changes should reduce metric load to one
third (3 metric values per minute instead of 9 values), and a similar
reduction for the RTI Collector metrics.
Local tests of JON agent metric performance show the following performance:
|| Disable this calltime metric and use averageResponseTime instead.
|| Reduce the collection interval to once a day or less frequently.
|| No change.
|| No change.
|| No change.
|| Change to once per hour.
The test collected all 4 RTI metrics (including the calltime) for each
RTI transaction, every minute which is 20,000 metrics per minute
(which equates to 45,000 metric values). The Agent Measurement
Subsystem reported < 5 seconds per minute spent collecting metrics
(which corresponds to CPU monitoring) with no late collections. The
average CPU load for the JON agent was < 2% with one or two spikes at
90-100% CPU for 1 second each (or spread over two seconds at 50-60%
each). There were about 3 intervals where the CPU usage is about
20-30% over the 30 minute test period.
- 4 CPU 1.8GHz EC2 virtual instance,
- 10 JVMs running EAP 6.2 in domain mode (= 10 RTI collectors),
- 5000 different transaction patterns (500 per each JVM/collector) at the rate of 7000 requests per minute,
- JON agent running with a heap of 512MB.
4.15 What RTI Versions support what JON and RHQ versions?
As of July 2014:
Note 1: Version 3.4.3 added JON 3.2.0ER, 3.2.0GA. From this point on each JON release is explicitly incompatible
from the RHQ release it is based on. To reduce product size, we removed support for RHQ 4.1 and JON 3.0.0, 3.0.1
but those are available upon request as plug-ins to the RTI Console.
| RTI Version
|| Release Date
|| RHQ Version(s)
|| JON Version(s)
|| Apr 2012
|| 3.0.0, 3.0.1
|| initial RHQ/JON support
|| Jun 2012
|| added RHQ 4.3,4.4, JON 3.1.0
|| Jan 2013
|| same as 3.3.2C
|| Jul 2013
|| JON 3.0.0/1,-3.1.0/2
|| added RHQ 4.5-4.8,JON 3.1.2
|| Sep 2013
|| JON 3.0.0/1,3.1.0/2
|| added RHQ 4.9
|| Feb 2014
|| JON 3.1.1-3.2.0GA
|| Note 1 below
|| Apr 2014
|| JON 3.1.1-3.2.0GA
|| no additional versions
|| Aug 2014
|| RHQ 4.4-4.8,4.9,4.12
|| JON 3.1.1-3.2.0GA
|| added RHQ 4.12
5.1 How do Enterprise Collectors get created?
The Java-based enterprise collectors that are started by
scripts--currently JBoss, JMeter, SoapUI?, and Tomcat--are "enabled"
for RTI by edit these scripts, after which collectors are
automatically created or activated as-needed whenever the script is
run. Thus, the way to create a script for these is to choose a host
and collector type, "edit" the script, and then (re)start the
So you need to:
That will edit the catalina.sh script to call
rti_enable_tomcat.sh. Then you need to restart Tomcat and it should
create a collector (or use the one you had previously) and you should
then see processes and data:
- log in as a user with write permission to the scripts
- open a shell and execute these two commands:
rti edit -t tomcat $CATALINA_HOME # for example
rti get_collector_processes -t tomcat
or you can run the console and it will issue the same kind of commands to update the Tomcat collector node.
Just to continue some more, for all the collector kinds except java
and rcp, we have to do this "enable" step that consists of editing the
start-up script for the tool to call the appropriate
rti list_collector_data -t tomcat
$RTI_HOME/bin/rti_enable_*.sh/bat script. That can be
done: automatically at installation on Linux (and in the future maybe
Windows) if the env variables are set up and the user opts to, from
the console using the Enable page in the New Collector wizard.
manually from an RTI shell with the "rti edit" command,
It's spread out like this because the user must have privilege to edit
the tool scripts and that may not be possible for the console user. In
those cases they will have to get privilege or do it on the server
manually or during installation.
After doing the enable/edit the user will have to restart any running
tool processes to get data. This is another reason they may need to do
the operation on the server with privilege.
5.2 Why do I get a permissions error when trying to create a JBoss collector from the RTI Console?
There are two operations done when you create a JBoss (or other
Java-based) collector: creating the collector objects within RTI, and
enabling the JBoss application. This second step, enabling JBoss,
requires write access to JBoss application files. That's why we
recommend you enable JBoss as a separate step, from the command line,
as a JBoss administrative user, as described in
the user's guide.
5.3 How do I apply RTI to my Apache httpd web server?
This capability is provided for Linux by the "mod_arm4" module and is
described in the text file
5.4 Is it possible to restrict a tracing to a "white list" of incoming URLs?
The mod_arm4 plugin for Apache httpd can be used to provide a white list capability:
That will allow tracing for listed IP addresses and not trace all
help in setting this up.
- Set the base (last) trace policy to trace none.
- Add earlier trace policies that match the IP addresses (one for each) that should be allowed with trace DEFAULT, ALL, SAMPLE, or whatever.
5.5 Where does RTI Enterprise create temporary files?
Version 3.4.1 of RTI manages temporary files differently from previous
versions. Older versions of RTI stored temporary working files in the
default temporary directory for that platform ($TMP for Linux and
%TEMP% for Windows). RTI now places (almost) all the temporary files
in $RTI_DATA/tmp. A few small temporary files may still be placed in
the default platform temporary directory, but the space requirements
are very small.
5.6 What metrics can we collect if we enable Apache (httpd) collecting in RTI?
If you want metrics about the Apache web server itself, you may be
able to get these from JBoss operations network. Use RTI plus the
modarm4 module for Apache httpd to provide deep-dive tracing (not
lightweight-profiling) of all transactions that go through the web
5.7 How do I use RTI with JMeter?
If your JMeter is on your desktop where you run your RTI Console, then
you can enable it directly from the RTI Console as described in
22.214.171.124 Enabling RTI Using the Console Wizard (just choose 'jmeter' instead of
'tomcat' as the collector type in section 126.96.36.199.2).
If your JMeter is remote relative to your RTI Console, then you
probably need to use putty to log in directly to the machine where
JMeter is running, as a user with write-access to
$JMETER_HOME/bin/jmeter.sh, and use
rti edit -t jmeter .... as
described in the user's guide section 188.8.131.52, "Enabling RTI Through the Command Line".
Either way, once JMeter is enabled for RTI, it will trace RTI transactions whenever you run it.
NOTE: Most users do notapply RTI to JMeter: they just enable their
web- and application servers for RTI, then drive their tests with
JMeter (or a similar) tool as before. The reason to enable JMeter
with RTI is to force every transaction sent by JMeter to be deep-dive
traced. Though there are ways to fine-tune the tracing, for
load-testing and longer-running test cases this generally collects a
lot more data than you want.
6 RTI Console
6.1 How do I copy a file to or from the RTI Console's Navigator view to a Windows folder?
When the RTI Console is first started, there is a shortcut created on
the desktop label "RTI Drop Box" that points to the RTI Drop Box/RTI
Data folder in the Workspace View at the lower left of the console
window. This is an easy way to access files from both within the RTI
Console and your Windows Explorer. In the absence of the RTI Drop Box link.
You can Copy out of Workspace Drop Box by copy and paste:
Copy into the Workspace Drop Box by drag and drop: Copy/paste as above
will work, but this is easier. Simply select the file in the Windows
Explorer, and holding the mouse down, drag it onto the folder (e.g.,
RTI Data) shown in the Workspace view.
- Click to select the file under RTI Drop Box/RTI Data in the Workspace View
- Type Ctrl-C or do right-click→Copy.
- In the destination windows folder type Ctrl-V, or do right-click→Paste.
6.2 Why does Linux RTI Console crash with a JVM core dump?
Check that your DISPLAY environment variable valid is valid. If Firefox can
work, then RTI Console should also.
6.3 Why does Linux RTI Console context help (opened by ? buttons) launch a browser window instead of appearing within the app like it's supposed to?
Probably because the installed version of xulrunner is newer than that
supported by the version of Eclipse SWT that RTI is built from (3.7/Indigo)
6.4 How do I fix Linux RTI Console Help not displaying due to error "Firefox is already running but is not responding."
The RTI Console is Eclipse-based, and uses the Eclipse help system. On Linux
this requires a Firefox browser. If Firefox is running you will need to stop
and restart it. If it's not running, delete
6.5 How can I improve the performance of the RTI console?
The performance and capacity of the RTI console is directly
proportionally to the size of the Java heap the console uses. The
32-bit RTI console defaults to a 1GB heap and the 64-bit version
defaults to a 2GB heap. These default sizes can be adjusted locally
as described below. However, making it larger doesn't necessary help:
It is very important that the Java heap fit in system memory and not
be paged, or performance will be severely impacted.
The following table summarizes Java heap size, required free memory, and trace capacity for the RTI console:
Note that the required memory is about 250MB above the heap size. RTI
events correspond to method calls traced, and RTI transactions
composed of RTI events. RTI transactions vary in complexity based on
the system traced, and so the number of RTI events per transactions
can vary between 10 to 200, on average--a value of 100 events per
transaction was used in the table above.
The performance of the RTI console can be improved by closing tabs
that are needed, particularly deep-dive trace tabs. In general you can
increase performance by closing unneeded deep-dive editors which tax
the Java heap. Closing all views (other than the Welcome view) works
best, as it also disposes of data shared by multiple views.
You can track the Java heap usage by the RTI console by
enabling the heap status as follows:
Windows→Preferences→General→Show heap status
| Java Heap
|| Required Memory
|| RTI Events
|| RTI Transactions
This will display a heap status widget in the status line of the RTI
console (along the bottom). The heap status shows the total Java heap
used and the current size of the heap. The Java heap may grow in size
up to the maximum specified (1GB for Windows and 32-bit Linux, 2GB for others).
You can increase (or decrease) the available heap by closing the RTI Console and
rti-console-install-path/RTIEEConsole/RTIAnalyzer.ini file to
change the size specified in the -Xmx option there. For example, if
you have 8MB of RAM on your machine and
you can change that to
to double the available memory.
6.6 Why does "Open Deep Dive View" open the wrong view sometimes?
This is a glitch in the Eclipse Windows UI with selecting items in the
navigator. I open dataset A by selecting it under Saved Trace Data. Then I
select on dataset B, choose 'Open'. and it re-opens A. If I double-click on B
it opens correctly. We can't figure it out either, so recommend using
6.7 In the Linux RTI Console, 'Summary' seems to do nothing, and Context help (? button) operations send the HTML output to a separate browser rather than in the help panel. Is there a fix for this?
Yes, this is a known bug in eclipse on newer Linuxes. The workaround is
to install WebKitGTK and then tell Eclipse's SWT to use that. If you're
using 64-bit RTI Console (rti-console-*-lx86_64-*) then:
or else if you're using 32-bit RTI Console (rti-console-*-lx86-*) then:
yum install webkitgtk.x86_64
yum install webkitgtk.i686
6.8 Is there a way to give a better name than the raw hostname or IP address to a host in the RTI Console?
Yes: Select a host, right-click, choose Properties. In the Properties dialog change the "Description" to what is wanted, then click OK.
Then right-click and Refresh the collector: The new Description will then appear instead of the original hostname for as long as that host is
defined in your local RTI Console workspace.
6.9 Why can't the RTI Console (or JON) find RTI on a host after it successfully connects there?
See OK, it's still not finding RTI, now what?.
6.10 Can an RTI Console user connect directly to a linux server without having a login account there?
No. The two modes are ssh login or through JBoss Operations Network. You could have multiple users connect using the same
login account dedicated just for that, if your security procedures allow this.
6.11 What's the best way to verify that the RTI Console is working in my environment?
These are covered in the Quick Start Guide and the RTI User Guide.
If these operations work, then you should be in pretty good shape.
- Try Connecting to a host or hosts
- Try getting metrics using the Analyze Resource performance Measurements link from the Welcome Page
- Try getting profiles form the hosts using the View Transaction Profile Snapshots link from the Welcome page
- Try starting a deep-dive trace using the Start Deep-Dive trace link from the Welcome page.
6.12 On Windows, the RTI Console sometimes becomes unresponsive. What do I do?
Use the X or context menu in the RTIAnalyzer window frame to close it.
When prompted, click 'Close the Program' and when Windows says,
"Searching for a solution..." click "Cancel". This can happen if RTI
runs out of memory (see
How can I improve the performance of the RTI Console.
However, there also is an intermittent deadlock in the Windows RTI Console related to
'Edit Configuration' which we've as yet been unable to isolate.
6.13 How can I change the JVM used by the Windows RTI Console?
If you already have a 64-bit Windows JRE in your PATH but for some reason the
RTI Console isn't finding it, OR you want to try a different JVM for some reason:
- Using "Run As Administrator" edit the file C:\Program Files (x86)\OC Systems\RTI\Console\RTIEEConsole\RTIAnalyzer.ini (the path above might be different if you changed it when installing)
- In that RTIAnalyzer.ini file, insert two lines BEFORE the "-vmargs" line:
6.14 I can connect to my server in the cloud successfully, but when I choose a connected host, the resulting entry is deactivated. How do I fix that?
$HOME/.ocs-rti-hosts is likely wrong, namingg one or more host(s) that don't really exist. This can happen if the IP
address/hostname changed but an old
.ocs-rti-info file was preserved. To fix this:
- Delete the deactivated host from your RTI Console
- For Openshift, Log into your server on the cloud (e.g., email@example.com) and hand-execute the command
hostname > ~/.ocs-rti-info.
- For AWS or other cloud setups, check all hostnames or IPs named to ensure that they're correct and running.
- In the RTI Console, reconnect to the cloud now: it will present you the correct host name(s) which when checked should work.
7 Events and Properties
7.1 Why does the order look wrong in the tree table when I sort by Elapsed Time
Sorting is done independently at each level of the transaction
tree. So all the Top-level transactions are sorted, then under each of
those, it's direct children are sorted, and so forth. So if you look
at an expanded tree it can look out of order. You can flatten the tree
into a single linear list by doing a filter on "/".
7.2 Why is the Elapsed Time for some erroneous transactions so long?
Some "failed" (light red color) transactions are so marked because no
stop event was ever seen. The stop time associated with them is the
time at which the tab or window is closed, which could be a long time
after it was started. The statistics associated with failed
transactions are not generally reliable.
7.3 What is the meaning of the Elapsed Time and Data Length values I see associated with a shockwave-flash file corresponding to a YouTube video?
This are the load time and size of the actual .swf file. The file may
then display an animation or streaming video which will not be
measured by RTI, since what you see is not interacting with the
browser itself in any way.
7.4 What does 304 means as a value for the "Cached" property?
For cached, 0 means "page requested, not cached"; 1 means "page
retrieved from cached, not requested"; and 304 means "request made
with 'If-Modified-Since' header in request"; and "304 Not Modified" was
returned, so the page was then retrieved from cache.
7.5 How can I obfuscate IP addresses and names in RTI-collected data?
Use the RTI Console to:
innocuous characters, and
- Export it as XML
- edit the XML file to substitute sensitive strings with similar
- load it again into a new dataset.
7.6 How do I get a complete date/time to show in Excel from RTI-exported CSV file?
RTI allows you to export selected transactions from a transaction tree
view in CSV format. When you open the exported .csv file with
Microsoft Excel, you see only hours and minutes -- the date isn't
Amazingly enough, Excel does not support automatically formatting a
date/time with millisecond accuracy from a CSV file! You need to do a
custom format in Excel to properly use the millisecond date/time
exported by the Console in the CSV file, as follows:
Click here for a larger view
Now column A gives the date/time with millisecond precision.
- Highlight column A
- Select "Format Cells..."
- On the Number tab select Custom under Category
- On the right, select any format in the list, say "m/d/yyyy"
- Edit the format immediately under "Type:" to read "m/d/yyyy h:mm:ss.000"
- Click OK.
7.7 Is there a limit on the length of string values logged in deep-dive?
Yes. Before version 3.4.5, string data such as SQL queries is truncated to 255 characters. Newer versions truncate at 1023.
7.8 How can I see long strings in the deep dive views?
The list views in the deep dive editor show only part of the string.
To show the full text of the string (up to the length limit above) you
- select a row, change to the Transaction Properties view tab in the lower left, and maximize that to see a full screen of data
- OR, Select the Property, right click and use Copy to Clipboard, then paste in another window to see the full text
8 Events Editor
8.1 How can I see methods from my application in the RTI call profile and deep-dive trace?
Use the Collector Configuration Editor as described
in the user's guide.
8.2 Does RTI support the blotz framework/API?
If you don't see RTI-blotz.edf in the Collector Configuration
Editor's "Events" tab, then the answer is probably no. But we might
be able to whip something up for you, or provide guidance on how you
could add it yourself. Please contact firstname.lastname@example.org and
8.3 Are there any EDFs that I absolutely should not uncheck?
No: If you really want RTI to do nothing at all in your enterprise
application, you can uncheck everything and your application will
still work. This is basically what "Disable RTI" does when applied to
But: Do not uncheck RTI-Monitoring.edf unless you want to disable all
the lightweight profiling, JMX metrics, JON metrics, and statprof.
In general, you should only disable EDFs if you're getting data about parts
of the application that you don't care about.
8.4 When I 'Inject' changes, when do they take effect?
Changes may take effect immediately, periodically, or only on restart, as follows:
- Changes to the list of selected EDFs and probes, and changes to the contents of those EDFs and probes, take place immediately upon injection, subject to minor delays if updated methods are currently being executed.
- Changes to the Trace Policy properties about what is logged under what conditions, are picked up approximately once per minute, more or less accurately depending on the frequency of data logging.
- All others take effect only when you restart the application.
8.5 What is the correct way of adding methods while creating a custom EDF file? The goal is to be able to add by package name and NOT by individual methods.
To add by package, you:
If you trace too many methods you will introduce too much overhead to give you
very accurate timings. Avoid low-level methods and favor API-level methods.
- right-click on the RTI event, just like adding a single method,
- the Add Method dialog opens.
- you type in a wildcard pattern in the Additional Method field (with the quotes):
- "com.my.package.*::*" // for all methods in all classes in a package
- "com.my.package.MyClass::*" // for all methods in a class
- "com.my.package.MyClass::myMethod(*)" // for a specific method
- click OK
8.5.1 What happens if I have two wildcards whose expansion overlaps?
If you have, for example:
MONITOR EDGE METHOD OCSTX com.ocsystems.example.ecommerce.*::* and
CHILD METHOD OCSOTHER com.ocsystems.example.*::*
You will get duplicate entries for those methods that match both wildcards, with
one appearing as the caller of the other.
8.6 How can I further configure RTI to collect more profile and trace data?
The general mechanism for specifying where RTI collects profile and
trace information is to create a custom EDF. This is generally done
using the RTI console Collector Editor where you can select specific
methods from your application and specify whether the method should be
a trace/profile point, and whether it should be a monitoring point.
For most situations this process will be quite sufficient, but the RTI
EDF file support a number of features which are not accessible from
the RTI Collector Editor (see the next question).
To use these features you will need to edit
the EDF file manually, outside of the RTI Collector Editor. Actually,
you can edit the EDF file as an ordinary text file in the RTI console
by saving and opening it as such.
If you want to edit an existing EDF file, first save it to the RTI console workspace:
If you want to create a new EDF file, create it in the RTI console workspace:
- right-click on the EDF file in the list of EDFs
- choose Save As.. in the context menu
- choose a destination in the RTI workspace (RTI Data folder)
- click OK
The file may be opened in the default system editor which is fine, or
you can choose to edit it in the embedded text editor in the RTI
- right-click on the RTI Data folder int he workspace
- choose New→File in the context menu
- specify the file snake with an extension of .edf
- click OK to create the file
The file will be available to edit.
- close the default editor, if open
- right-click on the new EDF file
- choose Open With→Text Editor from the context menu
8.7 What advanced directives are available in the EDF files?
The EDF file may contain blank lines, comment lines, and directives:
The most common and useful forms of directive lines are:
- blank lines are ignored
- comment lines start with # or DESCRIPTION and are ignored
- all other lines are considered to be directive lines
The first of these is for selecting application methods for monitoring,
tracing, and profiling. Use the MONITOR keyword to specify that the
method is a BTM: a business transaction that can produce metrics, and start
profile summary trees and deep-dive traces. Use the EDGE keyword to
specify that this method should start deep-dive traces
unconditionally--this is not very common. Use the CHILD keyword to
indicate that the method should be traced only if a caller starts the
trace--this is the most common form. The quoted-id specifies the
event class that appears in traces. The quoted-method-target
specifies to which application methods these actions should be
applied--these are generally method signatures but can be wildcard
The second form above is for specifying that custom probes should be applied
when the target methods are executed. These are primarily used for
built-in RTI probes for common frameworks. There are several useful
RTI probes that you might want to use when investigating application
- [MONITOR] [EDGE | CHILD] METHOD quoted-id quoted-method-target
- CUSTOM quoted-id quoted-method-target custom-probe-class-name
- CUSTOMBEAN quoted-id quoted-method-target custom-bean-class-name
An example of using the traceback probe is:
- RtiLogParametersProbe - to record method parameters in the deep-dive trace
- RtiLogTracebackProbe - to record a stack trace in the deep-dive trace
- RtiPrintParametersProbe - to print method parameters to the console
- RtiPrintTracebackProbe - to print a stack trace to the console
CUSTOM "traceback" "package.Class::method()" RtiLogTracebackProbe
The third form is for specifying custom probe beans, which contain
sets of cooperating probes. These are primarily used for built-in RTI
The quoted-method-target specifications have the following form:
Note that the separator between class and method names is a double
colon (::). Wildcards (*) can be used once in the class specification
and once in the method signature specification and should be the
rightmost character (on each side to the double colon). Be careful
using wildcards because they may match many methods, and the RTI
runtime limits the number that can be matched in one directive.
8.8 How do I add my own hand-written probe to my collector(s)?
Add a user EDF file in your RTI Console as described in the user's guide,
4.2. Configuring a Collector:
Once this operation completes, the collector(s) should start collecting additional information the next time you run your test.
- Open the RTI Console application on your desktop
- In the Workspace view in the lower left open "Drop Box" folder
- Drag your EDF to Workspace>Drop Box>RTI Data OR select RTI Data, right-click, select Import... to use eclipse import wizard
- Select an appropriate collector in the Collectors view on the upper left
- Right-click and and select Edit Configuration in the context menu
- Load your EclipseLink.EDF as described in 184.108.40.206.3. Loading an External EDF
- Make sure the EDFs you want to use are checked.
- Click Inject at the top of the Event editor window
- Select collector(s) to inject the change into, and click OK.
9 Browser (Chrome, Firefox, IE) Collector Plugins
9.1 Can I install the Browser Collector(S) without the Console?
Yes, and it will collect data from your browsers that can be viewed by
other RTI Consoles.
9.2 Can I look at the collected data without using the RTI Console?
The .rti files are XML text, but they're not very readable. Contact us
for an rti2csv tool which might help.
9.3 Why didn't my browser collect any RTI Events?
There are a few possible reasons:
RTI Firefox and Internet Explorer Collectors write under a sub-folder in %USERPROFILE%.
You can change this behavior by editing
- RTI doesn't support your version of Firefox. As of version 3.4.1, Firefox supports only version 4 or newer.
- You didn't restart your browser after the installation. Try that now.
- The RTI extension or add-on is disabled -- check the appropriate settings in your browser.
- The RTI System Tray is Disabled, causing RTI to specifically not collect Events. Make sure it is enabled:
- The plugin is broken or incompatible. Contact email@example.com so we can help you figure it out.
- There was a permissions problem writing to the expected output path:
and modifying the parameter RING_LOG_DIR to a safe location; for example
Restart Firefox and verify there are now .rti files contained in
9.4 Does a top-level browser navigation event such as IE-Navigation always correspond to an interactive user action?
Not always. On pages with sophisticated dynamic content, transactions
can be initiated that are impossible to relate back to the last
user-initiated transaction that caused them. However, if you have
reproducible output which doesn't look right, please
report a bug.
9.5 Where did my RTI data from last week go?
In its default configuration, the Browser collector only keeps data
for seven days or up to a limit of 32MB. See the next question.
9.6 How do I change how much or how long RTI data is saved?
If you want to keep more data, change the parameters. Here you can
change the values of TOTAL_RING_SIZE, RING_SIZE, and KEEP_DAYS,
NOTE You must restart your browser(s) to pick up the changed configuration.
9.7 What is an "ARM Ring?"
By a "ring" of files, we mean a fixed-size group of files, such that
each file is approximately the same size, and the oldest is deleted
when creating the newest. There is always one more file than the
stated ring size. For example, if ARM Ring Size is listed as 1,
there's never more than 1 file in addition to the one being written
to. If the Arm Ring File Size is 8388607, then when a transaction is
recorded which brings the current file size at or above 8MB, that file
is closed. If there's already an existing data file for the current
process it is deleted, and then new file with the next sequence number
9.8 How do I change RTI Browser Collector configuration?
Using the RTI Console, right-click on the Localhost→Chrome, Firefox or IE
collector, and choose Edit Configuration... to see the Properties
page. Note that the same RTI browser configuration is shared by all three
browser "brands" so changing the properties for Chrome changes them for IE and
The properties are are as described
9.9 How much data does RTI keep by default?
When you re-start your browser, it deletes data files changed
more than 7 days before. For each IE process, the default is to keep 2
8MB data files, or about 10,000 transactions. These settings can be
changed in the Properties page for the collector.
10 Reporting Bugs
We've built lots of support into RTI for collecting internal data to
help us figure things out, so here's what to do...
10.1 Where do I send bug reports?
Attach all data in any compressed format to an e-mail describing the
- Compose an new e-mail to firstname.lastname@example.org.
- Attach the rtiinfo-*.zip, rtilog.txt, or other files to the e-mail message
- Write in the message an explanation, as detailed as possible, describing what happened and why you think it's a problem.
- Verify that the attachment is there, then click Send.
10.2 How do I gather data about a bug seen using the RTI Console?
Under the RTI Console, there is Help→Problem Report. Try that first,
and send the file it tells you it creates.
10.3 How do I gather data about a bug against the Enterprise collectors?
The RTI Console's Help->Problem Report will gather data about all
connected hosts and collectors, but if the problem isnt' with
something you're currently connected to, then:
- Send the stdout/stderr (console) output from the RTI-instrumented application.
- For this reason it's important to try to capture this output whenever running with RTI.
- Send %PROGRAMDATA%\OCS\RTI\EE\log\rtilog.txt if its available.
- Compress data files with zip or tar+gzip if possible