TIBCO BusinessWorks 6.x integrates with different TIBCO and non-TIBCO systems like databases, TIBCO EMS Server, SOAP Service over HTTP/JMS, and TIBCO messaging frameworks like FTL. AppDynamics supports JMS, SOAP, JDBC and HTTP backend detection from TIBCO BusinessWorks out-of-the-box. However, non-standard and proprietary backend calls to TIBCO FTL are not detected by the AppDynamics Java Agent out-of-the-box. Backend instrumentation and custom-activity-correlation are needed to capture the end-to-end flow between upstream and downstream tiers in a single Business Transaction. Below is a solution that we successfully tested and implemented with a customer to instrument cross-tier FTL communication and correlate exit calls in TIBCO BusinessWorks 6.x.
For testing purposes, we used:
We had three TIBCO BusinessWorks Applications: PT, BT and TT. Each JVM used a unique TierName and all the three JVMs reported to the same application in the AppDynamics Controller.
PT is a simple REST Service implemented with TIBCO BusinessWorks 6.x that accepts API requests at http://localhost:8080. It sends an FTL request to BT and receives a FTL reply from BT using FTLRequestReply activity. It then provides the reply back to the end user.
BT uses a FTLSubscriber activity to receive the FTL request from PT, uses FTLRequestReply activity to send another request to TT and receive reply from TT, then replies back to BT using FTLReply activity.
TT uses a FTLSubscriber activity to receive the FTL request from BT, makes an outgoing HTTP call using SendHTTPRequest activity, gets back a response and then replies back to BT using FTLReply activity.
singularityheaderin both the FTL request and reply messages in order to preserve the cross-tier correlation. However, since FTL is a non-standard call, they would need to identify the proper place/method to inject/extract the
singularityheaderin the FTL message.
These 8 steps will guide you through how to instrument cross-tier FTL communication and correlation in TIBCO BusinessWorks 6.x.
1. Apply the attached
custom-activity-correlation.xml for all three JVMs (open the attached file named “Step 1 - custom-activity-correlation.xml.txt”). There are two producer and two consumer entries here. For this specific customer, we used three FTL activities:
custom-activity-correlation.xmlfile, the first producer and the second consumer are from this activity.
custom-activity-correlation.xmlfile, the second producer is from this activity.
custom-activity-correlation.xmlfile, the first consumer is from this activity.
You are putting the
singularityheader in the both producers and reading the
singularityheader in both consumers in the respective FTL calls invoked by these activities. If there is no FTLRequestReply and FTLReply activity involved in the BW code (no request-reply scenario) and you just have FTLPublisher and FTLSubscriber activities (only send-receive scenario), you can use the
custom-activity-correlation.xml in the attached file named “Step 1 - custom-activity-correlation.xml (only send-receive scenario).”
2. Make sure that the FTL message schema in the BW application has a new optional String Type element named
singularityheader added to it. You have to do this code change to make sure the application can understand the presence of this new header element in the FTL message and successfully parse this.
Add this new optional element for all the TIBCO BW 6.x nodes that are sending and receiving FTL messages.
Note: Make sure that this element is added at the end of your other existing elements in the FTL message schema or the application will fail to parse the FTL message with the
3. TIBCO BusinessWorks 6.x uses Jersey Server and by default, we exclude the above class in our code for async processing. In order to make sure thread-correlation happens properly, we must add the following include to the
JAVA_AGENT_HOME/verx.x.x.x/conf/app-agent-config.xml in the
fork-config section for all your FTL nodes. (Note: The field scrolls horizontally)
<include filter-type="EQUALS" filter-value="org.glassfish.jersey.server.ServerRuntime$AsyncResponder$2"/>
4. Add the Appagent node property
error-safety-rule-error-threshold=-1 for all nodes that are sending/receiving FTL messages.
Note: The custom correlation is applied on the nodes about 15 minutes after the application starts. If you start putting load on the application as soon as the nodes start, the
singularityheader will not be recognized and it will trigger an error in the Business Transaction. Eventually, the instrumentation will be neutralized and the
singularityheader will never be set. You need to add the above node property to make sure that the instrumentation is not neutralized during the first 15 minutes of the application startup when there is no
singularityheader set in the FTL messages.
5. Add Appagent node property
track-async-inside-custom-exit=true for all nodes sending/receiving FTL messages. This node property is used to track async calls inside the custom exit points.
6. If SpringBeans instrumentation is enabled for any of your TIBCO BW FTL Nodes (for BTs or SEPs), add the Appagent node property
ignore-exitcall-on-reentry=false for all nodes sending/receiving FTL messages. This will ensure that your custom exit calls are not lost due to SpringBeans instrumentation. Otherwise, you can simply disable SpringBeans instrumentation for BTs and SEPs on all such nodes.
7. If you are making outgoing WebService calls instead of HTTP calls in any of your nodes in your BW application, make sure that WebService type exit points are disabled on all such nodes. We have seen the FTL custom exit calls get lost when used with WebService type exits.
8. Start all the three TIBCO BusinessWorks 6.x JVMs with the AppDynamics Java Agent instrumentation and put load on the PT Tier. After around 15 minutes, you will see the complete end-to-end application logic in your AppDynamics Controller dashboard (see examples below).
Below are examples of Controller dashboards that display the complete end-to-end logic after following the steps above.