How do I instrument and correlate TIBCO FTL exit calls in TIBCO BusinessWorks 6.x?

[Biswajit.Nanda]

How do I instrument cross-tier FTL communication and correlate exit calls in TIBCO BusinessWorks 6.x?

Table of Contents

• Background
• Test Environment and Scenario
• Challenges
• Solution
• Example Dashboards

Background

TIBCO BusinessWorks 6.x integrates with different TIBCO and non-TIBCO systems like databases, TIBCO EMS Server, SOAP Service over HTTP/JMS, and TIBCO messaging frameworks like FTL.

AppDynamics supports JMS, SOAP, JDBC and HTTP backend detection from TIBCO BusinessWorks out-of-the-box. However, non-standard and proprietary backend calls to TIBCO FTL are not detected by the AppDynamics Java Agent out-of-the-box. Backend instrumentation and custom-activity-correlation are needed to capture the end-to-end flow between upstream and downstream tiers in a single Business Transaction.

Below is a solution that we successfully tested and implemented with a customer to instrument cross-tier FTL communication and correlate exit calls in TIBCO BusinessWorks 6.x.

Test Environment and Scenario

For testing purposes, we used:

• TIBCO FTL 5.2.x
• TIBCO BusinessWorks 6.x
• AppDynamics JavaAgent 4.5

We had three TIBCO BusinessWorks Applications: PT, BT and TT. Each JVM used a unique TierName and all the three JVMs reported to the same application in the AppDynamics Controller.

PT is a simple REST Service implemented with TIBCO BusinessWorks 6.x that accepts API requests at http://localhost:8080. It sends an FTL request to BT and receives a FTL reply from BT using FTLRequestReply activity. It then provides the reply back to the end user.

BT uses a FTLSubscriber activity to receive the FTL request from PT, uses FTLRequestReply activity to send another request to TT and receive reply from TT, then replies back to BT using FTLReply activity.

TT uses a FTLSubscriber activity to receive the FTL request from BT, makes an outgoing HTTP call using SendHTTPRequest activity, gets back a response and then replies back to BT using FTLReply activity.

Challenges

1. As mentioned above, unlike normal HTTP/JMS/JDBC backends, there is currently no out-of-the-box support in the AppDynamics Java Agent for TIBCO FTL exits.
   
   
2. Even if the customer was able to get the class/methods for TIBCO FTL exits to define custom exit points, there are two kinds of exit points:

• One type of exit point is implemented in the FTLRequestReply activity that sends a request message on a normal FTL endpoint.
• Another type of exit point is implemented in the FTLReply activity that sends a reply message on an FTL INBOX.

3. Even if the customer applied instrumentation on both types of exits as a custom exit, they would need to inject the singularityheader in both the FTL request and reply messages in order to preserve the cross-tier correlation. However, since FTL is a non-standard call, they would need to identify the proper place/method to inject/extract the singularityheader in the FTL message.

Solution

These 8 steps will guide you through how to instrument cross-tier FTL communication and correlation in TIBCO BusinessWorks 6.x.

1. Apply the attached custom-activity-correlation.xml for all three JVMs (open the attached file named “Step 1 - custom-activity-correlation.xml.txt”). There are two producer and two consumer entries here. For this specific customer, we used three FTL activities:
   
   • FTLRequestReply: It has a producer part and a consumer part. The producer part sends a request message on a normal FTL Endpoint and the consumer part receives a reply from an FTL INBOX. In the attached custom-activity-correlation.xml file, the first producer and the second consumer are from this activity.
     
     
   • FTLReply: It is a producer that sends a reply message on an FTL INBOX. In the attached custom-activity-correlation.xml file, the second producer is from this activity.
     
     
   • FTLSubscriber: It is just a consumer that receives requests from normal FTL Endpoint. In the attached custom-activity-correlation.xml file, the first consumer is from this activity.
     
     You are putting the singularityheader in both producers and reading the singularityheader in both consumers in the respective FTL calls invoked by these activities. If there is no FTLRequestReply and FTLReply activity involved in the BW code (no request-reply scenario) and you just have FTLPublisher and FTLSubscriber activities (only send-receive scenario), you can use the custom-activity-correlation.xml in the attached file named “Step 1 - custom-activity-correlation.xml (only send-receive scenario).”
     
     
2. Make sure that the FTL message schema in the BW application has a new optional String Type element named singularityheader added to it. You have to do this code change to make sure the application can understand the presence of this new header element in the FTL message and successfully parse this.
   
   
   Add this new optional element for all the TIBCO BW 6.x nodes that are sending and receiving FTL messages.
   
   

   NOTE: Make sure that this element is added at the end of your other existing elements in the FTL message schema or the application will fail to parse the FTL message with the singularityheader.

3. TIBCO BusinessWorks 6.x uses Jersey Server and by default, we exclude the above class in our code for async processing. In order to make sure thread-correlation happens properly, we must add the following include to the JAVA_AGENT_HOME/verx.x.x.x/conf/app-agent-config.xml in the fork-config section for all your FTL nodes. (Note: The field scrolls horizontally)
   

   <include filter-type="EQUALS" filter-value="org.glassfish.jersey.server.ServerRuntime$AsyncResponder$2"/>
   

4. Add the Appagent node property error-safety-rule-error-threshold=-1 for all nodes that are sending/receiving FTL messages.

NOTE: The custom correlation is applied on the nodes about 15 minutes after the application starts. If you start putting load on the application as soon as the nodes start, the singularityheader will not be recognized and it will trigger an error in the Business Transaction. Eventually, the instrumentation will be neutralized and the singularityheader will never be set. You need to add the above node property to make sure that the instrumentation is not neutralized during the first 15 minutes of the application startup when there is no singularityheader set in the FTL messages.

5. Add Appagent node property track-async-inside-custom-exit=true for all nodes sending/receiving FTL messages. This node property is used to track async calls inside the custom exit points.
   
   
6. If SpringBeans instrumentation is enabled for any of your TIBCO BW FTL Nodes (for BTs or SEPs), add the Appagent node property ignore-exitcall-on-reentry=false for all nodes sending/receiving FTL messages. This will ensure that your custom exit calls are not lost due to SpringBeans instrumentation. Otherwise, you can simply disable SpringBeans instrumentation for BTs and SEPs on all such nodes.
   
   
7. If you are making outgoing WebService calls instead of HTTP calls in any of your nodes in your BW application, make sure that WebService type exit points are disabled on all such nodes. We have seen the FTL custom exit calls get lost when used with WebService type exits.
   
   
8. Start all the three TIBCO BusinessWorks 6.x JVMs with the AppDynamics Java Agent instrumentation and put load on the PT Tier. After around 15 minutes, you will see the complete end-to-end application logic in your AppDynamics Controller dashboard. See Controller dashboard examples, below.
   

Example Dashboards

Below are examples of Controller dashboards that display the complete end-to-end logic after following the steps above.

• Application Dashboard
• Node Dashboards
• BT Dashboard
• BT Snapshot

Application Dashboard

Node Dashboards

BT Dashboard

BT Snapshot

[Joshua.Moesa]

Thank you! I managed to get this setup working on TIBCO BusinessWorks ContainerEdition apps deployed in Docker containers.

Some additional notes can be found here:
https://github.com/joshuamoesa/tibco-businessworkscontaineredition/tree/master/AppDynamicsDemo

[Claudia.Landivar]

@Joshua.Moesa, thank YOU for sharing this solution!

Claudia Landivar

Community Manager & Editor, Knowledge Base 

[Phuc.Duong Quang]

Hi Joshua.Moesa,

I'm facing with Tibco BWCE 6,

it is running via xxx.ear file not .jar file?

how to instrument it?