Processing Message Queues with IBM Websphere MQ
- Former user (Deleted)
- Former user (Deleted)
- Former user (Deleted)
Overview
IBM Websphere MQ is a messaging and queuing middleware provider that can be leveraged via the Java Message Service (JMS) specification. In this manner, Pentaho Data Integration exposes IBM MQ Consumer and Producer transformation steps allowing access to produce and consume messages through IBM Websphere MQ. IBM Websphere MQ provides this functionality through IBM's implementation of JMS, called the Websphere MQ classes for JMS.
As the MQ steps are layered atop the JMS Consumer and Producer transformation steps, there is overlap and great similarities between the documentation for JMS and IBM Websphere MQ. For the sake of usable documentation, some of the JMS documentation is repeated here.
With JMS (and as such, also with IBM Websphere MQ) it is possible to process messages in two basic methods:
Queues
A message can be sent to a specified MQ Queue and can be received by exactly one consumer. If there are no consumers available at the time the message is sent it will be kept until a consumer is available that can process the message. This is also know as a point-to-point model.
Topics
In Websphere MQ a Topic implements publish and subscribe semantics. When you publish a message it goes to all the subscribers who are interested - so zero to many subscribers will receive a copy of the message. Only subscribers who had an active subscription at the time the broker receives the message will get a copy of the message. This is also known as a publish/subscribe model.
Installation
You need to locate the Websphere MQ classes for JMS Java libraries from your IBM Websphere MQ installation. You can also find these libraries in your IBM Websphere MQ Client SupportPac. The Websphere MQ Java libraries version that the PDI plugin steps were built against is 7.0.1.9. The libraries that you must have available for distribution into the PDI JMS plugin are:
- com.ibm.mq.jar
- com.ibm.mqjms.jar
- com.ibm.mq.jmqi.jar
- dhbcore.jar
IBM licensing prevents Pentaho from distributing these libraries directly, hence the need to add them post install of Pentaho Data Integration.
To install the plugin to Pentaho Data Integration Spoon client:
- Make sure Spoon is shut down.
- Navigate your PDI installation to <pdi-install>/data-integration/plugins directory.
- Add the IBM Websphere MQ Java libraries listed above to the <pdi-install>/data-integration/plugins/pdi-jms-plugin/lib directory.
To install the plugin to the Pentaho Data Integration Server:
Follow the steps above. The location of the pdi-jms-plugin directory is different; look for the plugin directory under <pdi-server-install>/data-integration-server/pentaho-solutions/system/kettle/plugins directory.
To install the plugin to the Pentaho Business Analytics Server:
Follow the steps above. The location of the pdi-jms-plugin directory is different; look for the plugin directory under <ba-server-install>/biserver-ee/pentaho-solutions/system/kettle/plugins directory.
The following set of sample transformations should help get you started using your new messaging steps.
Sample Solution
In the following sample solution we use the Websphere MQ Consumer and Websphere MQ Producer steps and use Websphere MQ Version 7.0 http://activemq.apache.org as a Websphere MQ server.
This was tested with PDI EE Version 4.3 GA and Websphere MQ Version 7.0.
Setup for the sample solutions:
- Start up your IBM Websphere MQ Server.
- Have a PDI EE version 4.3.0GA or above available. If you don't have an Enterprise Edition, yet, an evaluation version can be downloaded.
- Download the attached sample transformations.
Testing QUEUES
Execute the transformation MQ Producer - QUEUE.ktr
This transformation creates 10 test messages and puts these into the message server queue JMSQueue.
You can see that the queue is created and the messages have arrived at the server. We used IBM MQExplorer to monitor our sample server.
Now retrieve the messages by the transformation MQ Consumer - QUEUE.ktr
After the execution, the messages are removed from the queue. See further commentary on this in the Notes on Transactional Behavior section.
Testing TOPICS
The ongoing for the topics is pretty much the same, but you need to start the consumer first! It is also possible to start multiple consumers as subscribers.
- Start the MQ Consumer - TOPIC.ktr
- Keep it running as a subscriber. You can have an indefinite subscriber by setting the receive timeout for the MQ Consumer step to 0 (zero means "never timeout").
- Start the JMS Producer - TOPIC.ktr
- You may monitor the topics in IBM MQExplorer or look at the file JMS_Topics_Received.xml. (Rows are written to this file only after you've stopped the MQ Consumer - TOPIC transformation, since the file is written with delay).
Notes on Transactional Behavior
The transactional behavior is set to AUTO_ACKNOWLEDGE by default in the IBM Websphere MQ steps: With this acknowledgment mode, the session automatically acknowledges a client's receipt of a message either when the session has successfully returned from a call to receive or when the message listener the session has called to process the message successfully returns.
This means, producers and consumers are decoupled from each other and these messages are loosely coupled and asynchronous.
By default, the messages are always removed from the queue after a successful receipt of the message by the JMS client - Pentaho Data Integration in our case. We are working toward a means of delaying the removal of the message until an acknowledgment has been made by the client. This feature is NOT in this release at the time of this writing, please see PDI-7403 for the feature request.