Processing Message Queues with Java Messaging Service (JMS)

Owned by Former user (Deleted)
Last updated: Dec 04, 2015 by Matt TuckerVersion comment

Overview

With JMS it is possible to process messages in two basic methods:

Queues
A message can be send to a specified JMS Queue and can be received by exactly one consumer. If there are no consumers available at the time the message is sent it will be kept until a consumer is available that can process the message. This is also know as a point-to-point model.

Topics
In JMS a Topic implements publish and subscribe semantics. When you publish a message it goes to all the subscribers who are interested - so zero to many subscribers will receive a copy of the message. Only subscribers who had an active subscription at the time the broker receives the message will get a copy of the message. This is also known as a publish/subscribe model.

Sample solution

In the following sample solution we use the JMS Consumer and JMS Producer steps that are delivered as part of the Pentaho Data Integration Enterprise Edition and use Apache ActiveMQ as a JMS test server.

This was tested with PDI EE Version 4.2 GA and ActiveMQ version 5.5.1

Please see also the Knowledge Base for reference documentation about the JMS Consumer and JMS Producer steps.

Setup you test solution

  1. Download ActiveMQ from the Apache site.
  2. Start the ActiveMQ server, e.g. by executing apache-activemq-5.5.1-bin\bin\activemq.bat
  3. After the server startup, you can connect with your web browser to the default port, e.g. http://localhost:8161/admin/ to see if the server is up and running correctly.
  4. Have a PDI EE version available. If you don't have an Enterprise Edition, yet, an evaluation version can be downloaded.
  5. Download the attached sample transformations.

Testing QUEUES

Execute the transformation JMS Producer - QUEUE.ktr

This transformation creates 10 test messages and puts these into the message server queue test_queue.

You can see that the queue is created and the messages have arrived at the server, you can monitor it by http://localhost:8161/admin/queues.jsp

Now retrieve the messages by the transformation JMS Consumer - QUEUE.ktr

After the execution, you see that the messages have been dequeued and are no longer pending:

You can redo this multiple times and can also do a preview on the Dummy step or process this further, e.g. for the first five rows:

Testing TOPICS

The ongoing for the topics is pretty much the same, but you need to start the consumer first! It is also possible to start multiple consumers as subscribers.

  1. Start the JMS Consumer - TOPIC.ktr
  2. Keep it running as a subscriber
  3. Start the JMS Producer - TOPIC.ktr
  4. You may monitor the topics in the Topics browser section or look at the file JMS_Topics_Received.xml (after you stopped the JMS Consumer - TOPIC transformation since the file is written with delay).

Notes on Transactional Behavior

The transactional behavior is set to AUTO_ACKNOWLEDGE by default in the JMS steps: With this acknowledgment mode, the session automatically acknowledges a client's receipt of a message either when the session has successfully returned from a call to receive or when the message listener the session has called to process the message successfully returns.

This means, producers and consumers are decoupled from each other and these messages are loosely coupled and asynchronous.

By default, the messages are always removed from the queue after a successful receipt of the message by the JMS client - Pentaho Data Integration in our case. We are working toward a means of delaying the removal of the message until an acknowledgment has been made by the client. This feature is NOT in this release at the time of this writing, please see PDI-7403 for the feature request.

References