JMS Clustering in WildFly and JBoss EAP

In this tutorial we will learn how to achieve High Availability in JMS clustering, comparing the configuration available with Hornetq messaging (AS 7 - WildFly 8 - JBoss EAP 6) and Artemis-mq (WildFly 10 - JBoss EAP 7)

In order to achieve high availability, you have the choice between using a shared store and data replication:

AWhen using a shared store, both live and backup servers share the same entire data directory using a shared file system. This means the paging directory, journal directory, large messages and binding journal. When failover occurs and a backup server takes over, it will load the persistent storage from the shared file system and clients can connect to it.

BWhen using data replication, the live and the backup servers do not share the same storage and all data synchronization is done through network traffic. Therefore, all (persistent) data traffic received by the live server will be duplicated to the backup.

HornetQ clustering

As most of the cluster configurations are done in Domain mode we will show both examples using a Managed Domain. The tweak is that, in order to discriminate between active nodes and backup nodes, using the same Profile, we need to use System Properties which will be overridden at Server level.

Shared Storage

Here is a sample Shared Storage configuration:

<paths>
        <path name="hornetq.shared.dir" path="/home/journal-A/bindings"/>
</paths>
. . .
    <subsystem xmlns="urn:jboss:domain:messaging:1.4">
        <hornetq-server>
            <persistence-enabled>true</persistence-enabled>
            <cluster-password>password</cluster-password>
            <backup>${jboss.messaging.hornetq.backup:false}</backup>
            <failover-on-shutdown>true</failover-on-shutdown>
            <shared-store>true</shared-store>
            <create-bindings-dir>true</create-bindings-dir>
            <create-journal-dir>true</create-journal-dir>
            <journal-type>NIO</journal-type>
            <journal-min-files>2</journal-min-files>
            <paging-directory path="paging" relative-to="hornetq.shared.dir"/>
            <bindings-directory path="bindings" relative-to="hornetq.shared.dir"/>
            <journal-directory path="journal" relative-to="hornetq.shared.dir"/>
            <large-messages-directory path="large-messages" relative-to="hornetq.shared.dir"/>
          ....
         </hornetq-server>
     </subsystem>

The key properties of this configuration are:

  • backup: true: marks a node as backup node. false: marks the server as an active node
  • shared-store: true : enables Shared Store. false: Uses data replication

Some other properties are the following ones:

  • failover-on-shutdown: true: allows failover to occur during “normal” shutdown. If false, failover will only occur on unexpected loss of node
  • allow-failback: true: a backup that is promoted to active will give control back if the original active node comes back online. If false, the backup node will remain the master node even if the original active node comes back.

In order to discriminate between the Active and Backup node, we will set the jboss.messaging.hornetq.backup property in the host.xml file. Here is the first server:

<server name="server-1" group="main-server-group" auto-start="true">
 	<system-properties>
      		<property name="jboss.messaging.hornetq.backup" value="false"/>
   	</system-properties>
       <socket-bindings port-offset="0"/>
</server>

Then on the other server jboss.messaging.hornetq.backup should be set to "true"

<server name="server-2" group="main-server-group" auto-start="true">
 	<system-properties>
      		<property name="jboss.messaging.hornetq.backup" value="true"/>
   	</system-properties>
         <socket-bindings port-offset="100"/>
</server>

Please notice that, for high availability purposes, the live server and the backup server must be installed on two separated physical (or virtual) hosts, provisioned in such a way to minimize the probability of both host failing at the same time. Also, highly available HornetQ requires access to reliable shared file system storage, so a file system such as GFS2 or a SAN must be made available to both hosts. HornetQ instances will store on the shared directory, among other things, their bindings and journal files. NFS v4 appropriately configured is also an option.

Data Replication

Here is a sample Data Replication configuration:

<subsystem xmlns="urn:jboss:domain:messaging:1.4">

    <hornetq-server>
	    <persistence-enabled>true</persistence-enabled>
	    <cluster-password>password</cluster-password>
	    <backup>${jboss.messaging.hornetq.backup:false}</backup>
	    <failover-on-shutdown>true</failover-on-shutdown>
	    <shared-store>false</shared-store>
	    <create-bindings-dir>true</create-bindings-dir>
	    <create-journal-dir>true</create-journal-dir>
	    <journal-type>NIO</journal-type>
	    <journal-min-files>2</journal-min-files>
	    <check-for-live-server>true</check-for-live-server>
	    <backup-group-name>NameOfLiveBackupPair</backup-group-name>
	   ....
  </hornetq-server>
</subsystem>

As you can see, the main difference is that we have setup the shared-store to false. Another property, named backup-group-name, links the active and backup servers together for replication. You need to configure the jboss.messaging.hornetq.backup property on the single nodes as we showed in the shared storage example

WildFly 10 - ArtemisMQ

The new messaging provider embedded in WildFly 10 is Apache Artemis MQ which is derived from the HornetQ project, recently donated to the Apache foundation Master. The mechanism for achieving high availability is the same, however a different subsystem (with different elements) will be used.

Shared Storage

Here is a sample Shared Storage configuration:

<profile name="full-ha-master">
        . . . .
        <subsystem xmlns="urn:jboss:domain:messaging-activemq:1.0">
            <server name="default">
                <cluster user="foo" password="bar"/>  
                <security-setting name="#">
                    <role name="guest" send="true" consume="true" create-non-durable-queue="true" delete-non-durable-queue="true"/>
                </security-setting>
                
                <shared-store-master failover-on-server-shutdown="true"/>
                
                <bindings-directory path="/home/journal-A/bindings"/>
                <journal-directory path="/home/journal-A/journal"/>
                <large-messages-directory path="/home/journal-A/largemessages"/>
                <paging-directory path="/home/journal-A/paging"/>
                <security-setting name="#">
                    <role name="guest" delete-non-durable-queue="true" create-non-durable-queue="true" consume="true" send="true"/>
                </security-setting>
                <address-setting name="#" redistribution-delay="0" page-size-bytes="524288" max-size-bytes="1048576" max-delivery-attempts="200"/>
                <http-connector name="http-connector" endpoint="http-acceptor" socket-binding="http"/>
                <http-connector name="http-connector-throughput" endpoint="http-acceptor-throughput" socket-binding="http">
                    <param name="batch-delay" value="50"/>
                </http-connector>
                <in-vm-connector name="in-vm" server-id="0"/>
                <http-acceptor name="http-acceptor" http-listener="default"/>
                <http-acceptor name="http-acceptor-throughput" http-listener="default">
                    <param name="batch-delay" value="50"/>
                    <param name="direct-deliver" value="false"/>
                </http-acceptor>
                <in-vm-acceptor name="in-vm" server-id="0"/>
                <broadcast-group name="bg-group1" connectors="http-connector" jgroups-channel="activemq-cluster"/>
                <discovery-group name="dg-group1" refresh-timeout="1000" jgroups-channel="activemq-cluster" jgroups-stack="udp"/>
                <cluster-connection name="my-cluster" discovery-group="dg-group1" connector-name="http-connector" address="jms"/>
                <jms-queue name="ExpiryQueue" entries="java:/jms/queue/ExpiryQueue"/>
                <jms-queue name="DLQ" entries="java:/jms/queue/DLQ"/>
                <connection-factory name="InVmConnectionFactory" entries="java:/ConnectionFactory" connectors="in-vm"/>
                <connection-factory name="RemoteConnectionFactory" reconnect-attempts="-1" retry-interval-multiplier="1.0" retry-interval="1000" block-on-acknowledge="true" ha="true" entries="java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector"/>
                <pooled-connection-factory name="activemq-ra" transaction="xa" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory" connectors="in-vm"/>
            </server>
        </subsystem>
   . . .
</profile>

As you can see, the main difference with HornetQ clustering is that we had to use a Profile instead of System Properties for setting up the Active/Backup node, as a matter of fact in ArtemisMQ the choice between Active and Backup is made at XML Element level. Here is the corresponding Backup node profile:

<profile name="full-ha-slave">
        . . . .
  <subsystem xmlns="urn:jboss:domain:messaging-activemq:1.0">
            <server name="default">
                <cluster user="foo" password="bar"/>  
                <security-setting name="#">
                    <role name="guest" send="true" consume="true" create-non-durable-queue="true" delete-non-durable-queue="true"/>
                </security-setting>

                <shared-store-slave failover-on-server-shutdown="true"/>

                <bindings-directory path="/home/journal-A/bindings"/>
                <journal-directory path="/home/journal-A/journal"/>
                <large-messages-directory path="/home/journal-A/largemessages"/>
                <paging-directory path="/home/journal-A/paging"/>
                <security-setting name="#">
                    <role name="guest" delete-non-durable-queue="true" create-non-durable-queue="true" consume="true" send="true"/>
                </security-setting>
                <address-setting name="#" redistribution-delay="0" page-size-bytes="524288" max-size-bytes="1048576" max-delivery-attempts="200"/>
                <http-connector name="http-connector" endpoint="http-acceptor" socket-binding="http"/>
                <http-connector name="http-connector-throughput" endpoint="http-acceptor-throughput" socket-binding="http">
                    <param name="batch-delay" value="50"/>
                </http-connector>
                <in-vm-connector name="in-vm" server-id="0"/>
                <http-acceptor name="http-acceptor" http-listener="default"/>
                <http-acceptor name="http-acceptor-throughput" http-listener="default">
                    <param name="batch-delay" value="50"/>
                    <param name="direct-deliver" value="false"/>
                </http-acceptor>
                <in-vm-acceptor name="in-vm" server-id="0"/>
                <broadcast-group name="bg-group1" connectors="http-connector" jgroups-channel="activemq-cluster"/>
                <discovery-group name="dg-group1" refresh-timeout="1000" jgroups-channel="activemq-cluster" jgroups-stack="udp"/>
                <cluster-connection name="my-cluster" discovery-group="dg-group1" connector-name="http-connector" address="jms"/>
                <jms-queue name="ExpiryQueue" entries="java:/jms/queue/ExpiryQueue"/>
                <jms-queue name="DLQ" entries="java:/jms/queue/DLQ"/>
                <connection-factory name="InVmConnectionFactory" entries="java:/ConnectionFactory" connectors="in-vm"/>
                <connection-factory name="RemoteConnectionFactory" reconnect-attempts="-1" retry-interval-multiplier="1.0" retry-interval="1000" block-on-acknowledge="true" ha="true" entries="java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector"/>
                <pooled-connection-factory name="activemq-ra" transaction="xa" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory" connectors="in-vm"/>
            </server>
        </subsystem> 
   . . .
</profile>

You will then assign each Profile to a Server group, so you will have a Master Server Group and a Backup Server Group:

<server-groups>
        <server-group name="master-server-group" profile="full-ha-master">
            <socket-binding-group ref="full-ha-sockets"/>
        </server-group>
        <server-group name="backup-server-group" profile="full-ha-slave">
            <socket-binding-group ref="full-ha-sockets"/>
        </server-group>
</server-groups>

Data Replication

Finally, here is a Profile which can be used for Data Replication on ArtemisMQ. At first the Replication Master Profile:

<profile name="full-ha-master">
      . . . .
      <subsystem xmlns="urn:jboss:domain:messaging-activemq:1.0">
            <server name="default">
                <cluster user="foo" password="bar"/>  
                <security-setting name="#">
                    <role name="guest" send="true" consume="true" create-non-durable-queue="true" delete-non-durable-queue="true"/>
                </security-setting>
                <address-setting name="#" dead-letter-address="jms.queue.DLQ" expiry-address="jms.queue.ExpiryQueue" max-size-bytes="10485760" page-size-bytes="2097152" message-counter-history-day-limit="10" redistribution-delay="1000"/>
                <http-connector name="http-connector" socket-binding="http" endpoint="http-acceptor"/>
                <http-connector name="http-connector-throughput" socket-binding="http" endpoint="http-acceptor-throughput">
                    <param name="batch-delay" value="50"/>
                </http-connector>
                <in-vm-connector name="in-vm" server-id="0"/>
                <http-acceptor name="http-acceptor" http-listener="default"/>
                <http-acceptor name="http-acceptor-throughput" http-listener="default">
                    <param name="batch-delay" value="50"/>
                    <param name="direct-deliver" value="false"/>
                </http-acceptor>
                <in-vm-acceptor name="in-vm" server-id="0"/>
                <broadcast-group name="bg-group1" connectors="http-connector" jgroups-channel="activemq-cluster"/>
                <discovery-group name="dg-group1" jgroups-channel="activemq-cluster"/>
                <cluster-connection name="my-cluster" address="jms" connector-name="http-connector" discovery-group="dg-group1"/>
                <jms-queue name="ExpiryQueue" entries="java:/jms/queue/ExpiryQueue"/>
                <jms-queue name="DLQ" entries="java:/jms/queue/DLQ"/>

                <replication-master/>

                <connection-factory name="InVmConnectionFactory" connectors="in-vm" entries="java:/ConnectionFactory"/>
                <connection-factory name="RemoteConnectionFactory" ha="true" block-on-acknowledge="true" reconnect-attempts="-1" connectors="http-connector" entries="java:jboss/exported/jms/RemoteConnectionFactory"/>
                <pooled-connection-factory name="activemq-ra" transaction="xa" connectors="in-vm" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory"/>
            </server>
        </subsystem>
</profile>

Next, here is a Profile which can be used for Replication Slaves Profile:

<profile name="full-ha-slave">
      . . . .
    <subsystem xmlns="urn:jboss:domain:messaging-activemq:1.0">
            <server name="default">
                <cluster user="foo" password="bar"/>  
                <security-setting name="#">
                    <role name="guest" send="true" consume="true" create-non-durable-queue="true" delete-non-durable-queue="true"/>
                </security-setting>
                <address-setting name="#" dead-letter-address="jms.queue.DLQ" expiry-address="jms.queue.ExpiryQueue" max-size-bytes="10485760" page-size-bytes="2097152" message-counter-history-day-limit="10" redistribution-delay="1000"/>
                <http-connector name="http-connector" socket-binding="http" endpoint="http-acceptor"/>
                <http-connector name="http-connector-throughput" socket-binding="http" endpoint="http-acceptor-throughput">
                    <param name="batch-delay" value="50"/>
                </http-connector>
                <in-vm-connector name="in-vm" server-id="0"/>
                <http-acceptor name="http-acceptor" http-listener="default"/>
                <http-acceptor name="http-acceptor-throughput" http-listener="default">
                    <param name="batch-delay" value="50"/>
                    <param name="direct-deliver" value="false"/>
                </http-acceptor>
                <in-vm-acceptor name="in-vm" server-id="0"/>
                <broadcast-group name="bg-group1" connectors="http-connector" jgroups-channel="activemq-cluster"/>
                <discovery-group name="dg-group1" jgroups-channel="activemq-cluster"/>
                <cluster-connection name="my-cluster" address="jms" connector-name="http-connector" discovery-group="dg-group1"/>
                <jms-queue name="ExpiryQueue" entries="java:/jms/queue/ExpiryQueue"/>
                <jms-queue name="DLQ" entries="java:/jms/queue/DLQ"/>

                <replication-slave allow-failback="true"/> 

                <connection-factory name="InVmConnectionFactory" connectors="in-vm" entries="java:/ConnectionFactory"/>
                <connection-factory name="RemoteConnectionFactory" ha="true" block-on-acknowledge="true" reconnect-attempts="-1" connectors="http-connector" entries="java:jboss/exported/jms/RemoteConnectionFactory"/>
                <pooled-connection-factory name="activemq-ra" transaction="xa" connectors="in-vm" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory"/>
            </server>
        </subsystem>

As for the Shared Storage, you will have to assign each Profile to a Server group, so you will have a Master Server Group and a Backup Server Group:

<server-groups>
        <server-group name="master-server-group" profile="full-ha-master">
            <socket-binding-group ref="full-ha-sockets"/>
        </server-group>
        <server-group name="backup-server-group" profile="full-ha-slave">
            <socket-binding-group ref="full-ha-sockets"/>
        </server-group>
</server-groups>

That's all! Enjoy High Availability on your JBoss EAP or WildFly messaging subsystem!

Related articles available on mastertheboss.com

How to configure a Queue in JBoss ?

This article has been moved here: JBoss JMS configuration

How to create a Queue with Jmx Console ?

  Bring up the JMX Console in your browser and look for the sect

JBoss JMS Queue example

The following article shows how to create a simple JMS Queue Prod

JBoss JMS Topic example

The following article shows how to create a simple JMS Topic Publ

JBoss HornetQ simple tutorial

HornetQ is an open source project to build a multi-protocol, embe

How do I configure a Queue/Topic to work in a cluster?

  JBoss AS 5 Just set the Clustered attribute to "true" in your

Follow us on Twitter