Atoms natively support message queuing. Messages are managed by a shared queue server embedded in the Atom Java process. This approach enables simple, reliable asynchronous process execution whereby a process invokes another process in a completely separate invocation.


The messaging system is implemented as follows:

  • Reusable queue components are configured at the account level. Each queue component specifies the configuration of a message queue, including its name and the messaging model with which the message queue can be used, either Point-to-Point or Publish/Subscribe.

  • An Atom’s shared queue server creates a message queue upon invocation of an Atom Queue connector Get, Send or Listen operation that specifies a queue component.

  • Messages consist of references to documents rather than actual document content. Each message references one or more documents. The document metadata in messages includes dynamic document properties.

  • Messages persist on a message queue until one of the following occurs:

    • The message is consumed.

    • The documents that the message references are purged.


The messaging system supports all Atom types including Molecules, which enable clustering, and Atom Clouds, which enable multi-tenancy.

To use the messaging system, you need to

  1. Create queue components.

  2. Configure the Atom’s shared queue server, if you are the Atom’s owner. The default configuration is likely to be suitable for your purposes, at least initially.

  3. Build and deploy processes that use Atom Queue connector operations that specify the queue components you created.

  4. Perform message queue management actions as needed.


While Atom message queuing is simple, robust and reliable, it lacks many of the features of an enterprise queuing system, such as message priorities and variable persistence guarantees. If Atom message queuing does not meet your requirements, consider installing your own messaging system and using the JMS connector.


Note: Atom message queuing is an optional feature. To have this feature enabled in your account, contact your Dell Boomi sales representative.


Benefits


The benefits of having a native message queuing implementation can be categorized as follows:

  • Asychronous communication — Processes writing and reading data execute independently of each other. Requests are batched in real-time. Enqueuing requests is more efficient than spawning an individual execution for each real-time message or writing messages to disk for later batch processing in a scheduled process.

  • Decoupling — Producers and consumers of messages are independent and can evolve separately at their own rate. Workflow is encapsulated into logical, reusable units — that is, separate processes — which are organized, maintained and deployed independently.

  • Multiple recipients — Messages can be sent to multiple recipients independently. Likewise, their receipt can be monitored independently.

  • Redundancy — Message queues can persist messages until they are fully processed.

  • Resiliency — Decoupling implies failures are not linked, thereby mitigating the risk of unreliable applications. A producer can continue to enqueue messages while consumers are temporarily disabled.


Typical usage scenarios

Following are typical usage scenarios for Atom message queuing employing message queues:

  • Consider a requirement for fully disconnected process execution. This requirement is common in the case of services with known reliability issues. Having a separate processing path enables more granular retries. In this scenario an Atom Queue connector Send operation would send failed documents to a Point-to-Point message queue in batches. In another process an Atom Queue connector operation — either Listen or scheduled Get — would receive the batches.

  • Consider a requirement for aggregate AS2 document processing — a batch process operating upon separate incoming documents. In this scenario the AS2 Shared Server connector would listen for incoming documents, and an Atom Queue connector Send operation would send them to a Point-to-Point message queue. In another process a scheduled Atom Queue connector Get operation would receive documents in batches.

  • Consider a requirement for dispersed document processing — incoming documents processed in parallel, with failed documents retried individually. In this scenario a primary process would have an Atom Queue connector Send operation to send documents to a Point-to-Point message queue in small batches, in some cases a single document. In another process an Atom Queue connector Listen operation would receive the batches and in subsequent steps route the documents for concurrent processing.

  • Consider a requirement to route messages between Atoms.

    • The server Atom would act much like an enterprise service bus (ESB). It would deploy two Web Services Server processes, one using an Atom Queue connector Send operation to send documents to a Point-to-Point message queue for consumption by client Atoms and the other using an Atom Queue connector Get operation to receive documents sent by client Atoms from a Point-to-Point message queue.

    • Each client Atom would deploy two Web Services SOAP or HTTP client processes, one using an Atom Queue connector Get operation to receive documents sent by the server Atom from a Point-to-Point message queue and the other using an Atom Queue connector Send operation to send documents to a Point-to-Point message queue for consumption by the server Atom.

  • Consider a requirement for a hub and spoke system in which documents are produced in the hub and made available for consumption on a variable population of spokes. In this scenario the publisher (hub) would have a scheduled process that executes an Atom Connector Send operation to send documents to a Publish/Subscribe message queue. At any given time the message queue would have zero or more subscribing message queues (spokes). Each subscriber would be executing a process in which an Atom Queue connector Listen operation would receive published documents. Because subscribers are unknown to the publisher, they can activate or deactivate at any time without necessitating a modification to the publishing process.


Limitations

Atom message queuing is subject to the following limitations:

  • Messages cannot be sent between accounts.

  • Messages cannot be sent between Atoms.

  • Message queues cannot be directly accessed from outside the Atom.


Listener management

You can view the status of listener processes that are deployed to an Atom, Molecule, or Atom Cloud to retrieve messages from a message queue by going to the Listeners panel in Manage > Atom Management. In this panel, you can also pause, resume, and restart listeners.


Dead letter queues

Dead letter queues contain undeliverable messages. Each ordinary message queue can have one associated dead letter queue.

Message delivery failure commonly occurs because of

  • network failure

  • purging of referenced documents

  • general message processing failure


When a failure occurs, the shared queue server will attempt to redeliver the message up to six more times, in intervals of 1, 2, 3, 5, 8, and 24 seconds. After six failures the message is sent to the appropriate dead letter queue. If the dead letter queue does not yet exist it is automatically created.

You should periodically analyze the contents of dead letter queues to identify the reasons for delivery failures.


Data storage

An Atom’s shared queue server’s backing disk store is as follows:

  • For a single tenant Atom or Molecule, the shared queue server’s runtime configuration information and message queues reside in the “queue” directory within the Atom installation directory.

  • For a multi-tenant Atom Cloud, the shared queue server’s runtime configuration information resides in the “queue” directory within the Atom installation directory. Message queues, on the other hand, reside in the “queue” directory within each account’s directory.

Note: Documents referenced in currently enqueued messages persist until a scheduled purge occurs. This is true even while the Atom or process is configured to enable purging of data immediately following process execution. When documents referenced in an enqueued message are purged, the message itself is also purged.


Monitoring the message queue service

The following metrics are available for monitoring an Atom’s message queue service:

  • overall status

  • message store disk usage

  • temporary data store disk usage

  • job scheduler store disk usage

  • memory usage


In order to monitor these attributes, you need to use a systems management tool, such as Zabbix, and use a JMX hook (a standardized interface for Java programs) — see the topic about using JMX to monitor your system, linked below.