Flow Activity Monitoring : Logging, Analazying and Monitoring Data of our integration flows camel using ElasticSearch, Logstash and Kibana.

Many times we receive requests like: Where's my message? Are you sur that my result has been routed to the good destination ? how many orders has been received from our partner? 

All monitoring solutions gives only part of answer. We have experienced the use of wireTap in our camel routes, but we have to design a module that consumes these messages and stock them into Database.

The Deal for us was : how we can design our flows without thinking to the monitoring ? how we can audit our integration flows already in production without any impact?

I think that the solution will be : Flow Activity Monitoring

1. Each camel flow deployed in JBoss Fuse are audited by Fuse BAI [REF-1], You have to develop a flow Backend BAI that consumes from endpoint vm:audit and log the content of the event using camel mdc logging. 
2. You have to setup a custom logging appender in JBoss Fuse in order to create a specific log for each camelContext deployed: 

e.g:
# Appender App 
log4j.appender.app = org.apache.log4j.sift.MDCSiftingAppender 
log4j.appender.app.key = camel.contextId 
log4j.appender.app.default = unknown 
log4j.appender.app.appender = org.apache.log4j.RollingFileAppender 
log4j.appender.app.appender.layout = org.apache.log4j.PatternLayout 
log4j.appender.app.appender.layout.ConversionPattern =% d {ISO8601} |%-5.5p |% X {camel.contextId} |% {X} camel.routeId |% X {camel.exchangeId} |% m% n 
log4j.appender.app.appender.file karaf.home = $ {} / log/apps/mediation- $ \ \ {camel.contextId \ \}. log 
log4j.appender.app.appender.append = true 
log4j.appender.app.appender.maxFileSize = 1MB 
log4j.appender.app.appender.maxBackupIndex = 10 

3. logstash [REF-2] listening to all logs in the follwing directory ${karaf.home}/log/apps/  ,parse and filter the message to header fields, Properties fields, Body, breadcrumb..etc, after we sent the output to ElasticSearch [REF-3].
4. Kibana for good graphic representation ;) 

Here's the schema for the solution :

Kibana dashboard :

Conclusion:
- Your mission-critical projects need management and monitoring. Today, is just possible using open source products: logstash, Elasticsearch and Kibana.

- The future previous version of the JBoss Fuse 6.1 allow to send automatically all information of the message to Elasticsearch by enabling insight-camel, Though the downside is this assumes fabric and 6.1;

Best regards,
Abdellatif BOUCHAMA (@a_bouchama)

• fr.linkedin.com/in/abouchama/

References
[REF-1] - https://github.com/jboss-fuse/fuse/tree/master/bai
[REF-2] - http://logstash.net/
[REF-3] - http://www.elasticsearch.org/

What's new in JMS 2.0 ?

In 2002 came out the first stable version of JMS version 1.1. More than a decade later, the specification has become widely used and today there are multiple implementations of open-source and commercial time.

Yet in eleven years, many vendors have developed new capabilities over those provided by JMS. 

Now the JMS 2.0 was finally released on 21 May 2013, Let's look at all the new features :

- DeliveryDelay

A property named DeliverDelay will be added in the JMS message header. This will enable a producer to specify a time interval before a message is delivered to a customer.

This property can be useful in delayed treatment, at the end of the day, for example.

- Sending a message asynchronously

The client calls the method send() and the API  makes her hand once the message has been sent successfully. With version 2.0, it will be possible to call this method asynchronously and have the hand immediately. The API then invoke a callback onAcknowledge () to indicate that the message has been sent.

The decision to use this feature will be on a case by case basis. My service of sending he needs to be more scalable as possible? Am I able to replay a flow error if it is detected asynchronously?

- JMSXDeliverCount becomes mandatory

The JMSXDeliverCount property to specify the number of times that a message must be delivered was optional in version 1.1. It becomes mandatory with version 2.0.

This non backward compatibility only affects the JMS provider, customers themselves are not affected by this change.

- Hierarchy of topics

A very interesting feature is the possibility to prioritize topics.

Suppose we have the following four topics:

CLIENTS.FRANCE.ACHAT
CLIENTS.FRANCE.VENTE
CLIENTS.USA.ACHAT
CLIENTS.USA.VENTE

It will be possible to use patterns for subscription JMS clients.

If you want to subscribe to all topics relating to French customers:

CLIENTS.FRANCE. *

Or all purchases:

CUSTOMERS. *. PURCHASE

This feature uses very well require the use of good governance to manage the naming of topics and types of data conveyed.

- Multiple customers on the same durable subscriber

Currently on a JMS queue, it is possible to have multiple consumers. The JMS provider use an algorithm to round-robin for load-balancing between the different customers.

But for JMS topics, it is not currently possible to have only one consumer in sustainable subscriber.
With version 2.0 it is possible to connect multiple clients on the same durable subscriber. The mechanism of round-robin will be also applicable to the topics to allow for better scalability.

- Batch mode

Although already implemented by some providers as webMethods JMS Broker, this feature will receive messages in batch mode. Namely the receipt by a JMS client several messages at once.

In 1.1, we had the following method:
void onMessage (Message message);

In 2.0 we have:
void onMessages (Message [] message);

Conclusion

JMS 2.0 proposes some very useful new features. The API will be greatly simplified and it will also allow the use of CDI for dependency injection.

High Availability for WebSphere Message Broker & Websphere ESB

It’s important to consider how to ensure the performance and availability targets for a WebSphere Message Broker implementation are met. As the most common implementation of Message Broker is as a central hub that all messages in a messaging architecture are processed though, it is a potential bottleneck and single point of failure for the whole WebSphere MQ Queue Manager network.

Below the topology that I proposed and implemented in a lot of clients:

The diagram shows the topology. It consists of:

1. Two brokers BK.1 and BK.2. Each broker executes a specific set of execution groups that run each process in a specific set of flows parallelized or not.

2. Two Queue Manager WMB.BK1 and WMB.BK.2 corresponding respectively to BK.1 and BK.2 brokers.

3. A DB2 database dedicated hosting the configuration repository shared brokers.

4. The configuration manager CM to administer the two brokers.

5. The queue manager WMB.CM dedicated to configuration manager

We have also to consider the following points that contribute to high availability of any ESB environment. For example:

• Reliable hardware
• Shared queues
• Heath monitoring
• Failover clustering
• Online backup
• Dual networks
• Reliable operating system
• Online reconfiguration
• Fast reboot
• RAID disks
• Crash recovery
• Fast start up
• IP takeover
• Documented procedures
• Practicing procedures

Lightweight ESB or ESB?

Lately, we see a lot of lightweight ESBs that begins to emerge, we can already list some solutions:, Apache Camel, MuleESB, Spring integration, WebSphere Message Broker.. Etc

The Deployment of lightweight ESB can be done in a container like JBoss AS, Jetty, Tomcat, WebSphere Application Server..etc, and it can be also in a message broker, which's directly embedded like Camel that is embedded by default in ActiveMQ.

The question that arises, which brick the lightweight ESB covers ?

The main features that one would expect from a lightweight ESB are:

• Connectivity
• Routing
• Mediation (Transformation, EIP patterns, Orchestration)
• Monitoring

Concerning the orchestration, only the technical orchestration is covered. Sometimes we use the lightweight ESBs for aggregation services, or enrichment of data, but it must be remembered that we should not regard the lightweight ESB as an orchestrator of process.

Idem for monitoring, only the technical monitoring is covered.

A lightweight ESB alone, does not setup the BPM or BAM.

Fuse ESB : How to configure Fuse ESB for authentication against an Active Directory server

Fuse ESB supplies a JAAS login module that enables it to use LDAP to authenticate users.

This tutorial will be focused on how we configure Fuse ESB for authentication against an Active Directory server.

The example defines a JAAS realm that uses the Active Directory server located at ldap://localhost:389 :

View of ldap-module.xml file to be copied into the Fuse ESB deploy/ directory:

<?xml version="1.0" encoding="UTF-8"?>

<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"

  xmlns:jaas="http://karaf.apache.org/xmlns/jaas/v1.0.0"

  xmlns:ext="http://aries.apache.org/blueprint/xmlns/blueprint-ext/v1.0.0">

  <jaas:config name="karaf" rank="1">

    <jaas:module className="org.apache.karaf.jaas.modules.ldap.LDAPLoginModule" flags="required">

      initialContextFactory=com.sun.jndi.ldap.LdapCtxFactory

      connection.username=user_service@company.com

      connection.password=password

      connection.url=ldap://localhost:389

      user.base.dn= OU=Users,DC=company,DC=com

      user.filter=(samAccountName=%u)

      user.search.subtree=true

      role.base.dn=OU=Groups,OU,DC=company,DC=com

      role.name.attribute=cn

      role.filter=(sAMAccountName=Operator)

      role.search.subtree=true

      authentication=simple

    </jaas:module>

  </jaas:config>

</blueprint>

NB: You have to change the value of karaf.admin.role in etc/system properties, the value should correspond to your Group mentioned in role.filter of the ldap-module.xml file.


The example allow the authentication to Fuse ESB, if the user is present in the Organisation Unit “OU=Users” and member of Operator Group.


The LDAP module is automatically activated when you put the file in deploy directory.


Enjoy !