Think before you put things on the Bus!

Recently after some business folks were pretty pissed off about not getting enough value out of SOA and the ESB not being able to handle certain processes properly, an architect at work asked me to write about the factors which should be considered while deciding what projects and solutions should be put on the enterprise service bus as services. I gave him a list which I am writing below. It is certainly not exhaustive and is specific to data integration solutions and the products which we have licenses for(Oracle Service Bus, Informatica PowerCenter+PowerExchange, Oracle Data Service Integrator). So here it is:

There are three different approaches we are using in our enterprise for data integration flows: (I will refer to these by their number in the reasoning below)

  1. Real time integration via SOA based messaging services (OSB, DSP/ODSI, Weblogic JMS)
  2. Real time integration via trigger based ETLs/ELTs(Informatica or Oacle Data Integrator)
  3. Batch processing via scheduled ETL jobs (Informatica)

Here is a list of factors/questions which I think we need to consider to choose between various data integration approaches.

1. What is the need of real-time integration of the data?

The IT processes are getting faster every day and every minute, but it is important to actually identify whether real-time integration is needed for any process which needs to be implemented. It is important because real-time data integration has a higher total cost of ownership. Keeping that in mind go for option 1 or 2 from above list only if the need for real-time integration of data is justified. In all other cases where there is no actual business need for real-time integration its best to go with option 3 which is cheaper, easier and faster to develop.

2. Size of data. In terms of size/message and overall size/day.

If the messages are very big(anything more than 1-2 MB/message) the transactions to send data to destination might run for a long time due to higher insertion time on database and higher network transfer time. This is not a very good business case for real-time integration via messaging services. It is wiser to keep such heavy transactions away from the shared bus.  For such scenarios it is better to go for option 2 if there is a justified business need of real-time integration otherwise option 3. If it is possible to split the messages into more number of smaller messages then SOA based messaging(option 1) can be used.

Also, if total amount of data being transferred per day is huge(anything more than 10 GB/day) the best approach is option 3. If we use option 1 for this kind of data integration then in case of any catastrophic event (destination being down etc) SOA layer will need to persist all the messages and storing such huge data on SOA layer(in a DB or on the file system) is a potential performance risk.

3. Message rate or Transactions per second

Usually TPS should NOT be deciding factor to choose between these 3 approaches if messages are small(average message size <100 KB). But in case where there are larger messages, TPS becomes a factor since processing of messages will be slow and throughput will be limited. Best option in that case would be approach 2. For example, if we have to send only 5000 messages of 2MB each every day, but all messages will come in a burst and within 10 mins(meaning ~8.3TPS during those 10 mins) then option 1 might not be the right way to go, rather use option 2 or even option 3 if timing/schedule of these bursts is fixed. While if these 5000 messages are spread across 2 hours (<1TPS) then option 1 can be used.

4. Synchronous or Asynchronous

For synchronous flows, where a response is required from the destination(for ex. Create order sent to Oracle EBS and an order number is required in real-time), Option 1 is the best approach. In this case it is necessary to ensure messages are small(even if means higher TPS). For asynchronous data flows there is a choice of all 3 options based on other factors.

5. Complexity/Logic needed on integration layer (this sometimes depends on message format being received from source system)

This becomes an important factor when dealing with big messages or when TPS is very high. Moreover it does not makes sense to put a lot of logic on the Bus(this includes heavy transformations of big messages). If required the logic should be moved to BPEL and Bus should only be used for routing and orchestration. Complex transformations (involving multiple levels of loops, lot of type conversions on XML documents running in tens of thousands of lines) eat out a lot of CPU cycles and may be a potential performance risk.

6. Scope of reuse

If the same data integration is to be reused for a lot of destinations(pub/sub kind of scenarios), it makes sense to use approach 1. Even if it means making changes to the data structure to reduce message size or asking the source systems to implement minor changes to make their message data structure simpler. Effort spent on making the messages smaller and simpler so that the solution can be put on the ESB will actually save a lot in future. Where there is a scope of reuse it makes sense to tweak other factors and put it on the Bus.

7. Performance of the destination

If the destination application/DB is performing very poorly(and it can’t be improved). For ex. if it takes more than 1 minute to process every message on the back-end(irrespective of the message size), then option 2 should be preferred. In such cases if option 1 has to be used then the response times of back-end system need to be improved either by performance tuning on that system or by reducing the size of messages. The response times are important for the services deployed on the bus because it is a shared environment. The longer one process keeps the threads occupied, more is the probability of having a large queue for the processor by the other requests.

There may be other factors as well but I feel the above are some of the most decisive ones. Moreover I have talked about them individually but almost always they all play a part together and can be tweaked together to deploy important flows on the bus.

For an example, consider that the scenario of 5000 2MB messages in 10 minutes is going to be used by a lot of applications across the enterprise. It makes sense to put it on the bus and make it accessible to multiple consumers. To make that possible either get rid of any transformations or try to move them out of the bus or make them simpler. Also make sure that the back-end application is not taking too long, if it is then make the process an asynchronous delayed response type for calling the back-end(use Sync To Async pattern if consumers want a synchronous transaction).

I have seen a lot of projects deployed on the bus where certain things were overlooked and they caused a lot of problems around the performance of the bus. Being a shared environment which almost always handles a lot of business critical processes, great care needs to be taken to maintain the bus. It will cost a lot more to maintain an environment which has a lot of unnecessary poorly designed services deployed wouldn’t it?

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: