Think before you put things on the Bus!

Recently after some business folks were pretty pissed off about not getting enough value out of SOA and the ESB not being able to handle certain processes properly, an architect at work asked me to write about the factors which should be considered while deciding what projects and solutions should be put on the enterprise service bus as services. I gave him a list which I am writing below. It is certainly not exhaustive and is specific to data integration solutions and the products which we have licenses for(Oracle Service Bus, Informatica PowerCenter+PowerExchange, Oracle Data Service Integrator). So here it is:

There are three different approaches we are using in our enterprise for data integration flows: (I will refer to these by their number in the reasoning below)

  1. Real time integration via SOA based messaging services (OSB, DSP/ODSI, Weblogic JMS)
  2. Real time integration via trigger based ETLs/ELTs(Informatica or Oacle Data Integrator)
  3. Batch processing via scheduled ETL jobs (Informatica)

Here is a list of factors/questions which I think we need to consider to choose between various data integration approaches.

1. What is the need of real-time integration of the data?

The IT processes are getting faster every day and every minute, but it is important to actually identify whether real-time integration is needed for any process which needs to be implemented. It is important because real-time data integration has a higher total cost of ownership. Keeping that in mind go for option 1 or 2 from above list only if the need for real-time integration of data is justified. In all other cases where there is no actual business need for real-time integration its best to go with option 3 which is cheaper, easier and faster to develop.

2. Size of data. In terms of size/message and overall size/day.

If the messages are very big(anything more than 1-2 MB/message) the transactions to send data to destination might run for a long time due to higher insertion time on database and higher network transfer time. This is not a very good business case for real-time integration via messaging services. It is wiser to keep such heavy transactions away from the shared bus.  For such scenarios it is better to go for option 2 if there is a justified business need of real-time integration otherwise option 3. If it is possible to split the messages into more number of smaller messages then SOA based messaging(option 1) can be used.

Also, if total amount of data being transferred per day is huge(anything more than 10 GB/day) the best approach is option 3. If we use option 1 for this kind of data integration then in case of any catastrophic event (destination being down etc) SOA layer will need to persist all the messages and storing such huge data on SOA layer(in a DB or on the file system) is a potential performance risk.

3. Message rate or Transactions per second

Usually TPS should NOT be deciding factor to choose between these 3 approaches if messages are small(average message size <100 KB). But in case where there are larger messages, TPS becomes a factor since processing of messages will be slow and throughput will be limited. Best option in that case would be approach 2. For example, if we have to send only 5000 messages of 2MB each every day, but all messages will come in a burst and within 10 mins(meaning ~8.3TPS during those 10 mins) then option 1 might not be the right way to go, rather use option 2 or even option 3 if timing/schedule of these bursts is fixed. While if these 5000 messages are spread across 2 hours (<1TPS) then option 1 can be used.

4. Synchronous or Asynchronous

For synchronous flows, where a response is required from the destination(for ex. Create order sent to Oracle EBS and an order number is required in real-time), Option 1 is the best approach. In this case it is necessary to ensure messages are small(even if means higher TPS). For asynchronous data flows there is a choice of all 3 options based on other factors.

5. Complexity/Logic needed on integration layer (this sometimes depends on message format being received from source system)

This becomes an important factor when dealing with big messages or when TPS is very high. Moreover it does not makes sense to put a lot of logic on the Bus(this includes heavy transformations of big messages). If required the logic should be moved to BPEL and Bus should only be used for routing and orchestration. Complex transformations (involving multiple levels of loops, lot of type conversions on XML documents running in tens of thousands of lines) eat out a lot of CPU cycles and may be a potential performance risk.

6. Scope of reuse

If the same data integration is to be reused for a lot of destinations(pub/sub kind of scenarios), it makes sense to use approach 1. Even if it means making changes to the data structure to reduce message size or asking the source systems to implement minor changes to make their message data structure simpler. Effort spent on making the messages smaller and simpler so that the solution can be put on the ESB will actually save a lot in future. Where there is a scope of reuse it makes sense to tweak other factors and put it on the Bus.

7. Performance of the destination

If the destination application/DB is performing very poorly(and it can’t be improved). For ex. if it takes more than 1 minute to process every message on the back-end(irrespective of the message size), then option 2 should be preferred. In such cases if option 1 has to be used then the response times of back-end system need to be improved either by performance tuning on that system or by reducing the size of messages. The response times are important for the services deployed on the bus because it is a shared environment. The longer one process keeps the threads occupied, more is the probability of having a large queue for the processor by the other requests.

There may be other factors as well but I feel the above are some of the most decisive ones. Moreover I have talked about them individually but almost always they all play a part together and can be tweaked together to deploy important flows on the bus.

For an example, consider that the scenario of 5000 2MB messages in 10 minutes is going to be used by a lot of applications across the enterprise. It makes sense to put it on the bus and make it accessible to multiple consumers. To make that possible either get rid of any transformations or try to move them out of the bus or make them simpler. Also make sure that the back-end application is not taking too long, if it is then make the process an asynchronous delayed response type for calling the back-end(use Sync To Async pattern if consumers want a synchronous transaction).

I have seen a lot of projects deployed on the bus where certain things were overlooked and they caused a lot of problems around the performance of the bus. Being a shared environment which almost always handles a lot of business critical processes, great care needs to be taken to maintain the bus. It will cost a lot more to maintain an environment which has a lot of unnecessary poorly designed services deployed wouldn’t it?


Business drama over, its time for some football

‘Finally!’ – That was the one word which was on every Reds fan’s mind yesterday.
October 15th, 2010 – A date which could be the turning point in the history of one of the greatest football clubs of Europe.
New England Sports Venture – New American owners of Liverpool which will have to work hard to live up to a lot of expectations.

The long running saga of Liverpool FC takeover finally came to a conclusion yesterday. This tale had all the makings of a bestseller. It had all the elements: Mighty rich businessmen from America, a powerful bank of Europe, one of the world famous sports clubs, a hugely passionate group of fans, court cases running across the Atlantic and of course greed, power, ego and money!

With NESV paying off all debts to RSB and taking control from the yank pair of Tom Hicks and George Gillett, the Reds fans sighed a breathe of relief. Although, whether another American owner will understand the nuances of successfully running an English football club remains to be seen. NESV has an impressive background as the organization which turned around the fortune of Boston Red Sox and ended the curse of the Bombino(for the unaware, Red Sox sold the greatest ever baseball player Babe Ruth to New York Yankees in 1918 and never won the world series again until the 2004 season, 2 years after NESV became the new owners of the club). All the Liverpool fans will be expecting nothing less of the new owners. But given Liverpool’s current situation it seems a herculean task at the moment. NESV will definitely not splurge money like the Sheikh at the helms of Man City because they know its not going to help much. What Liverpool needs is a complete rebuild under a capable manager. The current manager Roy Hodgson who took over the job only 3 months back certainly not made it easy for himself by applying extremely defensive tactics on the pitch which resulted in club’s worst start to a season in 57 years. He is a good manager but certainly lacks the skills to manage a big club. Top clubs usually rely on a more attacking strategy to win the games and remain on the top. Loss of Xabi Alonso and Macherano in last two seasons has not made it easier as well with Liverpool’s midfield being one of the weakest in a long time. Lack of a supporting striker for El Nino and a his own poor form is also a major concern.

It is expected that NESV will be providing a higher than average budget for January transfer window. Hodgson(or his replacement in case he is sacked) will be looking for a left back, a quality playmaker in the midfield and a proven striker to join the ranks but it seems unlikely that he will get budget for all three signings. It seems more probable that he might get around £20-25m for a striker and another £15-20m for a midfielder in this transfer window and a promise for more money during next year’s summer.

In any case, it seems that there will be a lot of changes at the Kop in coming months, how much these changes help getting the team back on track cant be said. Whatever happens the support of the fans is going to be there, now that they have seen the golden sky at the end of the storm, they are awaiting the sweet silver song of lark…

My mistakes

Where the life has brought me these days,
Searching for true friends in this endless maze,

Now thats pretty hard just like reading in the dark,
When you’ve had friends who were true to the mark,

And now It is becoming difficult to breathe everyday,
With the souls who were close are now so far away,

I wish I could go back to hearts which connected to mine,
With whom I enjoyed the time and drank some wine,

But that dream seems impossible to follow now,
Even if I do try hard, I cant figure out how,

Its hard to accept the mistakes I’ve made in my life,
and it doesn’t help at all that the fault was always mine,

With you guys my past may have been intrepid,
and without you my present seems so timid,

I am living this guilty life, and don’t know how I have the grit,
May be the thin ray of hope that says I can make it,

But I am ready to cling on to it as long as it takes,
Because I know that someday I will be able to make up for my mistakes…

What to do on a weekend if you are living in Bangalore?

I have tried a lot of things to have a good time on a weekend in Bangalore, some of which are listed below. I cant go out on a weekend because of work so not including that as an option. Let me know if you have any better ideas!
So here are the options:
1. Watch stupid movies on the tele which you wont be able to survive.
2. Go out for dinner but all you can get it tasteless food at super high prices.
3. Go out to watch a movie(obviously at very high prices) just to come back home, disgusted with the fact that the level of movies has fallen to such depths that its a torture to survive them in a theater.
4. Download and watch some good movies which you have already watched at least a dozen time.
5. Go out with friends to a pub where you wont feel like enjoying because of atmosphere which resembles a funeral at all times and the music played is simply shit.
6. Spend the day(and night) playing AOE with your roommate on the WLAN, while drinking beer, listening to your favorite music and eating some tasty home cooked dish. May be some time to read something as well.

I mostly opt for option 4 or 6.
So, what will you do if you were in the same situation eh?

Dollars Trilogy 3: The Good, The Bad and The Ugly (1969)

All right, time for the biggie now! The good, The Bad and The Ugly is the last part of the trilogy and definitely the best one. I would even go on to say that this is a movie of epic proportions. A grand stage for Sergio Leone to show his skills at making a perfect spaghetti western. There is no scope for any criticisms for this movie, everything is spot on. The movie is about three men after a huge sum of money and their journey which is intertwined with skirmishes with others and amongst themselves. All come up to one last three way gunfight between the three in a huge graveyard where they find the money. The story is much stronger and not as straight forward as we are used to see in a western. It keeps you glued to the screen an you never know which way it will go. I am not saying that its a top notch on edge of your seat thriller, i am just saying that it is one of the most thrilling westerns ever made and perhaps the best ever western.

Clint Eastwood(The Good) and Lee Von Cliff(The Bad) go on from where they left in For a Few Dollars More, they both play their roles to perfection, But it is Eli Wallach(The Ugly) who steals the show hands down. He plays the role of a crooked outlaw Tuco and he does portray the character wickedly enough. The best part is that even though they are supposed to be The Good, The Bad and The Ugly, there is no clear demarcation as such. All of them are just after money and care about nothing else but the money. Clint Eastwood does show glimpses of good side when he helps the dying army captain and fulfills his wish, but on the other hand he does wrong Eli Wallach in the first half, and again he is not your straight forward humbling cowboy hero who wont shoot first and follow the law. Similarly Eli Wallach is indeed the most wicked among the three but he can be considered on the brink of the bad as well as the good as well besides he is the funniest. Lee Von Cliff although is never shown deviating from his definition of The Bad, he is as menacing as a villain could be, but in a cool, calm and deadly way. He is not going to shout out loud or laugh like Gabbar but he will eat you alive with his stare but again he is shown as a man of principles, who will not deter from his job come what may.

Well, if you ask me what is the best part about the movie, you would be putting me into a lot of trouble, I really can’t pick anything without leaving behind the rest of it. But after pushing my teeny tiny brain to its absolute limits and cursing you countless times in between, I would have to say it is a tie between Eli Wallach’s portrayal of Tuco and the music. The music is so good that you still here it at a lot of places (and it has been abused in almost a 100 Bollywood movies i suppose). Ennio Morricone at his absolute best.
Rating: “Whoever double-crosses me and leaves me alive, he understands nothing about Tuco.” so you better go and watch(and like) this movie or your days are over outta 10!

Dollars Trilogy 2: For a Few Dollars More (1965)

Ok, this brings us to the second installment of the Dollars trilogy. The man with no name is back, this time as a bounty hunter. All the characteristic are exactly the same. He competes with another bounty hunter who took the profession after retiring from the army. Both of them are after the reward for a gang of outlaws headed by a dreaded leader Indio. They join their forces down the line to eliminate the gang and get the reward. There is also a slight twist in the end as Colonel has some hidden motives as well. Like all other westerns it has simple story and a stylish execution. Another great effort from Sergio Leone and Clint Eastwood, but Lee Von Cliff as Colonel Mortimer is an excellent addition to the cast. He does look cold blooded and a precise, calculating slayer.
The music and background score is better than Fistful of Dollars, and so is the direction. I think Sergio Leone improved with time and so did Ennio Morricone who has given some beautiful harmonica compositions here.

Rating: “Where life had no value, death, sometimes, had its price. That is why the bounty killers appeared” and they will surely give you an adrenaline high outta 10.

Dollars Trilogy 1: A Fistful of Dollars (1964)

This was the first one of the trilogy to come out. The movie tells the story of “Joe” a drifter gunman who comes to an almost deserted town and finds out that the town is torn apart by the the old rivalry between two powerful families and the only flourishing job is that of the undertaker.
Joe then uses his brain and the skills with gun to play both the families against each other and ultimately getting the town rid of the troublemakers and earning a lot of money on the way. The movie also shows the good side of Joe when he frees a couple being held captive by one of the bad families.
The movie is slow at times, but still I like the way he directs gun fights. Sergio Leone has the habit of building up the scenes, with long shots of people staring at each other with nice music playing in the background. Well, thats what western movies are about and I totally enjoyed the gun fights in the climax of this one. The background score is pretty cool and exciting which gives the finishing touch to the scenes. The only problem with the movie is that the story is very simple and straight, there is no guess work here, you know whats going to happen from the start itself.

Rating: “If a Man with a .45 fights a man with a rifle, the man with a pistol is a dead man!” Is it??? Nah, nay, never, the man with .45 always rocks outta 10 😉

Dollars Trilogy preface

Dollars trilogy is a set of three spaghetti western movies released during 1960’s directed by Sergio Leone and Clint Eastwood as the lead in the role of The man with no name.
The movies are not related to each other in any way and all the three have different characters and story. But still they are considered as a trilogy only because of the characters played by Clint Eastwood in them. He plays different characters but the mannerisms, style, clothing etc are exactly the same in all three. ‘The Good, The Bad and The Ugly’ is the last movie in the set and is the most famous one.

Clint Eastwood as The man with no name brings to life a character which I always consider one of the greatest superheroes to ever appear in movies. You will have to see him to believe it. He is quick to draw as quick a snake will bite and he can shoot better than all the Olympic gold medalists, he is super cool even in the most terrifying situations, smokes cigarillos and smokes it in a way that it appears he is almost chewing it, he is laconic, speaking very rarely and very softly whenever he does.
Now I know I have been calling him a superhero thus far, but actually he played an anti-hero sort of role in all three movies. He is not a typical all good cowboy like in the other western movies, he will shoot first if required, he will kill men without flinching an eyebrow if he thinks he needs to do so and above all he is out there to earn money not for social service. But he does care about people in trouble and usually ends up helping them while earning the dollars.
Whatever Clint Eastwood does seems grand! The man has got amazing looks to go with an excellent voice and piercing eyes. He truly is the most stylish hero Hollywood has seen thus far.

Now about the ‘No Name’ part, He is called Joe, Monco and Blondie in the three movies respectively. But these are all nicknames given by some other character given to him in all the three movies. All the three names are very common nicknames. Eastwood never reveals his name in any of the movie himself. In The Good The Bad and The Ugly there is a scene where an army man asks him for his name while he was enlisting in the army but he is reluctant to answer and the scene is shifted to something else. So even though there is a nickname given to his character in the trilogy that is not his actual name and thus he is a Man with no name. 🙂