Life and IT

Can your life be like a wave?

2011-08-24T20:38:00.002+05:30

My brother doesn't blog these days but I love this masterpiece of this blog -

http://uglyduckling1of9.blogspot.com/2009/05/its-amazing-how-ones-life-can-be.html#comments

It says that your life has the same graph/pattern as a single day in your life - like fractals - its recursive, repetitive and also that life is a series of events that sometimes double back on themselves (recursive) and sometimes result in unexpected flows.

Why I like this is for two reasons -

For about 10 years now, in my life, each time I think I have figured out what the bloody repeating pattern is and find a way to work around or live with or beat the problem - something else crops up.
This problem crops up with a frequency of 3 months, 6 months and 1 year
The pattern that repeats each time is the same - something comes up, something good happens or si get a result or someone does something, I begin to take interest in that and believe that will continue - and then someone or something brings the house down.

Its fun in a way and pain in many ways - but it always surprises me that maths can explain it very easily :)

Crowd sourcing your own unhappiness

2011-08-24T20:13:00.004+05:30

We all check Facebook - we all check it very often. Once in a while I come across a nice link on Facebook that I read and share, or a picture that I truly feel is good and appreciate it, or a witty comment. But 99% of the time I find stuff that if I take too much interest in, then I will be causing myself some unwanted sadness.

Its not bad things that make me sad, its too much happiness.

People have become so addicted to Facebook and social networking that they put up every small moment of their life into it - XYZ is at ABC with PQR, XYZ is feeling lazy, XYZ visits The-best-place-on-earth. Pardon me for being so damn rude, but sometimes I don't want to see all these happy events in your life - I just want to live in my own shell!

I know what you are saying - if you don't like it then get out of there! Yes I will - the day I truly fall sick. But before that, let me put across my point.

Back in the old days before all these websites, my cousins lived in Singapore. Obviously they visited a lot of places, were happy, had fun and took pictures and videos. I heard about all this, but never actually saw the pictures instantaneously. Being good in GK, I knew about Singapore and had an idea about the place in my head. Once in a year when these guys came to India, I would look forward to meeting them and then seeing the pictures or videos or whatever it was. There was some fun in that. Then came digital cameras and all the photo sharing and emailing - even then, there was a delay - so I saw their happy faces a week later as opposed to an year later.

In both these cases, I saw the pictures of the happy moments when I was in a peaceful state of mind. Not when I was pissed off.

Cut to today - almost all of us have a high pressure job. Everybody has tensions - real or imagined - and we are all trying to live a fast life. We have a moments attention span on anything. Inspite of all this, because of peer pressure we all check Facebook daily. Or Google+. This peer pressure is something that we refuse to acknowledge as real - it has become so much woven into our life. We do not even know that we are under peer pressure. We feel it is but natural that we should feel the urge and check these sites everyday, if you have a smart phone then even more often, and update statuses.

We will not go to a place, enjoy the place, come back, relax and write a nice paragraph - we will get into the car, tweet, post status, talk about the car, get into a conversation, reach the place and again tweet or post about it, tweet or post about the hot chick or hunk you saw, food , loo, tired, return journey and then that you are falling asleep dead tired.

Excuse me - did anyone else feel that there was more fun in writing a relaxed account rather than all these updates?

Imagine looking at your timeline and seeing all these updates - you feel people are doing a lot while you are doing nothing; Do the same thing while or after you had a bad day at work or after you spent the whole day alone; Or after you did not shop and are feeling bad. Does it hurt? What if you are the easily influenced kind? It hurts more then.

And once you are hurt, you hurt those around you with unwanted unhappiness.

The twist in the tale is that you and if you hurt someone then they too will post about the sadness which leads to a few more comments and even more sadness!!!!!

So - while we are supposed to get to know what the other person was doing, we don't stop there. Most of us don't - very few can stay aloof of all this.

And that's why I hate these platforms.

My over used ligaments - an anticlimax story

2011-05-14T16:10:00.003+05:30

"No fracture, but probably due to overuse" - This is the cause given by the doctor here about my knee pain issues. Its funny because I am just 28 and haven't really abused or overused my legs that much.

For the last couple of months I was training hard to participate in the Oxfam Trailwalker Japan 2011 - a 48 hour 100 km walk over some beautiful hiking trails in the Mount Fuji area. I was not fat, but was not fit either. I walked a lot,ate less and pushed myself to build my stamina. I graduated from 5 KM walks home to 16 Km walks around the imperial palace. I could not run or jog like all those hundreds going around the imperial palace, but I sure hoped that one day I could. On our practice walks for the event, we did a 28 KM and a more recent 54 KM walk - I did them both with just one issue that I could not continuously climb mountains. But I was making progress.

Yesterday, this whole affair has an anti-climax when I had to drop out of the event after just 18 KM due to a stinging pain in my knees. The day started well with us four going to the track and field at Odawara - the day was bright and sunny and hydration was the only thing on our mind. I was sure I would slow down somewhere on the walk but was looking to complete the thing in about 30 hours. By the time we got to the start of the uphill trail I had a strange feeling in my knee. I pushed on and in fact walked to the top of that 600m hill without stopping much on the way. I probably stopped once for a 30 sec break. Walked downhill from the top and started the next 800m climb and did it with a surprising amount of stamina and did not stop at all on most of the uphill bits. But then, 70% into it, the knee started to hurt with each step. And then from the ankle up to the knee along the side. What the fuck! By the time I got to the checkpoint at 18 KM, I was trailing my teammates by 300 meters and unable to catch up and a stinging pain on each step.

I probably became the only guy in the history of Oxfam to give up at that early stage!

I had to withdraw, but I decided to help the support crew. And when I saw my teammates today morning at the 80Km check point making good time and looking to beat the 30 hour mark, I did feel jealous and depressed at the same time.

I got into the train home and to my horror that by the time I had to get down after the 1 hour train ride, my knees were locked and paining. Dropped off my stuff at home and went to the hospital - thankfully no fracture but a damaged ligament or to put it in the words of the doctor, overused ligaments. I am told to rest and cool my knees for a few weeks.

Wasn't the kind of ending I was looking for to this event - but life gives you a bouncer every now and then.

Impressions of Tokyo on the day of the earthquake

2011-04-03T15:53:00.003+05:30

Unless you have been living in a long forgotten corner of the world where you don't have access to news, you probably know all too well about the recent earthquake, tsunami and nuclear plant accidents in Japan. March 11 2011 will be an unforgettable day in my life not because I experienced the biggest quake in my life but because I saw the kind of orderly and calm behavior that I have not seen elsewhere.

To put things into perspective, it was a normal day where we were all fighting our daily battles, some losing battles, and going about our work. It started with a slight jolt which became a shake and pretty soon we were swinging and well aware that this was not one of the usual tremors that we face every now and then. It was a minute but felt like a long time - under the desk, reaching for a ringing cell that has fallen down and trying to tell my wife that I will call her back in a minute - I don't know why I said I will call her back, but i definitely asked her to stay inside and safe.

By the time it stopped shaking, the trains had been stopped, the popular carrier which supports iPhone here was unable to take any calls, and the tsunami alert was in place. As we all gathered our emergency kits and helmets, it became clear that norther Japan had been hit by the tsunami and there was large scale devastation. I managed to reach my wife on email and tell her to come to my office and that we will walk home. She was 15 KM away on the street but could not really appreciate why I said we will walk home. The trains had stopped but she was hopeful that we could get a cab. I tried not to scare her - she would know soon how panicky it would be on the streets pretty soon.

There was news that some building was on fire, a refinery was burning, someones wife managed to save the 42 inch TV while everything else fell down, blackout, someone running to the day care to get his son, someone praying - it was a mess. The glass facade of my building began to creak and we moved towards the center of the building. By the time the security verified the building was safe and we were safest in the building than on the street, almost everyone was either planning to leave or thinking how the hell should they get home now. Down on the streets, it was full of people trying to get home. And this is where I began to admire Tokyoites.

Just to summarize,

People stood in line for taxis that were not available in large numbers due to the rush - 100 people waiting an hour or so in a line
People pooled for the cabs, some paid the fare on the meter when their stop came without bothering to split the money with others in the cab then and there.
People did not rush or push; cars did not honk in the traffic jam
Starbucks, Subway and all those places were open and friendly and no overpricing or shouting
The police and traffic police were out and helping people find the way around as they walked - you don't know which way to walk if you take the subway everyday
People helped each other and were friendly
There were 300 people waiting in a queue for a bus that would come every 15 mins and at max could take 50 people
Some people who could not walk home waited at office in hope that trains would resume soon - the emergency staff and security at the office promptly got blankets and emergency food supplies for everyone
In some offices, people formed groups depending on where they were going and one person lead the group like a tour guide and made sure everyone reached home.
There was no shouting or line breaking even when the few trains started late at night
Hotels did not overprice or throw people out, in fact they opened the lobbies for everyone for free
Apple store and other shops that sold PC's let people check email and charge phones etc

It was far from chaotic - it was just a large number of people walking. And in all the 6 KM that I walked home, I did not see one person behaving badly or losing calm. Not at the airports where flights were cancelled, not at the stations where the trains went nowhere.

And this continued till the next day afternoon when everyone finally managed to get home - some walked 30+ KM to get home or slept somewhere in some station for that night.

Most of all - media did not hype anything. Not a single scene of destruction or dead body was shown for at least 4 days following the disaster.

What I saw here made me so happy that I wonder why we cannot emulate the same in India ? Would it kill us if we thought of greater good or stopped being selfish and stayed calm for a while?

Throwing away ideas with the trash

2010-08-29T16:52:00.002+05:30

About 2 years ago I had an idea - why not put real estate ads and Google maps together to create a mashup? The idea was born as I browsed through GumTree sitting in London. The idea was simple, and I even registered a domain name for that on the spot - http://2-let.co.in - and an idea was born.

Trouble was that I was never exposed to web development on large scale. So I needed to learn something like Rails or Grails to be able to build this quickly. Sure, there were so many sites with the usual example program and mashup. Read them and off you go.

Troubles

I am the only guy with a clarity and interest
Others want to get the job done, get a beer and enjoy the weekend
The learning curve is steep and the work load in office is deep
The examples are too simple and I do not have the help of an architect

And with these the plan fell through the cracks.

Similar fate met another couple of ideas as well. At the end of my stint in London I threw away the notebook with all these scribblings.

Today, I still don't have time for such things - all I managed was a few java libraries that I host on sourceforge, few people download and no one cares to leave a feedback.

I read somewhere that Google now puts real estate ads on its maps in US or somewhere. Well someone implemented it.

Some day....

Reliability - gone to dogs

2010-08-29T16:36:00.002+05:30

In todays world of cloud computing and clusters, reliability is a hard thing to ask for and the mantra and buzz phrase doing the rounds is "failure is not unexpected and you need to build in such a way that you can ignore the failure and still retain system functionality" - whew!.

Google and Facebook really convinced us on this didn't they - build large server farms running clustered code on cheap commodity servers where memory and processing power are cheap, build code that can survive failure by virtue of duplication and replication and there you go - the best system on the planet!

How about Twitter going down so many times in a day? Well, scale it using some fancy cluster or cloud software and paradigm.

Great!

How about the real physical world? Batteries dying down or not charging, cables snapping, back paining etc etc? Do we scale ourselves using the Matrix?

Of late I have been hit by problems like the battery on my laptop discharging in half the time ( its a new less than 2 months old battery mind you!), cable snapping, shared rive write failures etc etc.

Why? Why cant you just buy a piece of hardware these days that works well for a good time!

I am not a stingy bastard expecting something to cost me pocket change and run for ages - I just want god damned company products to last their god damned printed life time!

I lost files the other day due to a shared drive write failure.

I lost files some time ago when a long life DVD failed because the plastic was peeling off!

Aaaaaaaaaaaaaaaaaaaaaaaaah!

The only advice that is available - free or paid - is to accept failure.

hmmmmmmmmmmmmmmmmmmmmmmmmm

PS: This laptop/netbook that I am writing this post on advertised 14 hours battery life; Taking it with a pinch of salt I expect it to give me 7 hours.

I am NOT getting that - and yeah ... the answer apparently is that I am crazy enough to run Linux on this. Fortunately the guy giving this was far far away from me.

Hajo 0.2 beta released

2010-08-28T16:48:00.004+05:30

The much delayed 0.2 beta of Hajo is now released. I had the code for this ready more than a month ago but with my moving to Tokyo and the settling down here, I finally found the time to get the release out. This version does not add much in terms of the interface but adds the two following things

An embedded Jetty server that published the underlying DB stats on port 1122
An option to start a multi threaded server in the Thrift code on the server - the simple tests that I did seemed to work, so this is now released. Need to test more on this for next release and add more interfaces.

The application can be downloaded from https://sourceforge.net/projects/berkeleydbtools/files/ .

iPhone 4 adventures

2010-08-23T18:11:00.004+05:30

This weekend I got my iPhone 4 - just 15 days after I reserved it. And that is nice considering that folks waited upto a month to get that here in Tokyo. What impressed me most about this entire process was the way the girl at Softbank Roppongi opened the box.

Yes, not the phone, or the features but just the way she opened it. Probably thats how Apple mandates that the box be opened, but it was not how I saw an iPhone 3GS being opened at the Apple store when I was here in February.

So, here is what happened, we went to the store and they found out 2 girls who could speak english to help us. It was easy in that store because that area is preferred by expats here. So after all the signing etc etc and selecting the phone number, this girl brings the cute little box with the iPhone, takes a small blade and neatly cuts the bottom of the plastic cover - along the edges not damaging anything and cutting where required. Then she neatly removes that rectangular piece of plastic, inverts the box and slowly lifts lid, slowly revealing the iPhone.

Imagine a guy in Vodafone Bangalore, tearing away the plastic, throwing it away, quickly opening the box and doing a practical quickly finished job of it - you wont really enjoy it. Even if it was a good looking girl ripping off the packing.

Thats why I like the opening of the phone much better than the phone itself - it was apple so the phone will be good, but how better can you make the act of opening the packet? You cant do better than this.

That apart, I am enjoying the phone, addicted to frequent tweeting facebook updates again. Checking mails and taking notes on Evernote - its fun.

Well there you go ... I am an apple fanboy now :)

Life in Tokyo

2010-08-14T20:13:00.002+05:30

I moved to Tokyo on 22 July 2010 to start a new chapter in my life with my job at Merrill Lynch. So far it has been good. Probably because I was here for a week in February this year and roamed around the city a lot, I do not feel out of place or new.

Last few weeks have been hectic. Getting here, getting ready for the office, first week at work, getting a new house, getting a bed from ikea and fixing it on my own - 5 hours, whew! etc etc.

Work is going good.

Life is good - managed to get Internet in this new house and so am now hooked and ready to go. Also managed to get wireless working on my Ubuntu netbook so thats a bonus.

Looking forward to some fun filled years in Tokyo - I am starting to learn Japanese and hope to be able to speak decent Japanese by end of the year - writing is another issue anyways :)

ScribeFire test!

2010-07-16T09:39:00.001+05:30

This is a test post using the ScribeFire extension in Google Chrome :)

Looks interesting and easy way to post blogs - thanks Google!

Edit 1: Fixed the font size.

ANTLR tutorial

2010-07-15T22:54:00.002+05:30

My latest muse is ANTLR - I have been trying to get some knowledge on compilers and how they are built, learn lexing etc etc for a while. I even bought the dragon book - but never made a headway.

Well, a month to kill before the next job does give you a good chance, so today after killing 3 weeks of this month on packing and Thrift, I have picked ANTLR.

This tutorial is awesome - nice and easy way to learn - do have look if you want to learn.

http://javadude.com/articles/antlr3xtut/

Working on the go

2010-07-08T11:40:00.002+05:30

When I was in London few years ago, a common sight was to see someone on the tube working on his laptop busy doing some work. People typing away on blackberry's and reading notes etc was more common. With the advent of the small netbook's it became more common to see people working on the tube.

The trend seems to be catching up in India as well, especially Bangalore. Bangalore transport, BMTC, has been running comfortable Volvo 8400 AC coaches for a few years now. Today there are a good number of these buses running on most of the major that pass through or end at the IT hubs of Electronics City and ITPL.So it is common to see techies on these buses. Also, the internet on the go USB devices are more common and less costly than before. So it is not surprising that I see folks on the bus with a laptop and working away to send personal mails etc on the commute to office. I did see one rare case where a guy was writing an email in Gmail offline mode - that's one neat way to use it. Even students and the young crowd use the laptops in buses. Food courts in big malls are another place, much like connecting from a McDonald's if it was in UK or US.

However, this throws up some issues. If a young guy or gal is using the laptop in the bus or a food court browsing photos or writing a mail to a loved one, even if someone did eavesdrop, it would be a small embarrassment - nothing really BIG. But what if someone happened to eavesdrop on official stuff? I mean if you are on the company transport bus, the fine. But on a public bus? The chief of Scotland Yard once had to quit because some over zealous photographer of an London evening paper happened to take a photo of him with some papers out in the open, instead of inside a file. This happened outside the gate of 10 downing street mind you, the prime ministers house and not some public bus.

At work we sign all kinds of NDA and secrecy agreements, so we need to be careful. I have friends in competing companies in sales positions. I speak with them about all the troubles at work and they talk as well. But we are always careful not to talk to each other about specifics. It helps because I am into technical things and not sales and there is little that I can offer or use from them.

To quote from Spiderman movies -'With great power comes great responsibility'. I guess that applies to technology as well and we need to learn to be careful.

Etcetera

2010-07-07T12:29:00.002+05:30

Like I mentioned in my previous post, I finally managed to configure my Linode and lock down the server and configure my SSH keys etc. I managed to write a few bash scripts and a groovy swing builder based script to checkout and build the code for Hajo. They work perfect! and its a good ego boost before I get to more serious work on my next job in a few weeks.

Next plans include setting up some form of CMS or maybe Mediawiki to host a website for all my pet projects. I have decided to group them under a name Supercoderz. This will be the umbrella for all my projects so far and the ones to come. And maybe 5 years down the line, become something big. Supercoderz is still in the alpha stage and nothing concrete to show yet.

Qutting jobs, getting a Linode 512 and looking forward to new things ... last 24 hours were a rush

2010-06-23T18:04:00.003+05:30

24 hours ago I quit my current job here in Bangalore, India. I am looking forward to moving to Tokyo, Japan for my next job. The process is in full swing and should be done soon and by this time next month, I should be there.

In another impulsive decision, which is becoming quite a habit these days, I signed up for a Linode 512 VPS on Linode. Last night I tried to follow all the guides and how-to's at once and had to rebuild my VPS 4 times. I always ended up locking myself out of the server from my machine, but could login using the AJAX shell. At about 1 AM IST, I gave up and got up at 6 AM IST to figure out why. Some time in my sleep I figured out what I might be doing wrong - I was installing Fail2Ban and DenyHosts and probably screwing the configuration. So, I got up and tried with a new build, and this time everything worked. I was able to do the usual user add stuff and sudo permission stuff, lock down root user ID and enforce key based authentication. I had a bit if trouble creating the keys but it went down well. I did not do anything out of the getting started notes at Linode, but only made sure that I did it correctly - not in a sleep :).

What next? I have about 30 days to enjoy or use to do something fruitful before my next job - so am planning to use these days to get some work done - work that I have been cribbing not having the time to do.

My latest shot at writing a killer tool has got 7 downloads - in 2 days - thanks to those who downloaded, would be better if you could let me know your feedback. So am planning to make this better and more functional.

CassandraAdapter needs some fixes to correct my goof up's in understanding if you can call it that. So that's next.

Once these two are done, I will revisit OpenJMSAdapter and add some more stuff there - although it looks like a dead end in terms of ideas that I am getting.

Lets see what I can achieve.

Ravan - my take

2010-06-21T19:13:00.003+05:30

OK, so I went to see Mani Sir's latest flick Ravan not heeding to repeated warnings from critics and friends and wife - I loved it.

Of late, there has been a dearth of movies that have a credible and good storyline. So I have been able to train myself to look at the presentation and technique. You anyways know the climax of the wafer thin plot. Recently I went to see Vedam, another telugu movie, and in the first half an hour found myself drawing parallels to flicks like Crash, Love Actually and others that deal in parallel interconnecting story lines. By interval I could predict the climax.

I remember seeing Dalapathi - my first Mani Sir movie. It was a fantastic movie and a take on Mahabharata. There were so many characters and so many twists in the tale. Each and every scene was full of emotion. The scene where Shobhana realizes that Rajnikant has married another lady, I think Bhanupriya - its a small scene where all the emotions experienced by Shobhana are portrayed in such a short scene.

Raavan - Abhishek and Aishwarya develop feelings for each others in a set of elaborate long scenes. The scene where he could have touched her, but holds his hands just inches away from her and slowly moves away - in the past this could have been handled in less than half that time. I remember quite a few movies where the hero would suddenly take back his hands in a jerk realizing where they were heading, the heroine would grab something to cover herself, a moment of awkwardness and that's it. Here its a choreographed sequence, well shot by Santosh Sivan.

So, why I liked this movie is because of the way it was presented. Simple story, you know what will happen, but each character is well cast, simple lines, no jingoism or over the top shouting. And the frames are brilliant. I remember a shot where a dragonfly was shown in full brilliant colours and a macro close up shot, there were water droplets and mist clearly seen as the hero enters the hut where Aishwarya was ties up, the sun around Aishwarya's face in the Behne De song.

Plus there are are many moments where you can see the action on the screen and feel the emotions going in the characters head - not a one shot awkward moment scene where you get the gist of the feel.

Like some review pointed out, its for the folks who appreciate presentation. Its more in line with international flicks like Offside by Jafar Panahi where the core is pretty simple, its just how it is presented. If you like that kind of cinema then Ravan is for you - otherwise - stay away.

Hajo - A simple RPC interface to Bekeley DB JE

2010-06-21T07:51:00.002+05:30

Last week I decided to learn Apache Thrift. Thrift is a simple cross language RPC framework that was open sourced by Facebook. It seemed interesting and as with any new product that excites me, I thought, its such a simple and brilliant idea - how come it did not strike anyone else before? Of course, I then realized that there are proprietary frameworks from Google, Microsoft and CISCO that they use internally.

Anyway, coming back to my learning, I decided to write a simple application to test out what I can do. Instead of doing the usual "Hello World" stuff, I decided to do something more meatier. I decided to build a RPC interface to Berkeley DB JE.

A few years ago we had to build a Spring+Tomcat based service to let the other applications in our architecture write to a single Berkeley DB JE instance. I took the same thing, and replace the interface with a Thrift service.

It took a few hours to complete and test. The code and functionality are very simple - it just supports three operations - insert, delete and get.

I have decided to call this Hajo - after a place in Assam, India.

The code for this is shared at Sourceforge, and the home page is at http://www.allamraju.com/hajo/. It is GPL licensed.

There is nothing much to look at evaluate in this simple piece of code, but if you ever had the need to build a system where multiple app's want to write to a single Berkeley DB JE instance and High Availability is an overkill, then you can try this.

Why I love Mercurial

2010-06-01T18:20:00.002+05:30

I started with the idea of writing my own projects about 2 years ago. I wanted to contribute to other projects as well, but considering my job and the schedule I keep, it is difficult to commit to making some fixes or improvements.

Anyway, writing my own projects put me in a situation that I needed to have a source control so that I can track my changes to code by some means other than having backup folders on the disk. I had a repository on sourceforge.net , but I did not want to be tethered to an internet connection always even if that was wireless.

I tried installing my own SVN repo - it worked for a while, then I did something and it got screwed. Also the fact that I had to install Apache HTTP Server and have it running in order to be able to commit and that I had to reserve space for a repository and a working copy somehow seemed like an overkill.

Even on the job, having to hold off not committing to the CVS or SVN repo until the code works was sometimes suffocating.

In comes Mercurial like a ray of sunshine on a cloudy day, and suddenly everything is easy.

This is all that I did

Installed Mercurial Windows binary
Made sure it is on path
To make life easy, installed Mercurial plugin for Netbeans
Opened Netbeans and my current project, right click on the project and selected share by Mercurial.

That's it! Then and there out of pure awesomeness, a source control repository got created in my own project folder with no effort!

Then I executed

hg add

All the files got added to the repository - just waiting to be committed. I then executed

hg commit -uhari

A notepad popped up asking for a commit message - keyed it in and saved, closed notepad and behold! the code is committed. Then I execute

hg log

And I can see a history of changes. Made few more changes and few more commits, then typed

hg serve

I browsed to http://localhost:8000 and I can see a neat web page showing the changes, the diff's and even a graph!

Awesome! Now to go and try some branching.

Learning Cassandra - Part 2

2010-06-01T18:15:00.002+05:30

The first draft of my experiment with Cassandra and writing my own client described here is ready.

Working about an hour each night for the last 3 days I have come up with a basic factory to get a Cassandra client and a simple DAO to to the insert and select operations using both default values for keyspace and column family etc and also user specified values. There are of course tests - these tests run against the Cassandra default config that comes with the download.

The code is hosted at bitbucket - http://bitbucket.org/hari/alpha/wiki/Home

Learning Cassandra

2010-05-30T21:40:00.003+05:30

Of late I have been interested in Document Oriented Databases and No-SQL databases. Last week I started looking at Cassandra - the one that runs Facebook. I have just started testing the waters. I am currently looking at the Thrift API and writing some code to test the connection and login mechanism. I did not want to use any existing Thrift client like jassandra - wanted to try and build something like that on my own, something like an adapter which I could later share with others.

I am also using this opportunity to try out Mercurial for my project and see if I can get a hang of branching - this is something that really freaks me out.

As of now no idea where this will finally lead, just hoping to add another nice piece of software to my arsenal.

Continuous Obsessive Tracking

2010-05-30T21:21:00.002+05:30

My current project is the fourth project in a row where I am playing the role of a lead. And the amount of tracking and the obsession with tracking has increased as a geometric progression from the first till the fourth.

In my first role as a lead, I was leading a bunch of folks with similar kind of experience - or the lack of it. And our tracking was limited to me coming in at 8 AM and checking my mail, calling up my counterpart in Plano, Texas and getting the priorities straight. In the evening, as long as I sent a chronological sequence of what we did and the results and there was no issues faced, all I did was send a mail by 9:30 AM their time. Period.

Simple and sweet. We never missed any deadline, no escalations even in UAT and the back to back releases went live.

In my second project as a lead, this time facing the client, we all agreed upfront on the criticality of the deadline and getting the result right and set out to work. There was micro management and the client would sit next to me asking doubts, mail me or call my architect at midnight or weekends and so all those things that you did in a tight project but there was never any shouting or escalation. We agreed, disagreed, compromised but never was an elaborate report to tens of stakeholders required. Just a dirty dozen of us were always on top of things and we never badgered the offshore guys for reports.

By this time I learnt to appreciate the business perspective of a project and costs and could see why they wanted to control so tightly.

The third project was in a mess right from the start - scope creep was the culprit and no one had any option to say no. So we tracked per hour internally, twice a day for the client and our delivery leads. But still the timings were human enough for us. Managers did have late night calls, still only in normal working hours.

My current project has been an exception. An increasing urgency to track. We used to send a report at 9 AM CST, this was advanced to 7:30 AM CST. Fine, only thing is that I now sent more addendum reports. Yesterday was interesting - someone took a report at 5:30 AM CST; we finished our updates by 6 AM CST; there was a small fire. The other vendors drop a mail or call up every hour to ask. And there is a need to keep a mail trail of all this tracking.

Over the last 3 years in these 4 projects, I have seen this exponential rise in the need to track. People no longer believe in the if-it-needs-time-it-will-take-time principle. Time costs and costs need to be down.

But continuous obsessive tracking is not going to help either, because there will be no updates. I would rather have a developer work on fixing the issue than summarize progress every 30 minutes. I would not be happy as a developer if I had to keep reporting rather than work.

Agreed its not that bad that I have to do 4 reports a day, but the rate at which it is going, it might just get worse soon.

Adventures with BW film

2010-05-04T14:49:00.002+05:30

Over the last couple of weeks I shot 3 BW film rolls and gave them for development yesterday. The verdict is out - although I got all the pictures, none of them turned out anything like I wanted. A few came close though.

Bad!

2000 bucks and no good result :(

Need more practice and more steady hands.

I have 2 more rolls, have to use them carefully on portraits instead of experimenting.

Good read

2010-04-17T08:31:00.000+05:30

Of late I have been reading Malcolm Gladwell a lot - I finished Outliers (second time) and Tipping Point and have started Blink. I need to get a copy of What The Dog Saw. I admit I am a little behind the world in terms of discovering these books, but that's the beauty of books - they are never old.

I went to his website today and read up a few articles at random from his New Yorker archive. These two are the ones that I loved the most from the few that I picked (one of them is from the year 2000!).

http://www.gladwell.com/2009/2009_07_27_a_cocksure.html

http://www.gladwell.com/2000/2000_08_21_a_choking.htm

Finally, something that I always said

2010-04-17T00:31:00.003+05:30

Read this article, and if you are an Indian settled outside India, an IT guy on his onsite trip or someone considering settling down abroad - give it a long, hard thought.

http://timesofindia.indiatimes.com/home/sunday-toi/special-report/Oz-us-A-banged-up-equation/articleshow/5826770.cms

No doubt we are welcome in other countries, but only as long as we are willing to mingle in, only as long as we don't flood the country, only as long as we have a legitimate reason to be there and we don't stick out, only as long as we live and let live.

The USA, UK or Australia are nice places to live or were nice places to live once because there was less population, the population was mature enough to do things the right way and follow rules and because the government and social welfare funds were not spread thin over a large population.

With so many people flocking to these places and settling down and demanding the same rights, the resources are spread thin and infrastructure strained. And we want to go there in loads, families, villages and towns want to just move to the foreign land - this is just not sustainable.

These countries were not founded as satellite townships for Indians!

Immutable Strings in Java

2010-04-13T22:45:00.000+05:30

Some nice simple explanations without the usual heavy talk

http://answers.yahoo.com/question/index?qid=20090315113401AAfNJx9

If only everything in life worked this way :)

2010-04-13T22:38:00.001+05:30

http://xkcd.com/726/

Short, simple and no rants

2010-04-13T22:20:00.001+05:30

Short, simple and no rants in my post for a change

http://ibnlive.in.com/news/uk-seeks-to-curb-migration-indians-to-be-hit/113229-2.html

How long before others follow suit?

The developer screwed the architects dream

2010-04-11T21:10:00.003+05:30

Does it sound familiar?

Did you ever hear in any meeting or at the water cooler that there was this great architecture designed by this great architect in his moment of great inspiration that got screwed because the wrong developer worked on it or because the developer did a patchy job?

Most of us have.

I walked into my office one day to overhear a discussion on why SOA was a great pattern, and yeah the SOA architects agree that it is tricky to implement, it is the developers who screwed the implementation. Great start for the day! I was thinking of retorting but kept quiet because there was no point.

A year or so ago I was part of a project to build a high volume system. Everyone right from the client side manager and developers to my technical architect agreed with the need for such a system and also to the fact that it needed to be a custom hand crafted solution much like a Ferrari engine. Or for that matter any great sports car. About 5 of us designed a system, each day painstakingly going over every if and but and every possible combination of boundary cases. Like they say, the devil is in the detail. We nailed down every last detail like the order of commits and the number of commits and by the time we went to code, we pretty much had an idea of how it would work.

12 Java services running as UNIX daemon processes, a web application reading a database cache every half a minute to display a real time status of the system, a 6 layer JMS server infrastructure that spanned from Americas to Japan and an API that helped about a 100 systems to pump a million messages an hour for 14 hours a day.

It worked.

It worked well.

And the bunch of developers who build this, apart from me and the architect, were more or less kids. Hell even I was a kid considering the client side team was seasoned veteran's in the world of software development.

There was a dream, it was realized by normal developers and it works.

We succeeded because
We wanted to succeed
We took 12 hours to build something if it required 12 hours
We kept it simple
We nailed the devil in the detail quite early
We did not rush
We were not afraid to throw away something and start all over again
We were not afraid of the client and told him the truth, no matter how hard it was to swallow
We did not start by defining a deadline, we started by defining what we want

Unfortunately, not every project works this way.

Some senior guy playing golf mentions that his systems are spread thin over a large user base and there is no central way to get a hang of things.

Genuine concern.

A consultant from a vendor or a technology guy from the same firm mentions that maybe we could use the latest hot thing in the market to build a solution.

Good.

The technology guys looks around if anyone in the company can do this, and probably because no one is there, or because it would cost to move people around to get this done internally sends it to the outsourcing department.

Hmmm.

They figure out whether to use the existing vendors, or new ones knocking on the doors or with a foot in the door already and send out an RFP.

There is no shortage of people willing to do things for less and faster.

There is a clamour to answer the RFP.

Who is quoting how much, and how fast are they willing to do it? Can we do it better? Can we leverage something that we already did or have in the cans? Can we pull better strings than the other guy?

And in all this the technical aspects are left out until the 13th hour. Finally they find an architect or a already implemented solution to leverage. They throw in some new things, a few terminologies, throw our specific details, anything that can get work done, add a splash of colour, logo and powerpoint

animation and finally find some smooth talker sales guy to sell the idea.

Anyone remember Armageddon and Bruce Willis?

"If we use Liferay portal , then we can plugin any web application or build a new one and release the new feature in a matter of days - trust me it is doable by just drag and drop"

A small demo hacked together by some guy doing a late night and the audience is hooked.

Truth is, if you want to replace the Liferay logo with your logo, it takes about a days effort - its embedded in code and uses a MS imaging library to cast a shadow etc etc and if you plugin any skinned web app to this, then your skins are screwed.

You need to build then in a certain way to plugin seamlessly to Liferay.

I am not saying Liferay is bad.

Assuming too much that things will work out of the box is bad.

If you base all your architecture on the fact that a certain thing can be developed just by drag and drop and you don't need to worry about transactions etc because the underlying platform takes care of everything else - good luck.

The platform was build by developers chasing an equally bad deadlines.

Ship this by yesterday or its your ass on the line!

Wow!

I once heard of a proposal where a supposedly good guy mentioned in his estimate that you can take a C++ based MQ 4.x application and migrate it to MQ 7.x in half a day. Yeah sure as hell you can. Install MQ 7.x, copy the application, change the paths and parameters and then hope that since you have been going to the church or temple or mosque every week and been a nice kid all the year, the changes in the libraries and platform from 4.x to 7.x will not matter and things will just run.

I don't know whom to pity.

The guy giving the estimate, the developer working on the thing or the architect or the guy at the water cooler who thinks that if he had better developers, it would have been a success like none before.

On a large enough time scale, the performance of any software gets screwed.

Its the environment, the loads, the network or simply how your code juggles around the bits and bytes.

If you take the analogy that getting a job done by a software is similar to a juggler juggling balls as he gets from point A to point B, then it will be a real shitty piece of software if the juggler dropped the balls every step of the way or threw them so high that he had to wait a long time to catch it again.

So all you managers,sales folks and architects out there, remember this

Don't go by the sales brochures of the products you use - there is a bunch of guys selling lies to meet targets just like you on the other side - and in many cases they sold you a decent lie or a fact that needs a better than average brain to implement correctly and that brain costs.
Writing a piece of code is tougher than coming up with the flow or the architecture
If you want to work backward from a deadline, then make sure you have a long way to walk.
Developers cannot start creating miracles after two weeks of training with good lunch.
Not everyone gets it.
You have approval to work and bill 25 hours a day, but the brain gets fuzzy after about 10 hours of work.
Nail the devil in the detail and not in the shade of blue on your powerpoint slide.
Some things cannot be done fast.
Some things cannot be done cheap.
Some things cannot be done by freshers right out of college.
The cute girl in your team can flash a confident convincing smile at the visiting client in a presentation but the same smile cannot get all the code to write itself.

The first folks to build software did so because they loved the challenge - that's why Steve Jobs and Steve Wozniak could sell their computer.

The next generation built software using the high level languages that made life a bit more easy.

The one after that were drawn in by OOPS, WWW etc etc.

The first batch of Indians were drawn probably by the Y2K bug and opportunity to fly to America, but they too were interested and challenged by the technical details.

Then the ERP packages came and people were drawn by the ease with which they could implement things.

Some of us were lured by the lie that Java has everything inbuilt and you can build almost anything using the SDK.

I was told that in TIBCO, I just needed to drag and drop.

The current batch of folks are coming because the job market is good, the pay is good, and everyone wants to go onsite.

My neighbours son makes a 100K a month, lives only on planes flying around the world and has never returned to India since he left a few years ago.

I want to do that,nothing less.

I want to go to US before I think of marriage so that I get good dowry.

With these kinds of ambitions in life and the work hard party harder attitude thrown in, most folks at any level don't bother diving into the details.

Sell fast, sell quick, get out on a profit.

Do something for an impossible deadline, work late hours, bill more, utilization shoots to 200%, show that as an achievement and mint more money.

>100% utilisation - how the fuck that even make sense, much less sound good?

Go onsite, smooth talk and sweet talk the client, manage him, settle down there.

And when the application goes down, catch the developer in the night shift or whatever it is.

If shit hits the fan, go to the water cooler and say - "It was great architecture, but the developers screwed it".

And then, start the same thing all over again.

How do you solve a problem?

2010-04-08T23:48:00.000+05:30

How you solve a problem and more importantly whether you solve it at all depends on one thing - how you start thinking about it.

Consider this problem - you need to build a simple user interface using a GUI toolkit.

If you posed this question to a guy who has no goddamn clue about programming you will get a blank stare or an inquiring counter question.

Ask a newbie to programming and you might hear him say he does not know it.

An expert will actually build it for you.

In the last six years in software industry , these are a sample of responses that I heard -

I am not familiar with this
I am not so good at programming
You should build this in visual basic , its just drag and drop
You should move to rich web based user interface
You should try ruby on rails, it builds the application automagically
Did you try xyz tool instead

A few times I have seen the guy actually mock up the screen on a piece of paper and talk about spacing etc. Very few occasions where they even talked about look and feel.

On one occasion they built a GUI so badly integrate to the windows look and feel that it looked gory.

I do not mean to say these approaches are wrong.

I am not saying these will succeed or these will fail.

This is how I probably will think in an ideal situation -

What is the target audience and what do they want?
Will it make sense to guild a desktop application and deploy that across so many systems and maintain it?
What platforms do we support and what sort of features do people expect
Which technology supports all the things that we want
How many folks know it, how ready to learn it is and what sort of paid or community support is available for it
Will these technologies and licences be accepted by the security and compliance teams

These six steps will merely let you pick something that will be accepted by one and all to a large extent and give you the buy in necessary to go ahead and solve the technical issues.

How u solve those is another story, but this is something you could try -

Choose a colour scheme that integrates into the platform or is consistent with the clients brand or logo
Ease of use with clear font
Field and text following the logical flow of thoughts when a person manually performs the functionality being implemented on that screen
Feedback - whether something in the input or the background process has failed or is taking too long - the user should know clearly and in advance
Consistency across all screens and pages
Acceptable response times

While this is no exhaustive list, it does set the general direction of thought.

Replace the GUI with a command line application or maybe a problem in a completely different domain other than the software industry - its the same thing - there is a set of simple questions that will put you in the correct direction and let you solve the problem.

What if you don't follow these?

Nothing - you will still get there but with a cost.

You decide if it is worth it.

Travelling rough

2010-04-04T22:14:00.002+05:30

Note: Of late, I have been writing only philosophical blogs about how the work place should be etc etc. And I must admit I sound pretty morbid. So, trying to write something a bit more cheerful here.

All of us travel and most of us want to travel in comfort with a definite plan in our mind.

I prefer to travel rough - get to the destination, by any means, enjoying a small amount of unpredictability or sense of adventure, not necessarily the most comfortable. I seem to have inherited this from my dad. He likes to travel and explore. He has gone to places without knowing anything in advance, got there, figured it out, got his work done and returned. And its very few times in all these years that I have seen him travel with an advance reservation. But he always has a plan in his head.

I do things somewhat similarly, I prefer to have some sort of a reservation apart from the plan.

Ever since I got a job and have had the chance to travel without the pressure of being in office at the end of the journey, I prefer to travel rough.

I have travelled from Hyderabad (where I lived) to Rajahmundry (my native place) during the day changing two or three buses over the 500 odd kilometres without a reservation. Get to the bus stop in Hyderabad, take a bus to Vijayawada - any bus that has a seat - get down in Vijayawada, eat something and catch a not so luxurious but fast bus and sit with the general public in the sweating sun and get to my destination by evening.

I did an even longer journey once - Bangalore to Vijayawada by bus, get down in early morning, wash my face somewhere in the bus stop, get onto another bus, spend three hours or so and get to Rajahmundry. On the return, same thing in reverse.

I enjoy this, and feel bad I am not doing the hitching a ride kind of journey - after all I just have a back pack most of the time. But folks at home are not impressed. My wife not one bit. She doesn't understand how I can travel in a bus for such long distances :-).

I need to drag her along with me once to show her the joys.

This week am making a bigger plan.

Bangalore to Tirupathi - I have some idea about Tirupathi.

Darshan in Tirupathi.

From there somehow get to Vishakapatnam.

Attend a wedding there - I don't know the city much.

From there to Eluru - still no concrete plan how.

Eluru to Vijayawada - this bit is easy, I have a bus every 15 minutes.

Then Vijayawada to Bangalore - I have a reservation for this.

A total of a 1000+ kilometres and 4 days.

Should be fun.

And am planning to take my camera along - yet to decide if I take the BW film or the digital one.

Delegation killed the spirit of tracking

2010-04-04T21:29:00.008+05:30

Almost all of us in the IT world have deliverables. And we report on our progress on a periodic basis. How frequently or infrequently we do this depends on the supervisor we have and the amount of visibility our deliverable has.

You are given a deliverable.

You start working on it and get a gut feel.

You tell this to your supervisor or whoever is tracking it for you.

He records it somewhere - paper or excel sheet etc.

Every now and then the two of you talk and update the status.

End of day you consolidate and go over the things and send a status report.

You are doing the hard work.

The person tracking you is also working hard - he has to track multiple people like you and maintain an up to date status.

In a normal situation this might be a cool process.

Work the full day and send an end of day report.

It gets complicated when there are multiple stakeholders who want regular updates.

To get over this we probably maintain the tracking and status in something like a Bugzilla or Quality Center - everybody can see what you updated and what you are working on.

It comes down to us to make sure that we update the latest status in this application.

But what do we update? A significant find? Or that we haven't found anything?

Why do we update? To share our findings? Or to let the world get the impression that we are working?

Should we concentrate on the update or the work?

Till sometime ago, when life in the IT lane was cool and we did not have every other team breathing down the neck of the other teams, you would come in, get a bunch of deliverables, start working on it, agree on periodic meetings and get things updated by the time of the meeting with the most latest and most sensible sounding update, along with an ETA.

But now, with tight deadlines and high percentages to achieve per day, people are asking for updates by the minute.

Imagine this - you walk in, check for defects in your tracking application, no defects, you open BBC to check news, and in comes a mail saying that defect number so and so is in new status for a long time - 2 minutes may be - and you are supposed to reply. You reply politely, your boss comes running to ask you why its not looked at, the test team boss walks in with a I am the person in control attitude complete with power dressing et al and asks you for an update every 15 minutes.

Whew!

What would you do? Send a mail every 15 minutes saying no major updates? Bother the guy working on the defect every 5 minutes or 15 minutes? Drive him crazy?

We need to learn to relax.

And the guy doing the tracking needs to apply some tact here. Its needless to say that the boss needs to support you in this.

People don't like to be tracked by the minute. Neither the tracker or the person doing the work on the defect will like to be asked for updates by the minute.

Human beings like to function with a bit of autonomy.

Provided the guy is averagely intelligent, he will prefer to be given a task and left alone to work on it. He will get back to you in case he is stuck. Yeah he might be dumb and not know that he has to ask in case of trouble or in today's case of frustrated employees, he might just not have the will to work. But these are small percentages.

Most of the guys will do their work, update you or ask you for help.

Small percentage will need to be pushed.

If you have always seen the kind that needs to be pushed, the please believe that the world is not all bad and your faith will be redeemed some day.

Any work place where individual pace and space are respected or acknowledged at the least will be a nice work place. Agreed that the bosses boss is breathing down the bosses neck, the boss has to show some result so he breathes down your neck and you in turn turn the heat on the poor developers ass - what can the developer do?

If he is any good, he will just quit and go to a better place. If its your bad day he might just shout at you.

There has to be a barrier somewhere that shields the folks below.

Your boss or you need to stand up, and say "folks, I know you need updates or results, we are working on it and it will take X amount of time if it merits X amount of time, so please relax and wait for our updates". You need to shield people under you. Don't let them waste time, shield them, ask them nicely and put a gentle pressure that accelerates the pace and still doesn't hurt anyone.

And this requires balls.

And this is where the beautiful concept of delegation ruins everything.

As a task climbs down the ladder from the management to the shop floor, it gets pushed from folks who can see the big picture to folks who can see the little discrepancies. At each level, the guy goes - "hmm, OK I don't know this, let me shove this down. This isn't part of my role". And so on so forth until it gets into the hand of someone who cant delegate it any more.

None of the folks above this poor chap had the balls to stand up to the pressure, and it don't matter if this guy stands up. There will be another guy willing to do it without standing up.

And the whole process repeats.

What was supposed to be a process becomes a pain.

The ability to judge when to delegate, when to push back, when to pressure people under you are matters of common sense that get refined with experience and little bit of education.

Only then can delegation or tracking will be effective.

The flow of ideas in a piece of code

2010-03-23T23:18:00.003+05:30

Try this exercise some day. You have a nice dream, you enjoy it. Then ask someone to dream a continuation to the same thing. And in case he does a shoddy job of it, ask a third person to take it to a proper conclusion. Why? Because you are needed for other things in life.

Impossible? Weird?

OK, try this then. Pick a topic, express your thoughts in a paragraph. Ask another guy to write the second para and another guy to write the third para. See if the essay makes sense and has continuity.

Weirder still?

Now imagine someone writing a long piece of code. Another writing or rewriting some part of it. Then a completely unknown guy coming and trying to fix this based on the code he can see.

What happens?

Defect.

A bad fix.

An escalation.

A delayed deployment.

You basically got where I am headed.

Code is a way to solve a problem. To solve a problem you need to define and quantify the exact steps that need to be executed. To do this, you need to know the problem domain and understand the scenario and you need to arrive at a possible set of steps and pick the best one depending on the time and resources and requirements you have. Making it future proof is another story altogether.

So if you need to get from the problem statement to the code statement that solves the problem, you need a single continuous thought flow. And you can achieve this the Archimedes way in your shower and say Eureka! or you can get into a room with a few like minded folks and agree on a way to join the dots.

But either way, once you agree to an approach and everyone can recite it in sleep, it stays that way. CMM and other such maturity models tell that if this approach to solve a problem is arrived at using certain tools, techniques and processes and is documented with sufficient examples, then it can be repeated again and again much like stamping out components from a mould.

So you have a project, working fine, well documented and following the above mentioned CMM stuff. So you get a guy to design some fix, another guy codes and tests it and then finally it fails in user acceptance testing. You grab a third guy to look at the defect and fix the issue, deadline being yesterday as usual. Yeah, it will work you are CMM certified, no problem whatsoever.

But it fails.

It fails for the same reason why Newton discovered gravity after seeing an apple fall and others merely satisfied their hunger or made a quick buck selling it in the market.

Human brain. although remarkably efficient, and made up of the same kind of neurons, acts differently for different people. Their neurons don't fire the same way. The second switch in the switchboard does not always turn on the light in every room of every house in the city.

When a problem is translated to code, it follows the flow of thoughts in the mind of the person coding it. The thoughts follow a correct logical path, we get working code. It gets diverted by the girl in the red dress or the handsome hunk 2 cubicles away and we get a defect. So if the person fixing your code two months from now has a look at your documentation of the code, he might not get the exact same idea unless guided explicitly.

He might get a better idea.

Or he might get it ass backwards.

If the flow of ideas is not correct, god save the fix.

We cant really eliminate this risk, but we can mitigate this. We need to understand that the person who wrote the code needs to be held accountable till the moment the code is deployed to production, at least, or maybe till way after that in the maintenance phase.

Of course, this means that there will be dependency, project deadlines will be pushed and the overall schedule will be delayed - meaning more cost. But that's what is good.

Tell a guy that he will have to pay for his today's carelessness tomorrow and he is less likely to be careless.

Same applies to code as well.

Hurrah! IT Jobs are back!! Are they?

2010-03-20T20:04:00.002+05:30

Couple of days ago I came across a link somewhere that said that CEO or some big shot at naukri.com, India's largest job site said in an interview that there were up to a million jobs in the market and that it was a great time to be looking for a job. And I wondered, really? Are so many jobs being created? Surely I did not hear the government opening some sort of a compulsory job guarantee scheme for the urban folks or the IT industry and definitely no news of any Indian IT company picking up a mega contract. I also heard in the news and saw around me that many people are quitting their companies for reasons like no promotion, change of promotion and career strategies and the most common reason of not getting a hike. It doesn't take much of an effort to put two and two together and realize that the jobs are not there because there is a demand, but they are there because the companies are expecting to see a good amount of attrition and so are hiring to make up for what they lose. If my company expects to see the demand for 5000 people in the next few months, then it surely expects to see 4000 people leave in the same time. So, there are no new jobs, just people moving around from one company to another, and just like the law of conservation of mass, everything remains the same.

Why are people moving around?

Back in 2004 when I got the job, it was after the downturn following the dot com bust, most of them in my batch at an Indian IT major was my senior who had waited for a year before getting a job. I went to interviews where I met my seniors. The market was opening up and jobs were plenty. I was put in a unit dealing with enterprise package implementation. The work sounded simple to most - lean the package, learn the modules, implement a few things well and you are a guru. After all there are only few ways in which you can implement something in a packaged software as opposed to a full blown programming language and its libraries. Pretty soon you would be teaching someone junior, boss would notice you, write something nice in your appraisal. And since the work was coming and people were less, you moved ahead in the job fast. Software Engineer to Team Lead in 3 years, Project manager in 5 years. I was pissed off when at 2.5 years 80% of my batch was made Team Lead and I wasn't. The fact that I did not have the maturity till another 7 months and a tight project later did not matter to me. Foreign trips where batch mates just quit and joined some other company for a higher package was common. And when I came back from my first 8 months in London and someone asked why I came back and if something was wrong, it seriously rubbed in the wrong way.

I was not alone, many of us grew through the last 5 years getting quick appreciations and fat hikes and onsite trips. Not many were grounded well enough to realize that the party would not last long enough. I got the first indications when I went oniste and my counterpart called me a baby. I had 4 years behind me, how dare he! Truth was, he was right. Compared to what he could, I was nothing and the appraisals I had behind me were not worth anything. 1.5 years later, I am happy I got an appreciative nod.

Its the same story with all of us who grew up fast, lived a fast life climbing the ladder and not realizing that at the top, we needed stuff. Orgasming about juggling spreadsheets and waxing eloquent about project management after just 5 years was not going to help us. Maybe with US clients, not with European ones. Not with clients looking for value for money, even when we do the job at a measly 11$ per hour.

I mean how many times in India have we haggled with the shopkeeper about the cost and how many of those occasions have we also been hell bent on getting the good thing for less?

Would a client running a billion dollar business pay your company millions so that you and your team could take your families abroad, pay your home loans, buy your dreams stuff and three months after you deliver the software goes bust forcing the guy to build it again? Or for that matter something going wrong and the 200$ per day Indian IT consultant calling up his offshore counterpart to get it fixed? I have been in situations where offshore sends a design for a screen that does not look like standard Windows application, has a horrible shade of blue and they feel it is a good job. Try telling the client its OK to have that shade. Obviously the guys aren't taught to appreciate and follow the nuances of GUI design or are too raw to understand that.

My boss told me to build a screen, I built it, so appreciate me, give me an A+. Any donkey can do that. The client wants you to do more than that and that's implicit. Think about aesthetics, ask around if you don't know, don't screw with colours and logos of the brand, use common sense and build something the client is not forced to throw away soon.

Sounds like I lost track of my core topic didn't I? I sound like I am venting my anger at someone or the system?

Nope.

All these problems are what is forcing the companies to revise their promotion policies. The client getting aware of the wastage and getting wise about the money he spends is what is causing them to stop your hikes - plus margins.

But folks haven't realized this. Oh crap, I did not get a hike here, let me go somewhere else. Talk well on the interview, sound tough and bargain for a hike, get about 40% hike and move on.Well done. You go and your company hires someone else. There is no shortage of dreamy eyed folks wanting to work for the popular brand of IT company. Ask any guy who works for a small in India but a good technical company who was asked by his prospective in-laws why he could not get a job in the BIG-4 of India?

People are moving from bigger to bigger and smaller to bigger companies,some are even moving out and getting hikes.

For a while everyone will be happy. Then what you ran away from, catches up with you. Like the tortoise catching up with the rabbit. Next year, again you will get a single digit hike, your boss will ask you to do more than turn up at 9, leave by 7. And then these people will migrate again. Add to that the inflation, the auto wallah, and every other guy you pay to live a free bird life in a city devoid of any responsibility or parental pressure/interference, all these folks, including me, will be in a position where the annual credit card spending overtakes the annual income.

There are no new jobs, its just musical chairs at a higher level of abstraction. And some day the music will stop and there will be no chair. No, there are no less chairs, just a few more freshers out of the college willing to work for a starting salary less than yours and a management which thinks coding is just typing.

Yeah, I know - I sound morbid.

Tokyo trip

2010-02-16T19:39:00.002+05:30

I am visiting my wife here in Tokyo for a week. The trip has been good so far. The weather is cold, windy and wet. Managed to catch some sun when we went to Kyoto over the weekend. Last two days were spent in visiting Odaiba, Meji Shrine and Shibuya in Tokyo. Managed to grab a Japaneese style lunch with my wife's boss.

Talking about Shibuya, if you saw any movie shot in Tokyo and saw a BIG intersection with many people and large screens - in the movie Jumper for example - that is the Hachiku intersection in Shibuya - worlds busiest intersection. See this video - its crazy. I crossed it a few times and sat in the starbucks overlooking the square.

Will upload the best pics in about a week.

OpenJMSAdapter 0.1 Beta : Ready for download

2010-01-30T17:11:00.004+05:30

I have uploaded the first version of OpenJMSAdapter today.

A very basic version of help and getting started is up at allamraju.com/openjmsadapter

Have a look, and let me know what you think.

Getting ready to release OpenJMSAdapter 0.1

2010-01-28T19:14:00.003+05:30

I managed to resolve my issues with running tests in Netbeans - partly a corrupted project and partly my stupidity. But finally everything is green.

I did some basic perf tests and with all my code between the send() call to the point where the message ends up on the JMS server, I am able to publish 110 message per second on my laptop and receive at more or less the same rate.

As of now, you can

Configure the adapter to send or receive messages on a queue or topic using Yaml configuration or a database table.
Send or receive BytesMessage, TextMessage or ObjectMessage.
Ready clients are available for ActiveMQ and OpenMQ servers. I know this will not be sufficient, so you can write your own client and plugin to the adapter and the low level code will do the rest for you as if you had a ready built client. Writing this client is trivial :).

The next steps for me are basically to write some examples and a getting started guide. I am not really planning to but might add the javadocs and comments to the code as well.

Hopefully by this time next week, the first file will be up for download on sourceforge.

OpenJmsAdapter - Roadblocks

2010-01-27T22:45:00.004+05:30

Something has gone wrong with my Netbeans. Over the Republic Day holiday, I could not run any test on OpenJmsAdapter! The reason strangely is that it cannot find the test hibernate config files that it could find so easily in the past. The files are not getting copied to the test folder under build!!!

This needs some head breaking this weekend. Hopefully, a fresh checkout should do the trick - in the past I have corrupted a couple of projects in my local copy and a fresh check out solved it.

OpenJmsAdapter - Draft version ready

2010-01-17T11:42:00.002+05:30

The first working draft of OpenJmsAdapter is ready. I spent the new year weekend coding the basics of the thing with a few tests and by last week managed to get all the layers of code ready. I started with ActiveMQ and built a client code that can send BytesMessage, TextMessage and ObjectMessage on destinations configured through a YAML file and maintain the sent/received sequence numbers. Then I managed to refactor out the common code and write a client for OpenMQ server with minimal effort.

The code had to be tweaked around a bit and the tests made my life easy. Basic performance tests on my kind of overloaded laptop showed about 120 messages published and received ( not simultaneously) with ActiveMQ.

There is still more work to be done before I put a release out. I need to put a SQL configuration apart from the YAML configuration. And make sure I have more tests and comments etc.

Will keep updating here.

Starting a new Project : OpenJmsAdapter

2009-12-28T21:39:00.002+05:30

After two days of deliberations and thinking I have decided to start a new project OpenJmsAdapter.

This is not actually entirely new - this was something that I thought of about an year or so ago and then put on hold due to the fact that I was working on something similar on my job and did not want to mix the two.

The idea is simple - we use JMS as a messaging backbone for enterprises. A lot of boiler plate code needs to be written to connect and publish/subscribe messages from the JMS server. Of course our life is made easy by frameworks like Spring that make it easy to ignore the low level stuff and just publish messages. However, these miss certain value add features like

Automatic application level message sequence numbering
Checking message sequencing (This does not make sense in JMS in most cases).
Publisher heart beat - is the publisher alive or not?

I am still coming up with ideas and fleshing the thing out.

The only thing that is certain at this point is that I have started this project in out and out TDD mode (as against my usual urge to go straight to code and test lazily) and am using Netbeans, with ActiveMQ for my local testing.

I have only managed to write and test two classes today - should commit something by new year.

500 mile email problem

2009-12-27T15:18:00.001+05:30

500 mile email problem - brilliant read. I was rolling laughing at this one.

Need ideas for a project

2009-12-26T16:35:00.002+05:30

Of late I have been thinking of devoting some of my spare time to developing software tools that will help me to polish my skills as well as provide some amount of financial incentive. More than a year ago I had spent a weekend coming up with a draft version of Console4Jython. I spent a few months after that actively adding more features but later got into loads of work and things took back seat. I want to start again. I am not thinking of developing Console4Jython into a bigger IDE - there are much better tools out there. I am looking at building some kind of a tool or framework that fits into the enterprise world. Or maybe even contribute to one of the existing projects. I can code comfortably in Java and want to improve my Python and learn some Ruby.

Let me know if anyone has any good ideas.

Industrialization of IT

2009-12-24T20:29:00.003+05:30

I am not very knowledgeable about this and so am open to corrections. Industrialization resulted in goods being manufactured fast and cheap due to economy of scale. Make 10000 units at once and you could save a lot of money by buying the raw material at once and using a decent set of machines. This led to more supply and things being available at a lower cost. There were countries that were big time consumers and then there were countries which were big time producers. The producers soon started to make more money selling these goods and started consuming themselves. Due to increased demand the prices increased and many things happened. And we now find ourselves facing today. Whether today is good or bad depends on who you ask. For me it is a mess simply because we cannot sustain it.

So, when I heard about strategic industrialization of software services industry - my blood boiled.

I have been in meetings where we go with an estimate of x number of guys working for y number of days and x*y being the total cost. The estimate is arrived at assuming that we can use our fresh catch of resources to get the things done fast by leveraging our prior experience and reusable artifacts. Then the client asks - "can you do this thing any cheaper and any more faster?". Thats another occasion when my blood boils.

A computer is a machine that we invented to execute a know set of logical steps repeatedly, at a speed that has been growing over the years and produce a repeatable result. But the logic itself needs to be arrived at with sufficient optimization built in in order to produce a decent result. And this requires a capable mind.

Agreed that past learnings and best practices can be leveraged, combined with a certain level of ability on part of the person creating the logic - we can get a good product that can be fine tuned. But, people forget that

The learning required and the amount of time needed to come up with a solution to a problem differs from problem to problem and cannot be put into s strict mathematical equation
A certain amount of intuition and common sense are needed along with ability and leverage to be able to produce a workable result
What works once will not work everywhere else
And finally, the human element involved here is a big factor that affects the final result

Software development, testing or for that matter maintenance cannot be compared to a car assembly. Even a car, which has few hundred parts, after being assembled with precision instruments and coming with extensive manuals on how to maintain it, and with suitable tools only runs as good as the owner and the mechanic are. A bad mechanic can screw a perfectly new car, and a bad owner can screw it much easily if he does not pay attention or run it properly. Software is a much more complicated piece of work and more easily prone to the error of the human element.

IT cannot be industrialized.

The industry has come down to a level where the vendors, based wherever they are in the world are charging the lowest possible to get projects done and still make a margin. All of them are quoting in the same range of price and soon will not be able to go anymore low. They are losing their price differentiation.

They will then start looking at the new differentiator - productivity. This depends on experience, expertise. And this is where the kinks in the armor will start to emerge. Not everyone can learn at the same rate. Not everyone will be able to get the same amount of work done in the same amount of time. Some will just not get it. Some will pull their hair, smoke a few cigarettes and go mad. After all we are human.

So from the crowd will emerge those who survive - the fittest. And when Darwin's theory shows itself in the IT industry, there will be a rush, a competition for these fit and productive men and women who can deliver what is needed in the time that is there. And sadly they cannot be cloned. They cannot be cast into reusable entities that provide the leverage to run IT businesses at a lower cost. And thus the cost will increase. We can no longer mass produce software at the low cost.

Well we could stop going lower or trying to find a new differentiator and stay with the current situation reaching an equilibrium - but thats not what the vendors got into business for - they would rather shut shop and exit.

So it is destined that we walk down the path to our doom.

Someone said somewhere that quitting when things are good is not s sign of being a loser.

The cost of living comfortably

2009-12-19T18:16:00.004+05:30

BBC happens to be my favourite source of news on the internet. I like the opinions expressed there. Today as I was browsing through the site and reading an item on the strike by British Airways cabin crew and the court stay order on the strike, one line struck me. The author mentioned that because BA had its hub airports in London and other high price cities, the staff would find it difficult to survive on the lower pay being proposed. If BA could move the hubs to places where the cots of living was less, then the staff would be able to survive even on the reduced pay.

Although happening far away from my current home in Bangalore, the problem sounds very familiar - the cost of living comfortably.

I grew up reasonably well off, never short of anything and never lacking in any of my needs. Probably the only time I felt I could have been better off was when I saw scores of my engineering batch mates heading to the US for higher education. But that was short lived. So, at home we are used to a very comfortable, but not high expenditure lifestyle. We spend, I splurge, but all within reasonable limits making sure we do not buy something that we will throw off very soon and always balancing the cost and the best thing available in the market. I don't mind spending 15000 yen to send flowers to my wife in Tokyo, but would definitely think twice before buying a pair of shoes at 7000 rupees. One is worth it, the other definitely not considering the dust and wear and tear the shoes will have to face when I am using them.

All of this comes at a cost - and that cost has been spiralling.

When I got a job five and a half years ago, a salary of 14,500 rupees was good enough to live. I lived in Bangalore, went home twice a month in an AC bus paying 1000 rupees for a round trip, religiously sent 5000 rupees home and even managed to pay my insurance premiums.

Over all these years I saw the prices rise. The house owner jacking up the rent by a few thousands because he saw in the paper that my company was giving a hike. The bus operator charging a 100% premium on the ticket price because of the rush and 99% of the passengers were IT folks, Inflation or simply because the new batches of students joining IT were willing to splurge.

Picture this - the fare for an auto for 3 kilometres in Bangalore with a normal meter comes to 25 rupees. With a faulty meter, the auto driver could subtly get this up to 35. Anything more would stand out. Then one fine day, some auto driver decides to stick out his neck and demand 50 rupees for the simple reason that the area is a bit interior to the main road and he might not get a passenger on the return trip. Common sense would demand that we question him and make him come for the normal fare. But we don't - being Indian to a certain extent means that we never stick our neck out an fight. The auto driver is fighting for an unfair cause but we wont fight for a fair reason. As a result, soon the auto fares are decided on the spot by the whim of the driver and not by the meter.

The same behaviour applies when the coffee guy decides to charge a few rupees more, the guy who sells you your lunch, the vegetable vendor, the paper boy and each and every person with whom you interact with using money as an instrument. The value and the cost of everything and anything is bloated and we paid.

Part of this stems from the fact that for a long time now, Indian mentality has been that a person spending more money is considered well off and held in high esteem. Couple this with the myth that IT is a well paying and respectable job and the hoards of people running out of college and straight into an IT company and from there to the United States of America. People were shocked when I came back from London. It did not make sense to some why I would even want to come back - was something wrong with me?

The result of this today is that it costs a lot to live in India. People in the non IT sectors have started earning better, sometimes more and people who run businesses that serve the large bachelor populations of young IT professionals are making more money. Couple this with the fact that IT professionals have become commodities who can deliver the clients requirements cheaper and faster and companies are vying with each other to do the work for lesser and lesser, we are truly heading for a mess.

I recently switched jobs, leaving a place where I was very well known for a place where I am relatively unknown as of now. I don't see much difference. The projects are being executed the same way, maybe a bit more disciplined. The folks are equally worried about the hike - the cost of living is increasing for them. Its the same thing, branded as a different company.

I am still counting my expenses, I am still worried about my loan payments, only change is that now I have a slightly bigger margin of error - which I am sure will erode in a few months.

What am I getting at?Surely I must have a point in this rant. Well, I do.

We as a nation, as a world should stop being obsessed with spending, and the ability to increase our ability to spend in a geometric progression. It simply is not sustainable considering the fact that the very organisations that want us to spend on services they sell are trying to pay us less and less to produce those services.

I am not able to live on a low salary and help my company fatten its margin simply because my company over the years has poisoned the ecosystem around it to become more and more costly and made life costly for me. So I switched and joined another organisation that is pretty much doing the same but is paying me enough. Few years later I run out of here to another place and then to another until finally there is no place to run. That day you will find me as a tramp on the roadside. Probably there will be many more tramps with me.

The cost of living comfortably is increasing day by day and will reach a point where it will be cheaper not to live.

It doesn't matter which country you have emigrated to or how much the exchange rate is.

There is a quote from The Matrix:

Agent Smith: I'd like to share a revelation that I've had during my time here. It came to me when I tried to classify your species. I realized that you're not actually mammals. Every mammal on this planet instinctively develops a natural equilibrium with the surrounding environment, but you humans do not. You move to an area, and you multiply, and multiply, until every natural resource is consumed. The only way you can survive is to spread to another area. There is another organism on this planet that follows the same pattern. A virus. Human beings are a disease, a cancer of this planet, you are a plague, and we are the cure.

True.

Change in workplace

2009-12-16T18:46:00.002+05:30

After 5.5 years at Infosys Technolgies, I have made my move. I am now employed at Accenture in their Bangalore office as a Team Lead and am looking forward to some fun work.

The decision to change was a difficult one that had to me made. The hunt for the job itself was another story. Had two or three small interviews that I did not clear for the simple reason that they did not excite me enough and I did not prepare well. Had the chance to get into a big investment bank, but unexpected delays made it infeasible.

The most fun part was my interview with Thoughtworks. They called me up on a Friday to do the initial phone screening by HR and after an hour he was convinced I was ready for their coding assignment. That weekend we had a major escalation at work and in between fixing and testing things, I found time to solve the Mars Rover problem - I even rewrote the thing twice and added a lot more tests than planned.

That code made the cut, and I was called for an in office interview. I did not make it, reason being I did not have the depth they expected (I shall reserve my comments on this), but the whole experience was fun. The logic and analytical test they give is good. My pairing round was miserable because a)I have never done pair programming and b) The guy was not thinking on the same lines and was putting brakes on my thought process. I am surprised they took me to the next round then found I was not up to their expectations - frankly I did not expect to get through after the pairing round.

But finally, I have joined Accenture. Hope I do well here.

Long time no blog

2009-07-31T19:22:00.002+05:30

I haven't been blogging at all for the last 3 months. I have been moved back to Bangalore for a project where I do not have any internet access due to client restrictions. And I have been staying in a single room accommodation with no internet connection because it is a very short term assignment. And am working my ass off.

All that is nothing great, I am getting married in two days on 2 August and am looking forward to taking a good two weeks off after that.

Hopefully by that time the project will complete itself automagicaly and I can return to doing things that I like.

Python saves the day!

2009-04-07T01:34:00.005+05:30

What was a rather dull and boring day turned into an interesting one thanks to Python!

I was basking in my holiday mood as I prepare to leave London after 1.5 years of highly satisfying development project. One of my team mates is doing a small project that involves taking a file, extracting the fields and creating a HTML report out of it. It was decided to use a transformation tool that we are using in another part of our project. This tool claims to be able to transform any input schema to any output schema. Our input schema is a raw flat file which looks like:

header
record
footer
header
record
record
...
footer

Pretty simple file - but we realized that the transformer tool is not able to handle this. I cannot discuss the specifics of the tool or the file, but it is a file that confirms to a widely used standard format. While we engaged the technical support to solve this, my manager started exploring alternatives. I looked at the file and said we could do it in Python using some basic regular expressions and string operations.

He let me give it a try, and here is what I did.

1. read the file and use regex to identify the kind of record represented by the line we read
2. extract the fields in each line using the simple string[x:y] operation
3. add these fields to a dict
4. create a list of each type of record and put the dict on that list
5. then using Cheetah, render a HTML page that can show this

Because I used the lists to hold the dicts, most of my operations in the template was looping over the list and printing a values from the dict. The data was part of the header records was put in global variables and passed to the template.

At the end of the day, in about 200 lines (thanks to me putting all that coding standards and OOPS into the code) we had a credible Plan B that worked. Also we could tweak this is a few minutes to work with another file that we needed to use!

So now in case we are left in the lurch by the technical support team, we have something that will work!

Issue with Ant Copy task

2009-03-22T23:25:00.005+05:30

This is something that I faced a few months ago. We had spent a few weeks developing build scripts for our deployment. We were entering the period where we prepared things for our final go live, and one of the suggestions was that we build a deployment script that could replace the current one. The existing one did not create a single tar file that could be extracted, it created two files - one with all the property files for all the services to be deployed and one with the jar files. Extracting these two one after the other was all that was needed, but it was not elegant.

So we asked a developer to merge these two scripts and thought that would be it. On the day we did the deployment to DEV, the services refused to start. Reason - corrupted files. We switched to plan B - used the old scripts and viola! the services start!

We duly fired a failure mail and decided to dissect things the next day. And after half a day when I figured it, I kicked myself.

Ant has a copy task that lets you perform token replacement. This basically lets you prepare files that can vary depending on environment etc and set the values when building the script. This is how it is used


< copy todir='../backup/dir'>
    < fileset dir='src_dir'/>
    < filterset>
      < filter token='TITLE' value='Foo Bar'/>
    < /filterset>
< /copy>

Apparently, this task does not differentiate between ASCII and binary files. It does the replacement on ANY file. Our original scripts were separate. So the jar files were copied in a copy task that did not do filtering. When we merged the scripts, we put all the files into a single task.

Once I moved the jar files and the property files into separate tasks, the scripts worked fine.

So it was a simple Ant task that was the issue!!

Street photography

2009-03-15T01:03:00.002+05:30

I was watching a programme called "The Genius of Photography"on BBC today. It gave me an insight into photography back in the old days. I loved all those black and white photos. It told me that I need to look at photography to capture life around me, how things are, how the world looks like. That's a good tip that I can use.

But there is a catch - I cant do street photography without getting into trouble. Just yesterday, I took my camera to work so that I could take a few shots around my office. I took my camera out in the evening and before I could take a picture, a security guard came to me and told me I could not take a picture without a pass. What? I was clicking pictures of a building and I got told off. What if I try to take a few pictures of people on the street? I am sure they will report me or take my film or my memory card.

The reason why people do that is not important to me. I just am not comfortable with the fact that I cannot just go out there and capture the scenes that I like. I guess I live in a different time now and I might need a license to own a camera one day.

Python Script: Converting strings to camel case

2009-03-13T21:19:00.004+05:30

A simple script to convert strings to camelCase


import sys

def processTokens(tokens):
    result='';
    for token in tokens:
        if token is not None:
            result=result+token.title()
    return result

def processString(string,separator=' '):
    li=string.split(separator)
    if li is not []:
        result=li[0].lower()
        result=result+processTokens(li[1:])
    return result

def getCamelCase(string,separator=' '):
    return processString(string,separator)


if __name__=="__main__":
    if sys.argv.__len__()<3:
        print 'usage: camelcase [input filename] [output filename]'
    else:        
        f=open(sys.argv[1],'r')
        o=open(sys.argv[2],'w')
        for line in f:
            o.write(getCamelCase(line))
        f.close()
        o.close()

Python Script: Extract strings matching a regex from a file

2009-03-10T16:41:00.003+05:30

A simple python script to extract strings matching a regex from a file.


import re
import sys

#initialize with some default pattern
mainpattern = re.compile('.*')


def matchLine(line):
    result=mainpattern.search(line)
    if result is not None:
        if result.group() is not None:
            print result.group()

def matchLines(inputfile):
    for line in inputfile:
        matchLine(line)

def processFile(inputfilename):
    inputfile=open(inputfilename,'r')
    matchLines(inputfile)

def main():
    processFile(sys.argv[1])

if __name__=="__main__":
    if sys.argv.__len__()<3:
        print 'usage: match [infile] [pattern]'
    else:
        mainpattern=re.compile(sys.argv[2])
        main()

Timeline of my life

2009-02-08T21:28:00.001+05:30

You can get your timeline at this site

1982	1983	1984	1985	1986	1987	1988	1989	1990	1991	1992	1993	1994	1995	1996	1997	1998	1999	2000	2001	2002	2003	2004	2005	2006	2007	2008	2009
EpochOn the 11th day of the 11th month, when the sun is in Scorpio and doctors are irritated and tired, a 5 kilo baby is born ( 1982 - 1982 )
The sleeping tigerThe boy grows up, learning thinsg fast and easy, apple of all eyes and troubling one and all ( 1982 - 1985 )
			The beginning of schoolingHe is admitted to school at the young age of 2.5 years, the telugu speaking principal has high hopes, but has a change of heart after he runs all over the school screaming and tells the principal rceipies for food that mom cooked at home. That was the beginning at Tiny Tots School Duliajan Assam ( 1985 - 1985 )
			Initial schoolingSettled down at Tiny Tots, studied well and moved at Kendriya Vidyalaya. Not sure or cannot recollect how much he studied but he did do quite afew things that are better not told at home, was not a bad boy buty was not innocent either ( 1985 - 1992 )
										The urge to excelStared getting interested in studies, painting etc etc and begins to have dreams ( 1992 - 1994 )
												The studious boyMoved to Hyderabad, learnt proper telugu, got class first, was even the leader of a class with 47 girls and 7 boys, and did everything a good boy would do. The evil streak died and he became a saint ( 1994 - 1998 )
																Pre universityRealized how much he doesnt know, and how scred he is - embarks on a journey to rectify things ( 1998 - 2000 )
												The computer bugWas intoduced to the C programming language by a cousin , is hooked immediately ( 1994 - 1994 )
																	The computer bug #2Is good at C now, decides to look towards Microsoft now, and starts using Visual Basic ( 1999 - 1999 )
																		The computer bug #3By the end of the milennium sets sights on Java, and gets the first computer he can play with without worries about spoiling it ( 2000 - 2000 )
																		EngineeringSwam through electronics and commnunications engineering, not sure whether he likes circuits or microprocessors, made friends and learnt many lessons of life, is officially labelled \"ITEM\" for being the odd one out in the class ( 2000 - 2004 )
																						The computer bug #4Has mastered Java, 8051 assembly and whole lot of things and thinks can conquer the world now ( 2004 - 2004 )
																						On the jobRealized there is a lot of things to be learnt on the job, is good at the job and is doing quite well, waiting to get married in August ( 2004 - 2009 )
																									Important Event #1In January decides to quit Infosys and join Oracle, but sticks back - not sure why ( 2007 - 2007 )
																									Important Event #2In Feb starts blogging and writing supercoder stories ( 2007 - 2007 )
																									Impotant Event #3In July asks the most important question ever asked: \"which building dear?\" ( 2007 - 2007 )
																										Important Event #4In August life changed completely when on Aug 15 he got engaged to his sweetheart ( 2008 - 2008 )
																											Mega Event!!Getting married on Aug 2 2009 ( 2009 - 2009 )
1982	1983	1984	1985	1986	1987	1988	1989	1990	1991	1992	1993	1994	1995	1996	1997	1998	1999	2000	2001	2002	2003	2004	2005	2006	2007	2008	2009

Extracting review comments with Python

2009-02-07T16:29:00.007+05:30

Often I need to exchange code over email with temmates who cannot access our source control repository due to network issues. In many cases, especially after a code review, I find that they have removed the review comments that I put in and I have no way to figure where I commented on what. This is also partly due to the not-so-good practice where I put the comments in the code using '//@@' as an indicator. To get over such cases, I wrote a simple Python script that reads all my Java source files and extracts these lines with '//@@' and puts them in a text file. This text file then serves as a rough guideline for me to review the code again.

Here is the script, hope it proves useful to someone else.


import re
import os

class Parser:
 def __init__(self,pattern,outfilename):
  self.pattern=re.compile(pattern)
  self.outfile=open(outfilename,'a')

 def walkDir(self,base_dir):
  for t in os.walk(base_dir):
   for f in t[2]:
    self.processFile("/".join((t[0], f)))
  self.outfile.close()

 def processFile(self,filename):
  self.outfile.write('Processiing file :'+filename+'\n')
  self.outfile.write('=============================\n')
  file=open(filename,'r')
  for line in file:
   if(self.pattern.match(line) is not None):
    self.outfile.write(line+'\n')


if __name__=="__main__":
 p = Parser('^.*//@@.*','testresult.txt')
 p.walkDir('./code_with_review_comments')

Let me know how useful you find it!

Perf stats with Python

2009-02-07T15:59:00.005+05:30

Last week we need to do a quick and dirty performance test for out Java application. The simplest way I figured was to instrument the logs with performance data for each critical step and for the overall end to end processing. These were simple log4j statements that used a set of predefined keys and logged - "****** key=value millisecs" - in the logs. We then put together a simple Python script to use regex and parse these log4j files, and spew out the statistics.

Instrumenting the Java logs is trivial, just decide the keys you want to use and put something like this:


long epoch = System.curTimeMillis();
// do your operation
logger.debug("******total_end_to_end="+(System.curTimeMIllis()-epoch));

Then to process the logs, use the below Python script


#import the regex package
import re

"""
        This class is a utility class to calculate the statistics of the
        data being recorded and store the data for reference.

        The stats reported are a simple average, min and max values
"""
class MinMaxAvg:
       
        def __init__(self):
                self.min=0
                self.max=0
                self.sum=0
                self.count=0
                self.data=[]

        def put(self,value):
                #if this is the first value, set it as both min and max
                if self.min==0 and self.max==0:
                        self.min=value
                        self.max=value
                #we have a new min value
                elif valueself.max:
                        self.max=value
                #sum the data so far
                self.sum=self.sum+value
                #and increment the count
                self.count=self.count+1
                #store for later use
                self.data.append(value)
        """
                returns the minimum value amongst all the values
                stored in this object
        """
        def getMin(self):
                return self.min
        """
                returns the maximum value amongst all the values
                stored in this object
        """               
        def getMax(self):
                return self.max
        """
                returns the number of values stored in this object
        """
        def getCount(self):
                return self.count

        """
                returns the simple average
        """
        def getAvg(self):
                return self.sum/self.count
        """
                returns the data used
        """
        def getData(self):
                return self.data

class ParseFile:
        """
                open the file, compile the egex pattern and
                set a objetc to track the stats
        """
        def __init__(self,filename,pattern):
                self.file=open(filename,'r')
                self.pattern=re.compile(pattern)
                self.minmaxavg=MinMaxAvg()

        """
                parse the file and send each line for further
                processing
        """
        def parse(self):
                for line in self.file:
                        self.parseLine(line)

        """
                apply regex and parse individual line
        """
        def parseLine(self,inputLine):
                #match the line to the regex
                line=self.pattern.match(inputLine)
                #if it is a match
                if line is not None:
                        #get the string representation
                        line=line.group()
                        #split at the first separator
                        line=line.split("******")
                        #split again to get the two bits that we need
                        data=line[1].split("=")
                        #get the category name
                        self.detail=data[0]
                        #and set the value to get the stats
                        self.minmaxavg.put(int(data[1]))

        """
                This method parses the file against the pattern and prints the
                stats to a file specified by the user
`       """
        def result(self,reportFile):
                #parse the input file
                self.parse()
                #open the output file
                f=open(reportFile,'a')
                #write the results
                f.write("========================================\n")
                f.write("results for "+str(self.detail)+"\n")
                f.write("minimum time " +str(self.minmaxavg.getMin())+"\n")
                f.write("maximum time " +str(self.minmaxavg.getMax())+"\n")
                f.write("Number of samples " +str(self.minmaxavg.getCount())+"\n")
                f.write("Average value " +str(self.minmaxavg.getAvg())+"\n")
                f.write("Data used "+ str(self.minmaxavg.getData())+"\n")
                f.write("========================================\n")

#if running stand-alone
if __name__=="__main__":           
        ParseFile('application.log','^.*total_processing_time_per_request=[0-9]*').result('report.txt')

Hope this is useful to someone!

More snow ....

2009-02-02T20:20:00.001+05:30

There is even more snow today!

Find a few pics .. this time from my DSLR :)

Canon 350 D, 70-300mm lens

http://www.flickr.com/photos/hari_259

Snow!!!!!!!!!!

2009-02-02T18:14:00.002+05:30

Its snowing here in London ... bigest snow in 18 years ... and for the first time in the 2 years that I am here ...I am having a blast .. unfortunately my DLSR was out of batteries ... so I am waiting for tomorrwo .. hopefuly to get my film developed :)

A few pics from our cell phones ..

Why I dont want to be perfect ...

2009-02-01T06:23:00.003+05:30

I am not perfect, and I don't want to be ... its a choice that I made ... because I am afraid I wont be able to keep up with peoples expectations ... afraid that I wont be able to run twice as fast just to retain my position ... Strange isn't it? I read it is not common or normal for a guy born in the sun sign scorpio saying that he is afraid and does not want to do something, but I guess scorpios are by nature hard to understand, so the person who said that might be wrong himself.

I am no loser or idiot or a case of sour grapes - I am a fairly successful software professional, one in the millions of software professionals in India, working for a dream top Indian company and in London on a project. I am good at what I do and I do that with dedication,100%, and have yet to come across a project that failed because of me. I just look for challenges in the job and am open to experiments. I dedicate almost 75% of my time to work or technology - things that keep my day job running like a well oiled machine. I can be seen so much with my laptop that my roomies feel sad for me that I am not doing much these days.

I have a life as well, I live and enjoy my work, I live and enjoy the comfort of the feeling that tomorrow morning when I go to office I am doing what I like, and I am good at it. Apart from this I like photographer and I play with my Canon Rebel (350D) and Canon EOS 300 cameras whenever I have time. I have about 3000+ pics , a pro flickr account and 4 hard bound books of my photos. I write stories, do things for my family and the special person who has made my life even more exciting.

Thats 100% of my life as on today.

I am doing well, but trouble is that I dont want to change those percentages.

When I picked up programming in high school it was like supercharging my brain. I did not have access to a computer but I always took a piece of paper and wrote down code on that, running it in my brain. By the time I left college I knew a lot of things that my friends only had an idea about or had just read the name in a courseware. So my IT job was a godsend - I could do what I like and at the same time help out at home with the responsibilities. But after 5 years I realized that there is an ocean out there, and I am not sure I will be able to swim it all and get to the other side. I do my job properly,but every time there is something that I havent done or heard of or learnt. So I put in an effort and learn it. When I started out this list was fairly short, today I dont know how long it is. Each manager that I work with adds to that list. And each time I reprioritize that list, someone just shuffles it for me.

Right now I am not perfect - not professionally,not in personal life. People have differing, often conflicting opinions about me. I am good, but on this road to perfection I do not want to get sucked completely into anything. I want to find that perfect balance where I am happy. Happiness is important for me.

Not many people ask me how happy I am - they ask my salary, my take home, my savings from my work in London, my promotion chances, why I am in the same company after 5 years and one question that dominates all these is - when will I get ready to move on to the next role? Is there any problem?

What is this mad rush for? Can't I just sit down, enjoy the moment, enjoy myself for a while? Why should there be a problem?

I am afraid that if I try to be perfect in any part of my life I might screw the other part - I cant drown myself in my job neither can I leave the job to go to hell and enjoy my life. I do not have that freedom or background. I need both , and I cannot be perfect if I want to balance both at the same time. I just like this middle ground where I go to do my job, make enough to live comfortably and save some for the future.

People call me mad for thinking this way. They point me to the people who are not as good as me, they tell me that if I dont do certain things they will overtake me and I will be left where I am currently. I need to run, I need to keep up. I need to do my job, learn new things, prepare for higher role, make sure I am ahead of that guy who I am supposed to be competeing with and do many many things - I dont have time for myself or to just sit and enjoy. They quote lines like "if you enjoy your work you will find time for everything", "learn to do things fast in short time", "busy man finds time for everything" etc etc.

No matter how fast pr perfect I might be in my work I am bound to get tired of this race. A guy with 10+ years under his belt can tell me to work smart, learn grow - but can he say that he never got sick of all this? I am sure he cannot.

That is why I dont want to be perfect - I am good at what I can do, and I want to draw the line, make my own rules, sit down and enjoy the moment - I want to be happy and content. To hell with the world!

A simple approach to setting up a maven repository

2009-01-31T20:54:00.005+05:30

At last, after I dont know how long I decided to try maven. I knew it is there and it is good,but I guess the final push for me to use this came from that fact that I wanted to manage my jar files better between all my personal projects. I had the trouble of syncing my Linux and Windows systems to the same version of jars while working on Console4Jython, so I decided to wisen up as I tried to finally kick start the other two projects.

Setting up maven locally was simple, and I am happy that now I can just depend on the maven central repository to get all the jars downloaded between my two boxes. I couldn't wait to migrate Console4Jython to Maven - but then I realized that there are jars that I want and are not available on the repository. One way would have been to search for the repositories for these and add them, but thankfully, I decided to set up my own repository and learn. Here is how it is:

Get a HTTP server somewhere on the network - I am using my web hosting on the internet
Create a folder called "myrepo" in the HTTP root of the server.
Inside this "myrepo" , create a folder called "m2-repo"
Check that you can access this from all the machines where you will work from, this might be necessary of your repo is on the internet and you might have to configure a proxy. I had to set up the web proxy

Now, I am assuming you have installed the jar to your local repository, if not you can do it by typing a command like

mvn install:install-file -Dfile=jyaml-1.3.jar -DgroupId=yaml -DartifactId=jyaml -Dversion=1.3 -DlocalRepositoryPath=C:\Users\hari\.m2\repository -Dpackaging=jar

Here I am trying to install the JYaml library. For further details you can refer to the maven documentation.

Once you have done this, you can go to your local repository and you will find a folder ~/.m2/repository/yaml. To set up the remote repository you will need to copy this structure to the m2-repo folder that you created previously. But before that you need to geneate an md5 for each file in this directory structure. I used the md5 for Windows tool from here.

Using this create the md5 files for all the files in the directory except the files names *.sha*.

Once that is done, copy the top level folder, in this case yaml, to the m2-repo folder. If you access the url http://myserver:port/myrepo/m2-repo, you should see the following structure

Now add this repository URL to your POM - it should be able to update the jar files.

Software bugs in my daily life - #1

2009-01-29T04:50:00.002+05:30

I am not a big fan of online shopping, but of late due to lack of any other option and partly due to my being in London, I am doing a bit of online shopping. And this is where I found a bug today, and this idea of documenting software bugs in my daily life.

Tomorrow is my mom's 50th birthday, and among other things that I did, the plan was to send her some flowers. So I went to the website of one of the well known florists in Hyderabad,India and logged in. Few minutes later, I had my choice ready and in the shopping cart. As I proceeded through the checkout, I was asked to enter a message on the enclosing card. I typed in a small message - and that is when the fun began.

The page now showed two next buttons. Normally I would expect "Add" or "Save" for the message, and a "Next" to proceed to the next page. Seeing two buttons labeled "Next", I was sort of surprised. I assumed, wrongly, that both mean the same thing - go to next page. So, wrongly again, I clicked the button at the bottom of the page. I was taken promptly to the next page. It looked OK, except that the message on the card as blank!

I looked around and found a back button. I click that, and am faced with a dialog saying page expired, resend data? I clicked OK and am taken to the same page from where I was trying to go back. The message obviously is still blank. But I notice that the delivery charge has doubled.Hmmm ... out of curiosity I hit back again and after hitting OK on the same dialog, I see the page and now the delivery charge is three times what it should be - the cost of the bouquet is still the same.Hmmm ... this time I hit F5 - and bingo the amount is now 4 times the original!

Being a developer myself I can imagine what kind of mistake the chap who coded it might have made. But nonetheless the page doesn't seem to have been tested properly.If it was not for my nitpicking habit if treble checking the costs on every purchase, I would not have caught this.

Thats not the end by the way. Because of the above error I decided to close the session and log in again. Thankfully they do not save the shopping cart between sessions. This time, they offered to load my pre-entered address. I was not really hoping anything here, but the form did get filled up incorrectly. Fields 1 to 4 were OK, 5 and 6 were blank and the 5 went into the 7 field. So fields 9 and 10 never got filled up and fields 5 to 10 are wrong.Hmmm ... by now I am hoping they dont reallys crew up something on my card details.

From then it was pretty much uneventful except for a minor issue with the mandatory form fields.

What appals me in this entire episode is the lack of quality. If it is a student who did this as a part time job, then poor chap is not being taught properly and needs to learn best practices;If it was a professional who was paid full time for this, then he is really bad and I would not trust anything in his hands. I am sure the florist does not have the resources or knowledge to test this stuff and must have trusted the guy who built it - and he did a not so neat job.

There will be guys doing mediocre jobs for as long as we have software, but we must atleast try to avoid mistakes, and if we need to really make a mistake, make a quality one - one that shouts, OK this could not have been foreseen!

Deja Vu

2009-01-03T03:45:00.002+05:30

For the last two weeks I have been working on a project that I left 2 years ago. Back then I was a bit player in a complex web of a project and never made any friends with the client team. I had issues with my manager who - lets be frank - was a complete screw up. I left after a few months when there was no need for me. And like any project that did not bring joy or learning, I blanked it from my brain.

But last two weeks were fun. I remember a few names and faces, but thats it. I was standing outside the office in the freezing London cold and a tall guy walks past me looking at me like he knows me. I start to try and recollect his name, but by the time it strikes me he is long gone. Later in the day I go up to meet the implementation team and this guy is sitting there. Even before I say anything he says - "I know this guy ... ". I smile and tell him I was around two years ago.

Again today I walk up to his colleague - I remember this guy, no way I can forget him, his knowledge is too good - and even before I tell him my name, he says "from Infosys ... " - I could only smile.

There are about 10 to 20 faces I know, pretty ladies I used to keep an eye out for etc etc and it is both funny in a way that I know so many things about the place. Its the first time in life that I am working a second time at the same place. I am enjoying it - and am hoping I have a happier ending this time around in this project.

Checks and validations

2009-01-03T03:17:00.002+05:30

The year 2000 bug was the thing that helped draw me towards programming. Thousands od programmers from India and all over the world worked to fix the issue and avoid a tragedy that thankfully never happened. When my cousin explained the issue to me, it seemed trivial, but nevertheless it made me realize how important having a suitable value or variable is.

On 31 Dec, I was busy most of the day working in the office - now new years bash for me, I was bashing a few issues in our first end to end connectivity test for the new application. The first end to end test is always painful. In the middle of all this bug bashing,a friend pointed me to a post about Microsoft Zune just dying - by thousands all over the world. As I learnt more details of the issue, it seemed stupid to me that such a stupid mistake as leap year check could happen. Well today when I saw the code here , I was laughing my ass off.

I see these bugs quite often in the work that I review and when I point it out, the other guy can just laugh. Once I had this guy in my team who built a portal for us. On his last day on the contract he wrote a fix and deployed it to UAT. He left and we never really bothered about the fix for a few weeks. Then one day we got abug saying that the hyperlink for the error detail was failing. It seemed silly - but when I looked deeper I found that it was failing only when the link text was alphanuneric - it would work when the link text was only numeric, whcih was 98% of the times. I called the guy and asked him - he was a pal, so the talk was fiendly. He laughed it off and told me which line to fix, I knew that already, but it seemed a sad state of affairs that something so silly and so obvious was left out - the guy could have paid some more attention and it would have saved us a bug.

In another case, we had a .NET application which did not have a dedicated support team - I was handling it alongside my other responsibilities. Most of the issues we ever got were due to lack of user training.So one evening when I get a call from the first line support saying they cannot export the data grid, I was inclined to push it back as a user error. But the pal on the other end was sure it was weird. So I go to look. I go there, open the grid and run a query that returned about 100 rows and export - OK. I run a query that returns 0 rows and export - I get a blank XML. Cool! Then he asked me to export 1,2 and 3 rows. One and three rows works fine, but 2 fails!!! It works for all n rows where n!=2! I still havent figured where it is going wrong!

Is it so hard to put some checks and validations in your code? Is it so hard to figure all the corner or conditional cases upfront? Its not hard, but it needs some effort - effort which might be easy but we are lazy to put in or effort that gets negelcted because we are running to meet an impossible deadline. Sometimes it is about another big issue that obershadows the main issue. Recently we did a major production rollout that impacted all users and broke compatibility. We had to lock out all users. The impact of failing to lock out all users and a user polluting the system due to a wrong version of the code was much much greater than a wrong fix, so I spent 3 hours making a fix, testing and releasing it to a few users in a controlled manner and one week planning the lock out strategy and fixning small leaks in our lock out code. The fix ran for a week with those few users without any issue - so we were happy. We had issues in the release to DEV but that was my Cruise Control scripts misbehaving. So three weeks and 2 pizza nights in the office we are there ! Except that the fix had a small validation error that screwed up three users. In the shadow of the lock out and with the aim of avoiding the lock out debacle, we screwed up the validation :(

Putting proper validations in code is not tough - if you make it a habit. Making it a habit is the pain.
Once you get into habit, its like riding a bike - you never forget.

I've been hacked and am pissed off ...

2008-12-25T02:58:00.004+05:30

I have this domain registered ... with about 50MB hosting ... I plan to use this to set up a family website ... because the domain name if my family name ... Initially I set up some static HTML ... OK .. good ... then I realized I could put some PHP ... so I go ahead .. look for a PHP CMS .. free one ... get in touch with the developer and tweak it a bit and deploy ...

that night ... some asshole hacked it and pointed to some porn ...

I was pissed ... I changed all passwords and went back to static pages ...

Yesterday ... I put one HTML page with some javascript using Jquery ... and today ... some stupid fuck hacks it again!!!!!!!!!

dont the sick fucks know not to bother with simple family pages!!! I am not writing my credit card details there!!

My experiments with setting up source control

2008-12-24T01:59:00.003+05:30

Today I had to set up Subversion(SVN) for my new project.The project is basically a small affair with two guys in London and two guys in Bangalore. The client has no interest in providing us a source control system, and we were maintaining the code through a web drive in my company.

Enter me - I was till last week in another place where we used SVN and Cruise Control on a daily basis. So when I saw an opportunity to teach a few things, I jumped in.

What we had was very basic - two desktop machines. No server, no backup and certainly no one willing to maintain my SVN.

I took the simplest approach

1. Download and install Apache HTTP server 2.2.x from here
2. Download and install/unzip the Apache bindings for subversion from here
3. Install Tortoise SVN from here

After this I had to do a few steps manually

1. Go to the folder where you unzipped the SVN binaries for Apache and look for mod_dav_svn.so and mod_authz_svn.so. Copy these to the modules folder under Apache
2. Copy the libdb*.dll and the intl3_svn.dll files from the bin folder where you unzipped the SVN Apache binaries and put them under the Apache bin folder
3. Add the bin folder under the SVN binaries folder to the system PATH

Now the next step is make some changes to the Apache httpd.conf file

Go to the LoadModule section and uncomment the lines corresponding to mod_dav_fs.so and mod_dav.so. Also add the following two lines at the end of the LoadModule section:

LoadModule dav_svn_module modules/mod_dav_svn.so
LoadModule authz_svn_module modules/mod_authz_svn.so

Now we need to tell Apache where our SVN repository is. Add the following to the end of your Apache httpd.conf file:



 DAV svn
 SVNListParentPath on
 SVNParentPath [Folder where all your repositories will be]
 #SVNIndexXSLT "/svnindex.xsl"
 AuthType Basic
 AuthName "Subversion repositories"
 AuthUserFile passwd
 #AuthzSVNAccessFile svnaccessfile
 Require valid-user

Just substitute the path under which you want to put all your SVN repositories.

The above XML tells Apache that you will authenticate with the SVN repository using apache user ID. So, go to the Apache home directory and run the following command to create a passwd file with a user ID and password:


bin\htpasswd -c passwd

If you already have a passwd file, then type


bin\htpasswd passwd

This will add a new user to the existing file.

With this we have set up everything on the Apache side. Go and restart the Apache server.If you open your browser and navigate to http://localhost/svn - it should ask you for a user id and password and should show an empty list of SVN repositories.

You need to set up a repository.

Go to the folder which you specified as the SVNParentPath. Create a folder called "Project" where Project is the name of the project for which you are setting up SVN. Now go into that folder and if you right click your mouse and go to the Tortoise SVN menu, you will see an option saying "Create respository here". Click that and a few files and folders appear in that folder. The repository is created.

You can create multiple folders under this parent path and create a repository in that folder.

Now if you browse to http://localhost/svn/Project , you can see the repository!

Of course you need to add some files to this.

You can do this through Tortoise SVN, or any other client or from Eclipse using Subclipse.

Leaking legacy code

2008-12-21T19:34:00.002+05:30

Recently I have been working on an application that has been around for years and has lots of legacy stuff to maintain compatibility. The fact that we maintained Java 1.3 compatibility for a long time and very recently moved to Java 1.4 is a good indication.

When we moved to Java 1.4, it was due to the fact that our new JMS implementation required atleast Java 1.4. We reworked on bits of code to get it to work with the new JMS infrastructure, but never really thought about the Java 1.3 to Java 1.4 migration issues. Java 1.4 was a big clean up release and we should have been wise. However, fortunately or unfortunatley we did not face any issues all through our testing. But when the users started using the code in different situations like JNI or App servers, shit hit the fan.

One of the recent issues is interesting.

Some one wanted to run the code in JBoss - we had never built it with that in mind, but did not see any issues. So we said - go ahead, feel free. They did it and in testing, the server crashed with an out of memory exception!

This was the first ever reported memory leak, so we were concerned. I ran the code in Tomcat and Resin, the memory grew by 10MB in 12 hours and most of it was strings. We were running in debug mode and we use lots of string operations all over the place - so we were not suprised.

But when we ran in JBoss we saw a large number of JMX objects - BasicMBeanRegistry in the heap. We saw that they kept increasing in number with time.

We have a JMX interface to our code and it was written ages ago in Java 1.3. We did not know - and still dont know - how it affected the JBoss server. But as a test we switched off the JMX and today I am told it is running for more than 72 hours with under 90MB of memory.

So something in the way our JMX code was written is not going well with JBoss.
I am hoping we will not have to rewrite the JMX bit to adhere with Java 1.5 or higher.

Any ideas?

Python

2008-12-19T05:38:00.002+05:30

Finally, I managed to convince my Technical Architect and good friend Sanjay that Python is a good language fit for web development - using Django and Google App Engine ...

I was in awe of Python for a while now and am playing with Django and the Google App Engine, but convincing Sanjay who was a Java fan was not easy. I am the developer who is easily attracted to new things, but he has the technical architects brain that dissects everything to pieces before he accepts it ... so .. Whew good win ..

Now he will make me sweat on a few projects that he is already thinking for us using Python and the App Engine

Marriage ...

2008-12-19T05:23:00.002+05:30

I will be getting married in August 2009 ... I am very much looking forward to it ...

Being an Indian means that the concept of an arranged marriage or the fact that girls get married by 25 and boys by 30 is not really alien to me. The concept of love marriages and the whole falling in love thing was in fact more alien for me till I was 18 years old. It was then that I realized that being in love is being utterly and totally comfortable in someones presence - either talking or just being silent. And I also realized that me being me, most people find it hard to adjust with me and expecting a girl that my parents choose to adjust with me will be tough.

I told my parents after I got a job that I want them to leave the decision of marriage to me. Coming from a 22 year old guy who just got his job, it did seem strange to them. And this year when I finally found the right person and told them, they just said OK. That was quick and easy.

And now I am getting married next year.

Over the last few years I have been working in London for my projects. I feel comfortable socializing here with my colleagues, and in my interactions I have come to realize and accept the fact that marriage is not the end goal that people are looking at when they get into a relationship. People can have 3 or 4 kids and still not be married - they are committed but not married.I am not shocked by this,but the reverse does not hold. I tell people I am getting married next year and the say "What?" - well they don't say it but the expression is the same. It feels kind of odd to them that at 26 I am getting married and looking forward to it.

I have just one answer to that - " thats the only way to some things in India!".

Agreed the generation has moved on and nowadays young people in India do get into relationships - physical as well - before marriage, but that is still a very small number. And for these guys and those who are more strict in their upbringing, the ultimate aim is still to settle down in a marriage. I cant just walk up to my folks and say - " I like this girl, I am moving in with her, and lets see how it goes" - I would like to but I wont, I cannot, it just is not "my way".

Ah well, some day in the future I might see my kids say this to me, but hey thats many many years away ... right now ... I spent a hell of an effort trying to convince my girl's parents, and am looking forward to August 2009!

Garbage In,Garbage Out

2008-12-18T04:24:00.002+05:30

"Garbage In, Garbage Out" or GIGO was one of the first computer acronyms that I learnt, and one that my cousin hammered into me. I should never expect the computer to identify the invalid data that I had entered and still give me a valid result. What this taught me was to check all the steps in my code and to always check for user stupidity as far as input data was concerned. I must admit I have been tempted to be complacent, I have succumbed and even paid the price, but nevertheless I have tried my best to adhere to this as much as I can.

Over the last one year, I have had the chance to work on a project involving a huge amount of coding. The amount of coding that we did is way above the usual coding done by my department. We are more into package tool implementation. So when projects like this come around, maniacs like me are in demand.

I realized during the course of last year that the quality of programmers in my company has fallen. And each time I read a story on the internet saying "if you want to eliminate the need for programmers, then outsource your project" or talking about the lack of quality in the code produced by developers in India/China and other outsourcing hubs, I am pained and the scenes that I see around me don't do anything to reduce the pain.

Garbage In, Garbage Out.

Back in the days when I started to write my first program, I was taught BASIC. C and Pascal were around, but I was taught BASIC so that I could appreciate the basic fact of programming - a set of instructions, nothing more.

When I did move to C and use #include and functions and recursion, I was merely applying structure to the basic set of instructions.Pointers, data structures etc were new and challenging topics and the fact that I crashed my code bu not freeing up memory was a learning step. So today when I write in Java and Python, I still remember how things work under the covers - Java and Python are written in C.

But the guys I work with nowadays are not aware of all this. Most of them were introduced to Java as a course in the semester, with a lab that taught that very basic programming tasks, nothing big or as a 5 day crash course in the company boot-camp. They know things,but never used them in anything big and most of all lack practice. Practice is what makes you good at programming, the more you practice, the more problems you face/analyze/solve the better you will be.

In the hurry to get up to speed on the latest buzz words, we are forgetting our basics. We don't teach the kids the best practices of programming, the way of analyzing a problem effectively and the optimum solution. We tell them the buzz words, a few samples,few lines of code using Eclipse or some such IDE and thats it - certified and project ready. In reality they cannot write 200 lines in one day.

No wonder when you approach a project with such resources, the results take ages to get to the desired quality. They finally get there because people realize things on the job,learn on the job and then deliver.

Is this the best way to go? Can we not improve the situation?

Did you stitch any code today?

2008-11-17T19:56:00.001+05:30

If I ask you - "Did you stitch any code today?" - will you be surprised?

You probably will be, but that's what we do everyday - if we write code for long enough we would have accumulated tried and tested methods which we use again and again in new programs that we write. We call it - "Reuse".

I call it stitching code - bits and pieces of used code strung together to make a bigger program.

There is no good reason for calling it so, it just came to me and my friends while discussing reuse one day.

With reuse, the problem is not only with identifying reusable bits of code, the problem is also that you want to store them somewhere, somewhere you can easily look at them and then put them together. It will be best if this "somewhere" was not tied to a single computer or an office LAN.

Everyone has their own means of managing their reusable bits of code. I used e-mail as a effective strategy - until I got to the current project where I had more than a hundred such pieces. I had close to 200 mails in my Gmail account and each time having to search them ( even using Google search) is not funny.

So what do I do??

Here is what I did. I built an application using Google App Engine. It authenticates you with your Google account, and lets you upload bits of code - any language. You can upload 100 such bits.
Obviously you name each snippet and attach a description. Because its on the Internet, you can use it just about anywhere you can access your email.

So if you are sitting in a cafe sipping coffee and using your net book to browse the Internet and your boss or friend calls you to ask for some code - or you just feel like doing something - you log in to this application, specify the names of all the bits that you want to stitch together and press a button!

You will see a program appear in front of your eyes using magic bits that you created not so long ago! And depending on where you are you can either execute it or just copy into a text file for running later.

This app is called "Stitch the code" and can be accessed at
http://stitchthecode.appspot.com

Do feel free to let me know what you think about it at anarahari@gmail.com !

Console4jython

2008-10-26T00:12:00.002+05:30

I have always been interested in starting a Open Source project on Sourceforge. I had once tried to join an existing project and be a contributor, but my work and other commitments do not let me follow a tight schedule for this kind of work.

A few months back when I was actively using Jython in my work,I realized that I could use the help of a small console that let me run commands from an interactive shell and let me save that history as a script. Another thing that troubled me was that after years of using Eclipse I was addicted to the intellisense.

So I set out to build such a console for myself - I did not bother to check if such a think already existed because I wanted to use this as a chance to build an open source tool of my own. So with a few days of effort I had a tool - Console4Jython - that I promptly put on Sourceforge.

Over the last few months whenever I have nothing to do I code a bit for this and have just released the 1.0.6beta version of it. It provides

1. A interactive shell in a SWING console
2. A command history
3. Code completion - nothing like Eclipse but it can show you all the methods on a Jython object
3. You can clear the screen
4. You can save the command history as a script
5. You can load a script and it will be executed
6. A bare bones editor from which you can run the script
7. You can run a script file by saying "exec [filename]"
8. You can specify a bootstrap script file
9. You can generate a wrapper script for any Java class with "parseclass [classname]"
10. You can generate the wrapper scripts for classes in a jar file with "parsejar [jarfile name] [package filter]"

Have a look at it, and if you like it - please let me know!

Work,deadlines,holidays and festivals

2008-10-09T01:26:00.002+05:30

Indian Software professionals work long hours.

This is a fact, and the exact reason for this varies. It is definitely not because we get paid for overtime or that we have to work long hours to make enough money with our low billing rates. Generally newcomers work late to either learn stuff or because they are still learning how to manage their time. Some people also work late because they have the "manager-from-a-different-world" who thinks that you are good only if you work late. We work on weekends and holidays also - again the reasons vary.

I had a long argument with my fiancée today about this - why do you have to work on a holiday. We both are the kind of people who take responsibility for our work and the team. We work 9 to 8 everyday. But we try to keep our weekends free.5 days, 9 to 8 do your best, enjoy the weekend. Working on holidays is OK if needed.

The reason we had the argument was because she is working tomorrow - on a festival holiday. Dussehra is a big festival in India and especially in our families it is a big celebration. The 10 days of Durga pooja culminates on Vijaya Dasimi which is tomorrow. And she has to work to run a regression test pack tomorrow so that they can show the results to the client sitting in Tokyo.

My question is - why should you work on a festival day? Cant you just tell the client that you will give the results a day late?? It definitely is not a end-of-the-world deadline tomorrow.

I myself am a team lead who has has to handle teams. We always maintained a rule that we will not work after a certain time and will definitely not work on certain days. Even when sitting at the clients office I have told the client that the deliverable will be delayed because it is a holiday in India and I do not expect people to come and work. None of my client side guys work for Christmas or Easter and they agreed to what I said.

It is not wrong to put you foot down and tell that you need to take a certain day off - we are humans and we depend on family and other factors as support and driving mechanisms to move ahead in life. It is by spending quality time at home and enjoying few days a year with the family for festivals and things like that, we keep ourselves sane and in a position to work. If we lose that then we lose our drive to work and ultimately we will fail.

Work and life might be synonymous in this competitive world, but we have to maintain a balance between the two. If it is not possible to keep this balance daily, we can at least make sure that we do not miss these holidays and take time off on these days.

Terracotta

2008-10-05T22:37:00.002+05:30

Last week I attended a talk by Terracotta founder and CTO Ari Zilka at the London Wicket User Group.
I had gone there because my technical architect and Sanjay had said Wicket was a good thing to learn about. I did get some idea about Wicket, but I was floored by Terracotta (www.terracotta.org).

I had heard about Terracotta from the guys on my project team and had visited the website. But I could not generate enough interest to sit and write some code. But after the talk, and especially after seeing the demo application, I had better idea about things. Today I finally sat down and wrote a few sample programs to try out Terracotta. It was fun and it was easy to do things from the comfort of Eclipse.

It was fun to able to take simple applications and cluster them with minimal effort. I wanted to try something bigger, so I promptly tried to cluster a Jython console - I ran into some errors. It says it cannot share the InteractiveConsole object for some reason - I have a big stack trace that I need to navigate to figure out exactly why.

But I can definitely see some applications for Terracotta in my work. I just need to convince my folks that it can be applied.

Vista (vs) Ubuntu (vs) Debian

2008-09-23T01:43:00.003+05:30

last night I was trying to think why I had only 80GB free on my 140GB disk running Vista? I tried cleaning up stuff I am not using including a 8GB Debian VM image and stuff like that. I even un-installed Office 2007 that I was not using .. still the space did not seem to go ... it grew to a mere 84GB ...

Then as I tried to remove another application, it prompted me for a restart and hello! the laptop wont boot up! After an hour of system recovery it started and behold! I can see 104GB free now ...

What the hell was Vista hogging 20GB for???

Anyways .. I have decided to install Linux as a dual boot on my laptop ... my decision was limited to Debian or Ubuntu ... I like Debain's stability from when I used it on my Linux only laptop ... but in this case I want to preserve the Vista partition .. so I will stick to Ubuntu which I can install with less risk of spoing things ... Last time I did Debain it was another story :)

Wish me all the best

Hello Delphi!

2008-09-23T01:35:00.002+05:30

Good projects are hard to come by ...
Projects that mentally stimulate you and gratify you are harder to come by ...
And both of these are less likely to happen in the first 4+ years of your job ...

They do happen .. but rarely ...

My current project is one such experience that I am not going to forget in this lifetime ...

Two weeks ago I had no clue what Delphi was .. except that it is related to Pascal .. which is again something I have no clue about ... I was a direct to C guy ...

Today I wrote my first Delphi console application and it works like a charm ... And I am sure in no other project would I be given such an opportunity to do something like this on my own .. learning on my own ...

Happy!

Installing MediaWiki - The easy and oversimplified approach

2008-09-14T23:14:00.004+05:30

This article is an attempt to provide a bare bones and sure to work approach to install MediaWiki and get it running and ready to use in a very very short time. This was the method that I found to be the best given that I was trying to set this up in under 2 hours.

So, here we go.

1. We will be using a Linux platform to get this job done.
2. If you have a Linux installation that you can use - perfect - I did this on Debian Etch, so I would suggest that.
3. If you do not have a Linux installation or Debian or any Debian derivatives, then you can do what I did.
4. Download VMPlayer from VMWare - this is a free virtual machine player. You can find it here [1]
5. Once you install this, please go ahead and get the Debian Etch 4.1 Virtual appliance from [2]
6. This is an 800MB plus download and will be the slowest part of our job - I downloaded this in 30 minutes on my broadband here in London
7. Unzip this to a disk with some 4GB space. Open VMPlayer, browse to this folder and open the virtual machine file. If it gives a lock warning select take lock and if it asks did you move or copy this machine, say you moved it.
8. Debian should boot in a VMPlayer window and you will see a login prompt. Login as "user" with password "user". You are root.
9. Select the desktop menu in the control panel at the top and select Administration->Synaptic Package Manager
10. In click on search on the Synaptic window and search for MediaWiki
11. Select MediaWiki 1.7 in the list shown. Click on the checkbox and say mark for installation. It will prompt you to tell that more dependencies are needed - click OK to install them also.
12. Now search for MySQL and select MySQL client and MySQL server and mark them for installation.
13. Click on apply on the Synaptic window
14. Let it install the stuff.
15. Once done, bring up a browser and go to http://localhost/mediawiki or the equivalent.
16. You should see a page saying Setup MediaWiki and a hyperlink to the set up.
17. Now fire up a terminal on the Debian box
18. type mysql and hit enter
19. You will be taken to the MySQL prompt.
20. Type this to create a DB called wikidb - "create database wikidb;"
21. Type this to create a user and give him necessary grants - "grant all on wikidb.* to wikiuser identified by wikiuser"
22. Type this to let the grant be effective immediately - "flush privileges;"
23. Commit changes by typing "commit;"
24. Return to the MediaWiki webpage and click on the hyperlink, it will show you a status saying you are ready to install MediaWiki and show you a form. Fill out the details and disable all e-mail related stuff. Enter the DB name as wikidb, user and password as wikiuser and leave the root id etc blank.
25. Hit install MediaWiki and wait for it to complete.

Thats it!

You are done!

I did this just now and am typing this article on the Wiki that I installed.

My Stories

2008-09-13T19:45:00.003+05:30

I have this habit of writing stories ... for a long time I had this habit of thinking up alternative endings and situations for films , but never really tried writing. But in 2007 after I started blogging on our company Intranet I came acros stories by Venu (see this) .. I loved the way he wrote complex stories in a simple and engaging way .. and I ventured into writing my own stories ... I have a collection of them hosted at whitenoise.sulekha.com

Please feel free to leave your comments ... and also watch that space for some more stories coming soon ...

A day at Lords

2008-09-13T19:42:00.004+05:30

Finaly I went to Lords .. the home of cricket .. and enjoyed the atmosphere there ... i just loved the way the audience enjoyed the game .. no shouting or slogans or anything ... but they enjoyed it and appreciated every player who made a contribution ...

I mainly concentrated on taking pictures ... I took a coupel of hundred pics ... but most of it was either repettitive or in some way not the best shot ..

I have uploaded some good ones here ... do have a look ...

Fourteen years with the computer ...

2008-09-11T03:12:00.001+05:30

For no good reason today I realized that it has been 14 years since I had my first encounter with a computer. At the first meeting, I was nervous and my computer stubborn. It was a 386 based PC with a 4MB extended RAM to run Windows 3.1 and had 170MB hard disk drive. My dad had installed Windows and wanted to show me few games. Unfortunately the mouse was not working and the "oh-so-obvious-these-days" shortcut of "Alt+F" was not known to me and my dad was still learning the ropes of Windows.

Well I did improve from that point. I had the privilage of booting all the 8 PC's in my school lab using the boot disk. That was because my lab incharge saw that I was both interested and comfortable. I used to do BASIC at school and once in a week or when my dad was free I got a chance to run a few commands on the PC at home. There was no Google to get code samples, and the GWBASIC reference we got with DOS was the only way to go. I still remember on the first day with GWBASIC, I wrote a program , saved it as something.BAS and executed it, and then tried to type a second program in the same file under the first program. My reasoning - why waste a file! My cousin who was teaching me was grinning and he thought I needed a lesson.

Slowly over the next two years he taught me C and basics of how the file system works, Viruses and anti viruses, how to care for a floppy and how to write a small game. Back then I had no access to anything other than the books in my hand and a pen and paper because most of the time my dad used the PC and I was not supposed to do anything stupid on the PC. I wrote lines and lines of code on paper and ran it in my head. C was tough to run in your head, but I did. Initailly I got everything backwards, but then once I got the hang of it it was cakewalk. By the time I was out of high school and laid my hands on a proper Windows 95 PC, I could get my program right the first time.This was 1998. C was good! but pointers had me for the next 3 years. Meanwhile I got a taste of Visual basic - back in its version 4.5 days and I was amazed by how much I could do easily. Although now after years of Java I dont think I will go to Visual Basic again.

Then I was away from computers for two years. In my first year of engineering, we had a C and data structures lab where I cracked pointers. Eureka! and I was the saviour of many souls who had to complete their lab work. I tried C and VB for two years. Then I did 8051 microcontroller programming for a semester on a small custom computer in the lab. That was fun! I wrote so much stuff that I could write machine code and finished my final lab exam in 30 mins flat out! Although this sounds good, it did not get ame any brownie points in my class - I was the geek who spoke an alien tongue. By the time I got to Java in final year, I was having fun. In out last semester we did a project where we coded a microprocessor in C and built a front end in VB.

Then came the Job, again I was the geek who spoke an alien tongue - C, Java and programming fundamentals were cakewalk. When they taught me Oracle , i just applied what I already knew about programs to SQL - its just a set of instructions, assignments, branchs and loops. Just dont get in a tangle.

I dont know if it is arrogance or ego or whatever, but I am never scared of a programming job adn always take it as something trivial. My first two projects in VB and Java, I did just that - took it as a simple job and did it well. I had the chance to try JUnit, Log4J and Python etc but never found a proper application for them in my projects. Then for two years I did projects that used packaged tools. These projects taught me a lot of other things about the job. So late last year when I got another Java project I was ready to go. Day 1 i realized that I am supposed to work with guys who have 20 yrs average experience in programming. It did hit me hard. I had to make effort to be recognized, I had to write good code and fast and in proper standards compliant manner to get then to even look at it. And each time I had just one shot - just like a sniper who can take just one shot at his kill.I have sobered down and mellowed in the last 10 months, but I have gained scars that will stay with me forever and guide me.

I grew up learning C and Java from playing around, not by Google. Today I do search for code on Google and other places, but still I do it my way. I wrote a small console for Jython few months back (http://console4jython.sourceforge.net), I wrote this in one night in a single class. Then I spent another 4 hours breaking it into more structured parts. I was happy, and I searched for any similar project on Google. I did find a few, and I felt I reinvented the wheel, but I was happy. Because I had a wheel that I liked. Maybe a few others will like this wheel, so I put it up on the net for people to use. The key here is the joy of creation. I felt the kind of thrill I felt when I wrote my first program. I also realized something. I was trying to learn Python for a while. I knew what and how to code, but was not getting that comfort feeling. I realized it was because I was not doing anything with my knowledge. I was not trying things. So I wrote a script to generate some XML we needed. I wrote something, cleaned it to standards, wrote some more and cleaned some more. In the end , I had something useful to be proud of.

What have I learnt in 14 years? Not much in the first 10 except how to get something right without Google to help you, but in the last four years on the job, yes a lot

Write some code - if you have an idea code it and see of it works or blows up. Doesnt matter fo you fry your PC, try first
Do this trying thing again and again even if you fail
Dont be afraid of code, It cannot come and bite you
Think, analyse and work out all possible scenarios
Dont believe in reuse - be prepared to rewrite stuff to suit your needs
Test
Test enough
Follow design patterns - dont follow sample code, follow the philosophy of the pattern or approach
Try new things
Try innovative approaches
Error handling
Failure and recovery mechanism
Performance, scalability, memory etc

Apart from this, it is just a few assignments, instructions, branches and loops.

So do I want to spend more time working with the computer?Yes, I dont see anysigns of me giving up as yet. The fun has just begun.

Building the terminator

2008-08-31T05:04:00.001+05:30

Before we go any further, please take a few minutes to read this blog from Google

http://googleresearch.blogspot.com/2006/06/extra-extra-read-all-about-it-nearly.html

We all are trying to build best of the class systems that take huge throughputs without complaining, do everything that we ask of them, give a unified view of the whole system using BPM and all the latest technologies and be superbly scalable by adding more resources and all that. We have he tools to make this possible and the resources to execute these implementations. In fact we have succeeded in achieving a lot.

But all this, is based on a program – a set of instructions, if-else blocks or loops that have been written by someone if not us, to provide us the tools to achieve our grand architecture. What if these tools are flawed? And we don’t realize it till it’s very late?

I was introduced to programming in my seventh class by my cousin who was a C expert. I learnt C at that age and tried a few things. I used to badger him with questions and get answers from him. One of the first BIG problems he told me was about was the Y2K problem – how using 2 digits or year had created a mess now. Countless number of programmers around the world did a lot of work to avoid a catastrophic event that could have happened as we stepped from 1999 to 2000. It was a great effort that succeeded. I used to read all that I could lay my hands on about this problem. Later as I went though engineering I realized how these small and seemingly insignificant things can bring down big systems. I blew up IC’s and motors in the lab by doing something that seemed perfect to me. I was corrected almost instantly when I saw the IC blow or the lab fuse burn out. I learnt. I watch discovery channel and see how sometimes big buildings come down because of a small flaw in a steel bar used on some floor. The thing here is – systems are prone to failure, and they will face that boundary condition that fails them sometime sooner or later, hopefully very very late maybe never.
The big question now is – did we belt and brace our system for that boundary condition?

In the IT world, we have moved past the Y2K bug that was the BIG thing and are moving towards the NEXT BIG things like Portals, SOA, Grid computing, million message per hour systems and all the high speed and high availability designs. We use tools and sometimes program our own tools. We do thousands of mappings and transformations or write thousands of lines of code. But do we belt and brace our programs? Or for that matter our XML or mappings or transformations?

In October 2007 I remember doing a release for our client, it was a CR and we did it at double speed. It was released over a weekend and we came on Monday to take over the monitoring from guys in the US. I was still sleepy and I noticed that something was wrong. Digging around, I found about a few hundred orders not processed in the system. Reason – some field that we mapped was null, and it gave an exception and so a few hundred orders got stalled. It took us an hour to wake the person responsible and get him to do the magic fix. It took me two hours to explain my lead why that silly mistake happened and he was not happy. I don’t blame him. And I did not try to explain my way out of the situation.

This was a small case, think of a messaging layer in a bank doing millions of messages in a day, what will happen if the variable used to hold the sequence number overflows? We could use long type, but then that too will overflow in a few years or if the message traffic really hits the fan.

We need to build air tight fail proof systems that will last longer than our lives. As engineers that is what we all have been taught to do. We can do that by doing simple things that safely secure the lowest components of the system – code level if we are coding or the mapping or XML or transformation level of we are using a tool.

This is just small list of possible things:

Implement the simplest possible solution to the problem: All of us, at some point of time are tempted to use all our knowledge to solve a problem. There was once a mail that I received. It showed about 10 ways to print “hello world” and it went from the obvious - print “hello world!” – to a cool solution using pointers. The simplest solution would have sufficed. Solving a problem does not necessarily have to satisfy the Geek in us, its fine as long as the problem is solved without getting complicated.

Documentation: Developers never comment their code - thats not correct, the good ones do comment. Neither do we comment our XML. But at the end of the day, this is what will save us. Imagine trying to understand and fix a piece of code with a gun to your head and the clock ticking away and you cant understand of the code does this or that. We should document every piece of solution that we implement and keep it as close the code as possible, not in some source control on the other side of the world. Document the code, the XML and the mapping (if you can). At the end of the day when there is a gun to your head, this is what will help you understand what you are doing or what you are supposed to do.

Defensive coding: See the below code,
if(a<5)
Print a

Let’s say some one decides that after printing “a”, you need to say hello, just to be courteous. You are not the person fixing it and this is what he does

if(a<5)
print a
print “hello!”

As you can see , its broken now. The correct code will be

If(a<5){
Print a
Print “hello!”
}
Although this may seem trivial, but this is what we need to avoid. Also we should validate all the parameters that are passed into any method – in fact don't use any input without validating it first. This list goes on, bottom line is – check stuff and play safe.

Error handling and Exception handling: I once met a person who was telling me that once in a while their application got stuck and would correct itself if restarted. It seemed strange because we did not see any exceptions, however on investigating the code we found that the problem was because an exception was being caught and not reported! Catching errors in a program or a work flow is one things, taking the correct remedial action or bubbling it back to the top level is another thing. It might not always be correct to ignore them or go on retrying for ever. Sometimes the wisest thing is to just throw it back up and stall any processing that might be going on.
Testing: We all test to check that the damn thing works. Do we test to see what happens if it comes across something unexpected? Do we test for all the invalid inputs? Do we test what happens when the connection or something fails and we need to retry? If we test all these, we are sure we have eliminated most of the bugs

Will to live: We all love Arnold in terminator, how he keeps going on and on and on and never fails. That's the will to live, systems we build should have the will to live. A connection gets closed unexpectedly, a disk gets unmounted, something else goes wrong, LAN cord gets pulled, no matter what the situation there will be a retry and recovery strategy. And if the system cannot recover, take a break, wait and then try again. Keep trying until it recovers a week later or maybe a month later.

This is just a small list of things that we can do and this needs to be supplemented by reviews, good practices and processes and by reuse of things we know work perfectly well.

At the end of the day its for us to decide whether we want to build a terminator style system that outlasts us and runs for years to come, or we build another Y2K problem that will blow sometime in the future.

Test driven development

2008-03-31T05:11:00.000+05:30

I saw a writing on one of my gym members t-shirt - "Bridges dont crash, why does software crash?"
The answer probably is because it isnt tested or because there was a untested boundary condition or maybe because
the world of software is more cruel than the elements of nature that the bridge needs to endure. I personally
would blame the systemic analmoly that is inherent in the programming of The Matrix. But I guess we can mitigate
this by testing, and testing enough.

"Test Driven Development" - this phrase scared the daylights out of me for quite some time. Like every new
developer I was more intersted in writing code and less worried about making sure it works. When I did start
testing my code it would be a couple of runs using an excel sheet or a piece of paper or nothing at all
depending on the level and necessity of documenting the results.

But then the only constant thing in the world hit me - change. I was forced into writing my own code and the
test cases for the code using Junit. It was not easy and it was painful and after a few attempts at resisting
things - I gave up. I did it initailly as a chore and despised it but soon I realized how good it can be.

Without getting preachy and too technical. I will try to explain the Test Driven Development paradigm in this
blog.

Lets take the example of a stored procedure - this is simple and is not language specific. Lets say there is a
huge database that needs to be mined for data and reports need to be generated based on the results. The data is
generic and the query will return different results for different input parameters. Each type of input is a
business scenario that can occur when the user is using the application. The user will access the stored
procedure from a simple tool. We will worry about the stored procedure only.

Each user input generates a report. Each input is a business use case. And the results being fetched successfully
mean that the business case is satisfied. The procedure executes a set of queries to return these results.

In Test Driven development, the test cases should be written before the code. Although this sounds not so
practical it is not very difficult. We are used to writing pseudo code. Soin this case we could write the
pseudo code for the stored procedure. A simple skeleton code that returns hard coded results for the different
inputs. Because we know what it is supposed to return to make the business case pass we can fix the output and
write a stored procedure to mock the final result that we expect. This is called 'Mocking' which is an important
aspect of Test Driven Development. We mock all systems that we cannot simulate by inputs or those that are not
ready for testing.

Then once we have a mock stored procedure we need to write a set of tests - one for each business case - that
sends the corresponding input and asserts (checks) that the correct output is received. The test should be a
valid test to reflect what we expect will be done in a normal usage scenario. These tests can be programs or
scripts that call the stored procedure.

Once we have a set of tests these will be run against the mock stored procedure that we have built. Because we
have rigged the procedure to return what we want - all test will pass. it would be surprising if this did not
happen.

Now the development of the procedure will start. We have a set of tests that can be run to verify that the
procedure performs what it needs to. While the procedure is being developed the database needs to be set up
to contain some data - data that we expect to be returned and data that we expect will not be returned by our
procedure. The best thing is to have real life data that has the necessary randomness and is not biased towards
the results that we expect.

The procedure will go though different stages in its development. It will move from very basic to the final
robust form. At each stage the test cases that we prepared should be run against the procedure. They should pass.
If they fail we know that the code is deviating from ecpected behavior.

The test cases are not god send and can be wrong. They can also change due to a change in the business case.
Along with the code the tests are also developed to make them more robust, increase code coverage by making sure
that they cover all possibilities and also be comprehnesive enough to ensure that all business requriements are
met.

The code and the tests go though their lifecycle and finally reach the ultimate D-day when the system can be said
to be completely ready for deployment. On that day the tests should pass 100% like they did on the first run with
the mock stored procedure.

Now we have a system that is deployed and a set of tests that verify what it needs to do. This will be deployed
and used over a period of time that might beless than or more than the time we spend on the project. Some day
someone who has no clue about what we did will need to fix the code. Depending on things like whether we
documented the code very well or documented it at all and a few other things he will do a kind of OK to a good
job of fixing it. How do we know he did not break it??? Simple - run the test pack on it and it should still pass.
Regression testing. This of course depends on whether the tests are still valid or not and whether they
themselves have been changed. But in the correct kind of scenario we have a way of proving that the system is
still functional.

Seems simple enough isn't it? There are many who argue that who will make sure that the tests are correct? Well
we as developers are supposed to do that. Is there a guarantee that this will result in bug free UAT or IT?? Yes
as long as the system is the only thing that is part of the architecture. If you have this system talking to
differnet other systems then there will be bugs in the integration that cannot be caught in this unit testing.

Quis custodiet ipsos custodes? is a Latin phrase from the
Roman poet Juvenal, variously translated as "Who watches the watchmen?",
"Who watches the watchers?", "Who will guard the guards?", "Who shall watch the watchers themselves?", or similar.

We can only make sure that we are approching the probability of it being perfect is 1. There are few things that
have a probablity of 1 like the sun rising or the night falling. We can only try to reach there and the day we
reach there we can truly say "We did it!".