The Deadly Silent Debt

I heard many times the concept of technical debt. I also realized it's terribly hard not to acquire it and it will keep increasing in your application all the time as you build it.

I will start with the first question...

What is technical debt?

Technical Debt is a detour from the right system to reduce development time.

It's called technical debt, because it behaves like a real monetary debt. You get a fully functional system without investing all the time and effort it needed to get it right. Just like a monetary debt, you need to return a capital plus interests. Depending on the amount of technical debt you acquire, the amount of interests will grow faster or slower. The sooner you pay your capital debt, the less you spend in interests. The longer it takes you to repay the technical debt, the interests will keep adding, making it at some point economically inviable to be repaid.

How do I acquire technical debt?

Well, as I said before, it's really easy to acquire it, temptation is always there to do the quick thing and keep going. I think there are four ways to increase your technical debt. A mix of intentionally, unintentionally, planned and unplanned.

Planned Intentional Technical Debt

The more formal definition of technical debt, implies it's consciously acquired to get something in the market faster to start making money sooner. That money that is made earlier on, will allow the company to survive or make the difference to repay the technical debt. Technical debt in this case, it's something planned and done intentionally. It's done as part of a business strategy which usually leads to a bigger profit. If it's planned correctly, and the risks are known it's not a bad thing.
Examples:

• We need to release the new version of the Iphone OS today, before Microsoft announces their Windows Phone 7 next week. We will add multitasking for Iphone 3GS now, and someday we will add it to the 3G too.
• We will release our next version of the application using the built-in user repository, and later on we will integrate it with CAS to support single sign on.

Planned Unintentional Technical Debt
Well, this is the the type of technical debt added into the system because of external constraints. It could be because you cannot get senior developers. It could be because another team in the organization needs something working in less than a week. It could be due to a corporate policy: We will only hire junior developers and train them.

We know we will be adding technical debt due to this constraints, so we plan for ways to reduce the capital we are borrowing. Also planning a way to repay the capital once the constraints are gone.


Examples:

• Junior members in the team might add technical debt because the lack of knowledge so we will do code reviews.
• Eventual tight deadlines might require the development team come up with a quick solution, and then refactor it to what we think is the right one.

Unplanned Intentional Technical Debt

In this case, I refer to the technical debt constantly added into the system. The unplanned intentional technical debt, is caused by constantly wrong desitions of management and the development team. This is the technical debt added when we are rushing things all the time. This is the type of debt added by the lack of a well defined development process or because it's not followed intentionally. This is the type of debt that rottens the guts of the systems and breaks the will of the ones who are trying to do the right system.
Examples:

• We will not be writing any test into the code because we are in a hurry this month. 
• Just add another if statement and move on, they want it today and we have a pile of work for 5 months waiting for us tomorrow.
• Submit the code now, I will do a quick check on the test server and then we promote it to production tomorrow.
• I need to leave early today, so I will submit my fix now, and someday I will fix it properly.
• I don't know how to use hibernate, so I will do it with a hardcoded SQL statement now, and hopefully someday somebody fixes it properly.

Unplanned Unintentional Technical Debt

This is very similar to the Planned Unintentional case. It's caused by the same constraints, but the big difference is that nobody plans or wants to spend time fixing them. This is due to resource constraints plus constant bad choices from the management team.
Examples:

• When we ask you to do the invoice module in two days, we thought you were doing it also right. 
• I cannot understand, you have 3 months of experience in java, and you never heard of the proxy pattern?. 
• I just added another IF there, there are already 30 of them in that method so I guess another one will do no harm.
• That java class with 10K lines of code started with just 5 lines a few years ago.

What can I do to reduce technical debt?

The first step would be to try to stop increasing the technical debt. I know, it's already a mess, all the developers complaint about it. Some of them would love to put the latest framework on it as it would be the silver bullet for your technical debt. And also, yes... you are right, each task will take you a lot longer now, as you need to fix countless problems for each thing you have to change. Yes, that's what you get by borrowing all the time and never paying it back. But, be sure that paying back some of it each time you have to do something, as little as it may seem, will make the difference

I can tell you how you can get more than 2 millions in technical debt. Borrowing one buck at a time. I can also tell you how you can repay it. Paying back one buck at a time. Of course, keep in mind it won't happen over night.

It's also very important to make technical debt as visible as possible. Having technical debt visible, will make the team to start taking it into account. Will help the team to stop increasing it.

Conclusions

I think planned debts, are not actually a problem in any company. Making technical debt part of your business strategy, when it's carefully planned, can be the difference of keeping your job in 2 months or not. Planning ways to mitigate technical debt your are not expecting and not doing intentionally is also a healthy way to do software development.

The unplanned debts are the problem. This is when technical debt is not considered and ignored by everybody. It becomes a deadly silent debt. It's deadly because the system will likely be considered to be rewritten even before it's finished, and it's silent because management doesn't want to hear about it. There is no worst deaf, than the one who doesn't want to hear. Without management knowing where you are, they will not plan ways to get you out of it.

Something is part of your behavior when you do it naturally and effortless. Something is part of your character, when it costs you more than you were willing to give for it. Managing and reducing technical debt must be engraved in management character, and not just part of it's behavior. That will be the only way for technical debt not kill your application.

Speed, it's all about speed!

In a previous post, I compared Agile methodologies with evolution in nature (Agile, Chaos and Evolution). There I compare the fitness of and organism with the amount of working software the team can deliver in an iteration. Increasing the amount of software delivered (I will call it velocity from now on) is equivalent to increasing life expectancy in nature.

This means velocity is critical for today's Software Development. If you want to last longer, you need to increase your velocity. The team needs to do everything they can to improve it.

Increasing the velocity, many times turns into a problem, rather than a solution. Instead of finding better and faster ways to do things, they just aim to cut what doesn't seem important, and you end up with thousands lines of code in production without any testing, documentation or even logic. That would be like trying to go faster with all your wheels flat. It might seem you go faster in the beginning, but you are putting in risk everybody on the car if the trip is long.

I will go over some concepts and show you some ways to improve your velocity without putting any life in risk. 

Closest Train Station

Imagine somebody asks you to take him to the train station. You can take him to any of the station of a particular train line. You will just look the closest one and take him there. You would never take him to the train station on the other side of the country! When you are doing software development, you are taking your customers to a train station. You need to find the closest one and go there. If you take them to the station on the other side of the country, you will waste a lot of work and time doing the train job. You need to take your customers to a point where they can keep going without you taking them all the way.

In this case, you are not affecting your speed, but you are traveling less distance, which in the end, achieves the same goal. You need to minimize the distance you have to take your customers. This is the equivalent of "the art of maximizing the amount of work not done" in the agile manifesto. The more things you don't do, the faster the customers get where they want.

Minimize Work in Progress

Work in progress are the tasks you are working at the same time. Any task you started working on and it's not yet used in production is considered work in progress. Having more WIP affects your velocity an quality. Another way to say the same to make it more generic is if "you have to still think or manage that story". Everybody on the team, will have to manage WIP, and waste time on it. If you do not manage it well, your quality and speed will decrease considerably. Also, it prevents people from being focused.

Once you finished working on a task, get it to the users as soon as possible, and stop thinking about it. This is in particular important, when you don't have automated regression tests and a good testing environment. Good testing policy could back up your work done 2 or 6 months later when the users get it.

WIP affects you not only by having to manage it, but also because it implies context switching. Context switching is a great source of bugs, and it reduces a lot your speed. You can easily verify this, with a simple and known exercise.

Prepare 4 pieces of paper and a stopwatch. With the first two pieces of paper write numbers in one and letters in the other one at the same time. Write a "1" in the first paper, and an "A" in the second all the way up to "10" and the letter "J". Measure the time it took you to complete it.
Then, with the other two pieces of paper, grab the first one and write numbers from 1 to 10, then the second and write letters from A to J, and again take the time it took you. You will clearly realize how context switching affects your speed and quality. As an example, it took me 29.82 secs to do the first part and only 16.23 sec the second part. It had also better quality, because I didn't have to rethink the letters every time I was writing them
You can imagine  context switching will be harder with complex development tasks. There will be more bugs because your context will not change completely and it will be harder to maintain it in your brain.

I think reducing WIP, increases your quality and velocity. I understand this it not easy, as there is always too many things to do, and they have to be done yesterday. But as I just showed you, doing all those things at the same time, will make you work more, think more and it will still take you longer. You will also do it with less quality.

Increase Quality to increase reusability and reduce rework
Lately, technology has been going really fast, allowing you to reuse almost any application you have and combine it with others to increase your flexibility and productivity. The only way to achieve this, is by building applications on top of other applications.

I could verify with many years of doing really large development projects, anything you didn't do a year ago, will take you two or three times the effort to correct it today. I will not also take you longer, but it will also have a great amount of bugs. All this because it was poorly requested, poorly developed, poorly tested, and poorly documented, so all the assumptions done a year or two ago to make it work will be lost.

Investing on doing the right thing, and doing it right in software development is the only way you will be able to build things on top of your application. If you don't do the right thing, you will spend a lot of time changing and fixing it, and probably throwing away a lot of what has already been done. This will not allow you to have the time to build something else on top of it.
If you don't do it right, you end with something complex, full of bugs, no documentation, no tests, no knowledge to allow you to build on top of it either. Both cases imply a lot of rework. Reducing rework translates in to doing less and reusing more. The only way to reduce rework is by making quality a pillar of your process.

Conclusions

Increasing your velocity is the only way to last longer on IT. Still, this doesn't mean doing it recklessly. Even an old lady can push the accelerator pedal to the bottom and go up to 150 miles, but I wouldn't like being on the car when it happens. You have to do like race drivers and practice all the time trying to do the perfect turn. Only trying to make it better you will be able to go faster. It will not happen on your first iteration, or the second. First try to get to a stable point, where you are doing the right thing right. Then practice will allow you to go faster and see better ways to do things. If you just stop doing important pieces to go faster, you only foolish yourself.

Value, what is value?

When you get to work with Agile methodologies, you start hearing a magic word all the time. The word "Value" in agile methodologies is what drives everything. What makes "Value" so magical? Lets see what is behind that word by starting with the  dictionary definition. There are many definitions there but I think this is the one that applies in this case:

"relative worth, utility, or importance" 

This is where everything gets hard with an agile methodology. The definition says it's a relative, but relative to what? Everybody will gets a difference sense of value, because everybody measures it with a different stick.

The business user measures value, in something that will allow him to work less and make the same or more money. A developer will see value on using the latest technology. A DBA will see more value on having a well defined DB schema. A Project Manager will see value following a process and become predictable. A QA guy will see value in a testable application. But, which of all those values is the one you should care about when working in software development? We got into a worst place, because everyone seems to have different objectives while they do their work.

Now, which of all the different values is the right one? Well, I think all of those at the same time, but each one with a relative worth. Now lets look at the first principle of the Agile Manifesto to get a better understanding of which value is important

"Our highest priority is to satisfy the customer
through early and continuous delivery
of valuable software."

This means the value the customer gets, it the most important one. So we need to build everything around that. Still, that is not something simple, and it takes a lot of effort to satisfy a customer. Who will put in all that effort to achieve this important goal? That's the whole team working together. But as we know, not all teams are able to overcome all the problems in the process of satisfying a customer. Something else should be added to this mix, and that is team commitment. Without the team working at full steam, it will be really hard to satisfy a customer and be profitable at the same time. There is also another principle in the agile manifesto related to this.

"Build projects around motivated individuals.
Give them the environment and support they need,
and trust them to get the job done."

Which is the trick for that? is there any recipe to build a committed team? That's something everybody struggles with, when they are building a team. I think the word "Value" comes into play again. To have motivated people in your team, they need to get value from the project.

How can you align each team member's value with the customer's value? I think that is what management should be focused on through the whole project. The best management is the one that achieves that. This is also not an easy task. Because management is not measured by value, it's measured by numbers. They are often so trapped into those numbers, that they forget what really matters.

"Value" can be many things, acts and even just feelings. I could be just being listened or listen. It can be a mountain touching the sky or bytes stored in a hard drive. The only way to find out what it is, it to talk, dig and look for it. This is probably the biggest challenge for a manager, find out where is value for everybody in the team and align that with the value for the customer. Once you get to that point, you will see a successful project all around.




References
http://www.merriam-webster.com/netdict/value
http://agilemanifesto.org/principles.html

The definitive Corporate IT Firefighter guide

A firefighter in real life, is a person dedicated to extinguish fires caused intentionally or unintentionally. The Corporate IT firefighter does the same. Here is a guide for all the IT firefighters around the globe.

The IT Firefighter theme
"All the effort is dedicated to the most important fire. Then we run to the next fire."

Here is the process IT firefighters follow everyday. We start with the most important thing a firefighter needs to know to be the best as his job. That's knowing :

How to prioritize a Fire?

Fires can have multiple sources on IT. The best way to prioritize fires is analyzing the source. The source of the fire is more important than the size of it, that's why the source is vital to determine the fire priority. A good practice is working with an organization chart. The organization chart is like the real life firefighter cloth, it prevents you from getting burnt too fast. The standard procedure is matching the name of the person that triggers the alarm with the names in the org chart. The higher the box matching that name, the higher the priority. All the fires you are not working on are kept in a backlog, which we cleverly named it the Burning backlog. We will focus then on the fire at the top of the backlog, while all the others wait in the burning backlog to be extinguished.

Once you get to be an experienced firefighter, you will develop improved methods to prioritize backlogs. I met a couple of IT firefighters that used the title of the person reporting a fire but unfortunately they are not working on IT anymore. This is because titles lately became meaningless in companies. For example the only tester in the company now is called Vice President of Quality Assurance. So as a general rule, don't pay attention to titles.

Also you might run into the case, where two fire alarms were triggered by two people at the same level. The best way to solve this case, is to look who has more boxes below him in the org chart. It's also highly important if your name appears in one of those boxes, because that could change the more boxes rule.

With this guidelines, you will be able to successfully prioritize fires in your corporation. 

Fire Alarm response procedure

Whenever there is a new alarm, you need to prioritize it. If the priority is higher than your current fire, you need to grab all your stuff and run to fight the new one. Remember our theme, we alway fight the most important fire. Wasting time in the other fires is not well seen in the IT Firefighting Union, and could get as bad as lose your annual bonus if there is one.

Once the new fire has been prioritized, you need to send a response to the person who set the alarm to calm him during this troubling time. Your response should be "We are working on it, and we will have a solution as soon as possible". That will make you look good, even when you are not even close to work with it or know what the problem is.

Make it a learning experience
Corporations usually have two policies. One is the "no training" policy, and the other is the "no training now" policy. Either way, you will not get any training for most of your tasks, if not all. So, what do you need to do to learn from experience?
When you are working on a fire, you need to be really dedicated, and not just look at the fire, but what caused it. So you will start learning what you should never do in your applications. Things like:

• Catching exceptions and never log any error.
• Building your own internal web framework based on struts.
• Your own XML parser or ORM tool.
• Your own IoC lightweight container.

All those might have a reasonable explanation when they decided to do them, but in the end, you learn they become a big source of fires. Once you see a big source of fire, look at it, understand it, and search somewhere what would be the best way to do it, so it doesn't cause more fires. That will be the closest you will get to training unfortunately. If you are lucky you will get a chance to properly fix them.

Live with it and climb the corporate ladder
Being on Corporate IT is not an easy job. Just like a real life firefighter. Nobody wants to pay for your work or care about your well being. But as soon as they get into a fire, they will call you right away and thank you gratefully once you extinguished it.

Once you become popular in the corporation, everybody will call you and talk about you. If you always follow these simple guide and keep up with the hard work, you future is assured and you will get high in the corporate ladder.

Seriously... if you are working as a firefighter in IT, don't sign up for the union. Become a planner, and work on plans. Planning is looking ahead, firefighting is looking backwards.

What am I doing here?

Have you ever been on a project where everything seems to be wrong? A project when even doing the right thing seems wrong? A project where nobody does one thing completely without problems? A project that is so complex and big that nobody is able to see how the pieces fit together? A project where as soon as there is a change in any piece everything else start failing? A project where nobody feels responsible for anything? A project where managers just want you to do something with one goal "make him look good in front of upper managers"? A project full of wishes but no requirements? A project you are not proud of being part of? A project everybody ask themselves... What am I doing here? Well, I have been in projects like this, and I would like to share my thoughts on it.

I would like to start with a story.

The Story

There was an ancient community living on a desert. They always saved a lot of water on the rainy season because they knew they had to survive with it when the dry season comes. They did that year after year for long time. They had to work really hard, and moderate their use of water to be able to survive.

A particular year, an elder of the community started to practice a ritual. A ritual dedicated to the god of rain. It consisted on stepping over a small rock, and pray to the god of rain very hard. If his prayer were good, then a big rain will start. The elder knew when there was about to be a rain, and he used that trick to convince people the god of rain existed. Therefore many people started to believe in the god of rain, and thinking he will be able to give them rain on the dry season.

That year, the community saved lot of water for the dry season, while the elder keep praying to the god of rain before each rain. Somehow that year they had rains over the dry season, so everybody was really happy and celebrated. They finally found a solution to their problem. They just have to pray over a small rock to the god of rain. The god of rain was going to give them rain whenever they need it.

The dry season didn't appear for many years. It was a great time for the community, because they finally got time to do other things, things they always wanted to do. After a couple of years, everybody forgot about the dry season or saving water.

A few years have passed, and they started to stop praying to the god of rain, because they had more important things to do. They were doing things like looking for the best outfit or coffee in town. Unfortunately, a dry season came back, and the whole community started to run out of water. They thought it was because they stop praying, so they started praying to the god of rain again. Another month passed, and they look at the elder for answers, what is happening? Why is not raining now? We are running out of water. The elder had the answer for them, "you are not praying hard enough, or you are praying over the wrong rock". Everybody run for find a rock to pray on it as hard as they could, but still they couldn't get any single drop of water from the sky.

A few months later, everybody was desperate for water. Everybody in the community blamed everybody else, because other don't pray as hard as they. The elder kept telling them, they were not praying right, so the god of rain got mad at them. Many people from the community left it in search for water somewhere else. The ones who stayed died before the next rainy season arrived.

What does the story has to do with my failing project?

You might be asking, "What does the story has to do with my failing project?"

Blame everybody else

The first thing that happens when project starts failing is everybody points their fingers to others. People blame others, because nobody followed the process as hard and good as they did. Nobody wants to be responsible for anything, because they know it will be used against them. Everybody always have a reasonably good excuse not to do things right. Nobody is able to backup what they did in the project. Bottom line, nobody and everybody at the same time is responsible for the project failure. As you go up in the hierarchy, more weight you have in the results of the project, so more responsible you should feel.

A process leading to nothing
Nobody ever asked themselves if there is actually a god of rain. They blamed how others followed a process dedicated to a god that does not exist. None was capable to realize there is no god of rain, and they had to start collecting water again to save them. Nobody wanted to challenge the elder just like nobody challenges managers. And if you do have the guts to do it, most of the time your manager will put you aside, because that is not the right attitude. They will also tell you it is because you are not praying hard enough. It will never be because they are asking you to go in the wrong direction, in a direction that only creates more problems. It seems like being wrong is the worst sin for a manager.

The Parasites
There will be plenty of villagers telling their peers, "I always knew this was going to happen". That was their excuse to do nothing, neither collecting water, nor praying over rocks. I'm sure you have this type of people in your failing project. They are parasites, they live from others effort, and they are no able to do anything on their own. Everything is always wrong, or others made it so complex, that they are not able to explode all their potential. They limit themselves to do the least possible, because they do not feel valued. They could also spend their time doing a side project for their aunts to organize weekend dressmaking workshops. The only way to get value from parasites is to push and control them all the time. You might need to evaluate how much time a manager has to waste to have a parasite doing something right.


The Followers
These are the guys who just follow orders. They do their work, and leave on time. They are committed to do what is asked, but they will not seek for any highness at work. They are there to do what they are paid for. With a good direction, followers will be enough to save your project. Still, as soon as the project gets into trouble, they quickly search for other places to work. Their commitment to the project increases as they get benefits from it. If they are not enough benefits for them to justify staying, they will leave.

The Lifesavers
If you are lucky, there will also be a few member unconditionally committed to the project. They will be overloaded, trying to get the water out of the sinking boat. The parasites will stick to these guys, because they know those can help them. Lifesavers will be able to fix whatever they can't. The lifesaver will assume the responsibility of the job because that's what they naturally do. It is easy to recognize a lifesaver, will be the one everybody asks questions, the one everyone runs for help. He might not be able to do things perfectly, but he is able to do things. Once the lifesaver passes his limits, he will leave searching for water somewhere else. If you are not able to create an environment to maintain them, you will only keep the parasites that are not even able to leave. Once you run out of lifesavers, the project will just sink faster.

Process Managers
There is an elder telling them which process to follow and they weren't doing it right. Same thing you get from your manager when the project is failing. They define or inherit a process to follow for the project, and they think by just following the process they will lead to results. They only focus on the process adherence, instead of looking to add value to the project.

They spent their time updating reports, and asking for time logs, status updates, and looking at to-do lists. They do not care what the lists are about, or why you had to spend twice as time as you should. They never review what has to be done, just review if the process was followed. They do not attempt to create a process to add value and reduce waste. They just aim to copy something they learn from internet or on a two day course. If there is no value in the project coming from managers, then for sure the project will fail.

The only way they know to save a project from failing is to push the team harder. Sometimes this works, because parasites need to be pushed to get them doing something. Most of the time, they only get something from the lifesavers, putting those on the verge to leave.

Value comes from people, not from the process. The process will only help to increase or decrease the value from people. If your team cannot create value, then the process is useless and might just waste the little value the team has to give. If you cannot get value from the team, then you need to stop, reconsider and start again.

Is leaving the only option?

First think what a project is. A project is created with an objective to generate value, with a time frame and limited resources. Projects are used in companies just to give work a frame or boundaries, to make it measurable and manageable.

Can a team provide products or services without doing it in a project? Yes, they can, and they are probably better without projects, because they can really focus on what is needed when it's needed.  They will be completely based on value driven work.


Some companies are project driven, some are value driven, some are wish driven and some are not driven at all. If a project does not have an objective to create value, a time frame, and resources well defined, then you should reconsider your project, and call it a wish.

Wishes are the main cause of failure, because a wish is hard to express and understand. A wish is not measurable or manageable. They are hidden as projects and the lack of measure gives management the sense of everybody is not working hard enough. Management sets an impossible end date to make the lazy team work hard. That generates the sense of failing to all the team. This causes every body to run and trash things to get to that impossible date making things worst.

Your management expects everything to be done by the end of the time frame without committing to the project at all stages. Your management only focuses on process adherence and not on how to get more value from the team. Your are working on a wish without realistic planning and measurable goals. Your company does not create an environment to keep the guys who can make a difference. You should start asking yourself  what am I doing there?

Offshore Development - Dream or nightmare?

I've been working doing offshore development for 5 years already. Through those years, I learned some really valuable lessons. In this post I will try to go over the benefits and problems it arises.
I will start with the first question you should ask yourself before considering offshoring.

Why do you want to do offshore development?
There could be many answers to this question. You could answer you are an NGO doing charity, giving people across the glove a good salary and a good life, because they are from a country with few opportunities. You could be an open source application and find a great committer in a remote location. You could say it's because you cannot find enough skilled people to do the work you need in your own country. But most of the cases it's just to cut costs. You want the same you get from a developer in your own country, but pay him a lot less. Even if it’s because you need to scale up and down as you need, offshoring is about cutting costs. People waiting until you want to scale up costs money too, same as scaling down when you don’t need them anymore. Yes, this does not sound idealist. It is not to make the world a better place. It's just to save money for wealthy rich people. Ok, I think I made the cost cutting idea clear. I'm not against cutting costs; actually I would love to be able to do the same with taxes. Now, this takes us to the second question you should ask yourself.

Do the savings justify all the hassle you get into by offshoring?
I would say it does. That's why many companies are offshoring right now, and more and more after the global crisis are trying to do it. Salaries of offshore developers range from 1/2 to 1/5 compared to developers in Europe or United States; this is depending on which country you offshore to. By just looking at initial numbers, it seems like it's the best deal of the world. Why wouldn't you save a lot of money on your overpaid IT salaries?  So far things had never been a dream with your local developers. That takes you to your third question.


What do I have to lose by offshoring?
The worst thing that could happen, it is to waste all the money you invest in the project. Normally, you will not get what you wanted. A very few times, you will get a decent application that meets your needs. You could also risk losing all your local people who knows about the application, the business and the project if you do no make them part of the bigger picture.  There is one thing for sure; many things will be more complex to do. If you are not able to overcome the increased complexity, you are more likely going to get something really different than what you ask for. Development will also take a lot more time than you initially thought it would.

Which things are more complex by offshoring?

The most complex thing is communication. Communication is not only about being able to talk over the phone in this case. It is being able to send a message, and that message being understood on the other side of the world. It’s not only about speaking the same language; it’s about being able to share the same language. If the team does not have a common language, everything else will be harder to share. By doing offshore development, you are already starting in disadvantage. The offshore team usually not only speaks a different language, they also haven't met personally the onsite members. If the team members are not able to leverage themselves by sharing their knowledge, then the distance between local and offshore will increase.

The time zone difference is also a factor that could complicate things. If you share most of your working hours with the offshore, you can consider them nearshore. If you shre little or none working hours with your offshore team, then you think about farshroe. Nearshore is simpler, and easier to manage compared when you do farshore.

If you want to do software development, you need to reduce as much as you can the differences between local and offshore teams. Reducing the differences will allow you to create a common language faster. First, think about one team, and not “them” and “us”. If you start it like this, you are already making differences. If you cannot trust your offshore team to do design, why would you trust them to implement your design? The more differences you are not reducing, the harder the communication gets. If you like lean software development, think about differences between local and remote teams as waste.

Culture is also a big difference to overcome. If you say “are you crazy?” some people could think they are doing everything wrong, when you actually wanted to tell them it is great. Some cultures also make people shy, because they are not able to freely express their feelings.  They reflect that on their work too, so you need to be twice the time more open to suggestions that you would normally do with people from your own country to get the desired effect. Again, building a common language is critical.

Which are the consequences of not having a common language?
The offshore team will go much slower and also going in wrong directions. Going in the wrong direction in software development makes the team work twice to get it back on track. This could mean your local team has to do this work because the offshore team will keep going in the wrong direction, or investing more time having your offshore doing it better. This could be the main cause offshoring your project could be more expensive than doing it right locally.

A lot of time will be wasted in communication to assure the offshore team understands what they need to do. This could mean endless meetings between local and offshore members discussing and trying to understand what is needed. If the team shares the same language communication time is reduced greatly and is more effective. Also endless meetings over the phone drive local and offshore teams crazy too. Nobody likes to be in a meeting over the phone for 6 hours, especially when it is in a different language. People get lost in the middle of the meeting. Details get more important than important things. Then you end up with your offshore team working on pointless details instead of things which add value to your business.

The offshore team will not be able to add value to the application. The offshore team only does improvements when they come from the local team as the direction to go. They do not know enough to improve things on their own, and communicate them effectively. Over time, any improvement attempt will stop, and they will only work on the requirements as they come. You need to get to the point where the offshore team is able to suggest and implement changes and fixes. If you are not at this point, you need to keep working on building a common language.

How do I build a common language?
The best way to build a common language, is to have your team interact. This does not mean talk over the phone, it means getting them to know each other. The best way to share things is by face to face contact. So the best thing is to get the team in the same place for some time and have them working together. This should be as mixed as possible, and aim to have everybody interacting with everybody as much as possible. Sometimes having the whole team traveling and staying onsite could be expensive. If you cannot afford it, then try to get onsite the offshore key players. They would be able to solve team differences faster when they are back at home. This also should not be once in a lifetime; bounds need to be reinforced periodically. This gives the offshore team a sense of belonging which is really important. If you don’t think you are part of something, you will hardly commit to it at a point where you make a difference.

Video conference is also very important, because it gives faces to the voices. It’s easier for a person to relate to another face than just to a voice on the phone or a signature on an email. It will help also to share the body language, which is sometimes as important as the words coming out from a mouth.

People on IT are known to be hard to deal with. They are not very eloquent, they do not have great acting skills; they are not best speakers. Why would you think it will be different in your case? If you do not invest on building a common language, the communication problem will be really hard to overcome. Initiatives will lose momentum right after they start making things harder to change the course of the project.

Is Offshore development a dream or a nightmare?
It will certainly not be a dream. It will allow you to reduce costs, making some projects possible. It will allow you to access really good development teams at an affordable price. If you are able to build a common language for the whole team, you will be able to reduce your costs considerably, and also be able to reduce the time things take to be developed. Building a common language is the most important thing when doing offshore, if you get to do it, you will get great results. If you omit this step in your offshore development, things will more likely turn into a nightmare.

As I said in the beginning, offshore is all about cutting costs but if you cut them too much, you will get the undesired effect. Frequent travels, reducing differences between local and offshore teams, video conferencing do make a difference when you build a common language. Not having a common language makes things take a lot longer, be more expensive and less likely to be what you expected.

Is it possible to do distributed Scrum or Agile software development?
I would not ask if it is possible, I think it is a must. As I said before, building a common language is all about the people interacting. Scrum and Agile, are in their roots about the same. Having daily short focused meetings allow people to build a common language. Visibility of project status and progress is clear, because you measure it with working software. Many practices from agile methodologies are of great help, such us continuous integration, testing, pair programming. These practices also help to build a common language, and reduce time spent trying to see which change broke everything since the last time the code was tested all together.

Waterfall is about interacting through documents. Documents are the worst way to build a common language. Documents hide differences, and they are often seen at the end, when you are already late to mitigate them. Maintaining documents is a tedious task and costs money too. You will also spend a lot of time on long meetings trying to explain what the document is about, instead of discussing what it's needed. Visibility of progress and project state becomes very fuzzy, I could send a document saying everything is about to be done, when it’s not close to it.

Conclusions
Doing offshore development is not only about the salary of your IT people. If you just look at that, you will only get the false perception of cost reduction. If you do not invest on building a common language, there are other things you will have to consider. Development will take longer than you initially thought. You will get a different product than you expected. Offshore development is not a dream or a nightmare. It is just like anything else, if you do it right, it will resemble a dream, if you do it wrong, it will be closer to a nightmare.

Agile, Chaos and Evolution

Let's get started with my simplified version of evolution. I'm not an expert on this, but Internet can turn any mortal with some time into anything.


Evolution in nature
Imagine you have a bunch of cells forming an organism. Each cell has a copy of the organism's DNA. The fitness measure how successful is an organism in its life, . The better the organism does during its life in the environment it is, the higher the fitness. A mutation is defined as cells having different DNA than the parent cells, due to a wrong copy of the DNA.

While the organism is in a stable environment, it will reach an equilibrium fitness value. When the organism is close to its equilibrium with the environment, the cells that mutate fast are suppressed, favoring the ones that mutate slowly.

A chaotic environment is defined as an environment that changes randomly and you can't predict it's future with certainty. It's also been seen that in a chaotic environment the organisms will favor the cells that mutate fast. Those will be picked by the organism in its effort to adapt to the environment. Once the organism starts adapting to the environment, fitness will stabilize, and the organism will go back to favor the less mutating cells.

Having too many mutations does not always mean the mutated cells will increase the organism fitness, it might take it to death too. Some mutations are good, and those should be the ones the organism needs to keep in order to survive to the chaotic environment. There is a cellular mechanism, which attempts to fix bad mutated DNA in order to prevent diseases and increase overall organism fitness.

Actually this is natural selection, the organisms who can adapt will survive more time, being able to reproduce more. While the ones with a lower fitness have lesser chances and time to reproduce, disappearing over time.

Some organism might found a perfect fitness for the current environment, conquering the world very fast. Later on there could be another environment change that could cause that particular organism, to quickly die, or other organism could mutate and feed from the previous organism. Perfect is enemy of good in this case. Perfect means everything is in order, everything works as it's supposed. If something is not in order or does not work, everything falls apart. While good searches for redundancy allowing the organism to survive if something does not work. It's better to be good at many things, than great at only one.

Evolution in software development
Now imagine your development team is the organism. The people in the team are the cells forming the organism. The organism DNA, which has been copied to each cell, is your development process. Each person has a specific copy of the DNA encoding. Management is the cellular mechanism in charge to control the cells do not mutate with the wrong copy of the DNA.


Predefined DNA
Some methodologies allow you to define the development process. This is similar to create a DNA for your organism. They let you know how the DNA should look like. You can make some changes, adapt it to your environment and use it to your organism. The person or team creating the DNA will attempt to create it perfectly. The DNA is in the organism before the cells that make it. If those cells cannot create an exact copy of the DNA, you need a strong cellular mechanism, to enforce the exact copy, or remove the bad cell of your organism, like a cancer.

Once you get into a chaotic environment, the cells will mutate at a faster rate, thus management will have more work to enforce the perfect copy of the DNA. This is because changing the DNA is expensive. It's expensive because it's complex and it was defined by a 3rd party. It takes more effort to change it than enforce it. The energy of the organism will be wasted on preventing mutations of all kind, reducing the ability of the organism to adapt to the changes in the environment.

Fitness is measured by achieving milestones in the schedule. In a chaotic environment those milestones have to change. You have to rework your project plan, your schedule, etc. to keep you milestones up to date. If you are not able to keep up, this causes wrong measurements of the fitness. Often the resulting fitness will only be seen at the end of the project, when it's probably too late to fix it.


Evolutionary DNA
Looking at Agile approaches I can see the DNA is not injected or defined by 3rd parties. It's grown from the cells in the organism. Each cell gives a piece of their DNA, to generate the organism DNA. The way to measure the fitness of the organism, is working production code at the end of the iteration. Once the iteration is done, the cells are able to improve the DNA causing mutations, exploring better alternatives to their particular environment. Once they get to their next iteration, they will be able to check if the mutation improved or reduced their overall fitness. The cellular mechanism should focus on preventing bad mutations, the ones that reduce overall fitness.

Agile methodologies only define a basic ceremony, which initially you need to follow, unless you are very confident of what you are doing. Once your team starts getting close to the equilibrium fitness, means everybody knows what they have to do, without have to introduce more changes in the DNA. This is the point where you will start favoring less mutating cells. This is the point where you want to get to, have a self organized team, which can react to the development needs.

Why Agile rules are so basic and simple? It is because they aim to set the most basic healthy DNA, from which you can evolve. Why it's so hard to implement them? This is because evolution is not easy. At first, all your cells will fight to make it to the organism DNA. You should start from the most basic healthy organism, and evolve to overcome the problems. Do not start from an advanced DNA, because it is complex to change, and you will end up with a powerful cellular mechanism to enforce it instead of focusing on evolving healthy.

Mutating the DNA and enforce it, is no longer a management task. Management can focus on avoiding bad mutations, which could cause diseases in the organism. This is not always easy, because there are a lot of cells in the universe which have a really twisted DNA. Starting from a wrong DNA, could turn your project into failure because you will need lots of iterations to get to a point where things start happening. If you do not have good cells in your team with healthy DNA, you need to invest precious time and money on training, and aim to maintain your good ones.


Healthy evolution
When you start from the basic organism, to evolve in the right path you need to focus on the basic agile principles:

-Focus on improving your fitness with continuous delivery of valuable working software for the customer.

-Focus on decoupled code to prepare your application for changes. Do not create your own internal framework to solve common problems because it increases the complexity when they are not common anymore.

-Focus on testing, to prevent new changes from breaking your application, reducing your fitness in the future.

-Focus on refactoring, change your code base to be simpler, better and avoid duplication. Learn and architect your application through this.

-Focus on having motivated and healthy cells, because that's what makes better the organism as a whole.

-Focus on interactions and knowledge share. If the cells do not know what to do, they will easily mutate in wrong DNA. This will also give you redundancy, allowing your organism to survive when it lost the head.

How Agile helps to evolution?
There are very important pieces in scrum that will allow and even force you to evolve. One of those is time-boxing. This is why you have time-boxed iterations. This is why all your meetings should be time-boxed. This is why you need to set a limit in the time you spend discussing issues. This is another factor that creates a needed amount chaos for the team to speed up evolution. If you couldn't get any production code at the end of your iteration, you need to evolve. If you couldn't get enough information from the users in the meeting to continue, you need to evolve. If you couldn't prioritize all the tasks you needed for the next iteration, you need to evolve. If all your team is locked waiting on the only person being able to do a task, you need to evolve.

You need to remove obstacles as fast as possible. This creates a better environment for the organism to evolve healthy. If your problems are not quickly overtaken, you team will mutate with wrong DNA in an attempt to continue with evolution, causing bad long term effect and diseases.

Have a small and well prioritized list of tasks to do next. Continuous big shifts in the direction of the project will accelerate mutations. This will create constant chaos, too many mutations and bad fitness measurements to ensure things are in the right path. You will end up with a bad DNA, and no compass to guide you in your evolution.

The team is what makes everything. If your cells are not healthy, they will make your organism sick. Fitness will be reduced, not because of wrong DNA, because the organism is sick. Aim to keep them happy. If you want your puppy to do all the tricks you want, you either need a big stick or bacon in your pocket. The big stick never works in IT, people leave their companies just because they don't like a framework the company is using, imagine if you scream at them. A good piece of bacon is what works. It does not mean overpay your employees, there are many ways to find out what would motivate them, spend time in this, it's important.

Why Agile does not solve my problems?
As I said before, Agile focused on the most basic healthy organism which you can start from and principles you have to follow to evolve healthy. It cannot impose a solution to all your problems, because it will not be able so solve others problems. If Agile would present a complex DNA for you to solve everything, you will not be able to change it and you would need again strong management to enforce it.

It's not paradise; you will get into many problems too. Where to go? What do I have to change to be better? You cannot keep changing everything in your DNA after each iteration. In the end, you will get nothing, even less than if you enforced any other more complex DNA.

If you solve problems you don't have, you are wasting energy from the organism in things that do not increase your fitness. Aim to solve the problems that reduce or prevent your fitness to be higher.

I said you need to evolve to overcome your problems. Sometimes the hardest thing is to realize what your problems are. If you don't know which your problems are, you need to spend more time reviewing the roots of those. If you are not able to deliver working code at the end of the iteration, aim to do the smallest thing possible, and grow from there. If testing of current code is impossible, aim to create code that can be tested, and decouple pieces as you can. Doing the simplest thing exposes your problems clearly. Doing complex things will easily hide them.

You know your problems, but they seem hard or impossible to solve. This case is simpler, try to waste as little as possible to overcome them. For this type of problems, seek help in Lean Software development principles.

Conclusion
Start from the most basic DNA and evolve to overcome problems. Do not start from complex solutions, or you will need strong management to enforce it, wasting energy. Aim to reduce waste from your organism. Aim to be healthy and do anything to increase your fitness; this is in the end, what will allow you to last longer.

sources:
http://www.obitko.com/tutorials/genetic-algorithms/index.php
http://www.talkorigins.org/faqs/evolution-definition.html
http://www.jstor.org/pss/3067634
http://evolution.berkeley.edu/evolibrary/home.php

Software Development is like buying a car, You never get what you wanted

I like the analogy between software development and buying a car. There is a car you would like to have, there is a car you can have, and there is the car you get. Unless you are rich, the "car you like" is never the car you "can pay". And most of the time the "car you get" is not the "car you can have". Why does it happens the same in software development? There is an application the users would like to get, there is an application you could potentially build, and there is an application you end with.

Imagine when you ask anybody, which type of car they like, without any budget restriction. They will be describing the car they would like to have, because they do not know the cost of it, they don't have to think about anything else other than the features they want, "red, wicked fast, leather seats, chick magnet, starts with an F and ends with errari".

You realize that is impossible, so you make some suggestions on what is feasible with the resources you have. Then you show them what they could have, "it's blue, not so fast, does not scare chicks, starts with a T and ends with oyota".

Then you start looking deeper about the car and what you can get, things that are not needed anymore, the other things are needed like a big cargo area, it should also be very light, doors are not critical, the budget gets smaller and smaller as you spend money and time going over the bigger cargo area and low weight constraints, so you end up with a Citroën Méhari.


Once you appear with the car at the front door, they will complain about it, it's not as fast as they expected, it doesn't heat the seats, it doesn't have air conditioner, etc. And even when you actually do not like the car, because it's not what you expected either, you have to start explaining them why. "Well the budget was low and we had to get the car very fast, and we spend a lot of time and money going over the changes you wanted, like the cargo area and the weight constraints, so this is what we could come up with."


Not only you have to deal with that, you also have to deal with the mechanics behind the car. Your good mechanic will complain, because it's not fuel injection, which is the last trend in cars, and the guru mechanic who created the initial piece of the warp core is gone for a better paycheck to a company building "The Enterprise". Your internship mechanic was not be able to understand the warp core and broke it, so he replaced it with a 2 cylinder engine.

Have you seen this happening in your projects?
The business team just expects results, without committing to the project because they are too busy making money. They don't even care how you do things as long as they are done, and complain to the CIO when anything is not what they expected.
Managers get lost with endless list of development tasks, and spent most of their time updating project plans after every meeting with the business. They also create tasks for everything the business team ask for and put them with the highest priority.
The development team is creating the warp core because it's a technological challenge but not working on the reports because those are just SQL queries with a nice format. The code is crappy because the development team had to stop in the middle of the warp core and mix it with some solar panels because somebody said solar power is the way to go, and then changed to a 2 cylinder engine to make it at least work.


The craziness I just described is day to day job for many people. This craziness creates the perfect alibi for everybody not committed to the project remain in the company, just doing the least they can like parasites. This craziness prevents the good people to be able to change things for better, because it takes them 4 times the effort of what it should be, lose hope and eventually leave. This also makes management not to trust employees, because the parasites are silent all the time without any idea, and the ones committed and with ideas have to work extra time to try to make them real, and sometimes it just turn impossible. A general sensation of failure is created making things harder to change.

Many companies would like to have a Ferrari at the price of a Méhari. Those do not like to pay the price of maintaining a Ferrari either. It's not the same insurance fee, not the same mechanics, and it's not the same oil consumption. Some other companies are rich, and can afford the price of a Ferrari, even when they spend 5 times the actual price of it.

So, what do you need to get the application you want?

• Good people in your company trying to do the right thing. 
• Commitment to higher quality. 
• Good management pointing the right direction as a company with common goals for business and IT. 
• Good communication between and within teams.

If you lack any of them, you will never get the car you want, or the application you want. You will only get some pieces of it and most of the time a lot less than you expected. You will always get a Méhari.

Is Agile only for fearless companies?

Why is it so hard to implement an Agile methodology in some places when it looks so simple?. I think this has to do with fear, and how fear affects people. Let me show what happens many times with this dialog. It's between an Experienced Project Manager (PM), and a Agile Mentor (AM) in a big company:

- PM: When will we be done? I told them we would have things done by last friday, and they want to kill me already!
- AM: This is the first question managers ask when they turn to agile. There are many ways to know that, you can use your old method if it ever worked out for you, but it doesn't seem to work this time. The best thing to do, is to be real about it.
- PM: What do you mean with "be real"?
- AM: it means, you cannot break the sound barrier in a chariot, no matter how many horses you want to use or have a baby in 1 month with 9 women.
- PM: That is far from my reality, so... what do you actually mean?
- AM: That means, you can only guess, and the less you know the worst the guess will be.
- PM: If I only guess, then Agile is not being of much help! Give me back my money!
- AM: Sorry, no refunds... but the answer you are looking for is, this is an empirical process, meaning, you learn from observing.
- PM: I can observe that you are doing nothing for me so far, so tell me when I will be done?
- AM: Easy answer, if your team velocity is 10, and you have your users stories in your backlog estimated by the team for a total of 200 story points, it will take you about 20 weeks.
- PM: my stories are not estimated by the team yet, so that doesn't work for me.
- AM: see, agile is already paying for itself, that means you need to at least estimate your user stories to know when you will be done.
- PM: Ok, imagine I get the user stories right, and the team estimates them in the story time meetings, and it works as you said, that means we will be done in 20 weeks, not a day more!
- AM: No... you still don't get the essence of it, it will be highly impossible to have it by that exact date, since that was a guess when you knew little.
- PM: WHAT! Then... you owe me money boy!, this is not what the business want to hear! They want results!
- AM: You just say it, results, not lies!
- PM: Ok, you got me on that one, then, What do I do?
- AM: Ok, after each iteration, you will show them the progress of the team, we will make this very clear. Also you will know the updated team velocity, allowing you to recalculate the remaining work, and give them an updated competition time. Which will be more accurate than last time.
- PM: But... hmmm...then... using an agile methodology means things will take longer than expected, we are not able to do more with less as it promises.
- AM: Well, there is truth in those words, but... that is because you expected way too much, so... using an agile methodology, makes that very clear for you right after the first iteration, and will allow you to adjust the curse of the project to avoid bigger problems.
- PM: I'm starting to get the picture I think... but...then in a week or two, I will have to go to the business and tell them we will not be able to make it on time.
- AM: Well, I think that's great, it's better to fail earlier than later.
- PM: Nobody likes to fail!, why do you say that??
- AM: Because now it's a small failure, but will allow you to discuss with the business, show them the problem, and adjust what is needed to prevent a bigger failure later when there is no turning back.Now we have a methodology which will help you and them to make better choices, because you know things will go wrong, way before they get worst, so you only lost a little.
- PM: I'm starting to like that, but... still, I'm scared to tell them we will not make it on time, we need to change things. What should I do?
- AM: You need to tell them, they should know right away. The faster they know, the better the solution will be.
- PM: But, they will tell me we will not make more with less being agile, what should I answer to that?
- AM: Only with good management you get better results, because better ideas are the ones that lead to do more with the same or less. You now count with a great tool to know your problems faster, to make things better, avoid failure and lead the change the company needs to shine over the rest. You are not better because you are good hiding problems, you are better when you solve them with better ideas.

Fears

People doing software development know they need to progress. Many times, their way to show it is just committing to an impossible date. They will start pushing everybody to get to it, and many times sacrificing things that are important for the business. So using an agile methodology, they will realize they cannot do what they promised right after the first iteration, and they think they are failing. The first reaction is the same in many cases, hide the problem and try to do anything to show things are working by the end date, even if it's missing critical parts for everything to work or making the development team work tons of extra hours. This is fear, the fear to fail cause people to do crazy things instead of using their heads.

Other times, dates are set by the business, and instead of creating a realistic plan and focus on what is really needed, and leave nice to have for later, people just push to get everything done. Everybody lose focus of what are the important things, lot of time is spent on things which could be never used. You might end with a really nice splash screen, but it doesn't process any data because you couldn't finish the process part. It's very hard to say no we cannot do everything you want by the end date. You need to talk to the users and find out what will give them the most out of the application, and focus on those. This requires the development team to work closely to come up with a good idea on what has to be built or changed to solve the problem. Without good communication allowing the team to know what is needed and have them committed, it will be very hard to solve the problem. Fear of saying "no we can't", gets you in muddy place which will be hard to come out, and harder as more time goes by and you remain silent.Event if you are doing an agile methodology, if you don't speak up, nothing changes, you are in the same place as before.


Fear to Testing and Test Driven Development. Many people claim to be test infected once they started doing it. Unfortunately not everybody has been in touch with Testing and TDD to get infected. So telling a guy who has been doing development for years and years you have to do TDD and write Unit Tests for all the new code, will cause as much fear as telling a guy who does TDD not to write any test for his code. Specially if your are always on the run on your projects dealing with a long list of issues and tight deadlines.

Also, when you are on your introspective meeting, many people is afraid to talk about problems they see. e.g: Rob never run his tests before committing his code to the repository breaking CI build. Or the specs and details we get from the initial discussions with Peter have nothing to do with all things we had to do during the iteration making it take more time just because he didn't think of this before. We are not getting enough training.  There are always things to improve and fear giving feedback, won't let you get better. You should encourage people to give feedback to their peers, and at the same time not making it personal.

So changing the way people is used to do things causes many fears, it takes courage and training to overcome them. Why it takes courage and training? you might ask. I think it takes courage to put your self in a path to the unknown, and it takes training, to make things known.

Many big companies lack of both, they don't give any training and they do not have people with the courage to deal with many others frighten to change. I think those places could be really hard to implement an Agile methodology. I think if your are in those cases, you should aim to build a special team, probably with people from outside with experience in Agile projects, and start adding the people that has already been in the company, specially the referents in different projects. Once you took fear out of them and they are comfortable with agile methodology, TDD, testing, CI, etc., you can start building more teams.

I've seen the slow and gradual change of existing teams and processes is too hard and almost impossible if team members are not convinced of the change. They always end in a crazy intermediate approach. They claim they are doing "agile" but they just ignore many agile principles. Those intermediate approaches should be based on the agile principles to consider agile. Also, if there is no powerful sponsor, the team will be trying things and taking the risk of failing on their own. Fear will beat them as soon as they start running into issues as they get into too many unknowns, and they will just fall back to the same way of working as before.

I would ask, why are people afraid to change would use an agile methodology? I think if you are not changing for better, you are changing for worst, and specially on IT, where everything else changes so fast.