Myth: Agile ensures Project Success

I've been working on IT for a long time, and used many methodologies during that time, a little bit of everything. Many times the projects went fine, many others they were a failure, and others they just became an endless list of bugs to solve.

So I always wondered, what went good? What went bad? And what caused so many problems? I realized that most of the problems, were not due to the methodology, it was due to people not doing they were supposed to do, or doing it poorly.
When I say people, it's not just the development team, it's everybody needed to make things work, and so this includes everybody in the company, and in some projects, people outside the company too. 
When I say people not doing what they were supposed, I don't mean they are lazy or just not skilled enough (even this could be for some of those), it's just that they didn't know enough at the time they had to do their jobs to assure everything will go fine when somebody else pick it from where it was left.

I started to think on many of the agile concepts, and why they will help you to be better at your projects. At a glance, this is what I see most people think of Agile.

1. I can just stop doing what I don't like to do, because I think it does not add any value.
2. I can just start coding, because it doesn't say I have to do analysis or design, like it will magically work out.
3. Only highly skilled people can do agile, because not skilled team members will not know how to do something that works and it's ready for production in 1, 2 or 3 weeks depending on your iteration.
4. I need these last minute issues solved in the iteration finishing tomorrow, and if we need to, we can increase the iteration length to fit this change because the business says it's very important. And repeat this over and over again.

I think all those thoughts can cause many things to fail, and they just show a lack understanding on how things should be performed to produce an application that will solve people's needs. Knowing how to build applications, doesn't mean technically be able to do it. Most of the time, agile projects fail, because the people involved do not see how everything must work together to make things right, and just skip many important things when they do software development.

What do I get when I look at the basic agile concepts? There is always more than meet the eyes, as most of those are written by people who actually do good systems, and they know how to do it, so they often take for granted many things normal people trying to implement agile should do already. Here is how I see some of these concepts are linked together and why they are important. I will not exhaustive cover all the aspects of Agile, but the ones I currently find more important. I will start with Production ready software.

Production ready software
For example, one of the key concepts is to "produce production ready working software at the end on the iteration". On many companies, working software just means move the code to production to start making it work and pay for itself.
Now... this does not mean just code something and move it to production without any thinking. You need to review the problems you would have if that code fails, or has bugs, or makes the company pay more or lose a lot of money.


Now, can you live with bugs in your application? I think you can, as I haven't seen any application without a bug in it. But there are bugs, and BUGS!!!!, a mistake in a calculation causing the company to lose a lot of money is a BUG!!!, and then you might have an input for a number in your screen, which does not display a proper error message when you enter a letter, and that is considered a bug which you can actually ignore many times without impacting the business.

This doesn't mean you can ignore everything, if your app, is used by millions of people, and that wrong error message cause a million calls to your call center every day, it will not be a bug any more, it can become a BUG!, so it's just a matter on how important things are for the business you are working for. This is not an excuse to do things wrong, the better things are done in the beginning, the better the app will be to maintain, fix, and generate trust.

Testing
Having said that you can live with bugs in your app, but you cannot live with BUGS!!! In your app, what will allow you to be sure you will not have BUGS!?
The answer for that is testing. You need to test your code, and test the things that are important for the business as much as you can.

Now, this does not mean I need to test everything in my app, even when it is good because you will have less bugs. Still you might invest too much money and time testing things which are not important for the business, and that might cause your system to fail too, as you never deliver new functionality, just more tests.

In everything there is a trade off, and you need to constantly revise that trade off. Lean software development guidelines will tell you to test enough. How much is enough? This is an answer that comes from the business you are in. You need to be sure no BUGS are in your system.

Testing is good, but not just because you will be more certain your code will work in production, many times you hear the expression that testing is your safety net. This means your will be more certain your changes will not break anything important in your application, so it will allow you to make changes without thinking of your application as a whole, with all the interactions it might have with other pieces of your code, you can focus on solving the problem you need to solve, and have more time to think on better ways to solve it.

Test Driven Development
Regarding testing, many times you hear developers saying that testing will take them too much time, or, whenever they have to change some lines of code, they need to change all the tests for those lines too, causing them to work twice.

I think this is what happens most of the time when you have tests in your app, but... why?
This is because you are writing test for your code, and not test for your requirements. Many would say... that code comes from requirements, so if I test the code they will test the requirements, but this is false unfortunately. This is pure logic... requirements => Code <=testing, so you are testing your code and not your requirement, and this is because the way you produce your code.

When you start coding, you have a lot of technical baggage in your head, with code you already written, and many times you trend to apply the same solutions you applied to similar problems before, or if you are lucky, some more experienced guy will tell you a better solution. With this, you might be writing tests to factories, or you need to build a hash table with many things in it to pass it to your code to be able to execute it. So, you end up coupling your test to the actual implementation of the requirement, and not the test to the requirement itself, so even when a requirement doesn't change, you might end up changing many test just because your implementation has to change.

But... how do I write tests against the requirement and not to the code itself then? To write test to your requirements, I say you should use Test Driven Development (TDD) practice, because you will write your tests before you think of your code and how to implement it, decoupling your tests from dirty implementation code. This has many other benefits too, which you can include, you need to understand your requirements before creating a test for it, you can discuss the test with the users to see if they are valid, and you are deferring your commitment to think how to write your code, and also you think how you want to use your code.

This has many implications, because, the code itself is no longer important, because you already defined the 2 most important things of it. First, what it should do? And second, how do you use it? Writing the code is now just a problem on looking for the best way to solve that, and even somebody with little skill should be able to write the code to satisfy the 2 things that defines it, even poor quality code will be ok, if it meets the other 2, so, the code itself is just code, and you should be able to change it as you please to improve it.

TDD will also allow you to break a solution into smaller steps, and as Kent Beck said, it will allow you to manage fear, fear of getting lost in the middle thinking of one piece of code that solves everything. You come up with a lot of code to solve problems which might not actually be there, making things more complex. It allows you to solve things step by step, and incrementally change your code to solve your problems. Also the good thing is that you can define how big your steps are.

Refactoring
Changing the code as I'm pleased is a really powerful tool, because I can put anything in there, and be sure it's still doing what it was meant to. So, what does refactoring means, other than reviewing your code and make it more understandable, to make it perform better in some cases?
It means a lot more in most of the cases. I haven't seen many people doing everything right the first time they do it, especially when you have less skilled team members, it forces them to think on doing things better, and not on the only way they know.

This is the time the developers need to take to improve their code and also their skills, because they are already sure they already solved the problem, so now they only can peacefully focus on improve what they have done, without the rush to get it done.

This is not an optional step in the process, the code is not done, until its meets your quality standards and duplicate code is removed. If you are not refactoring, you are leaving more and more work for the future, and this step is the one that adds quality to your project. Just like in anything, you cannot add quality at the end of your project; it needs to be part of your process.

Imagine living on a building, where all the things are in place, but as soon as you open your door, everything starts to fall apart, then your only way out would be start over.

I think many will agree with me on the following. If you have been in a company for a long time, you always keep hearing, we need to rewrite this system. Because it's full of problems and the ones who did it are gone, and understanding it is impossible. That's where all your projects will end up if you are not improving your code as you build it.

The same problem will be repeated so many times and fixing it will take you a long time. But if you have been improving your code as you create it, you can find better ways of doing things, specially avoiding the same errors over and over again.

Continuous Integration
Ok, my code is tested, and it meets quality standards, how I am able to ensure the code will be production ready when I finish an iteration? It's not a matter of running the entire tests the last day, and if they pass, then deploy your app. You will be less likely to finish on time if you think that way. If you integrate all your code your last day, you will see many problems to solve and it will be too late already.

Many agile methodologies encourage continuous integration. What does this mean? This means you need to ensure your application of bad code or bugs to prevent you from finding surprises when it's too late.

Ensure everything might be too much, but again, this depends on your business, to which level of integration you need to get. If you look at lean software development, try to ensure enough to feel comfortable. Whenever you experience a problem, and the problem starts repeating itself, the best thing to do, is to put something in your continuous build, so the issues is no longer repeated, instead of manually fix it every time.

How Agile does ensures Project Success then?

Continuous integration is like testing and refactoring, you invest time now, to avoid problems in the future. Do I care about those problems? It depends on what you do, and who you are, but... there is something you should never do... is to ignore the consequences of your acts.

If you say you are doing Agile, and you are not doing testing, refactoring and continuous integration to a level that allows you to reduce the risks of failing, then you are not doing agile. You are just ignoring your problems and kicking the ball ahead all the time, and the ball gets bigger and bigger, and it will get harder and harder to kick the ball ahead again. All this untill the only way out is to start over.

What agile promotes is do whatever your business need to continue, defer the commitment to fix it, and then FIX it when you get a chance and the business lets you. Sometimes you will find this very hard, as you are always behind of schedule, things change at the last minute, and you are always on the run to get things working. Sometimes that's the way it has to be, but you need to be very clear on how you raise this as a problem in your company, and those changes will cause in the future, and pray for them to understand.

Also it states that you have to focus on the most important thing for the business. Does it include building a whole Web Framework to solve their problems as the first thing? I don't think so, at least not for most of the project. Always focus on solving business problems, and not the technical problems. They are not IT guys for the most part, so they count on you to tell them what they need.

I stated that most of the problems were due to people not knowing enough of what they had to do at the time they had to do it. Why is agile different of other ways of development software? Well, it aims to reduce this problem in many different ways, but mainly following a basic ceremony with specific meetings

1. Iteration Planning
2. Daily Meetings
3. Review/ Feedback meeting
4. Introspective or process related meeting

Iteration Planning meeting.
In this meeting you will commit to a set of issues to solve, based on what you know at that time.

Daily meetings
You need to report progress or the lack of it, and what is in your way to finish. This meeting is to avoid getting to your last day of your iteration without doing anything because X didn't give you the right paper or any good excuse you can come up with.

Review/Feedback meeting
At the end of the iteration, you will display what has been done, how it's working, or not, but in both cases gives you feedback, well or bad. Did you finish everything you said you would? Is everything working as they expected? How many problems did you find? Take note of all the feedback you get.

Introspective or process related meeting
Look at all the notes you got from the review meeting, and see how you must change your process to solve the bad things, and leverage the good things. This meeting will tell you if you have to test more, add more validations in your continuous integration piece, and add more people to finish on time or anything you need to solve those problems.

Always the problem is the process and not people. If there is a developer causing more problems than solutions, you need to find out what the problem is. Then think how you can put something in place in the process to prevent those problems to be found the last day.

Training could be the problem, better specs, or specs in a different language... probably nobody realized that guy only speaks Guarani, or he never got a paycheck for the last 6 months. So... everything needs to be considered here, and you have to change your process for the things you can within the project scope of action, and raise the problems like salary to the appropriate channels in your company.

The faster the feedback, the faster people knows what they are doing wrong, and what needs to be changed in order to improve. This is also guided by the golden rule, fail earlier, fail often. This doesn't mean you have to fail on purpose, this is related to go, do it, and see what went wrong and fix it; instead of plan what you think you have to do for a long time to then start doing it.

I've seen a company that spent 1 year in a project to create an ERD diagram for their DB, to then use it in another project. By the time I did a demo of a product, they showed me 600 pages ERD impossible to understand unless I spend another year going over it. I'm sure that by the time they actually start with the project, that ERD will be outdated for the most part, and they just spent too much money on it to get nothing or very little from it.

Conclusion
Being agile or adopting agile practices will not ensure success of your project.

It will allow you to make things clearer since your first iteration.
It will allow you to identify problems earlier, and put something in place to solve them. If you just ignore them, they will just keep growing and growing.

It will allow you to adapt to changes faster, as you identify the changes as they appear.

If you just maintain status quo, you will fail as in any other methodology. You must have one constant in your software development. Continuous Improvement. Don't waste time in things not important. I heard many times, if you are not going forward, you are going backwards. I just hope when you are going backwards is because you are about to run forward faster :-)