Software Engineering: The problem with the production line

Is software engineering the best approach for developing software? Does it apply for the majority of the software projects or just a very few of them?

Software Engineering was an answer for the perceived "software crisis", back in 1968, in the First NATO Software Engineering Conference and it was created to solve the problems of extremely large NATO and U.S. Department of Defence projects. In the majority of these projects, the hardware was still being designed and with no hardware to test, there was plenty of time to investigate requirements and write the software specifications. Hardware controlled by the software was generally worth billions of dollars like in the case of space shuttle and the Safeguard Ballistic Missile Defence System. People's lives and national security were also at stake.

The IEEE Computer Society's Software Engineering Body of Knowledge defines "software engineering" as:

Software engineering is the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software, and the study of these approaches; that is, the application of engineering to software.

Software Engineering can be very effective when developing safety critical systems like the one for the space shuttle as described on "They Write The Right Stuff" - Fast Company article, from 1996:

The last three versions of the program - each 420,000 lines long - had just one error each. The last 11 versions of this software had a total of 17 errors. Commercial programs of equivalent complexity would have 5,000 errors.

Of course that this looks really impressive but there is more to it:

Money is not the critical constraint: The group's $35 million per year budget is a trivial slice of the NASA pie, but on a dollars-per-line basis, it makes the group among the nation's most expensive software organizations.

Another extract from this same article when they were discussing the process:

And the culture is equally intolerant of creativity, the individual coding flourishes and styles that are the signature of the all-night software world. "People ask, doesn't this process stifle creativity? You have to do exactly what the manual says, and you've got someone looking over your shoulder," says Ted Keller (senior technical manager). "The answer is, yes, the process does stifle creativity."

Many of the NATO and US Department of Defence took years, some over a decade to complete. Many had hundreds of people involved and almost half of the time was spend in requirements and design specifications, with uncountable reviews and approval cycles. Of course that, due to their reliability and quality, many are still in use today.

Software Engineering for the Masses

More and more hardware became cheap and business of all sizes needed software in order to survive and be competitive. The difference was that a the big majority of these businesses couldn't afford to pay $35 million dollars per year and neither wait for too many years to start benefiting from their software. Also, many of those software projects were not manipulating expensive hardware or dealing with life-threatening situations. Edward Yourdon, in his book Rise and Resurrection of the American Programmer, wrote:

I'm going to deliver a system to you in six months that will have 5,000 bugs in it - and you're going to be very happy!

It was clear that software engineering processes had to be adapted in order to satisfy a more impatient and lower budget legion of businesses.

The "Good Enough Software" Era

Over the decades, many new software engineering processes and methodologies were created in order to make software development faster and cheaper. The most adopted were the ones based on iterative and incremental development that evolved into the agile software development.

Agile software development and "good enough software" were an amazing improvement in bringing costs down, mitigating risks with quicker feedbacks and much faster time to market.

Regardless the methodology or process used, software projects are still failing. Some fail because they were over-budget, others because they were not delivered on time, others failed to satisfy the requirements and business goals, et al.

The main problem is that for decades, software development was seen as a production line. That's the software engineering perspective of software development. Processes and methodologies are generally much more focused in making this production line more productive, creating different management and organisational styles than actually trying to make the employees more capable. Developers are treated as mere brain-damaged programmers and are at the bottom of the food chain.

Good enough is not always good enough

Using agile, lean or any methodology is not enough. It is all about the people involved in the project and their willingness to succeed and be proud of what they produce. If developers are seen as the least important and cheapest members of a software project, the maximum that this team is going to produce is mediocre software, regardless of methodology.

A software project would have much better chances to succeed with "good people with a bad process" than "mediocre people with a good process". Good people would always find a way to improve the process, be productive and produce something that they are proud of. Mediocre people accepts whatever is established, even when it is not good enough, and will produce just what they were asked for.

In a software project, if a company wants a good software, they will need good and empowered software developers. Instead of 10 mediocre people with a good process, it would be better to have 3 or 4 good people and empower them to deliver the project.

A process, imposed by managers and people that have no clue how to write software, will just guarantee that mediocre software is predictably delivered (if lucky). On the other hand, a team of self-organised and great developers would have a better shot at creating a more efficient way to produce great software, constantly trying to improve they way they work.

A process should never be more important than the people. Managers should facilitate the work of great software developers and not tell them what to do. It should always be harder to replace a good software developers than a manager since they are the ones that know the system inside-out. Managing a few well motivated, well paid and good professionals is always easier than manage many mediocre people. 

Software development is a creative and highly skilled profession that takes years to master. While software development is treated like a production line, projects will continue to fail.

Source

Software Craftsmanship: The New Imperative - ISBN 0-201-73386-2, 2002
http://en.wikipedia.org/wiki/Software_engineering
http://en.wikipedia.org/wiki/Software_crisis
IEEE Standard Computer Dictionary, ISBN 1-55937-079-3, IEEE 1990
"They Write The Right Stuff", Fast Company, http://www.fastcompany.com/magazine/06/writestuff.html
Safeguard Program: http://en.wikipedia.org/wiki/Safeguard_Program
Stephenson, W. E. "An analysis of the resources used in the SAFEGUARD system software development"
Edward Yourdon  - http://yourdon.com/about/ 
http://en.wikipedia.org/wiki/Iterative_and_incremental_development 
http://en.wikipedia.org/wiki/Agile_software_development

Which type of barista are you?

A colleague from New Zealand once was telling me about a type of coffee that was originated there. It's called Flat White Coffee. We were discussing about the difference between this coffee and all the other types of coffee with milk. Eventually we started talking about the quality of the drink and what makes it be better or worse. He mentioned that, of course, the quality of the beans and milk are very important for a good coffee but what makes a good coffee an excellent coffee is the barista's ability.

What makes a coffee to be a rubbish coffee then? If the coffee grains are rubbish, you will have a rubbish coffee. However, for a flat white coffee, coffee is just one variable in the equation. There are other variables like how the grains are roasted, steaming the milk at the right temperature, not adding sugar, how the milk is poured, the microfoam on top of the drink, etc. See the distinction from cafe con leche for details. Anyway, the point is, a rubbish (or careless) barista, with rubbish coffee will produce a rubbish coffee drink.

Rubbish Coffee

Clearly a business that serves a rubbish coffee will not survive if their main business is to sell coffee drinks. Also, this business will never attract customers that really appreciate a good coffee.

Average Coffee

Producing an average coffee is easy. You buy average coffee grains, hire an average barista and, hey presto, you have an average coffee drink. There is nothing wrong with an average coffee drink if you are not in the coffee business. However, if you are in a coffee business, your business will be just another one. No one will remember you and chances are that you will have occasional customers, but not regulars.

Good Coffee

Producing a good coffee is not that simple. However, it does not need to be expensive. You don't need to buy the best quality ingredients to be able to make a good coffee. Best quality ingredients are expensive and inevitably will make your coffee drinks more expensive as well. You can mitigate this situation hiring a good barista, or maybe someone that has the potential and willingness to become a good one. Clearly, the barista that prepared the coffee above is trying his best to make a good coffee. It's still not perfect (purely looking at the pattern on top of the drink) but it is definitely a much better coffee than the normal and average one that you get everywhere. It is clear that the barista cares about it and eventually he will be able to produce a very good coffee.If you are in the coffee business, producing anything less than a good coffee is just unacceptable.

Great Coffee

And then you have the great coffee, that is made with great coffee beans, carefully roasted, prepared in a very good coffee machine and by a great barista. Great baristas are proud of their ability to prepare a great coffee and they would not work for too long for a business where coffee making is not treated with the deserved respect.

Now imagine that you are the barista. But instead of coffee, you produce code. Imagine that the the coffee grains, milk and coffee machine are the tools you use to produce your code like the computer language, the IDE, the database, etc. Instead of serving coffee for your customers, you are producing software that will be used by your team mates, project sponsors, the company they (or you) work for, external clients, etc. 

Like in the flat white coffee example above, of course that the tools we use are very important when producing a good software. However, more importantly, it is the quality of the software engineers that counts. A good barista can make a good coffee even when using average coffee grains, due to his or her ability to combine ingredients and prepare the drink. A bad barista can ruin the coffee even if he or she is using the best quality coffee beans. The biggest difference between the two baristas is how much they care about each cup of flat white coffee they prepare. Their pride and willingness to achieve the best pattern on top of each drink. The great feeling of achievement when they produce a great one. The happiness to see returning customers, queuing and waiting their turn to order the coffee that he, skilfully, prepares.

So, which type of barista are you?