Testing legacy: Hard-wired dependencies (part 1)

When pairing with some developers, I've noticed that one of the reasons they are not unit testing existing code is because, quite often, they don't know how to overcome certain problems. The most common one is related to hard-wired dependencies - Singletons and static calls.

Let's look at this piece of code:

public List<Trip> getTripsByUser(User user) throws UserNotLoggedInException {
    List<Trip> tripList = new ArrayList<Trip>();
    User loggedUser = UserSession.getInstance().getLoggedUser();
    boolean isFriend = false;
    if (loggedUser != null) {
        for (User friend : user.getFriends()) {
            if (friend.equals(loggedUser)) {
                isFriend = true;
                break;
            }
        }
        if (isFriend) {
            tripList = TripDAO.findTripsByUser(user);
        }
        return tripList;
    } else {
        throw new UserNotLoggedInException();
    }
}

Horrendous, isn't it? The code above has loads of problems, but before we change it, we need to have it covered by tests.

There are two challenges when unit testing the method above. They are:

User loggedUser = UserSession.getInstance().getLoggedUser(); // Line 3  
   
tripList = TripDAO.findTripsByUser(user);                    // Line 13

As we know, unit tests should test just one class and not its dependencies. That means that we need to find a way to mock the Singleton and the static call. In general we do that injecting the dependencies, but we have a rule, remember?

We can't change any existing code if not covered by tests. The only exception is if we need to change the code to add unit tests, but in this case, just automated refactorings (via IDE) are allowed.

Besides that, many of the mocking frameworks are not be able to mock static methods anyway, so injecting the TripDAO would not solve the problem.

Overcoming the hard-dependencies problem

NOTE: In real life I would be writing tests first and making the change just when I needed but in order to keep the post short and focused I will not go step by step here .

First of all, let's isolate the Singleton dependency on it's own method. Let's make it protected as well. But wait, this need to be done via automated "extract method" refactoring. Select just the following piece of code on TripService.java:

UserSession.getInstance().getLoggedUser()

Go to your IDE's refactoring menu, choose extract method and give it a name. After this step, the code will look like that:

public class TripService {

    public List<Trip> getTripsByUser(User user) throws UserNotLoggedInException {
        ...
        User loggedUser = loggedUser();
        ...
    }

    protected User loggedUser() {
        return UserSession.getInstance().getLoggedUser();
    }
}

Doing the same thing for TripDAO.findTripsByUser(user), we will have:

public List<Trip> getTripsByUser(User user) throws UserNotLoggedInException {
    ...
    User loggedUser = loggedUser();
    ...
        if (isFriend) {
            tripList = findTripsByUser(user);
        }
    ...
}  
 
protected List<Trip> findTripsByUser(User user) {
    return TripDAO.findTripsByUser(user);
} 
 
protected User loggedUser() {
    return UserSession.getInstance().getLoggedUser();
}

In our test class, we can now extend the TripService class and override the protected methods we created, making them return whatever we need for our unit tests:

private TripService createTripService() {
    return new TripService() {
        @Override protected User loggedUser() {
            return loggedUser;
        }
        @Override protected List<Trip> findTripsByUser(User user) {
            return user.trips();
        }
    };
}

And this is it. Our TripService is now testable.

First we write all the tests we need to make sure the class/method is fully tested and all code branches are exercised. I use Eclipse's eclEmma plugin for that and I strongly recommend it. If you are not using Java and/or Eclipse, try to use a code coverage tool specific to your language/IDE while writing tests for existing code. It helps a lot.

So here is the my final test class:

public class TripServiceTest {
        
    private static final User UNUSED_USER = null;
    private static final User NON_LOGGED_USER = null;
    private User loggedUser = new User();
    private User targetUser = new User();
    private TripService tripService;

    @Before
    public void initialise() {
        tripService  = createTripService();
    } 
        
    @Test(expected=UserNotLoggedInException.class) public void 
    shouldThrowExceptionWhenUserIsNotLoggedIn() throws Exception {
        loggedUser = NON_LOGGED_USER;
                 
        tripService.getTripsByUser(UNUSED_USER);
    }
        
    @Test public void 
    shouldNotReturnTripsWhenLoggedUserIsNotAFriend() throws Exception {             
        List<Trip> trips = tripService.getTripsByUser(targetUser);
                 
        assertThat(trips.size(), is(equalTo(0)));
    }
        
    @Test public void 
    shouldReturnTripsWhenLoggedUserIsAFriend() throws Exception {
        User john = anUser().friendsWith(loggedUser)
                            .withTrips(new Trip(), new Trip())
                            .build();
                 
        List<Trip> trips = tripService.getTripsByUser(john);
                 
        assertThat(trips, is(equalTo(john.trips())));
    }

    private TripService createTripService() {
        return new TripService() {
            @Override protected User loggedUser() {
                return loggedUser;
            }
            @Override protected List<Trip> findTripsByUser(User user) {
                return user.trips();
            }
        };
    }        
}

Are we done?

Of course not. We still need to refactor the TripService class. Check the part two of this post.

If you want to give it a go, here is the full code: https://github.com/sandromancuso/testing_legacy_code

Working with legacy code

Context

Large organisations' systems may have from tens of thousands to a few million lines of code and a good part of those lines is legacy code. By legacy code I mean code without tests. Many of these systems started being written many years ago, before the existence of cool things and frameworks we take for granted today. Due to how some systems are configured (database, properties file, proprietary xml) , we cannot simply change a class constructor or method signature to pass in the dependencies without undertaking a much larger refactoring. Changing one piece of code can break completely unknown parts of the system. Developers are not comfortable in making changes in certain areas of the system. Test-first and unit testing is not widely used by developers.

The Rule

In our commitment to make the existing systems better, more reliable, easier to deal with (changing and adding more features), we established the following rule: We can not change existing code if it is not covered by tests. In case we need to change the existing code to be able to write the tests, we should do it only using the automated refactoring tools provided by our IDEs. Eclipse and IntelliJ are great for that, if you are a Java developer like me. 

That means, before we make any change, we need to have the current code under test, guaranteeing that we don't break its current behaviour. Once the existing code is under test, we then write a new test for the new feature (or change an existing test if it's a change in existing behaviour) and finally we are can change the code.

Problem

There is an eternal discussion on forums, mailing lists and user groups about TDD being responsible for the increase or decrease of the team's velocity. I don't want to start this discussion here because we are not doing TDD for majority of the time. With legacy code, we spend the majority of our time trying to test existing code before we can do TDD to satisfy the new requirement. And this is slow. Very slow. It can be, in our experience, somewhere between 5 to 20 times slower than just making the change we need and manually test it.

You may be asking, why does it take so much longer? My answer is: If you haven't, try unit testing all the possible execution paths in a 2000+ line class? Have you tried to do it with a 1000 line method, full of multiple return statements, hard-wired dependencies, loads of duplication, a big chain of nested IFs and loops, etc? Believe me, it will take you a bloody long time. Remember, you can't change the existing code before writing tests. Just automated re-factorings. Safety first. :-)

Why are we "under-performing"?

Quite often, after start following this rule, management will enquire why the team is under-performing. Although it is a fact that the team is taking longer to implement a feature, this is also a very naive view of the problem. When management and development teams talk about team's velocity, they just take into consideration the amount of time taken to write the code related to a few specific requirements.

Over the time, management and teams just forget how much time they spend in things that are caused by the lack of quality and tests in their code. They never take into consideration that there is absolutely no way that developers would be able to manually test the entire system and guarantee they didn't break anything. That would take days, for every piece of code they write or change.  So they leave that to QA.

By not increasing the test coverage, the team will never have a regression test suite. Manually testing the system, as it grows in size and complexity, will start taking longer and longer by the QA team. It will also be more error prone, since testing scripts need to be kept in sync with the current behaviour of the system. This is also waste of time.

Also, just hacking code all the time will make the system more fragile and way more complicated to understand, which will make developers to take longer to hack more code in.

Because we can't press a button and just unit test a specific piece of code, the team is forced to spend a lot of time debugging the application, meaning that the team needs to spend time compiling and deploying it. In certain systems, in order to test a small piece of code, you need to rely on other systems to trigger something so your system can respond. Or you need to manually send messages to a queue so your piece of code is invoked. Sometimes, depending of the complexity of your system and it's dependencies, you can't even run it locally, meaning that you need to copy your application to another environment (smoke, UAT, or whatever name you use in your company) in order to test the small piece of code you wrote or changed. And when you finally manage to do all that to execute that line of code you just added, you realise that the code is wrong. Sigh. Let's start over.

Since I started practising TDD, I realised that I very rarely use my debugger and very rarely have to deploy and run my application in order to test it.

So basically, when people think we are spending too much time to write a feature because we were writing tests for the existing code first, they are rarely considering the time spend elsewhere. I've seen it many times: "We don't have time to do it now. Let's just do a quick fix.". And then, contradicting the laws of physics, more time is created when bugs are found and the QA phase needs to be extended. Panic takes over: "Oh, we need to abort the release. We can't go live like that.". Exactly.

The graphic above is not based in any formal research. It is purely based on my own experience during my career. I also believe that the decrease in the team's velocity is related to the state of the untested code. There are untested code out there that is very well written and writing tests to it is quite straightforward. However, in my experience, this would be an exception.

In summary, if you are constantly applying the "Boy Scout Rule" and always keep improving the code you need to touch, over the time you will be making your code better and you will be going faster and easier. If you don't do that, you will have a false impression of good velocity at start but gradually and quite often without noticing, you will start to slow down. Things that used to take a few days, now take a few weeks. Because this lost of velocity happened slowly (over months or even years), no one really noticed until it was too late and everything now take months.   

Drifting away and losing focus

One interesting side effect of constantly making legacy code better, writing tests and refactoring, is that it is too easy to be carried away. More than once I've seen pairs, while trying to clean a piece of code, drifting away from the task/story they were working on. That also happened to me quite a few times. We get carried away when making progress and making things better.

Very recently I was asked by one of the developers: "When do we stop?". My answer was: "When we finish the task.". In that context, that meant that the focus is always on task at hand. We need to keep delivering what was agreed with our product owners (or whoever the person in charge of the requirements are). So, make the system better but always keep the focus on the task. Don't go crazy with the refactoring trying to re-write the whole system in a few days. Do enough to finish the task and try to isolate the parts that are still not good enough. Baby steps. 

A more positive side effect

Another very common problem in legacy systems is the lack of understanding about the system and also about the business. Quite often we find situations where the developers and business people are long gone and no one really knows how the system behave.

Writing tests for the existing code force us to understand what it does. We try to mine some hidden business concepts in the middle of the procedural code and try to make them very explicit when naming our tests. That is a great way to drive our refactoring later own, using the business concepts captured by the tests. Quite often, we also get the business people involved, asking them questions and checking if certain assumptions make sense.

In writing the tests and refactoring the code, the previously completely unknown behaviour of the system is now well documented by the tests and code.

Quality is a long term investment

In a recent conversation, we were discussing how we could measure our "investment" in quality. A quick and simple answer would be that we could measure that by the number of bugs and the velocity that the teams are delivering new requirements. However, this is never too simple. The question is, why do you think you need quality? Which problems do you have today that makes you think they are quality related? What does quality mean anyway?

When working with legacy, after many years, we tend to forget how things were in the past. We just "accept" that things take a long time to be done because they have been taking a long time to be done for a long time. We just accept a (long) QA phase. It's part of the process, isn't it? And this is just wrong.

Constantly increasing the quality level in a legacy system can make a massive difference in the amount of time and money spend on that project. That's what we are pursuing. We want to deliver more, faster and with no bugs.

It's up to us, professional software developers, to revert this situation showing a great attitude and respect for our customers. We should never allow ourselves to get into a situation where our velocity and quality of our work decreases because of our own incompetence.

A change in attitude - Legacy code

Attitude is a little thing that makes a big difference.  ~Winston Churchill 

 
Not long ago, I gave a talk about Software Craftsmanship where I asked who liked to work on greenfield projects. Almost everyone raised their hands. Then I asked who liked to work with legacy code. Besides one or two friends that were there, almost everyone kept their hands down.
 
It is always nice to start a project from scratch, be able to choose the technology, use the latest frameworks and have a lot of fun writing code without worrying about breaking existing features or having to understand existing code. Working on greenfield is great, mainly with an experienced and disciplined team using TDD since day one. Progress flows naturally and quickly.
But as soon as we are working with code written by people that are long gone, no tests, no documentation, we go mental. We notice we are going mental by the number of WTF we say during the day. We may get moody and start hating to work on a specific system or in parts of it. Frustration becomes a constant.

A change in attitude

If you don't like something change it; if you can't change it, change the way you think about it.  ~Mary Engelbreit

 
Although I wrote "we" and "us", that was really how I used to feel when working with legacy code. However, in the past few years, I learned many things. An obvious one is that moaning, complaining and cursing won't make my life easier or better. If I want things to be better I need to do something about it.
 
Today, when I look at legacy code, instead of moaning and getting frustrated, my attitude is to try to understand it and make it better, constantly applying the Boy Scout Rule.
As my friend David Green said in a twitter conversation, what I agree 100%, improving and understanding legacy code can be massively rewarding. The process of trying to make sense of a big ball of mud seems daunting but if we just focus in small parts of it, one at a time, and start improving them (writing tests, extracting methods and classes, renaming variables, etc), bit by bit, things become much easier and enjoyable. 

Working with legacy code is almost like solving a big jigsaw puzzle. We don't do that all at once. We start by separating the pieces into groups, often starting with the edges and corners and then separating other small pieces by colour, pattern, etc. We now have a few (logical) groups and we start to form a higher level model in our head. What before was a bunch of random pieces (or a big ball of mud), now is a bunch of smaller groups of random pieces. Still not very helpful or encouraging, but nonetheless some progress. Bit by bit, we start working on one of these groups (parts of the code) and we start putting some pieces together (writing tests for the existing code, which will help with our understanding of the code, and refactoring it).

Once we start putting some pieces together, we start seeing a small part of our jigsaw puzzle picture. We get excited because now it's getting real. It's starting to make sense and we are happy we are making progress. The more pieces we put together the more excited we are about finishing the jigsaw puzzle. The more pieces we put together, the easier it gets to put more pieces together. And that's exactly the feeling I've got when working with legacy code now. For every piece of code I make better, more I want to make the whole code better. The feeling of achievement is really rewarding. What before was something I could barely read and took ages to understand, now reads like a story and every developer can understand it with no effort at all. Maybe when a good part of the code is stable (covered by tests, loosely coupled, well defined responsibilities, etc), I could even introduce the cool and new frameworks that I always wanted to use. I could upgrade some library versions. I could even throw a lot of code away and replace it with a framework because now my code is clean and modularised and replacing one part of the system will not break another. I don't even need to envy my friends working on greenfield projects any more. 

Different challenges 

Another cool thing about legacy code is that it forces you to think in a different way. In greenfield projects, when developing a new feature, we write a test and start implementing stuff. In legacy, sometimes you can't even instantiate a class in order to test it due to all its dependencies. A change in one place can cause an impact in obscure parts of the system. We have an option here. We can see these challenges as a pain in the ass or can be see them as very interesting problems to solve. I prefer the latter.

The professional side 

Although it is OK to have fun and enjoy what we are doing, we always need to remember that we are professionals and being paid to write software. Software is an asset and a lot of money and time is invested on it. As every investment, clients want to maximise the return of their investment. The more we improve and keep the software clean, the longer the client will be able to benefit from it. Stretching the lifespan of a software also maximises the client's return of investment.

Conclusion

At the end of the day, it does not matter if you are working on a green or brownfield project. It's your attitude towards your job that will make it more or less enjoyable.

Re-drawing my own map: A new milestone

For every step you take towards mastery, your destination moves two steps further away. Embrace mastery as a lifelong endeavour. Learn to love the journey.

--George Leonard, Mastery

It is with a mixture of sadness and excitement that I would like to announce that, after over five years, I'm leaving Valtech on the 18th of May. The decision was not an easy one and took me months to figure out what the next step in my long road would be. As I said in a recent interview, I love working for consultancy companies and that's the main reason I spent over ten years (two-thirds of my career) working as a consultant.  

Working for Valtech

Valtech is a fantastic company to work for and had a huge impact in my personal and professional life. During my time there I had the opportunity to work on a great variety of projects, different companies, different industries and different technologies. Most importantly, I had the opportunity to meet and work with a lot of great people that helped me to become a much better professional.

If there is one thing that I will never complain about Valtech is that I did not have recognition for the work I've done. I have started at a Valtech in a relatively junior role, according to Valtech's grade scheme, and bit by bit, with a lot of support and trust from my colleagues I was given more and more responsibility and gradually climbed my way up to one of the most senior positions.

There is no such a thing as a perfect company but if I had to point out the best thing about Valtech, I would say that, beyond the shadow of a doubt, it is its people. People that I learnt a lot from, that helped me to feel at home in the UK, that helped me with personal issues, people that challenged me, that pushed me to my limits, that gave me constructive criticism, that trusted me and empowered me to do my job, people that helped me to be better.

I may not be in the office or in a client site full time any more but I will always be around. As it happened before (I took a year off to work on a startup idea and came back) I'll always be around for drinks and events.

Special thanks

I would like to thank you everyone at Valtech that had to put up with me for all this time. I know I can be a pain in the neck quite often sometimes. :-)

It's always tricky to mention names since there is always the risk of people left out feeling a bit upset but I feel that I need to thank some people in a more personal level, so apologies for the ones I did not mention. In no particular order: Toby Mckenzie for caring about me and every single consultant during tough times, all his herculean effort to keep every one happy, finding us good projects; David Draper for challenging many of my beliefs and opinions, for the many advices, for supporting my involvement with user groups and events and for the effort in making Valtech a place of excellence; Mashooq Badar for the fantastic time we had together in many projects, for all the things I learnt from him and for making me open my mind about so many things. Ah, and for all the Blazing hot! moments; Phil Hall for the support and keeping the doors open to me and LSCC; Kevin Harkin (I can't believe he does not use twitter) for the fun we had together, for his friendship, advices, ranting sessions and memorable nights at the pub; Andrew Rendell for his professionalism, trust and for being a great role model for every Valtech consultant; And last but not least, Akbar Zamir, for pushing me and challenging me to be better, for all the advice, trust, knowledge and help, for being a great career manager and most important of all, for being a great friend.

Thanks for the great five years that I spent there. It was an absolute pleasure to work with all of you and be part of this great company.

The future

It’s not just a question of conquering a summit previously unknown, but of tracing, step by step, a new pathway to it.

--Gustav Mahler, musician and composer

I'm joining UBS as a senior developer at a director level, starting on the 23rd of May. Due to my involvement with the software craftsmanship movement, this came as a surprise to many people, including myself, mainly because investment banks tend to be almost a hostile environment for agile and software craftsmanship initiatives. When I started re-drawing my own map, investment banks were an avenue that I was not considering to explore.

As I said before, choosing my next step was not easy. I had a few things in mind that were non-negotiable: I wanted different challenges, that means, things that I haven't done before, keep having fun and loving my job, a potential long term commitment where I would have time enough to put into practice many ideas and beliefs and most importantly, have a long term career as a software developer but with a lot of space to keep growing as a professional.

I was fortunate to have had many opportunities during this time but the majority of them could not satisfy all the items above. I was determined to keep doing what I had been doing throughout my entire professional life that is just to work for companies that I really want to work for, I mean, companies that would be able to offer me what I was looking for at that point in time. For me, that's the best way to keep fuelling the passion that I have for what I do. Joining a company just because of money is and always has been totally out of question.

UBS came along with a very interesting proposition. They want to improve the quality of their software and recognise that agile and software craftsmanship are a great way to get there. They were interested in people with no previous investment bank experience, what is very unusual for an investment bank. They want people that can think different, that are passionate and can help them drive this transformation. I had five interviews and was very pleased to see so many people striving to be and do better.

As far as I understand, my main role will be to work as a hands-on developer, embedded in a team, helping to improve quality, leading by example and mentoring other developers. They also expect me to give internal talks, training, promote events, disseminate passion and promote the craftsmanship values and techniques. In the future I'll be working with other teams in the UK and in other offices around the world. But make no mistake. I'll have a hell of interesting and tough challenges ahead of me and I hope to live up to all their expectations.    

Besides that, I'll keep running the London Software Craftsmanship Community (LSCC) alongside my friend David Green and try my best to give something back to the wider and great community of software developers out there that some many times I benefited from.

Thanks everyone for being part of long road journey.

Open-source developers deserve respect

I was recently reading Gojko Adzic's blog post called How is it even possible for code to be this bad? I must admit that I was very sad to see tremendous lack of respect towards the Hudson/Jenkins community and towards open source software developers in general.

Firstly I would like to say that any person out there that is working on an open source project deserves a lot of respect, mainly the ones working on projects that bring so many benefits to so many companies and developers around the world. The velocity that our industry moves forward and evolves is, in general, because of many open source initiatives. It's because of thousands of developers that work on the their spare time, for free, to create software that will make the lives of many other developers, companies and users much easier. These people are kind enough to offer their work to all of us and are humble enough to ask and accept help from many other developers around the globe.

One thing we need to understand is that every project has a history and many things, throughout the lifespan of a project, can be responsible for the success or failure of a project. It's always easier to criticise something instead of trying to help and make it better. As far as I'm aware, I don't think that anyone in the Hudson/Jenkins community ever claimed that their code base is the best example of quality software ever written and we all should learn from them. If that was the case, probably it would justify such harsh words written in the original blog post. 

The only acceptable form of criticism towards any open source project or developer is constructive criticism. 

If we think that a open source project is below standard and/or not good enough, we should either not use it or we should contribute to make it better.

However, I do understand the point Gojko is trying to make. Yes, I agree that good code is not the only thing that makes a project be successful. I think we all could name a few projects that we delivered where clients and/or users were relatively happy but the code base was a bit messy. There are always exceptions to the rules. Trying to get our code as perfect and as clean as possible does not guarantee that our projects will be successful and having a messy code does not always mean that our project will fail either. We all know that. Even my 5 month-old baby girl knows that.

However, I totally disagree with Gojko's statement that "[Hudson's success with "bad" code] is close to the final proof that God doesn't exist for the whole craftsmanship debate". It is almost like saying that Agile is rubbish and everyone should forget about it just because some waterfall projects succeeded. It is like saying that we all should forget about good principles of software development just because some projects succeeded with messy code. In summary, is like trying to get some exceptions and transform them into rules. There are many ways and many things that can contribute to the success of a project. Software craftsmanship is one of them and a very important one. In a software project, the most important thing is the software itself and looking after its quality is an obligation of every developer involved in it.

Although software craftsmanship on its own may not be enough to guarantee the success of a project, the lack of it can be reason for its failure. Single-handed.

Another important point we should ask is the definition of a "successful project" and in which context or environment but I'll leave that for another post.

If exceptions invalidate rules and that is the way we should look at things, does God exist for anything in software development?

Working for consultancy companies

Recently I was approached by the guys from the Graduate Developer Community (GDC) where they asked me for an interview about life in a consultancy company. Their idea, that came out from an email I sent to the London Java Community (LJC) ages ago about different career paths, was to interview professionals that followed different career paths and publish these interviews on a website called GDC Careers so that graduates and people starting in our industry could have an idea about what is available out there in terms of jobs and career paths.

The interview was originally published on the GDC Careers but I'll also be publishing it here in full. I hope it helps any professional that would like to know more about life on consultancy companies.

NOTE: Throughout my career I've worked for three consultancy companies and this is just my general view of all of them. Each project is a different project and different consultants, working for the same consultancy company but in different projects, can have a completely different view of their employers.

Sandro Mancuso has been working as a software developer since 1996 but started writing code for pure enjoyment way before that. Although he has worked for software houses and startups, he spent the majority of his career working for international consultancy companies where he had the opportunity to work on a great variety of projects and across many different industries. He has a BSc in Computer Science and a MSc in Distributed Objects. He is a Co-founder of the London Software Craftsmanship Community (LSCC).

Title – What is your job title?

Senior Consultant.

What is your role about?

My role is to work with clients, helping them to identify what they really need, advise them, provide options, and help them to achieve whatever they want to achieve in the most efficient way.

OK, I know. That was not very specific or helpful. This is because the role of a consultant can vary quite a lot. Different consultants may choose different career paths. Some follow a more business oriented career, advising clients on strategy, marketing, investment, finance, etc. Some follow a more process oriented career, emphasising project management, Agile Coaching, training, business analysis, requirements, etc. Others prefer a more technical career path, working more often as developers, technical architects and team leaders, mentoring, etc. In my case, I took the more technical path so normally I do what I like doing, that is writing code, regardless of my position in the project. Although I was a hands-on developer on all my assignments, many of them had elements of mentoring. Some times I also got involved in customised training courses for clients, like TDD, Agile, etc.

I work mainly in Agile Java projects. Technologies vary quite a lot since every now and again we are working for different clients, different industries and completely different systems. Sometimes clients already have something in place and the technology stack is already defined. In this case, we often use whatever they are using to start with but we always try to make suggestions if we see that other technologies or frameworks should be used instead. Some other projects, mainly greenfield ones, we generally have freedom to choose the technology stack we judge is the best for the problem. 

What are the best/most positive parts of the job/industry?

In my view, the best thing about being a consultant is the amount of technologies and business domains that you are exposed to during your career. Because of the frequency that we change projects, we end up being exposed to a lot of interesting problems quite early in our careers, giving us a great understanding of many different software projects.

Networking is definitely another very positive aspect of being a consultant. As we move from project to project, client to client, we end up meeting a lot of people, in different organisations, what can be very useful as you progress in your career. The more people you work with or for, the more options you will have in the future.

Another important point is that, as a consultant, you need to be sellable. This may sound a bit negative, but if fact, what it means is that you need to be at the top of your game, knowing the latest technologies, methodologies and trends. The more you know, the easier it will be for a consultancy to place you in a project (sell you to a client). As the consultancy companies recognise that, in general they have different mechanisms (training, conferences, internal talks, events, etc) that you can take advantage of to improve your skills.

Because your peers are also moving from project to project and using different technologies in each project, it is amazing what you can learn from them. The variety of information shared among consultants about projects, businesses, methodologies, technologies, etc, is absolutely priceless. There is always someone doing something different somewhere.

What are the negative parts to the job/industry?

The main complain that consultants have is that in general you don’t have much saying on where you are going to be sent to. Once you decide you want to work for a consultancy company, you need to be aware that travelling is always a possibility. A few lucky consultants get to stay for a long time on projects close to their homes but in general this is not the case. If travelling or long commute is not an option for you, you really need to think hard before becoming a consultant. For those living in big cities, like London for example, in general this is not a big problem since the majority of the clients will be there anyway.

According to your level of seniority, years in the company and history of projects you delivered, you can refuse to go to some projects, in case you don’t like the project (less often) or if the commute is bad (more often). However, this is a card that needs to be played very wisely since you don’t get to play it more than once in a short period of time.

In some projects, you may be seen as an outsider by a few permanent employees. Some feel threatened by the presence of a “consultant”, since you may be “exposing” problems in the current process and software. It’s part of our job to deal with that and do whatever we can to be seen as another team member, a person that shares the same goals, and not an outsider.

Career Path

What is the standard career path/qualifications?

This varies quite a lot since consultancies may offer different type of services and they will need people that are more specialised in some areas than in others. Services offered may include software development, project management, training, coaching, business advice (strategy, finance, investment, etc), auditing, etc. Some consultancies tend to work more on the “consulting” side of things and tend to engage with clients at a more senior level (CEOs, CTOs, Directors, etc). Other consultancies are more focused on software delivery and tend to engage with clients at lower levels (directors, project managers, head of departments, etc). And, of course, some consultancies do a bit of both.

I’ll just talk about consultancies that focus more on software delivery since that’s my background.

My title is Senior Consultant but this is a reasonably vague title. Many consultancies, internally, distinguish their consultants by grades. Each grade has different ranges of salaries and responsibilities and we take on responsibilities on projects, in general, according to our grade.

For people aspiring a more technical career, in the entry levels you are expected to have a good knowledge of a programming language (in our case Java or .Net), most common frameworks, good understanding of Object-Oriented Programming and you would be working as a team member (developer). As you move up through the grades, you may take roles where you will be leading teams, be the technical architect, enterprise architect, Agile coach, etc. At the higher grades, besides all the technical knowledge you will also have some extra responsibilities like writing technical proposals, pre-sales, project inceptions, project management, manage clients expectation, etc. However, some consultants manage to reach the higher grades and remain totally hands on. That’s the path I’m following.

To be successful as a consultant, you need to be a well-rounded professional. Just knowing a programming language well is far from being enough to be a good consultant. You are expected to have knowledge in many different areas since you never know in which project you will end up and which position in the project you will have. And regardless what the project is, you need to be ready. Of course that we all know that is simply impossible to know everything that is out there but at least you need to show that you have the motivation and drive to learn whatever is thrown at you.

Besides the technical skills, another thing that is very important for a good consultant is soft skills, due to the amount of iteration with clients. I’m still working to improve mine. :)

What are the prospects?

Over the years, after many different projects, a consultant will have a good understanding of different types of software projects, technologies and also will have met a lot of people working for many different companies. This gives the consultant a good variety of options for the future. 

In your experience are you aware of any differences your role has between industries/sectors?

In terms of clients we work for, yes, there are huge differences. Working for clients where software is not their core business (a big supermarket chain, a media company, etc) is totally different from working for clients where software is their core business or is an extremely important part of it (software products, internet companies, telcos, etc). There’s also the investment banks, financial companies, government, gambling, etc. Each client will demand a different type of service and a different type of expertise. Some clients will ask a consultancy to help creating the process, roles and responsibilities, requirements, definite the technology stack, etc. Full freedom. Other clients, which have a more mature software capability, may ask a consultancy to provide help with a very specific problem. Sometimes they just need some good developers for a limited period of time. They may want someone in there to coach and mentor their own developers. That means, the role can vary quite a lot, depending on the client.

Reflection and The Future

What was it like coming into the industry?

Before joining my first consultancy company, I had worked for two small software houses and I didn’t know much about it. Job was announced in one of the main Brazilian newspapers. 800+ developers applied and just 34 were hired after a month long selection process, 4 phases, including a two week training and group dynamics. My two previous companies combined had in total 10 people. This consultancy company had more than one thousand employees just in Sao Paulo. Many thousands around the world. I was completely lost and a bit scared by the size of the projects. Before I was writing software for local shops and businesses and there I was working in projects for multinational companies and governments. Complete shock. New technologies, new people, massive projects, multiple teams involved, clients speaking different languages, etc. After a few months, I was quite comfortable there and knew that I had made the right choice. I guess that I was very lucky there, firstly because I was assigned to work on a very technical and capable team and secondly because for two and a half years I had the best mentor (my boss) that I could ever have asked for.

However, joining a modern and more agile consultancy company today is totally different, much easier and less traumatic, even when working on big projects for big clients. In general you will be working in smaller teams (even on long term projects) and following agile methodologies. The technologies used are accessible to every one. When I started, back in the 90ies, we couldn’t just go to a website, download certain technologies and install on our PCs. Many of the technologies were proprietary. There’s absolutely nothing to fear about joining a consultancy company today and the world is a much better place now.

Do you have any thoughts on the future of your role/industry?

I think that while there is a need for software development, consultancies and consultants will always have a place. However, the time where consultants were the only or the best specialists in the market are over. Due to the internet, open source projects, social networking, user groups, conferences, shared videos, podcasts, etc, any one, anywhere in the world, has access to the latest technology at the minute it is released. Consultants now, more than ever, need to be at the top of their game. Consultancy companies will need to do their best to get the best professionals in the market in order to survive. They will need, more than ever, to show to clients that they have the best people to work on their projects.

What advice would you give someone entering your industry?

The following advices are valid for any developer starting his or her career, regardless if he wants to be a consultant, the industry or career path he will choose:

- Firstly and most importantly, enjoy and be proud of what you do. Software development is a very cool profession. During your career, there will be many people out there that will be using and benefiting from stuff you created. And this is definitely really cool.

- If you think you studied hard during university, think again. You haven’t even started studying hard yet. Software development IS NOT a 9 to 5 profession. Be prepared to study hard, outside working hours, for the rest of your life.

- Always aim for jobs where you are going to learn more and not earn more. Preferably a place where you could have a good mentor. Be patient because money will come. If you are good and keep learning, you will have a very decent life throughout your career. If you just aim for the money instead of learning, you may get a good money now and be unemployable and out of the market in a few years time.

- Put a lot of effort on learning the fundamentals of software development instead of just learning the latest framework. If you have a good foundation, you will be able to learn any new technology much faster.

- Contribute to open source projects, learn from their code base and take the opportunity to communicate with other contributors.

- Always have a pet project (you may be excused if you are actively contributing to an open source project). Pet projects are great for you to try different technologies, techniques and methodologies. The cool thing about a pet project is that you have the power to do whatever you want, whenever you want, making it quite fun to work on. You don’t need to finish a pet project, in fact, you never will.

- Join you local user groups. The amount of stuff you learn and people you meet is just priceless.

Have you come across anything or anyone that has helped you move forward in the industry?

Definitely the mentor I had early on in my career. He was an exceptional developer and like a father for all of us in the team. He could hit us as hard as he wanted but no one else was allowed to do it. He shielded his team (us) from all the external heat and pressure. Quality was non-negotiable. He was a true believer that “how it’s done is as important as having it done“, phrase that I use as my blog’s subtitle. He believed in the apprenticeship model, although we’ve never used the term. It was there and then that I learned the true values of what today is called software craftsmanship. I worked for him and on that team for two and a half years and by far it was, if not the best, one of the best teams that I've been part of. My mentor gave me everything I needed to become the professional I am today.

Twitter: @sandromancuso
blog: http://craftedsw.blogspot.com
LSCC: http://www.londonswcraft.com

Frustrations and aspirations of a software craftsman

For a while I've been thinking about what makes me like or dislike a project. Having spent a very big part of my career working for consultancy companies, I was exposed to many different environments, industries, team sizes, processes and technologies. There were projects that I absolutely loved, some projects were OK and some were a real pain.
 
There were even a couple of times in my career where I questioned myself if the choice of being a software craftsman and keep walking the long road would be the best thing for my sanity.

What makes me dislike a project?

Well, there are many factors. Here are just a few but is far from being a complete list:

• Bureaucracy is something that can be really frustrating. That includes process for the sake of process, innumerable cycles of approvals, tortuous and long test and deployment cycles, pointless documentation, and all that anti-agile stuff. 
• Old technologies or the "wrong" technology for the job is always demotivating. We love new toys. There is nothing more annoying when the technology stack is imposed on the team. "You must use these tools from Oracle or IBM. But, hey, don't look like that. You have support if you need it."
• Lack of autonomy and credibility. "You are just the developers here. You don't make decisions. You do what you are told to do. There are much smarter people here to worry about the _real_ problems. And by the way, you don't have admin rights to your PC and you can't access a few websites either."
• Uninteresting domain. It's always difficult to find motivation to build a great software if you don't like what the software does or don't really believe in the business idea. 
• Demotivated people. How can we find motivation and have team spirit when your colleagues attitude is: "Oh, I just turn up to work, keep my head down and do what I'm told. If something goes wrong, it's not my fault." 
• Finger pointing and highly competitive environment, where no one plays as a team. This is an environment where everyone wants to be the boss, they are always looking for a scapegoat and the less work they do, the more they delegate, the better. If something goes wrong, it would never be their fault. If something goes well, they take all the credit. 
• Arrogant and unskilled people. Arrogance many times is used as a self-defence mechanism in order to hide the lack of skills a person may have. "I don't need to read any books. I think all these new technologies and methodologies are crap. I've been doing this for years. I know what it is best."
• Software factory concept. "We need to go faster. Let's throw more developers here. Which ones? Doesn't matter. Just hire some monkeys."
• Mortgage-Driven Development
• Project managers that think they are the most important member of the team
• Very deep hierarchy
• You can't help those who don't want to be helped.
• I really could go on forever here.... 

So what is really the problem here?

When I first mentally thought about all these items, I realised that almost all the things that make me dislike a project are related to people. Yes, people. One of the few exceptions are uninteresting domain and old technologies. So even if I make sure that I don't work on projects related to subjects I have no interest and that use the latest technologies, the people involved may make it very frustrating. Have you ever been on a project where you thought that the project had everything to be a great project but for some reason it was a pain?

After this analysis, things were not looking good, so I decided to look at all the projects I really enjoyed. It was when I realised that in some of them we didn't use the latest technologies and in a few of them I was not exactly passionate about the domain either. One or two were even quite bureaucratic. So, why did I enjoy them? What was in there that made me put them among the projects I liked the most? Once again, the answer was "people".

The good projects and what I always would like to find

My favourite projects had quite a few things in common but the most important ones were passion, craftsmanship, friendship and trust.

Passion

It's not because you like something that you are going to be good at it. However, to be really good at something, you must have a passion for it.

The best people for a job are the ones that love the job. This in the essential quality that drives people to be successful in whatever they decide to do. A passionate person will do whatever is in his or her power to keep acquiring skills and do the best they can. Passionate people bring innovation, they question what they are doing, they want to contribute, they want to get involved, they want to learn. They want to succeed. Passionate people CARE.

Craftsmanship

The only way to go fast is to go well - Uncle Bob Martin

In all my favourite projects, the focus on quality and the willingness to see users satisfied and using the system was always a big thing. The whole team was focused in delivering the best project we could, taking into consideration all the constraints we were under. It was clear to all of us that to be successful we had to be pragmatic. We also always believed that how it is done is as important as having it done.
Software craftsmen use the right tools for the job, are skilful, are pragmatic, care about the quality of their work, care about their reputation and want to delight every single one of their customers and users. Software craftsmen CARE.

Friendship

So far I've been talking about passionate people and care. However we know that people have different opinions and there are many ways to do the same thing. Now imagine a room full of very passionate people that don't really get along. In the best scenario, there will be some memorable arguments. In the worst, they will kill each other.

Friendship is the answer to that. The importance of social events for a team is enormous, even if it is just a few drinks once or twice a month. Like it or not, we spend more time with our colleagues than with our own family, so it is important that we have the friendliest environment possible. Having lunch together at least a few times per week is another thing that can help improve this friendship. Team members need some time together where they are not always just talking about work. Team members need time to know each other. 

Among friends people feel comfortable to speak their minds. Working with friends helps to improve the quality of the discussions and no one is worried to expose ignorance or give suggestions. Friends help each other, friends learn from each other, friends CARE for each other. 

Trust

Once people can demonstrate their commitment and passion, can demonstrate competence, willingness to learn and contribute and are able to deliver the best software possible, trust is easily established. With trust you gain autonomy and are free to decide what it is best for the project and to do your job well. With trust you can be more effective when delegating or sharing tasks. With trust we can remove all the bureaucracy that impedes that we do our job efficiently.   

Aspirations

I just wish I could find the things I described above in all projects. Companies should aim to hire passionate and skilful people. People that can contribute to their projects and organisation. People that are willing to share and learn.

Unfortunately not always we will find all that. In those cases, the only thing we can do is to try our best to change the environment around us. We can try motivate people and share our passion. We can be nice to everyone, respect our colleagues and promote an environment where everyone feels comfortable to ask for help, to help each other and share knowledge.

With great people we can overcome any obstacle and have an environment where every morning, when we wake up, we would think: "Yay! Today I'll have another great day at work."