New Blog @ codurance.com/blog

Hi all,

Quite a few people asked me if I had stopped blogging. The answer is NO. In November 2013, Mashooq Badar and I started Codurance and since then I only blogged on our company website.

Please visit http://codurance.com/blog for my latest blogs.

Apologies for taking more than one year to add this message here.

Cheers
Sandro

Starting a Software Craftsmanship Community

Two people that meet regularly are already a community. The first thing to know if you want to start a community is that it is much simpler than you think. You just need another person that is keen to meet regularly and a public space like a coffee shop or a quiet pub. That's it. You already have your community going. Everything else that follows in this blog post are just things that helped us to grow the London Software Craftsmanship Community (LSCC) from two people to the largest software craftsmanship community in the world, but they are not essential to have a great and healthy community.

Community website

This is an important thing. Don't try to run your community from a mailing list; make sure you have a proper website. One of the points of being part of a community is to know who else is in the community. As a member, I want to go to the website and see when the next events are. I want to be notified in advance about the events, and be reminded about the events one or two days before they happen. I want to know who else is going to the event. I want to place myself on a waiting list in case the event is full. I want to tell others that I'm going to the event. I want to see the profile of the people attending the event so I can have more focused conversations during or after the event. I want to have a mailing list where I can ask questions and engage in discussions.

The good news is that there are a few websites out there that, for an annual subscription, you can have all that. We decided to use Meetup.com (used by the majority of the Software Craftsmanship communities around the world) and I highly recommend it. Don't waste your time trying to create your own website, at least not in the first year or so. Focus only on the quality and content of your meetings. Don't waste your time doing things that are already done for you. Websites like Meetup.com will take care of all the admin work for you: controlling registrations, email, waiting list, reminders, etc. (In case you are wondering, I don't get any commission from Meetup.com, however, now that I thought about it, I probably should.:))

Your first meeting

Some communities started with a big first meeting. Others started quite small and without much noise. As you probably guessed, there is no right or wrong. In London, we decided to give a talk on Software Craftsmanship. The London Java Community (LJC) announced our talk to their members. SkillsMatter, the training centre where the talk happened, also announced on their website and for our huge surprise, 120 people showed up. The Paris Software Craftsmanship Community (PSCC) invited me to give a talk for their kick off meeting and they had more than 50 people.

It helps if you can have a big first meeting; potentially inviting someone that you know that can give a good talk on Software Craftsmanship but that is not really necessary. Many other communities started really small and now they are doing really well. It's important that we understand that a Software Craftsmanship community is not about numbers or well-known people. A Software Craftsmanship community is about passionate software developers sharing and learning from each other, regardless if there are 5 or 1000 of them.

Start with simple meetings

The simpler the meetings are, the better. When starting a Software Craftsmanship community, chances are that you won't have much help. We are all busy people, with families, work, and many other activities. If we make the job of organising a community a burden, we will end up giving up. To minimise the chances for that to happen, make sure that, at least in the beginning, your meetings are extremely simple to organise.

For almost one year, we just had one meeting a month: The Software Craftsmanship Roundtable. This meeting was limited to 30 people. Back then, I was working for [Valtech](http://www.valtech.co.uk/) and they were kind enough to let me use their office space from 6:30pm to 9pm. They also paid for pizzas and soft drinks.

The Software Craftsmanship Roundtable

The Software Craftsmanship Roundtable is a very easy meeting to organise. At the start of the meeting, we ask all attendees if they want to give a lightning talk (five minutes for the presentation, two minutes for questions) or/and propose a topic for a group discussion. Attendees write their lighting talks or topics for discussion in a Post-It and stick them in a white board. While we are eating some pizza, people present their lighting talks. The facilitator, normally the community organiser, keeps track of time. Once all the lightning talks are finished (if any), we ask each person that proposed a topic, in turn, to give a 30 seconds explanation of what they want to discuss. Once all topics are explained, we take a vote. Every one raises their hands for the topics that they want to discuss (they can vote for more than one topic) and the most voted topic gets discussed. Sometimes we discuss a single topic in one night; sometimes we discuss three or four. Depending of how people vote, we some times split in two groups and then have a retrospective at the end. Every 30/45 minutes (use your gut feeling) you ask the group if they want to keep discussing the same topic or pick another one.

The great thing about the round table is that whichever topic is chosen, we know we are going to have a good discussion since the majority of the people voted for it. Topics can vary immensely, from tools and good practices to continuous deployment, from design and architecture to interview techniques, from how to convince teammates to adopt TDD to functional languages.

The overhead of organising a roundtable is very small. You just need to make sure you have a venue. If you don't have an office space, do it in a coffee shop or pub. Food is totally optional. If you have it, great, if you don't have it, it's OK. Don't let food sponsorship be the reason for not having a meeting. I'm sure people wouldn't mind to eat before or after the meeting and pay for it themselves. :)

Hands-on session

Hands-on sessions tend to be slightly more difficult to organise. This is the second type of meeting we introduced. Arguably, it should have been the first type of meeting to have but that was what we did.  Hands-on sessions require a venue where people can not only sit down but also have a place to put their laptops. That means that not always a pub or another public place can be a good choice. A proper office space is always a better option.

Over the years we had many different types of hands-on session. Some of them required a lot of preparation; some of them were dead simple to organise. We also had a mix of very experienced and not so experienced people running the sessions. They were all great.

For the first few hands-on sessions, keep it simple. Choose a kata, do it yourself a few times, and don't tell anyone which kata you've chosen. All attendees need to bring their own laptops with their favourite languages and testing frameworks installed. Have all attendees in a room and ask them to pair up with someone; preferably someone that they don't know or they never paired before. At least one person in the pair needs to have a laptop configured with their favourite language and testing framework. For people starting with TDD, it helps if they pair with people a bit more experienced. Give them time, between one and two hours, with a few breaks (5 minutes) for sharing, and towards the end, run a retrospective. While the pairs are working on the kata, walk around, observe what the pairs are doing, and help the pairs that are stuck. In the final retrospective, ask the pairs which approach they used, what they've learn, which technologies they used, etc. As you know what the majority of the pairs have done, you can poke people to share what they've done. :)

There are many different types of things you can do in a hands-on session but I'll leave them to another post.

Rhythm 

A community needs to have a rhythm. A heartbeat. For example, our Software Craftsmanship Roundtable meetings are always on the second Tuesday of the month, at the same time, in the same place. Our hands-on sessions are always on the last Wednesday of the month. Code & Coffees are always every other Tuesday in the same coffee shop.

It's important that members of the community know when and where the meetings are going to happen so they can organise themselves to be there. A good rhythm keeps the community alive. When choosing a venue, make sure you can run meetings there every month. Once you run the first meeting in a new venue, it is much easier to run subsequent ones. Finding venues is not always easy, so make sure you don't need to do it every month.

In case you can't be in the meeting, make sure someone else can facilitate the meeting. Try never to break the rhythm. Having long gaps between the meetings will make people loose interest.

Make sure that you never have an empty calendar, that means, as soon as the a meeting is finished, you should have the date for the next meeting. People tend not to join a group if there are no meetings scheduled.

Technology Specific vs. Technology Agnostic

There are quite a few Software Craftsmanship communities that have many technology specific meetings. By that, I mean, they have a hands-on session to learn Erlang, or to try a JavaScript framework, or a specific NoSQL database. From what I know, that is working well for them.

In LSCC, we made a commitment from day one that all our meetings would be technology agnostic, and for the past three years we never had a single meeting that was related to a specific technology. There are a few reasons for that:

1. We didn't want to alienate a single member of our community
2. We wanted to focus on the foundations of software development and not in specifics of a language or framework.
1. each developer can choose the language they want to use and who they want to pair with.
3. If the community has monthly meetings and you (as a member) are not interested in the technologies chosen for the meetings, you may not attend the meetings for many months; you may even leave the community.
4. We are fortunate to be in London, a city that has user groups and technical communities for all the main technologies out there. No point in doing what they already do well.

There is no right or wrong here and you need to use your gut feeling. Listen to the people coming to the meetings and try to provide what they want. However, at the very beginning, they probably won't know what they want, so you will need to make the call. Technology agnostic has better chances to attract more developers, from all sorts of backgrounds, which in turn will make the community more diverse. And that is exactly what we always wanted.

Focus on learning, sharing, and having fun

Learning, sharing, and having fun with like-minded people are the most important things in any technical community. As an organiser, make sure that this is your main focus. Growth is not important. Number of people attending the meetings is not important. Whoever comes is the right people. If everyone enjoys the meetings, they will invite their friends, they will tell other developers, so don't focus on growing your community. That will happen as a side effect.

Associates

In LSCC we have the LSCC Associates. Associates are some of the LSCC core members that I invited to help me with the organisation of the community. They are people that come to the vast majority of our meetings, are extremely passionate, talented, and fully understand the Software Craftsmanship ideology. The difference between them and the organiser and the associates is the level of commitment. Associates help whenever they can while the organiser doesn’t really have this option. Some associates will quickly step up and can easily become co-organisers.

As the community grows, you, as an organiser, will need help. Also, there is a limit for what you can do; the community needs variety. The community should never be about the organiser. The community should be about the community. Having a good group of people sharing the organisation should be one of your priorities as an organiser.

We have a separate Google group for the Associates where we discuss everything that is related to the community: ideas for a session, who is going to run it, new companies interested in sponsoring (venue, food, etc), new types of meeting, and many other things. At the beginning you may want to have one or two people to help you. As the community grows, you will need quite a few more.

Associates bring variety and enthusiasm. They keep the community alive and healthy. LSCC would never be so awesome without all the LSCC Associates.

Good neighbours

If there is any other user groups or technical communities in your city, go and speak to them. One of the reasons for the LSCC success is our proximity to other communities. The idea for starting LSCC happened during a conversation I had in one of the London Java Community meetings, and from day one we had the support from the LJC organisers. LJC, back then, already had more than 1000 members and they announced many of our events to the LJC members. We also have a good relationship with other local user groups. In the early days, we made sure that none of our meetings would clash with meetings from other user groups so we all could go to each other meetings.

Protect your community

It's your responsibility, as an organiser, to protect your community. As the number of developers grows, you will see many recruiters and marketeers approaching your community. They will try to use the community's mailing list to advertise their conferences, products, companies, etc. Protect your members and the community mailing list. At LSCC we don't allow any recruiters or people that are not directly involved with a software project to join. For that, membership is by approval only, and I do my best make sure that just people that can contribute to the community will join. We ask every member to write a brief description about who they are. We don't accept memberships where people don't write anything about themselves. A community is not a community if every person in the community doesn't have a profile. Currently I decline between 5 to 10 memberships a month. They are normally recruiters and people trying to advertise their stuff. Sometimes I even Google people and search them on LinkedIn to make sure they are really developers.

There is nothing that the community will gain with people spamming our mailing list. The only thing that will happen is that members will leave the community because of spam.

Have fun

I know that there is a lot in this post and maybe I made it sound far more complicated than it really is. Organising a community is a lot of fun and has many benefits. You will learn more than you can ever imagine and you will also be helping many other developers to learn as well. That, on its own, is more than enough reason to start a community.

Keep it simple and get more people involved. If you follow that, you will have a successful community.

If you start a Software Craftsmanship community, let us know. All of us from LSCC will be more than happy to help. I wish you all the best with your new community and, above all, I hope you have a loads of fun.

You can email me at sandromancuso at gmail dot com or find me on Twitter @sandromancuso

Test Deafness

A few months ago I was talking to a musician friend of mine. He is also Brazilian, same age as me, and moved to the UK on the same year I did. As we were talking about music, I mentioned I like punk rock. He said he likes punk rock as well, but as a musician, he listens to a bit of everything. The conversation was going well until he asked me what my favourite bands were. "Legiao Urbana" is my favourite Brazilian band I said. "Seriously?" he said with a puzzled face. "They are rubbish, man."

As he was a friend (he became an enemy after that stupid comment), I thought to myself: How can I make him suffer for the rest of his life? Luck for him, I remembered that I was not a Brazilian savage anymore. I was British now and I had to act like one: "Oh, are they? Thanks for sharing that with me. Would you mind telling me what your favourite bands are? More tea?" He then named a few bands, including some Forró bands. Shock! Horror!! Blasphemy!!! I really wanted to kill him right there. Fuck the British citizenship. How could a guy, that also liked punk rock music, say that the band I liked was rubbish and then name some stupid Forro bands as his favourite bands?

After quite a long list of swear words pronounced in a very loud tone, I asked him to elaborate. All songs of Legiao Urbana, he said, are played with three or four chords maximum. Their lyrics are great, but they are very poor musicians. The Forro bands are totally the opposite. The lyrics suck but no one cares. Their are great musicians that focus on creating music for people to dance.

That conversation made me realise something really important. If you are a person like me, good music is related to good and strong lyrics. For a musician like my friend, good music is about the techniques used by other musicians when playing their instruments, regardless of the lyrics. For a person that likes to sing, she may appreciate opera, even if she doesn't have a clue about what the words mean.

But what does it have to do with tests?

You cannot expect to produce quality code just by listening to your test. If you don't know what good code looks like, you are pretty much test deaf. Musicians have a trained ear to listen to each instrument individually while the music is playing. They can also imagine, quite precisely, how different instruments could be combined to create new music.

Walking around asking other developers to listen to their tests, as if this advice alone would make them produce quality code immediately, doesn't work. It may make us look smart (or at least feel smart) but it does not really help the other developers receiving the advice. It's too vague.

If we want to produce quality code, we should study the concepts and techniques that lead to it: Domain Driven Design, Clean Code, SOLID principles, design patterns, coupling, cohesion, different programming languages and paradigms, architecture, just to name a few. Once we get a good understanding of all these things, we will have an implied knowledge about what constitutes good code. This implied knowledge is what may cure our test deafness, allowing us to listen to our tests in order to produce quality code. 

My birthday wish list

Today is my birthday. Yay! And since today is all about me, I will choose what I want as a present. As a developer, here's what I want:

1. I want developers to be empowered to do whatever they need to do to satisfy the real business needs and delight their customers.
2. I want developers to be accountable for the decisions they make, and not by decisions that are made for them. 
3. I don't want to see developers going through endless meetings trying to prove why they shouldn't use the technologies or architecture defined by an ivory tower architect. In fact, ivory-tower architects should be an extinct species (visit your nearest National History Museum if you want to see one). 
4. I want developers to know the truth. If a decision was made because of political reasons, please tell us that that is the case. We will still be unhappy but at least we will find it easier to digest. 
5. Although we are happy to justify and explain every technical decision we, the development team, make, we don't want to have any person that is not part of the development team making technical decisions. 
6. I don't want to see developers working with technical people that think that their role is to define the architecture of a project. I want developers to work with people that focus on delivering the simplest solution to a problem; satisfying functional and non-functional requirements. If we can achieve that without writing software, even better. And no, that doesn't mean quick and dirty.
7. In case we need to write software to solve a business problem, I want developers to craft solutions in a way that changes are embraced and the business will never be slowed down by technical incompetence.
8. I want developers that can build applications that will provide a good return on investment to the business. I don't want to see applications being decommissioned after a few years because they became a pile of crap; the maintenance cost is so high that it is cheaper to rewrite. 
9. I want to work with developers that are passionate and care about what they do. Every single day I want to speak to my colleagues and learn something new from them, the same way they should learn something new from me. 
10. I want to work with people (not just developers) that embrace changes and provide business agility. I don't want to keep embarrassing myself every time I need to tell stakeholders that the small change they want will take a few months to be done (due to the crap application/architecture/process we have). 
11. I want to work in a place where we can choose the best technology or tool for the job; without being told that we cannot do that because our people don't have the skills. "They need to be trained." (sigh)
12. Instead of being told that we need to build a new feature or even an entire new application using specific technologies, we would appreciate if we were just given the business requirements. I would love to see that developers are trusted to provide the best solution for the problem.
13. I would like to see the people behind all the bureaucracy and stupid processes being blamed by the damage they are causing. 
14. I would like to see all the 9-to-5 employees being replaced by just a few passionate, talented, well-paid and empowered professionals. 
15. I wish all companies recognised software development as a full career and not as just the initial steps for management. Yes, this is stupid.
16. I wish that every organisation paying for a software project understood the principles of Software Craftsmanship and the importance to have real software craftsmen working on their projects. 

You may be thinking that I'm not being reasonable. After all, my birthday is almost over and of course no one will be able to give me all the things I want today, as a birthday present. You are right. I agree. And because of that, I won't add all the many other wishes I have to the list above. I could go on forever. 

The good news for me is that I'm really smart and I always have a plan B. Although I'm a disappointed that I won't get all these things for my birthday this year, my plan is to be a good boy for the rest of the year and maybe Santa will bring me some of these for Xmas. 

Screencast: Testing and Refactoring Legacy Code

In this screencast I take a small piece of legacy Java code that contains the most common problems found in much larger legacy code bases. The objective is to first write tests to understand what the code does and then refactor it to make it better. The code contains Singletons, static calls and behaviour that does not belong there. It also has some design issues.

As an advice, I always recommend that we should never "touch" the production code as we retrofit tests, that means, we should not change it typing into the production file. However, as normally legacy code is not written with testing in mind, sometimes we need to change the production code in order to write tests for it. I address this scenario explaining how we can do that in a very safe way.

A common question when developers want to make legacy code better is "Where do we start?" I also address that explaining the how the approaches for testing and refactoring legacy code are the opposite from each other.

Besides a few other things, I also cover the use of code coverage tools to help us testing the code, how often we should be committing, how to fix a possible problem with the design in very small steps and how to stay in the green while refactoring our code.

Last but not least, I show how our tests and production code be easily written in a way that it captures the business requirements.

Although it is a bit long, I hope you all enjoy it.

There are two minor things I forgot to do while doing this exercise. Let me know if you spot it. :)

The Wrong Notion of Time

No one wakes up in the morning and say "Today I'm gonna screw up. Today I'm gonna piss my boss and all my team mates off writing the worst code I could possibly write". Well, there are always exceptions but normally no one does that. If that is the case, how come Agile projects are now failing? How come do we still have the same old problems?

A Technical Debt Story

Some time ago I was in this project and one of the developers chose to work on a brand new feature. For the implementation of this new feature, we did not need to touch any of our existing code, besides very few things just to wire the new feature into the application. After a day or so, I offered to pair with him. Naturally, since I had just joined him, I asked him to give me an overview of what the feature was about. He promptly explained it to me and I asked him to show me where he was so we could continue. After he finished showing the code to me I made a few observations since it was not clear to me that his code was reflecting what needed to be done - according to his previous explanation. Basically, the language the he used to explain me the business feature was not in sync with the language that he had used in the code and I could also see some code that was not really necessary for the implementation of that feature. I also noticed that there were no tests. When I asked him about that he said _It is working now and I may need that extra code in the future. Let's add this refactoring you are proposing and the unit test to the technical debt backlog. I need to finish this feature. How crazy is that? That was a brand new feature. We should be reducing technical debt as we went along instead of adding more of it. However, this developer somehow felt that it was OK to do that. At the end of the day we had a technical debt backlog, didn't we? That was supposedly an Agile team with experienced developers but somehow, in their minds, it was OK to have this behaviour. Perhaps one day someone would look at the technical debt and do something about. Possibly. Maybe. Quite unlikely. Nah, will never gonna happen.
 

But we want to do the right thing

But we all want to do the right thing. We do not do these things on purpose. However, over the time, I realised that we developers have a wrong notion of time. We think we need to rush all the time to deliver the tasks we committed to. Pressure will always be part of a software developer life and when there is pressure, we end up cutting corners. We do not do that because we are sloppy. We normally do that because we feel that we need to go faster. We feel that we are doing a good job, proving the business the features they want as fast as we can. The problem is that not always we understand the implications of our own decisions.
 

A busy team with no spare time

I joined this team in a very large organisation. There were loads of pressure and the developers were working really hard. First, it took me days to get my machine set up. The project was a nightmare to configure in our IDEs. We were using Java and I was trying to get my Eclipse to import the project. The project had more than 70 maven projects and modules, with loads of circular dependencies. After a few days, I had my local environment set. The project was using a heavyweight JEE Container, loads of queues and had to integrate with many other internal systems. When pairing with one of the guys (pairing was not common there but I asked them if could pair with them) I noticed that he was playing messages in a queue and looking at logs. I asked him what he was doing and he said that it was not possible to run the system locally so he had to add loads of logs to the code, package, deploy the application in the UAT environment, play XML messages into one of the inbound queues, look at the logs in the console and try to figure out what the application was doing. Apparently he had made a change and the expected message was not arriving in the expected outbound queue. So, after almost twenty minutes of changing the XML message and replaying it into the inbound queue, he had an idea of what the problem could be. So he went back to his local machine, changed a few lines of code, added more logs, changed a few existing ones to print out more information and started building the application again. At this point I asked if he would not write tests for the change and if he had tests for the rest of the application. He then told me that the task he was working on was important so he had to finish it quickly and did not have time to write tests. Then he deployed the new version of the application in UAT again (note that no one else could use the UAT environment while he was doing his tests), played an XML message into the inbound queue and started looking at the logs again. That went on for another two days until the problem was fixed. It turned out that there were some minor logical bugs in the code, things that a unit test would have caught immediately.
 

We don't have time but apparently someone else does

Imagine the situation above. Imagine an application with a few hundred thousand lines. Now imagine a team of seven developers. Now imagine ten of those teams in five different countries working on the same application. Yes, that was the case. There were some system tests (black box tests) but they used to take four hours to run and quite often they were broken so no one really paid attention to them. Can you imagine the amount of time wasted per developer per task or story. Let's not forget the QA team because apparently testers have all the time in the world. They had to manually test the entire system for every single change in the system. Every new feature added to the system was, of course, making the system bigger causing the system tests to be even slower and QA cycles even longer. Debug time was also getting bigger since each developer was adding more code that all the others would need to debug to understand how things work. Now thing about all the time wasted here, every day, every week, every month. This is all because we developers do not have time.

Dedicated Quality Assurance teams are an anti-pattern. Testers should find nothing, zero, nada. Every time a tester finds a bug, we developers should feel bad about it. Every bug found in production is an indication of something that we have not done. Some bugs are related to bad requirements but even then we should have done something about that. Maybe we should have helped our BAs or product owners to clarify them. By no means I am saying that we should not have testers. They can be extremely valuable to explore our applications in unexpected ways that just a human could do. They should not waste their time executing test plans that could be automated by the development team.

Business want the features as soon as possible and we feel that it is our obligation to satisfy them - and it is. However, business people look at the system as a whole and so should we. They look at everything and not just the story we are working on. It is our job to remove (automate) all the repetitive work. I still remember, back in the 90ies, when debugging skills was a big criteria in technical interviews. Those days are gone. Although it is important to have debugging skills, we should be unpleasantly surprised whenever we need to resort to it and when it occurs, we need to immediately address that, writing tests and refactoring our code so we never need to do that again. 

Using time wisely

Our clients and/or employers are interested in software that satisfy their needs, that works as specified and is easy to change whenever they change their minds. It is our job to provide that to them. The way we go about satisfying their expectations, normally, it is up to us. Although they may mention things like automated testing and Agile methodologies, what they really want is a good value for their money when it comes to the investment that they are making in a software project. We need to use our (their) time wisely, automating whatever we can - being tests or deployment procedures - instead of thinking that we may not have time to do it. We can always quantify how much time we are spending in repetitive tasks and even get to the extend of showing them how much time is being spent over a period of time in those activities. Before implementing any new feature or task, we should spend some time preparing our system to accept the changes in a nice way, so we can just _slide that in_ with no friction, and making sure that whatever we write can be easily tested and deployed. When estimating our work, we should always take this into account as **part of the time** that will take us to do it instead of having the false impression that we will be going faster if we treat them as a separate task since, chances are, they may never get done and the whole time will be slowed down because of that. The less time we waste manually testing (or waiting for a long automated test suite to run), debugging, dealing with a huge amount of technical debt, trying to get your IDE to work nicely with your fucked up project structure, or fighting to deploy the application, the more time we have to look after the quality of our application and make our clients happy.  

Note: The teams I mentioned above after a lot of hard work, commitment, support from management and a significant amount of investment (time and money) managed to turn things around and are now among the best teams in the organisation. Some of the teams managed to replace (re-write) an unreliable in-house test suite that used to take over three hours to run with a far more reliable one that takes around 20 minutes. One of the teams is very close to achieve a "one-button" deployment and has an extensive test suite with tests ranging from unit to system (black box) that run in minutes and with code coverage close to 100%.

Testing legacy code with Golden Master

As a warm up for SCNA, the Chicago Software Craftsmanship Community ran a hands-on coding session where developers, working in pairs, should test and refactor some legacy code. For that they used the Gilded Rose kata. You can find links to versions in java, C# and ruby here and for clojure here.

We ran the same session for the London Software Craftsmanship Community (LSCC) early this year and back then I decided to write my tests BDD-style (I used JBehave for that). You can check my solution here.

This time, instead of writing unit tests or BDD / Spec By Example to test every branch of that horrible code, I decided to solve it using a test style called Golden Master.

The Golden Master approach

Before making any change to the production code, do the following:

1. Create X number of random inputs, always using the same random seed, so you can generate always the same set over and over again. You will probably want a few thousand random inputs.
2. Bombard the class or system under test with these random inputs.
3. Capture the outputs for each individual random input

When you run it for the first time, record the outputs in a file (or database, etc). From then on, you can start changing your code, run the test and compare the execution output with the original output data you recorded. If they match, keep refactoring, otherwise, revert back your change and you should be back to green.

Approval Tests

An easy way to do Golden Master testing in Java (also available to C# and Ruby) is to use Approval Tests. It does all the file handling for you, storing and comparing it. Here is an example:


For those not familiar with the kata, after passing a list of items to the GildedRose class, it will iterate through them and according to many different rules, it will change their "sellIn" and "quality" attributes.

I've made a small change in the Item class, adding a automatically generated toString() method to it:
The first time the test method is executed, the line:

Approvals.verify(getStringRepresentationFor(items));

will generate a text file, in the same folder where the test class is, called: GildedRoseTest.should_generate_update_quality_output.received.txt. That mean, ..received.txt

ApprovalTests then will display the following message in the console:

To approve run : mv /Users/sandromancuso/development/projects/java/gildedrose_goldemaster/./src/test/java/org/craftedsw/gildedrose/GildedRoseTest.should_generate_update_quality_output.received.txt /Users/sandromancuso/development/projects/java/gildedrose_goldemaster/./src/test/java/org/craftedsw/gildedrose/GildedRoseTest.should_generate_update_quality_output.approved.txt

Basically, after inspecting the file, if we are happy, we just need to change the .received with .approved to approve the output. Once this is done, every time we run the test, ApprovalTests will compare the output with the approved file.

Here is an example of how the file looks like:

Item [name=Aged Brie, sellIn=-23, quality=-44]
Item [name=Elixir of the Mongoose, sellIn=-9, quality=45]
Item [name=Conjured Mana Cake, sellIn=-28, quality=1]
Item [name=Aged Brie, sellIn=10, quality=-2]
Item [name=+5 Dexterity Vest, sellIn=31, quality=5]

Now you are ready to rip the GildedRose horrible code apart. Just make sure you run the tests every time you make a change. :)
Infinitest

If you are using Eclipse or IntelliJ, you can also use Infinitest. It automatically runs your tests every time you save a production or test class. It is smart enough to run just the relevant tests and not the entire test suite.  In Eclipse, it displays a bar at the bottom-left corner that can be red, green or yellow (in case there are compilation errors and the tests can't be run).

With this, approach, refactoring legacy code becomes a piece of cake. You make a change, save it, look at the bar at the bottom of the screen. If it is green, keep refactoring, if it is red, just hit CTRL-Z and you are back in the green. Wonderful. :)

Thanks

Thanks to Robert Taylor and Balint Pato for showing me this approach for the first time in one of the LSCC meetings early this year. It was fun to finally do it myself.

The best approach to software development

Today, talking about doing a big design up-front (BDUF) sounds a bit ridiculous, right? Who would do that? That's not craftsmanship, is it?

However, in the past, that would be considered the norm. Writing requirement documents, drawing architectural and very low level detail diagrams was the right thing to do. Well, that’s what very smart guys proposed on the 1968 NATO Software Engineering Conference and it worked for NASA and the US Department of Defense. I’m sure they know what they are doing and if it works for them, it will definitely work for our small CRUD application or one page website. And then it happened. It became a religion and the majority of projects in the following decades were developed like that.

No, but not nowadays. We've learned the lesson, right? We wouldn't make this mistake again.  

After watching a few talks in conferences and InfoQ, we understood that this is not a good thing. We’ve also read in some books that we should do TDD. The design should emerge from tests.

And of course, we should adopt an Agile methodology as well. Let’s adopt Scrum to start with. There are many books about it, certifications and even entire conferences dedicated to it. Of course we should adopt TDD an Scrum because that’s the best approach to manage and develop and software.

Oh, but what about all this lean stuff? Eliminate waste, limit work in progress, system thinking, theory of constrains, Kanban. I heard it worked really well for Toyota so we should definitely do that as well. Why? Jesus, you just don’t get it. Of course that’s best approach to manage and develop software.

God, how could I forget? I was also told that I really should speak to my customer. We should really discuss the requirements so we can have a better understanding of what we should build. Are you using BDD? No!!! Wow! How can you develop software without that? Why should you use it? The answer is simple. That’s the best approach to manage and develop software. Don’t tell me that you thought that BDD was a testing tool. You are so yesterday. That was version one. BDD version three is all about communication. It's about software development done right. Yes, I know it sounds alien but apparently we are supposed to speak to people. God, how on Earth we haven’t thought about that before? How did we develop software all these years? If you don’t use BDD, you are just doing it wrong. Why? Because that’s the best approach to manage and develop software. Duh!

Outside-In TDD, Inside-Out TDD, ATDD, Classic TDD, London School TDD? Really? Are you still discussing that? Don’t tell me that we you are still writing tests. What? Why are you wasting time writing unit tests? It doesn’t make sense any more. You should spike and stabilize. What if you don’t know what you are doing or where you are going? What if you just want to explore your options? Why are you writing tests? Oh, I get it. You were told that this was the best approach to manage and develop software. Nah, forget that. Unit tests are for losers. We write small services and just monitor them. If they are wrong, we just throw them away and re-write. And THAT is the best way to manage and develop software.

Architecture and design patterns? What??? Who are you? My grandfather? Scrap that. That’s for programmers from the 80ies and 90ies. In the real world we have our design emerging from tests. No, stupid. Not using normal TDD. God, which planet do you live? We use TDD As If You Meant It. We use this technique and PRESTO, the design emerges and evolves nicely throughout the life span of the project regardless of how many developers, teams and design skills. Every one can see code smells, right?

And what about DDD? Domain Driven what? Nah, never heard about it. Hmm.. hold on. I think I heard something about it many years ago, but probably it was not important enough otherwise we would have more people today saying that DDD is the best approach to manage and develop software.

Noooo. No no no no. No, I didn't hear that. Get out of here. Did you just say that you are still using an Object-Oriented language? STATICALLY-TYPED???? No, sorry. This conversation is a waste of my time. It's because of people like you that our industry is shit. The only way for this conversation to get even worse is if you tell me that you still use a relational database. Haven't you heard that functional programming is a MUST DO and just retards and programmers from the 80ies and 90ies use relational databases. Functional languages, NoSQL databases... Repeat with me. Functional languages, NoSQL databases. What a douche bag.

Ah, trying to be a smart ass? Yeah, functional programming appeared long ago in the 50ies and developers in the 60ies and 70ies preferred to use OO instead. But you don't know why, do you? DO YOU? They did use OO because they were a bunch of hippies that didn't take anything seriously. They were that sort of people that went to Woodstock, got high on LSD and had brain damage after that. No, don't take this whole OO stuff seriously. We are finally getting back to reality now. Functional programming and NoSQL databases are definitely the best approach for software development.

Dogmatism, religion, context, curiosity, inquiring mind and pragmatism

Before I conclude, I just want to clarify that by no means I'm criticizing any person or group of people behind some of the methodologies, technologies or techniques mentioned above. These people have done an amazing job thinking, practicing and sharing their own ideas of how software development could be done in a better way and for that we should all be grateful. Our industry if definitely better with their contribution.

My main criticism here is about how the vast majority of developers react to all these things. It is not just because someone, somewhere wrote a book, recorded a video or gave a talk in a conference about something that it will make that thing right, in all contexts. Quite often, we fail to question things just because the people promoting it are relatively well known. We fail to understand the context where a methodology, technology or technique should be best suitable for. We fail, quite often, to use our own judgement because of the fear to be ridiculed by our colleagues. We should stop being dogmatic and religious about things. This just leads to stupid decisions. Doing things for the sake of doing or because someone else said so is just plain wrong and stupid. 

Being a good developer means to be inquisitive, curious and pragmatic. Never religious. Never dogmatic. Curious means that we should be eager to learn about all the things mentioned above and many many more. Inquisitive means that we should investigate and question all the things we learn. Pragmatic means that we should choose the right tools, being technologies, methodologies or techniques, for the job.

Context matters. 

Whenever you see people saying that we should or shouldn't do something, ask them why. Ask them about the context where they tried to do (or not to do) what they are saying. 

Software development is not the same thing of producing cars. Once the car is ready, you don't go back to the manufacturer and ask them to add another wheel or put the engine somewhere else. Developing software for expensive hardware is not the same thing as developing a simple web application with two pages. Hardware has an specification that you need to code against. Quite often, you don't even have access to the hardware because it is just not built yet. The cost of a bug in production is not the same for all applications. The cost of a few bugs in a social networking or cooking website can be very different from the cost of a few bugs in a trading or financial system processing millions of transactions per day. Working with a small team, every one co-located and with easy access to customers is very different from working on a project with 10+ teams spread in 5 countries and different timezones. 

Read and learn as much as you can. However, don't assume that everything you read or watch applies in every context. Make informed decisions and trust your instincts.

The bad news is that there is no best approach to software development. Maximum we could say is that there are certain technologies, methodologies and techniques that are more suitable to a specific context.

In case you are really trying to find the best approach to software development in general, I hope you don't get too frustrated and good luck with that. If you ever find it, please let us know. It's always interesting to know about different approaches. Maybe unicorns really exist. Who knows?



This post was inspired by conversations during many London Software Craftsmanship Community (LSCC) Round-table meetings, conversations during SoCraTes 2012 and also conversations with colleagues at work. Thanks everyone for that.

Test-driving Builders with Mockito and Hamcrest

A lot of people asked me in the past if I test getters and setters (properties, attributes, etc). They also asked me if I test my builders. The answer, in my case is it depends.

When working with legacy code, I wouldn’t bother to test data structures, that means, objects with just getters and setters, maps, lists, etc. One of the reasons is that I never mock them. I use them as they are when testing the classes that uses them.

For builders, when they are used just by test classes, I also don’t unit test them since they are used as “helpers” in many other tests. If they have a bug, the tests will fail.

In summary, if these data structures and builders already exist, I wouldn’t bother retrofitting test for them.

But now let’s talk about doing TDD and assume you need a new object with getters and setters. In this case, yes, I would write tests for the getters and setters since I need to justify their existence writing my tests first.

In order to have a rich domain model, I normally tend to have business logic associated with the data and have a richer domain model. Let’s see the following example.

In the real life, I would be writing on test at a time, making them pass and refactor. For this post, I’ll just give you the full classes for clarity’s sake. First let’s write the tests:

Now the implementation:


This case is interesting since the Trade object has one property called inboundMessage with respective getters and setters and also uses a collaborator (reportabilityDecision, injected via setter) in its isReportable business method.

A common approach that I’ve seen many times to “test” the setReportabilityDecision method is to introduce a getReportabilityDecision method returning the reportabilityDecision (collaborator) object.

This is definitely the wrong approach. Our objective should be to test how the collaborator is used, that means, if it is invoked with the right parameters and if whatever it returns (if it returns anything) is used. Introducing a getter in this case does not make sense since it does not guarantee that the object, after had the collaborator injected via setter, is interacting with the collaborator as we intended.

As an aside, when we write tests that are about how collaborators are going to be used, defining their interface, is when we are using TDD as a design tool and not just simply as a testing tool. I’ll cover that in a future blog post.

OK, now imagine that this trade object can be created in different ways, that means, with different reportability decisions. We also would like to make our code more readable and we decide to write a builder for the Trade object. Let’s also assume, in this case, that we want the builder to be used in the production and test code as well. 

In this case, we want to test drive our builder. 

Here is an example that I normally find when developers are test-driving a builder implementation.

Now let’s have a look at these tests.  

The good news is, the tests were written in the way developers want to read them. That also means that they were “designing” the TradeBuilder public interface (public methods). 

The bad news is how they are testing it.

If you look closer, the tests for the builder are almost identical to the tests in the TradeTest class. 

You may say that it is OK since the builder is creating the object and the tests should be similar. The only different is that in the TradeTest we instantiate the object by hand and in the TradeBuilderTest we use the builder to instantiate it, but the assertions should be the same, right? 

For me, firstly we have duplication. Secondly, the TradeBuilderTest doesn’t show it’s real intent. 

After many refactorings and exploring different ideas, while pair-programming with one of the guys in my team we came up with this approach:

So now, the TradeBuilderTest express what is expected from the TradeBuilder, that means, the side effect when the build method is called. We want it to create a Trade and set its attributes. There are no duplications with the TradeTest. It is left to the TradeTest to guarantee the correct behavior of the Trade object.

For completion’s sake, here is the final TradeBuider class:



The combination of Mockito and Hamcrest is extremely powerful, allowing us to write better and more readable tests.

Testing multiple properties with single assertion

Every time I was trying to test an object's properties I was neither satisfied writing very verbose tests nor in using some of the out of the box hamcrest matchers. Although using the matchers was a big help, I never managed to make them read the way I wanted.

Another thing that was very important to me, I wanted to have a single assertion per method and a very descriptive description if the test did not pass.

I've decided to write my own matcher and hopefully it will be useful to other people. So, that's what I've done:

BeanMatcher

Hamcrest matcher to match multiple attributes of an object within a single assertion.

How to use it

NOTE: Make sure you are using org.hamcrest.MatcherAssert.assertThat instead of the JUnit one.

If you run this test, you will get a message like

Now, change the age check to

   property("age", greaterThan(60)) 

And you should get:

Testing object graphs

You can also do this

I use a combination of two matchers to do that:
- BeanMatcher: Provides the "has" method responsible to group all the property matchers.
- BeanPropertyMatcher: Provides the "property" method.

I expect to make more changes to them, so for the most up-to-date version, please check BeanMatcher on my github account.

Enjoy!!!

Extract, Inject, Kill: Breaking hierarchies (part 3)

In part one I explain the main idea behind this approach and in part two I start this example. Please read parts one and two before reading this post

Although the main ideas of Extract, Inject, Kill is already expressed, it's good to finish the exercise just for completion's sake. Here is where we stopped:

Let's have a look at the VoucherPricingService, that now is the only concrete class at the bottom of our hierarchy. 

Note that it uses the VoucherService class to calculate the voucher value.

Before anything, let's write some tests to VoucherPricingService.java

Once thing to notice is that the User parameter is not used for anything. So let's remove it.

Now it is time to user the Extract, Inject, Kill on the VoucherPricingService. Let's Extract the content of the VoucherPricingService.applyAdditionalDiscounts(double, String) method and add it to a class called VoucherDiscountCalculation. Let's call the method calculateVoucherDiscount(). Of course, let's do that writing our tests first. They need to test exactly the same things that are tested on VoucherPricingService.applyAdditionalDiscounts(double, String). We also take the opportunity to pass the VoucherService object into the constructor of VoucherDiscountCalculation.

If you noticed, when doing the extraction, we took the opportunity to give proper names to our new classes and methods and also to pass their essential dependencies to the constructor instead of using method injection.

 

Let's now change the code in the VoucherPricingService to use the new VoucherDiscountCalculation and see if all the tests still pass.

Cool. All the tests still pass, meaning that we have the same behaviour, but now in the VoucherDiscountCalculation class, and we are ready to move to the Inject stage.

Let's now inject VoucherDiscountCalculation into PricingService, that is the top class in the hierarchy. As always, let's add a test that will test this new collaboration.

And here is the changed PriningService.

Now it is time to kill the VoucherPricingService class and kill the PricingService.applyAdditionalDiscounts(double total, String voucher) template method, since it is not being used anymore. We can also kill the VoucherPricingServiceTest class and fix the PricingServiceTest removing the applyAdditionalDiscounts() method from the testable class.

So now, of course, we don't have a concrete class in our hierarchy anymore, since the VoucherPricingService was the only one. We can now safely promote UserDiscountPricingService to concrete.

That is now how our object graph looks like:

Our hierarchy is another level short. The only thing we need to do now is to apply  Extract, Inject, Kill once again, extracting the logic inside UserDiscountPricingService into another class (e.g. UserDiscountCalculation), inject UserDiscountCalculation into PricingService, finally kill UserDiscountPricingService and the calculateDiscount(User user) template method.  UserDiscountPricingService, 

Since the approach was described before, there is no need to go step by step anymore. Let's have a look at the final result.

Here is the diagram representing where we started:

  After the last Extract, Inject, Kill refactoring, this is what we've got:

The cool thing about the final model pictured above is that now we don't have any abstract classes anymore. All classes and methods are concrete and every single class is independently testable. 


That's how the final PricingService class looks like:


For a full implementation of the final code, please look at https://github.com/sandromancuso/breaking-hierarchies

Note: For this three part blog post I used three different approaches to drawing UML diagrams. By hand, using ArgoUML and Astah community edition. I'm very happy with the latter. 

Extract, Inject, Kill: Breaking hierarchies (part 2)

In part 1 of this post I explain the problems of using the template method in deep class hierarchies and how I went to solve it. Please read it before reading this post.

Here is a more concrete example in how to break deep hierarchies using the Extract, Inject, Kill approach. Imagine the following hierarchy.

Let's start with the StandardPricingService. First, let's write some tests:

Note that I used  a small trick here, extending the StandardPricingService class inside the test class so I could have access to the protected method. We should not use this trick in normal circumstances. Remember that if you feel the need to test protected or private methods, it is because your design is not quite right, that means, there is a domain concept missing in your design. In other words, there is a class crying to come out from the class you are trying to test.

Now, let's do the step one in our Extract, Inject, Kill strategy. Extract the content of the calculateProductPrice() method into another class called StandardPriceCalculation. This can be done automatically using IntelliJ or Eclipse. After a few minor adjusts, that's what we've got.

And the StandardPriceService now looks like this:

All your tests should still pass.

As we create a new class, let's add some tests to it. They should be the same tests we had for the StandardPricingService.

Great, one sibling done. Now let's do the same thing for the BoxingDayPricingService.

Now let's extract the behaviour into another class. Let's call it BoxingDayPricingCalculation.

The new BoxingDayPriceService is now

We now need to add the tests for the new class.

Now both StandardPricingService and BoxingDayPricingService have no implementation of their own. The only thing they do is to delegate the price calculation to StandardPriceCalculation and BoxingDayPriceCalculation respective. Both price calculation classes have the same public method, so now let's extract a PriceCalculation interface and make them both implement it.

Awesome. We are now ready for the Inject part of Extract, Inject, Kill approach. We just need to inject the desired behaviour into the parent (class that defines the template method). The calculateProductPrice() is defined in the PricingService, the class at the very top at the hierarchy. That's where we want to inject the PriceCalculation implementation. Here is the new version:

Note that the template method calculateProductPrice() was removed from the PricingService, since its behaviour is now being injected instead of implemented by sub-classes.

As we are here, let's write some tests for this last change, checking if the PricingService is invoking the PriceCalculation correctly. 

Great. Now we are ready for the last bit of the Extract, Inject, Kill refactoring. Let's kill both StandardPricingService and BoxingDayPricingService child classes. 

The VoucherPricingService, now the deepest class in the hierarchy, can be promoted to concrete class. Let's have another look at the hierarchy:

And that's it. Now it is just to repeat the same steps for VoucherPricingService and UserDiscountPricingService. Extract the implementation of their template methods into classes, inject them into PricingService, and kill the classes.

In doing so, every time you extract a class, try to give them proper names instead of calling them Service. Suggestions could be VoucherDiscountCalculation and PrimeUserDiscountCalculation.

There were a few un-safe steps in the re-factoring described above and I also struggled a little bit to describe exactly how I did it since I was playing quite a lot with the code. Suggestions and ideas are very welcome.

For the final solution, please check the last part of this blog post.

NOTE
If you are not used to use builders in your tests and is asking yourself where the hell aProduct() and aShoppingBasket() come from, check the code in here:

ProductBuilder.java
ShoppingBasketBuilder.java

For more information about the original problem that triggered all this, please read part 1 of this blog post.
In part 3 I finish the exercise, breaking the entire hierarchy. Please have a look at it for the final solution. 

Extract, Inject, Kill: Breaking hierarchies (part 1)

Years ago, before I caught the TDD bug, I used to love the template method pattern. I really thought that it was a great way to have an algorithm with polymorphic parts. The use of inheritance was something that I had no issues with. But yes, that was many years ago.

Over the years, I've been hurt by this "design style". That's the sort of design created by developers that do not TDD.

The situation

Very recently I was working on one part of our legacy code and found a six level deep hierarchy of classes. There were quite a few template methods defined in more than one of the classes. All classes were abstract with the exception of the bottom classes, that just implemented one or two of the template methods. There were just a single public method in the entire hierarchy, right at the very top.

We had to make a change in one of the classes at the bottom. One of the (protected) template method implementations had to be changed.

The problem

How do you test it? Goes without saying that there were zero tests for the hierarchy.

We know that we should never test private or protected methods. A class should "always" be tested from its public interface. We should always write tests that express and test "what" the method does and not "how". That's all well and good. However, in this case, the change needs to be done in a protected method (template method implementation) that is part of the implementation of a public method defined in a class six level up in the hierarchy. To test this method, invoking the public method of its grand grand grand grand parent we will need to understand the entire hierarchy, mock all dependencies, create the appropriate data, configure the mocks to have a well defined behaviour so that we can get this piece of code invoked and then tested.

Worse than that, imagine that this class at the bottom has siblings overriding the same template method. When the siblings need to be changed, the effort to write tests for them will be the same as it was for our original class. We will have loads of duplications and will also need to understand all the code inside all the classes in the hierarchy. The ice in the cake: There are hundreds of lines to be understood in all parent classes.

Breaking the rules

Testing via the public method defined at the very top of the hierarchy has proven not to be worth it. The main reason is that, besides painful, we already knew that the whole design was wrong. When we look at the classes in the hierarchy, they didn't even follow the IS-A rule of inheritance. They inherit from each other so some code could be re-used.

After some time I thought: Screw the rules and this design. I'm gonna just directly test the protected method and then start breaking the hierarchy.

The approach: Extract, Inject, Kill

The overall idea is:

1. Extract all the behaviour from the template method into a class.

2. Inject the new class into the parent class (where the template is defined), replacing the template method invocation with the invocation of the method in the new class.

3. Kill the child class (the one that had the template method implementation).

Repeat these steps until you get rid of the entire hierarchy.

This was done writing the tests first, making the protected template method implementation public.

NOTES

1. This may not be so simple if we have methods calling up the stack in the hierarchy.

2. If the class has siblings, we have to extract all the behaviour from the siblings before we can inject into the parent and kill the siblings.

This probably is too complicate to visualise, so in part 2 of this post I'll be giving a more concrete example.