Friday 10 December 2010

Readable Tests: Separating intent from implementation

Very recently, I was working on a test class like this:

public class AnalyticsExpirationDateManagerTest extends TestCase {

 private static final long ONE_HOUR_TIMEOUT = 1000 * 60 * 60;
 private static final long TWO_HOUR_TIMEOUT = ONE_HOUR_TIMEOUT * 2;
 
 private Map<Parameter, Long> analyticsToTimeout;
 private long defaultTimeout;
 
 private Parameter minTimeoutParam;
 @Mock private CacheKeyImpl<Parameter> cacheKey;

    @Override
    protected void setUp() throws Exception {
     MockitoAnnotations.initMocks(this);
     
     this.minTimeoutParam = new Parameter("minTimeout", "type");
     
     when(cacheKey.getFirstKey()).thenReturn(minTimeoutParam);
     
     this.analyticsToTimeout = new HashMap<Parameter, Long>();
     this.defaultTimeout = 0;
    }
 
 public void
 testGetExpirationDateWhenAnalyticsToTimeoutsAndCacheKeyAreEmpty() {
  AnalyticsExpirationDateManager<Long> manager = 
    new AnalyticsExpirationDateManager<Long>(analyticsToTimeout, defaultTimeout);
  Date date = manager.getExpirationDate(cacheKey, 0L);
  assertNotNull(date);
 }
 
 public void 
 testGetExpirationDateWithMinimunTimeoutOfOneHour() {
  this.analyticsToTimeout.put(this.minTimeoutParam, ONE_HOUR_TIMEOUT);
  Collection<Parameter> cacheKeysWithMinTimeoutParam = new ArrayList<Parameter>();
  cacheKeysWithMinTimeoutParam.add(this.minTimeoutParam);
  when(this.cacheKey.getKeys()).thenReturn(cacheKeysWithMinTimeoutParam);
  
  AnalyticsExpirationDateManager<Long> manager = 
   new AnalyticsExpirationDateManager<Long>(analyticsToTimeout, defaultTimeout);
  Date date = manager.getExpirationDate(cacheKey, 0L);

  assertNotNull(date);
  Calendar expirationDate = Calendar.getInstance();
  expirationDate.setTime(date);
  
  Calendar currentDate = Calendar.getInstance();
  
  // Check if expiration date is one hour ahead current date. 
  int expirationDateHour = expirationDate.get(Calendar.HOUR_OF_DAY);
  int currentDateHour = currentDate.get(Calendar.HOUR_OF_DAY);
  assertTrue(expirationDateHour - currentDateHour == 1);
 }
 
 public void 
 testGetExpirationDateWhenCacheKeyIsNullAndDefaultTimeoutIsOneHour() {
  CacheKeyImpl<Parameter> NULL_CACHEKEY = null;
  AnalyticsExpirationDateManager<Long> manager = 
   new AnalyticsExpirationDateManager<Long>(analyticsToTimeout, ONE_HOUR_TIMEOUT);
  Date date = manager.getExpirationDate(NULL_CACHEKEY, 0L);
  
  assertNotNull(date);
  Calendar expirationDate = Calendar.getInstance();
  expirationDate.setTime(date);
  
  Calendar currentDate = Calendar.getInstance();
  
  // Check if expiration date hour is the same of current date hour.
  // When cache key is null, system date and time is returned and default timeout is not used.
  int expirationDateHour = expirationDate.get(Calendar.HOUR_OF_DAY);
  int currentDateHour = currentDate.get(Calendar.HOUR_OF_DAY);
  assertTrue(expirationDateHour - currentDateHour == 0);
 }
 
 public void 
 testGetExpirationDateWithDefaultTimeout() {
  // Default timeout is used when no time out is specified.
  Collection<Parameter> cacheKeysWithoutTimeoutParam = new ArrayList<Parameter>();
  cacheKeysWithoutTimeoutParam.add(new Parameter("name", "type"));
  when(this.cacheKey.getKeys()).thenReturn(cacheKeysWithoutTimeoutParam);

  AnalyticsExpirationDateManager<Long> manager = 
   new AnalyticsExpirationDateManager<Long>(analyticsToTimeout, ONE_HOUR_TIMEOUT);
  Date date = manager.getExpirationDate(cacheKey, 0L);
  
  assertNotNull(date);
  Calendar expirationDate = Calendar.getInstance();
  expirationDate.setTime(date);
  
  Calendar currentDate = Calendar.getInstance();
  
  // Check if expiration date is one hour ahead current date. 
  int expirationDateHour = expirationDate.get(Calendar.HOUR_OF_DAY);
  int currentDateHour = currentDate.get(Calendar.HOUR_OF_DAY);
  assertTrue(expirationDateHour - currentDateHour == 1);
 }
 
 public void 
 testGetExpirationDateWhenMinTimeoutIsSetAfterCreation() {
  AnalyticsExpirationDateManager<Long> manager = 
   new AnalyticsExpirationDateManager<Long>(analyticsToTimeout, ONE_HOUR_TIMEOUT);
  manager.setExpirationTimeout(this.minTimeoutParam.getName(), TWO_HOUR_TIMEOUT);
  
  Date date = manager.getExpirationDate(cacheKey, 0L);
  
  assertNotNull(date);
  Calendar expirationDate = Calendar.getInstance();
  expirationDate.setTime(date);
  
  Calendar currentDate = Calendar.getInstance();
  
  // Check if expiration date is two hour ahead current date. 
  int expirationDateHour = expirationDate.get(Calendar.HOUR_OF_DAY);
  int currentDateHour = currentDate.get(Calendar.HOUR_OF_DAY);
  assertTrue("Error", expirationDateHour - currentDateHour == 2);
 }
 
}

Quite frightening, isn't it? Very difficult to understand what's going on there.

The class above covers 100% of the class under test and all the tests are valid tests, in terms of what is being tested.

Problems

There are quite a few problems here:
- The intent (what) and implementation (how) are mixed, making the tests very hard to read;
- There is quite a lot of duplication among the test methods;
- There is also a bug in the test methods when comparing dates, trying to figure out how many hours one date is ahead of the other. When running these tests in the middle of the day, they work fine. If running them between 22:00hs and 00:00hs, they break. The reason is that the hour calculation does not take into consideration the day.

Making the tests more readable

Besides testing the software, tests should also be seen as documentation, where business rules are clearly specified. Since the tests here are quite messy, understanding the intention and detecting bugs can be quite difficult.

I've done quite a few refactorings to this code in order to make it more readable, always working in small steps and constantly re-running the tests after each change. I'll try to summarise my steps for clarity and brevity.

1. Fixing the hour calculation bug

One of the first things that I had to do was to fix the hour calculation bug. In order to fix the bug across all test methods, I decided to extract the hour calculation into a separate class, removing all the duplication from the test methods. Using small steps, I took the opportunity to construct this new class called DateComparator (yes, I know I suck naming classes) using some internal Domain Specific Language (DSL) techniques.

public class DateComparator {
 
 private Date origin;
 private Date target;
 private long milliseconds;
 private long unitsOfTime;
 
 private DateComparator(Date origin) {
  this.origin = origin;
 }
 
 public static DateComparator date(Date origin) {
  return new DateComparator(origin);
 }
 
 public DateComparator is(long unitsOfTime) {
  this.unitsOfTime = unitsOfTime;
  return this;
 }
 
 public DateComparator hoursAhead() {
  this.milliseconds = unitsOfTime * 60 * 60 * 1000;
  return this;
 }
 
 public static long hours(int hours) {
  return hoursInMillis(hours);
 }
 
 private static long hoursInMillis(int hours) {
  return hours * 60 * 60 * 1000;
 }
 
 public boolean from(Date date) {
  this.target = date;
  return this.checkDifference();
 }
 
 private boolean checkDifference() {
  return (origin.getTime() - target.getTime() >= this.milliseconds);
 }
}

So now, I can use it to replace the test logic in the test methods.

2. Extracting details into a super class

This step may seem a bit controversial at first, but can be an interesting approach for separating the what from how. The idea is to move tests set up, field declarations, initialisation logic, everything that is related to the test implementation (how) to a super class, leaving the test class just with the test methods (what).

Although this many not be a good OO application of the IS-A rule, I think this is a good compromise in order to achieve better readability in the test class.

NOTE: Logic can be moved to a super class, external classes (helpers, builders, etc) or both.

Here is the super class code:

public abstract class BaseTestForAnalyticsExperationDateManager extends TestCase {

 protected Parameter minTimeoutParam;
 @Mock protected CacheKeyImpl<Parameter> cacheKey;
 protected Date systemDate;
 protected CacheKeyImpl<Parameter> NULL_CACHEKEY = null;
 protected AnalyticsExpirationDateManager<Long> manager;

 @Override
 protected void setUp() throws Exception {
  MockitoAnnotations.initMocks(this);
  this.minTimeoutParam = new Parameter("minTimeout", "type");
  when(cacheKey.getFirstKey()).thenReturn(minTimeoutParam);
  this.systemDate = new Date();
 }

 protected void assertThat(boolean condition) {
  assertTrue(condition);
 }
 
 protected void addMinimunTimeoutToCache() {
  this.configureCacheResponse(this.minTimeoutParam);
 }
 
 protected void doNotIncludeMinimunTimeoutInCache() {
  this.configureCacheResponse(new Parameter("name", "type"));
 }
 
 private void configureCacheResponse(Parameter parameter) {
  Collection<Parameter> cacheKeysWithMinTimeoutParam = new ArrayList<Parameter>();
  cacheKeysWithMinTimeoutParam.add(parameter);
  when(this.cacheKey.getKeys()).thenReturn(cacheKeysWithMinTimeoutParam);
 }
}

3. Move creation and configuration of the object under test to a builder class

The construction and configuration of the AnalyticsExpirationDateManager is quite verbose and adds a lot of noise to the test. Once again I'll be using a builder class in order to make the code more readable and segregate responsibilities. Here is the builder class:

public class AnalyticsExpirationDateManagerBuilder {
 
 protected static final long ONE_HOUR = 1000 * 60 * 60;

 protected Parameter minTimeoutParam;
 private AnalyticsExpirationDateManager<Long> manager;
 private Map<Parameter, Long> analyticsToTimeouts = new HashMap<Parameter, Long>();
 protected long defaultTimeout = 0;
 private Long expirationTimeout;
 private Long minimunTimeout;

 private AnalyticsExpirationDateManagerBuilder() {
  this.minTimeoutParam = new Parameter("minTimeout", "type");
 }
 
 public static AnalyticsExpirationDateManagerBuilder aExpirationDateManager() {
  return new AnalyticsExpirationDateManagerBuilder();
 }
 
 public static long hours(int quantity) {
  return quantity * ONE_HOUR;
 }
 
 public AnalyticsExpirationDateManagerBuilder withDefaultTimeout(long milliseconds) {
  this.defaultTimeout = milliseconds;
  return this;
 }
 
 public AnalyticsExpirationDateManagerBuilder withExpirationTimeout(long milliseconds) {
  this.expirationTimeout = new Long(milliseconds);
  return this;
 }
 
 public AnalyticsExpirationDateManagerBuilder withMinimunTimeout(long milliseconds) {
  this.minimunTimeout = new Long(milliseconds);
  return this;
 }
 
 public AnalyticsExpirationDateManager<Long> build() {
  if (this.minimunTimeout != null) {
   analyticsToTimeouts.put(minTimeoutParam, minimunTimeout);
  }
  this.manager = new AnalyticsExpirationDateManager(analyticsToTimeouts, defaultTimeout);
  if (this.expirationTimeout != null) {
   this.manager.setExpirationTimeout(minTimeoutParam.getName(), expirationTimeout);
  }
  return this.manager;
 }

}

The final version of the test class

After many small steps, that's how the test class looks like. I took the opportunity to rename the test methods as well.

import static com.mycompany.AnalyticsExpirationDateManagerBuilder.*;
import static com.mycompany.DateComparator.*;

public class AnalyticsExpirationDateManagerTest extends BaseTestForAnalyticsExperationDateManager {

 public void
 testExpirationTimeWithJustDefaultValues() {
  manager = aExpirationDateManager().build();
  Date cacheExpiration = manager.getExpirationDate(cacheKey, 0L);
  assertThat(dateOf(cacheExpiration).is(0).hoursAhead().from(systemDate));
 }
 
 public void 
 testExpirationTimeWithMinimunTimeoutOfOneHour() {
     addMinimunTimeoutToCache();  
  manager = aExpirationDateManager()
      .withMinimunTimeout(hours(1))
      .build();
  Date cacheExpiration = manager.getExpirationDate(cacheKey, 0L);
  assertThat(dateOf(cacheExpiration).is(1).hoursAhead().from(systemDate));
 }
 
 public void 
 testExpirationTimeWhenCacheKeyIsNullAndDefaultTimeoutIsOneHour() {
  manager = aExpirationDateManager()
      .withDefaultTimeout(hours(1))
      .build();
  Date cacheExpiration = manager.getExpirationDate(NULL_CACHEKEY, 0L);
  // When cache key is null, system date and time is returned and default timeout is not used.
  assertThat(dateOf(cacheExpiration).is(0).hoursAhead().from(systemDate));
 }
 
 public void 
 testExpirationTimeWithDefaultTimeout() {
  doNotIncludeMinimunTimeoutInCache();
  manager = aExpirationDateManager()
      .withDefaultTimeout(hours(1))
      .build();
  Date cacheExpiration = manager.getExpirationDate(cacheKey, 0L);
  assertThat(dateOf(cacheExpiration).is(1).hoursAhead().from(systemDate));
 }
 
 public void 
 testExpirationTimeWhenExpirationTimeoutIsSet() {
  manager = aExpirationDateManager()
      .withDefaultTimeout(hours(1))
      .withExpirationTimeout(hours(2))
      .build();
  Date cacheExpiration = manager.getExpirationDate(cacheKey, 0L);
  // Expiration timeout has precedence over default timeout.
  assertThat(dateOf(cacheExpiration).is(2).hoursAhead().from(systemDate));
 }
 
}


Conclusion

Test classes should be easy to read. They should express intention, system behaviour, business rules. Test classes should express how the system works. They are executable requirements and specifications and should be a great source of information for any developer joining the project.

In order to achieve that, we need to try to keep our test methods divided in just 3 simple instructions.

1. Context: The state of the object being tested. Here is where we set all the attributes and mock dependencies. Using variations of the Builder pattern can greatly enhance readability.
manager = aExpirationDateManager()
                .withDefaultTimeout(hours(1))
                .withExpirationTimeout(hours(2))
                .build();

2. Operation: The operation being tested. Here is where the operation is invoked.
Date cacheExpiration = manager.getExpirationDate(cacheKey, 0L);

3. Assertion: Here is where you specify the behaviour expected. The more readable this part is, the better. Using DSL-style code is probably the best way to express the intent of the test.
assertThat(dateOf(cacheExpiration).is(2).hoursAhead().from(systemDate));

In this post I went backwards. I've started from a messy test class and refactored it to a more readable implementation. As many people now are doing TDD, I wanted to show how we can improve an existing test. For new tests, I would suggest that you start writing the tests following the Context >> Operation >> Assertion approach. Try writing the test code in plain English. Once the test intent is clear, start replacing the plain English text with Java internal DSL code, keeping the implementation out of the test class.

PS: The ideas for this blog post came from a few discussions I had during the Software Craftsmanship Round-table meetings promoted by the London Software Craftsmanship Community (LSCC).

Tuesday 7 December 2010

A basic ActiveRecord implementation in Java

Recently I was looking for different ways to develop applications in Java and thought that it would be interesting trying to use ActiveRecord in my persistence layer instead of implementing the traditional approach with a DAO. My idea is not to create my own ActiveRecord framework but use the existing ones as a support for this approach.

The scope: Write functional tests that could prove that the methods save and delete on an entity work. I'll use an entity called Traveller for this example.

The technology: I chose to use the following frameworks: Spring 3.0.5, JPA 2.0, Hibernate 3.5.3, AspectJ 1.6.9, JUnit 4.8.2, Maven 2.2.1, Eclipse Helios and MySQL 5.x

I'll be omitting things that are not too important. For all the details, please have a look at the whole source code at:

https://github.com/sandromancuso/cqrs-activerecord

Let's start with the test class:

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(
  locations={
    "file:src/test/resources/applicationContext-test.xml",
    "file:src/main/resources/applicationContext-services.xml"    
  })
@TransactionConfiguration(transactionManager = "myTransactionManager", defaultRollback = true)
@Transactional
public class TravellerActiveRecordIntegrationTest extends BaseTravellerIntegration {
 
 @Test public void 
 testTravellerSelfCreation() {
  assertThereAreNoTravellers(named("John"), from("England"));
  
  Traveller traveller = aTraveller().named("John").from("England").build();
  traveller.save();
  
  assertThereIsASingleTraveller(named("John"), from("England"));
 }
 
 @Test public void
 testTravellerEdition() {
  Traveller traveller = aTraveller().named("John").from("England").build();
  traveller.save();
  
  traveller.setName("Sandro");
  traveller.setCountry("Brazil");
  traveller.save();
  
  assertThereAreNoTravellers(named("John"), from("England"));
  assertThereIsASingleTraveller(named("Sandro"), from("Brazil"));  
 }
 
 @Test public void 
 testDeleteTraveller() {
  Traveller traveller = aTraveller().named("John").from("England").build();
  traveller.save();

  traveller.delete();
  assertThereAreNoTravellers(named("John"), from("England"));
 }
 
}

A few things to notice about this test class:
- As this test is meant to insert and delete from the database, I set it to always rollback the transaction after each test, meaning that nothing will be committed to the database permanently. This is important in order to execute the tests multiple times and get the same results.
- The test class extends a base class, that has the methods assertThereAreNoTravellers and assertThereIsASingleTraveller. The advantage of doing that is that you separate what you want to test from how you want to test, making the test class clean and focused on the intention.
- Note that I also used the Builder pattern to build the traveller instance instead of using the setter methods. This is a nice approach in order to make your tests more readable. 

The Traveller entity implementation

So now, let's have a look at the Traveller entity implementation.

@Entity
@Table(name="traveller")
@EqualsAndHashCode(callSuper=false)
@Configurable
public @Data class Traveller extends BaseEntity {

 @Id @GeneratedValue(strategy=GenerationType.AUTO)
 private long id;
 private String name;
 private String country;
 
}

The Traveller class is a normal JPA entity as you can see by the @Entity, @Table, @Id and @GenereratedValue annotations. To reduce the boiler place code like getters, setters, toString() and hashCode(), I'm using Lombok, that is a small framework that generate all that for us. Just add a @Data annotation.

The real deal here is Traveller's super class BaseEntity (for a lack of inspiration to find better name). Let's have a look:

@Configurable
public abstract class BaseEntity {

 @PersistenceContext
 protected EntityManager em;

 public abstract long getId();
    
 public void save() {
  if (getId() == 0) {
   this.em.persist(this);
  } else {
   this.em.merge(this);
  }
  this.em.flush();
 } 
 
 public void delete() {
  this.em.remove(this);
  this.em.flush();
 }
 
 public void setEntityManager(EntityManager em) {
  this.em = em;
 }
 
}

The BaseEntity class has quite a few things that can be discussed.

The save() method: I've chosen to have a single method that can either insert or update an entity, based on its id. The problem with this approach is that, firstly, the method has more than one responsibility, making it a bit confusing to understand what it really does. Secondly it relies on the entities id, that needs to be a long, making it a very specific and weak implementation. The advantage is that from the outside (client code), you don't need to worry about the details. Just simple call save() and you are done. If you prefer a more generic and more cohesive implementation, make the getId() method return a generic type and split the save() method into a create() and update() methods. My idea here was just to make it simple to use.

EntityManager dependency: Here is where the biggest problem lies. For it to work well, every time a new instance of a entity is created, either by using new EntityXYZ() by hand or when it is created by a framework (e.g. as a result of a JPA / Hibernate query), we want the entity manager to be injected automatically. The only way I found to make it work is using aspects with AspectJ and Spring.

Configuring AspectJ and Spring

My idea here is not to give a full explanation about the whole AspectJ and Spring integration, mainly because I don't know it very well myself. I'll just give the basic steps to make this example work.

First add @Configurable to the Entity. This will tell that the entity will be managed by Spring. However, Spring is not aware of instances of the entity, in this case, Traveller, being created. This is why we need AspectJ. The only thing we need to do is to add the following line to our Spring context xml file.

<context:load-time-weaver />

This makes AspectJ intercept the creation of beans annotated with @Configurable and tells Spring to inject the dependencies. In order to the load-time weaver (LTW) work, we need to override JVM's class loading so that the first time our entity classes are loaded, AspectJ can kick in, the @Configurable annotation is discovered and all the dependencies are injected. For that we need to pass the following parameter to the JVM:

-javaagent:<path-to-your-maven-repository>/.m2/repository/org/springframework/spring-instrument/3.0.5.RELEASE/spring-instrument-3.0.5.RELEASE.jar

The snippet above is what we must use if using Spring 3.0.x. It works fine inside Eclipse but apparently it has some conflicts with Maven 2.2.1. If you run into any problems, you can use the old version below.

-javaagent:<path-to-your-maven-repository>/.m2/repository/org/springframework/spring-agent/2.5.6/spring-agent-2.5.6.jar

Another thing that is a good idea is to add the aop.xml file to your project, limiting the classes that will be affected by AspectJ. Add the aop.xml to src/main/resources/META-INF folder.


 
  
  <include within="com.lscc.ddddemo.model.entity.*" />
  <include within="com.lscc.ddddemo.model.entity.builder.*" />
  <exclude within="*..*CGLIB*" />
 

 



NOTE: The exclude clause is important to avoid some conflicts during the integration test. AspectJ sometimes tries to do some magic with the test classes as well causing a few problems and the exclude clause avoids that.

Making the integration test work

I'll be using MySQL, so I'll need a database with the following table there:

CREATE TABLE `traveller` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(45) NOT NULL,
`country` varchar(45) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB 

As I'm using JPA 2.0, we need a persistence.xml that must be located at src/main/resources/META-INF


 
    
        com.lscc.ddddemo.model.entity.Traveller
 
        
            <property name="hibernate.show_sql" value="true" />
            <property name="hibernate.format_sql" value="true" />
 
            <property name="hibernate.connection.driver_class" value="com.mysql.jdbc.Driver" />
            <property name="hibernate.connection.url" value="jdbc:mysql://localhost:3306/lscc-ddd" />
            <property name="hibernate.connection.username" value="root" />
 
            <property name="hibernate.c3p0.min_size" value="5" />
            <property name="hibernate.c3p0.max_size" value="20" />
            <property name="hibernate.c3p0.timeout" value="300" />
            <property name="hibernate.c3p0.max_statements"
                    value="50" />
            <property name="hibernate.c3p0.idle_test_period"
                    value="3000" />
        
    
 


In our Spring configuration, we also need to tell Spring how to create an EntityManager, providing the EntityManager Factory:


        
    
        <property name="persistenceUnitName" value="testPU" />
    

    <tx:annotation-driven transaction-manager="myTransactionManager" />
 
    
        <property name="entityManagerFactory" ref="entityManagerFactory" />
    
       


In order to separate unit tests and integration tests in my Maven project, I've added a different profile for the integration tests in my pom.xml:


   with-integration-tests
   
    
     
      org.apache.maven.plugins
      maven-surefire-plugin
      
       always
       -javaagent:/Users/sandro.mancuso/.m2/repository/org/springframework/spring-agent/2.5.6/spring-agent-2.5.6.jar
      
      2.5
      true
      
       
        integration-tests
        integration-test
        
         test
        
        
         
          **/common/*
         
         
          **/*IntegrationTest.java
         
        
       
      
     
    
   
  


So now, if you want to run the integration tests from the command line, just type:

mvn clean install -Pwith-integration-tests

If running the tests from inside Eclipse, don't forget to add the -javaagent paramenter to the VM.

Conclusion

So now, from anywhere in your application you can do some thing like:

Traveller traveller = new Traveller();
traveller.setName("John");
traveller.setCountry("England");
traveller.save();

The advantage of using the ActiveRecord:
- Code becomes much simpler;
- There is almost no reason for a DAO layer any more;
- Code is more explicit in its intent.

The disadvantages:
- Entities will need to inherit from a base class;
- Entities would have more than one responsibility (against the Single Responsibility Principle);
- Infrastructure layer (EntityManager) would be bleeding into our domain objects.

In my view, I like the simplicity of the ActiveRecord pattern. In the past 10 years we've been designing Java web applications where our entities are anaemic, having state (getters and setters) and no behaviour. So at the end, they are pure data structures. I feel that entities must be empowered and with techniques like that, we can do it, abstracting the persistence layer.  

I'm still not convinced that from now on I'll be using ActiveRecord instead of DAOs and POJOs but I'm glad that now there is a viable option. I'll need to try it in a real project, alongside with Command Query Responsibility Segregation (CQRS) pattern to see how I will feel about it. I'm really getting sick of the standard Action/Service/DAO way to develop web applications in Java instead of having a proper domain model.

By the way, to make the whole thing work, I had loads of problems to find the right combination of libraries. Please have a look at my pom.xml file for details. I'll be evolving this code base to try different things so when you look at it, it may not be exactly as it is described here.

https://github.com/sandromancuso/cqrs-activerecord

Interesting links with more technical details:
http://nurkiewicz.blogspot.com/2009/10/ddd-in-spring-made-easy-with-aspectj.html
http://blog.m1key.me/2010/06/integration-testing-your-spring-3-jpa.html

Monday 27 September 2010

Bad Code: The Invisible Threat

One of the first things said by the non-believers of the software craftsmanship movement was that good and clean code is not enough to guarantee the success of a project. And yes, they are absolutely right. There are innumerable reasons which would make a project to fail ranging from business strategy to competitors, project management, cost, time to market, partnerships, technical limitations, integrations, etc.

Due to the amount of important things a software project may have, organisations tend not to pay to much attention to things that are considered less important, like the quality of the software being developed. It is believed that with a good management team, deep hierarchies, micro management, strict process and a large amount of good documentation, the project will succeed.

In a software project, the most important deliverable is the software itself. Anything else is secondary.

Many organisations see software development as a production line where the workers (developers) are viewed as less skilled than their high-qualified and much more well paid managers. Very rarely companies like that will be able to attract or retain good software developers, leaving their entire business on the hands of mediocre professionals. 


Look after your garden

Rather than construction, programming is more like gardening. - The Pragmatic Programmer

Code is organic, not mechanical. Like a garden, code needs constant maintenance. For a garden to look nice all year round, you need to look after its soil, constantly remove weeds, regularly water it, remove some dead plants, replant new ones, trim or re-arrange existing ones so they can stay healthy and look nice as whole. With basic regular maintenance, the garden will always look great but if you neglect it, even if for a short period, the effort to make it nice again will be much bigger. The longer you neglect it, the harder it will be to make it look nice again and you may even loose some or all of your plants.

Code is no different. If code quality is not constantly looked after, the code starts to deteriorate. Bad design choices, lack of tests and poor use of languages and tools will make parts of the code to rot. Bit by bit other parts of the code will also be contaminated up to the point that the whole code base is so ill that it becomes extremely painful to change it or add new features to it.

The Invisible Threat

When starting a greenfield project, everything is great. With a non-existent code base, developers can quickly start creating new features without the fear of breaking or changing any existing code. Testers are happy because everything they need to test is new, meaning that they don't need to worry about regression tests. Managers can see a quickly progress in terms of new features added and delivered. What a fantastic first month the team is having. 

However, this is not a team of craftsmen. This is a team of average developers structured and treated like unskilled production line workers. 

As time goes by, things are getting messier, bugs start to appear (some with no apparent explanation) and features start taking longer and longer to be developed and tested. Very slowly, the time to deliver anything starts to stretch out. But this is a slow process. Slow enough that takes months, sometimes, over a year or two to be noticed by the management.


It's very common to see projects where, at the beginning of a project, a feature of size X takes N number of days to be implemented. Over the time, as more bad code is added to the application, the same feature X (or a feature of the same size) takes much longer to be implemented than it used to take at the beginning of the project. As the quality of the code decreases, the amount of time to implement a new feature, fix a bug or make a change increases. The lower the quality, the higher the number of bugs, the harder is to test and less robust and reliable the application becomes. 

Some people say that they just don't have time to do it properly but, in general, a lot more time and money is spent later on on tests and bug fixing.  

Hostage of your own software

When the code base gets into the situation where changes or additional features take too long to be implemented or worse, developers and managers are scared to touch existing code, an action must be taken immediately. This is a very dangerous situation to be since business progress is being impeded or delayed by the software instead of being helped by it.

To keep business progress, schedule and budget under control, high quality code needs to be maintained at all costs.

Organisations may need to cancel the implementation of some features or postpone changes just because of the amount of time and money that they may cost to be built. Having poor quality of code responsible for it is totally unacceptable. 

The biggest problem here is that bad code is invisible to everyone besides developers. Other members of the team will just realise that something is wrong when it is too late. This means that it is the developers responsibility to look after the quality of the code. Some times, developers expose the problem to project managers but the request for having some time to "re-factor" the code is often ignored for various reasons, including a lack of understand of the impacts of bad code and the inability of developers to explain it. On the other hand, when developers come to a point where they need to ask for some formal time to do refactoring, this means that for one reason or another, they neglected the code at some point in the past.

Hire craftsmen not average developers
 

With the amount of literature, tools, technologies, methodologies and the infinite source of information available on the web, it is just unacceptable to have a team of developers that let the code to rot.

Craftsmen are gardeners and are constantly looking after the code base, quickly refactoring it without fear since they are strongly backed by a good battery of tests that can test the entire application in just a few minutes. Time constraints or change in requirements will never be used as excuses for bad code or lack of tests due to the good design principles and techniques constantly used throughout the application.

Having an empowered team of craftsmen can be the difference between success and failure of any software project.

Quality of code may not guarantee the success of a project but it can definitely be the main invisible cause of its failure. 

Tuesday 14 September 2010

Beyond the manifesto: The Software Craftsmanship Attitude

Being an aspiring software craftsman goes way beyond than just saying it. I'll quote my own definition of software craftsmanship from my previous post.
Software craftsmanship is a long journey to mastery. It's a lifestyle where developers choose to be responsible for their own careers and for improving their craft, constantly learning new tools and techniques. Software Craftsmanship is all about putting responsibility, professionalism, pragmatism and pride back into software development.
Software craftsmanship is all about attitude. The attitude of raising the bar of professional software development starting with our own skills and professionalism.

The responsibility shift

Not long ago, I was speaking to a developer and he was complaining about his company, saying that they didn't have a good career plan, that he did not have any training, that he was not given the opportunity to learn new technologies and, of course, that he was not paid enough. Apparently, from his perspective, his employer was responsible for his career.

Imagine that we need a doctor. Would we pay a doctor to learn while he cut us open or give us a diagnosis? Would we pay an engineer to learn while he draws the plan for our new house? Would we go to a concert and pay the musician to learn how to play the guitar during the show? What about a chef in a restaurant?

So why is the employer's obligation to pay us training courses and pay us to learn new technologies and tools while we are working on a project? Should the employers be responsible for what we learn and what we don't learn.

Software development is not a 9 to 5 profession. To be a professional software developer, we need to take our own time and money to keep learning and improving. As professionals, we should be paid for what we know, our ability to learn fast and for the quality of the work we do. We own our careers and are responsible for them. Working for a customer/employer that helps us with our career in terms of training, books, conferences, seminars, etc, is great but should be considered a bonus.


... but how can we learn and keep ourselves up-to-date?

Different developers have different preferences but here is a list of ways I find useful.

Literature 

Books, many books. Having your own library is essential. Books give you a good overview of a specific technology or subject quickly. No, it does not mean you will be proficient and next day you will be a specialist. What a book will give you is an understanding of what a technology or subject is about. It will be up to you then to decide if you want to practice what you've learned and become proficient. If you don't have the habit, try reading 3 to 4 technical books per year. Once you get the habit, try one per month. The most common excuse is that you don't have time. The cool thing about books is that you can read them during periods of  "dead" time, like on the tube, bus, in your dentist's waiting room, on the bed before going to sleep, toilet, etc.  

Blogs are now one of my favourite types of reading. They tend to fit more the software craftsmanship model since they are much more personal and in general, related to personal findings, opinions, successes and failures. Reading blogs from more experienced professionals and subject matter experts is a good, quick and free way for apprentices and journeymen to learn from multiple master craftsmen at the same time. But don't think that just experienced professionals should write blogs. Every software developer should write his or her own blogs, sharing their experiences and findings, helping to create a great community of professionals.  

Technical websites are also good in order to keep yourself up-to-date with what's going in the market. New technologies, techniques, etc.

Practice, practice, practice

Pet project(s) is for me, by far, the best way to learn and study. A pet project is a real project but without the boring bits. There are no deadlines, does not need to make money, you control the requirements and most importantly, you use the technologies and methodologies you want, whenever you want, wherever you want. You are the boss. Pet projects give something for you to focus and help you to understand why and how you can use certain technologies. It gives you the experience you need to apply what you've learned into real projects. Pet projects are meant to be fun. 

Contributing to open source projects can also be a great thing. There are thousands of them out there. Find a project that is related to what you want to learn or know more about and download the source code. Start running and reading the tests, if any. Inspect and debug the code. If you want to contribute, start small. Add some documentation and write some tests. Then, check the list of features to be implemented and pick a simple one and give it a go. You can also propose and implement a new and small one to start with.

Pair-programming can be a great experience. Many developers are afraid to try or think they will feel uncomfortable. That's what I thought as well before I tried. Today I really enjoy it. Pair-programming gives you an opportunity to discuss your ideas, to learn new tricks and techniques from your pair and the resulting code is much better. Pairing with someone from your team is good but pairing with someone that you barely know can be a very interesting experience.

If learning a new language or new technique like TDD / BDD or trying different approaches to OOP or functional programming, try a code kata. Code kata is a small exercise that in general can be solved in a few minutes or in a few hours. Code katas were created to help developers to focus in a problem while improving their skills. You can find a good source of code katas at codingkata.org and codekata.pragprog.com

Community and Events

Be part of your local community. Software craftsmanship is all about a community of professionals, learning and sharing with each other and elevating the level of professionalism and maturity of our industry. Join your nearest user groups and participate in their events. User groups are the best way for making contacts, sharing ideas and learning things. Many user groups promote free talks, coding activities and social events. A great aspect of being part of a community if the feeling that you are not alone. There are many people out there having the same problems you've got and many others that will happily share their solution to it.

Raising the bar

The main changes proposed by the software craftsmanship movement are related to the developers attitude. In summary, it's about being proud about your work, the code you produce and the software you create. It's about constantly trying to improve your skills and learn new ones.

An aspiring software craftsman is intolerant of bad code will constantly apply the Boy Scout Rule

If you find that the good code you wrote one year ago is still good enough today, it means you didn't learn anything during this period and this is unacceptable.

Productive partnerships

The attitude of a software craftsman goes beyond good code. The relationship between a software craftsman and his or her customer (or employer) is of a productive partnership and not employer / employee. As a productive partnership, it's our role to constantly question the requirements and propose improvements and new features. It's our job to warn our customers about the problems we see. Due to the knowledge of the code and technology, developers are well positioned to help their customers improve their business.

However, sometimes some customers or employers don't want or don't see the advantages of this partnership and will treat developers as production line workers that should just do what they are told. Coding monkeys. In cases like that, any aspiring software craftsman should move on and find another job. Staying in a place where your skills are not appreciated is a career suicide.  

Raising the bar of our industry is on our own interest. The better we become in writing code and delivering valuable software, the better our lives will be, professionally and financially. 

PS: If  you are in London or nearby, join the London Software Craftsmanship Community.

Friday 3 September 2010

Software Craftsmanship

So what is software craftsmanship? 

A better metaphor: In a very simplistic way, we can say that software craftsmanship is a better metaphor for software development than software engineering, as I wrote in a previous post. Software craftsmanship sees software as a craft and compares software developers to the medieval blacksmiths. Apprentices would work with more experienced blacksmiths, travelling from place to place, working with and for different masters, learning different tools and techniques, improving their craft until the point they were good enough to become master themselves. (There is more to it, but let's keep it simple for now).

Wikipedia Definition:  is an approach to software development that emphasizes the coding skills of the software developers themselves. It is a response by software developers to the perceived ills of the mainstream software industry, including the prioritization of financial concerns over developer accountability. (read in full)

I personally don't like too much the Wikipedia's definition. It's very dry and I don't think it captures the essence of what being a software craftsman means to a software developer.

A more personal definition: Software craftsmanship is a long journey to mastery. It's a lifestyle where developers choose to be responsible for their own careers and for improving their craft, constantly learning new tools and techniques. Software Craftsmanship is all about putting responsibility, professionalism, pragmatism and pride back into software development

A software craftsman cares and is proud of his or her work and is extremely professional and pragmatic when it comes to its implementation.



The Software Craftsmanship Movement

The software craftsmanship movement is basically an evolution of ideas that started probably in the late 90ies, early 2000 with the publication of The Pragmatic Programmer by Andy Hunt and Dave Thomas (1999) and Software Craftsmanship: The New Imperative by Pete McBreen (2001). In 2008, Uncle Bob proposed "Craftsmanship over Execution" (originally Craftsmanship over Crap) as the fifth value for the Agile Manifesto. In 2009, the Manifesto for Software Craftsmanship was created, defining the values of the movement and international Software Craftsmanship conferences emerged in the US and UK.

In the manifesto, the essence of the software craftsmanship is capture in its subtitle: Raising the bar. The manifesto was idealised by very experienced developers that had enough of project failures mainly caused by poor management, ill processes and, of course, badly-written code.

Developers are taking the matter into their own hands and are trying to change how the industry sees software development not just proposing new and revolutionary processes but showing customers that they care about what they do and that they want to work together with their customers in order to produce great and long-lived software.  

The values of the Software Craftsmanship Movement

Not only working software, but also well-crafted software
Working code is not good enough. Think on a 5 year old application (it could be 2 or 10) where we are scared to change some of its parts (fix bugs, add new features, etc.) because we don't understand how it works and have no confidence that we will not break anything else. This application is working software but is it good enough? Well crafted software means that regardless how old the application is, developers can understand it easily, side effects are well known and controlled, high test coverage, clear design, business language well expressed in the code and adding or changing features does not take longer that it used to take at the beginning of the project, when the code base was small.

The code must be maintainable and predictable. Developers must know what is going to happen when changing the code and must not fear to change it. Changes should be localised and not cause impact in other parts of the application. Tests will guarantee that nothing else was broken.

Not only responding to change, but also steadily adding value
This is not just about adding new features and fixing bugs. This is also about constantly improving the structure and cleanliness of the code. The software must be seen as an asset and the constant maintenance of it will make it more valuable during its lifetime, instead of letting it rot and devalue. 

The Boy Scout Rule (defined by Uncle Bob) states that we should always let the code a bit cleaner than we found it. This is a paraphrase of Boy Scouts rule: leave the camp cleaner than you found it. 

If customers want to keep benefiting from adding and changing features quickly, they will need high quality code to enable them to do it.  


Not only individuals and interactions, but also a community of professionals
This is somehow related to the idea of apprentices, journeymen and masters, where software craftsmanship masters will mentor apprentices and help them in their journey. The software craftsmanship community is responsible for training the next generation of professionals. Knowledge and ideas must be shared and discussed within the community in order to keep moving the software development industry forward. 

Not only customer collaboration, but also productive partnerships
Software craftsmen need successful projects to build their reputation and are proud of their achievements. Successfully delivering high quality software is essential for any software craftsman journey. With this in mind, software craftsmen will do whatever they can for a project to succeed. They don't act like simple employees that just do what they are told to do. They want to actively contribute to the success of the project, questioning requirements, understanding the business, proposing improvements and productively partnering with their customers or employers. This is an interesting shift of perspective, if you like, and the advantages for the customer and for the project success are enormous. A well-motivated team has a much bigger chance to make any project succeed. However, if the customer is not prepared to have this partnership and sees software development as an industrial process and the least important part of the project, this customer will never have real software craftsmen working for him for too long. Not getting involved with the business, not questioning requirements, not proposing improvements and not knowing the customers needs is not a partnership. Real software craftsmen make the customer needs, their needs.

Conclusion

Recently, Uncle Bob Martin said during an interview:
The original torch of the Agile message has changed hands, and is now being carried by the Software Craftsmanship movement. These are the folks who continue to pursue the technical excellence and professionalism that drove the original founders of the Agile movement.

The Software Craftsmanship Movement is another step forward towards better and healthier software projects. Instead of just focusing on processes, it focuses on the quality of the software that is produced and most importantly on the attitude and competence of the people involved.


Software Craftsmanship brings pride and professionalism into software development.

Sunday 22 August 2010

Software Engineering: The problem with the production line

Is software engineering the best approach for developing software? Does it apply for the majority of the software projects or just a very few of them?

Software Engineering was an answer for the perceived "software crisis", back in 1968, in the First NATO Software Engineering Conference and it was created to solve the problems of extremely large NATO and U.S. Department of Defence projects. In the majority of these projects, the hardware was still being designed and with no hardware to test, there was plenty of time to investigate requirements and write the software specifications. Hardware controlled by the software was generally worth billions of dollars like in the case of space shuttle and the Safeguard Ballistic Missile Defence System. People's lives and national security were also at stake.

The IEEE Computer Society's Software Engineering Body of Knowledge defines "software engineering" as:
Software engineering is the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software, and the study of these approaches; that is, the application of engineering to software.
Software Engineering can be very effective when developing safety critical systems like the one for the space shuttle as described on "They Write The Right Stuff" - Fast Company article, from 1996:

The last three versions of the program - each 420,000 lines long - had just one error each. The last 11 versions of this software had a total of 17 errors. Commercial programs of equivalent complexity would have 5,000 errors.
Of course that this looks really impressive but there is more to it:
Money is not the critical constraint: The group's $35 million per year budget is a trivial slice of the NASA pie, but on a dollars-per-line basis, it makes the group among the nation's most expensive software organizations.
Another extract from this same article when they were discussing the process:
And the culture is equally intolerant of creativity, the individual coding flourishes and styles that are the signature of the all-night software world. "People ask, doesn't this process stifle creativity? You have to do exactly what the manual says, and you've got someone looking over your shoulder," says Ted Keller (senior technical manager). "The answer is, yes, the process does stifle creativity."
Many of the NATO and US Department of Defence took years, some over a decade to complete. Many had hundreds of people involved and almost half of the time was spend in requirements and design specifications, with uncountable reviews and approval cycles. Of course that, due to their reliability and quality, many are still in use today.

Software Engineering for the Masses

More and more hardware became cheap and business of all sizes needed software in order to survive and be competitive. The difference was that a the big majority of these businesses couldn't afford to pay $35 million dollars per year and neither wait for too many years to start benefiting from their software. Also, many of those software projects were not manipulating expensive hardware or dealing with life-threatening situations. Edward Yourdon, in his book Rise and Resurrection of the American Programmer, wrote:
I'm going to deliver a system to you in six months that will have 5,000 bugs in it - and you're going to be very happy!
It was clear that software engineering processes had to be adapted in order to satisfy a more impatient and lower budget legion of businesses.

The "Good Enough Software" Era

Over the decades, many new software engineering processes and methodologies were created in order to make software development faster and cheaper. The most adopted were the ones based on iterative and incremental development that evolved into the agile software development.

Agile software development and "good enough software" were an amazing improvement in bringing costs down, mitigating risks with quicker feedbacks and much faster time to market.

Regardless the methodology or process used, software projects are still failing. Some fail because they were over-budget, others because they were not delivered on time, others failed to satisfy the requirements and business goals, et al.

The main problem is that for decades, software development was seen as a production line. That's the software engineering perspective of software development. Processes and methodologies are generally much more focused in making this production line more productive, creating different management and organisational styles than actually trying to make the employees more capable. Developers are treated as mere brain-damaged programmers and are at the bottom of the food chain.

Good enough is not always good enough

Using agile, lean or any methodology is not enough. It is all about the people involved in the project and their willingness to succeed and be proud of what they produce. If developers are seen as the least important and cheapest members of a software project, the maximum that this team is going to produce is mediocre software, regardless of methodology.

A software project would have much better chances to succeed with "good people with a bad process" than "mediocre people with a good process". Good people would always find a way to improve the process, be productive and produce something that they are proud of. Mediocre people accepts whatever is established, even when it is not good enough, and will produce just what they were asked for.

In a software project, if a company wants a good software, they will need good and empowered software developers. Instead of 10 mediocre people with a good process, it would be better to have 3 or 4 good people and empower them to deliver the project.

A process, imposed by managers and people that have no clue how to write software, will just guarantee that mediocre software is predictably delivered (if lucky). On the other hand, a team of self-organised and great developers would have a better shot at creating a more efficient way to produce great software, constantly trying to improve they way they work.

A process should never be more important than the people. Managers should facilitate the work of great software developers and not tell them what to do. It should always be harder to replace a good software developers than a manager since they are the ones that know the system inside-out. Managing a few well motivated, well paid and good professionals is always easier than manage many mediocre people. 

Software development is a creative and highly skilled profession that takes years to master. While software development is treated like a production line, projects will continue to fail.

Source
Software Craftsmanship: The New Imperative - ISBN 0-201-73386-2, 2002
http://en.wikipedia.org/wiki/Software_engineering
http://en.wikipedia.org/wiki/Software_crisis
IEEE Standard Computer Dictionary, ISBN 1-55937-079-3, IEEE 1990
"They Write The Right Stuff", Fast Company, http://www.fastcompany.com/magazine/06/writestuff.html
Safeguard Program: http://en.wikipedia.org/wiki/Safeguard_Program
Stephenson, W. E. "An analysis of the resources used in the SAFEGUARD system software development"
Edward Yourdon  - http://yourdon.com/about/ 
http://en.wikipedia.org/wiki/Iterative_and_incremental_development 
http://en.wikipedia.org/wiki/Agile_software_development

Saturday 7 August 2010

Which type of barista are you?

A colleague from New Zealand once was telling me about a type of coffee that was originated there. It's called Flat White Coffee. We were discussing about the difference between this coffee and all the other types of coffee with milk. Eventually we started talking about the quality of the drink and what makes it be better or worse. He mentioned that, of course, the quality of the beans and milk are very important for a good coffee but what makes a good coffee an excellent coffee is the barista's ability.

What makes a coffee to be a rubbish coffee then? If the coffee grains are rubbish, you will have a rubbish coffee. However, for a flat white coffee, coffee is just one variable in the equation. There are other variables like how the grains are roasted, steaming the milk at the right temperature, not adding sugar, how the milk is poured, the microfoam on top of the drink, etc. See the distinction from cafe con leche for details. Anyway, the point is, a rubbish (or careless) barista, with rubbish coffee will produce a rubbish coffee drink.

Rubbish Coffee
Clearly a business that serves a rubbish coffee will not survive if their main business is to sell coffee drinks. Also, this business will never attract customers that really appreciate a good coffee.

Average Coffee
Producing an average coffee is easy. You buy average coffee grains, hire an average barista and, hey presto, you have an average coffee drink. There is nothing wrong with an average coffee drink if you are not in the coffee business. However, if you are in a coffee business, your business will be just another one. No one will remember you and chances are that you will have occasional customers, but not regulars.


Good Coffee
Producing a good coffee is not that simple. However, it does not need to be expensive. You don't need to buy the best quality ingredients to be able to make a good coffee. Best quality ingredients are expensive and inevitably will make your coffee drinks more expensive as well. You can mitigate this situation hiring a good barista, or maybe someone that has the potential and willingness to become a good one. Clearly, the barista that prepared the coffee above is trying his best to make a good coffee. It's still not perfect (purely looking at the pattern on top of the drink) but it is definitely a much better coffee than the normal and average one that you get everywhere. It is clear that the barista cares about it and eventually he will be able to produce a very good coffee.If you are in the coffee business, producing anything less than a good coffee is just unacceptable.

Great Coffee
And then you have the great coffee, that is made with great coffee beans, carefully roasted, prepared in a very good coffee machine and by a great barista. Great baristas are proud of their ability to prepare a great coffee and they would not work for too long for a business where coffee making is not treated with the deserved respect.

Now imagine that you are the barista. But instead of coffee, you produce code. Imagine that the the coffee grains, milk and coffee machine are the tools you use to produce your code like the computer language, the IDE, the database, etc. Instead of serving coffee for your customers, you are producing software that will be used by your team mates, project sponsors, the company they (or you) work for, external clients, etc. 

Like in the flat white coffee example above, of course that the tools we use are very important when producing a good software. However, more importantly, it is the quality of the software engineers that counts. A good barista can make a good coffee even when using average coffee grains, due to his or her ability to combine ingredients and prepare the drink. A bad barista can ruin the coffee even if he or she is using the best quality coffee beans. The biggest difference between the two baristas is how much they care about each cup of flat white coffee they prepare. Their pride and willingness to achieve the best pattern on top of each drink. The great feeling of achievement when they produce a great one. The happiness to see returning customers, queuing and waiting their turn to order the coffee that he, skilfully, prepares.

So, which type of barista are you?

Wednesday 23 June 2010

One team, one language

On a previous post I was discussing, among other things, how code often doesn't represent the business properly. The code "satisfies" the business requirements but doesn't express them very well. The main reason for that is because we, developers, like to abstract business terms and rules into technical implementations and patterns.

Very often, we discuss the user stories (requirement documents, use cases, whatever the methodology used is) with the "domain experts" and as soon as we understand what needs to be done, we map the requirements to a technical design (actions, services, entities, helpers, DAOs, etc) that is completely meaningless to the domain experts. Sometimes, even among developers themselves, different names and expressions are used to refer to the same thing. The main reason is that different developers talk to different domain experts and come up with different abstractions. As a result,  the usage of different terms to describe requirements leads to confusion, duplication of code and unpredictable behaviour in the system.

The first step towards an expressive and domain-focused design is to have a common language among ALL members of the team. ALL means ALL: developers, domain experts (business analysts, users, product owner, etc), testers, project manager and anyone else involved in the project.

Developers very often say that domain experts don't understand objects and database and because of that, they need to "translate" business requirements into software design. However, we developers don't understand the business as well as the domain experts do, what more often than not, leads to imperfect and confusing abstractions.

The Ubiquitous Language

A language structured around the domain model and used by all team members to connect all the activities of the team with the software.
The Ubiquitous Language is one of the most important things, if not the most, in Domain-Driven Design. The main idea is that the whole team speaks a single language, that is the business language.



Business terms related to the software to be implemented must enter the ubiquitous language and each of these terms must be understood clearly by all members. During requirements gathering sessions and planning meetings, these terms must be captured and made available to everybody. Technical terms from the development team and business terms not relevant for the piece of software being implemented MUST NOT enter the ubiquitous language. 

Capturing the ubiquitous language

In order to capture the ubiquitous language, it is mandatory that you work on a iterative software development environment. Trying to capture the ubiquitous language up-front could straitjacket the whole process, inhibiting team members to make the necessary changes along the way. As the language is used to express the business requirements, it is natural that it evolves during the lifetime of the project, where new terms are added, deleted and also re-defined.


Methodologies like Extreme Programming (XP) says that the only documentation should be the code. Not even comments on the code are appreciated, since they can easily get out of sync with the code. The code should be the only documentation since it is the only one that represents exactly what the system does.

On the other hands, we have UML (Unified Modeling Language), that in theory, should be a great candidate to document the ubiquitous language since the whole purpose of UML was to document requirements and express them in a language that is common to developers and business people.

In summary, showing code during discussions with domain experts, testers and other members of the team during a design session is not exactly a fantastic idea. Also, using just UML, because of its bureaucracy, rules and details is also a bad idea. The whole UML notation could easily straitjacket the creative process during the exercise.

There are many discussions about what would be the best way to capture the ubiquitous language. My preferred way is to draw diagrams (boxes and arrows mixed with some well understood UMLish notation) where each box represent a "domain object" (aka domain concept). A domain object can be anything that is expressed by the business, like client, organisation, product, route specification,  sales system, etc. They would all be boxes. Add to it a few arrows linking the boxes and with just a couple of words explaining how they related to each other. Sometimes a mixture of a class and sequence diagram (or an active diagram) can be very helpful, but don't get to picky about any notation. Preferably, draw on a white board, take a picture and store that on the wiki. For further sessions, just open the wiki and re-draw just the bit of the design that is important to the feature being discussed. Make the necessary adjusts on the white board, take another picture and stored it on the wiki again. There is much more to that, if you want to dive into agile modeling, but I will leave it to another post.

This goes way beyond transforming nouns and verbs into classes and methods. Taking the examples above, for example, client, organisation and products could be transformed in entities; route specification could be a strategy class used by a routing service (that would also need to be added to the diagram and to the ubiquitous language); sales system would be an external system that we need to integrate to, etc.

Making the code more expressive

The code should reflect all concepts exposed by the model. Classes and methods should be named according to the names defined by the domain. Associations, compositions, aggregations and sometimes even inheritances should be extracted from the model. 

Sometimes, during implementation, we realise that some of the domain concepts discussed and added to the model don't actually fit well together and some changes are necessary. When it happens, developers should discuss the problems and/or limitations with the domain experts and refactor the domain model in order to favour a more precise implementation, without ever distorting the business significance of the design. 

Ultimately, the code is the most important artefact of a software project and it needs to work efficiently. Regardless of what many experts in the subject say, code implementation will have some impact on the design. However, we need to be careful and very selective about which aspects of the code can influence design changes. As a rule, try as much as you can to never let technical frameworks limitations influence your design, but as we know, every rule has exceptions.

A common implementation problem that very often get in the way is mapping objects to databases using ORM tools. In this case, bending the model a little bit in favour of a more realistic relation among entities is not a bad thing. Just make sure that changes like that are represented in the model and understood by everyone involved.    

DOs and DON'Ts 
  • Don't try to model everything. Focus on the core of your application. We are not working on a waterfall or Unified Process project here.
  • Try to model just the key concepts of the domain problem;
  • Do not clutter your models with too much details. Keep it focused on the main responsibilities;
  • Limit your discussions and changes in the model to the business concepts (domain objects) related to the user story being discussed;
  • Don't add architectural concepts like DAOs, Actions, etc. We are not writing implementation diagrams. 
  • As your application grows, break the application into multiple models (domains), explicitly defining the context and boundaries within which a model applies. This is called Bounded Context in Domain-Driven Design.
  • Avoid thinking purely on the implementation when designing your model. Understanding and modeling the business is the most important thing here. 
Challenges of a Model-Driven Design

Model-Driven Design (MDD) is more an art than a science. It takes a lot of practice and willingness to get it going and get it right. Refactoring towards deeper insights must be seen as a positive and essential part of the project development. TDD and continuous integration are also essential for any agile and domain-driven application.

The goal of MDD is having the software expressing a deep and supple design.
Last buy not least, developers with good Object-Oriented Design skills are needed in the project. The lack of design skills could easily transform ANY software project in a total failure in the long term.
Source

Saturday 12 June 2010

The Wolf in Sheep's Clothing

In the last few years, I've noticed that the majority of the projects that I've participated roughly followed the same design. Speaking to colleagues and friends, the great majority said that they were also following the same design. It's what I call "ASD" design (Action-Service-DAO).

by Action we mean a Struts Action, Spring Controller, JSF backing bean, etc. Any class that handles an action triggered by the user interface.

We can argue that there is nothing really wrong about this approach since if we map that to a three-tier architecture, that is followed by the majority of the applications, the ASD classes would be in the right places, keeping a good isolation between the tiers.


One of the problems of this approach is when services are created randomly, with different granularities, low cohesion and with a weak representation of the business domains. Services end up being function libraries, almost like an utility class where all "functions" related to a specified "module" are grouped. When it comes to DAOs, the situation is not very different. DAOs are developed in a way that they become utility classes where sometimes we find a single DAO with all queries, inserts, delete, updates for an entire module or sometimes you find one DAO per entity. Either way, regardless what the query returns (or what it is its intention), or any rules related to updates or deletes, the methods will go to the same class. As the application grows, the code becomes something like this:


Looking at the picture above, can we really say that it is object-oriented programming? As services and DAOs don't strongly represent business concepts, the code becomes procedural. I don't want to bang on about the advantages of OOP over Procedural code. I believe that too many people have already discussed that over the past 30 years. My point here is to discuss what a "design" like that can do to an application.

Following bad examples


Unfortunately, in software development, bad examples are easier to be followed than good examples. With a non-expressive business model like that, the services get overloaded with methods. When adding new features to the application, developers need to go through all the methods of a service to see if there is already a method that does what they need. Due to the lack of cohesion and method explosion in the services, many developers will just add a new method in there, that can be very similar to existing ones, instead of re-factor the existing ones. This leads to services with confusingly similar methods and a lot of duplication. As a chain of "bad" events, the more methods a service has, the more classes depending on it the application will have. Many dependencies means that re-factoring the class would be harder, discouraging any one willing to improve the quality of the code.

Duplication does not happen just inside a single service. It's also very common to find different services with very similar methods, if not the same. In general, this is due for the lack of clarity on the responsibility of each service. According to the feature that developers are working on, they will choose a service to add (or reuse) the methods they need. If the responsibility and granularity of each service is not clear, business rules related to a certain area of the business will be spread all over the place since different developers will think that the method will belong to different classes.  


Exposing DAOs to the presentation tier

Another side effect of this approach is that since the DAOs are also "utility classes" with no business meaning (it just has an architectural meaning), some developers can easily expose the DAOs to the actions, without going through the services. Let's give more meaningful names and more details about the Action 4 and DAO 4 shown above. Let's call them ClientAction and ClientDAO.



Here the user interface needs to display a list of clients. ClientDAO has a method that returns a list of clients. In theory this may make sense. The most used argument in favour of this approach instead of adding a "ClientService" in between the ClientAction and the ClientDAO is that the service would have a method that just delegates the call to the DAO. The service layer, in this case, would be considered redundant and just pointless extra work. 


Looking at this isolated scenario, having a service in the middle really looks a bit overkill and unnecessary. However, we will probably want to do more things with a client. We will probably want to create a new client, delete or update an existing client.


Here is where the problems begin. Very rarely, we have a pure CRUD application. In general, many business rules have to be performed before or after any changes are made in the persistence tier (generally a database). For example, before inserting a new client, a credit check needs to be performed. When deleting a new client, we need to make sure that there is no outstanding balance for that client and close his or her account. We also need to archive all the orders for this client.  Whatever the application's business rules dictate. Multiple entities (or database operations) may be involved in a operation like that. We need to be able to define transaction boundaries. Besides CRUD operations, we will also have all business methods like check if client has credit, add orders to a client, etc. This is too much for a single DAO to handle. DAOs should not perform any business logic and should be hidden from the presentation tier. It should be hidden (encapsulated) even from different services. DAOs should just deal with the persistence. Business logic and transaction boundaries should be controlled by a class with business responsibilities, that in the case of this "poor" design, it would be a service.   



So even having a few methods in the service that just delegate the responsibility to a DAO, it is still a price worth paying. The service should hide (encapsulate) the details of it's implementation. A client code does not need to know how and where the data is persisted.

OK, enough of this. Exposing DAOs to the presentation tier is WRONG. Let's move on.

Moving away from the procedural code (baby steps)

The first thing to do to move away from the procedural code is to make your code more expressive and aligned to the business. We need to narrow the gap between developers and domain experts (business analysts, users, product owner or whoever knows the business rules) so we all start speaking the same language. The code must be written in a way that it expresses the business and not just architectural concepts that means nothing to the business. Actions, Services and DAOs are technical terms with no connection to any business term.

There are a few techniques that can be applied (or even combined) in order to make it possible. In future posts, I'll be talking about some of these techniques in more details. I know, it's frustrating that I will not tell you right now how to do it after have spending all this time criticising the ASD procedural design. The important thing for now is to know that what looks a good solution, since it is used by many different people and in many different projects, may not be as good as it seems.

In the meantime, there is something that we can start doing to our code in order to reduce the mess and make the necessary refactorings easier in the future.

Preparing your code for an easier transition

The following advices will help us to get our code to a point where it can be easily refactored into a more domain (business) focused approach in the future. It will still be a bit procedural but will be much more well organised and a notion of components (business components) will start to emerge.

1. Identify key concepts (domains) in your application. (e.g. Client, Order, Invoice, Trip, Itinerary, etc)
2. You can create / refactor services for each one of the key domains.
3. Try to keep a well balanced granularity for all services.
4. Avoid services for small parts of the domain. For example, do not create a service for line items. Line items should be handled by the OrderService.
5. Services are not allowed to manipulate multiple domains (with exceptions of whole-part relations - compositions)
6. DAOs are encapsulated by the services and never accessed by any other class.
7. Not every service must access a DAO, but any DAO must be accessed by one and only one service.
8. Services must delegate operations to other services if the operation is related to a different domain from the one the service is handling.
9. Services talk to each other. DAOs never talk to each other.
10. Parts (like a line item) are never exposed by a service. Service always exposes the whole (like Order). Parts are accessed via the whole. (e.g. order.getLineItems();)

With the following rules, our procedural code starts looking more like meaningful objects, with a reasonably well defined interface and some significance in terms of business.

  
As mentioned before, this is still a bit procedural but this design already solve a some of the problems discussed earlier like having more meaningful and specific services. The responsibility and boundaries of each service are more defined, making it easier for developers to look for implemented methods and re-use what it is in there, reducing duplication. 

In future posts I'll be talking about the next steps towards a more expressive and business focused code, less procedural and more object-oriented.  


If you are curious about the next steps and can't wait for my posts, have a look at:
http://domaindrivendesign.org/
http://en.wikipedia.org/wiki/Behavior_Driven_Development
http://en.wikipedia.org/wiki/Model-Driven_Architecture
http://en.wikipedia.org/wiki/OOD