When given a choice we prefer to adopt an Agile approach. We believe that following an Agile approach leads to a far superior final product. We also believe that Agile is much less risky. So this post is unashamedly biased in favor of Agile.

First, a brief description of the two types of projects:



Waterfall-type project:

A waterfall project typically has a set deadline for set functionality.  Many businesses like this type of project because it seems predictable and makes it easy to get a budget approved for the project.

Process: Plan, Build, Test, Done.

Problems:

• Trying to predict accurately while software development is very unpredictable.
• User Requirements Specifications are usually insufficient and open to interpretation, which leads to errors.
• Scope Creep is time-consuming to manage and often “sneaks” in due to unclear specifications. This puts pressure on the development team to deliver more functionality in the same amount of time.

Agile-type project:

An agile project is typically seen as a recurring monthly expense instead of a big capital expense. Features are delivered frequently  (typically every two weeks) and the project carries on for as long as it is delivering useful functionality.

Process: Plan, Build, Test, *Repeat Repeat Repeat

Problems:

• Unable to promise specific features at a specific deadline far in the future.
• Difficult to work with businesses with a “waterfall” mindset (most big old businesses).
• Difficult to get project approval when hard-and-fast promises can’t be made.

Advantages:

• Deliver what’s useful, not “what the spec said”.
• Deliver useful features early.
• Detect problems early.
• Embraces change.

Why Projects Fail

Projects can be considered failures for many reasons. The most common reasons that I have observed are:

1. Budget Overruns and Late Delivery – these two usually go hand-in-hand.
2. Incorrect Functionality delivered – the client expected one thing and the programmers built something else.
3. Technical Problems – for example: the system is too slow to use, or it doesn’t integrate with another critical system in the software ecosystem.

Below is a list of ways to combat these three problems in either a Waterfall or Agile project.

Problem 1: Budget Overrun/ Late Delivery

Waterfall:

• Add extra time to mitigate risk. For example: estimate 13 hours for a 10 hour task. Not optimal. Expensive.

• Track progress against milestones to ensure the project is progressing on schedule. (The problem with milestones is that they can give a false sense of security. When you think something is “Done” it might not really be. “It’s done! (except for the styling, or integration with external system X)”)
• Work overtime if needed to catch-up.

Agile:

• Frequent delivery means most important features get delivered early.

• If less important features are late it is not such a big deal. The system is constantly improving and maturing.

Problem 2: Technical Problems

Waterfall:

• Do lots of prototyping in the first stage of the project to try and spot technical challenges.

• If caught late it may be too late to do anything about it. Major losses.

Agile:

• Will be picked up early due to continuous early delivery. Project can then go a different direction, or cancel project if absolutely necessary (at least it will be early).

Problem 3: Wrong Functionality (incorrect assumptions)

Waterfall:

• Write very detailed user requirements specification. (The problem with User Requirements is that they give a false sense of security. Now matter how explicitly they are written, they are still open to individual interpretation and interpretation may not be what the author intended.)
• If caught late it may be too late to do anything about it. Major losses.

Agile:

• Agile embraces change. Frequent delivery ensures that the correct functionality is delivered and any “mistakes” are detected early. The client is continually evaluating the system and can change direction as required.

In the dynamic software development industry you need to experiment, adapt and find out how to be super-efficient without sacrificing quality. It’s a challenge.

In this series of blog posts I will describe in detail the things we do and why those things work so well.

Problem Nr1: Broken Code

Any reasonably sized software application (or website) is essentially a complicated machine composed of inter-related components. A large part of a software engineer’s job is to simplify that complexity by compartmentalizing  parts of the system and writing code that does what it’s supposed to but is also understandable . After all – how can someone maintain your code if they struggle to figure out what it does?

Yet, despite all our best efforts Errors still creep into the application. For example: I may innocently make a small change to how a financial calculation is done, and unwittingly break several other web pages that I did not realize would be influenced by my change. And the breakage may only be noticed several days later, making it very difficult to try and determine why the pages broke.

Unfortunately this is just the nature of software development: “Bugs Happen”.

Wouldn’t it be great if there was a way to protect against breaking code? And to detect bugs as soon as possible?

Problem Nr2: Being afraid to change code

The world is moving at a rapid pace. Businesses change and need to adapt to be able to take advantage of gaps in the market.

Nobody understands this better than a software developer who has had to change how a system worked even before they were done implementing it completely first time ’round.

Unfortunately changing code can be scary, because your change may break existing features without you realizing it. So often developers try to make changes in a way that is unlikely to break something else by “sticking” the code on the side of the system where it interacts with as little other code as possible. This obviously results in a poorly integrated solution that is difficult to maintain. It results in bad code.

Wouldn’t it be great if existing code were easy to change? That developers could respond rapidly to business demands and fearlessly change the entire solution to better fit the business requirements?

The Solution: Unit Tests

We believe Automated Unit Tests help alleviate the problems mentioned above. Automated Unit Tests promote developer confidence and helps protect against bugs.

When we code a new feature, we write two types of code:

1. Feature Code: Code that is the feature requested by our customer.
2. Automated Test Code: Code that tests the Feature code.

A single website that we  build will be covered by hundreds of unit tests. We run these tests several times a day while we are busy extending or modifying the client’s code. We use a test runner developed by JetBrains. Its output looks as follows:

Have a look at the output of the test run above:

• For the client project we ran 534 tests. It took less than a minute to run all the tests.
• Tests are nested underneath each other. Different sets of tests test different parts of the system.
• 1 test failed and the message at the bottom describes why. When a problem like this is detected it has to be fixed immediately.

The important thing to note here is that we always run all the unit tests, as opposed to only testing the small part of the system that the programmer is busy changing. This means that the entire system is tested several times a day for defects.

What we test

A software solution is made up of layers where top layers depend on bottom layers. For example (this is simplified):

• The User Interface layer displays web pages to the user. It relies on…
• The Business Logic layer that knows about business entities and how they behave. It relies on…
• The Data Access layer that knows how to interact with a database to save and load data.

One way to test would be to test the User Interface Layer and let it interact with the layers below it.

We do not test this way. Testing all the layers together leads to slow tests that are brittle and hard to configure. It also means that the causes of failures are difficult to track down because the bug could be anywhere in numerous layers.

We prefer to test layers and individual components in isolation. We do this by supplying fake implementations of the layers on which the code depends. For example:

Testing layers individually means our tests execute quickly and errors are very easy to pinpoint. We prefer writing lots of little tests as opposed to a few big ones.

A single “Business Logic” test may do the following things:

1. Configure “Fake Data Access” to return certain data when called – For example: return a fake Person when called.
2. Call a “Business Logic” operation – For example: calculate investment income for a Person.
3. Check result of the operation – was a calculation done correctly?
   3a.  Flag the test as “Failed” if the calculation did not yield the expected result.

Conclusion

We believe in writing unit tests because:

• Unit tests are much faster than manual testing. The entire system can be tested several times a day.  (although we obviously still do lots of manual testing to pick up issues that unit tests can’t detect)
• Unit tests give developers confidence to make changes and keep the system clean and easy to maintain.
• Unit tests save time (it’s very time consuming to find and fix broken features once the code has gone “Live”).
• Unit tests make everyone sleep better at night.

As Global Vision made the decision to develop the Adora front-end using Microsoft Silverlight (at the time a very brave move – Silverlight 1.1 was VERY limited compared to Silverlight 3!), New Media Labs was asked to provide some assistance in the development process.  With full-time in-house User Experience (UX) designers and interface designers from Stonewall+, this project has really brought to light the power and flexibility of the Silverlight platform.

Design of Adora

The application is built along a classic Model-View-ViewModel (M-V-VM) architecture and utilises a custom-built framework for this that supports all the required elements.  We had a number of technical challenges to overcome along the way, mainly related to data storage and using the same Domain model on both the Client- and Server-side.  Most of these issues are posts in their own right and will be covered in the near future.

Adora consists of many layers, each with its own set of technologies.  Starting from the client and working our way to the backend:

• Client
  Silverlight 3, utilising the Silverlight Toolkit and numerous custom-built controls.
• Services
  The application services are provided by WCF services that are hosted on the server.  These are communicated with using a custom-built data-contract serialiser.
• Data storage
  Data storage is provided by SQL Server 2008 and data goes through many stages of filtering and matching to ensure maximum data integrity.
• Messaging
  A messaging and scheduling engine resides on the server side to handle all email and SMS communications with consumers.  Messages received are linked to the consumer’s profile.

The Development Process

The goal is to deliver a product that is not only scalable, flexible and fast but is also pleasant to use with minimal or no user training.  With a team consisting of UX designers, developers, graphic designers and QA testers, we are continuously and iteratively tweaking the design.  Using an Agile approach has allowed us to make fast-paced design changes whilst ensuring that the end result not only looks good, feels good and performs well but also matches the expectations of the Product Owner.

We currently employ a mix of various established practices – test driven development, peer reviews, daily stand-ups and whiteboard sessions are some of the things that we do every day.

Interface Design

Adora’s interface borrows heavily from concepts used by Windows Explorer to allow users to work in a familiar environment.  For example, “folders” represent brands and “files” represent elements such as email lists, creatives or even communication schedules. 

Adora Explorer interface

Wireframe designs are created by our UX designers and the initial development work to implement functionality is done by the developers and testers.  Whilst this is taking place, the interface design is UAT-tested by the UX designers – the interface is then sent to Stonewall+ for “polishing” and styling.  Due to Silverlight’s dynamic binding model, it is simple for the developers to drop the finished interface on top of the functional elements built.

All work with consumer populations is graphical – that is, instead of reams of confusing numbers that may not appear to have any significant proportions or relationships, Adora presents interactive graphs.  A user with little or no SQL knowledge is able to build up complex queries very quickly with this system and it provides a “quick-glance” overview of the consumers that the user is currently communicating with. 

Marketing List builder interface design

Adora Query Language

Central to Adora’s success is our own query language that we have developed: Adora Query Language, or AQL.  With a syntax that is similar to SQL, AQL allows developers to retrieve consumer related information using queries that are business oriented.  If that doesn’t excite you, how about this:

SELECT firstname, lastname, age FROM brandname WHERE gender=’male’ AND age>25

The result of this query is a list of consumers within the “brandhouse” brand that are male and over 25 years of age.  A complete subset of SQL is supported, including ORDER BY, GROUP BY, INTERSECT, UNION and many more.  On top of this, we have our own extensions to the language that allow us to apply pagination to data results or to even group data into ranges.  The result?  Our developers can focus on working with data, not on retrieving data.  It is envisioned that future versions of Adora will allow 3^rd party vendors to create add-ins to the Adora system using AQL.

What’s next?

Still to come – details on the technical challenges faced and more screenshots!  Beta 1 is scheduled for release in January 2010. 

Repeat orders from existing clients, proving once again that NML is more than a supplier they are a partner that delivers.

Microsoft is now working in collaboration with New Media Labs to generate a case study out of this project. The project makes a good case study for Microsoft’s new Rich Internet Application technology known as Silverlight. This project is being billed as the largest Silverlight implementation in Africa.

Domain Driven Design was utilised extensively to model the specific banking domain. The Service Layer Data Contract and Service Contract models were modeled using the web service software factory.

The site has now moved into user acceptance phase and should be launched shortly.

Our base layer CMS was used for the development. Often when you tell clients you own a CMS they are skeptical – “why would you want to own or build a CMS when there are so many out there”. The reality is, a site like this is not only managing textual content it is managing the business content too. In other words you have to manage the data for each investment fund, the allocations of the fund and price performance over time. These are the kinds of systems NML has become experts in developing.

NML do not claim to own a packaged CMS or CMS framework. Packaged and Framework are words we’ve become skeptical of. We have a base layer of CMS code , that has been designed with integration in mind, to tie the company’s business model into the content management system. Our CMS allows for the usual features such as adding pages, changing navigation and publishing workflow.

The social elements are tied into the business model of this division of the bank, making this a complex development and integration exercise.

This will be NML’s second large Silverlight implementation and is being touted by Microsoft as the largest implementation in the EMEA region. Even thought this project is in the early stages it is being looked at closely as a case study by Microsoft.

Given the number of databases and stored procedures the system has to tie into Microsoft Repository Factory was the natural choice for Middle Tier. The service layer was developed using the Web Service Software Factory and the WCF .NET3.5 implementation technology.

The video content was originally going to be provided by youtube. But due to the complex authorisation workflow process brandhouse required, we decided to build our own video encoding and streaming service. The videos were then encoded using the Silverlight2.0 Encoder (apparently the first production purchase of this software in South Africa).

This was a very fun and exciting project. The application has come out of the testing phase so watch this space…

New Media Labs will provide architecture and development expertise on Global Vision’s Adora product. Adora is a marketing automation and CRM tool that will supercede the product built for Diageo (the consumer goods companies with portfolio of world-famous drinks brands).

The application will be built as a Rich Internet Application which will utilize the Silverlight presentation layer technology developed by Microsoft.

Adora is a marketing automation software providing:
1) Multichannel Campaign Management (Distribution)
2) Enterprise Feedback Management (Collection)
3) Business Intelligence (Analysis)

Adora’s goals are:
To become the leading marketing technology provider to CPG / FMCG businesses globally and to become a recognized leader in the marketing technology space as defined by number and caliber of customers, global footprint, industry awards, and industry analyst opinion.

1) Short term: make ADORA one of the top 5 Marketing software products in the Business-to-Consumer space worldwide within the next 3 years.

2) Medium term: make ADORA the accepted leading software product for businesses wishing to extend the marketing capabilities of MS CRM within the next 5 years.

3) Long term: make ADORA one of the top 5 marketing software products in the market worldwide in the next 8 years.