The Agile Balanced Scorecard

We recently published a nice article in the online journal "Projects@Work" on frameworks for Agile metrics. An excerpt is below.

Leading and Lagging Indicators
There are a number of frameworks for measuring performance on software development projects, including leading/lagging indicators, efficiency/effectiveness, and the Balanced Scorecard. Is it possible to combine the best of these approaches into a unified system that can be implemented on larger Agile projects?

The U.S. Federal Reserve Board keeps close tabs on two high-level measures of economic performance: GDP and inflation. While growing GDP and simultaneously keeping inflation in check are the end goals, the FRB looks at many other measures as well. That is because GDP and inflation are “lagging indicators” of economic performance - they confirm longer-term trends but cannot predict them. For example, it took macro economists a full year to figure out that we were already in the current recession. So it would be foolish to focus on only these numbers because by the time they indicate that a change is required, it is already too late.

Enter “the leading indicators.” Leading indicators are measures that often give early signals as to where the economy is heading. By looking at such measures as housing starts, durable goods orders, warehouse inventories, unemployment rates, etc., the FRB can see early trends in where GDP and inflation may be headed and decide what actions to take.

What does all of this have to do with Agile and software development? Managers in corporate settings also need to think about leading and lagging indicators of software development performance. They are expected to develop some foresight into what is to come and take proactive measures as appropriate. What metrics can act as early warning indicators of future software development performance - and which can tell us whether or not we were ultimately successful? We will discuss some potential leading and lagging indicators that may be used to assess software development success, particularly in the Agile development space.

Effectiveness vs. Efficiency

Yet another way to think about software development metrics is to consider the sometimes-opposing forces of effectiveness and efficiency. Efficiency measures are focused on how well we planned and used our resources to deliver our product. Budgets, schedules, labor hours, average time per software release, average spend per software release, SLOC, etc., are common examples of software efficiency metrics - here are many more. Effectiveness measures, on the other hand, indicate whether or not our IT solutions meet our business needs. Examples include customer acquisition and retention rates, customer satisfaction, revenue or cost savings, and profitability.

In developing a measurement system for IT projects, we need to think about how to assess the following two questions: “Did we build the right thing?” - our effectiveness - and “Did we build it the right way?” - our efficiency. If you build the right product at the right time, you will have a high likelihood of success and can then work to improve your efficiencies. But if you build a product that few customers want, you can be highly efficient and still suffer a massive loss.

IT has a history of putting more emphasis on efficiency metrics than on effectiveness metrics - but I believe that effectiveness trumps efficiency almost every time.

The Balanced Scorecard

Robert Kaplan and David Norton introduced the Balanced Scorecard approach to organizational strategy design and measurement in 1992. As the name suggests, the idea is that organizations need to balance their strategies and their measurement systems across a spectrum of parameters to achieve long-term success. Since then, the framework has grown into a strategic management system that can be scaled up and down the organization and also used for holistic organizational measurement across the following four basic categories.

Financial - Measures of financial performance. At the corporate level, these might be revenues, profit, earnings per share, etc.

Customer - Measures of customer satisfaction including customer growth, retention, satisfaction, equity, and others.

Innovation and Learning - Measures of internal skills and capabilities. Examples here might be the percentage of revenue from new products, level of education in our workforce, number of training hours delivered to employees, etc.

Internal Business Processes - Measures of processes usually in terms of efficiency such as time-to-market, inventory, throughput, defect rates, etc.

The Balanced Scorecard was designed to be scalable across the organization. That is, high-level corporate measures along these four categories could be pushed down to the business-unit level and functional levels, and also rolled up and integrated organizationwide. In this system, individual departments could contribute to financial performance, customer satisfaction, internal business process improvement and innovation and learning. These four categories are just suggestions, and organizations should modify them to fit their particular needs.

An Agile Balanced Scorecard

All three of these frameworks - leading and lagging indicators, efficiency and effectiveness, and the Balanced Scorecard - have merit. But is it possible to achieve a holistic view across all of these important measurements and make it work at the program and project level where most of the work actually gets done?

The Agile Balanced Scorecard takes the best of these approaches to form a unified measurement system at the project and program level. This is done by modifying its traditional four categories to better align with the way we typically think about projects. Our categories are:

Product: Are we building high-value products that are in demand by our customers?

Financial: Are we achieving planned-for financial results?

Team:Are we working well together as a team and addressing the needs of the various stakeholders?

Schedule:Are we on track to deliver within the schedule commitments?

You could stick with the original four categories of the scorecard, use these modified categories, or define your own when setting up your own scorecard.

For more details and some sample metrics, see the article at Projects@Work.

How Not to Do Resource Management

Every now and then, we will get the interesting question of how to do "resource management" in an enterprise that is adopting Agile. "How can I allocate people 100% to a single team for so long when we have so many projects going on?"   I like to reply that this is actually the wrong question.  Instead of answering the question of how to do resource management, we like to say that the right question is "How can I not do resource management?"  How can I avoid it altogether or at least greatly reduce it?

There are 2 problems in the question above.   The first is the assumption that we need to move people around to staff projects.  The second is related to the number of simultaneous projects.

Moving People Around

There is a tendency to think of projects as the primary unit of organization.   In this view, the project is static and we move people and resources to the project in order to staff it.  When the project is finished, we return the people and resources to the central pool and make them available for the next project.   There are several issues here mostly related to the pe

ople side of things.   Treating people as individual pieces that can be moved around like pieces on a chessboard does not go a long way towards creating high performance teams.  You will recall that teams go through a life cycle of maturity:  forming, storming, norming, performing, and all of that, and this maturation process takes time.  By the time the team is really performing, usually after quite a few months, the project begins to wind down and we break the team up and send staff off to other projects where they have to start the process all over again.  As soon as we get a high performing team we break it up!

In the Agile world, it is common to think of the team as the central unit of organization and we move projects to the team.   In this model, teams go through the maturation process but once they achieve a high performing state, we keep them working together for as long as possible. The team members know each other well, they know their strengths and weaknes

ses, they have learned how to communicate and collaborate with one another and they have hopefully worked out their major differences.  Why would you want to break that up?  Instead, look at the projects that are in the queue and assign the project to the team that is in the best position to deliver from a capacity and skills perspective.

Specialists

Some projects demand the use of specialists who are in limited supply.  In those cases, you may need to share specialists across many teams and the teams will need to adjust their Agile schedules around the availability of the specialists.   Moving only the specialists around is a lot easier than shuffling everyone.  And oh, by the way, if you don't have enough specialists to serve all of the projects, then you have an obvious bottleneck that needs to be addressed.

Too Many Projects

Finally, the biggest problem in most organizations is too many projects and too little focus. Just like having too many cars on the highway, having too many projects in flight is a guarantee for SLOW DELIVERY.  There is plenty of inventory management and queueing theory to support this and we all know it intuitively.   Most IT organizations have a lot of projects going on, hence a lot of spending going on, but not a lot of delivery happening.  Each project (or car) is inching slowly along, fighting for attention and limited resources.  The solution of course is take some cars off of the highway so that the remaining cars can fly!  But that requires the political strength to say "I'm sorry but we are fully allocated and we cannot support your project right now.  We will queue it up for the next available team."  Easy to say, harder to do. This takes strong, decisive management to truly manage demand against capacity in order to keep from overloading the system.  Agile can help however.   Since Agile does require that teams spend a significant amount of time together, it is difficult for them to support many simultaneous projects.  Agile teams are better able to focus and get the project done quickly.   By moving towards Agile on a large scale in your organization, projects will have to be better prioritized and some will have to go into the queue while others get the focus they deserve.  And this is a great step towards achieving throughput and delivery instead of high-spend and never-ending projects.

Mistake Proofing and The Role of QA

Mistake Proofing

Sometimes, we find that we've been doing something for so long, that the way that we do it becomes taken as as a given.  Such is the relationship between software developers and QA testers.  It has unfortunately become standard practice for the developers to write the code and the testers to find the bugs.   This is a common although inefficient method of quality assurance in which defects are allowed to enter the system and then testing is used to try to discover them and have them removed.  A far more efficient and effective method is to borrow from the concept of "mistake proofing" from lean manufacturing systems.   The idea behind Mistake Proofing is to take steps to help ensure that defects never enter the system in the first place.   This leads to higher quality of course but also to lower testing times, lower testing resources, and less re-work.   

In the waterfall world with its functional silos, it is natural for development and QA to have a somewhat adversarial relationship at times.  This is part of the nature of big hand-offs across functional boundaries.  But as we all know, on Agile teams, testers are first-class members of the team who should be working very closely with developers, analysts, etc.   But what exactly is the nature of that work?   How exactly do developers and testers work closely together?   I feel that one of the primary goals should be mistake-proofing.   Mistake-proofing on a software development project involves:

a)   Having developers and testers and analysts discuss feature requirements in great detail so that they all develop a common understanding of the requirements.

b)  Discussing in some detail how exactly we are going to test each feature.  Here is where the tester can add great value to the development process and help in the mistake-proofing process.  The tester tells/discusses with the developer:

• The test scenarios that will be used
  
• The test data that will be used
  
• The boundary conditions that exist and what should happen in those cases
  
• Any use-cases or other work-flow paths that will be taken in order to get to the test-case at hand. 
  
• What the system should do in the event of an error condition
  
• Test automation needs
• etc

In discussing the requirements and testing at this level of detail, we begin to treat test cases as first class requirements that the developer must take into account.   By knowing exactly how the code will be tested, the developer can write code to pass the tests and thereby keep defects from ever entering the system.   This is like getting the questions to the final exam before the final exam;  you ought to be able to pass a test if you know the questions in advance. 

The Role of QA

In this model, the QA role transforms from finding defects to:

1. Specifying how to keep defects out of the system.   
   
2. Providing independent verification that the system works per the requirements
   

 The result; fewer functional defects.  And fewer functional defects means less time and resources spent finding them, tracking them, fixing them, and re-testing them. QA should then be able to focus more of their time on other higher value testing such as integration testing, end-to-end testing,  etc., which are often short-changed in the pursuit of functional defects.