There is a confusion between the terms audit and assessment. Actually I was confused by these two words until not so long ago, so I figured it might help to shed some light on this ambiguity.

An audit is a process to verify the conformance of an entity to a given standard. It consists in gathering evidence of conformance or nonconformance. Therefore, it results in a success or a failure with regard to the scope of the audit. Verification and inspection are synonyms of audit.

Example of audit: Verify that the project management practices comply with a set of best practices defined by an instance of the Project Management Body of Knowledge (PMBOK).

An assessment, on the other hand, evaluates the efficiency and/or effectiveness of an entity and results in a measure of its performance with regard to the scope of the assessment. The goal of the assessment is to gain understanding in order to formulate a recommendation to improve the entity's situation. Appraisal and evaluation are synonyms of assessment.

Example of assessment: Evaluate the CMMI level of an entity.

Note that both audit and assessment can use the same base material, such as a given project management methodology, but their goal and the way they use the reference material will differ.

We can also distinguish audits from assessments by looking at:

• Attribution: An audit might explicitly point at specific people or groups of people as the cause for noncompliance (attribution). An assessment does not evaluate individuals (non-attribution). Of course, the result of an assessment might still be used to infer responsibility for failure or low scores.
• Fault: Whereas an audit results in a success or a failure (fault), an assessment usually provides a score that does not express success or failure (no-fault). With an assessment, one can freely decide which score constitutes a success or a failure if they want to, but this is done to identify the areas that should be improved and not to judge people.

As you can see, the difference doesn't lie only in the goal and in the process, but also in the mindset.

Additionally, we have to mention that the two words audit and assessment generate very different reactions. Audit is associated with distrust and punishment whereas assessment is often seen as an opportunity for improvement. Unfortunately I have personally witnessed audits disguised as assessments. Instead of resulting in a healthy improvement plan the results have been used to blame people and terminate projects.

Just like with best practices and professional certifications, most of us have a love/hate relationship with key performance indicators (KPI). It's certainly compelling to be able to measure how good we are at delivering projects. But the non-repeatability and variability of projects makes it elusive at best. Nonetheless I decided to give it a try in order to better structure the software development audit methodology I'm currently working on.

First let's have a look at the characteristics that every KPI should have. The SMART criteria really help:

• Specific (= objective)
• Measurable (= easy to measure)
• Attainable (= appropriate range)
• Relevant (= directly linked to business goals)
• Time-bound

For example, "the number of projects over budget vs. total number of projects, during the last month" would make a decent KPI.

Second, we categorize the KPI. I've found that the balanced scorecard approach is very useful. Instead of applying it to an entire organization, we can apply it to the department/team delivering software solutions, or even to a single project. The balanced scorecard is composed of four perspectives. In our case, we assume that the overarching goal of the organization is to achieve sustainable profitability, therefore we place the finance perspective at the top. In order to achieve this goal, we need to satisfy - better, to delight - our customers (whether they are internal or external to the organization), which is represented by the customer perspective. Assuming that the quality of results tends to reflect the quality of processes and the quality of people, we understand that the internal processes and learning and growth perspectives make that possible.

Third, we identify KPI for each perspective:

Financial

• ROI, short-term
• ROI, long-term
• Cash flow

Customer

• Value: value is more to the point than scope. Scope refers to compliance to a project contract, whereas value refers to how well the delivered projects does what it's suppose to do in terms of business value. Plenty of projects deliver the agreed-upon scope but fail to deliver the value they were supposed to.
• Quality
• Timeliness
• Relationship and communications

Internal processes

• Quality management
• Cost, schedule, and scope management: although being on-cost, on-budget, and on-schedule does not garantee project success (see also the value KPI), I believe that these should be managed properly. Note that this KPI is about management and not compliance.
• Productivity
• Collaboration and communications

Learning & growth

• Team satisfaction
• Process improvement
• Technical excellence

That's the easy part. Now we should identify indicators that are directly linked to each KPI. Indeed these KPI are high-level. We need indicators that are closer to reality and easier to measure. If we look at the ROI KPI for example, we might define underlying indicators that measure how well we calculate ROI with standard methods, at the right time, and compared to other projects in same or other organizations. The ROI KPI would be composed of these underlying indicators.

It's a work in progress. More on this in later posts.

Developing the agile way doesn't necessarily mean that your clients will suddenly adhere to the agile philosophy. In particular, clients are used to buy a product developed for a fixed price and delivered at a predetermined date. Working with agile, we commit to a predetermined budget and delivery date, but we agree that the scope can (read: will) change. It's no wonder that at first sight agile sounds like a bad deal to the clients. This is also the reason that most contracts remain traditional and still mention the three traditional constraints of scope, budget, and deadline. It also demonstrates that most clients do not care whether we develop with agile or not. They want what they ordered, period.

In many situations, the product is developed with an agile framework, but the contract remains traditional, creating a gap between the team's objectives (delivering value that satisfies the client) and the client's expectations (adhering to the terms of the contract). This road is heading for conflict.

But then, what happens when the effort for a feature has been underestimated to the point that the agreed-upon scope cannot be completed within budget and schedule? How can we solve this (common) problem?

Read more »

Just like I mentioned the importance of discipline in Lean in a previous post, in his book Agile Project Management: Creating Innovative Products Jim Highsmith expresses why discipline is a key enabler for agile.

Some critics consider agile methods to be ad hoc, undisciplined, and, ultimately, technically inferior. They are wrong. First, for traditionalists, who thrive on process, procedure, and documentation, the informality of agile methods is interpreted as undisciplined. Nothing could be further from reality. Discipline is doing what you said you would do. Using this definition, many organizations with elaborate processes fail - they have volumes of processes, procedures, forms, and documents that are systematically ignored. Agile organizations have less-elaborate process structures, but they tend to follow the ones they have.
- Jim Highsmith

There is much discussion as to how to transition from the Project Management Body of Knowledge (PMBOK) to Agile or - and that's wiser - how to combine them into a sensible solution.

I just wanted to share links to articles written by Michele Sliger:

Relating PMBOK Practices to Agile Practices - Part 1
Relating PMBOK Practices to Agile Practices - Part 2
Relating PMBOK Practices to Agile Practices - Part 3
Relating PMBOK Practices to Agile Practices - Part 4

Although we can map agile practices to PMBOK processes and activities, it is a futile effort. Indeed the key difference resides in the approach: PMBOK is process-centric whereas agile is people-centric. To give you an idea, topics such as team dynamics and motivation make up only 20 pages of the 300+ page (without appendix) PMBOK guide, and the word "collaboration" appears only twice.