Risk management is a critical part of project management. However, in the real world, it's simply not cost-effective to manage risks thoroughly over all aspects of the projects. Similarly, when configuring the project management processes, we have to determine which areas deserve more attention than others. The more risks in a given area, the more controls we need.

When initiating a project, I always thought it would be great if we had some kind of guidelines as to what areas we should focus on based on the characteristics of the project. Indeed if we look at a project management method such as PMBOK, RUP, or Prince2, we notice it's rather exhaustive, if not abstract, compared to our specific needs. The task of customizing the method according to our needs can be daunting. Hence the idea to look at how risks affect different project management areas based on characteristics. More generally, the characteristics typically translate into increased workload, higher occurrence of changes, and of course increased risks.

For example, a project with a large team might require more work in managing communications, schedule, costs, human resources, and execution, while a project with high technical complexity will require special attention to manage quality, schedule and costs.

The following table summarizes my first thoughts on the subject.

* Relatively to team's experience.

Low indicates that the area deserves some specific attention. High indicates that it requires great attention. Otherwise (no indication), the are doesn't deserve any special attention.

IT governance is about the stewardship of IT resources on behalf of the stakeholders who expect a return from their investment. It comes from the need that "corporate managers should be working on behalf of shareholders to allocate business resources to their optimum use (Alan Greenspan)."

Governance drives the need for compliance. The compliance requirements can be either internal or external to the organization. Internal compliance requirements refer to a set of rules that the organization defined for itself based on the belief that it will reduce risks and increase performance. For example, an organization could decide that it has to comply with CMMI level 3, in which case it will set this standard as an internal compliance requirement. On the other hand, external compliance requirements are usually imposed by external bodies such as governments. For example, a government can mandate organizations to comply with a set of rules such as Sarbanes-Oxley (SOX).

Ultimately, governance is a top-down approach driven by financial results and reporting. It is supposed to have financial impact and is defined at the corporate level and implemented throughout the lower levels. SOX is an excellent example. The goal of SOX is to ensure the accuracy of financial reports. We might wonder what's the relationship between SOX and IT governance? Given the critical role that IT plays in organizations, it is not difficult to imagine (see Wikipedia: IT controls and the Sarbanes-Oxley Act).

To comply with Sarbanes-Oxley, organizations must understand how the financial reporting process works and must be able to identify the areas where technology plays a critical part. In considering which controls to include in the program, organizations should recognize that IT controls can have a direct or indirect impact on the financial reporting process. For instance, IT application controls that ensure completeness of transactions can be directly related to financial assertions. [...] Application controls are generally aligned with a business process that gives rise to financial reports.
- Wikipedia, Information technology controls

We can ask ourselves how can an organization adopt a structured approach to implement controls in order to comply with regulations such as SOX? To do so, it can rely on popular IT governance frameworks, models, standards, or collections of best practices such as:

• IT Infrastructure Library (ITIL) focuses on services management.
• Control OBjectives for Information and related Technology (COBIT) provides a set of generally accepted information technology control objectives for day-to-day use by business managers and auditors.
• ISO 27001 and ISO 27002 essentially focus on the IT security. ISO 27002 and COBIT complement each other.
• CMMI is a model for process improvements. CMMI-DEV specifically addresses product and service development processes.

However, these IT governance frameworks only describe what should be done to address IT governance needs, in the forms of criteria, objectives, levels, and so on that express compliance requirements. Therefore, organizations have to translate these requirements in practices for their specific context. Real-world implementation includes business processes, technologies, information systems, and so on. One way to help implement the required controls is to rely on product and service development governance methods (also called project governance). The two most popular product and service development methods are Project Management Body of Knowledge (PMBOK) and Projects in Controlled Environments (PRINCE2). PMBOK and Prince2 are widely regarded as standard project management methods, PRINCE2 being used mostly in Western Europe and PMBOK predominantly in North-America.

Today is your first day on a consulting mission to assess software development practices. Okay, you have experience, you know a bunch of methodologies, and you are familiar with standards, but what would be your very first actions as a newcomer who doesn't know anyone or anything in this organization?

Tough spot.

Here's a hint: it's all about people.

At first sight it may seem surprising that the success of assessments and audits depends more on the ability to relate with people than on the knowledge of methodologies or standards. Although the later is important, it doesn't matter how good you are at it if you can't create good relationships with people. Indeed, to assess you need information, and to collect information you need people to collaborate with you - willingly.

With this in mind, let's see what steps we can take to start this mission.

Meet with the sponsor to ensure commitment and understand the mission

• Gain understanding of the business objectives behind the assessment.
• Establish the scope of the mission, including what has to be assessed and what will not be assessed.
• Identify the stakeholders and how to access them.
• Agree on the deliverables and on the planning.
• Determine the mission's constraints, such as deadline and costs.
• Summarize the meeting with a document, ideally a project charter.

Pitfalls:

• Don't assume that the description of the mission you received beforehand is accurate, even if it is detailed.
• Be ready to challenge the sponsor's assumptions. The sponsor might have a preconceived idea of the assessment process and even of the result. Kindly remind him that you have been called to perform an independent review, not to find evidence for predefined results.
• Avoid stretching the scope based on what you discover. If the assessment points to issues beyond the scope, report this finding to the sponsor.
• Don't confuse the official organization's structure (titles and functions) with the way people interact in reality (roles, influence, and informal channels of communication). Create your own view of the structure and analyze how stakeholders interact with each other.

Meet with the stakeholders to create a sound foundation for collaboration

You have to defuse resistance. The second you step in, the resistance bomb starts ticking. In order to avoid having to overcome resistance later on, take the initiative to communicate.

• Explain why the assessment has been requested.
• Explain how the assessment will be carried out.
• Emphasize the opportunity for improvement and personal development that the assessment will create.
• Make sure that everyone understands that the assessment's objectives are to improve results, not to evaluate people. All information collected are anonymous.
• Be part of the team. Use "we" and "us", not "I" and "you".
• Handle questions.
• Summarize the meeting with a document and send it to all stakeholders to demonstrate willingness to collaborate and transparency.

Pitfalls:

• Don't base your communication on emails or documents. Written communication is used to prepare (for example to present a meeting's agenda) and to report (for example to summarize a meeting's conclusion), not to create trust or enable collaboration. No amount of documents can replace an in-person kick-off.
• Don't communicate without prior consent from the sponsor. Review the content with him beforehand.
• Don't introduce yourself as being called by executives. It will create suspicion and hinder the collection of information.
• Don't provide too much details. Keep it simple. Provide some additional details when answering questions.

Identify the sources of information required to perform the assessment

• Break down the mission's objective in subgoals in order to list the information you will need, then identify people who can help you with each category of information.
• Focus on people as the primary source of information.
• Contact each person individually to personalize communication and to confirm the person's roles and availability to help with the assessment.
• Ask for help to locate documentation.

Pitfalls:

• Don't trust documentation. Documents are seldom current, and what they describe seldom reflects reality.
• Be wary of people who systematically send you documents to avoid meeting with you in person.
• Don't waste your time parsing the files by yourself to find documentation. Ask stakeholders for help.

Be accessible

• Provide your contact information.
• Set up your office among the team. If you have a separate office, keep the door open.
• Be available.

Pitfalls:

• Don't become overly friendly. It might compromise your objectivity and your credibility.

There is a confusion between the terms audit and assessment. Actually I was confused by these two words until not so long ago, so I figured it might help to shed some light on this ambiguity.

An audit is a process to verify the conformance of an entity to a given standard. It consists in gathering evidence of conformance or nonconformance. Therefore, it results in a success or a failure with regard to the scope of the audit. Verification and inspection are synonyms of audit.

Example of audit: Verify that the project management practices comply with a set of best practices defined by an instance of the Project Management Body of Knowledge (PMBOK).

An assessment, on the other hand, evaluates the efficiency and/or effectiveness of an entity and results in a measure of its performance with regard to the scope of the assessment. The goal of the assessment is to gain understanding in order to formulate a recommendation to improve the entity's situation. Appraisal and evaluation are synonyms of assessment.

Example of assessment: Evaluate the CMMI level of an entity.

Note that both audit and assessment can use the same base material, such as a given project management methodology, but their goal and the way they use the reference material will differ.

We can also distinguish audits from assessments by looking at:

• Attribution: An audit might explicitly point at specific people or groups of people as the cause for noncompliance (attribution). An assessment does not evaluate individuals (non-attribution). Of course, the result of an assessment might still be used to infer responsibility for failure or low scores.
• Fault: Whereas an audit results in a success or a failure (fault), an assessment usually provides a score that does not express success or failure (no-fault). With an assessment, one can freely decide which score constitutes a success or a failure if they want to, but this is done to identify the areas that should be improved and not to judge people.

As you can see, the difference doesn't lie only in the goal and in the process, but also in the mindset.

Additionally, we have to mention that the two words audit and assessment generate very different reactions. Audit is associated with distrust and punishment whereas assessment is often seen as an opportunity for improvement. Unfortunately I have personally witnessed audits disguised as assessments. Instead of resulting in a healthy improvement plan the results have been used to blame people and terminate projects.

Just like with best practices and professional certifications, most of us have a love/hate relationship with key performance indicators (KPI). It's certainly compelling to be able to measure how good we are at delivering projects. But the non-repeatability and variability of projects makes it elusive at best. Nonetheless I decided to give it a try in order to better structure the software development audit methodology I'm currently working on.

First let's have a look at the characteristics that every KPI should have. The SMART criteria really help:

• Specific (= objective)
• Measurable (= easy to measure)
• Attainable (= appropriate range)
• Relevant (= directly linked to business goals)
• Time-bound

For example, "the number of projects over budget vs. total number of projects, during the last month" would make a decent KPI.

Second, we categorize the KPI. I've found that the balanced scorecard approach is very useful. Instead of applying it to an entire organization, we can apply it to the department/team delivering software solutions, or even to a single project. The balanced scorecard is composed of four perspectives. In our case, we assume that the overarching goal of the organization is to achieve sustainable profitability, therefore we place the finance perspective at the top. In order to achieve this goal, we need to satisfy - better, to delight - our customers (whether they are internal or external to the organization), which is represented by the customer perspective. Assuming that the quality of results tends to reflect the quality of processes and the quality of people, we understand that the internal processes and learning and growth perspectives make that possible.

Third, we identify KPI for each perspective:

Financial

• ROI, short-term
• ROI, long-term
• Cash flow

Customer

• Value: value is more to the point than scope. Scope refers to compliance to a project contract, whereas value refers to how well the delivered projects does what it's suppose to do in terms of business value. Plenty of projects deliver the agreed-upon scope but fail to deliver the value they were supposed to.
• Quality
• Timeliness
• Relationship and communications

Internal processes

• Quality management
• Cost, schedule, and scope management: although being on-cost, on-budget, and on-schedule does not garantee project success (see also the value KPI), I believe that these should be managed properly. Note that this KPI is about management and not compliance.
• Productivity
• Collaboration and communications

Learning & growth

• Team satisfaction
• Process improvement
• Technical excellence

That's the easy part. Now we should identify indicators that are directly linked to each KPI. Indeed these KPI are high-level. We need indicators that are closer to reality and easier to measure. If we look at the ROI KPI for example, we might define underlying indicators that measure how well we calculate ROI with standard methods, at the right time, and compared to other projects in same or other organizations. The ROI KPI would be composed of these underlying indicators.

It's a work in progress. More on this in later posts.