Dealing with Shadow IT

“There’s this guy on the 14th floor who created a giant Excel spreadsheet with macros and pivot tables to produce sales statistics. That was a few years ago and now the thing – or rather different versions of it – spread to several places. Can you believe top management is actually using these numbers? And, guess what, they’re quite happy with it. Yet it seems no one knows precisely where the numbers come from.”

Shadow IT refers to IT components that are considered illegitimate in an organization. Note that legitimacy is a point of view, usually the IT department’s.

How are shadow IT assets born?
In order to deal with shadow IT, we need to understand why and how it’s created. The creation of shadow IT assets can either be spontaneous or planned.

• Spontaneous: the culprit does not realize he's creating shadow IT. Such assets are usually created or brought in on the spur of the moment. Typical spontaneous shadow IT assets include Excel spreadsheets, Access databases, reports and extractions from legitimate IT applications.
• Planned: the perpetrator intentionally bypasses IT department and related compliance requirements. The justification is usually that IT won't do it, can't do it, is too slow, expensive, or simply has bad history. And anyway once it’s there IT will have to deal with it, possibly even maintain it, however unhappy it makes them.

Strengths and weaknesses of shadow IT
Wikipedia does a good job of summarizing the positive and negative characteristics of shadow IT. I’d like to emphasize two aspects, however.

First, shadow IT is entirely at odds with the relatively recent trend – and sometimes mandatory requirement – to comply with standards such as Sarbanes-Oxley (SOX), because it prevents tracing the data that ends up in financial statements. As a consequence, organizations in which shadow IT is involved in any data flow impacting financial statements (which is most IT applications) take a huge risk of making decisions based on incorrect data, not to mention not complying with regulations. IT being instrumental in most business processes, one might argue that shadow IT could be the single biggest threat to corporate governance.

Second, the cost of shadow IT is difficult to evaluate and therefore grossly underestimated. Shadow IT being, well, in the shadow, its cost – whether implementing assets or maintaining them – doesn’t show up explicitly in any financial or managerial accounting document. If it did, no doubt many managers who allowed it in the first place would have decided against it. As you know, if you can’t see the cost it looks practically free.

Dealing with shadow IT
It would be tempting to simply outlaw all IT done outside of IT department in an organization. But for reasons mentioned above it would create a lot of frustration and wouldn’t work in the end anyway. Shadow IT is bad. Some shadow IT assets should never have been created. Others deserve being created but should have been implemented within the rules so avoid several of the problems they now cause.

Consequently, dealing with shadow IT requires a two-pronged approach:

• Dealing with existing shadow IT assets
• Preventing the creation of shadow IT assets

1. Dealing with existing shadow IT assets

For each shadow IT asset (or part of, or category of), perform a cost-benefit analysis that should minimally include the three following alternatives:

1. Retire shadow IT asset without replacing it
2. Migrate features of shadow IT asset to existing legitimate IT asset
3. Make shadow IT asset legitimate by making it compliant

Criteria typically include cost (migration/compliance, and subsequent maintenance costs), timeframe, and loss of business/user benefits (mainly in case of retirement without replacement).

2. Preventing the creation of shadow IT assets

To prevent the creation of shadow IT assets that are knowingly created (“planned”), we have to remove the excuses for them being created in the first place. One way of seeing it is that shadow IT is competing with legitimate IT.

• Simple and fast intake process to make it easier for the requestor to have her initiative evaluated and started in the legitimate IT asset creation process
• Make sure solution truly satisfies business needs. Instead of the typical client-provider relationship, client is involved during the whole process.
• Competitive/transparent pricing and delivery timeframe shows that IT deserves its position as preferred provider of IT in the organization.
• Open up to external providers if need is justified but IT can’t do it within acceptable budget or timeframe.

Project management obviously has a big role to play in preventing shadow IT. Indeed these aspects are part of the many responsibilities of project management.

To prevent the creation of shadow IT assets that are spontaneously created, we have to inform potential perpetrators.

• Define what is an IT asset vs. a simple document. Is an Excel spreadsheet an IT asset? It would be absurd to think that Excel documents would be created and managed only by IT people. But what if it has a macro that connects to a database? That’s more tricky. It’s important that the IT authority determines objective criteria that define IT assets and communicate these criteria throughout the organization.
• Audit: small, spontaneously created shadow IT assets usually live under the radar. Therefore it’s important to frequently audit the organization in order to identify shadow IT assets.

If shadow IT assets are created despite above measures, there is no excuse: penalize it. For example, do not provide support or maintenance, and do not allow legitimate IT assets to interfacing with it. It sounds harsh but it’s important to have strong deterrents. There will always be people considering “what would happen if I do it anyway?” with total disregard for shadow IT costs and risks.

Pervasive IT
There’s something wrong in what I’ve written above. It assumes the traditional view that IT assets should be created and managed solely by and within an IT department, and that other departments strictly have client status and don’t know or do any IT.

I believe that this situation is becoming increasingly uncommon and is at odds with important trends in IT. Consider that there is a grey zone between IT skills and non-IT skills. Many people who are not in IT possess the skills and the tools to create and modify IT assets on their own. Instead of preventing them to do it, what stands for the IT authority should leverage it. If this point of view doesn’t convince you, consider that many of the best software products have been created by users of these products who happened to have the IT skills to develop them.

In order to enable non-IT people to create and manage IT assets, we have to dissociate the process of creating and managing IT assets from the people who are doing it. In other words, we have to acknowledge that IT is becoming pervasive in the organization.

In this model, the IT authority is the owner of a process that makes sure IT assets are created and managed according to standards. Strategic areas representing the compliance standards such as enterprise architecture, IT security, and others should be under direct supervision of the IT authority, but other roles such as analysis, development, and testing could be taken over by clients and users for the best result.

No doubt there is much more to say and to write about shadow IT. I’ve walked most of my career in IT and management witnessing the awkward, even distrusting relationship between IT and the rest of the organization. I believe that shadow IT is the most tangible consequence of this poor relationship, which is why it’s so important to address it.