Shadow IT: Balancing the Risks and Benefits Every Organization Faces
Abstract
Self-service reporting is the promised land of the day. Whether it’s Tableau, Cognos Analytics, Qlik Sense, or another analytics tool, all vendors seem to be promoting self-service data discovery and analysis. With self-service comes Shadow IT. We posit that all organizations suffer from Shadow IT lurking in the shadows, to one degree or another. The solution is to shine a light on it, manage the risks and maximize the benefits.
Overview
In this white paper we’ll cover the evolution of reporting and the dirty secrets that no one talks about. Different tools require different processes. Sometimes even ideologies. Ideologies are “the integrated assertions, theories and aims that constitute a sociopolitical program.” We’re not going to get sociopolitical but I can’t think of a word to convey a business and IT program. I would consider the Kimball-Inmon database divide an ideological debate in a similar way. In other words, your approach, or the way you think, drives your actions.
Background
When the IBM 5100 PC was state of the art, $10,000 would get you a 5-inch screen with a built-in keyboard, 16K RAM and a tape drive 
weighing in at just over 50 pounds. Suitable for accounting, this would be connected to a free-standing disk array the size of a small filing cabinet. Any serious computing was still done via terminals on a mainframe timeshare. (image)
“Operators” managed the daisy-chained PCs and controlled access to the outside world. Teams of operators, or later-day sysadmins and devops, grew to support the ever-growing technology. The technology was big. The teams that managed them were bigger.
Enterprise management and IT-led reporting have been the norm since the beginning of the computer era. This ideology was built on the stodgy, conservative approach that “The Company” manages the resources and will provide you with what you need. If you need a custom report, or a report in a timeframe that was out-of-cycle, you need to submit a request.
The process was slow. There was no innovation. Agile didn’t exist. And, like the ancient clerical pool, the IT department was considered overhead.
In spite of the downsides, it was done for a reason. There were some benefits to doing it this way. There were processes in place which everyone followed. Forms were completed in triplicate and routed through interoffice mail. Data requests from throughout the organization were sorted, shuffled, prioritized and acted upon in a predictable manner.
There was a single data warehouse and a single enterprise-wide reporting tool. Canned reports created by a central team provided a single version of the truth. If the numbers were wrong, everybody worked from the same wrong numbers. There’s something to be said for internal consistency. 
Management of this way of doing business was predictable. It was budgetable.
Then one day 15 or 20 years ago, all that exploded. There was a revolution. Computing power expanded. Moore’s Law – “the processing power of computers will double every two years” – was obeyed. PCs were smaller and ubiquitous.
More companies started to make decisions based on data rather than the gut instincts they had used for so many years. They realized that the leaders in their industry were making decisions based on historical data. Soon the data became near real time. Eventually, the reporting became predictive. It was rudimentary at first, but it was the beginning of using analytics to drive business decisions.
There was a shift to hiring more data analysts and data scientists to help management understand the marketplace and make better decisions. But a funny thing happened. The central IT team did not follow the same trend as the shrinking personal computers. It did not immediately become more efficient and smaller.
However, in response to the decentralized technology, the IT team also began to become more decentralized. Or, at least roles that had traditionally been part of IT, were now part of business units. Analysts who understood data and the business were embedded in every department. Managers started asking their analysts for more information. The analysts, in turn, said “I’ll need to fill out the data requests in triplicate. The earliest it will be approved is at this month’s data prioritization meeting. Then it may take a week or two for IT to process our request for data – depending on their workload. BUT,… if I could just get access to the data warehouse, I could run a query for you this afternoon.” And so it goes.
The shift to self-service had started. The IT department eased its grip on the keys to the data. Vendors of reporting and analytics began to embrace the new philosophy. It was a new paradigm. Users found new tools to access data. They discovered that they could bypass the bureaucracy if they just got access to the data. Then they could perform their own analysis and reduce turnaround time by running their own queries.
Benefits of self-service reporting and analytics
Providing direct access to the data to the masses and self-service reporting solved a number of problems, 
- Focused. Purpose-built tools which were easily accessible replaced a single, dated, multi-purpose legacy reporting and analytics tool to support all users and answer all questions.
- Agile. Previously, the business units were hampered by poor productivity. Access to only last month’s data led to the inability to work agilely. Opening up the data warehouse shortened the process allowing those closer to the business to function more rapidly, discover important trends and make decisions more quickly. Thus, increased velocity and value of data.
- Empowered. Instead of users having to rely on the expertise and availability of others to make decisions for them, they were given the resources, authority, opportunity, and motivation to do their work. So, users became empowered using a self-service tool which could free them from the reliance on others in the organization for both access to the data and the creation of the analysis itself.
Challenges of self-service reporting and analytics
However, for each problem self-service reporting solved, it created several more. The reporting and analytics tools were no longer managed centrally by the IT team. So, other things that weren’t problems when a single team managed reporting became more challenging. Things like quality assurance, version control, documentation and processes like release management or deployment took care of themselves when they were managed by a small team. Where there were corporate standards for reporting and data management, they could no longer be enforced. There was little insight or visibility into what was happening outside of IT. Change management was non-existent. 
These departmentally controlled instances functioned like a shadow economy which refers to business that occur ‘under the radar’, this is Shadow IT. Wikipedia defines Shadow IT as “information technology (IT) systems deployed by departments other than the central IT department, to work around the shortcomings of the central information systems.” Some define Shadow IT more broadly to include any project, programs, processes or systems which are outside of the control of IT or infosec.
Whoa! Slow down. If Shadow IT is any project, program, process or system that IT doesn’t control, then it’s more pervasive than we thought. It’s everywhere. To say it more bluntly, every organization has Shadow IT whether they acknowledge it or not. It just comes down to a matter of degree. An organization’s success in dealing with Shadow IT depends largely how well they address some key challenges. 
- Security. At the top of the list of issues created by Shadow IT is security risks. Think macros. Think spreadsheets with PMI and PHI emailed outside of the organization.
- Higher risk of data loss. Again, because of inconsistencies in implementation or processes, each individual implementation may be different. This makes it difficult to prove that established business practices are being followed. Furthermore, it makes it difficult even to comply with simple audit requests of usage and access.
- Compliance issues. Related to audit issues, there is also an increased likelihood of data access and data flows, making it more difficult to comply with regulations like Sarbanes-Oxley Act, GAAP (Generally Accepted Accounting Principles), HIPAA (Health Insurance Portability and Accountability Act) and others
- Inefficiencies in data access. Even though one of the problems which distributed IT tries to solve is speed to data, unexpected consequences include hidden costs to non-IT workers in finance, marketing, and HR, for example, who spend their time debating the validity of data, reconciling to their neighbor’s numbers and trying to manage software by the seat of their pants.
- Inefficiencies in process. When technology is adopted by multiple business units independently, so, too, are the processes related to their use and deployment. Some may be efficient. Others not so much.
- Inconsistent business logic and definitions. There is no a gatekeeper to establish standards, inconsistencies are likely to develop because of a lack of testing and version control. Without a unified approach to data or metadata the business no longer has a single version of the truth. Departments can easily make business decisions based on flawed or incomplete data.
- Lack of alignment with corporate vision. Shadow IT often limits the realization of ROI. The corporate systems in place to negotiate vendor contracts and large-scale deals are sometimes bypassed. This can potentially lead to excess licensing and duplicate systems. Further, it disrupts the pursuit of organizational goals and IT’s strategic plans.
The bottom line is that the good intentions of adopting self-service reporting led to unintended consequences. The challenges can be summarized into three categories: governance, security, and business alignment.
Make no mistake, businesses need empowered users leveraging real-time data with modern tools. They also need the discipline of change management, release management and version control. So, is self-service reporting/BI a hoax? Can you find a balance between autonomy and governance? Can you govern what you cannot see?
The Solution
The BI Self-Service Spectrum
A shadow is no longer a shadow if you shine a light on it. In the same way, Shadow IT is no longer to be feared if it’s brought to the surface. In exposing Shadow IT, you can take advantage of the benefits of self-service reporting that business users demand while at the same time reducing risk through governance. Governing Shadow IT sounds like an oxymoron, but is, in reality, a balanced approach to bring oversight to self-service. 
I like this author’s analogy (borrowed from Kimball) of self-service BI/reporting likened to a restaurant buffet. The buffet is self-service in the sense that you can get anything you want and bring it back to your table. That’s not to say that you’re going to go into the kitchen and put your steak on the grill yourself. You still need that chef and her kitchen team. It’s the same with self-service reporting/BI, you will always need the IT team to prepare the data buffet through extraction, transformation, storage, securing, modelling, querying, and governing.
An all-you-can-eat buffet may be too simple of an analogy. What we have observed is that there are different degrees of participation of the restaurant kitchen team. With some, like the traditional buffet, they prepare the food in the back and lay out the smorgasbord when it’s ready to eat. All you have to do is load your plate and take it back to your table. This is the Las Vegas MGM Grand Buffet or the Golden Corral business model. At the other end of the spectrum, are businesses like Home Chef, Blue Apron and Hello Fresh, that deliver a recipe and the ingredients to your door. Some assembly required. They do the shopping and the meal planning. You do the rest.
Somewhere in between, perhaps, are places like Mongolian Grill that have prepared the ingredients but set them out for you to select and then give your plate of raw meat and vegetables to the chef to put it on the fire. In this case, the success of the end result depends (at least in part) on you to select a mix of ingredients and sauces which go together well. It also depends on the preparation and quality of the food you have to select from, as well as the skill of the chef who sometimes adds his own touches. 
The BI Self-Service Spectrum
Self-service analytics is much the same. Organizations with self-service analytics tend to fall somewhere on the spectrum. On one end of the spectrum are organizations, like the MGM Grand Buffet, where the IT team still does all the data and metadata preparation, selects the enterprise-wide analytics and reporting tool and presents it to the end-user. All the end-user needs to do is to select the data elements he wants to see and run the report. The only thing self-service about this model is that the report isn’t already created by the IT team. The philosophy of organizations that use Cognos Analytics falls on this end of the spectrum.
Organizations which more closely resemble the meal kits delivered to your door tend to give their end-users a “data kit” which includes the data they need and choice of tools with which they can access it. This model requires the user to better understand both the data and the tool to get the answers they need. In our experience, companies that leverage Qlik Sense and Tableau tend to fall into this category.
Enterprise tools like Power BI are more like the Mongolian Grill – somewhere in the middle.
Although we can generalize and place organizations that use various analytics tools at different points of our “BI Self-Service Spectrum”, the reality is that position may change due to several factors: the company may adopt new technologies, user competence may increase, management may dictate an approach, or the enterprise may simply evolve to a more open model of self-service with more freedom for the data consumers. In fact, the position on the spectrum may even vary across business units within the same organization.
The Evolution of Analytics
With the shift toward self-service and as organizations move to the right on the BI Buffet Spectrum, traditional dictatorial Centers of Excellence have been replaced with collaborative communities of practice. IT may participate in these matrixed teams which help socialize best practices across delivery teams. This allows the development teams on the business side to maintain some autonomy while working within the corporate boundaries of governance and architecture. 
IT must remain vigilant. Users creating their own reports – and in some cases, models – may not be aware of data security risks. The only way to prevent potential security leaks is to proactively search for new content and evaluate them for compliance.
The success of governed Shadow IT is also about the processes that are in place to ensure security and privacy policies are complied with.
Self-Service Paradoxes
Governed self-service analytics reconciles the polar forces pitting freedom against control. This dynamic plays out in many areas of business and technology: speed versus standards; innovation versus operations; agility versus architecture; and departmental needs versus corporate interests.
Tools for managing Shadow IT
Balancing risks and benefits is key to developing a sustainable Shadow IT policy. Leveraging Shadow IT to uncover new processes and tools that could allow all employees to excel in their roles is just smart business practice. Tools with the capability of integrating with multiple systems offer companies a solution that can appease both IT and the business.
The risks and challenges raised by Shadow IT can be greatly mitigated by implementing governance processes to ensure that quality data is available to all who need it via self-service access.
Key Questions
Key Questions IT Security Should Be Able to Answer Related to Shadow IT Visibility and Control. If you have systems or processes in place to answer these questions, you should be able to pass the Shadow IT section of a security audit:
- Do you have a policy which covers Shadow IT?
- Can you easily list all of the applications being used within your organization? Bonus points if you have information on version and fix level.
- Do you know who modified the analytic assets in production?
- Do you know who is using Shadow IT applications?
- Do you know when the content in production was last modified?
- Can you easily revert to a previous version if there are defects in the production version?
- Are you able to recover individual files easily in case of disaster?
- What process do you use for decommissioning artifacts?
- Can you show that only approved users accessed the system and promoted files?
- If you discover a flaw in your numbers, how do you know when it was introduced (and by whom)?
Conclusion
Shadow IT in its many forms is here to stay. We need to shine a light on it and expose it so that we can manage the risks while taking advantage of its benefits. It can make employees more productive and businesses more innovative. However, the enthusiasm for the benefits should be tempered by security, compliance, and governance.
References
How to Succeed with Self-Service Analytics Balancing Empowerment and Governance
Definition of Ideology, Merriam-Webster
Definition of Shadow Economy, Market Business News
Shadow IT: the CIO’s perspective
Single version of the truth, Wikipedia
Succeeding With Self-Service Analytics: Verify New Reports
The IT Operating Model Evolution
