The chasm between Business and IT is well documented and has existed since the first punch-card mainframe dimmed the lights of MIT to solve the ballistic trajectory of WWII munitions. Since that time, great leaps in data collection, storage, connectivity, and processing power have made IT infrastructure ubiquitous. You’re not even in the game if you don’t have an IT group.
But the productivity gains have not kept pace with the investments. The newest hype-driver, analytics, claims that it will finally deliver the goods. But can it?
Some studies suggest that analytics projects have an 80% failure rate. That’s abysmal. In the next few articles, we’ll look at reasons for this failure. We’ll start by describing the roles, then talk about process, and finally identify the characteristics of world-class analytics.
A helpful starting point is to imagine your analytics dream team. Who would you hire, and what would their roles be? I suggest that there are five distinct job descriptions:
Data Steward – this skillset is alive and well in most organizations. Almost everyone has a data warehouse, talks about the ETL process, and has had discussions around the business rules of cleaning up and storing their data. The data steward will use tools such as SQL Server, MySQL, Oracle, and if she’s a superstar, she’ll dabble in Python and web scraping and know the difference between Hadoop and Map Reduce.
Analytic Explorer – this skillset is a tough one to find. It requires math, statistics, and modeling along with a healthy dose of creativity and skepticism. These are the people who can spin straw into gold or write tomorrow’s news today. His job is to explore your data, combine it with sources outside the firewall, and distill it down to insights that will support your most critical decisions. He’ll use tools such as Excel, R, MATLAB, ArcInfo, SAS, Tableau, and SPSS. If he’s a superstar, he’ll know all about Bayes, Optimization, and the difference between precision, accuracy, and skill.
Information Artist – This is the role of a creative. Her goal is to sell the results to the decision-maker. And the lack of emphasis on this skillset is one of the reasons analytics is such a failure (and why Apple is such a success). Edward Tufte – the godfather of data visualization - speculates that the lack of good data design caused the Challenger space shuttle tragedy. Think of this person as being as crucial as your sales force. In fact, that is their job – to sell the right answer. Excel and PowerPoint can suffice, although the more skilled will use a variety of tools from Google Earth to Adobe Illustrator to D3. If she’s a superstar, she’ll be as comfortable talking about the math behind the visuals as she is talking about the psychology behind her design.
Automator – If the Explorer finds the path through the dark forest to the fountain of youth, and the visualizer designs a beautiful bottle for the elixir, then the Automator turns that path into an eight-lane highway and builds a factory to bottle that stuff as soon as it comes out of the ground. His job is to operationalize the work of the Explorer and Visualizer. He makes sure that results are timely and fast. He adds scale. He might use traditional coding methods like C# or .NET or he might fiddle with Ruby or Objective C. Or he might even be the guru of Business Objects, Microstrategy, or D3.
The Champion – The champion stands with one foot in the land of “gut feel”, and the other planted firmly in the side of “evidence”. She can speak the language of the geeks, and translate it to that of the battle-hardened general. She believes strongly in data-driven decision making, but also recognizes the value of deep domain experience. She’s tireless in her efforts to sculpt the processes of the organization to support analytics. She aims to harvest the brightest insights from the sharp young analysts and the cleverest hacks from the wily old veterans. Her focus is adoption, and if she’s a superstar, she’ll make you believe that this analytics thing was your idea in the first place.
So that’s your dream team: a steward, an explorer, an artist, an automator and a champion.
But there’s a problem. This team rarely exists in the wild. Most companies hire the Data Steward, and then try to do the rest through a major software implementation. Unfortunately, the software is not meant to explore and discover. And it was designed by engineers who don’t understand the psychology of data visuals. It’s like expecting your bookkeeper to be your CFO. Sure they can both do accounting, but you won’t be happy with the results.
In other instances, organizations will try to shoehorn engineers into the roles “in their spare time”. Again, with neither the training nor the time to explore the data or design the results, they’re doomed to fail. These skillsets are distinct, and they shouldn’t be ignored.
So what’s the canny company to do? If you’re extremely lucky, you’ll find the unicorn of the 21st century known as the Data Scientist, pay her a quarter million, and watch the magic happen. (A data scientist can do all five roles.) Or you can try to develop these skills in-house. Or you can hire contractors – perhaps engaging a consulting firm to take on the Explorer or Visualizer roles for a time. Or you can outsource the whole thing.
What’s important is that you recognize that each of these roles is necessary. Neither software nor “Dave in Engineering” can replace them. Happy hunting.