An article recently posted on Harvard Business Review declares the Data Scientist as “The Sexiest Job of the 21st Century“. The authors, Thomas Davenport and DJ Patil are both familiar names to me, especially Patil, and, as expected, I found the article to be an interesting read. (It is also an easily palatable read, suitable for sending along to parents and friends who are still a little confused by what it is I do).
One passage, in particular, caught my interest:
Considering the difficulty of finding and keeping data scientists, one would think that a good strategy would involve hiring them as consultants… But the data scientists we’ve spoken with say they want to build things, not just give advice to a decision maker. One described being a consultant as “the dead zone—all you get to do is tell someone else what the analyses say they should do.” By creating solutions that work, they can have more impact and leave their marks as pioneers of their profession.
As a data scientist* who operates as a consultant, I found this thought-provoking. Is hiring a data scientist as a consultant a good strategy for a company? Is it true that most data scientists are averse to consulting because they cannot make as much impact as a consultant than as a full-time employee? I certainly can’t speak for other data scientists, but here are some of my thoughts.
Many data science projects are well suited for consulting.
There are many indicators to help an organization decide when to outsource a project and when to handle it “in-house”. I’ve worked for many years in the world of technical consulting, and to me a significant percentage of data science projects are well-tailored for outsourcing.
- Data science is usually not related to an organization’s core competency. A business that is good at making widgets may not be well equipped to build a team and develop processes for doing data science.
- Many data science projects involve validating an idea before it is put into production. A consultant is often the right person to efficiently investigate the feasibility of an idea and determine its potential return-on-investment. An outsider will have the emotional detachment and political freedom to declare whether the project is well-grounded and realistic, and what it will take to execute the vision. Once validated, a business can make an informed decision about whether to build the product in-house.
- Hiring a full-time data scientist can be difficult and time-consuming, especially when the individuals recruiting aren’t equipped to evaluate such candidates, and data scientists command a high salary. Much time and resources can be saved by first validating and developing a strategy with a consultancy.
There is also another important consideration.
“Data scientist” is a very broadly defined category. An experienced statistician with some programming skills, an experienced programmer with some some knowledge of machine learning, a veteran business analyst with proficiency in big data architecture — all these may truthfully call themselves data scientists. That the term is overloaded causes problems in the context of recruiting, and there’s another consequence.
A data science project is composed of many different components and many different phases. Data exploration, confirmatory analysis, translating hypothesis to business strategy, communicating yet-to-be-developed data-centric ideas to executives, architecting and developing production-ready systems, optimizing and scaling infrastructure. Each of these requires very different skill-sets, yet most organizations find themselves hiring a data scientist or several data scientists without an understanding of which skills will be needed, when, and for how long. A strategic approach is very important.
For example, an organization may bring on a consultant to do those things that require specialization and need to be done only once; hire a permanent data scientist for long-term tasks and tasks that require intimate knowledge of secure internal data; and train existing technical teams to handle some of the development and maintenance of the data science product in production.
A consultant enables organizations to explore or experiment with an idea (or develop new ideas) with less risk and investment.
[UPDATE: There are, of course, many disadvantages of using a consultant over hiring in-house. Employees have more intimate knowledge of the business and the data. There are important considerations related to data confidentiality and related legal restrictions in many industries. And there are the more general pitfalls of outsourcing, about which much is written elsewhere. I do not mean to imply that hiring a consultant for data science is always the right thing to do — just that there are many scenarios in which it is.]
Data Exploration is fun!
Switching perspectives to that of the data scientist, there are many reasons to choose consulting over working full-time for an employer. For me, one of the most important is simply that … it’s fun!
Patil and Davenport quote a data scientist who clearly gets the most satisfaction out of building finished products, but the beginning stages of the data product development cycle are equally rewarding. There is a unique challenge in gaining a broad understanding of the client organization and their business goals, in exploring the available data and their latent potential. One must develop hypotheses, find creative ways to test them. There’s both a focus in trying to achieve the objective, and a creative license in methodology. Often there’s an opportunity to find an unforeseen innovative use for the data.
Although truly data-driven companies will continually explore new ideas in their data, it is usually in the early exploratory phases of a project that I learn the most and feel the most rewarded for my work. As a consultant, I am able to maximize the amount of time I spend on my favorite data science tasks.
Yes, there is a cost to this luxury, and the quoted data scientist makes a good point about what he calls the ‘dead zone’ and the impact of building lasting solutions. But for me, at this point of my career, I’m very happy with the trade-off — and I would imagine that many other data scientists are too.
*I usually don’t refer to myself as a ‘data scientist’, but that’s a discussion for another day.