This is part 4 of a 4 part series on DataOps
What is DataOps?
DataOps is an emerging idea, with many perspectives. For this post, I am using the following definition is from Hitachi-Vantara:
DataOps is enterprise data management for the AI era. Now you can seamlessly connect your data consumers and creators to rapidly find and use all the value in your data. Data operations is not a product, service or solution. It’s a methodology, a technological and cultural change to improve your organization’s use of data through better data quality, shorter cycle time and superior data management.
The most significant challenge to implement DataOps is going to be cultural, not technology. The following protocols are suggested to address some of the main cultural barriers that exist.
Dark Matter Data
The exact percentage varies, but studies agree that corporations are barely utilizing their data assets. As the volume of data grows, there are growing companion efforts to manage and find new value in existing data assets. And the efforts are not keeping up.
DataOps is the response to managing and finding new value in your data assets. DataOps is more a set of principles and framework than technology. As such, the required cultural changes will make or break a DataOps effort. This series of posts identified some of the cultural challenges. I ask that you bear one thing in mind as you take your DataOps journey:
There will be times, no matter how much data you have, you will not have all that is impacting what you do and you will never be able to get it.
I call this Dark Matter Data.
When studying the movement of celestial bodies, the math has some difficulties. To account for these difficulties, the existence of Dark Matter has been postulated. To greatly oversimplify, Dark Matter is something that we cannot perceive directly but it influences the movement of objects in space.
It is not to be confused with Dark Data – which is data you have but did not know you have. (Although it may feel like Dark Matter at times).
Dark Matter Data is data you will never have, cannot obtain, but is influencing what happens to your organization. Our cultural fondness for numbers, the quantification of everything, creates a strong sense of certainty in our decisions. As DataOps succeeds, your Advanced Analytics capabilities grows, and your company becomes more data driven it is important to manage the expectations.
The existence of Dark Matter Data means:
- Sometimes you will have no idea why something happens. It just does. And that is fine. It is data, not magic.
- Unintended consequences will still happen. There are no decisions without risk.
- There is a world of difference in choosing what color your customer may like versus a new surgery technique. Strive for an appropriate level of probability and certainty. There are diminishing returns to increase certainty.
- Just like the movement of planets and stars, DataOps is continuous, it never ends.
- Dark Matter Data will always exist.
Interested in learning more about DataOps? We recommend the DataOps.NEXT virtual conference by our premier partner Hitachi-Vantara. There are 4 tracks:
- Optimize the Data Fabric
- Build and Manage Data Pipelines
- Improve Data Governance and Agility
- Expand Analytics and Machine Learning
There is no cost to join live or to watch replays for 90 days.
Date: Thu 14 May, 2020
Time: 9.00am EST / 1.00pm UTC
Where: Online
