Disparate Data Sources

Author: Robin V. Horth

January 2008

“The exponential growth in the amount of data collected in research has created an urgent technical challenge for computer scientists to develop infrastructures to integrate metadata sources and data storage in a query-compatible, data-intensive environment.” Source: Pacific Northwest National Laboratory Operated by Battelle for the U.S. Department of Energy

The quote above is a very small example of how the technology sector is working toward solving problems associated with disparate data sources.  These problems manifest as redundant data, inaccuracies introduced by untimely updates in one data source or another and inconsistent data based upon who entered the data, when and which disparate sources of information the data resides in.   There are only two ways to solve the problem.  One must either integrate the data sources by integrating to fewer data systems or consolidate the data into one data source that is designated as the authoritative source.  

Program and project teams fight the same battle on a smaller scale – but the same problem nonetheless.  Our tendency is to propagate several tools that are designed to solve specific problems or automate individual internal workflows.  Each of the tools tends to be populated with redundant data that may also be resident in other tools.  Results invariably include confusion (what’s the most current, accurate data, and what’s the authoritative data source?), extra work (someone must administer each of the disparate tools), and most importantly – redundant work (I’m doing work to administer this tool and you’re doing the same work to administer another tool).  

The solution to this problem is to consolidate data as much as possible into a single tool, and to use the consolidated tool as your authoritative source of data and/or information.