The Next Decade of Business Intelligence

Author: Jodie Heflin, Lucrum

January 2010

In 1951, the US Census bureau received their first computer followed by the IRS in 1952.  Computers became available for the home in the late 70’s and early 80’s.  Data hit the web in the early 90’s and computing became truly mobile when the first PDAs arrived in the late 90’s.

As computers became more mainstream, companies started to get very good at “getting data in”.  Programmers developed better methods for designing databases.  We learned how to manipulate the processes and we learned how to spread the load across the servers.  These advances allowed for consumers to add more and more data to the company systems.  We started collecting transactions from retailers and capturing every SKU.  We asked our sales staff to add all of their contact information to CRM systems.  We started adding every asset to our asset management systems.  We had so much data that the inevitable happened…the CFO’s and CEO’s started asking for information about all of the data that we had amassed.

In order to “get the data out”, new reporting tools emerged.  These tools allowed a new crop of programmers and analysts to rapidly deploy flat reports.  We got creative and added dynamic parameters, but still the C’s wanted more.  The queries were slow and sometimes brought the transactional systems to a crawl.  We looked to our data models and found ways to index them but still it wasn’t fast enough.  A better method for accessing our data was desperately needed.  Enter data warehousing…

The first Business Intelligence books were published in 1991 (Inmon) and 1996 (Kimball).  The Data Warehouse Institute (TDWI) started in 1995.  Although in the 70’s and 80’s there were people talking about data warehousing, Business Intelligence and Data Warehousing practices didn’t really start until the mid-90’s.  These methods proclaimed that the best way to get the reporting to be better, faster and more dynamic was to get it out of the transactional systems we had worked so hard to optimize and to move it into a new dimensional model.  Additionally companies started developing special tools aimed specifically at the new DW market.  These companies proclaimed that all you had to do to solve your BI problems was to attach their tool to your model and voilà!  Instant BI!

The dimensional models were so much better and faster.  However, we discovered that the traditional waterfall method of development that worked so well for creating stable transactional systems didn’t address the ever-changing needs of the business.  Addressing those needs and being able to change course mid-stream is critical to BI success.

60 years later, what have we learned?  We’ve learned that BI needs to be done fast enough to change with the business.  Two year-long BI projects are no longer effective.  To make BI effective, we have to get better at fast, short cycles.  BI needs to be agile, flexible and highly iterative.  Adapting SCRUM methods into BI development is one step. We’ve also realized that BI is more than a tool sitting on top of a dimensional model.  True Business Intelligence is achieved when flexible, scalable models are built quickly in a highly iterative fashion and address constantly changing business questions.

In addition to addressing the way that we manage our DW implementations, we are also starting to evaluate how the models themselves are built.  Some companies are evolving beyond Kimball and Inmon method’s.  There is a “new player” in the field of DW and his name is Dan Lindstedt.  Dan’s view of DW is called The Data Vault.  Endorsed by Bill Inmon (which speaks volumes), his method is a hybrid approach to the ODS and is “a hybrid approach encompassing the best of breed between 3rd normal form (3NF) and star schema.”  It appears that we can start addressing real-time BI with this model.

For companies developing BI solutions in 2010, they must be sure to create models which support the business, give the business tools that are easy to understand, and model the data in a way that allows for real-time BI.  This will provide the C-level executive with something beyond static data.  Turning data into meaningful, fluid information that can provide these execs with the knowledge that they need to run their business more effectively.

www.lucruminc.com

uptime
search | login