Data Warehousing, Data Vault as evolutionary step
•
4 likes•3,005 views
Presentation held on may 6th 2011 on the Advanced Dat Vault seminar in the Netherlands: Abstract: Data Warehousing is still being ridiculized by popular literature and opportunistic vendors (and sometimes analysts) in the Business Intelligence domain - they tend to call it 'traditional' as opposed to their 'silver bullet technology'. However, data warehousing has evolved and Data Vault - although undervalued by many - is fueling this evolution. Data Vault methodology enables architects to (finally) embed data warehousing into the Enterprise Architecture, something they struggled with the last 15 years. In the Netherlands, Data Vault sky rocketed innovation in the data warehousing scene. Accelerators in terms of frameworks and software are being build by experts in the field and several product vendors picked up on it. The presentation of Ronald Damhof will discuss briefly the evolution of data warehousing as it stands today, the position of Data Vault in the Enterprise Architecture, the different species/forks that exist in Data Vault and the automation that comes with it.
Recommended
Recommended
More Related Content
More from Prudenza B.V
More from Prudenza B.V (8)
Recently uploaded
Recently uploaded (20)
Data Warehousing, Data Vault as evolutionary step
- 1. “It is not the strongest of the species that survives, nor the most intelligent that survives. It is the one that is the most adaptable to change” Charles Darwin Data Vault is anevolutionary step Data Vaultfirmlypositioned data warehousing in EA Data Vaultforksinto different species Andyes, we can speed up evolution
- 4. 3:1 mileage Expensive to buy Slow…. Inflexible Carbon based, polluting! Need specialists to drive Need specialists to repair
- 5. It can go anywhere It can go anytime It can be used by anyone Very fast!! Of course, extremely energy efficient However…. It won’t fly
- 6. Sorry, not more than 100 miles Oh, charging takes like 5 hours Batteries are somewhat expensive Prone to (electric) failure No infrastructure yet
- 7. Utilize new methods and technologies Make it effective with todays legacy Utilizing todays infrastructure As well as adapting to new ones Retain the good, get rid of the bad Make it efficient, better mileage Make it durable Make it repeatable Make it affordable Make it Agile Make it fit in the environment
- 8. What products are asked? What are the quality characteristics? How are these products made?
- 9. What
- 10. Product & Services What
- 11. What Compliant Adaptible Sustainable Decoupled Centralized Standardized & Industrialized Effective
- 12. How ‘Calculating risk’ Source ‘Yield modules’ Source ‘Customer segmentation’ Semantic gap
- 13. How 4. Generate (BI) products 3. Enrich and cleanse data 2. Register & (anchorize data) 1. Get the raw (uncut) data Information Delivery Proces
- 14. Recipient End-user (Local) 4 4 4 4 4 Data & function service 3 3 3 3 3 Information Delivery process 2 2 2 2 2 1 1 1 1 1 Generic BI proces (Central) Data sources(internal & external)
- 15. Key Design Decisions Adaptable Sustainable Compliant Centralized
- 16. Adaptable Effectiveness Sustainable Decoupled Compliant Centralized
- 17. Key Design Decisions Compliant Adaptable
- 18. View: Component view 1 2 3 4 Company xxx data warehouse & Business Intelligence Domain 4 Sources BI apps Reports 3 2 Source store 1”, 2” Business View, Data feeds BI AppsAnalysis 1 Enterprise Data Warehouse BI Apps Ad-hoc Function, ‘How’ External sources Data, ‘What’ ‘Where’, ‘Whom’
- 19. Sourcestore to BV Sourcestore to product Source to product EDW (DV) Adaptable Sustainable Compliant Decoupled Effective Standardized Centralized
- 20. View: Component view 1 2 3 4 Company xxx data warehouse & Business Intelligence Domain 4 Sources BI apps Reports 3 2 Source store 1”, 2” Business View, Data feeds BI AppsAnalysis 1 Enterprise Data Warehouse BI Apps Ad-hoc Function, ‘How’ External sources Data, ‘What’ ‘Where’, ‘Whom’
- 21. Administrative process Information Delivery Process Decision- & control Data & Information recipients Generate& Distribute Enrich Register (& anchorize) Attain Proces PDCA DV basedData Warehouse Systems(internal &external) Information products Compliance reporting Risk Management Supply/Data Demand/Function Performance Management Data products Businessrules Supply chain optimization Staging Fraud detection Market basket analysis Control / Metadata
- 23. Performance of the systems has direct effect on efficiency of users
- 24. Most core business activities are ‘one line’
- 25. Systems work is mostly maintenance
- 27. Performance of the systems has direct effect on efficiency of users
- 28. New systems promise major processand service improvement
- 29. New systems promise major costreduction
- 31. Performance of the system has no direct effect on efficiency of users
- 32. Company canquicklyrevertto manual procedures
- 34. New systems promise major costreductions
- 35. New systems will close significant cost, service or process performance withcompetiters
- 36. IT constitutes more than 50% of capitalspending
- 37. IT makes up more than 15% of total corporate expensesData Vault & Governance Strategy Need for reliability in IT Need for innovation with IT
- 39. Externalfocused
- 42. Control of assets
- 44. Information
- 45. IT Services
- 46. Efficiency / Cost Control
- 47. Security
- 48. Avoid surprises
- 49. Manage risks
- 51. Stable ArchitectureData Function Need for reliability in IT Need for innovation with IT
- 52. Data Vault & Self ServiceThe development model Central functiondevelopment CentrallycoordinatedInfrastructuredevelopment Gedelegeerde Ontwikkeling Localfunctiondevelopment Localfunctiondevelopment Selfservice development Delegateddevelopment Selfservice development Delegateddevelopment Function (Opportunisticdevelopment) Data (Systematic development) Data CentrallycoordinatedICT development
- 54. 1 - Classic Data Vault Business Transaction System Data Vault Data Marts Staging Out Business Transaction System Generic Business Rules Rule Vault Structure transformation Hub = business keys Business rule execution Structure and value transformation Standardized Centralized Adaptable Effectiveness Sustainable Decoupled Compliant ? ?
- 55. 2 - Source Data Vault Business Data Vault Staging Vault Business Transaction System Data Marts Business Transaction System Staging Vault Structure transformation No integration, Hub=surrogate keys Persisting staging in DV format Business rule execution Integration DV modelled Structure transformation Standardized Centralized Adaptable Effectiveness Sustainable Decoupled Compliant ? ? ?
- 56. Source Source 100% Semantic gap Business DV Source Staging DV Source Staging DV 100% Semantic gap Still the source Integration, cleansing, consolidation Business rule execution upstream ?? DV modelled
- 57. Source Source 100% Semantic gap Data Warehouse Business DV Source Source Staging DV Source Source Staging DV 100% Semantic gap Still the source Integration, cleansing, consolidation Business rule execution upstream ?? DV modelled
- 62. Information maturity; conceptual business process knowledge present
- 63. Scale and complexity of data and/or organization
- 64. Relatively big semantic gap between Source Requirement
- 65. Factors that contribute to (2)
- 66. Scale and complexity of data and/or organization
- 67. Bad data quality
- 68. Business keys hard to identify
- 69. Information maturity; no conceptual business process knowledge present
- 70. Urgency
- 73. 1b – Classic Data Vault Business Transaction System Data Vault Data Marts Staging Out Business Transaction System Generic Business Rules Rule Vault Business Transaction System Data Vault Data Marts Staging Out Business Transaction System Generic Business Rules Rule Vault Structure transformation Light integration on the business keys Specific business rule execution Structure and value transformation Consolidation
- 74. Speeding up Data Vault Evolution?
- 78. 80-20 rule
- 79. Not everything can be generated, be real
- 80. Truly understand the mechanics
- 81. Do not underestimate the complexity
- 82. It is a pretty steep learning cycle
- 83. PoC, PoC, PoC
- 84. Make it yourself
- 85. Then “generate”
- 86. Generation software only in combo with modeling software and ETL software
- 88. Aim is to be 100% declarative
- 90. More advanced; generating XML code for 2nd gen. ETL tooling
- 93. Both DV species will live next to others
- 94. Technology will push the envelope
- 95. Automation typologies will be pushed even more towards metadata driven automation
- 96. WE – the people – need to evolve:
- 97. We need more talented engineers
- 98. We need more focused education
- 99. We need more knowledge sharing
Editor's Notes
- Shared