SlideShare a Scribd company logo

(Deprecated) Slicing the Gordian Knot of SOA Governance

•
•
Ganesh Prasad
Ganesh Prasad

This document has been superseded by "Dependency-Oriented Thinking: Volume 2 - Governance and Management". Please download that instead: http://slidesha.re/1fEjz7A

1 of 116
Download to read offline
“Slicing the Gordian Knot of SOA Governance” Version 0.99.1, November 2012 © Ganesh Prasad This work is licensed under Creative Commons Attribution-No Derivs 3.0 Australia http://creativecommons.org/licenses/by-nd/3.0/au/ Confessions of a back-to-basics SOA architect
Table of Contents Synopsis.................................................................................................................................6 Part I – Why The Industry Is Doing SOA Governance All Wrong..........................................7 Timidity in Asserting the All-Encompassing Scope of SOA...............................................7 Confusion between Governance and Management..........................................................9 Obesity of the Governance Function...............................................................................10 Part II – Elements of a Common Sense Approach..............................................................11 Recognising “Services” as a Weasel Word.....................................................................11 Back to First Principles and the Notion of “Dependencies”.............................................12 Why are Dependencies so Important?............................................................................14 Case Study 1 – Take it or Leave it: Monopsony and Choice......................................15 Case Study 2 – Culture Shock....................................................................................16 Case Study 3 – The Weakest Link of a Supply Chain................................................17 Case Study 4 – Locked Out: Key Person Risk............................................................17 Case Study 5 – The Stress and Strain of an Engineer's Life......................................18 Case Study 6 – It Goes Without Saying......................................................................19 Case Study 7 – “ASSUME” Makes an ASS of U and ME...........................................20 Case Study 8 – What Makes a Good Technology Platform?......................................21 An Architectural Framework to Analyse Dependencies..................................................24 BAIT as Layers of Dependencies................................................................................24 TOGAF Artifacts as Dependency Relationships.........................................................25 Part III – A Practical Guide To Governing And Managing Dependencies............................27 Agencies and Vehicles.....................................................................................................27 Key Roles....................................................................................................................27 Key Bodies...................................................................................................................30 How Many Committees?.........................................................................................32 Functions and Processes................................................................................................35 One-time Processes....................................................................................................35 Initial Dependency Review......................................................................................37 Recurring Processes...................................................................................................37 Periodic Dependency Review ................................................................................37 Remediation Program Review................................................................................37 Remediation Program Proposal..............................................................................39 BAU Program Management....................................................................................39
New Initiative Appraisal ..........................................................................................39 Processes in Steady State..........................................................................................43 Governance and Management Check-lists.....................................................................44 Classification of Dependencies...................................................................................44 The List of Check-Lists................................................................................................45 A Basic Governance Check-list...................................................................................45 A Basic Management Check-list..................................................................................46 Fundamental Enterprise Dependencies......................................................................47 Enterprise Dependency Check-lists........................................................................48 Business Layer Dependencies....................................................................................51 Modelling the Business with Domain-Driven Design..............................................51 Realism around Reuse............................................................................................52 Business Layer Dependency Check-lists................................................................55 Application Layer Dependencies.................................................................................58 Cohesion and Coupling...........................................................................................59 A Fun Exercise in Applying the High Cohesion Principle........................................62 Criteria for High Cohesion.......................................................................................66 Process, Product and Service.................................................................................68 Application Layer Dependency Check-lists.............................................................70 Information (Data) Layer Dependencies.....................................................................73 Data on the Outside versus Data on the Inside......................................................73 Aspects of Low Coupling.........................................................................................74 The Domain-specific Data Dictionary......................................................................75 The Internal Data Model (“Data on the Inside”)......................................................76 The Interface Data Model (“Data on the Outside”).................................................77 Message Data “on the wire”....................................................................................78 Information (Data) Layer Dependency Check-lists.................................................81 Technology Layer Dependencies................................................................................85 Implementing Products...........................................................................................86 Implementing Services............................................................................................86 Implementing the Nouns and Verbs of the Interface Data Model...........................87 The Three Core SOA Technology Components......................................................88 How Not to Use a Broker........................................................................................93 Implementing the Adverbs of an Operation............................................................94 Technology Layer Dependency Check-lists............................................................96
Bringing about Desired Behaviour – Velvet Glove or Iron Hand?...................................99 Summary and Conclusions................................................................................................101 Contributions of this White Paper..................................................................................102 Defining Terms...........................................................................................................102 Restoring Potential....................................................................................................102 Identifying Gaps.........................................................................................................102 Simplifying Tasks.......................................................................................................103 Doubling the Pay-off..................................................................................................103 Potential Criticism of This Approach..............................................................................104 The Weight of Tradition.............................................................................................104 Making Mountains out of Molehills............................................................................104 Drawing a Long Bow.................................................................................................105 A Bridge Too Far........................................................................................................105 About the Author................................................................................................................106 Acknowledgements............................................................................................................106 Appendix A – SOA Governance and Management – An Issue of Definition.....................107 Appendix B – Lessons from Cadet Camp (or Why SOA is Like a Snakepit).....................109 Appendix C – Core Entities and Dependencies in the TOGAF 9 Model...........................111 Appendix D – Artifacts In the TOGAF 9 Model..................................................................112 Appendix E – References..................................................................................................115
Synopsis “SOA Governance” as practised in the industry today suffers from three major problems: 1. “SOA Governance” is wrongly understood to be the governance of SOA as an IT function, rather than the application of SOA principles to the governance of all levels of the enterprise. Most of the potential benefits of SOA as an organising principle for the enterprise therefore go unrealised because of this overly narrow, technology-focused interpretation. 2. There is widespread confusion between the terms “governance” and “management” even among SOA practitioners. Many of the processes associated with “SOA Governance” are in fact routine management functions, and true governance tasks requiring direction are often neglected, resulting in costly yet preventable mistakes. 3. The processes associated with “SOA Governance” (which are more often management than governance) are needlessly heavyweight. Given the overly limited scope of “SOA Governance” as mentioned in Point 1 above, these processes soak up a disproportionate share of an organisation's resources without corresponding benefit. This white paper aims to do the following: 1. Redefine both Governance and Management in simple and unambiguous terms so that there is no confusion between the two functions and both can be performed effectively. 2. Raise the scope of SOA Governance and SOA Management beyond their current narrow technology focus so that SOA thinking can be applied at all levels to improve the agility, cost and risk profile of an organisation. 3. Recommend a comprehensive yet lightweight approach involving a few core roles and bodies, a minimal set of processes and a manageable set of checklists to enable both SOA Governance and SOA Management to be performed cost-effectively. 6
Part I – Why The Industry Is Doing SOA Governance All Wrong Timidity in Asserting the All-Encompassing Scope of SOA 1 “SOA Governance” does not mean the governance of SOA, any more than “scientific thinking” means “thinking about science”. We know of course that “scientific thinking” means a different way of thinking about everything, i.e., adopting a rigorous, analytical, evidence-based approach to understanding every aspect of the universe without exception. Scientific thinking is about applying science to thinking, not the other way around. In exactly analogous fashion, “SOA Governance” is about applying SOA thinking to governance, not about applying governance to SOA. So how does one “think SOA”? We need to understand that SOA is an organising principle that impacts every aspect of the enterprise. It is not a set of technology products or even an approach to deploying technology components. The word “technology” refers to the implementation of business logic2, and most “SOA Governance” activities in organisations that would describe themselves as “doing SOA” relate to implementation-related activities such as setting development and environment standards, reviewing the design of SOAP-based web services and Business Process Management (BPM), controlling versions of services, establishing policies around the use of an Enterprise Service Bus (ESB), using a registry/repository to centralise information about services, etc. All of these are in fact routine management activities, with a narrow focus on technology to boot. These cannot be called SOA Governance at all! The definition of SOA we propose is “the science of analysing and managing dependencies between systems”. Systems need not be computer systems, and neither are dependencies restricted to technology. Dependencies exist at any level of business, human relations or technology, as we will show using diverse examples. The term “managing dependencies” as used above is shorthand for “eliminating needless dependencies and formalising legitimate dependencies into readily understood contracts”. 1 Service-Oriented Architecture 2 “Technology” may conjure up visions of advanced computer systems, but even a manual ledger to record transactions is technology. Indeed, it would seem like high technology to a race without paper! 7
In a previous white paper published under the aegis of WSO2 (“Practical SOA for the Solution Architect”3), we showed how a solution design that is tightly coupled at the data layer can completely negate the benefits of expensively procured SOA technology (e.g., ESBs, registries, etc.) Such situations are unfortunately quite common because practitioners often take a technology-only approach to SOA and do not see the dependencies that exist between systems at different levels. The fault lies not with SOA itself but in our misunderstanding of SOA as being limited to technology. We need to start seeing SOA as a way of thinking about dependencies, not just within the technology realm or even at the level of data, but across the board. Unless we as an industry adopt “Dependency-Oriented Thinking”, the returns on our SOA investments will remain anaemic4. It would seem that industry analysts and prominent vendors have knowingly or otherwise misled us for over a decade with the conceptual model illustrated below, and this view has stood in the way of our ability to realise the full benefit of SOA. Every expert unfailingly issues the standard disclaimer that SOA is not about technology, but the opposite message gets dog-whistled through the emphasis on products to manage web services, and ultimately prevails. In a later section, we hazard an explanation for this. Corporate Governance Corporate Resources IT Governance IT Resources SOA Governance SOA Resources Fig. 1 – The limited (and limiting) view of SOA Governance as a subset of IT Governance (itself a subset of Corporate Governance) – a view endorsed by industry thought leaders such as the Burton Group5 and IBM6. 3 Downloadable from http://bit.ly/stbfiA 4 In retrospect, DOT may have been a better term than SOA. 5 The Burton Group's Research Director speaks on SOA and Governance: http://bit.ly/HFdF6t 6 IBM's definition of SOA Governance: http://ibm.co/qm46 8
To repeat, SOA is not a subset of IT. The benefits of SOA thinking when applied more broadly are: 1. An improvement in business agility because of minimal dependencies between systems and consequently minimal “friction” that can impede change 2. Sustainably lower operating costs for the same reasons 3. A significant reduction in operational risk because of better-understood dependencies and fewer surprises Clearly, we have lost out on some major benefits by defining SOA as narrowly as we have. Confusion between Governance and Management “Governance” is an over-used (and abused) term. One of the unfortunate side-effects of the collapse of corporations like Enron and WorldCom in 2001-2002 was the sudden popularity of the term “governance” in popular discourse. The word “governance” has acquired such a cachet today that it often tends to be used just for effect, even when what is meant is just plain old “management”. So let's set these terms apart right away. Governance is ensuring that the right things are done. Management is ensuring that things are done right. In spirit, governance is about the ends, or the “What”; management is about the means, or the “How”. This is a fundamental distinction that is key to correctly implementing what we recommend in this white paper, hence it is worth spending some effort to understand this thoroughly. An analogy with the functions of a company's board of directors and its executive management will make the distinction between corporate governance and corporate management more concrete. A decision on whether the company should enter a new market is a decision for the board, because it pertains to the fundamentals of what the company wants to be and whether or not the move is in the best interests of its shareholders. In other words, this is about doing the right thing (i.e., governance). Once the decision is taken to enter a new market, executive management is responsible for retooling the resources of the company to enable it to compete effectively in that 9
market. This is about doing things right (i.e., management). Governance and management decisions apply at lower levels of the organisation as well. Projects often have steering committees as well as working groups. Steering committees are comprised of key stakeholders and they tend to make governance (i.e., “what”) decisions – objectives, scope, success criteria, etc. They also monitor adherence to these parameters. Working groups are teams of hands-on people assigned to the project. They make the day-to-day management (i.e., “how”) decisions that will enable them to solve routine problems and achieve the objectives that have been set by the steering committee. The steering committee is only concerned with the ends and not the means. The working group is responsible for the means. In general, one can tell whether a given decision is about governance or about management by thinking about how those decisions could be judged in hindsight. If the verdict is likely to be “right” or “wrong”, then it's a governance decision. If the assessment is likely to be one of a spectrum (e.g., “excellent”, “good”, “fair” or “poor”), then it's a management decision. Obesity of the Governance Function Organisations that have embraced the requirement for “SOA Governance” generally have committees, processes and tools to bring discipline to the way services are designed, built, deployed and managed. After all, this is how the governance of SOA is commonly interpreted. Some organisations have an “Integration Centre of Competence” that is tasked with defining standards and controlling the introduction of new services into the ecosystem. While these are sensible measures, they impose a rather high overhead especially given their limited technology-only focus. A centralised approving authority is an overworked bottleneck, and so agility and operational cost are the first casualties. Many organisations consequently see very modest improvements in their operational cost and efficiency as a result of moving to SOA. The economies and dis-economies largely neutralise each other, and the heavyweight governance processes shoulder a large portion of the blame. If projects in your organisation are constantly trying to find short-cuts around a centralised authority and a large part of this latter group's efforts are around reigning in such rogue projects, it could be a sign that the organisation is chafing under heavyweight governance. 10
Part II – Elements of a Common Sense Approach Recognising “Services” as a Weasel Word No matter how often we parrot the mantra “SOA is not about technology”, we end up with a technology-only view of SOA while we're looking elsewhere. How does this happen? Our opinion is that there is a slippery slope that we embark on once we agree to talk about “services”. From “services” to “web services”, and from web services to a technology-only view of SOA and SOA Governance is then just a natural progression, as illustrated below: “SOA is not about technology” “Service-Oriented Architecture is a view of the organisation as a collection of reusable services” “We should expose all business functionality through Web Services” “It's good practice to proxy all Web Services through an ESB and use a registry/repository to manage them, especially as their numbers start to grow.” “We need some governance around the use of all this technology” “SOA assets are owned by IT, so SOA Governance is an IT function” Fig. 2 – The slippery slope that leads to the view of SOA as a part of IT There is a place for the concept of “services” in what we lightly call “Service-Oriented” Architecture. But the wide-angle lens of dependencies shows us that there's much more to SOA. We need to derive our view of “services” much more systematically and rigorously than we have been used to doing. So let's look at this process more closely. 11
Back to First Principles and the Notion of “Dependencies” The four most important principles of SOA are dependencies, dependencies, dependencies and dependencies7. We're not being entirely flippant in saying this, because there are four distinct “layers” in an organisation where dependencies need to be managed. These layers, as we will explain shortly with the help of a formal framework, are Business, Applications, Information (Data) and Technology 8. SOA Principles (Eliminate needless dependencies between systems, formalise legitimate dependencies into well-understood “contracts”) Manage dependencies Business Layer Manage dependencies Manage dependencies Application Layer Manage dependencies Manage dependencies Information (Data) Layer Manage dependencies Manage dependencies Technology Layer Fig. 3 – The True Scope and Concerns of SOA - More than just a few Web Services managed by a section of IT! We believe that for an organisation to be effective in achieving its goals, an acceptance of this dependency-based view of SOA is essential. Any discussion of SOA Governance and SOA Management has to start from this basis. Since SOA is all about the management of dependencies, SOA Governance should be seen as deciding what dependencies are legitimate and SOA Management as deciding how to manage dependencies. Both these aspects are important, so even though SOA 7 To paraphrase the 3 rules of real estate: “Location, location, location”. 8 This is commonly referred to as the “BAIT Model”. 12
Governance seems to have all the mind-share, this white paper will talk about SOA Management9 with equal emphasis. Ironically, traditional approaches to “SOA Governance” are usually about SOA Management (i.e., the “How”). So here are our simple and readily-understandable definitions: • SOA Governance is determining what dependencies are legitimate at every layer of the organisation and identifying what existing dependencies fall outside this set. • SOA Management deals with how to remediate illegitimate dependencies at every layer of the organisation, how to formally document and communicate legitimate dependencies and how to prevent recurring violations. We will break these down into specific tasks in Part III, but for now, the following high-level illustration will summarise how we arrive at these definitions. SOA (The science of analysing and managing dependencies) Governance SOA Governance (What are the right What dependencies are legitimate? things to do?) What existing dependencies fall outside this set? SOA Management Management How do we remediate illegitimate dependencies? (How do we do How do we formalise legitimate dependencies? things right?) How do we prevent recurring violations? Fig. 4 – SOA Governance and SOA Management at a glance This is the core philosophy behind our approach to SOA Governance and SOA Management in this white paper. 9 In the literature, we often come across the term Service Management. This term derives from the view of SOA as a subset of IT rather than as a set of organising principles for the entire enterprise. Our term “SOA Management” is more comprehensive. 13
Why are Dependencies so Important? We believe that a deep understanding of dependencies makes all the difference between a successful organisation and one that is less so. Sun Tzu said 10 many centuries ago, “Know thyself, know thy enemy – a thousand battles, a thousand victories.” By “knowing” oneself and one's enemy, he was really talking about dependencies, because strengths as well as weaknesses are dependencies. What is it that makes you and the other guy tick? Fast cavalry? Long range artillery? What are the things that can trip you up? Lack of supplies (logistics)? Troops weakened by dysentery? What buttons can be pushed to make you or the other guy behave in a predictable way? Siege of a prestigious fort? Abduction of a corps commander's son? All of these are dependencies. All of these relate to “knowing thyself” or “knowing thy enemy”. In that classic marketing strategy book of the '80s, “Marketing Warfare”, authors Al Ries and Jack Trout offer this nugget of wisdom to their clients who are looking for the most effective way to compete against their rivals - “Don't attack a weakness that is a weakness. Attack the weakness inherent in a strength.” Their idea is that weaknesses that are purely weaknesses can be fixed, but weaknesses that are an inherent part of a competitor's strength cannot be fixed without compromising that strength. As an example, they suggest that any company trying to compete against a giant like Campbell Soup should attack not the price of the soup (since a larger company can easily drop its prices long enough to drive a smaller competitor out of business), but something peripheral like the metal cans in which the giant corporation packages its products. The enormous investment in metal can packaging is a weakness inherent in the strength of Campbell Soup's scale of operations, and this investment cannot easily be thrown away. A competitor positioning themselves as using either environmentally friendlier packaging or packaging that is less suspect from a health perspective would have a better chance than one competing on price. In other words, Trout and Ries suggest attacking dependencies that a competitor cannot easily rid themselves of. That's how important dependencies are to the life-and-death struggle of companies in the marketplace. To further reinforce the importance of dependencies to an organisation, we provide a few real-life case studies11 which clearly demonstrate two things – one, that dependencies are 10 The Art of War 11 The examples are authentic but have been anonymised, not so much to protect the guilty as to protect the authors and their associates from accusations of breach of confidentiality. 14
not restricted to information technology, and two, that a lack of understanding of dependencies can cost an organisation dearly. Case Study 1 – Take it or Leave it: Monopsony 12 and Choice A medium-sized commercial organisation in a developing country was experiencing strong growth, and began hiring more staff to meet the demands of its expanding business. One of its hires was a seasoned manager from another industry, and his eyes immediately picked up an aspect of his new employer's business that its owners had been oblivious to. He was shocked to discover that although the volume of growth was strong, profits were not growing at all. On the contrary, many of the projects were actually losing money. It took him very little time to home in on the problem. Almost all of the company's business was coming from government contracts, which had been its traditional mainstay. Government contracts had two features that were both very unfavourable to suppliers. One was the policy of always awarding contracts to the lowest bidder, which squeezed margins to the bone. The second was the one-sided payment policy, under which the bulk of the payment for a project would only be released on completion of the work, with a significant amount withheld for a 12 month warranty period, leaving the supplier with a huge financing gap while the project was being executed and potential liability for months after completion. The newly hired manager immediately began to implement a strategy to diversify the company's client base. As he moved to other markets, he was able to trade in on the company's reputation and credibility to command higher premiums from clients in the private sector, generally bypassing the tender process altogether. He also concentrated on short delivery cycles and 100% payment on completion. In four years, 30% of the company's business was from the non-government segment. More dramatically, 85% of its profits came from this segment too. Moral of the story: The company had a legitimate dependency on its reputation for quality and reliability because it was impossible to win contracts without a good track record. However, it had developed an unnecessary dependency on government contracts through sheer history or habit. The new manager, coming from outside the company, had no such mental blinkers and could see this dependency that the owners weren't even aware of. He then leveraged the company's legitimate dependency as an explicit marketing tool to win 12 http://en.wikipedia.org/wiki/Monopsony 15
new business, simultaneously reducing its unnecessary dependency on government contracts with their unfavourable commercial terms. The results were spectacular. Case Study 2 – Culture Shock When a new General Manager took over at a company specialising in turnkey engineering projects, he was immediately struck by the lack of standardised process in his new organisation. No two projects, however similar, followed the same process. Project execution was ad hoc and freewheeling, with lower level managers and engineers making informal decisions in oral negotiations with their counterparts in their client and supplier organisations. True, the business seemed to be running smoothly, but the lack of process seemed to be a ticking time-bomb. The GM came down hard on this laidback culture. He began to insist on standardised processes and timetables for all projects. Instead of informal agreements between operating staff, the company began to send out formal communications of intent, spelling out what was going to be done and when, and what the other party was expected to do. Suppliers were kept on a tight leash and offered little room to negotiate on terms. Within months, dramatic results began to be seen. Project schedules began to slip. Clients began to complain loudly. Suppliers were less accommodating when plans had to change. Things became bad so rapidly that the manager was fired and replaced with one of his senior direct reports who was familiar with the old way of doing things. It took many more months to bring the situation back to normal. It turned out that in the turnkey engineering business, every client and every site is different and requires a different approach. Flexibility is key, because there are lots of unknowns and unexpected developments during the course of a project's execution. Two important aspects of managing these changes are sufficient autonomy for local staff and maintenance of good working relationships between key people, on the client side as well as on the supplier side. A one-size-fits-all approach to process does not work. Neither does a rigid and formal approach to managing external relationships. Moral of the story: The company had a unavoidable dependency on local and individual conditions for every project. The impacts of this dependency could only be mitigated through flexibility, and this was achieved through a conscious culture of autonomy to staff “at the coalface”, mutual respect and adjustment between business partners, and regular 16
and informal communication between key people. The new general manager wrongly saw the informal agreements between people as a failure to formalise dependencies into contracts, and he then tried to replace these with more standardised processes and rigid contractual agreements, but these measures removed the cushion that the company had developed to shield it from its fundamental dependency on local conditions. This case study provides a useful contrast with the previous one because what looks like an unnecessary dependency may not in fact be so. It takes astute observation to understand dependencies and to leverage or manage the legitimate ones while eliminating those that are truly unnecessary. Case Study 3 – The Weakest Link of a Supply Chain An Australian construction company based in Sydney fabricated aluminium and timber windows for the building industry. The company was careful to hedge against currency fluctuations as well as the price of aluminium. But no such arrangements were in place for timber which was sourced from a supplier based in the neighbouring state of Victoria. What the construction company didn't realise was that Chile and Malaysia are the two major sources of timber in the world, and that the Victorian supplier imported timber exclusively from Chile. In February 2010, a major earthquake in Chile disrupted the supply of timber, resulting in worldwide timber shortages and a steep rise in prices. Since the supplier could no longer source timber in a timely or cost-effective manner, the Windows fabricator in turn could not deliver windows to its builders, resulting in delays, losses and unhappy customers. Moral of the story: If the construction company had looked beyond its immediate Australian supplier and studied the entire supply chain for its products, it would have realised the critical dependency it had on Chilean timber. Once formally recognised, it could then have done a number of things to mitigate that dependency, such as hedge against the price of timber, force its supplier to hold sufficient inventory, or diversify its supplies across Chile and Malaysia. Since it neglected to understand and mitigate the effects of this dependency, it paid a heavy price. Case Study 4 – Locked Out: Key Person Risk A small value-added, niche-market IT distributor had a sales team of 6 to 7 people headed 17
by a national manager. The company had recently changed owners, and the new owner had not had enough time to familiarise himself with the company's customers. The national manager was suddenly poached by a competitor. He not only took his entire sales team with him, but also destroyed all records at his old company before leaving, wiping computers clean and removing all business documents. The owner was faced with the closure of his new business, since he had no records of any of his customers. Although his national manager's betrayal could have been criminally prosecuted, he did not have the time or the resources to pursue litigation. This story did however end happily. The owner put together a fresh team under a new and dynamic national manager, and they painstakingly won back all their customers over the next five years. The predatory competitor went bust. Moral of the story: The business had a critical dependency on its customer relationships, physically manifested as its database of customers and records of interactions with them. This dependency was not recognised and formalised as a contract, e.g., a customer database that was regularly backed up and protected against loss or damage. The owner's failure to formalise and manage this dependency cost him heavily. Case Study 5 – The Stress and Strain of an Engineer's Life A company supplying and installing power generators in buildings encountered a problem that had not been anticipated or designed for. A generator installed in the basement of a twelve storey building was hooked up to an exhaust pipe to transport hot fumes to the outside through a shaft in the core of the building. The metal exhaust pipe ran horizontally for 6 metres along the ceiling of the basement and then turned upwards in a 90-degree L-joint to run a further 40 metres to the top of the building through the central shaft. This was the first time the company's engineers had deployed a generator in this configuration, since all their previous engagements involved ground-level generators with short exhaust pipes. The engineers noticed a problem after installation. Whenever the generator was in operation and began to push out hot exhaust fumes, the 40 metre column would expand with the heat, pushing down on the L-joint and forcing the horizontal section to arch downwards almost half a metre. The pipe would return to its position once the generator 18
stopped and the pipe cooled, but this repeated movement due to expansion and contraction was a certain recipe for failure of the pipe due to fatigue. Ultimately, the engineers came up with a solution to replace the rigid L-joint with a much looser (though still airtight) bell-shaped container into which both the horizontal and vertical sections of the pipe could extend. A system of rollers allowed both sections of the pipe to expand smoothly into the “bell” and contract equally smoothly out of it, with neither section's movement impacting the other. Although the company had to spend extra time and money to implement this solution (which they could have built into the initial design had they anticipated the problem), they managed to avert a future, potentially more expensive accident. Moral of the story: The two sections of the generator's exhaust pipe had an unnecessary dependency on each other's thermal expansion due to the rigid L-joint between them. Once this dependency was eliminated through a more loosely-coupled expansion mechanism, the problem disappeared. Case Study 6 – It Goes Without Saying At a large bank in the Middle East, a sincere and hard-working British expat Business Analyst was put in charge of a major branch automation project. The bank's tellers were using mainframe “green screen” terminals, and these were overdue for replacement with a more modern interface. The BA worked hard for many months, interviewing bank staff and management, gathering requirements, creating questionnaires for vendors, issuing RFIs and RFPs, conducting evaluations and negotiations, etc., and finally selected a vendor that seemed superior in every respect. Although he made presentations from time to time to management to keep them informed of his progress, he was doing everything virtually single-handed. Finally, after about a year of hard work, and just before the contract was to be awarded to the successful vendor, the BA organised a demonstration of the branch automation system for the bank's senior management and branch heads. Partway through the impressive demonstration, one of the managers asked, “Does the system support Arabic?” The BA was stunned. He turned to the vendor representative, who looked embarrassed. 19
No, the system did not support Arabic. The commotion and head-shaking that greeted this admission made it abundantly clear that the deal was dead. The poor BA was later spotted sitting at his desk with his head in his hands. The search for a branch automation system had to resume almost from scratch, at a huge cost in time and money. Moral of the story: Although the BA had taken great pains to gather detailed requirements from a wide cross-section of bank staff, Arabic language support was considered to be such an obvious requirement that no one had actually spelt it out. Being a recently-arrived expat, the BA didn't consider it either. And so the most obvious and important requirement for the system went unmet and ruined a year's worth of work. In other words, a legitimate dependency was not formalised in the form of a contract (i.e., the requirements document). Its absence was therefore not noticed until too late and resulted in a nasty surprise. Case Study 7 – “ASSUME” Makes an ASS of U and ME A multinational systems integrator was engaged to deliver a comprehensive Corporate Internet Banking solution to a major Australian bank. Most of the functionality of the application would leverage the bank's mainframe-based product systems, and these would be exposed as services through middleware. What remained to be sourced was a suitable web-friendly front-end product with the required rich user interface. The SI evaluated a few packaged products and selected one for its client. The overall project was a multi-million dollar undertaking with about a hundred people working on the program. There were business analysts, project managers and a large number of technical staff working on the mainframe and middleware, along with a smaller team of front-end designers and developers. It was known that the selected product had been originally designed to work against data in a relational database, but the product vendor's architects had assured the SI that it was relatively easy to graft the front-end onto the middleware services that were being built. A few months into the project, the SI's front-end designers got their first look at the source code of the front-end product, and one of them noticed references to a “Row ID” sprinkled throughout the code. The vendor's front-end developers innocently explained that this referred to the row number in the database. The SI team was aghast. “But there's no database! You know this has to work against services talking to a mainframe!” 20
A meeting of the SI's senior architects and project managers was hastily called. The vendor's senior architects were summoned from overseas. Not having a detailed knowledge of the code, they had not realised that the relational database assumption would be so deeply rooted in their product's design, and they estimated that it would take a further 3 months to fix. All knew that a 3 month delay was unacceptable to the client. Ultimately, the vendor relationship was terminated, the SI's project manager was fired, and the SI barely salvaged the contract after demonstrating a custom-built working prototype to the client. The fiasco threatened to derail a prestigious project and came perilously close to rupturing a highly visible, multi-million dollar relationship between a multinational systems integrator and one of its biggest Australian clients. Moral of the story: The front-end application had hard-coded its dependency on a relational database into every element of its user interface. The vendors' architects had not adequately documented how this dependency should have been isolated, and those who documented the implementation had not adequately recorded the extent of the dependency. Either of these “contracts” would have alerted people to the extent of the dependency. No wonder even the vendors' architects were blind-sided. Case Study 8 – What Makes a Good Technology Platform? To round out this section, let's end with a generic example from the realm of technology – the notion of a “platform”. A platform is a technology environment that provides all the necessary run-time support for applications. These could be virtual machines, interpreters, dynamically linkable libraries, etc. Let's look at five platforms that have been popular over the last decade and a half. i. The Windows operating system is a platform for native Windows applications. While its ubiquity was attractive to Independent Software Vendors (ISVs), there were a few problems with this platform. One was the shift from 16-bit to 32-bit around 1995, which rendered quite a few third-party applications unable to run on the newer version. Another persistent problem was known as “DLL hell”, because of the dependencies of applications on versions of dynamically linked libraries. An upgrade to an underlying system library could and often did break dozens of third-party apps. A third issue was that Microsoft exploited “undocumented Windows 21
APIs” in its own applications that rival app vendors could not use, and so it was not a level playing field. ii. Linux (like any Unix variant) has the notion of symbolic links that it uses to advantage to manage its dynamically linked libraries (called “shared object files”). Shared object files have major version numbers and minor version numbers. Changes in minor version numbers represent changes that do not break the API. Changes in major version numbers represent changes that do break the API. Symbolic links are used to hide changes in minor version numbers but not in major version numbers, so applications that declare their dependencies in terms of the symbolic links will be shielded from irrelevant changes but rightly impacted when the API changes. Also, old and new versions of files can simultaneously exist to support old and new versions of applications. Shared Object Symbolic link Description version libxxx.so.2.1 libxxx.so.2 Original version that applications are dependent on. libxxx.so.2.2 libxxx.so.2 Change in shared object's minor version number is not reflected in the symbolic link, so dependent applications are (correctly) not impacted. Both shared objects can exist simultaneously, but only one is “active”. libxxx.so.3.1 libxxx.so.3 Change in shared object's major version number is reflected in the symbolic link, so dependent applications are impacted, as they should be. Both symbolic links can exist simultaneously, supporting old and new versions of applications. Table 1 – Linux and the use of symbolic links to abstract insignificant version changes iii. The Java Virtual Machine is a platform that runs Java bytecode. The Java language has undergone numerous version revisions, but the JVM has hardly changed in 15 years. Compiled bytecode from 15 years ago runs on the JVM just as well as compiled bytecode from the latest version of Java. iv. The browser is a platform too. In the mid- to late-nineties, developers were discouraged from writing applications that relied on Javascript running on the browser, because there were at least two broad variants of browser platforms – 22
Netscape's and Microsoft's – and even the versions of each browser implemented different versions of the HTML and JavaScript standards (if there were standards at all). It is only since very recently that the browser can be relied upon as a platform that will run applications in a predictable and consistent way, independent of browser vendor. v. Apple's products have a reputation for ease of use, but they are also highly 'closed'. Standard input/output mechanisms like USB ports are not supported, and very little can be achieved by a device without integration to iTunes, iCloud or both. For users who don't mind the restrictions that the Apple platform places, everything “just works”. The competing Android platform is more open, but also a bit more chaotic in terms of multiple versions and multiple implementations by device vendors. Moral of the story: For a platform to be considered a good one, applications must, above all, be able to rely on consistent and stable behaviour. Irrelevant changes must not be allowed to impact applications. There must be considerable latitude in the definition of what constitutes a change, so that the bulk of changes within the platform can be “absorbed” without exposing applications to them. The five examples above showed various approaches to the construction of a stable platform, with varying degrees of success. We can see that the notion of “dependencies” underlies all of them. Summary We have now seen several dramatic examples of how unappreciated dependencies caused, or threatened to cause, problems of reduced agility, increased cost and increased risk to organisations. Not all of these have to do with technology, much less with “SOA technology”, and the choice of examples has been deliberate in that regard. Yet the failure to understand the potential impact of dependencies is the common thread in all these stories. A basic level of dependency-oriented analysis would have highlighted the costs and risks in each of these cases, saving the concerned organisations significant amounts of time, money and risk. Having therefore established the crucial importance of dependencies to an organisation's viability and success, let us see how we can approach the analysis of dependencies in a systematic way. This is what SOA Governance and SOA Management are really about. 23
An Architectural Framework to Analyse Dependencies We need a framework to analyse dependencies before we determine how to govern and manage them. Rather than reinvent the wheel, let's look at some existing and well-understood models to see if they suit our requirements. BAIT as Layers of Dependencies The BAIT model is a popular architectural framework that decomposes an organisation into four layers – Business, Applications, Information (Data) and Technology. BAIT has the right idea about layers, but it is traditionally focused on identifying the entities that exist at each layer and the relationships between those entities. There is no strong emphasis on the aspect of dependencies, even when dealing with relationships. When we adopt the BAIT model to aid us in SOA Governance and Management, we have to adapt it to focus on dependencies, as follows: Traditional concerns of The BAIT Model applied to the BAIT Model SOA concerns Business Layer Business-Level Business Entities (Intent) Dependencies Application Layer Domain Dependencies Applications and Systems (Internal Cohesion) (High Cohesion Principle) Information (Data) Layer Interface Dependencies Domain Data Models (Interfaces/Integration) (Low Coupling Principle) Technology Standards, Technology Layer Technology Dependencies Software and Hardware (Implementation) Fig. 5 – Adapting the BAIT Model to address SOA concerns In other words, we prefer to think of BAIT's four layers as representing four “I”s, referring to the (1) Intent, (2) Internal Cohesion (grouping of business functions into highly cohesive “applications”), (3) Interfaces/Integration between logical functions with low coupling, and (4) physical Implementation of all the components of the logical business organisation. The emphasis on cohesion and coupling is particularly relevant, as we will see in later sections. 24
TOGAF13 Artifacts as Dependency Relationships The TOGAF framework of Enterprise Architecture builds on the BAIT model and describes generic entities in each layer along with their inter-relationships. This is extremely valuable to us when embarking on SOA Governance and SOA Management because the hard work of understanding the components of an organisation and how they fit together has already been done. However, just like with BAIT itself, TOGAF as it stands is not a perfect framework that can be used without adaptation to our treatment of SOA Governance and SOA Management. It does not inherently support the dependency-oriented organisational transformation that we would like to model and plan for. SOA is a single isolated chapter in The Open Group's 780-page TOGAF 9 book, which betrays a traditionalist view of SOA as a subset of IT rather than as the science of analysing and managing dependencies across the board. However, nothing prevents us from using a dependency lens to view the detailed set of entity relationships identified by TOGAF and derive a more applicable model for our requirements. We'll show how to do that in Part III of this document. TOGAF defines some core concepts at the four layers and lists out some important and standardised documents (“viewpoints” in the TOGAF terminology) that provide useful information about these concepts and their interactions. TOGAF uses the term “catalog” to refer to any list. Similarly, it uses the term “matrix” for tables that show the relationship between exactly two concepts, and “diagram” for any depiction of how more than two concepts are related. (a) (b) (c) Fig. 6 – (a) a TOGAF “catalog”, or list of entities; (b) a TOGAF “matrix”, or table of two-entity relationships; (c) a TOGAF “diagram”, or a depiction of multi-entity relationships. If we interpret “relationships” as “dependencies”, matrices become two-entity dependencies and diagrams become multi-entity dependencies. The set of TOGAF-defined matrices and diagrams at each of the four BAIT layers then gives us a ready-made check-list of the dependencies we need to be mindful of within those layers as well as between layers. 13 The Open Group Architecture Framework: http://www.opengroup.org/togaf/ 25
We consciously adapt TOGAF's set of artifacts to align them with a SOA view of the enterprise because TOGAF is not inherently dependency-oriented. In other words, the entity dependencies we study will correspond to our SOA version of the BAIT Model rather than the traditional one. SOA Version of BAIT Model TOGAF Artifacts Business Layer Business-Level (Intent) Dependencies Application Layer Domain Dependencies Core entities, (Internal Cohesion) (High Cohesion Principle) Two-entity dependencies and Information (Data) Layer Interface Dependencies Multi-entity (Interfaces/Integration) (Low Coupling Principle) dependencies Technology Layer Technology Dependencies (Implementation) Fig. 7 – TOGAF artifacts classified by BAIT level We can now begin to see why the traditional view of “SOA Governance” is so limited. SOA Governance SOA Management (“What dependencies are (“How do we manage legitimate?”) dependencies?”) Business Traditional “SOA Governance” concerns Application Development and environment Information (Data) standards, reuse metrics, Web Service version management, repository structures to store Technology WSDL, schema and policy files, etc. Fig. 8 – Traditional “SOA Governance” concerns are more correctly SOA Management, and a subset at that. Little effort is focused on the governance question, “What dependencies are legitimate?” Clearly, our entire approach needs nothing less than an overhaul. It's time to look at how SOA Governance and SOA Management should in fact be done. 26
Part III – A Practical Guide To Governing And Managing Dependencies In the previous two parts of this white paper, we critiqued the industry's approach to SOA Governance and suggested how we could do it better by considering the elements of a more rational approach based on the analysis of dependencies. In this part of the white paper, we will show how to put these elements together into a lightweight method. We'll recommend some key roles and bodies that will be responsible for the functions of SOA Governance and SOA Management, the minimal set of processes they need to carry out, and a manageable set of check-lists that they should use for these purposes. Agencies and Vehicles Key Roles We believe that the primary agency to apply dependency-oriented thinking at every level of an organisation is the Enterprise Architecture function. In organisations with an Enterprise Architecture group, there are usually architects with specialised skills who are tasked with defining the business architecture, application architecture, data architecture and technology architecture, so the BAIT Model is already an operational tool. These specialised architects are the people who need to spearhead a new way of approaching SOA Governance and SOA Management. The role of Enterprise Architecture under this model is to analyse the four layers of the organisation specifically from a dependency focus, first to determine the “what” (the legitimate dependencies that should exist at each level as well as the unnecessary ones that do currently exist) and then to guide the “how” (the programs of work required to align the organisation to the set of legitimate dependencies that were identified). One of the suggestions we have for the Enterprise Architecture function to assist them in transitioning to this world-view is to include professionals from certain other disciplines, even if on a part-time basis. Project Managers, Risk Managers and Contract Lawyers are all used to looking for dependencies in their own areas, and they can inject some of the fresh blood that the traditional Enterprise Architecture group needs to make them effective in a SOA sense. 27
In organisations with a reasonably effective architecture function, here's how business intent is transformed into working systems today: Fig. 9 – Key roles and their traditional concerns Here's the subtle but important change we would like to see: Fig. 10 – Key roles and their recommended concerns It's not as if architects don't already think about dependencies. It's just that dependencies need to be promoted to be a primary, top-of-mind item in all discussions and decisions, 28
especially where the business and IT are involved. Every individual who is part of the decision chain from business objective to implementation, no matter what their level, needs to be sensitised to the importance of dependencies. That's what it means to inculcate “SOA Thinking” within an organisation, and architects need to don the mantle of evangelists to spread this philosophy. The perspectives of project managers, risk managers and contract lawyers could be very useful additions to their skill set, since these give them a “fresh pair of eyes” with which to recognise the diverse set of dependencies impacting the enterprise. SOA Governance and SOA Management become far easier to execute when the organisational culture understands the importance of dependencies. 29
Key Bodies While the Enterprise Architecture function is key to our recommended approach, SOA Governance and SOA Management require the active participation of many different roles and functions to be effective. The business and IT are crucial stakeholders, as we saw. It is essential to involve multiple roles in the SOA Governance and SOA Management functions, and this calls for new organisational structures. We are all too aware of the reputation of committees as inefficient bureaucracies 14, but the multi-disciplinary nature of the tasks at hand, together with the fact that these are ongoing responsibilities, dictate the need for one or more standing bodies rather than ad hoc teams. Broadly, two groups of people are required to carry out the required functions: • a “Dependency Governance Committee” • a “Dependency Management Committee” Now this is a functional classification, and these two logical groups could be realised as several physical committees, e.g., one or more at each level of the BAIT model plus one at the Enterprise level. Some committees could also take responsibility for more than one BAIT layer, as long as they are clear which layer they are dealing with at any given time. The decision on how many physical committees to have is entirely up to the organisation in question, but we would strongly recommend against combining governance and management functions within any single committee. That would defeat one of our primary goals, which is to perform both functions effectively and without confusion. Additionally, although it's the committees' functions that are important and not their names, we would recommend that the term “dependency” be explicitly used in their names in order to inculcate a dependency focus within every person in the organisation. We've lost a decade's worth of benefits thanks to poor naming (“Service-Oriented” as opposed to “Dependency-Oriented”), so let's call a spade a spade this time, however corny it may sound. A committee should either be a “Dependency Governance Committee” or a “Dependency Management Committee”, with prefixes as appropriate to the BAIT level and/or business unit. 14 Two of our personal favourites are “A camel is a horse designed by a committee” and “A committee is a group of people who individually can do nothing, but who get together to decide that nothing can be done.” 30
In a nutshell, • The Dependency Governance Committee is responsible for defining which of the dependencies at the level they are responsible for are legitimate and which are not. They are also responsible for prioritising the programs of work to remediate dependencies. • The Dependency Management Committee is responsible for costing, planning and initiating the programs of work that will bring and keep the organisation in line with the model validated by the Dependency Governance Committee and according to the priority decided by that committee. The Dependency Governance Committee should have a fair representation of Enterprise Architects, with specialists in each of the BAIT layers. At least some of the members of the Dependency Governance Committee should have a background in one of the non-IT professions listed earlier 15, because these professionals are trained to look for dependencies. Depending on the layer of the BAIT model that the committee is responsible for, there should be representation from senior executives, business managers, functional heads and technology specialists. This may mean separate committees, or a single committee with some fixed members and a number of others who attend only the sessions that are relevant to them. Importantly, the Dependency Governance Committee must have authority and access to funding to be able to approve programs of work, otherwise the organisation will be unable to remediate the dependencies identified as invalid. One of the reasons why we expended so much ink in Part II to present 8 separate case studies was to hammer home the idea that attention to dependencies saves real money. Business unit heads should therefore not be churlish about funding dependency remediation tasks, and their representation on Governance committees serves more than just a decorative function! The Dependency Management Committee should comprise people skilled in costing and planning, typically project managers and functional heads. They may need representation from Enterprise Architecture to keep them focused on the dependency aspect while they attend to their more familiar roles of costing and planning programs of work. Again, there will be a need to have different people engaged for different layers of the BAIT model, so there could be considerable variety in the structure and/or composition of the committee(s). 15 Project managers, risk managers and contract lawyers. 31
How Many Committees? The dependency governance and management functions can be performed by just two committees in a small organisation, since they can do justice to the complexity of the entire organisation themselves. As organisations become larger, the committees will need to become more specialised, as the following examples illustrate. Dependency Governance Dependency Management Committee Committee Fig. 11 – Committees for a small organisation Enterprise Dependency Enterprise Dependency Governance Committee Management Committee Business Dependency Business Dependency Governance Committee Management Committee Application & Information Application & Information Dependency Governance Dependency Management Committee Committee Technology Dependency Technology Dependency Governance Committee Management Committee Fig. 12 – Committees for a medium-sized organisation Enterprise Dependency Enterprise Dependency Governance Committee Management Committee Business Dependency Business Dependency Governance Committee Management Committee Application Dependency Application Dependency Governance Committee Management Committee Information Dependency Information Dependency Governance Committee Management Committee Technology Dependency Technology Dependency Business Governance Committee Management Committee Units / Geographies Fig. 13 – Committees for a large organisation 32
If this set of committees is beginning to look too bloated, our audacious claim is that these are the only governance and management structures that an organisation will ever need, so this could in fact represent a drastic simplification to the plethora of management bodies that currently exist in a given organisation. Let's look at a few special cases to support this claim. Security dependencies: Security is an important aspect of business operations, and information security traditionally tends to be treated as an IT concern. However, security concerns are relevant at different levels of the BAIT model. Perimeter security, for example, may be relevant only at the level of technology, since it is completely opaque to higher layers. Data Leakage Protection applies at all levels from the business layer down, since it is impacted by business processes, application design, data interchange as well as technology-related aspects like USB device control and disk encryption. PCI-DSS 16 compliance requirements may or may not impact business processes but will certainly impact application design, data storage and encryption/tokenisation technology. The dependency governance and management committees at these various levels will address and manage the security concerns listed above, as well as any others that arise from time to time. Non-technology domains that are not line-of-business, such as Human Resources, Accounting, Finance or Legal, also fit into this dependency-oriented model. Human Resources dependencies: Clearly, an organisation that employs many more contractors than permanent employees has a very different dependency profile than one with the opposite composition. The contractor-based organisation can downsize more readily in an economic downturn and re-hire when required, but it has a more tenuous hold on IP due to the less “sticky” nature of employee knowledge. The contractor-based organisation will have to invest more heavily into knowledge-management systems to mitigate against their risk of losing critical business and operations knowledge. The organisation with a higher proportion of permanent employees will have to be more cautious about hiring in an economic upturn because of their larger financial exposure arising from retrenchment benefits they may have to pay out when downsizing. A business dependency governance committee and business dependency management committee focused on the HR business unit will be able to formalise these concerns and evolve mitigating strategies. 16 Payment Card Industry Data Security Standard 33
Finance dependencies: An organisation that is funded mainly by debt has a different dependency profile to one funded mainly by equity. A debtor and a creditor represent dependencies of different kinds, and every financial instrument that is issued or purchased imposes a different flavour of dependency on the enterprise. Financial controllers make decisions based on such dependencies all the time, whether or not they use the word “dependency”. Regardless, the business dependency governance and management committees for the Finance business unit will standardise these functions in a consistent corporate style. In short, these varied examples support our claim that SOA, being all about dependencies, is not about technology but is an organising principle for the enterprise and lends itself to a simple and consistent form of governance and management. Form follows function, and the structure of the required governance and management committees follows the discipline of dependency-oriented thinking. [It may be argued that the bulk of a modern enterprise's business logic tends to be coded into software-based systems with very little residual manual processing, so “it's all IT anyway”. Let us refute this argument with a simple analogy. Just because all legal documents in a country are in the English language does not imply that a mastery of English is all that is required to understand or to draft legal documents. One needs to understand law. In similar fashion, even if every piece of business logic in an organisation is implemented within a technology platform, understanding how to govern and manage technology alone is not sufficient. The principles behind how business logic and business data are identified and organised need to be understood, and the analysis of dependencies is a crucial tool to achieve this understanding.] 34
Functions and Processes Broadly speaking, the processes required for SOA Governance and SOA Management are either one-time or recurring. To keep an otherwise dry topic light and readable, we will illustrate what occurs during each of these processes through imaginary conversations between various parties. The players in all these scenarios are the following: G – Dependency Governance Committee M – Dependency Management Committee B – Any Business Unit One-time Processes When an organisation embarks on doing SOA Governance and Management the way this white paper recommends, there will be a one-time activity (which could of course be broken up into phases for manageability) on the part of the Dependency Governance Committee to do two things: 1. Agree on the set of approved dependencies at each level of the organisation as suggested by the BAIT model; 2. Identify the dependencies that actually exist, specifically the ones that fall outside the set of approved dependencies above. The latter list (the set of actual dependencies that are not approved) is the input to the Dependency Management Committee to plan a program of work to remediate. 35
Governance Functions Management Functions Publication: List of Approved Dependencies One-Time Dependency List of Dependencies to be remediated Review Update: List of Approved Dependencies Need not Periodic Remediation align to Review Dependency Program Business cycle Review Proposal Project List of cycle Dependencies to be remediated List of programs, Remediation grouped and costed Program Review Business Business Requirements Driver (Objective) Initiative Details New New Business Initiative Initiative Project Review Proposal cycle Non-approval: List of Dependencies in violation Update: List of Approved Dependencies Approval BAU Program Management Approval: Prioritised list of programs, with funding Report: Dependencies remediated Dependency-related Process Existing Business Process Fig. 14 – Dependency Governance and Management Processes 36
Initial Dependency Review Scenario 1: G: “We've done a review and decided that the only legitimate dependencies are these. There are many actual dependencies we've found in our systems that are not in this approved set. They need to be eliminated.” M: “OK, we'll come back with a proposal for a program of work to remediate them.” Recurring Processes There are recurring processes within both the governance and management functions. Recurring governance processes are reviews to re-validate the set of legitimate dependencies at any level and to identify afresh the set of existing dependencies that fall outside of this approved set. The latter set (which is expected to shrink with each iteration) is then an input to management committees to initiate programs of work to remediate. Prioritisation and approval of these proposed programs of work is also a governance function. Periodic Dependency Review Scenario 1: G: “Given the changing nature of our business since the last review, some new dependencies are now deemed legitimate and some old ones no longer are.” M: “OK, we'll make a note of the new set of approved dependencies and come back to you with a proposal for a program of work to eliminate the ones that are no longer approved.” Scenario 2: G: “Some new dependencies that are not in the approved list seem to have “snuck into” our systems since our last review. These need to be eliminated ASAP.” M: “We don't understand how that could have happened. We'll come back to you with a proposal for a program of work to remediate them.” Remediation Program Review M: “We've put together a proposal for a set of programs that will remediate the unapproved 37
dependencies that you identified. We've worked out the costs and benefits of each of them. Please tell us how you would like to prioritise them for execution, and please approve funding for the ones you want done in the current period.” G: “We think you should initiate the following subset of programs in the current reporting period and defer the remaining for a future period. The funding for the programs in the current period is hereby approved.” M: “We'll initiate these programs right away.” 38
Recurring management processes are intended to identify programs of work to remediate the invalid dependencies identified by the governance committee(s) and to manage these programs once approved by them. Remediation Program Proposal Scenario 1: M (internal conversation): “Given the set of existing dependencies that were identified by the Dependency Governance Committee as not being on the approved list, let's work out the costs and benefits of eliminating them, and group these tasks into cohesive programs of work for them to prioritise and approve.” BAU Program Management Once a Dependency Remediation Program is approved, it is managed in exactly identical fashion to regular business projects. These are not to be treated as projects of secondary importance or nice-to-haves because of a mistaken characterisation of SOA as technology. On the contrary, the involvement of business and technology representatives and proper positioning of the benefits of these initiatives by Enterprise Architecture should ensure that they are taken equally seriously as programs initiated by the business. After all, unfixed dependencies mean significant amounts of real money, as we have seen earlier. The business initiatives detailed in the next section also feed into Business-As Usual (BAU) program management alongside dependency remediation programs. The standard discipline of conducting Post-Implementation Reviews (PIRs) to evaluate the success of a program should be conducted for dependency remediation initiatives as well. Granted, the benefits are not always immediate but recurring, but it should be possible to assess if the dependencies slated to be eliminated were in fact removed, and a fresh assessment of the costs and benefits of the initiative should be conducted to report back to the Dependency Governance Committee. New Initiative Appraisal An event that could trigger a fresh governance activity is when there is a business driver (objective) necessitating a program of work, the proposal is put forward, and the governance committee evaluates if this introduces any fresh dependencies that are not on the approved list. (Of course, any additional dependencies may be justified, in which case 39
the approved list is updated, or may even make some existing dependencies unnecessary, which again results in updates to the approved list). The proposal may need to be amended to avoid introducing dependencies that are not approved. A new initiative could be on the business side (e.g., entering a new market, introducing a new product, acquiring a company, outsourcing or re-insourcing a business function, changing a business process, signing up a new partnership, etc.) The initiative could also be on the technology side (e.g., buying and installing a new product system, developing a new system in-house, outsourcing or re-insourcing a technology function, changing the technology implementation of a function or process, decommissioning a system or application, undertaking a turnkey project, etc.) All of these changes have dependency implications that have to be reviewed before the initiatives concerned can be approved. A large part of the technical and business “debt” that is incurred by organisations is due to a failure to assess new initiatives for their dependency implications. The following process is meant to plug that loophole. Scenario 1: B: “Here's our proposal for a new project.” G: “It appears that your project is about to eliminate some dependencies. That's good. Project approved.” B: “Thanks, we'll initiate the project.” M: “We'll make a note that the set of approved (and existing) dependencies will shrink after this project.” Scenario 2: B: “Here's our proposal for a new project.” G: “Your project introduces a new set of dependencies, but they do seem to be valid, so we'll expand the set of approved dependencies. Project approved.” B: “Thanks, we'll initiate the project.” M: “We'll make a note of the expanded set of approved dependencies.” 40
Scenario 3: B: “Here's our proposal for a new project.” G: “Your project will introduce new dependencies that are not valid. We're afraid we can't approve it. Go back and re-think your processes and/or designs so that they stay within the set of approved dependencies.” B: “OK, we'll get back to you with a reworked proposal.” M: (“Nothing for us to do.”) Incorporating a Dependency Focus into New Initiative Appraisal The business case for any program of work should include its impact on dependencies (positive or negative), in addition to the standard calculations of cost-to-implement and business benefit. Dependencies carry a hidden yet heavy cost in terms of business agility, operating cost and operational risk, so we would like to have these made explicit and brought to the notice of approving authorities. Introducing dependencies (legitimate or not) incurs cost and risk, and removing them has the opposite effect. This is where the work done by governance committees in identifying dependencies will form a useful input. The following page has a simplified form that illustrates this. 41
New Initiative Appraisal Form Cost Business Benefit Dependency impacts (+/-) Description of proposed project CapEx OpEx (3 Yrs) Business agility Operating costs Operational risk (3 Yrs) (3 Yrs) (3 Yrs) assessment Remove replication of customer (200,000) (25,000) 90,000 50,000 80,000 Changes from address data by consolidating it (25,000) 90,000 50,000 80,000 medium to low within system X where it belongs (25,000) 90,000 50,000 80,000 Approved by: Approval date: Fig. 15 – An indicative format for a “New Initiative Appraisal Form” The form above illustrates how the hidden benefits from the removal of dependencies (or conversely, the hidden costs of introducing fresh dependencies), once quantified, could sway the business case for a project either way. In this example, the business case for a proposed project doesn't by itself stack up because the total business benefit over a 3 year ROI horizon ($270,000) is less than the cost incurred ($275,000). However, the benefit goes up enormously once the improvements to the organisation's agility, cost and risk profile are taken into account. A dependency focus therefore drives behaviour that is desirable over the longer term. This is the kind of change we would like to see in the way organisations evaluate programs of work. It represents a “least-dependency” model of doing business which keeps them lean and low-risk.
Processes in Steady State The recurring processes around dependency review followed by remediation are expected to wind down as the culture of dependency-sensitivity begins to take hold and the number of exceptions correspondingly reduces. In steady state, the only processes that we are likely to see are New Initiative Proposal, New Initiative Review and BAU Program Management, since these processes will subsume the dependency-related activities that are the sole focus of the other processes. Periodic Dependency Reviews will still take place, but with any luck, no deviations will have occurred and hence no remediation programs will need to be initiated. Governance Functions Management Functions Business Business Requirements Driver (Objective) Initiative Details New New Business Initiative Initiative Project Review Proposal cycle Non-approval: List of Dependencies in violation Update: List of Approved Dependencies Approval BAU Program Management Dependency-related Process Existing Business Process Fig. 16 – Ideal Dependency Governance and Management Processes in “Steady State” 43
Governance and Management Check-lists With our governance/management bodies and processes in place, it's time to look at what they will actually do. Here is a recommended approach based on a detailed analysis of organisational layers along with a sample check-list of questions for the committees to ask. Classification of Dependencies Consider a set of dependencies between entities within a BAIT layer (or across layers). Entity 1 dependency 1 dependency 3 dependency 2 Entity 4 Entity 3 Entity 2 dependency 4 Entity 5 Fig. 17 – An unclassified set of dependencies These dependencies may be further categorised into (say) two broad groups because they are themselves related in some way. A governance or management committee may then assign responsibility for these two dependency groups to two separate teams. Dependency Group 1 (covers dependencies 1 and 2) Entity 1 dependency 1 dependency 3 dependency 2 Entity 4 Entity 3 Entity 2 dependency 4 Entity 5 Dependency Group 2 (covers dependencies 3 and 4) Fig. 18 – Dependency Groups containing cohesive sets of dependencies 44
The Dependency Governance Committee(s) and Dependency Management Committee(s) may want to organise themselves into specialised teams to study and monitor smaller, cohesive sets of dependencies. We will call this intermediate level of granularity (i.e., somewhere between a single dependency and all the dependencies that may exist at a BAIT layer) a “Dependency Group”. Although not identified in TOGAF, a Dependency Group helps to provide a “system” view of related dependencies. In the following sections, we will document not just the set of dependencies suggested by TOGAF at each BAIT layer, but also our suggested classification of these dependencies into Dependency Groups that can be assigned to specialised teams as their responsibility. The List of Check-Lists Our check-lists will be numbered according to the following scheme: G(overnance) M(anagement) Basic G-00 M-00 E(nterprise) EG-00 EM-00 B(usiness) BG-00 BM-00 A(pplication) AG-00 AM-00 I(nformation) IG-00 IM-00 T(echnology) TG-00 TM-00 A Basic Governance Check-list Regardless of the level of the BAIT model at which governance is sought to be applied, there are some standard questions that will need to be asked: G-01 What are the core entities involved? G-02 What are the legitimate dependencies between them? G-03 What dependencies can we identify in the current situation that fall outside this set of legitimate dependencies? There will of course be more specialised governance questions depending on the BAIT layer concerned, and we will cover them under the appropriate sections. 45
A Basic Management Check-list In similar fashion, there are some standard management questions that need to be asked at every level of the BAIT model: M-01 How do we express the dependencies between entities as suitable contracts? M-02 How will we enforce contracts? M-03 How will we monitor adherence to contracts? M-04 How do we re-engineer our existing systems to align them with the target model? Every BAIT layer will add its own specialised management questions to this list, and we will examine each under its corresponding section. When an organisation's governance and management committees are staffed with appropriate experts and when they adapt to the dependency-oriented way of thinking, they will be able to formulate more relevant questions to add to these check-lists. 46
Fundamental Enterprise Dependencies Before we even go into the four BAIT layers, we need to understand some fundamental dependencies at the overall organisational level as illustrated below. The reason why the “enterprise” has been split from even the business layer is that most organisations are highly silo-ed at the business unit level (from immediately below the CEO). Hence enterprise concerns need to be addressed at a level higher than those of business units. Statement of Purpose Goal vision and alignment mission Value Chain responsibility external External Stakeholder assignment Enterprise dependencies Entity Core Roles governing principles Business Principle Architectural Principle Policy Framework Fig. 19 – Dependencies and Dependency Groups at the Enterprise level (TOGAF) The Dependency Governance Committee at this level is necessarily high-powered. It should consist of the Head of Architecture and the CIO engaging with the CEO and executive management. This is the only body that can authoritatively decide on the legitimate dependencies at the enterprise level. 47
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance
(Deprecated) Slicing the Gordian Knot of SOA Governance

Recommended

A Real Time Application Integration Solution
A Real Time Application Integration SolutionA Real Time Application Integration Solution
A Real Time Application Integration SolutionMatthew Pulis
 
The Defender's Dilemma
The Defender's DilemmaThe Defender's Dilemma
The Defender's DilemmaSymantec
 
Cms user manual
Cms user manualCms user manual
Cms user manualcsscctv2011
 
Hacking.pdf
Hacking.pdfHacking.pdf
Hacking.pdfAmbuj Sharma
 
It Sector Risk Assessment Report Final
It Sector Risk Assessment Report FinalIt Sector Risk Assessment Report Final
It Sector Risk Assessment Report FinalHongyang Wang
 
Rand rr2364
Rand rr2364Rand rr2364
Rand rr2364JA Larson
 
By d ui_styleguide_2012_fp35
By d ui_styleguide_2012_fp35By d ui_styleguide_2012_fp35
By d ui_styleguide_2012_fp35Александр Луганский
 
Whitepaper on distributed ledger technology
Whitepaper on distributed ledger technologyWhitepaper on distributed ledger technology
Whitepaper on distributed ledger technologyUnder the sharing mood
 

Recommended

A Real Time Application Integration Solution
A Real Time Application Integration SolutionA Real Time Application Integration Solution
A Real Time Application Integration SolutionMatthew Pulis
 
The Defender's Dilemma
The Defender's DilemmaThe Defender's Dilemma
The Defender's DilemmaSymantec
 
Cms user manual
Cms user manualCms user manual
Cms user manualcsscctv2011
 
Hacking.pdf
Hacking.pdfHacking.pdf
Hacking.pdfAmbuj Sharma
 
It Sector Risk Assessment Report Final
It Sector Risk Assessment Report FinalIt Sector Risk Assessment Report Final
It Sector Risk Assessment Report FinalHongyang Wang
 
Rand rr2364
Rand rr2364Rand rr2364
Rand rr2364JA Larson
 
By d ui_styleguide_2012_fp35
By d ui_styleguide_2012_fp35By d ui_styleguide_2012_fp35
By d ui_styleguide_2012_fp35Александр Луганский
 
Whitepaper on distributed ledger technology
Whitepaper on distributed ledger technologyWhitepaper on distributed ledger technology
Whitepaper on distributed ledger technologyUnder the sharing mood
 
Implementing and auditing security controls part 1
Implementing and auditing security controls part 1Implementing and auditing security controls part 1
Implementing and auditing security controls part 1Rafel Ivgi
 
Siem & log management
Siem & log managementSiem & log management
Siem & log managementRafel Ivgi
 
Data science book
Data science bookData science book
Data science bookJOHNSONTIRKEY7
 
Interview questions
Interview questionsInterview questions
Interview questionsAamirJadoon5
 
Span derivés gb_200802 _2__tcm6-44568
Span derivés gb_200802 _2__tcm6-44568Span derivés gb_200802 _2__tcm6-44568
Span derivés gb_200802 _2__tcm6-44568stratforcms
 
Improving software testing efficiency using automation methods by thuravupala...
Improving software testing efficiency using automation methods by thuravupala...Improving software testing efficiency using automation methods by thuravupala...
Improving software testing efficiency using automation methods by thuravupala...Ravindranath Tagore
 
Hacker techniques, exploit and incident handling
Hacker techniques, exploit and incident handlingHacker techniques, exploit and incident handling
Hacker techniques, exploit and incident handlingRafel Ivgi
 
Advanced web application hacking and exploitation
Advanced web application hacking and exploitationAdvanced web application hacking and exploitation
Advanced web application hacking and exploitationRafel Ivgi
 
Guidebook: Supportive Housing Asset Management
Guidebook: Supportive Housing Asset ManagementGuidebook: Supportive Housing Asset Management
Guidebook: Supportive Housing Asset Managementkhals
 
Configuration-Release management
Configuration-Release managementConfiguration-Release management
Configuration-Release managementRavindranath Tagore
 
White Paper Guide For Developing Security Plans
White Paper Guide For Developing Security PlansWhite Paper Guide For Developing Security Plans
White Paper Guide For Developing Security Plansbdana68
 
ES410 Report
ES410 ReportES410 Report
ES410 ReportMatt Dent
 
Java secure development part 1
Java secure development part 1Java secure development part 1
Java secure development part 1Rafel Ivgi
 
R handbook - from Installation to Text Analytics
R handbook - from Installation to Text AnalyticsR handbook - from Installation to Text Analytics
R handbook - from Installation to Text AnalyticsATI_2205
 
3rd Year Formula Student Frame Project Report
3rd Year Formula Student Frame Project Report3rd Year Formula Student Frame Project Report
3rd Year Formula Student Frame Project ReportJessica Byrne
 
Getting started in Transmedia Storytelling
Getting started in Transmedia Storytelling Getting started in Transmedia Storytelling
Getting started in Transmedia Storytelling Robert Pratten
 
Book telecommunications
Book telecommunications Book telecommunications
Book telecommunications Bendito Freitas Ribeiro
 
bkremer-report-final
bkremer-report-finalbkremer-report-final
bkremer-report-finalBen Kremer
 
Graduation Report
Graduation ReportGraduation Report
Graduation Reportzied khayechi
 
BizTalk Practical Course Preview
BizTalk Practical Course PreviewBizTalk Practical Course Preview
BizTalk Practical Course PreviewMoustafaRefaat
 
E participation study
E participation study E participation study
E participation study LG ICT Network by South African Local Government Assocations SALGA
 
Soa In The Real World
Soa In The Real WorldSoa In The Real World
Soa In The Real Worldssiliveri
 

More Related Content

What's hot

Implementing and auditing security controls part 1
Implementing and auditing security controls part 1Implementing and auditing security controls part 1
Implementing and auditing security controls part 1Rafel Ivgi
 
Siem & log management
Siem & log managementSiem & log management
Siem & log managementRafel Ivgi
 
Interview questions
Interview questionsInterview questions
Interview questionsAamirJadoon5
 
Span derivés gb_200802 _2__tcm6-44568
Span derivés gb_200802 _2__tcm6-44568Span derivés gb_200802 _2__tcm6-44568
Span derivés gb_200802 _2__tcm6-44568stratforcms
 
Improving software testing efficiency using automation methods by thuravupala...
Improving software testing efficiency using automation methods by thuravupala...Improving software testing efficiency using automation methods by thuravupala...
Improving software testing efficiency using automation methods by thuravupala...Ravindranath Tagore
 
Hacker techniques, exploit and incident handling
Hacker techniques, exploit and incident handlingHacker techniques, exploit and incident handling
Hacker techniques, exploit and incident handlingRafel Ivgi
 
Advanced web application hacking and exploitation
Advanced web application hacking and exploitationAdvanced web application hacking and exploitation
Advanced web application hacking and exploitationRafel Ivgi
 
Guidebook: Supportive Housing Asset Management
Guidebook: Supportive Housing Asset ManagementGuidebook: Supportive Housing Asset Management
Guidebook: Supportive Housing Asset Managementkhals
 
Configuration-Release management
Configuration-Release managementConfiguration-Release management
Configuration-Release managementRavindranath Tagore
 
White Paper Guide For Developing Security Plans
White Paper Guide For Developing Security PlansWhite Paper Guide For Developing Security Plans
White Paper Guide For Developing Security Plansbdana68
 
ES410 Report
ES410 ReportES410 Report
ES410 ReportMatt Dent
 
Java secure development part 1
Java secure development part 1Java secure development part 1
Java secure development part 1Rafel Ivgi
 
R handbook - from Installation to Text Analytics
R handbook - from Installation to Text AnalyticsR handbook - from Installation to Text Analytics
R handbook - from Installation to Text AnalyticsATI_2205
 
3rd Year Formula Student Frame Project Report
3rd Year Formula Student Frame Project Report3rd Year Formula Student Frame Project Report
3rd Year Formula Student Frame Project ReportJessica Byrne
 
Getting started in Transmedia Storytelling
Getting started in Transmedia Storytelling Getting started in Transmedia Storytelling
Getting started in Transmedia Storytelling Robert Pratten
 

What's hot (17)

Implementing and auditing security controls part 1
Implementing and auditing security controls part 1Implementing and auditing security controls part 1
Implementing and auditing security controls part 1
 
Siem & log management
Siem & log managementSiem & log management
Siem & log management
 
Data science book
Data science bookData science book
Data science book
 
Interview questions
Interview questionsInterview questions
Interview questions
 
Span derivés gb_200802 _2__tcm6-44568
Span derivés gb_200802 _2__tcm6-44568Span derivés gb_200802 _2__tcm6-44568
Span derivés gb_200802 _2__tcm6-44568
 
Improving software testing efficiency using automation methods by thuravupala...
Improving software testing efficiency using automation methods by thuravupala...Improving software testing efficiency using automation methods by thuravupala...
Improving software testing efficiency using automation methods by thuravupala...
 
Hacker techniques, exploit and incident handling
Hacker techniques, exploit and incident handlingHacker techniques, exploit and incident handling
Hacker techniques, exploit and incident handling
 
Advanced web application hacking and exploitation
Advanced web application hacking and exploitationAdvanced web application hacking and exploitation
Advanced web application hacking and exploitation
 
Guidebook: Supportive Housing Asset Management
Guidebook: Supportive Housing Asset ManagementGuidebook: Supportive Housing Asset Management
Guidebook: Supportive Housing Asset Management
 
Configuration-Release management
Configuration-Release managementConfiguration-Release management
Configuration-Release management
 
White Paper Guide For Developing Security Plans
White Paper Guide For Developing Security PlansWhite Paper Guide For Developing Security Plans
White Paper Guide For Developing Security Plans
 
ES410 Report
ES410 ReportES410 Report
ES410 Report
 
Java secure development part 1
Java secure development part 1Java secure development part 1
Java secure development part 1
 
R handbook - from Installation to Text Analytics
R handbook - from Installation to Text AnalyticsR handbook - from Installation to Text Analytics
R handbook - from Installation to Text Analytics
 
3rd Year Formula Student Frame Project Report
3rd Year Formula Student Frame Project Report3rd Year Formula Student Frame Project Report
3rd Year Formula Student Frame Project Report
 
Getting started in Transmedia Storytelling
Getting started in Transmedia Storytelling Getting started in Transmedia Storytelling
Getting started in Transmedia Storytelling
 
Book telecommunications
Book telecommunications Book telecommunications
Book telecommunications
 

Similar to (Deprecated) Slicing the Gordian Knot of SOA Governance

bkremer-report-final
bkremer-report-finalbkremer-report-final
bkremer-report-finalBen Kremer
 
BizTalk Practical Course Preview
BizTalk Practical Course PreviewBizTalk Practical Course Preview
BizTalk Practical Course PreviewMoustafaRefaat
 
Soa In The Real World
Soa In The Real WorldSoa In The Real World
Soa In The Real Worldssiliveri
 
irmpg_3.7_python_202301.pdf
irmpg_3.7_python_202301.pdfirmpg_3.7_python_202301.pdf
irmpg_3.7_python_202301.pdfFernandoBello39
 
Отчет из Германии о 4й промышленной революции
Отчет из Германии о 4й промышленной революции Отчет из Германии о 4й промышленной революции
Отчет из Германии о 4й промышленной революции Sergey Zhdanov
 
Solmanfocusedbuild
SolmanfocusedbuildSolmanfocusedbuild
SolmanfocusedbuildGhassen B
 
SAP MM Tutorial ds_42_tutorial_en.pdf
SAP MM Tutorial ds_42_tutorial_en.pdfSAP MM Tutorial ds_42_tutorial_en.pdf
SAP MM Tutorial ds_42_tutorial_en.pdfsjha120721
 
Dimensional modeling in a bi environment
Dimensional modeling in a bi environmentDimensional modeling in a bi environment
Dimensional modeling in a bi environmentdivjeev
 
Deployment guide series ibm tivoli access manager for e business v6.0 sg247207
Deployment guide series ibm tivoli access manager for e business v6.0 sg247207Deployment guide series ibm tivoli access manager for e business v6.0 sg247207
Deployment guide series ibm tivoli access manager for e business v6.0 sg247207Banking at Ho Chi Minh city
 
Deployment guide series ibm tivoli access manager for e business v6.0 sg247207
Deployment guide series ibm tivoli access manager for e business v6.0 sg247207Deployment guide series ibm tivoli access manager for e business v6.0 sg247207
Deployment guide series ibm tivoli access manager for e business v6.0 sg247207Banking at Ho Chi Minh city
 
Reinventing the City
Reinventing the CityReinventing the City
Reinventing the CityDean Pallen
 

Similar to (Deprecated) Slicing the Gordian Knot of SOA Governance (20)

bkremer-report-final
bkremer-report-finalbkremer-report-final
bkremer-report-final
 
Graduation Report
Graduation ReportGraduation Report
Graduation Report
 
BizTalk Practical Course Preview
BizTalk Practical Course PreviewBizTalk Practical Course Preview
BizTalk Practical Course Preview
 
E participation study
E participation study E participation study
E participation study
 
Soa In The Real World
Soa In The Real WorldSoa In The Real World
Soa In The Real World
 
irmpg_3.7_python_202301.pdf
irmpg_3.7_python_202301.pdfirmpg_3.7_python_202301.pdf
irmpg_3.7_python_202301.pdf
 
This is
This is This is
This is
 
Pentest standard
Pentest standardPentest standard
Pentest standard
 
Отчет из Германии о 4й промышленной революции
Отчет из Германии о 4й промышленной революции Отчет из Германии о 4й промышленной революции
Отчет из Германии о 4й промышленной революции
 
Industry 4.0 Final Report, National Academy of Science and Engineering of Ger...
Industry 4.0 Final Report, National Academy of Science and Engineering of Ger...Industry 4.0 Final Report, National Academy of Science and Engineering of Ger...
Industry 4.0 Final Report, National Academy of Science and Engineering of Ger...
 
Solmanfocusedbuild
SolmanfocusedbuildSolmanfocusedbuild
Solmanfocusedbuild
 
Sg246399
Sg246399Sg246399
Sg246399
 
SAP MM Tutorial ds_42_tutorial_en.pdf
SAP MM Tutorial ds_42_tutorial_en.pdfSAP MM Tutorial ds_42_tutorial_en.pdf
SAP MM Tutorial ds_42_tutorial_en.pdf
 
It project development fundamentals
It project development fundamentalsIt project development fundamentals
It project development fundamentals
 
Dynamics AX/ X++
Dynamics AX/ X++Dynamics AX/ X++
Dynamics AX/ X++
 
Ppm7.5 demand cg
Ppm7.5 demand cgPpm7.5 demand cg
Ppm7.5 demand cg
 
Dimensional modeling in a bi environment
Dimensional modeling in a bi environmentDimensional modeling in a bi environment
Dimensional modeling in a bi environment
 
Deployment guide series ibm tivoli access manager for e business v6.0 sg247207
Deployment guide series ibm tivoli access manager for e business v6.0 sg247207Deployment guide series ibm tivoli access manager for e business v6.0 sg247207
Deployment guide series ibm tivoli access manager for e business v6.0 sg247207
 
Deployment guide series ibm tivoli access manager for e business v6.0 sg247207
Deployment guide series ibm tivoli access manager for e business v6.0 sg247207Deployment guide series ibm tivoli access manager for e business v6.0 sg247207
Deployment guide series ibm tivoli access manager for e business v6.0 sg247207
 
Reinventing the City
Reinventing the CityReinventing the City
Reinventing the City
 

More from Ganesh Prasad

Workshop Slides - Introduction to Dependency-Oriented Thinking" Feb 15, 2014,...
Workshop Slides - Introduction to Dependency-Oriented Thinking" Feb 15, 2014,...Workshop Slides - Introduction to Dependency-Oriented Thinking" Feb 15, 2014,...
Workshop Slides - Introduction to Dependency-Oriented Thinking" Feb 15, 2014,...Ganesh Prasad
 
Dependency-Oriented Thinking Sydney Workshop Brochure and Schedule (Feb 15 2014)
Dependency-Oriented Thinking Sydney Workshop Brochure and Schedule (Feb 15 2014)Dependency-Oriented Thinking Sydney Workshop Brochure and Schedule (Feb 15 2014)
Dependency-Oriented Thinking Sydney Workshop Brochure and Schedule (Feb 15 2014)Ganesh Prasad
 
50 data principles for loosely coupled identity management v1 0
50 data principles for loosely coupled identity management v1 050 data principles for loosely coupled identity management v1 0
50 data principles for loosely coupled identity management v1 0Ganesh Prasad
 
Sofea in a soa ecosystem v0 4
Sofea in a soa ecosystem v0 4Sofea in a soa ecosystem v0 4
Sofea in a soa ecosystem v0 4Ganesh Prasad
 
Life above the service tier preso v1 0
Life above the service tier preso v1 0Life above the service tier preso v1 0
Life above the service tier preso v1 0Ganesh Prasad
 
Life above the_service_tier_v1.1
Life above the_service_tier_v1.1Life above the_service_tier_v1.1
Life above the_service_tier_v1.1Ganesh Prasad
 

More from Ganesh Prasad (7)

Enterprise REST
Enterprise RESTEnterprise REST
Enterprise REST
 
Workshop Slides - Introduction to Dependency-Oriented Thinking" Feb 15, 2014,...
Workshop Slides - Introduction to Dependency-Oriented Thinking" Feb 15, 2014,...Workshop Slides - Introduction to Dependency-Oriented Thinking" Feb 15, 2014,...
Workshop Slides - Introduction to Dependency-Oriented Thinking" Feb 15, 2014,...
 
Dependency-Oriented Thinking Sydney Workshop Brochure and Schedule (Feb 15 2014)
Dependency-Oriented Thinking Sydney Workshop Brochure and Schedule (Feb 15 2014)Dependency-Oriented Thinking Sydney Workshop Brochure and Schedule (Feb 15 2014)
Dependency-Oriented Thinking Sydney Workshop Brochure and Schedule (Feb 15 2014)
 
50 data principles for loosely coupled identity management v1 0
50 data principles for loosely coupled identity management v1 050 data principles for loosely coupled identity management v1 0
50 data principles for loosely coupled identity management v1 0
 
Sofea in a soa ecosystem v0 4
Sofea in a soa ecosystem v0 4Sofea in a soa ecosystem v0 4
Sofea in a soa ecosystem v0 4
 
Life above the service tier preso v1 0
Life above the service tier preso v1 0Life above the service tier preso v1 0
Life above the service tier preso v1 0
 
Life above the_service_tier_v1.1
Life above the_service_tier_v1.1Life above the_service_tier_v1.1
Life above the_service_tier_v1.1
 

(Deprecated) Slicing the Gordian Knot of SOA Governance

  • 1.
  • 2. “Slicing the Gordian Knot of SOA Governance” Version 0.99.1, November 2012 © Ganesh Prasad This work is licensed under Creative Commons Attribution-No Derivs 3.0 Australia http://creativecommons.org/licenses/by-nd/3.0/au/ Confessions of a back-to-basics SOA architect
  • 3. Table of Contents Synopsis.................................................................................................................................6 Part I – Why The Industry Is Doing SOA Governance All Wrong..........................................7 Timidity in Asserting the All-Encompassing Scope of SOA...............................................7 Confusion between Governance and Management..........................................................9 Obesity of the Governance Function...............................................................................10 Part II – Elements of a Common Sense Approach..............................................................11 Recognising “Services” as a Weasel Word.....................................................................11 Back to First Principles and the Notion of “Dependencies”.............................................12 Why are Dependencies so Important?............................................................................14 Case Study 1 – Take it or Leave it: Monopsony and Choice......................................15 Case Study 2 – Culture Shock....................................................................................16 Case Study 3 – The Weakest Link of a Supply Chain................................................17 Case Study 4 – Locked Out: Key Person Risk............................................................17 Case Study 5 – The Stress and Strain of an Engineer's Life......................................18 Case Study 6 – It Goes Without Saying......................................................................19 Case Study 7 – “ASSUME” Makes an ASS of U and ME...........................................20 Case Study 8 – What Makes a Good Technology Platform?......................................21 An Architectural Framework to Analyse Dependencies..................................................24 BAIT as Layers of Dependencies................................................................................24 TOGAF Artifacts as Dependency Relationships.........................................................25 Part III – A Practical Guide To Governing And Managing Dependencies............................27 Agencies and Vehicles.....................................................................................................27 Key Roles....................................................................................................................27 Key Bodies...................................................................................................................30 How Many Committees?.........................................................................................32 Functions and Processes................................................................................................35 One-time Processes....................................................................................................35 Initial Dependency Review......................................................................................37 Recurring Processes...................................................................................................37 Periodic Dependency Review ................................................................................37 Remediation Program Review................................................................................37 Remediation Program Proposal..............................................................................39 BAU Program Management....................................................................................39
  • 4. New Initiative Appraisal ..........................................................................................39 Processes in Steady State..........................................................................................43 Governance and Management Check-lists.....................................................................44 Classification of Dependencies...................................................................................44 The List of Check-Lists................................................................................................45 A Basic Governance Check-list...................................................................................45 A Basic Management Check-list..................................................................................46 Fundamental Enterprise Dependencies......................................................................47 Enterprise Dependency Check-lists........................................................................48 Business Layer Dependencies....................................................................................51 Modelling the Business with Domain-Driven Design..............................................51 Realism around Reuse............................................................................................52 Business Layer Dependency Check-lists................................................................55 Application Layer Dependencies.................................................................................58 Cohesion and Coupling...........................................................................................59 A Fun Exercise in Applying the High Cohesion Principle........................................62 Criteria for High Cohesion.......................................................................................66 Process, Product and Service.................................................................................68 Application Layer Dependency Check-lists.............................................................70 Information (Data) Layer Dependencies.....................................................................73 Data on the Outside versus Data on the Inside......................................................73 Aspects of Low Coupling.........................................................................................74 The Domain-specific Data Dictionary......................................................................75 The Internal Data Model (“Data on the Inside”)......................................................76 The Interface Data Model (“Data on the Outside”).................................................77 Message Data “on the wire”....................................................................................78 Information (Data) Layer Dependency Check-lists.................................................81 Technology Layer Dependencies................................................................................85 Implementing Products...........................................................................................86 Implementing Services............................................................................................86 Implementing the Nouns and Verbs of the Interface Data Model...........................87 The Three Core SOA Technology Components......................................................88 How Not to Use a Broker........................................................................................93 Implementing the Adverbs of an Operation............................................................94 Technology Layer Dependency Check-lists............................................................96
  • 5. Bringing about Desired Behaviour – Velvet Glove or Iron Hand?...................................99 Summary and Conclusions................................................................................................101 Contributions of this White Paper..................................................................................102 Defining Terms...........................................................................................................102 Restoring Potential....................................................................................................102 Identifying Gaps.........................................................................................................102 Simplifying Tasks.......................................................................................................103 Doubling the Pay-off..................................................................................................103 Potential Criticism of This Approach..............................................................................104 The Weight of Tradition.............................................................................................104 Making Mountains out of Molehills............................................................................104 Drawing a Long Bow.................................................................................................105 A Bridge Too Far........................................................................................................105 About the Author................................................................................................................106 Acknowledgements............................................................................................................106 Appendix A – SOA Governance and Management – An Issue of Definition.....................107 Appendix B – Lessons from Cadet Camp (or Why SOA is Like a Snakepit).....................109 Appendix C – Core Entities and Dependencies in the TOGAF 9 Model...........................111 Appendix D – Artifacts In the TOGAF 9 Model..................................................................112 Appendix E – References..................................................................................................115
  • 6. Synopsis “SOA Governance” as practised in the industry today suffers from three major problems: 1. “SOA Governance” is wrongly understood to be the governance of SOA as an IT function, rather than the application of SOA principles to the governance of all levels of the enterprise. Most of the potential benefits of SOA as an organising principle for the enterprise therefore go unrealised because of this overly narrow, technology-focused interpretation. 2. There is widespread confusion between the terms “governance” and “management” even among SOA practitioners. Many of the processes associated with “SOA Governance” are in fact routine management functions, and true governance tasks requiring direction are often neglected, resulting in costly yet preventable mistakes. 3. The processes associated with “SOA Governance” (which are more often management than governance) are needlessly heavyweight. Given the overly limited scope of “SOA Governance” as mentioned in Point 1 above, these processes soak up a disproportionate share of an organisation's resources without corresponding benefit. This white paper aims to do the following: 1. Redefine both Governance and Management in simple and unambiguous terms so that there is no confusion between the two functions and both can be performed effectively. 2. Raise the scope of SOA Governance and SOA Management beyond their current narrow technology focus so that SOA thinking can be applied at all levels to improve the agility, cost and risk profile of an organisation. 3. Recommend a comprehensive yet lightweight approach involving a few core roles and bodies, a minimal set of processes and a manageable set of checklists to enable both SOA Governance and SOA Management to be performed cost-effectively. 6
  • 7. Part I – Why The Industry Is Doing SOA Governance All Wrong Timidity in Asserting the All-Encompassing Scope of SOA 1 “SOA Governance” does not mean the governance of SOA, any more than “scientific thinking” means “thinking about science”. We know of course that “scientific thinking” means a different way of thinking about everything, i.e., adopting a rigorous, analytical, evidence-based approach to understanding every aspect of the universe without exception. Scientific thinking is about applying science to thinking, not the other way around. In exactly analogous fashion, “SOA Governance” is about applying SOA thinking to governance, not about applying governance to SOA. So how does one “think SOA”? We need to understand that SOA is an organising principle that impacts every aspect of the enterprise. It is not a set of technology products or even an approach to deploying technology components. The word “technology” refers to the implementation of business logic2, and most “SOA Governance” activities in organisations that would describe themselves as “doing SOA” relate to implementation-related activities such as setting development and environment standards, reviewing the design of SOAP-based web services and Business Process Management (BPM), controlling versions of services, establishing policies around the use of an Enterprise Service Bus (ESB), using a registry/repository to centralise information about services, etc. All of these are in fact routine management activities, with a narrow focus on technology to boot. These cannot be called SOA Governance at all! The definition of SOA we propose is “the science of analysing and managing dependencies between systems”. Systems need not be computer systems, and neither are dependencies restricted to technology. Dependencies exist at any level of business, human relations or technology, as we will show using diverse examples. The term “managing dependencies” as used above is shorthand for “eliminating needless dependencies and formalising legitimate dependencies into readily understood contracts”. 1 Service-Oriented Architecture 2 “Technology” may conjure up visions of advanced computer systems, but even a manual ledger to record transactions is technology. Indeed, it would seem like high technology to a race without paper! 7
  • 8. In a previous white paper published under the aegis of WSO2 (“Practical SOA for the Solution Architect”3), we showed how a solution design that is tightly coupled at the data layer can completely negate the benefits of expensively procured SOA technology (e.g., ESBs, registries, etc.) Such situations are unfortunately quite common because practitioners often take a technology-only approach to SOA and do not see the dependencies that exist between systems at different levels. The fault lies not with SOA itself but in our misunderstanding of SOA as being limited to technology. We need to start seeing SOA as a way of thinking about dependencies, not just within the technology realm or even at the level of data, but across the board. Unless we as an industry adopt “Dependency-Oriented Thinking”, the returns on our SOA investments will remain anaemic4. It would seem that industry analysts and prominent vendors have knowingly or otherwise misled us for over a decade with the conceptual model illustrated below, and this view has stood in the way of our ability to realise the full benefit of SOA. Every expert unfailingly issues the standard disclaimer that SOA is not about technology, but the opposite message gets dog-whistled through the emphasis on products to manage web services, and ultimately prevails. In a later section, we hazard an explanation for this. Corporate Governance Corporate Resources IT Governance IT Resources SOA Governance SOA Resources Fig. 1 – The limited (and limiting) view of SOA Governance as a subset of IT Governance (itself a subset of Corporate Governance) – a view endorsed by industry thought leaders such as the Burton Group5 and IBM6. 3 Downloadable from http://bit.ly/stbfiA 4 In retrospect, DOT may have been a better term than SOA. 5 The Burton Group's Research Director speaks on SOA and Governance: http://bit.ly/HFdF6t 6 IBM's definition of SOA Governance: http://ibm.co/qm46 8
  • 9. To repeat, SOA is not a subset of IT. The benefits of SOA thinking when applied more broadly are: 1. An improvement in business agility because of minimal dependencies between systems and consequently minimal “friction” that can impede change 2. Sustainably lower operating costs for the same reasons 3. A significant reduction in operational risk because of better-understood dependencies and fewer surprises Clearly, we have lost out on some major benefits by defining SOA as narrowly as we have. Confusion between Governance and Management “Governance” is an over-used (and abused) term. One of the unfortunate side-effects of the collapse of corporations like Enron and WorldCom in 2001-2002 was the sudden popularity of the term “governance” in popular discourse. The word “governance” has acquired such a cachet today that it often tends to be used just for effect, even when what is meant is just plain old “management”. So let's set these terms apart right away. Governance is ensuring that the right things are done. Management is ensuring that things are done right. In spirit, governance is about the ends, or the “What”; management is about the means, or the “How”. This is a fundamental distinction that is key to correctly implementing what we recommend in this white paper, hence it is worth spending some effort to understand this thoroughly. An analogy with the functions of a company's board of directors and its executive management will make the distinction between corporate governance and corporate management more concrete. A decision on whether the company should enter a new market is a decision for the board, because it pertains to the fundamentals of what the company wants to be and whether or not the move is in the best interests of its shareholders. In other words, this is about doing the right thing (i.e., governance). Once the decision is taken to enter a new market, executive management is responsible for retooling the resources of the company to enable it to compete effectively in that 9
  • 10. market. This is about doing things right (i.e., management). Governance and management decisions apply at lower levels of the organisation as well. Projects often have steering committees as well as working groups. Steering committees are comprised of key stakeholders and they tend to make governance (i.e., “what”) decisions – objectives, scope, success criteria, etc. They also monitor adherence to these parameters. Working groups are teams of hands-on people assigned to the project. They make the day-to-day management (i.e., “how”) decisions that will enable them to solve routine problems and achieve the objectives that have been set by the steering committee. The steering committee is only concerned with the ends and not the means. The working group is responsible for the means. In general, one can tell whether a given decision is about governance or about management by thinking about how those decisions could be judged in hindsight. If the verdict is likely to be “right” or “wrong”, then it's a governance decision. If the assessment is likely to be one of a spectrum (e.g., “excellent”, “good”, “fair” or “poor”), then it's a management decision. Obesity of the Governance Function Organisations that have embraced the requirement for “SOA Governance” generally have committees, processes and tools to bring discipline to the way services are designed, built, deployed and managed. After all, this is how the governance of SOA is commonly interpreted. Some organisations have an “Integration Centre of Competence” that is tasked with defining standards and controlling the introduction of new services into the ecosystem. While these are sensible measures, they impose a rather high overhead especially given their limited technology-only focus. A centralised approving authority is an overworked bottleneck, and so agility and operational cost are the first casualties. Many organisations consequently see very modest improvements in their operational cost and efficiency as a result of moving to SOA. The economies and dis-economies largely neutralise each other, and the heavyweight governance processes shoulder a large portion of the blame. If projects in your organisation are constantly trying to find short-cuts around a centralised authority and a large part of this latter group's efforts are around reigning in such rogue projects, it could be a sign that the organisation is chafing under heavyweight governance. 10
  • 11. Part II – Elements of a Common Sense Approach Recognising “Services” as a Weasel Word No matter how often we parrot the mantra “SOA is not about technology”, we end up with a technology-only view of SOA while we're looking elsewhere. How does this happen? Our opinion is that there is a slippery slope that we embark on once we agree to talk about “services”. From “services” to “web services”, and from web services to a technology-only view of SOA and SOA Governance is then just a natural progression, as illustrated below: “SOA is not about technology” “Service-Oriented Architecture is a view of the organisation as a collection of reusable services” “We should expose all business functionality through Web Services” “It's good practice to proxy all Web Services through an ESB and use a registry/repository to manage them, especially as their numbers start to grow.” “We need some governance around the use of all this technology” “SOA assets are owned by IT, so SOA Governance is an IT function” Fig. 2 – The slippery slope that leads to the view of SOA as a part of IT There is a place for the concept of “services” in what we lightly call “Service-Oriented” Architecture. But the wide-angle lens of dependencies shows us that there's much more to SOA. We need to derive our view of “services” much more systematically and rigorously than we have been used to doing. So let's look at this process more closely. 11
  • 12. Back to First Principles and the Notion of “Dependencies” The four most important principles of SOA are dependencies, dependencies, dependencies and dependencies7. We're not being entirely flippant in saying this, because there are four distinct “layers” in an organisation where dependencies need to be managed. These layers, as we will explain shortly with the help of a formal framework, are Business, Applications, Information (Data) and Technology 8. SOA Principles (Eliminate needless dependencies between systems, formalise legitimate dependencies into well-understood “contracts”) Manage dependencies Business Layer Manage dependencies Manage dependencies Application Layer Manage dependencies Manage dependencies Information (Data) Layer Manage dependencies Manage dependencies Technology Layer Fig. 3 – The True Scope and Concerns of SOA - More than just a few Web Services managed by a section of IT! We believe that for an organisation to be effective in achieving its goals, an acceptance of this dependency-based view of SOA is essential. Any discussion of SOA Governance and SOA Management has to start from this basis. Since SOA is all about the management of dependencies, SOA Governance should be seen as deciding what dependencies are legitimate and SOA Management as deciding how to manage dependencies. Both these aspects are important, so even though SOA 7 To paraphrase the 3 rules of real estate: “Location, location, location”. 8 This is commonly referred to as the “BAIT Model”. 12
  • 13. Governance seems to have all the mind-share, this white paper will talk about SOA Management9 with equal emphasis. Ironically, traditional approaches to “SOA Governance” are usually about SOA Management (i.e., the “How”). So here are our simple and readily-understandable definitions: • SOA Governance is determining what dependencies are legitimate at every layer of the organisation and identifying what existing dependencies fall outside this set. • SOA Management deals with how to remediate illegitimate dependencies at every layer of the organisation, how to formally document and communicate legitimate dependencies and how to prevent recurring violations. We will break these down into specific tasks in Part III, but for now, the following high-level illustration will summarise how we arrive at these definitions. SOA (The science of analysing and managing dependencies) Governance SOA Governance (What are the right What dependencies are legitimate? things to do?) What existing dependencies fall outside this set? SOA Management Management How do we remediate illegitimate dependencies? (How do we do How do we formalise legitimate dependencies? things right?) How do we prevent recurring violations? Fig. 4 – SOA Governance and SOA Management at a glance This is the core philosophy behind our approach to SOA Governance and SOA Management in this white paper. 9 In the literature, we often come across the term Service Management. This term derives from the view of SOA as a subset of IT rather than as a set of organising principles for the entire enterprise. Our term “SOA Management” is more comprehensive. 13
  • 14. Why are Dependencies so Important? We believe that a deep understanding of dependencies makes all the difference between a successful organisation and one that is less so. Sun Tzu said 10 many centuries ago, “Know thyself, know thy enemy – a thousand battles, a thousand victories.” By “knowing” oneself and one's enemy, he was really talking about dependencies, because strengths as well as weaknesses are dependencies. What is it that makes you and the other guy tick? Fast cavalry? Long range artillery? What are the things that can trip you up? Lack of supplies (logistics)? Troops weakened by dysentery? What buttons can be pushed to make you or the other guy behave in a predictable way? Siege of a prestigious fort? Abduction of a corps commander's son? All of these are dependencies. All of these relate to “knowing thyself” or “knowing thy enemy”. In that classic marketing strategy book of the '80s, “Marketing Warfare”, authors Al Ries and Jack Trout offer this nugget of wisdom to their clients who are looking for the most effective way to compete against their rivals - “Don't attack a weakness that is a weakness. Attack the weakness inherent in a strength.” Their idea is that weaknesses that are purely weaknesses can be fixed, but weaknesses that are an inherent part of a competitor's strength cannot be fixed without compromising that strength. As an example, they suggest that any company trying to compete against a giant like Campbell Soup should attack not the price of the soup (since a larger company can easily drop its prices long enough to drive a smaller competitor out of business), but something peripheral like the metal cans in which the giant corporation packages its products. The enormous investment in metal can packaging is a weakness inherent in the strength of Campbell Soup's scale of operations, and this investment cannot easily be thrown away. A competitor positioning themselves as using either environmentally friendlier packaging or packaging that is less suspect from a health perspective would have a better chance than one competing on price. In other words, Trout and Ries suggest attacking dependencies that a competitor cannot easily rid themselves of. That's how important dependencies are to the life-and-death struggle of companies in the marketplace. To further reinforce the importance of dependencies to an organisation, we provide a few real-life case studies11 which clearly demonstrate two things – one, that dependencies are 10 The Art of War 11 The examples are authentic but have been anonymised, not so much to protect the guilty as to protect the authors and their associates from accusations of breach of confidentiality. 14
  • 15. not restricted to information technology, and two, that a lack of understanding of dependencies can cost an organisation dearly. Case Study 1 – Take it or Leave it: Monopsony 12 and Choice A medium-sized commercial organisation in a developing country was experiencing strong growth, and began hiring more staff to meet the demands of its expanding business. One of its hires was a seasoned manager from another industry, and his eyes immediately picked up an aspect of his new employer's business that its owners had been oblivious to. He was shocked to discover that although the volume of growth was strong, profits were not growing at all. On the contrary, many of the projects were actually losing money. It took him very little time to home in on the problem. Almost all of the company's business was coming from government contracts, which had been its traditional mainstay. Government contracts had two features that were both very unfavourable to suppliers. One was the policy of always awarding contracts to the lowest bidder, which squeezed margins to the bone. The second was the one-sided payment policy, under which the bulk of the payment for a project would only be released on completion of the work, with a significant amount withheld for a 12 month warranty period, leaving the supplier with a huge financing gap while the project was being executed and potential liability for months after completion. The newly hired manager immediately began to implement a strategy to diversify the company's client base. As he moved to other markets, he was able to trade in on the company's reputation and credibility to command higher premiums from clients in the private sector, generally bypassing the tender process altogether. He also concentrated on short delivery cycles and 100% payment on completion. In four years, 30% of the company's business was from the non-government segment. More dramatically, 85% of its profits came from this segment too. Moral of the story: The company had a legitimate dependency on its reputation for quality and reliability because it was impossible to win contracts without a good track record. However, it had developed an unnecessary dependency on government contracts through sheer history or habit. The new manager, coming from outside the company, had no such mental blinkers and could see this dependency that the owners weren't even aware of. He then leveraged the company's legitimate dependency as an explicit marketing tool to win 12 http://en.wikipedia.org/wiki/Monopsony 15
  • 16. new business, simultaneously reducing its unnecessary dependency on government contracts with their unfavourable commercial terms. The results were spectacular. Case Study 2 – Culture Shock When a new General Manager took over at a company specialising in turnkey engineering projects, he was immediately struck by the lack of standardised process in his new organisation. No two projects, however similar, followed the same process. Project execution was ad hoc and freewheeling, with lower level managers and engineers making informal decisions in oral negotiations with their counterparts in their client and supplier organisations. True, the business seemed to be running smoothly, but the lack of process seemed to be a ticking time-bomb. The GM came down hard on this laidback culture. He began to insist on standardised processes and timetables for all projects. Instead of informal agreements between operating staff, the company began to send out formal communications of intent, spelling out what was going to be done and when, and what the other party was expected to do. Suppliers were kept on a tight leash and offered little room to negotiate on terms. Within months, dramatic results began to be seen. Project schedules began to slip. Clients began to complain loudly. Suppliers were less accommodating when plans had to change. Things became bad so rapidly that the manager was fired and replaced with one of his senior direct reports who was familiar with the old way of doing things. It took many more months to bring the situation back to normal. It turned out that in the turnkey engineering business, every client and every site is different and requires a different approach. Flexibility is key, because there are lots of unknowns and unexpected developments during the course of a project's execution. Two important aspects of managing these changes are sufficient autonomy for local staff and maintenance of good working relationships between key people, on the client side as well as on the supplier side. A one-size-fits-all approach to process does not work. Neither does a rigid and formal approach to managing external relationships. Moral of the story: The company had a unavoidable dependency on local and individual conditions for every project. The impacts of this dependency could only be mitigated through flexibility, and this was achieved through a conscious culture of autonomy to staff “at the coalface”, mutual respect and adjustment between business partners, and regular 16
  • 17. and informal communication between key people. The new general manager wrongly saw the informal agreements between people as a failure to formalise dependencies into contracts, and he then tried to replace these with more standardised processes and rigid contractual agreements, but these measures removed the cushion that the company had developed to shield it from its fundamental dependency on local conditions. This case study provides a useful contrast with the previous one because what looks like an unnecessary dependency may not in fact be so. It takes astute observation to understand dependencies and to leverage or manage the legitimate ones while eliminating those that are truly unnecessary. Case Study 3 – The Weakest Link of a Supply Chain An Australian construction company based in Sydney fabricated aluminium and timber windows for the building industry. The company was careful to hedge against currency fluctuations as well as the price of aluminium. But no such arrangements were in place for timber which was sourced from a supplier based in the neighbouring state of Victoria. What the construction company didn't realise was that Chile and Malaysia are the two major sources of timber in the world, and that the Victorian supplier imported timber exclusively from Chile. In February 2010, a major earthquake in Chile disrupted the supply of timber, resulting in worldwide timber shortages and a steep rise in prices. Since the supplier could no longer source timber in a timely or cost-effective manner, the Windows fabricator in turn could not deliver windows to its builders, resulting in delays, losses and unhappy customers. Moral of the story: If the construction company had looked beyond its immediate Australian supplier and studied the entire supply chain for its products, it would have realised the critical dependency it had on Chilean timber. Once formally recognised, it could then have done a number of things to mitigate that dependency, such as hedge against the price of timber, force its supplier to hold sufficient inventory, or diversify its supplies across Chile and Malaysia. Since it neglected to understand and mitigate the effects of this dependency, it paid a heavy price. Case Study 4 – Locked Out: Key Person Risk A small value-added, niche-market IT distributor had a sales team of 6 to 7 people headed 17
  • 18. by a national manager. The company had recently changed owners, and the new owner had not had enough time to familiarise himself with the company's customers. The national manager was suddenly poached by a competitor. He not only took his entire sales team with him, but also destroyed all records at his old company before leaving, wiping computers clean and removing all business documents. The owner was faced with the closure of his new business, since he had no records of any of his customers. Although his national manager's betrayal could have been criminally prosecuted, he did not have the time or the resources to pursue litigation. This story did however end happily. The owner put together a fresh team under a new and dynamic national manager, and they painstakingly won back all their customers over the next five years. The predatory competitor went bust. Moral of the story: The business had a critical dependency on its customer relationships, physically manifested as its database of customers and records of interactions with them. This dependency was not recognised and formalised as a contract, e.g., a customer database that was regularly backed up and protected against loss or damage. The owner's failure to formalise and manage this dependency cost him heavily. Case Study 5 – The Stress and Strain of an Engineer's Life A company supplying and installing power generators in buildings encountered a problem that had not been anticipated or designed for. A generator installed in the basement of a twelve storey building was hooked up to an exhaust pipe to transport hot fumes to the outside through a shaft in the core of the building. The metal exhaust pipe ran horizontally for 6 metres along the ceiling of the basement and then turned upwards in a 90-degree L-joint to run a further 40 metres to the top of the building through the central shaft. This was the first time the company's engineers had deployed a generator in this configuration, since all their previous engagements involved ground-level generators with short exhaust pipes. The engineers noticed a problem after installation. Whenever the generator was in operation and began to push out hot exhaust fumes, the 40 metre column would expand with the heat, pushing down on the L-joint and forcing the horizontal section to arch downwards almost half a metre. The pipe would return to its position once the generator 18
  • 19. stopped and the pipe cooled, but this repeated movement due to expansion and contraction was a certain recipe for failure of the pipe due to fatigue. Ultimately, the engineers came up with a solution to replace the rigid L-joint with a much looser (though still airtight) bell-shaped container into which both the horizontal and vertical sections of the pipe could extend. A system of rollers allowed both sections of the pipe to expand smoothly into the “bell” and contract equally smoothly out of it, with neither section's movement impacting the other. Although the company had to spend extra time and money to implement this solution (which they could have built into the initial design had they anticipated the problem), they managed to avert a future, potentially more expensive accident. Moral of the story: The two sections of the generator's exhaust pipe had an unnecessary dependency on each other's thermal expansion due to the rigid L-joint between them. Once this dependency was eliminated through a more loosely-coupled expansion mechanism, the problem disappeared. Case Study 6 – It Goes Without Saying At a large bank in the Middle East, a sincere and hard-working British expat Business Analyst was put in charge of a major branch automation project. The bank's tellers were using mainframe “green screen” terminals, and these were overdue for replacement with a more modern interface. The BA worked hard for many months, interviewing bank staff and management, gathering requirements, creating questionnaires for vendors, issuing RFIs and RFPs, conducting evaluations and negotiations, etc., and finally selected a vendor that seemed superior in every respect. Although he made presentations from time to time to management to keep them informed of his progress, he was doing everything virtually single-handed. Finally, after about a year of hard work, and just before the contract was to be awarded to the successful vendor, the BA organised a demonstration of the branch automation system for the bank's senior management and branch heads. Partway through the impressive demonstration, one of the managers asked, “Does the system support Arabic?” The BA was stunned. He turned to the vendor representative, who looked embarrassed. 19
  • 20. No, the system did not support Arabic. The commotion and head-shaking that greeted this admission made it abundantly clear that the deal was dead. The poor BA was later spotted sitting at his desk with his head in his hands. The search for a branch automation system had to resume almost from scratch, at a huge cost in time and money. Moral of the story: Although the BA had taken great pains to gather detailed requirements from a wide cross-section of bank staff, Arabic language support was considered to be such an obvious requirement that no one had actually spelt it out. Being a recently-arrived expat, the BA didn't consider it either. And so the most obvious and important requirement for the system went unmet and ruined a year's worth of work. In other words, a legitimate dependency was not formalised in the form of a contract (i.e., the requirements document). Its absence was therefore not noticed until too late and resulted in a nasty surprise. Case Study 7 – “ASSUME” Makes an ASS of U and ME A multinational systems integrator was engaged to deliver a comprehensive Corporate Internet Banking solution to a major Australian bank. Most of the functionality of the application would leverage the bank's mainframe-based product systems, and these would be exposed as services through middleware. What remained to be sourced was a suitable web-friendly front-end product with the required rich user interface. The SI evaluated a few packaged products and selected one for its client. The overall project was a multi-million dollar undertaking with about a hundred people working on the program. There were business analysts, project managers and a large number of technical staff working on the mainframe and middleware, along with a smaller team of front-end designers and developers. It was known that the selected product had been originally designed to work against data in a relational database, but the product vendor's architects had assured the SI that it was relatively easy to graft the front-end onto the middleware services that were being built. A few months into the project, the SI's front-end designers got their first look at the source code of the front-end product, and one of them noticed references to a “Row ID” sprinkled throughout the code. The vendor's front-end developers innocently explained that this referred to the row number in the database. The SI team was aghast. “But there's no database! You know this has to work against services talking to a mainframe!” 20
  • 21. A meeting of the SI's senior architects and project managers was hastily called. The vendor's senior architects were summoned from overseas. Not having a detailed knowledge of the code, they had not realised that the relational database assumption would be so deeply rooted in their product's design, and they estimated that it would take a further 3 months to fix. All knew that a 3 month delay was unacceptable to the client. Ultimately, the vendor relationship was terminated, the SI's project manager was fired, and the SI barely salvaged the contract after demonstrating a custom-built working prototype to the client. The fiasco threatened to derail a prestigious project and came perilously close to rupturing a highly visible, multi-million dollar relationship between a multinational systems integrator and one of its biggest Australian clients. Moral of the story: The front-end application had hard-coded its dependency on a relational database into every element of its user interface. The vendors' architects had not adequately documented how this dependency should have been isolated, and those who documented the implementation had not adequately recorded the extent of the dependency. Either of these “contracts” would have alerted people to the extent of the dependency. No wonder even the vendors' architects were blind-sided. Case Study 8 – What Makes a Good Technology Platform? To round out this section, let's end with a generic example from the realm of technology – the notion of a “platform”. A platform is a technology environment that provides all the necessary run-time support for applications. These could be virtual machines, interpreters, dynamically linkable libraries, etc. Let's look at five platforms that have been popular over the last decade and a half. i. The Windows operating system is a platform for native Windows applications. While its ubiquity was attractive to Independent Software Vendors (ISVs), there were a few problems with this platform. One was the shift from 16-bit to 32-bit around 1995, which rendered quite a few third-party applications unable to run on the newer version. Another persistent problem was known as “DLL hell”, because of the dependencies of applications on versions of dynamically linked libraries. An upgrade to an underlying system library could and often did break dozens of third-party apps. A third issue was that Microsoft exploited “undocumented Windows 21
  • 22. APIs” in its own applications that rival app vendors could not use, and so it was not a level playing field. ii. Linux (like any Unix variant) has the notion of symbolic links that it uses to advantage to manage its dynamically linked libraries (called “shared object files”). Shared object files have major version numbers and minor version numbers. Changes in minor version numbers represent changes that do not break the API. Changes in major version numbers represent changes that do break the API. Symbolic links are used to hide changes in minor version numbers but not in major version numbers, so applications that declare their dependencies in terms of the symbolic links will be shielded from irrelevant changes but rightly impacted when the API changes. Also, old and new versions of files can simultaneously exist to support old and new versions of applications. Shared Object Symbolic link Description version libxxx.so.2.1 libxxx.so.2 Original version that applications are dependent on. libxxx.so.2.2 libxxx.so.2 Change in shared object's minor version number is not reflected in the symbolic link, so dependent applications are (correctly) not impacted. Both shared objects can exist simultaneously, but only one is “active”. libxxx.so.3.1 libxxx.so.3 Change in shared object's major version number is reflected in the symbolic link, so dependent applications are impacted, as they should be. Both symbolic links can exist simultaneously, supporting old and new versions of applications. Table 1 – Linux and the use of symbolic links to abstract insignificant version changes iii. The Java Virtual Machine is a platform that runs Java bytecode. The Java language has undergone numerous version revisions, but the JVM has hardly changed in 15 years. Compiled bytecode from 15 years ago runs on the JVM just as well as compiled bytecode from the latest version of Java. iv. The browser is a platform too. In the mid- to late-nineties, developers were discouraged from writing applications that relied on Javascript running on the browser, because there were at least two broad variants of browser platforms – 22
  • 23. Netscape's and Microsoft's – and even the versions of each browser implemented different versions of the HTML and JavaScript standards (if there were standards at all). It is only since very recently that the browser can be relied upon as a platform that will run applications in a predictable and consistent way, independent of browser vendor. v. Apple's products have a reputation for ease of use, but they are also highly 'closed'. Standard input/output mechanisms like USB ports are not supported, and very little can be achieved by a device without integration to iTunes, iCloud or both. For users who don't mind the restrictions that the Apple platform places, everything “just works”. The competing Android platform is more open, but also a bit more chaotic in terms of multiple versions and multiple implementations by device vendors. Moral of the story: For a platform to be considered a good one, applications must, above all, be able to rely on consistent and stable behaviour. Irrelevant changes must not be allowed to impact applications. There must be considerable latitude in the definition of what constitutes a change, so that the bulk of changes within the platform can be “absorbed” without exposing applications to them. The five examples above showed various approaches to the construction of a stable platform, with varying degrees of success. We can see that the notion of “dependencies” underlies all of them. Summary We have now seen several dramatic examples of how unappreciated dependencies caused, or threatened to cause, problems of reduced agility, increased cost and increased risk to organisations. Not all of these have to do with technology, much less with “SOA technology”, and the choice of examples has been deliberate in that regard. Yet the failure to understand the potential impact of dependencies is the common thread in all these stories. A basic level of dependency-oriented analysis would have highlighted the costs and risks in each of these cases, saving the concerned organisations significant amounts of time, money and risk. Having therefore established the crucial importance of dependencies to an organisation's viability and success, let us see how we can approach the analysis of dependencies in a systematic way. This is what SOA Governance and SOA Management are really about. 23
  • 24. An Architectural Framework to Analyse Dependencies We need a framework to analyse dependencies before we determine how to govern and manage them. Rather than reinvent the wheel, let's look at some existing and well-understood models to see if they suit our requirements. BAIT as Layers of Dependencies The BAIT model is a popular architectural framework that decomposes an organisation into four layers – Business, Applications, Information (Data) and Technology. BAIT has the right idea about layers, but it is traditionally focused on identifying the entities that exist at each layer and the relationships between those entities. There is no strong emphasis on the aspect of dependencies, even when dealing with relationships. When we adopt the BAIT model to aid us in SOA Governance and Management, we have to adapt it to focus on dependencies, as follows: Traditional concerns of The BAIT Model applied to the BAIT Model SOA concerns Business Layer Business-Level Business Entities (Intent) Dependencies Application Layer Domain Dependencies Applications and Systems (Internal Cohesion) (High Cohesion Principle) Information (Data) Layer Interface Dependencies Domain Data Models (Interfaces/Integration) (Low Coupling Principle) Technology Standards, Technology Layer Technology Dependencies Software and Hardware (Implementation) Fig. 5 – Adapting the BAIT Model to address SOA concerns In other words, we prefer to think of BAIT's four layers as representing four “I”s, referring to the (1) Intent, (2) Internal Cohesion (grouping of business functions into highly cohesive “applications”), (3) Interfaces/Integration between logical functions with low coupling, and (4) physical Implementation of all the components of the logical business organisation. The emphasis on cohesion and coupling is particularly relevant, as we will see in later sections. 24
  • 25. TOGAF13 Artifacts as Dependency Relationships The TOGAF framework of Enterprise Architecture builds on the BAIT model and describes generic entities in each layer along with their inter-relationships. This is extremely valuable to us when embarking on SOA Governance and SOA Management because the hard work of understanding the components of an organisation and how they fit together has already been done. However, just like with BAIT itself, TOGAF as it stands is not a perfect framework that can be used without adaptation to our treatment of SOA Governance and SOA Management. It does not inherently support the dependency-oriented organisational transformation that we would like to model and plan for. SOA is a single isolated chapter in The Open Group's 780-page TOGAF 9 book, which betrays a traditionalist view of SOA as a subset of IT rather than as the science of analysing and managing dependencies across the board. However, nothing prevents us from using a dependency lens to view the detailed set of entity relationships identified by TOGAF and derive a more applicable model for our requirements. We'll show how to do that in Part III of this document. TOGAF defines some core concepts at the four layers and lists out some important and standardised documents (“viewpoints” in the TOGAF terminology) that provide useful information about these concepts and their interactions. TOGAF uses the term “catalog” to refer to any list. Similarly, it uses the term “matrix” for tables that show the relationship between exactly two concepts, and “diagram” for any depiction of how more than two concepts are related. (a) (b) (c) Fig. 6 – (a) a TOGAF “catalog”, or list of entities; (b) a TOGAF “matrix”, or table of two-entity relationships; (c) a TOGAF “diagram”, or a depiction of multi-entity relationships. If we interpret “relationships” as “dependencies”, matrices become two-entity dependencies and diagrams become multi-entity dependencies. The set of TOGAF-defined matrices and diagrams at each of the four BAIT layers then gives us a ready-made check-list of the dependencies we need to be mindful of within those layers as well as between layers. 13 The Open Group Architecture Framework: http://www.opengroup.org/togaf/ 25
  • 26. We consciously adapt TOGAF's set of artifacts to align them with a SOA view of the enterprise because TOGAF is not inherently dependency-oriented. In other words, the entity dependencies we study will correspond to our SOA version of the BAIT Model rather than the traditional one. SOA Version of BAIT Model TOGAF Artifacts Business Layer Business-Level (Intent) Dependencies Application Layer Domain Dependencies Core entities, (Internal Cohesion) (High Cohesion Principle) Two-entity dependencies and Information (Data) Layer Interface Dependencies Multi-entity (Interfaces/Integration) (Low Coupling Principle) dependencies Technology Layer Technology Dependencies (Implementation) Fig. 7 – TOGAF artifacts classified by BAIT level We can now begin to see why the traditional view of “SOA Governance” is so limited. SOA Governance SOA Management (“What dependencies are (“How do we manage legitimate?”) dependencies?”) Business Traditional “SOA Governance” concerns Application Development and environment Information (Data) standards, reuse metrics, Web Service version management, repository structures to store Technology WSDL, schema and policy files, etc. Fig. 8 – Traditional “SOA Governance” concerns are more correctly SOA Management, and a subset at that. Little effort is focused on the governance question, “What dependencies are legitimate?” Clearly, our entire approach needs nothing less than an overhaul. It's time to look at how SOA Governance and SOA Management should in fact be done. 26
  • 27. Part III – A Practical Guide To Governing And Managing Dependencies In the previous two parts of this white paper, we critiqued the industry's approach to SOA Governance and suggested how we could do it better by considering the elements of a more rational approach based on the analysis of dependencies. In this part of the white paper, we will show how to put these elements together into a lightweight method. We'll recommend some key roles and bodies that will be responsible for the functions of SOA Governance and SOA Management, the minimal set of processes they need to carry out, and a manageable set of check-lists that they should use for these purposes. Agencies and Vehicles Key Roles We believe that the primary agency to apply dependency-oriented thinking at every level of an organisation is the Enterprise Architecture function. In organisations with an Enterprise Architecture group, there are usually architects with specialised skills who are tasked with defining the business architecture, application architecture, data architecture and technology architecture, so the BAIT Model is already an operational tool. These specialised architects are the people who need to spearhead a new way of approaching SOA Governance and SOA Management. The role of Enterprise Architecture under this model is to analyse the four layers of the organisation specifically from a dependency focus, first to determine the “what” (the legitimate dependencies that should exist at each level as well as the unnecessary ones that do currently exist) and then to guide the “how” (the programs of work required to align the organisation to the set of legitimate dependencies that were identified). One of the suggestions we have for the Enterprise Architecture function to assist them in transitioning to this world-view is to include professionals from certain other disciplines, even if on a part-time basis. Project Managers, Risk Managers and Contract Lawyers are all used to looking for dependencies in their own areas, and they can inject some of the fresh blood that the traditional Enterprise Architecture group needs to make them effective in a SOA sense. 27
  • 28. In organisations with a reasonably effective architecture function, here's how business intent is transformed into working systems today: Fig. 9 – Key roles and their traditional concerns Here's the subtle but important change we would like to see: Fig. 10 – Key roles and their recommended concerns It's not as if architects don't already think about dependencies. It's just that dependencies need to be promoted to be a primary, top-of-mind item in all discussions and decisions, 28
  • 29. especially where the business and IT are involved. Every individual who is part of the decision chain from business objective to implementation, no matter what their level, needs to be sensitised to the importance of dependencies. That's what it means to inculcate “SOA Thinking” within an organisation, and architects need to don the mantle of evangelists to spread this philosophy. The perspectives of project managers, risk managers and contract lawyers could be very useful additions to their skill set, since these give them a “fresh pair of eyes” with which to recognise the diverse set of dependencies impacting the enterprise. SOA Governance and SOA Management become far easier to execute when the organisational culture understands the importance of dependencies. 29
  • 30. Key Bodies While the Enterprise Architecture function is key to our recommended approach, SOA Governance and SOA Management require the active participation of many different roles and functions to be effective. The business and IT are crucial stakeholders, as we saw. It is essential to involve multiple roles in the SOA Governance and SOA Management functions, and this calls for new organisational structures. We are all too aware of the reputation of committees as inefficient bureaucracies 14, but the multi-disciplinary nature of the tasks at hand, together with the fact that these are ongoing responsibilities, dictate the need for one or more standing bodies rather than ad hoc teams. Broadly, two groups of people are required to carry out the required functions: • a “Dependency Governance Committee” • a “Dependency Management Committee” Now this is a functional classification, and these two logical groups could be realised as several physical committees, e.g., one or more at each level of the BAIT model plus one at the Enterprise level. Some committees could also take responsibility for more than one BAIT layer, as long as they are clear which layer they are dealing with at any given time. The decision on how many physical committees to have is entirely up to the organisation in question, but we would strongly recommend against combining governance and management functions within any single committee. That would defeat one of our primary goals, which is to perform both functions effectively and without confusion. Additionally, although it's the committees' functions that are important and not their names, we would recommend that the term “dependency” be explicitly used in their names in order to inculcate a dependency focus within every person in the organisation. We've lost a decade's worth of benefits thanks to poor naming (“Service-Oriented” as opposed to “Dependency-Oriented”), so let's call a spade a spade this time, however corny it may sound. A committee should either be a “Dependency Governance Committee” or a “Dependency Management Committee”, with prefixes as appropriate to the BAIT level and/or business unit. 14 Two of our personal favourites are “A camel is a horse designed by a committee” and “A committee is a group of people who individually can do nothing, but who get together to decide that nothing can be done.” 30
  • 31. In a nutshell, • The Dependency Governance Committee is responsible for defining which of the dependencies at the level they are responsible for are legitimate and which are not. They are also responsible for prioritising the programs of work to remediate dependencies. • The Dependency Management Committee is responsible for costing, planning and initiating the programs of work that will bring and keep the organisation in line with the model validated by the Dependency Governance Committee and according to the priority decided by that committee. The Dependency Governance Committee should have a fair representation of Enterprise Architects, with specialists in each of the BAIT layers. At least some of the members of the Dependency Governance Committee should have a background in one of the non-IT professions listed earlier 15, because these professionals are trained to look for dependencies. Depending on the layer of the BAIT model that the committee is responsible for, there should be representation from senior executives, business managers, functional heads and technology specialists. This may mean separate committees, or a single committee with some fixed members and a number of others who attend only the sessions that are relevant to them. Importantly, the Dependency Governance Committee must have authority and access to funding to be able to approve programs of work, otherwise the organisation will be unable to remediate the dependencies identified as invalid. One of the reasons why we expended so much ink in Part II to present 8 separate case studies was to hammer home the idea that attention to dependencies saves real money. Business unit heads should therefore not be churlish about funding dependency remediation tasks, and their representation on Governance committees serves more than just a decorative function! The Dependency Management Committee should comprise people skilled in costing and planning, typically project managers and functional heads. They may need representation from Enterprise Architecture to keep them focused on the dependency aspect while they attend to their more familiar roles of costing and planning programs of work. Again, there will be a need to have different people engaged for different layers of the BAIT model, so there could be considerable variety in the structure and/or composition of the committee(s). 15 Project managers, risk managers and contract lawyers. 31
  • 32. How Many Committees? The dependency governance and management functions can be performed by just two committees in a small organisation, since they can do justice to the complexity of the entire organisation themselves. As organisations become larger, the committees will need to become more specialised, as the following examples illustrate. Dependency Governance Dependency Management Committee Committee Fig. 11 – Committees for a small organisation Enterprise Dependency Enterprise Dependency Governance Committee Management Committee Business Dependency Business Dependency Governance Committee Management Committee Application & Information Application & Information Dependency Governance Dependency Management Committee Committee Technology Dependency Technology Dependency Governance Committee Management Committee Fig. 12 – Committees for a medium-sized organisation Enterprise Dependency Enterprise Dependency Governance Committee Management Committee Business Dependency Business Dependency Governance Committee Management Committee Application Dependency Application Dependency Governance Committee Management Committee Information Dependency Information Dependency Governance Committee Management Committee Technology Dependency Technology Dependency Business Governance Committee Management Committee Units / Geographies Fig. 13 – Committees for a large organisation 32
  • 33. If this set of committees is beginning to look too bloated, our audacious claim is that these are the only governance and management structures that an organisation will ever need, so this could in fact represent a drastic simplification to the plethora of management bodies that currently exist in a given organisation. Let's look at a few special cases to support this claim. Security dependencies: Security is an important aspect of business operations, and information security traditionally tends to be treated as an IT concern. However, security concerns are relevant at different levels of the BAIT model. Perimeter security, for example, may be relevant only at the level of technology, since it is completely opaque to higher layers. Data Leakage Protection applies at all levels from the business layer down, since it is impacted by business processes, application design, data interchange as well as technology-related aspects like USB device control and disk encryption. PCI-DSS 16 compliance requirements may or may not impact business processes but will certainly impact application design, data storage and encryption/tokenisation technology. The dependency governance and management committees at these various levels will address and manage the security concerns listed above, as well as any others that arise from time to time. Non-technology domains that are not line-of-business, such as Human Resources, Accounting, Finance or Legal, also fit into this dependency-oriented model. Human Resources dependencies: Clearly, an organisation that employs many more contractors than permanent employees has a very different dependency profile than one with the opposite composition. The contractor-based organisation can downsize more readily in an economic downturn and re-hire when required, but it has a more tenuous hold on IP due to the less “sticky” nature of employee knowledge. The contractor-based organisation will have to invest more heavily into knowledge-management systems to mitigate against their risk of losing critical business and operations knowledge. The organisation with a higher proportion of permanent employees will have to be more cautious about hiring in an economic upturn because of their larger financial exposure arising from retrenchment benefits they may have to pay out when downsizing. A business dependency governance committee and business dependency management committee focused on the HR business unit will be able to formalise these concerns and evolve mitigating strategies. 16 Payment Card Industry Data Security Standard 33
  • 34. Finance dependencies: An organisation that is funded mainly by debt has a different dependency profile to one funded mainly by equity. A debtor and a creditor represent dependencies of different kinds, and every financial instrument that is issued or purchased imposes a different flavour of dependency on the enterprise. Financial controllers make decisions based on such dependencies all the time, whether or not they use the word “dependency”. Regardless, the business dependency governance and management committees for the Finance business unit will standardise these functions in a consistent corporate style. In short, these varied examples support our claim that SOA, being all about dependencies, is not about technology but is an organising principle for the enterprise and lends itself to a simple and consistent form of governance and management. Form follows function, and the structure of the required governance and management committees follows the discipline of dependency-oriented thinking. [It may be argued that the bulk of a modern enterprise's business logic tends to be coded into software-based systems with very little residual manual processing, so “it's all IT anyway”. Let us refute this argument with a simple analogy. Just because all legal documents in a country are in the English language does not imply that a mastery of English is all that is required to understand or to draft legal documents. One needs to understand law. In similar fashion, even if every piece of business logic in an organisation is implemented within a technology platform, understanding how to govern and manage technology alone is not sufficient. The principles behind how business logic and business data are identified and organised need to be understood, and the analysis of dependencies is a crucial tool to achieve this understanding.] 34
  • 35. Functions and Processes Broadly speaking, the processes required for SOA Governance and SOA Management are either one-time or recurring. To keep an otherwise dry topic light and readable, we will illustrate what occurs during each of these processes through imaginary conversations between various parties. The players in all these scenarios are the following: G – Dependency Governance Committee M – Dependency Management Committee B – Any Business Unit One-time Processes When an organisation embarks on doing SOA Governance and Management the way this white paper recommends, there will be a one-time activity (which could of course be broken up into phases for manageability) on the part of the Dependency Governance Committee to do two things: 1. Agree on the set of approved dependencies at each level of the organisation as suggested by the BAIT model; 2. Identify the dependencies that actually exist, specifically the ones that fall outside the set of approved dependencies above. The latter list (the set of actual dependencies that are not approved) is the input to the Dependency Management Committee to plan a program of work to remediate. 35
  • 36. Governance Functions Management Functions Publication: List of Approved Dependencies One-Time Dependency List of Dependencies to be remediated Review Update: List of Approved Dependencies Need not Periodic Remediation align to Review Dependency Program Business cycle Review Proposal Project List of cycle Dependencies to be remediated List of programs, Remediation grouped and costed Program Review Business Business Requirements Driver (Objective) Initiative Details New New Business Initiative Initiative Project Review Proposal cycle Non-approval: List of Dependencies in violation Update: List of Approved Dependencies Approval BAU Program Management Approval: Prioritised list of programs, with funding Report: Dependencies remediated Dependency-related Process Existing Business Process Fig. 14 – Dependency Governance and Management Processes 36
  • 37. Initial Dependency Review Scenario 1: G: “We've done a review and decided that the only legitimate dependencies are these. There are many actual dependencies we've found in our systems that are not in this approved set. They need to be eliminated.” M: “OK, we'll come back with a proposal for a program of work to remediate them.” Recurring Processes There are recurring processes within both the governance and management functions. Recurring governance processes are reviews to re-validate the set of legitimate dependencies at any level and to identify afresh the set of existing dependencies that fall outside of this approved set. The latter set (which is expected to shrink with each iteration) is then an input to management committees to initiate programs of work to remediate. Prioritisation and approval of these proposed programs of work is also a governance function. Periodic Dependency Review Scenario 1: G: “Given the changing nature of our business since the last review, some new dependencies are now deemed legitimate and some old ones no longer are.” M: “OK, we'll make a note of the new set of approved dependencies and come back to you with a proposal for a program of work to eliminate the ones that are no longer approved.” Scenario 2: G: “Some new dependencies that are not in the approved list seem to have “snuck into” our systems since our last review. These need to be eliminated ASAP.” M: “We don't understand how that could have happened. We'll come back to you with a proposal for a program of work to remediate them.” Remediation Program Review M: “We've put together a proposal for a set of programs that will remediate the unapproved 37
  • 38. dependencies that you identified. We've worked out the costs and benefits of each of them. Please tell us how you would like to prioritise them for execution, and please approve funding for the ones you want done in the current period.” G: “We think you should initiate the following subset of programs in the current reporting period and defer the remaining for a future period. The funding for the programs in the current period is hereby approved.” M: “We'll initiate these programs right away.” 38
  • 39. Recurring management processes are intended to identify programs of work to remediate the invalid dependencies identified by the governance committee(s) and to manage these programs once approved by them. Remediation Program Proposal Scenario 1: M (internal conversation): “Given the set of existing dependencies that were identified by the Dependency Governance Committee as not being on the approved list, let's work out the costs and benefits of eliminating them, and group these tasks into cohesive programs of work for them to prioritise and approve.” BAU Program Management Once a Dependency Remediation Program is approved, it is managed in exactly identical fashion to regular business projects. These are not to be treated as projects of secondary importance or nice-to-haves because of a mistaken characterisation of SOA as technology. On the contrary, the involvement of business and technology representatives and proper positioning of the benefits of these initiatives by Enterprise Architecture should ensure that they are taken equally seriously as programs initiated by the business. After all, unfixed dependencies mean significant amounts of real money, as we have seen earlier. The business initiatives detailed in the next section also feed into Business-As Usual (BAU) program management alongside dependency remediation programs. The standard discipline of conducting Post-Implementation Reviews (PIRs) to evaluate the success of a program should be conducted for dependency remediation initiatives as well. Granted, the benefits are not always immediate but recurring, but it should be possible to assess if the dependencies slated to be eliminated were in fact removed, and a fresh assessment of the costs and benefits of the initiative should be conducted to report back to the Dependency Governance Committee. New Initiative Appraisal An event that could trigger a fresh governance activity is when there is a business driver (objective) necessitating a program of work, the proposal is put forward, and the governance committee evaluates if this introduces any fresh dependencies that are not on the approved list. (Of course, any additional dependencies may be justified, in which case 39
  • 40. the approved list is updated, or may even make some existing dependencies unnecessary, which again results in updates to the approved list). The proposal may need to be amended to avoid introducing dependencies that are not approved. A new initiative could be on the business side (e.g., entering a new market, introducing a new product, acquiring a company, outsourcing or re-insourcing a business function, changing a business process, signing up a new partnership, etc.) The initiative could also be on the technology side (e.g., buying and installing a new product system, developing a new system in-house, outsourcing or re-insourcing a technology function, changing the technology implementation of a function or process, decommissioning a system or application, undertaking a turnkey project, etc.) All of these changes have dependency implications that have to be reviewed before the initiatives concerned can be approved. A large part of the technical and business “debt” that is incurred by organisations is due to a failure to assess new initiatives for their dependency implications. The following process is meant to plug that loophole. Scenario 1: B: “Here's our proposal for a new project.” G: “It appears that your project is about to eliminate some dependencies. That's good. Project approved.” B: “Thanks, we'll initiate the project.” M: “We'll make a note that the set of approved (and existing) dependencies will shrink after this project.” Scenario 2: B: “Here's our proposal for a new project.” G: “Your project introduces a new set of dependencies, but they do seem to be valid, so we'll expand the set of approved dependencies. Project approved.” B: “Thanks, we'll initiate the project.” M: “We'll make a note of the expanded set of approved dependencies.” 40
  • 41. Scenario 3: B: “Here's our proposal for a new project.” G: “Your project will introduce new dependencies that are not valid. We're afraid we can't approve it. Go back and re-think your processes and/or designs so that they stay within the set of approved dependencies.” B: “OK, we'll get back to you with a reworked proposal.” M: (“Nothing for us to do.”) Incorporating a Dependency Focus into New Initiative Appraisal The business case for any program of work should include its impact on dependencies (positive or negative), in addition to the standard calculations of cost-to-implement and business benefit. Dependencies carry a hidden yet heavy cost in terms of business agility, operating cost and operational risk, so we would like to have these made explicit and brought to the notice of approving authorities. Introducing dependencies (legitimate or not) incurs cost and risk, and removing them has the opposite effect. This is where the work done by governance committees in identifying dependencies will form a useful input. The following page has a simplified form that illustrates this. 41
  • 42. New Initiative Appraisal Form Cost Business Benefit Dependency impacts (+/-) Description of proposed project CapEx OpEx (3 Yrs) Business agility Operating costs Operational risk (3 Yrs) (3 Yrs) (3 Yrs) assessment Remove replication of customer (200,000) (25,000) 90,000 50,000 80,000 Changes from address data by consolidating it (25,000) 90,000 50,000 80,000 medium to low within system X where it belongs (25,000) 90,000 50,000 80,000 Approved by: Approval date: Fig. 15 – An indicative format for a “New Initiative Appraisal Form” The form above illustrates how the hidden benefits from the removal of dependencies (or conversely, the hidden costs of introducing fresh dependencies), once quantified, could sway the business case for a project either way. In this example, the business case for a proposed project doesn't by itself stack up because the total business benefit over a 3 year ROI horizon ($270,000) is less than the cost incurred ($275,000). However, the benefit goes up enormously once the improvements to the organisation's agility, cost and risk profile are taken into account. A dependency focus therefore drives behaviour that is desirable over the longer term. This is the kind of change we would like to see in the way organisations evaluate programs of work. It represents a “least-dependency” model of doing business which keeps them lean and low-risk.
  • 43. Processes in Steady State The recurring processes around dependency review followed by remediation are expected to wind down as the culture of dependency-sensitivity begins to take hold and the number of exceptions correspondingly reduces. In steady state, the only processes that we are likely to see are New Initiative Proposal, New Initiative Review and BAU Program Management, since these processes will subsume the dependency-related activities that are the sole focus of the other processes. Periodic Dependency Reviews will still take place, but with any luck, no deviations will have occurred and hence no remediation programs will need to be initiated. Governance Functions Management Functions Business Business Requirements Driver (Objective) Initiative Details New New Business Initiative Initiative Project Review Proposal cycle Non-approval: List of Dependencies in violation Update: List of Approved Dependencies Approval BAU Program Management Dependency-related Process Existing Business Process Fig. 16 – Ideal Dependency Governance and Management Processes in “Steady State” 43
  • 44. Governance and Management Check-lists With our governance/management bodies and processes in place, it's time to look at what they will actually do. Here is a recommended approach based on a detailed analysis of organisational layers along with a sample check-list of questions for the committees to ask. Classification of Dependencies Consider a set of dependencies between entities within a BAIT layer (or across layers). Entity 1 dependency 1 dependency 3 dependency 2 Entity 4 Entity 3 Entity 2 dependency 4 Entity 5 Fig. 17 – An unclassified set of dependencies These dependencies may be further categorised into (say) two broad groups because they are themselves related in some way. A governance or management committee may then assign responsibility for these two dependency groups to two separate teams. Dependency Group 1 (covers dependencies 1 and 2) Entity 1 dependency 1 dependency 3 dependency 2 Entity 4 Entity 3 Entity 2 dependency 4 Entity 5 Dependency Group 2 (covers dependencies 3 and 4) Fig. 18 – Dependency Groups containing cohesive sets of dependencies 44
  • 45. The Dependency Governance Committee(s) and Dependency Management Committee(s) may want to organise themselves into specialised teams to study and monitor smaller, cohesive sets of dependencies. We will call this intermediate level of granularity (i.e., somewhere between a single dependency and all the dependencies that may exist at a BAIT layer) a “Dependency Group”. Although not identified in TOGAF, a Dependency Group helps to provide a “system” view of related dependencies. In the following sections, we will document not just the set of dependencies suggested by TOGAF at each BAIT layer, but also our suggested classification of these dependencies into Dependency Groups that can be assigned to specialised teams as their responsibility. The List of Check-Lists Our check-lists will be numbered according to the following scheme: G(overnance) M(anagement) Basic G-00 M-00 E(nterprise) EG-00 EM-00 B(usiness) BG-00 BM-00 A(pplication) AG-00 AM-00 I(nformation) IG-00 IM-00 T(echnology) TG-00 TM-00 A Basic Governance Check-list Regardless of the level of the BAIT model at which governance is sought to be applied, there are some standard questions that will need to be asked: G-01 What are the core entities involved? G-02 What are the legitimate dependencies between them? G-03 What dependencies can we identify in the current situation that fall outside this set of legitimate dependencies? There will of course be more specialised governance questions depending on the BAIT layer concerned, and we will cover them under the appropriate sections. 45
  • 46. A Basic Management Check-list In similar fashion, there are some standard management questions that need to be asked at every level of the BAIT model: M-01 How do we express the dependencies between entities as suitable contracts? M-02 How will we enforce contracts? M-03 How will we monitor adherence to contracts? M-04 How do we re-engineer our existing systems to align them with the target model? Every BAIT layer will add its own specialised management questions to this list, and we will examine each under its corresponding section. When an organisation's governance and management committees are staffed with appropriate experts and when they adapt to the dependency-oriented way of thinking, they will be able to formulate more relevant questions to add to these check-lists. 46
  • 47. Fundamental Enterprise Dependencies Before we even go into the four BAIT layers, we need to understand some fundamental dependencies at the overall organisational level as illustrated below. The reason why the “enterprise” has been split from even the business layer is that most organisations are highly silo-ed at the business unit level (from immediately below the CEO). Hence enterprise concerns need to be addressed at a level higher than those of business units. Statement of Purpose Goal vision and alignment mission Value Chain responsibility external External Stakeholder assignment Enterprise dependencies Entity Core Roles governing principles Business Principle Architectural Principle Policy Framework Fig. 19 – Dependencies and Dependency Groups at the Enterprise level (TOGAF) The Dependency Governance Committee at this level is necessarily high-powered. It should consist of the Head of Architecture and the CIO engaging with the CEO and executive management. This is the only body that can authoritatively decide on the legitimate dependencies at the enterprise level. 47