Understanding cloud ecosystem value and architecture via a “strategic pyramid”
The “strategic pyramid” shown above is a useful way to visualize the different areas of focus for organizations, partner consultancies, and individual technologists working in the Microsoft Cloud.
In my recent piece, Strategic thinking for the Microsoft Cloud, I wrote about how wave periods between transformative technological advances are shortening (e.g., the time between the advent of “platform-first” and then AI breaking on the beach just a few short years later) and the industry’s shift from activity-based to system-based value. I then set forth an approach to baking a more strategy-led approach into an organization’s ongoing cloud technology journey.
I’ll transition, now, to more specifically discuss what organizations should be doing, a model so to speak for how organizations—customers and consultancies—and individual practitioners should be prioritizing their work and investments in the Microsoft Cloud. The imperative here could not be greater. Technological advancements in this space are now moving on timelines that in some instances can be measured in weeks. Not months. Not years. But weeks. This both accelerates and is accelerated by the shift to system-based value. In other words, getting the platform ecosystem right in an organization is both necessary to creating the greatest likelihood that the organization can absorb rapid innovation, whilst simultaneously creating the conditions that drive that rapid innovation forward.
The IT industry faces a major challenge in that too many people and organizations are focused on the wrong things. Or, to be more precise, too many organizations have misallocated their focus up and down the value chain, prioritizing workload implementation either at the expense of or out of ignorance to architecting strategic foundations, building the platform ecosystem, and creating the conditions for success. Too much energy remains focused on building apps ‘n stuff, and not enough energy is devoted to accelerating the building of apps ‘n stuff in an exponential way.
That’s a bit esoteric, so let’s visualize this phenomenon as the “strategic pyramid” shown here. In many organizations—both end customer and partner consultancy—and certainly in the talent market, almost all of the focus is applied to “implementing workloads” shown atop the pyramid. A good slice of energy is devoted to “creating the conditions for success”, but most organizations apply less and less rigor the deeper in the pyramid we go.
In plain English: Almost everyone is up top, but the real action is increasingly down low.
We’ll consider each layer below to understand why this is a problem, and how organizations should be reallocating their focus deeper into the pyramid in order to improve their prospects higher up.
Leveling Down
Implement Workloads
Atop the pyramid we have activities to Implement Workloads, a big category of work in which we should include:
Implementing core business applications such as Microsoft Dynamics 365 (or SAP, or SalesForce, or Workday, etc.)
Extending those core business systems I’ve mentioned above
Building complex custom applications
Migrating and modernizing legacy applications
Building data and AI-driven end-user components
Automating processes across the organization
There are likely more, but I think this is a good sample. Notice that the above list is not filled with simple or technically unsophisticated work. Implementing core business applications can require immense industry or functional—think accounting or supply chain—knowledge, and people have recently become so interested in AI precisely because it is becoming ever-more accessible to and usable by them.
This is where the bulk of cloud technology and IT resources are focused. It’s what much of the talent in the market professes to do for a living: building apps, implementing modules, rolling out productivity and document management services, creating reports and dashboards, etc., etc., etc. (the three etceteras are intentional). It’s where I see the vast majority of effort in most consultancies and partners focused, and for seemingly good reason, because it’s also what I see many business decision makers really obsessed with. This is all highly rational, in that implementing workloads is the tip of the spear, so to speak, we can characterize it as the plane on which most users interact with the technology. In other words, I don’t see many non-technical people (i.e., “business users”, in which I include a great many executives outside of IT) asking about data governance, but plenty of them get quite vocal about lousy apps that don’t meet their needs.
The problem here is not one of importance, but of misaligned focus. Across the industry and in many organizations there is far too much energy applied to the “implement workloads” layer at the expense of the deeper, more foundational layers. This is problematic for several reasons that I will discuss below.
Let’s get this one out of the way first… Implementing workloads is where much of the professional talent is, so it is the easiest thing to insource or shift to other geographies. Whilst end customer organizations justifiably see this as a positive, it is terrible news for partner firms and individual practitioners.
It has hitherto been difficult to achieve exponential efficiency gains here. Yes, there are opportunities to build templates and re-usable components, or employ process improvements such as building an “app factory” rather than running each development effort as a separate project, but these approaches tend to offer more linear rather than exponential efficiency gains. In other words, re-usable components and app factories are tools that help teams of developers work faster or to a higher level of quality. To—say—double your velocity you need to nearly double your developer talent pool. This is usually impractical due either to constraints on budget or constraints on available talent, and in any case as IT organizations grow they can often hit ceilings of diminishing return whereby teams become so large that they become unwieldy and even less efficient.
Which leads us to some more recent evolutions that are already changing the game in a more exponential way…
Low-code is making development much more efficient, both when placed in the hands of professional developers using it to work faster, and when placed in the hands of so-called “citizen developers” building apps in the productivity (tier 3) and important (tier 2) workload categories. Either accelerating workload implementation in the hands of pros or shifting it to citizens using what have become quite mature low-code and no-code tools available in the form of Microsoft Power Platform and its competitors allows organizations to more efficiently squeeze a tremendous amount of value into their cloud journey (read this!).
And if you are unconvinced by the above, consider this: AI is getting quite good at building apps based on human direction (hence the quip I heard recently, “English is the new programming language”), so in the near to medium-term I think we’re looking at “implementing workloads” becoming one of the things most accessible to AI-driven disruption. This is part and parcel to another quip that’s floating around the industry: “Software has eaten everything, and now AI is eating software”.
So, to sum up, “implementing workloads” is immensely important. In fact it is the entire point of our strategy, what we’re working towards. But it is perhaps the single area most likely to be done by less and more flexible professional talent, and accelerated—perhaps exponentially so—thanks to the contributions of citizen developers and AI.
Create the Conditions for Success
The above conditions have—either explicitly or accidentally—led a good number of practitioners, partner consultancies, and end customer organizations to shift their focus in recent years to activities that we’ll call Creating the Conditions for Success. In many cases this has been dressed up as governance, specifically what in this context we’ll call application governance (so that we may distinguish it from data governance), which I have sometimes characterized it as “infrastructure and governance of that infrastructure”. This has been a positive shift in thinking because it focuses effort on enabling the implementation of workloads, and therefor strikes at achieving more exponential efficiency, greater quality, alignment towards greater business value, and more cohesive practices across the organization that ultimately reduce technical debt and buy down risks to security and compliance.
I broadly define “create the conditions for success” in terms of five pillars that I increasingly see consultancies offering and customers buying via almost a managed services model (indeed, I think this is the direction in which traditional IT “managed services” needs to evolve):
Platform Management focused on the tooling (e.g. Azure Monitor, Managed Environments, CoE Starter Kit, etc.) and organizational processes needed to effectively manage the Microsoft Cloud and its component major platforms;
Enterprise Architecture focused on identity, environmental architecture, re-usable components and how they work together, and the care and feeding of our data ecosystem;
Application Lifecycle Management focused on automating and strengthening the integrity of the development process, including source control, deployment automation, standards, and related activities;
Mature Security Model focused on security accreditation within the organization, platform-level security, user management (very much a companion to the aforementioned “identity”;
User and Organizational Empowerment focused on enabling users to thrive in the ecosystem, including business onboarding to new technologies, nurturing the citizen developer community, and—of course—good ‘ole help desk style IT support.
Creating the conditions for success is an incredibly worthy layer of our strategic pyramid because it really bridges the gap between some of the oft-unseen things happening at the core of our cloud ecosystem and the pointy-end-of-the-spear workloads with which most of our constituents interact.
It also, I believe, happens to logically follow some of the patterns acting on workload implementation that we discussed earlier. In other words, work around the pillars listed just above is done by a sizable albeit smaller industry talent pool than workload implementation, it is likely less subject to being eaten by AI in the short term, though AI will surely enable its practitioners to do more with less in fairly short order, e.g., to identify security vulnerabilities, or deploy and manage logical environments based on patterns or standards. The real takeaway here, though is that investment to create the conditions for success unlocks the potential for significant value to be achieved in the “implement workloads” layer above.
Build the Platform Ecosystem
We are now into a layer of our pyramid that I find astonishingly few organizations or their partners explore in a particularly coherent way. Building the Platform Ecosystem is about creating and maturing the core platform elements required for the ecosystem to thrive, i.e., all the goodness happening in the “Create the Conditions for Success” and the “Implement Workloads” layers. As the bridge between your strategic foundations and the upper layers, it is in building the platform ecosystem that we really unlock the potential cloud transformation and exponential value to the organization. This is, in effect, where we create the deep architecture and build out the “ecosystem map” that I introduced here.
Let’s give this some concrete meaning with specific examples below.
Landing Zone deployment and integration wherein we design and deploy the technical components of, say, an Azure or Power Platform landing zone and then—critically—knit these components together. This is increasingly important as Microsoft technologies converge, for (a simple) example in using common identity management in the form of Azure Active Directory across the cloud, or that you may be building Power Platform solutions in managed environments, but those solutions might be built using Azure Application Insights, or their underlying Dataverse may be pushing a subset of data to an Azure Data Lake.
Institutional Data Model (IDM), just as Microsoft publishes its “Common Data Model” (CDM), which provides a set of common tables and other data elements such as “Contact” or “Account” that are used across many organization, so too should an organization develop its own IDM to establish common tables and data elements across its own estate. For example, a shipping company may wish to establish a standard by which it describes “vessels” as structured data, similar with an investment bank describing “deals”, or a department or ministry of defense describing its command and control hierarchy. The IDM can grow significantly over time, but it can also be packaged and deployed throughout the estate so as to accelerate app dev (“implementing workloads”) whereby developers build their apps atop the IDM without having to create the data model from scratch every time. This also, of course, improves master data management and integration between workloads (see below).
Core platform services and data governance bring coherence to the ecosystem in a number of ways. First (and there is a lot of overlap here with the landing zone discussion above), we have enterprise management and governance services such as Azure Purview, Azure Active Directory, Key Vaults, Azure Monitor, and Security Center (non-exhaustive list) that are used across the ecosystem. Second, we have data integration (both logic, e.g., logic apps, functions, event grid, service bus, etc., and data integration such as Azure Data Factory) and storage approaches such as master data management (MDM), lake, warehouse, lake house, etc. In the past we built much of this as one-offs driven by individual workloads, i.e., “we have this workload that we need to integrate with this other workload, so let’s build a mechanism to do so”. But the best practice approach in the platform-first era is to build these as core platform services that can be re-used or gently modified across an entire portfolio of workloads, that is to say, for example, an integration layer that establishes patterns of integration between nodes and enterprise master data.
Integrating workloads across the estate is enabled by all of the above. To be clear, not all workloads have a requirement for integration like this. Productivity workloads—at least at their outset before they become popular or widely used within the organization—will often be built by end user citizen developers and left in peace. But the more critical the workload is to the functioning of the organization, the more likely the need to integrate it with something else. Think of it like assembling a vast jigsaw puzzle where each workload is a piece. Building workloads atop an IDM, deploying them to an established landing zone, and plugging them into core data platform services and data governance makes this task less burdensome, but architects and ecosystem know-how are required to fit the pieces into place.
These activities part and parcel to building the platform ecosystem are absolutely foundational to strategy-led success in the platform-first era. They represent a shift in value from the implementation of individual workloads to the maturation of an ecosystem of capabilities that enable (let’s call it) your cloud center of excellence to create the conditions for success, and a combination of professional developers and consultants, citizen developers, IT pros, and—yes—AI to implement workloads.
True to our model, the expertise required to build the platform ecosystem is also much harder to come by in the talent market and therefor much more difficult to insource, outsource, citizen-source, etc. It is also the least susceptible, at least in the medium-term, to replacement by AI. Again, AI in the hands of experienced technologists will surely remove some burden and tediousness in parts of these activities, but the expertise and alignment to strategic intent (see next section) required to build a thriving platform ecosystem is not likely to become an unaccompanied process any time soon.
Architect the Strategic Foundations
We’ve reached the base—the foundation—of our strategic pyramid, which returns us to how we started this piece: I’ve already published an in-depth essay on this topic, how to go about establishing, then refreshing, the organization’s cloud strategy. Again, see Strategic thinking for the Microsoft Cloud. Let’s take a moment to re-consider these core activities, though:
Executive vision whereby we define the organizational vision for the technology at the CXO level to guide technical adoption, which in turn becomes the “north star” that guides everything downstream (or up the pyramid, as it were);
Ecosystem mapping whereby we map the most important apps, data sources, and user experiences to be built… predictably, this is the blueprint for our higher order (but still core) “build the platform ecosystem”;
Workload prioritization whereby we gather key business and IT stakeholders to compile challenges or workloads to be built with the technology—critically—aligned to the executive vision, which itself becomes the roadmap to “workload implementation”;
Reference architecture, by which I mean the organization’s reference architecture for the platform, to be built as the landing zone and matured over time, so we can think of this as a more technical, infrastructure-focused companion to the ecosystem mapping referenced above;
Data ecosystem mapping to understand how each component technology must integrate with the organization’s data ecosystem, which is to say that this is similar to straight-up ecosystem mapping but with an explicit focus on identifying and mapping the known components of the organization’s data landscape—for example—nodes in a master data management scenario;
Strategic renewal and refresh so that we regularly revisit our strategic thinking whenever the organization’s dynamics seem to dictate (at least quarterly, if we’re being regimented about it), and to refresh at least annually.
Implications
This change in mindset, from that which places a premium on workload implementation to that which emphasizes strategic foundations and the building of a cloud ecosystem, has deep implications for Microsoft, its customers, consultancy partners, and individual practitioners.
Implications for Microsoft
Let’s begin with Microsoft, who for all its vast technical investment in actualizing the concept of “one cloud” made up of multiple component platforms—Azure, Power Platform, Microsoft 365—is still organized in a way that can be counterproductive to the strategy we are crafting and the pyramid metaphor we’re using to visualize it.
This phenomenon exists mostly in “the field”, that is to say, the sales and account management organizations that put the tech in the hands of Microsoft’s customers. To be sure, the product engineering groups are also organized around technology categories, e.g., business applications or Azure data services, but it is in the sales and account management teams where I see my Microsoft colleagues often focusing nearly exclusively on their own designated technology set. That’s rational from the perspective of those individuals, for they are often compensated on the success of the technology they represent. Account executives are, in theory, tasked with bringing representatives of different platforms together to form a coherent whole for their customers, but I too often find that these efforts are surface deep and fail to build any sort of integrated, cloud-spanning strategy.
The good news is that—whilst bucketed around categories—I have found Microsoft’s product engineering groups to be pretty good at developing technologies that play well with one another and support the strategic vision we’ve discussed here.
But there is more that Microsoft can do to make this magic happen on the ground. Building whole-cloud architecture incentives into field compensation would go a long way, as would cross training or more purposefully rotating colleagues across different categories. Increasing the prominence (and capacity) of technical strategists and architects tasked with whole-cloud ecosystem design would be tremendous, as well. Finally, Microsoft ought to reconsider how it positions the technology in the industry community. Introducing interdisciplinary tracks for those earning its celebrated Most Valuable Professional (MVP) award would create new incentives for Microsoft’s most prominent technical practitioners and evangelists, as would encouraging community programming such as conferences that meaningfully bring together content from across Azure, Power Platform, and Microsoft 365.
Implications for Customers
Customer organizations have a tremendous amount of power here to shape their own destiny thanks to the significant investments Microsoft has made in convergence across its own portfolio (albeit, as discussed above, sometimes lacking in field-level execution).
First and foremost, organizations that take seriously the paradigm shift visualized by the strategic pyramid—and, importantly, prioritize the foundational “architect the strategic foundations” and “build the platform ecosystem” activities—are most likely to thrive in the years to come because they have crafted a flexible ecosystem able to both absorb and accelerate successive waves of innovation.
This means that procurement practices, both hiring for internal talent and contracting partner services, need to be less obsessed with implementing workloads than they are with creating the conditions for success, i.e., enabling those workloads to be built, building the platform ecosystem, and architecting the strategic foundations. This way of working will likely involve blending traditional requirements-based procurement and project-based delivery with models that involve retainers for expertise, institutional CI/CD, and a focus on outcomes rather than deliverables.
Organizations should work from roadmaps and act purposefully in converging their IT activities. For example, considering app migration and modernization as an interdisciplinary team sport involving Azure, Power Platform, Microsoft 365, and middle-ground services such as integration rather than treating the Azure app modernization roadmap as a separate work stream from the Power Platform app modernization roadmap. In many cases, this requires a change in how IT is organized and funded. Moving away from teams rigidly siloed along technology category lines funded by rigid budgets that target specific pots of money to specific baskets of requirements can unlock significant opportunities for innovation.
I discussed several of the above concepts in my early-2023 piece The Tyranny of the Deliverable, so give that a read if the above interests you.
Implications for Consultancy Partners
Big or small, partners are symbiotic with both Microsoft and its customers. Unfortunately, almost every partner that I encounter is absolutely obsessed with the “implement workloads” layer of the pyramid, which is to say almost all of them fight one another tooth and nail for the work that is most easily replicated and therefor the hardest through which to differentiate (not to mention the work that likely is subject to the most intense pricing pressure).
This partner behavior portends big implications for nearly every stakeholder in that it often results in the technology’s capabilities being underutilized when implemented, steers customers (who are the clients of these partners) towards technology siloes rather than away from them, and reinforces a talent dynamic in the industry that decouples tip-of-the-spear app development and implementation from deeper ecosystem architecture and strategic imperative.
Just as customers have considerable power to shape their own destiny here, so too do partners have considerable opportunity to lead the market. Seizing this opportunity requires them to (and by the way, customers should use this guidance as a discriminator when selecting partners):
Steer customers towards whole-of-cloud strategy that includes, by whichever name, ecosystem mapping, workload prioritization, reference architecture, data ecosystem mapping, strategic renewal and refresh all driven by executive vision;
Build upon that strategic thinking with expertise in landing zone deployment and integration, construction of institutional data models, core platform services, enterprise data governance, and the integration of workloads across the estate;
Translate that ecosystem into enablement through services in platform management, enterprise architecture, application lifecycle management, security, and user and organizational empowerment.
Note the pleasing alignment of the above with our strategic pyramid.
Otherwise partners are just selling steering wheels to customers who are asking for a way to move freely about town.
Yes, continue to implement workloads. Continue to differentiate yourself with the industry knowledge required to build applications and implement software products to solve specific business challenges. But lead with strategy and prize the foundations upon which those workloads rest. I can nearly guarantee, based on a lot of experience, that the partner who earns its customer’s trust deep in the pyramid will be the first partner they call on to implement workloads alongside the citizen developers, geographically dispersed pros, and AI now joining the fray.
Implications for Individual Practitioners
These are the folks in the talent market. They are developers, IT pros, consultants, product managers, technical luminaries, etc. whose collective body of knowledge ultimately shapes what’s possible in the organizations they serve, the industries in which they specialize, and in the cloud technology community in which they work. The implication for these technologists is clear: The deeper in the pyramid that you cultivate your expertise and experience, the more long-term value you offer the organizations you serve, the more adaptable to the yet unknown implications of the AI revolution, the low-code revolution, the platform-first revolution, and the-on-and-on-it-might-go you become.
Embrace—do not resist—the transition to system-based value from activity-based value. As I wrote in Strategic thinking for the Microsoft Cloud:
“The truly meaningful impact of what humans offer in the IT industry is moving from performing activities to understanding, constructing, and building systems of strategic value. Something like moving from assembling information to creating a framework for understanding as-yet-unknown information, or moving from building an app to architecting an ecosystem… The real action is increasingly in working with a sense of strategic imperative to assemble pieces of value across whatever system in which you are dealing.”
It's not necessary that you be the architectural genius behind every cloud, but if you’re not retooling yourself to work and contribute in this era, then you’re falling from the sky.
Conclusions
I have written this piece as the opening chapter to a white paper on which I am working with several luminaries from across Microsoft. Expect me to share Architecting an Ecosystem a bit later this year, wherein we will share principles and best practices for architecting and building cloud ecosystems within an organization. Think of it is a “double click” on the bottom two layers of our strategic pyramid wherein we will explore strategies for knitting together integration and data storage, core platform services, unstructured data, application development, implementation of core business systems, and data distribution to build platform-first ecosystems. I am immensely excited about this and other related in-flight work.
In the meantime, and every time ;-), I urge you to read my recent essays as a coherent thesis…
Tyranny of the Deliverable proposed the idea that most organizations are struggling to realize big value from Power Platform specifically and other cloud technologies generally because of non-technical outmoded ways of doing business.
Strategic thinking for the Microsoft Cloud outlined a framework for infusing strategy into IT decision making and technical execution in an era where AI aspires to eat everything and wave periods between major technological sea changes are shorter.
Finally, the Strategic Pyramid presents a model for ordering our priorities in the platform-first, AI-everywhere, era of cloud technology where the highest value is found deepest in the pyramid on the opposite end of the spectrum from where almost everyone has spent years focusing.
Ensure that you’re doing your best work there.
