Author’s Note: I've covered the Principles of Ecosystem-Oriented Architecture (EOA) and Mapping your Cloud Ecosystem in previous articles. These are essential pre-reading as you dig into the piece below focused specifically on the Public Sector.

We’ll now make the concepts discussed in those previous articles more real in context of Public Sector organizations. To do so, I want to spend some time speaking less about technology and begin describing workloads that incorporate functions and scenarios upon which a typical agency might rely on its cloud ecosystem to perform.

Workloads

I considered writing this in the form of agency-specific scenarios, in other words, a separate section devoted to what this might look like in various Public Sector domains including (based on my expertise) civilian government, federal or central government (terminology varies between countries and regions), local and regional services, regulatory agencies, military and defense, law enforcement, and even academia.

Eventually, I concluded that doing so would turn this article into a book. So, instead, I’ve created another composite that we’ll use to work through these concepts. You may find in reading that there are one or two specific functions that your agency isn’t particularly concerned with, but on the whole, I think you’ll easily see something of your own organization here whether you are in the business of administering services, running metro systems, regulating industries from gambling to telecommunications, administering agricultural policy, building roads, soldiering, policing, or teaching kids.

*Architectures like this are common across the public sector, with many workloads spread amongst technologies like SAP, SQL, SharePoint, Excel, and many other data services.*

The above diagram shows (icons noted in black) an assortment of workloads common across many Public Sector organizations. These workloads have grown over time as point solutions implemented largely in isolation of one another. Their specific data storage technologies differ between organizations, but I’ve used a combination of Excel, SAP, proprietary databases, SharePoint lists, MySQL, and SQL Server to provide a representative sample.

Job Applicants

The place where it all starts for employees and volunteers eager to join the organization. Think about recruiting, candidate sourcing, interview and offer processes, pre-arrival preparation with recruits before their first day, personnel on-boarding, and all the related activities undertaken to progress someone from being unknown to the organization into their first day as a colleague.

Personnel Management

We could have easily called this “Human Resources”. Examples here include personnel records, background checks, pay and benefits, awards, reasonable accommodations for colleagues’ unique circumstances, and other HR-related processes.

Medical Readiness

Though not a requirement in many private sector settings, some Public Sector agencies find themselves managing the medical readiness and recovery of their workforces due to the organization’s mission deploying personnel (be they sailors, scientists, or aid workers) quite far afield. Examples include core electronic medical records (EMR), vaccinations, psychological screening, clinic scheduling and management. Some agencies are conversely in the business of providing medical care directly to the citizens or constituents (e.g., disaster victims) they serve. I’ve personally built software solutions that ensure scientists in extreme latitudes are psychologically fit to spend many months in darkness, that soldiers deploying to far-flung regions are properly vaccinated, or that disaster victims are properly attended to once they’ve arrived in a safe haven.

Training

I have found that Public Sector training needs can range from the mundane tracking of mandatory training to more complex functions around competency or skills, languages, curriculum management, course delivery, and integration with a full-blown learning management system (LMS).

For example, Public Sector organizations—particularly in high-risk domains such as military and law-enforcement—are generally subject to much more rigorous management of curriculum, tracking of training status, and related activities. It is often insufficient to know that someone completed a required course, rather, it is necessary for the agency to know which version of the curriculum, even down to the level of an individual lesson, a training graduate completed. This capability is not a common feature in your run-of-the-mill learning platforms, so composable architectures are very welcome here.

I’d elaborate with more examples from different Public Sector domains, but again, I am trying to stay in article territory here and not turn this into a book.

Performance

“Performance Management” is one of those domains that means different things to many people, but to paint a picture let’s think of performance in this context as the management of employee performance along the lines of career progression, goal setting, employee appraisal, annual review, and disciplinary actions.

Operations

Many of these agencies are tasked with a set of operational missions or functions diverse as arresting criminals, transporting supplies, fighting wars, caring for patients, and collecting trash (or “rubbish”, depending on which country’s Public Sector in which you find yourself). These functions are diverse so as to lend themselves poorly to monolithic point solutions. They’re often highly inter-related so as to require data integration with other systems, and usually highly specific to the organization so as to make them great candidates for building your own solutions rather than shoe-horning yourself into the process that a vendor built for you.

Depending on the organization’s mandate, “Operations” might also include what we’ll call “constituent services”, activities serving, regulating, or otherwise interacting with their organization’s public—citizens, regulated industries, etc. The private sector might call this “customer relationship management” (CRM), but in the Public Sector context we’re talking about IT functions to enable efficient management of citizen casework, regulatory infractions, refugee resettlement, and—gosh—this list would grow incredibly lengthy were I to not cut it off here.

Facilities

The pieces begin to fall into place. Once we’re entrusting our cloud ecosystem and the composable solutions with such rich personnel data, a natural extension is to further leverage the technology investment to manage the facilities that this workforce inhabits. I’m thinking here of the real estate footprint, e.g., buildings, campuses, and other physical locations, requiring day-to-day management and long-term upkeep, and security functions such as who goes where, and with which badge.

Retirement

We’ve come full circle with “job applicants” above onto workloads I’ve broadly grouped into the “Retirement” category. These could include projected departures and the workforce planning that this necessitates: employee offboarding, administration of retirement benefits (a reality for many, but not all Public Sector agencies), case management for departed colleagues, and the cultivation of the organization’s alumni network.

Now, in nearly every organization I’ve worked with—prior to their transition to an ecosystem-oriented architecture, of course—individual applications in their fragmented collection of point solutions require significant amounts of common data. Personnel data offers a great example, because each of the workloads discussed above require some degree of data or knowledge about the people working in the organization. So, IT organizations build “spaghetti web” point-to-point integrations between data stores using a variety of tools including Power Automate, scattershot use of actual integration tools (event, logic, or batch integration), Excel, and even what we used to jokingly call “sneaker net”, in other words, manually moving data from one system to the next via physical media.

This copying of data—scratch that, this making copies of copies of data—often results in a catastrophic proliferation of (among other things) personally identifiable information (PII). Indeed, our scenario below has resulted in 8x copies of the same PII.

*Unfortunately, these common ad hoc architectural patterns create significant risk and compliance issues by creating copies (and copies of copies) of often sensitive or personal data.*

An ecosystem-oriented approach

A more ecosystem-oriented approach is shown below, wherein we’ve adopted Dataverse as the single source of truth for transactional application data, using it as a data orchestration layer that makes data securely available for use in Copilot, data governance via Microsoft Purview, and virtualized storage in OneLake where the organization’s data can then be used in onward scenarios such as indexing for enterprise search or RAG, warehousing, and analytical workloads that will perform far better than Power BI hitting the application data store directly.

*Microsoft Dataverse serves as a data orchestration layer for workloads across a cloud ecosystem, securely regulating the storage and onward distribution of data integrated with other services.*

I’ll add a note here that traditional architecture might be tempted to argue that “ERP can do all of this, so why not use SAP shown in our first diagram across more workloads?”

This argument is seductive, but wrong.

First, many organizations have become trapped by overengineered ERP solutions. They’ve so extensively customized their ERP over time that replacing it proves to be monumentally expensive down the line. Second, this monolithic app approach is antithetical to ecosystem-oriented architecture because it fundamentally ignores the “Principle of Composability”: The whole ecosystem is greater than the sum of its parts. Modularize workloads to be added or removed with little impact to the whole.

Rather, our cloud ecosystem must:

Disaggregate these functions, many of which would be considered “core business systems” in their respective organizations, from one another so that significant change in one workload does not have a domino effect in the others;
Provide operational insight, analytics, and reporting to decision makers at a cadence that is appropriate to the task, i.e., reporting on training completion can probably wait an hour whereas knowing who is coming and going from your campus may require a real-time solution;
Share data with one another, ideally from mastered data sources, so as to reduce copying data (or copying copies of data) from one point solution to the other; remember, data is the lifeblood of our cloud ecosystem;
Extract value from data, the organization’s collective knowledge, in the form of AI-infused workloads or predictive analytics, e.g., “where shall we stage supplies for the upcoming flood season, based on historical flood patterns and our own supply chain?”

"Reference Ecosystem" for the Public Sector

Think back to the Reference Ecosystem shared earlier as we combine this with the workload map of our notional Public Sector organization discussed above. We’ll link them in the ecosystem map below that I’ve built around these common Public Sector scenarios.

I've modified the "reference ecosystem" discussed in a previous article to be specific to what I commonly see across many public sector organizations, which has produced a "Reference Ecosystem for the Public Sector".

This ecosystem map, like the Reference Ecosystem, is a composite of many organizations. Also, remember the Principle of Artistry, and keep in mind that I could have taken this in any number of other directions. Let’s hit the highlights (moving generally left to right on the map) with those caveats in mind.

We’re migrating several legacy core business systems into a modern ecosystem-oriented architecture. Legacy ERP has been disaggregated back down to its financial management core, with capabilities for HR and facilities moved out of ERP and into their own composable solutions. Note that we’ve also moved sensitive operational data out of SharePoint—making that data far more secure—retired on-premise SQL Server that would have been difficult to weave into our modern data platform, and have moved constitution services out of Excel, which would have been difficult to manage at scale.

We’ve not eliminated SAP entirely, but we have moved the functions that were logical to keep in SAP into a modern cloud-based solution. Otherwise, we have standardized on Power Platform for the development of many core business systems. Our key thinking here is that getting data out of its legacy stores and into Dataverse allows us to more easily integrate that data to our Data Distribution Neighborhood and its onwards distributive workloads (more on that later). Finally, we opted to move constituent services into Dynamics 365 because (in the case of this organization) we felt that D365’s out of the box features were a good fit for these specific constituent services needs. We may have gone in a different direction with a different organization; this ecosystem map is illustrative, not authoritative. The key takeaway here is that we are leveraging Dataverse as the data orchestration and storage service for our core business systems wherever feasible.

This organization had previously allowed data warehouses to sprout up in a rather haphazard way, resulting in four main warehouses each using a different cocktail of technologies. They were hydrated by legacy core business systems that will eventually be sunset, so we will similarly direct the modernized core business systems not to the legacy data warehouses, but to OneLake (more on this in a moment), thus fully integrating their data into the Data Distribution Neighborhood.

Unstructured data is migrated from legacy network file storage—which is still shockingly common—into a combination of Microsoft 365 (SharePoint and OneDrive) and Azure Blob Storage, which will in turn be indexed and made available for onwards distribution.

Our Tier 2 (“important”) and Tier 3 (“productivity”) apps reside within the Application Portfolio Neighborhood, where we will standardize to the greatest extent possible with Dataverse as our primary transactional data service, though expect many personal productivity apps to store their data in a combination of SharePoint and Excel. That’s just fine as long as we are rigorously governing that neighborhood to prevent data leakage. There’s no need to migrate every legacy app in our portfolio, rather, we can integrate them via the Integration Neighborhood as required, modernizing them when the need (and budget) arises.

Data from our core business systems and application portfolio alike is integrated by the services found in our Integration Neighborhood. The specific, neighborhood-level architecture here will be different in every organization, but I’ve left it alone in this ecosystem map.

Finally, we arrive in the Data Distribution Neighborhood. There’s a lot going on here. Most of our data will eventually flow downstream to land in OneLake. This technology is part of Microsoft Fabric and is built atop Azure Data Lake Storage (ADLS) Gen 2. It’s a fantastic technology that also offers a good example of how we can apply the Principle of Following the Money. Microsoft is investing heavily here. OneLake has a 1:1 relationship with the Azure tenant, in other words, there is one OneLake per tenant. There are not zero. There are not two. One. The idea is that organizations manage and govern their lake data here using a strong security model, rather than copying data from data lake to data lake, a long-standing process that has tended to create copies of copies of data (and so on). From there, data will be distributed to “data products” that the organization has decided to build, among them:

Indexing the data estate to support both search of information across the organization and to use in more advanced AI scenarios such as augmenting a LLM (large language model) via a RAG (retrieval augmented generation) pattern, which is a story for another time;
Distribution to external partners and other agencies via API;
Consumption by analytical and AI-infused workloads, of which I’ve offered several examples. These include a Copilot that provides insights and talking points for executive leadership, an operational dashboard through which the agency acquires insights as to the performance of its ongoing operational missions, a “born in AI” workload providing predictive insights to colleagues planning for emergency response contingencies, a workforce planning dashboard used by HR, and a ChatGPT-like chatbot that can be embedded on the agency’s website to provide a more rich experience for constituents interacting with the organization.

Lastly, our Core Platform Services Neighborhood remains unchanged from the Reference Ecosystem, though I do want to call out the importance of Microsoft Purview in getting a handle on data governance across the ecosystem. This is an essential piece of technology for a future-ready cloud ecosystem.

But why Dataverse as the data orchestration and storage service for our core business systems?

The answer is in the word orchestration, for Dataverse isn’t a database in the traditional sense, per say. Rather, Dataverse is a technical service that quite smartly orchestrates much of the storage, logic, integration, security, and other needs for the applications built atop it. In particular, Dataverse is the bridge between an app and the downstream data distribution of the app’s data. It’s difficult to show in a diagram, but implicit to our Integration Neighborhood is the use of what Microsoft calls a “shortcut” to surface data from Dataverse into OneLake. Think of it like a shortcut to a file in OneDrive or on your desktop. We’re not outright copying data into OneLake, which has historically contributed to the copies-of-copies-of-data problem, but rather virtualizing and caching that data in the lake such that we can perform lake- like operations on it without actually duplicating it from its source. This capability to easily use transactional application data from source within the data platform is both critical to the modern cloud ecosystem and unparalleled in alternative technologies.

We’ll explore this more in the next section tackling EOA and artificial intelligence.

Let’s finally assess this architecture through the lenses offered by our principles of ecosystem-oriented architecture.

Principle of Platform First

This architecture employs platform-level technologies at every turn, in our Core Plat- form Services Neighborhood and beyond. We’ve relied on the Data Distribution Neighborhood to consolidate data, in turn allowing us to retire legacy data warehouses. We’ve integrated data using the repeatable, reusable patterns and components in the Integration Neighborhood, and have standardized on key technologies like Dataverse throughout the architecture.

Principle of Composability

Disaggregating capabilities and data away from monolithic ERP means that workloads can be customized, developed anew, or even removed from the ecosystem all together with minimal impact on the whole.

Principle of Evolution

You can’t see it in the diagram, but the transition to this EOA need not occur all at once. We could even leave our legacy SAP and data warehouses in place for some time, preferring instead to focus on building a minimum viable Data Distribution Neighborhood into which we gradually—evolutionarily—pipe data from our newly modernized operations, medical, constituent services, and application portfolio workloads.

Principle of Restraint

There’s a lot that we could have done here, but did not. We could have developed an entirely custom ERP, but modern SAP in the cloud was just fine for our financial workloads. We could have heavily customized a Dynamics application to meet the needs of our operational workloads, but we concluded that a custom app was better. The inverse applies to our employ of Dynamics 365 for the constituent services workloads. We could have gone real-time on the analytical workloads, but instead have blended several of the Synapse technologies to meet a range of insight needs. We made these choices by practicing the…

Principle of Artistry

…Which suggests that, though the ecosystem map we’ve drawn shares many elements in common with the Reference Ecosystem, it is the product of the artistry in combining technologies to suit the organization’s needs.

Principle of Following the Money

The cloud ecosystem shown in this map relies heavily on some of Microsoft’s biggest investments including a Data Distribution Neighborhood built atop Fabric technologies, the widespread use of Dataverse, and the use of Purview to govern data across the ecosystem.

A future-ready Public Sector

We’ll conclude by exploring how EOA equips organizations to leverage artificial intelligence, in particular, AI acting on an agency’s own data.

The diagram below reshuffles the various workloads from our earlier example. On the right we have the collection of Tier 1 “core business systems” discussed at length above, all atop Dataverse as their application data service.

*Our common public sector workloads with Dataverse, integrated to a RAG pattern common in many artificial intelligence scenarios.*

Dataverse then shortcuts that data into OneLake. Meanwhile, on the far left we see an application user experience (UX)—e.g., a mobile, tablet, or web app, built with Power Apps, in this case—that provides an end user the ability to interact with our AI workload.

The application sitting beneath the UX queries the knowledge contained in Azure AI Search’s index (as derived from the data sources on the right). It then passes that prompt and knowledge to Azure AI services to generate an appropriate response to be fed back to the user.

This is what is called a RAG pattern, which stands for “retrieval augmented generation”. The precise methods of causing AI to act upon an organization’s data are both varied and constantly evolving, but whether RAG or some other technique, similar principles apply: Consolidate the organization’s data into storage technologies that are accessible to AI, index that data, and pass that data to AI services that are able to reason over that data and generate a response.

If you’re interested in further exploring the topic of organizational maturity for artificial intelligence, I encourage you to read my September 2024 white paper, Crafting your Future-Ready Enterprise AI Strategy, Second Edition.

The key is that ecosystem-oriented architecture creates the conditions for these types of patterns to succeed, and has made the agency future-ready in its flexibility.

Organizations across the economy and around the world have spent decades kicking the proverbial data can down the road, opportunistically storing data in whichever service was cheapest—or simply present—implementing workloads that are disconnected or integrated as point-to-point rather than via platform-level integration services, and paying little to no mind of data governance.

This article could have easily been four times longer, the basis for a semester-long class, at minimum. So, I’ll leave you with this.

The transition to EOA feels difficult at first, particularly in Public Sector organizations with lengthy budget planning cycles that lack a tradition of viewing technology leaders as strategic leaders. But as wave periods between major innovation become shorter, so too does the need for IT leaders across government to become strategic leaders of their organizations.

After all…

Wave periods between major innovation in the cloud are growing shorter. We no longer have the luxury of waiting it out, of adopting later. Cloud ecosystems built on strategic foundations create the conditions to absorb successive waves of change.

Applying Modern Ecosystem-Oriented Architecture in PubSec