The ABSOLUTE Basics of Data Architecture

Taking a looooong step back from all the “markitecture” and seeing the big picture (or actually, just one picture)

For us living the data life, things like “data warehouses” and “data lakes” and “data lakehouses” and “datalakewaremarts”¹ are the bread and butter of our daily lives. But we tend to forget a couple of things:

• First, not everyone lives and breathes data. Many people are in this business only partly, or passingly, or reluctantly – they just need to do some data stuff to get on with their daily lives. We can’t expect everyone to follow all the terminology and pedantic discussion about the semantics of, amongst other things, “semantics”.

• Second, sometimes it’s difficult to see the forest for the trees even if you are in this business for good. With the level of technological detail being casually thrown around in the discussions in various conferences, forums, LinkedIn, and even here on Substack, it’s often feels like we’re missing even the trees for the individual leaves².

In the Finnish language, we have an idiom “vääntää rautalangasta”, loosely translated “to twist from iron wire”, meaning that occasionally you have to make an actual wire frame out to make some concept understood. My favourite variant of that saying is “vääntää ratakiskosta”, where instead of wire, we’re bending railroad tracks. You probably understand what the difference is!

In this article, I want to look at Data Architecture in the way it was taught to me many moons ago. Those wire frames were made out of very thick steel bars, so that when you took a long step back, you could see how the big picture is supposed to work without losing yourself into minute details. So don’t expect anything here that would immediately help you deliver that pipeline; instead, lean back and forget the daily toil for a second.

The reason for this approach is that I want to take into account the two issues listed above; non-data people trying to figure out what’s going on, and data people who are in too deep to remember what’s going on. There’s a bunch of articles coming up later that will go a bit deeper into how I think about various architectural topics, and I hope what you’re about to read will give some context to my thinking on all that!

Zooming out

To truly understand Data Architecture, we must zoom out. Leave the use case you’ve been working on. Leave the platform you’ve built. Zoom out further. Further! Are you still thinking of warehouses and lakes?! FURTHER STILL, I SAID FURTHER! No houses of any kind, no hydrological features visible? There you go. Hold that zoom level for a moment.

For many decades, since the earliest work of Bill Inmon & co, we’ve generally had an approach to data that could be summarized as follows:

To serve data about our business to various analytical use cases, it’s best to take data out of the operational systems it’s born in, and to bring it all together in some other place.

Think about it at this level for a second. Data is created when someone types in the details of an invoice. This data’s primary purpose is to ensure people pay us what we’re owed. This we call the operational (or primary) use of data.

If we want to analyze the last twelve months worth of invoices, and connect that to our customer data, to perhaps see who has a tendency of missing their due dates, we are talking about the analytical (or secondary) use of data.

It used to be that the operational systems where data is born were optimized for dealing with individual invoices (or whatever data elements). Asking the system for the last 12 months of invoices meant a huge burden which might hinder the actual day-to-day operations, and we didn’t want to risk that! To a large degree, this is still the case (though not necessarily for performance reasons always). Operational systems are optimized for operational data use.

This article is not a history lesson, so I’m not going to go through all the various solution patterns of data extracts and whatnot during the last 50 years. Suffices to say, over time people realized three things:

1. To avoid bothering the operational systems, it’s generally a good idea to take the data out before we start doing analytics.

2. To enable more sophisticated analytics, we need to be able to combine data from multiple operational systems.

3. To enable the same data being used for different kinds of analytical use cases, it’s better to make it reusable.

I told you to zoom out earlier, didn’t I? Now what does this look like, when you zoom out enough: we have many sources for data, we take the data out, combine it, and serve that combined data to many data consumers?

It looks like this:

Oh crap, we zoomed out too far!!

We have to go back! Zoom in a bit…

A bit more…

Now enhance…

There we go. This is the bowtie. That’s the only picture you need in order to understand Data Architecture.

Everything’s a bowtie if you squint

As I mentioned in the intro, the bowtie was how Data Architecture was introduced to me maybe 15 years ago. I’m not sure if that is how it’s taught nowadays, or if it ever was taught like that for the majority of data people, but for me it helped to put things in their right place.

Think about it:

• You have multiple sources for data: these are the left side of the bowtie.

• You have one place in which data is stored and combined for analytical use: this is the knot in the middle.

• You have multiple use cases for the data, for which you might create all kinds of use case-specific datasets: this is the right side of the bowtie.

Multiple inputs, one thing in the middle, multiple outputs. Now here’s the thing: at this level of abstraction, pretty much every Data Architecture innovation for at least the last 35 years is a bowtie.

• Data Warehouse? That’s the knot in the middle, where data is stored and combined.

• Data Lake? That’s the knot in the middle, but this time data is only stored physically in one place and never properly combined with other data.³

• Data Lakehouse? That’s the same as the Data Warehouse, except that due to marketing reasons we must use a different name when the underlying storage technology is different.

• Data Virtualization? Wait, does anyone even remember Data Virtualization anymore? It was pretty much the same thing, except this time data wasn’t physically moved anywhere.

• Data Mesh? Well... we’ll get back to this some other time, but honestly this might be one of the few ideas where the traditional bowtie is not exactly there. Instead, you have multiple bowties, per domain.⁴

And so on and so forth - fill in your Data Architecture term du jour.

Why the knot matters

The reason why Data Warehousing was such a big thing originally was that the knot in the middle played two very important roles. Without going too deeply into Bill Inmon’s criteria for a Data Warehouse (although you probably still should read up on those, as it’s as important as ever to get those right: you can start from this rant), we can focus on the following:

• The knot stores data in one place (this doesn’t necessarily need to be one physical place, it can be just a collection of references, like a library), so that it’s easier to access: you don’t need to go hunting for different datasets in wildly different places.

• The knot combines data from multiple sources: you don’t need to do it for your use case, and then for your colleague to repeat the same for theirs, and so on. This we call data integration.⁵

Make the data accessible and findable from one place, and make it integrated so that data elements from multiple systems that tell us about the same things (events, places, people, whatever) are pre-combined, saving the data consumers a lot of time.

Everything in Data Architecture is based on this. Just the exact methods, models, and technologies differ.

The knot, however you choose to build it, is the piece that keeps the bowtie in shape. It’s what makes Data Architecture valuable. Without it, we’re just building endless point-to-point solutions, reinventing the wheel over and over again.

The basics of Data Markitecture

When you look at any data tech vendor maps (or, heaven forbid, the MAD landscape), you will find technologies focusing on specific parts of the bowtie: some like to focus on the left-hand side, making it simpler to get data into the knot. Some focus on the knot in the middle, making it simpler to combine data sources and to store the results. Some focus on the right-hand side, making it simpler to distribute data to various use cases. Some try to do many things at the same time, or to orchestrate the entire thing. Whatever tool you pick, chances are you can place it somewhere in the bowtie.

When you’re a software vendor, you need to figure out your differentiation. How is your technology different from the 100s of other players in the same space?

If you’ve built your tool in a crowded space, this is much more difficult. Purely technological improvements are not that good. “20% faster” is not very exciting! So you start to create a sub-category.

A lot of the buzz around Data Architecture is vendor-driven differentiation. The basics of Data Architecture have not changed; it’s still the bowtie when you zoom out enough. But coming up with a NEW AND DIFFERENT! way of doing the bowtie is good for marketing, so we have endless churn of books and conferences and diagrams and terms. This is Data Markitecture – marketing-driven architecture.

I have rather strong opinions on what is Markitecture and what is not, and you might have caught a whiff of that in the preceding paragraphs. But what I personally think about Architectural Pattern XYZ is not that important; what’s important that we all are able to see the forest for the trees and to place all this terminology and technology in its context.

What’s next in Data Architecture?

Most of the buzz in the business comes from either new technology that hits an existing piece of the bowtie, or (as mentioned) vendor-driven Markitecture. These might be cool and interesting, but they don’t change the fundamentals.

Occasionally, properly new ideas appear. These are just more difficult to see, partly because they tend to hit at the level of abstraction we don’t generally perceive in our daily work, and partly because they are sometimes so esoteric that they never get real traction. But sometimes the odd ideas catch on, and things change for real.

Personally, I’m really looking forward to someone way smarter than me figuring out a way to break down the bowtie for good. The whole operational vs. analytical dichotomy is, in my opinion, getting more and more useless and harmful by the day. Analytics is getting fed back into operations all the time. It feels like many of the operational/analytical boundaries are now maintained simply due to organizational friction, more than technological necessity. If you’ve worked in corporate IT for 20 years you have an established way of thinking that might be difficult to change (and it’s not because you’re stupid or anything, it’s just that everyone’s thinking gets stuck in a rut very easily over time).

But as long as the bowtie isn’t blown up, it’s good to be aware that some things haven’t changed.

Thank you for reading! I have a few architectural topics waiting on the backlog, so it was good to get this piece out. If you’d like to read more of this kind of stuff, please consider subscribing! Also, you might like to share this article, or perhaps write a comment about what has been your tortured metaphor way of understanding Data Architecture.

Until next time - cheerio!

1. Obviously, this is not a thing... yet?

2. Though here on MY Substack we don’t have this problem, because all of you reading this are clearly super smart and excellent people altogether!

3. Actually combining data from different sources in a Data Lake is left as an exercise to the data consumer, which of course could never work for the vast majority of the actual use cases actual people have in actual non-FAANG organizations, and this is why the whole Data Lake craze was outright silly and why you don’t really see many Data Lakes existing by themselves without any support from a Data Warehouse (or similar) anymore.

4. But then again, in a Mesh you’ll need to have a metadata setup that allows various data consumers to find and access all the various data products created by various domains from a single place, so perhaps that is like a metabowtie?

5. Note that here “data integration” means integrating data with other data coming from a different source. Sometimes people will also, confusingly, call it “data integration” when they move the same data from A to B. I’ll write a proper rant on this, too, at some point!