The Small Problem & the Big Problem - Part 1: Introducing the Problems

The problem spaces at solution and enterprise level

What you’ll find in this piece and its upcoming part 2 is the basis of my entire philosophy regarding data & AI work and everything in it, be it data modeling, data governance, platforms, product management, operating models, technology choices, anything. Every project I do, every client I consult, every team I work with - as well as pretty much every data-related text I write or presentation I give - I always try to think in terms of solving both the Small Problem and the Big Problem. And it seems to be helpful.

This is not to claim that I’m about to write something really profound or groundbreaking, or that this would be some kind of a fancy framework¹. Rather, it’s just a way of thinking that works for me, and I feel overall this is rather simple and sort of obvious if you just take a few steps back…

…but being able to take those few steps back seems often a luxury when you’re deep down in the weeds, trying to figure out why the quarterly reports are once again broken. For many reasons, often even stating the obvious can be helpful! So bear with me, as I’ll try to explain how I think about data and why it feels like it’s making sense.

The two problem spaces

Every activity we do in data & AI happens in one of these two problem spaces². You are either:

1. delivering a single solution, or

2. looking after the big picture.

Now, you might think that this is very straightforwad: all the development activities where you code stuff goes into bucket 1, and all that documentation and governance work and so on goes into bucket 2. No, not that simple!

Both the solution-level problem space and the big picture, i.e. enterprise, problem space include different kinds of work - it can be coding or documentation or operating models. It’s a question of target scope.

At the solution level, our target scope is an individual solution. Someone requested a dashboard, we deliver a dashboard. In addition to the practical development work, I count things like the roles and operating model of the specific team in question (are they using Scrum or something?) and all the solution-level governance (no matter how implicit and unofficial it is) in this scope. That’s because all this work is targeting at the delivery of this particular dashboard/API/ML model.

At the big picture level, our target scope is the entire enterprise. This is work relating to questions like how we organize our data teams, what competences and technical capabilities we need, what kind of technology stacks we use, do we have shared solutions for common problems to be used across solutions, do we know what data is where and who takes care of it… The enterprise level is about organizing everything around the solution scope so that many solutions could be consistently delivered now and in the future. The specifics of an individual solution’s delivery are not part of the thinking here.

Simple so far?³ Good - forward!

The Small Problem

First I must state this: I’m not calling this the “Small Problem” because I’d consider it less important. I call it “Small” because the target scope is small, i.e. we’re in the solution-level space.

At that solution-level space, all our activities (again irrespective of whether the activity is technical or something else) should focus on solving this:

How do we deliver this solution efficiently so that it has maximal positive impact?

A few points:

• “Solution” of course refers to the specific thing we’re developing right now.

• “Efficiently” means that we find a way to balance the three constraints of money, time, and scope of delivery without endangering quality too much.

• “Positive impact” refers to actual business value being created with, through, and because of the solution - I wrote about the Last Mile Problem recently⁴, you might want to check that out to see some thinking about where value is really created (hint: the technical engineering artefact isn’t it).

In effect, the Small Problem is all about getting a solution out and ensuring that the business need is fulfilled. And then the people involved start with the next solution, and the next, and the next… the Small Problem is repeatedly solved⁵ & encountered again, from project to project.

The Big Problem

The Big Problem is Big because its target scope is, of course, the whole enterprise.

Now, you might expect me to start talking about some vague high-level governance terms or (heaven forbid!) data-drivenness, but I’m not going to do that. Instead, I define the Big Problem in connection to the small problem, but just lifted up in terms of scale. Like this:

How do we ensure that all solutions throughout the organization are consistently delivered efficiently and have maximal positive impact?

Again, a few definitions:

• “All solutions throughout the organization” means, really, all. Not just the things that a particular team develops, not just the things that the IT department has been asked to do: everything.⁶

• “Consistently” means two things: 1. that successes can be repeated, i.e. the delivery process is consistent because we know what works in this organization and everyone can use the same best practices to their benefit, and 2. that solutions are interoperable, i.e. the solutions that are built are consistent because we have shared practices for e.g. tooling and metadata management so that everything fits together (instead of wildly disparate siloed solutions).

So this means that the enterprise-level Big Problem is really just the Small Problem scaled up. It’s about supporting and managing all those individual solutions and their development; herding cats, as it were, to drive strategic-level impact across the organization (even when the real-value impact of an individual solution might be minuscule in the grand scheme of things).

A large part of why I don’t define the enterprise-level problem in terms of “data-drivenness” or some other nice-sounding term is that I believe that anything we do on the enterprise level that doesn’t impact the solution level is fundamentally pointless. Think about it: it’s the solutions that solve business problems! There is no inherent value in enterprise-level governance (or toolboxes or platforms or documentation) if it doesn’t positively affect the solutions we create. So if you say that our goal at the enterprise level is to be “data-driven”, I ask “how does that change what we do at the solution level”.

Now, you might think that I’m perhaps somehow downplaying the enterprise level.⁷ Far from it; quite the opposite, in fact! My view is that most of the problems organizations face with data & AI these days (and have faced in the past, and will face in the future) are actually caused by failure to solve the Big Problem adequately. And it’s not just the massive corporations, it’s pretty much everyone beyond a ~10 person data team.

Which one are you solving today?

I tend to have most of my assignments on the enterprise level, because that’s where my deep skills are. In fact, nowadays it’s rather rare for me to work exclusively on the Small Problem for some individual delivery project. You, dear reader, are perhaps like me. Or perhaps you work on solution delivery all the time, and rarely have the opportunity to think about the enterprise level?

Both are fine. We need people who are deep experts at the solution level, and people who are deep experts at the enterprise level.

But the magic lies in being able to see both, and to be able to connect them. If you do either in isolation, you’ll miss the forest for the trees⁸.

Up next: the critically important feedback loops

There’s something important still missing from the picture - in fact, something so important, that without it the whole thing is pointless. I’m talking about feedback loops between the two levels, enterprise and solution. It’s these feedback loops that make or break the whole game, and it’s these feedback loops that most often are missing.

If you’d like to be notified when the next part comes out (which should be in a week or so), please consider subscribing to Common Sense Data!

Until next time - cheerio!

1. Sorry, there won’t be a SP-BP© certification exam, training courses, or an official whitepaper, all sold separately at insane prices, turning me into a millionaire (although it’s not that I don’t like money).

2. I’m deliberately avoiding the term “domain” here, because it’s already overloaded with various meanings, and I want to save it for when I write about Mesh-y stuff!

3. By the way, if you find yourself working on something that doesn’t fit into either of these problem spaces, I… don’t know what to tell you. What on Earth do you even do for a living?!?

4. Honestly, I didn’t mean to have both of these posts to be about Problems of various kinds, it just happened! I guess it’s good to have the problem statements out in the beginning, so that the future posts on various methods and possible solutions make more sense.

5. Hopefully solved.

6. I am of course still talking about solutions in the data & AI area and not absolutely everything, but really, if you keep taking more steps back, the thinking kind of still works.

7. If you happen to know me professionally, you are probably not thinking that at all.

8. Haven’t I used this metaphor already like five times?