Your Data Has a Shape

Everyone - whether discussing technology or wine - can gain an understanding of those characteristics, and more importantly how they affect the success of a product and the team creating it.

In this post, I’m going to walk you through how intimately understanding a fundamental property of our data allowed us to cut the delivery time of a feature by more than 75%.

The punchline: your data has a shape. And, to be clear, I mean that literally. That’s a big, powerful idea, and I won’t pretend that it isn’t. To some people reading this, the point will be obvious; to others, it might be borderline mind-blowing; and, for most, the idea will probably fall somewhere in between.

We had a business need that required what looked like a big change to our system. This enhancement would affect not only some core calculations, but also how our team would manage the data, and how that data would be stored. That’s a lot of change, and we needed it as fast as possible (of course).

To adapt in a way that would reconcile these competing priorities was going to take at least two 2-week sprints. I couldn’t see any way around it. And that wasn’t making any of us happy.

It was at this point that a nice glass of a California Rhône Blend sounded very appealing!

But one thing we have going for us at Sommsation is really open dialogue. People respect expertise, but we also push and question each other outside of our respective areas of focus– but only up to the point where doing that is useful. It’s a fine line that I personally think we nail.

While we were discussing the intersection of business goals and technical realities, it seemed like some of our highest priorities were pulling directly against each other, and that we would be forced to accept one unappealing tradeoff or another. We needed to expand optionality, but we also wanted to keep our machine as simple as we could. We needed to optimize speed-to-market, but we also had to retain our functionality and usability.

As people were sort of stress-testing my thinking and asking me about different ways to think about some of the constraints we were facing, I found myself thinking about the shape of our data.

And, for people who aren’t already used to thinking in terms of data having a shape, I think it could be useful to visualize a Rubik’s Cube to kind of get the intuition here. Each of the Cube’s axes represents a category we were working with (so, three dimensions), and each of the smaller, colored cubes represents a piece of data that has a value in each of the three categories we’re working with.

Now, like the Rubik’s Cube, the data we were working with happened to be three-dimensional. However, unlike the Rubik’s Cube (which has dimensions of 3 x 3 x 3), our data could be thought of as a rectangular solid with dimensions of 7 x 25 x 12.

So, as we were discussing what we were trying to do to increase the features that our API could deliver, I was also kind of playing with the shape of our data in my head, and it dawned on me that the feature we most urgently wanted to add had a length of 12, and that our most urgent new use case probably didn’t require the feature that (also!) had a length of 12 in our current structure. If this was right, we would be able to bridge the new functionality in half the time that it would take us to add the new feature from scratch.

I explained my hypothesis to the team, and there was pretty quick agreement. While we will eventually need to expand to more dimensions by repurposing (i.e., relabelling) the axis that already had a length of 12, we had a quick way forward. We could build the more advanced products in parallel, while delivering much more quickly on the immediate business need. We could get the priority feature delivered within one 2-week sprint, rather than two or more.

Anyhow, I’d love to go way into the vines on this topic with some of the more technical readers of this post. Whether you’re technical or not, though, I’d love to hear from you in terms of whether you think I’ve accomplished the goal I set for myself in introducing a powerful and practical concept with a minimum of unnecessary detail or technical jargon.

Now, about that Rhône blend…

Kind regards,

Matt Hessinger,

Chief Technology Officer