I never really had to think about actual business data until about five years ago. Before that, like a lot of professionals, I dealt with lots of numbers, but none of them actually mattered because they were all basically vanity metrics that never got to the root of anything. I tracked things like open rates and leads created via campaign X and all of that, but as someone in a low level support role, or product marketing & copywriting at a small company, I never had to connect the elements of my work to — literally — how we made money.
When I joined Contactually in 2014, it was the first time a majority of a company’s revenue went through something I was so directly involved in — in that case, content and eventually product marketing, which by the time I left involved getting people through the trial phase of things and into a monthly subscription. Sure, there were sales people moving things along for certain clients, and eventually working on enterprise deals, but during my relatively short time there (about a year and a half), we made most of our money through self-service, and I was supposed to make or work on a lot of the things that made self-service work, from a business perspective.
Working with a cheaper, self-service business model also significantly expanded the scale of the numbers I did have to think about. We needed thousands of leads to get to our goals, and they came in (and left) very quickly. I spent a lot of time looking at spreadsheets, and “the leads” were sort of an amorphous concept of success to me, versus an amalgamation of actual people. While I didn’t like, freak out and run out of the building screaming, I never felt like I had a real grip of what was going on, and instead sort of poked at various marketing tactics that were supposed to work while watching large numbers move up or down in ways that mostly seemed disconnected. I did not really know what I was doing, but I learned a lot, even if it was mostly just learning what I needed to get better at.
I joined FiscalNote in 2015 to go into a very different form of product marketing that was closer to what I knew — basically glorified sales enablement for fairly large ($10k+) annual contracts. My work teed up a large sales team; it wasn’t itself a revenue funnel. I knew how to do this sort of marketing-as-a-service (and how to think about it), so it was immediately less stressful. However, at the time, FiscalNote was still a very young company, and I was sitting in the room and providing supporting material for lead generation and sales processes that — the more I looked — were clearly leaning on absolutely atrocious data practices. Eventually, through what I can only assume was some sort of clerical error, I ended up being responsible for large parts of our marketing department, and eventually the entire thing (probably another clerical error). There was no looking the other way at that point.
When I talk about “atrocious data practices”, I’m not talking about any one person’s individual incompetence, and I don’t mean that we were any worse with data than anyone else. What was horrifying is that we were actually pretty normal, and the people working with awful data were all pretty smart. But that doesn’t mean we weren’t often doing incredibly stupid things. Because while almost everyone was smart, almost no one was cynical about data, technology, or how the two were related.
The number one culprit for us — and I think for many people — was that we constantly looked at data exclusively from the top down. What I mean is that at some point, someone set up a giant Salesforce instance, or a Marketo instance, or whatever, and created some basic, automatic process for names, or companies, or something to get sucked into one or more of these systems. The automation works, the names go in there, there’s a ton of metadata (probably TONS), and it all looks very robust and thorough. Welcome to BIG DATA.
If you’re like everywhere I worked, the initial purpose of this system is to organize Sales, because if salespeople aren’t organized and busy, they will kill you and eat you, and whether it’s two or two hundred, you probably hired too many of them because someone built a capacity model that said “sales head = $X”. Everyone does this, because to some degree it’s true, but companies with big growth targets and some amount of traction tend to overstate it. So the salespeople create tons of “opportunities”, or “deals”, or whatever the basic unit of potential revenue is at your office, and includes exactly the bare minimum of information and updates required for them to be paid out by whatever the specifics of their compensation plan requires. So almost immediately, you end up with tons of zombie data, old opportunities, incorrect information, and outright lies from people who usually get fired after a while. Again, unless you’re entirely self-service, this is your basic unit of business data.
Ghost in the Machine
At this point, something weird happens. Maybe it’s because leaders come and go, and institutional knowledge disappears, or maybe it’s that there’s some amount of success, and it becomes taboo to really think about whether any of these systems were set up appropriately. But before you know it, the key decision makers — and maybe everyone — has becomes completely disconnected from what this data actually is. Don’t get me wrong, people will acknowledge “data problems” in general, and create projects like “clean up Salesforce”, and even spend money and time on it. But for some reason, at a certain stage it becomes incredibly rare and crazy-sounding for anyone with any seniority to want to get in and try to basically model out the business manually, without BI tools or whatever. But that means no one actually understands the data — engineering, or accounting, or an SF admin know the computations that happen (but not why, or what it’s supposed to represent in the real world), and business people look at overall performance and look for ones that make it look like whatever has been bothering them is, in fact, a problem.
So you get business assessments of things like “marketing-influenced pipeline” being exceptional, or exceptionally poor, or important, or unimportant, and some hoodie-wearing goober like me is tasked with responding right in the middle of a really good daydream/reminiscence about the 2008 Eastern Conference Semifinals.
And when I do get asked that — again, because I am a hoodie-wearing goober — I usually ask a simple question.
“Do we actually know what marketing-influenced pipeline is?”
Incredulous faces. MBA syllabi flicker to life in the minds of people desperate to reference them. “You see, what we’re talking about is the amount of opportunity value that was specifically driven by a marketing activit—“
“No, I know that. I mean, how is that calculated at this company, and…”
(this one always ends badly)
“… can I see an example?”
I want to be really clear. I do not think I am smarter than the fictional co-worker archetype I’m lampooning here. I may think I like better music, I may think my jokes are better, and I may even think I have better taste in typography than these people (I do think this), but I do not actually think I’m smarter, because I am a moron about all kinds of things.
But what I’m really good at is detecting well-intentioned self-delusion. I don’t know why this is. Maybe I should write about that someday. But detecting well-intentioned self-delusion is both an incredibly useful AND incredibly difficult skill in corporate America, because corporate America — and I include any startups beyond “garage level”, and certainly any startups with serious venture backing here — often gets through tough days entirely with caffeine and well-intentioned self-delusion.
The business analytics industry, and the B2B software industry in general, knows this. Hell, I know this, because — very meta — I work in marketing spinning up elaborate, plausible scenarios for how we can make your self-delusion slightly more well-intentioned and slightly less (more?) deluded.
And all of this is possible because people, especially decision makers, simply cannot resist looking at data exclusively from the top down.
No, You May Not See an Example
Every executive loves a good dashboard. Why is that? In theory, it’s because it gives you the executive holy grail of buzzwords… the “30,000 foot view” of a business, a function, or whatever. But what’s really interesting, and maybe even more important, is what it doesn’t tell you, which is specific instances of any of the trends, rates, or totals its reporting. Oh sure, it’ll show you “top five sources of whatever”, or “biggest deals of the quarter”, but we’re talking about a really small, featured subset of data, and often things that you already know intimately. Who sees something call out their third biggest source of revenue and goes “huh, never heard of that one”?
Instead, in a surprising amount of scenarios, you get things like total numbers — or even better, month over month changes — with no way to drill down into them directly from what you’re looking at (which means you won’t go look). Salesforce is notorious for this, or at least it should be, because it’s one of the worst offenders while still allegedly being some kind of machine learning powered business wizard. But more times than I wish to remember, I literally could not drill down into the individual data that added up to some key business metric we expected to base a decision on, unless I exported it to an Excel file and worked on it myself. And when I find something weird, you know what I hear?
“Can you put that in a Salesforce report and send me the link?”
And the thing is, I don’t blame anybody for asking that, because something like Salesforce, or your CRM or ERP or whatever is supposed to be your system of record. People should be skeptical of off-the-reservation analysis in a spreadsheet by anyone with an agenda, which is everyone, including me. But when your system of record makes it really difficult, or sometimes actually impossible, to audit your data piece by piece, you quickly end up with an analysis and decision making culture that’s a messy combination of a few sales anecdotes from the loudest people in the company, and a top-down view of metrics full of unvetted, invisible assumptions.
I don’t really know how to solve this institutional problem, but I did find a way to get my arms wrapped around what the hell was going on — I used this crazy Google Sheets plugin to manually suck in every single lead, and tons of metadata, from Salesforce into a tab called “TEH DATAZ”, and then I built a bunch of elaborate queries in with Google’s kind of amazing query language to pull in things from different tabs and do a bunch of math for me. I’m not really a spreadsheet person, but I am a reality person, and the thing about having all my math in that stupid, bloated spreadsheet was that when some total seemed odd, or important, I could go check every individual record that comprised that total. And I did, all the time! It was incredibly informative, and literally every week I found everything from bullshit duplicates that were inflating marketing numbers (whoops, my bad guys) to piles of qualified leads that never got sent to anyone, or were marked as “unqualified” by a 23 year old, probably by accident because they had to get through thirty required drop-down fields in Salesforce.
I’ve fixed a lot of stuff with this irritating, stubborn insistence on working with both big and “little” data, but I always get the feeling — wherever I go — that people resent it. And the thing is, I know it’s not just my annoying face they resent (again, different blog post for that topic), because as soon as I go up to the 30,000 foot view again, people are so happy to hear what I have to say.
The difference is that when I say those things, I’m happy too, because I actually believe it, because I insisted on understanding the individual pieces behind the trend before I started thinking about the trend itself. I think we could all be a little happier, a little less defensive, and work for significantly more productive organizations if we stopped trying to avoid that part of the process.