I was a SI/BI engineer for a while; let me chime in to defend my people just a little bit :)

Certainly there are lots of bullshit metrics, there is often very little desire to audit data or even do a by hand sanity check once and a while.

That being said, there were _many_ engineers who actually gave a fuck about making sure we had telemetry in actionable, meaningful, and appropriate places, and unfrotunately the gap in analytics would take place far over their heads, when the upper management would produce documents with all the beautiful charts and graphs, communicating.... absolutely nothing.

Graphs with mixed axis scaling (log vs non log) graphs with mixed units, conclusions that are a total stretch from the data that's there and ignoring obvious conclusions that don't line up with what the managers want to say.

For there to be USEFUL BI (and such a thing certainly can exist) there needs to be a "this isn't bullshit" mindset up and down the whole stack, not just in engineer land.

While what I said was actually an amalgamation of true stories I've seen myself, I was certainly exaggerating a bit :)

I think a lot of things come together for this pattern, which definitely happens quite a lot. Best predictor for this scenario is a clear divide between product and BI people, usually exacerbated by the fact that non-technical people get hired for BI.

Group bias by backend engineers who see BI as "just some point and click" which isn't really that challenging (but usually much better paid, as they are catering sales/bizdev/C-level which is always closer to the money) also doesn't help.

Also, as you say, often times the value of BI doesn't trickle back down the chain and that way, tracking is at most a second thought for application engineers when going prod.

Having built two analytics stacks myself (and seen the perspectives from BI, backend, sales and marketing alike) these are exactly the drivers we tackled first at our current company.

Our recipes against this common failure are: Marketing directly working with product engineers for their tracking requirements (with just some coaching from tracking pros) - and dual-using our analytics stack (Snowplow -> Redshift) for both operations (user segmenting, push notifications) and business intelligence.

This is usually considered a big no-no in BI circles which all tend to duplicate data to be on the safe side, but it helps immensely to make sure product and engineering are just as interested in data quality as the BI guys, as they depend on the very same data.

It's certainly not for everyone, but for us, it works really well.

wow.

i've only ever worked at small companies, and i guess to avoid things like this is why, but i can't help but wonder how common this is, and if in fact no one has ever used BI effectively. i find that hard to believe but really i have no idea, and its an interesting thought, makes you want to chuckle or shake your head or both.

Of course I wouldn't go so far as to say no one has ever used it effectively, but it seems like in the vast majority of cases, it's bullshit all the way down.

It's a surreal experience when you realize how executives are making decisions based on something that, after a moment of critical thought and common sense, clearly has no more relevance than some integers pulled out of rand(). Chuckle and shake your head is indeed about all you can do.

Just try not to visibly smirk if you're ever involved in acquisition talks. Play along.