2 Key Lessons From Facebook’s Video Views Metrics Fiasco

People have short term memory (or selective memory), when they can’t remember things they will resort to how they think something should be. Recently Facebook was in the hot seat because of this very reason.

Facebook metrics definition issue

fb-video-metricsFacebook has a metrics called “Video View” for video ads.  In this metric they only counted the video as viewed if it was watched more than 3 seconds by the viewer.  In other words, if someone watches a video for 2 seconds then that video view won’t be counted as a view in this metric.

Facebook also has another metrics, called “Average duration of Video views”, the “standard” definition of it should be Total Time spent watching video divided by Total Viewers. However, that’s not how Facebook defined it.  In Sept Wall Street Journal reported that Facebook “vastly overestimated average viewing time for video ads on its platform for two years.”  This lead to an apology from Facebook

About a month ago, we found an error in the way we calculate one of the video metrics on our dashboard – average duration of video viewed. The metric should have reflected the total time spent watching a video divided by the total number of people who played the video. But it didn’t – it reflected the total time spent watching a video divided by only the number of “views” of a video (that is, when the video was watched for three or more seconds). And so the miscalculation overstated this metric. While this is only one of the many metrics marketers look at, we take any mistake seriously.

As per DM News article, Facebook did state the definition when it rolled out this metric two years ago.  So it was not actually doing anything wrong.  It was a case of short term memory issue.

“The problem, as critics put it, is a problem of omission. While Facebook very clearly states that it’s only counting views as any video-play event that lasts longer than three seconds, it does not go out of its way to explicitly beat readers over the head with the fact that this definition of a “video view” applies equally to the calculation of average duration of video views.”

If Facebook product team had read my posts from 2012 on “Creating a culture of analytics” then they might have likely avoided this “scandal”. The two issues that Facebook dealt with were the exact same ones I talked about in my posts. To recap, here are the gist of those two posts:

Lack of standard definitions for the metrics causes people to report different numbers for supposedly same metrics, leading to confusion and total lack of trust in data.  No trust in data means that nobody is going to use the data to make strategic decisions and there goes all your efforts to create a culture of Analytics.

Having standard definitions is not as easy as it sounds.  It starts from you and your team having a clear understanding on how to calculate various metrics.   Some seemingly simple metrics can be calculated in various different ways and all of those ways might be right but getting one standard way of calculating those removes any confusion and gets everybody on the same page.

  • People have short term memory.  In my 2012 post, titled  Dealing with Short-Term Memory: Creating a Culture of Analytics,  I wrote:

    We all make assumptions from time to time; sometime we state them clearly and sometimes we just assume in our own head. We then operate under those assumptions.  In context of Analytics, one such assumption is that everybody knows what the goals and KPIs are.  We might have defined them on the onset of the program, campaign, beginning of month, quarter, year etc., but once those are defined we start to assume that everybody knows about them and is operating keeping those goals in mind.

    Well the truth is that people have short term memory. They do forget and then start to interpret the KPIs, defined based on those goals, in their own way.  As the Analytics head/analyst/manager, it is your job to constantly remind stakeholders of the goals and KPIs. 

Two Lessons

This fiasco provides two great lesson for all the Digital Analytics teams.

  1. Clearly define your metrics and make sure the underlying metrics and calculations are clear in your definition.
  2. Don’t make any assumptions, people have short term memory. Just because you stated a definition of a KPI in past does not mean everybody will remember it and know how tit was calculated. It is your job to make sure anybody using your metrics/KPI can get to the definition and calculations right away.

Questions? Comments?

4 Data Ownership Questions You Should Ask: Creating a Culture of Analytics

These days most of the marketing solutions are provided as a service. These solutions send emails on your behalf, server ads on your site , serve your ads on other sites/networks, collect your web Analytics data, collect your social media data, collect usage of customer on social media platforms, trade your cookies etc. You get the idea.

As a result, most of your marketing data resides with 3rd party vendors and outside your company’s environment.  In some case you might have an explicit agreement with the company that allows you to have ownership of your data (e.g. Omniture, ExactTarget etc.) while in other cases you might implicitly assume that you have the ownership of data (e.g. Google Analytics, Facebook etc.).  Either way the data resides with someone else.  This lack of direct ownership of your data could potentially pose a threat to your data driven culture.  I am not saying that all of sudden you will lose all your data (though that is also possible) but there is a potential risk.

Source: http://mimiandeunice.com/2011/01/06/ownership/

In order to ensure that you are in control of the situation, you need to carefully evaluate your “Data Ownership” risks and have a well thought out plan to mitigate the risk. Here are few question you need to ask

  1. What if the vendor(s) gets bought by one of your competitors?
  2. What if one of the free tools all of a sudden disables your account because of some violation (perceived or actual) of their policy? (See What I Learned When Facebook Disabled My Account)
  3. What if the vendor has a data breach?
  4. What if you want to move to another vendor?
    1. What will happen to your historical data?
    2. Will you have access to all you historical data? For how long?
    3. Will you be able to port your data into your inhouse system?
    4. Will you be able to port your data into new vendor system?
    5. What will be the cost of porting your data?

Tell a Story with the Data: Creating a Culture of Analytics

Many organizations fail to create a culture of analytics, not because they don’t understand the value of analytics, but because the analysts fail to craft a good and relevant story with the data.  We have all seen reports filled with pretty charts, graphs and numbers.  After going through all the data if the audience doesn’t have a clear understanding of what the reports are telling and what they should do with those charts and graphs then the analyst has failed in his/her job, no matter how pretty the reports are.

Let’s face it, numbers are interesting for a while but a lot of numbers without a story leaves audience either unsatisfied or bored.  However a well-crafted story with data weaved in can keep your audience involved. It is still the data presentation but with a solid story that ties the data back to the business objective.

Data Analysts need to become better story tellers if they want to build a culture of analytics at their organization. Here are some of the elements of a good data story

  1. What are we observing
  2. Why does it matter
  3. How does it look compared to past performance, competitor, baseline, goals etc.
  4. Are we going to be able to achieve the goals
  5. What is effective and what is not
  6. What can we do today
  7. What did we learn that we can apply in future

Anything that is not relevant to the goals/objective of the business/stakeholder should be removed from the story (move detailed data to the appendix).  The KPIs should form the foundation of the data story.

Focus on the story rather than the tables of endless data and you will find that the people are willing to listen.

Other posts in the series

Dealing with Short-Term Memory: Creating a Culture of Analytics

We all make assumptions from time to time; sometime we state them clearly and sometimes we just assume in our own head. We then operate under those assumptions.  In context of Analytics, one such assumption is that everybody knows what the goals and KPIs are.  We might have defined them on the onset of the program, campaign, beginning of month, quarter, year etc., but once those are defined we start to assume that everybody knows about them and is operating keeping those goals in mind.

Well the truth is that people have short term memory. They do forget and then start to interpret the KPIs, defined based on those goals, in their own way.  As the Analytics head/analyst/manager, it is your job to constantly remind stakeholders of the goals and KPIs.  All it takes in one or two extra slides in your weekly/monthly reports/presentation to remind people about what the goals are and how you are measuring them.

Needs, requirements etc. do change too and when that happens it your job to reassess the goals and KPIs to make sure they are still valid.  If not then you need to bring up the issue in front of all the stakeholders. You need to drive redefining the KPIs.

 

Other posts in the series

 

————————————————————————————————————————————-

Standard Definitions of Metrics: Creating a Culture of Analytics

Lack of standard definitions for the metrics causes people to report different numbers for supposedly same metrics, leading to confusion and total lack of trust in data.  No trust in data means that nobody is going to use the data to make strategic decisions and there goes all your efforts to create a culture of Analytics.

Having standard definitions is not as easy as it sounds.  It starts from you and your team having a clear understanding on how to calculate various metrics.   Some seemingly simple metrics can be calculated in various different ways and all of those ways might be right but getting one standard way of calculating those removes any confusion and gets everybody on the same page.

Let’s take an example and see how many ways you can calculate “COST”.  How do you calculate cost?

In case of Search Marketing, I am sure you are taking actual amount paid to Google or Bing. Right?  So that is actual media spend. But what about the cost you pay to your agency for running and optimizing those campaigns?  Where do they factor in? If all you are doing is Media cost then what about Display Advertising?  Is your Agency commission part of your cost? This agency is running and optimizing the campaigns so I am sure you are using that all up cost.  What about your internal email lists? What is the cost of that?   What is the cost of Social Media campaigns?  How do you calculate those? To have one definition of Cost you should calculate it in the same way across all media but most likely you have different way of calculating cost for different media/tactic.

Some more examples:

  1.  Conversion Rate? Is it measured in terms of visits, visitors, new visitors, non-customers or customers?
  2. How do you calculate a bounce? Is it page views based? Is it action based? Is it time based?

If your team is not clear on how to do this then how can you expect others in your organization to understand these metrics and trust the data. Creating a culture of Analytics requires trust in data and that trust requires standard definitions.

 

Other posts in the series