Is the Pinterest Way to Measure Ads the Right Way?
A cutting-edge data science model can only be created if the impact is measured properly. Pinterest upgraded everyone's preferred impact measurement metric, CTR, after carefully analyzing the problems.
Every Data person adores the CTR measure as it is:
- Simple to explain to leadership, product managers do not question it, and measuring it is straightforward: Divide the number of clicks by the total number of impressions.
- CTR data is easily available, so it will be easy to calculate, track, and analyze.
- And, as a bonus, lots of advertisers want to have high CTRs too. So, you’ve aligned your advertiser and user interests!
So, just throw a lot of advanced machine learning at the problem to predict and maximize CTRs and you’re done, right? Not so fast!
CTR Suffers From Some Serious Shortcomings. “CTR Is Not the Key to User Engagement.”
CTR just reports on clicks on a certain item, regardless of whether it is relevant or not. Some may claim that the user clicked because the content or ad was relevant, however, this is not always the case; it could be due to positional bias, clickbait, or other factors. These flaws will be discussed in further depth below. CTR is only useful for measuring short-term gains without consideration of user engagement. As a result, we require a metric to track user engagement.
“You can fool all the people some of the time and some of the people all the time but you cannot fool all the people all the time” - Abraham Lincoln
Problem 1 - Positional Bias
Ads higher up on the page are more likely to be clicked, but this does not imply that the user finds them more valuable. If you fail to take this into account, you can conclude that users prefer the ads at the top of the page when, in reality, they are just more inclined to pay attention to them.
Problem 2 - CTR Promotes Clickbait
Items having a high click-through rate (CTR) may be "clickbaity" — they entice the user to click without providing anything helpful, instead sending them to ad-loaded slideshows that generate revenue for the advertiser but annoy the user.
Problem 3 - CTR Ignores Other Signals
CTR doesn't imply positive interaction. The person may still despise that specific piece of content. Hide Ads or drop-offs are powerful hate signals that CTR does not take into account.
Above is a Google Ads example to capture explicit user dislike feedback. No point in showing a strong dislike signal ad with high CTR; it will eventually put the user off the platform someday.
Problem 4 - Some Things Are Not Meant to Be Clicked
Ads that do not require a click, such as a video, brand awareness, or pitch ads, such as an Independence Day discount at a nearby store or a short video of a new car launch.
Okay..! We have now listed the problems with CTR but what is the Solution?
Going Beyond CTR
They adjusted their ads measurement metric to address the difficulties with CTR mentioned previously. A metric that incorporates engagement and more accurately captures user experience.
User Metric:
Let's discuss the Numerator first - Weighted Engagement on Ads
Instead of just looking at CTR, we look at a weighted average of different actions — Clicks, Hides (Stop ads), Watch Time (on Video Ads), and Saves, among others — on the ads.
Consider two ads: B is clicked more often than A, but it is hidden by a lot more people.
Conclusion - One can readily conclude that ad type A is superior to ad type B because it receives more engagement and fewer users hide or disconnect with it while CTR alone shows the opposite picture.
If we choose the actions and weights carefully, we can address concerns 2, 3, and 4 with CTR.
Coming to the Denominator - Weighted Engagement on Neighbouring Organic Content
To address Problem 1 (Position Bias) - We need to dig a bit deeper and understand that the CTRs drop on ads and organic content as users scroll through.
LEARNING (This stands true in the majority of businesses) - If an ad or Feed Content on the 10th Position of the carousel gets a similar CTR to one on the 1st position, this means the ad on the 10th position is better. One can verify this hypothesis of positional bias issue with A/B testing on the platform as well.
To normalize the effect of positional bias on ads content, we compare the average engagement rate on ads to that of the organic content in the spots before and after. This is where the denominator comes in.
Positional Bias Normalise with Organic Content Neighbours
Experimenting with Above setup and New Ads Measuring Metric
Compare CTR vs New Ad Metric
From the above table, we can infer that Treatment has lower CTR than Control but a lot better on our Weighted Engagement metric.
Conclusion
I understand that digesting and practically applying this new metric can be difficult because CTR has been used to gauge ads and content relevance for a long time, but change brings growth.
NOTE: If your ads monetization model is purely dependent on CTR, you can place a higher weight on Clicks when taking weighted engagement scores, but deleting engagement aspects and relying solely on CTR would make users unhappy in the long run.
Reference
Is there anything you would like to add? Please feel free to share your views and feedback in the comments section below.
Connect, Follow or Endorse me on LinkedIn if you found this read useful. To learn more about me: visit Here
This newsletter is now read by more than 4500+ subscribers. If you are building an AI or a data product or service, you are invited to become a sponsor of one of the future newsletter issues. Feel free to reach out to [email protected] for more details on sponsorships.
I am nominated for the HackerNoon 2022 Noonies, Vote for me: https://www.noonies.tech/2022/programming/2022-hackernoon-contributor-of-the-year-data
Connect 1:1 Meeting here: https://topmate.io/shaurya
Data Science Book Recommendations:
[1] The Book of Why
[2] Naked Statistics
Also published here
