I once heard a marketer teach the following strategy:
- Create a Video Views campaign and have people watch 95% of a long video for pennies on the dollar.
- Retarget this audience with conversion ads selling a product.
- Collect the đź’°đź’°đź’°
I tried it and it looks like magic!
Thousands of people “watch” a long video about your product or niche.
But even though I am a big fan of Meta Ads, it sounded too good to be true…
Machine learning is only as good as the data you feed it. Models LOVE to find a shortcut to maximize their rewards.
For ThruPlay Video Views campaigns, a reward is finding someone to watch 15 seconds of a video.
But when you have billions of users accessing your platform every day, your algorithm can quickly learn who is most likely to forget a tab open with a video playing in the background, or leave a video playing on their phone unattended.
So it left me thinking…
Are thousands of people really watching 95% of a long, technical video about machine learning, on social media?
I Designed The Following Experiment To Find Out
Everything italicized is taken from Meta’s tooltips and the Business Help Center.
It’s well-known that the first 3 seconds of a video ad can make or break it.
Companies like Harmon Brothers spend money testing multiple 3-second intros and measuring the thumb-stopping ratio: the number of 3-second video views divided by the number of impressions.
A higher ratio means the video is more engaging and likely to generate more conversions.
If people don’t even watch the video, they are unlikely to click and accept your offer.
Knowing that, what is the most boring video you can create? A long, black screen video without audio.
If this video shows up in your feed unsolicited, as most ads do, you will scroll past it without even noticing.
If you are so bored that you decide to investigate it, you will skip around to see if there is any content after the black screen.
So let’s create a campaign to advertise a boring video that no one has a sane reason to watch!
I used FFMPEG to generate a 60-minute, 720p video with no audio.
After that, I created a Video Views campaign optimizing for ThruPlays.
What is considered a ThruPlay?
“ThruPlay is an optimization and billing option for video ads through the Video Views objective.
ThruPlay allows advertisers to optimize and choose to pay only for ads that are played to completion for videos shorter than 15 seconds.
For longer videos, ThruPlay optimizes and bills for ads that are played for at least 15 seconds.”
It doesn’t mean 15 continuous seconds. Reading further it states:
“For longer videos, you pay when someone watches at least 15 unique seconds of your video.”
My understanding is: if the user skips around and the sum of watched segments is 15 or more seconds, it will compute as a ThruPlay.
It’s fair to expect a non-zero amount of ThruPlays because of unattended devices and skippers.
I am more interested in video views that reach 95% of the full length of the video.
I picked broad targeting. Anyone 18-65 in Brazil. The placement was Facebook video feed only, as this is the best place to get people to watch a long video on Facebook for the least cost.
To avoid people that know me skipping around thinking it’s a real ad, I excluded all my retargeting audiences: anyone that visited my website in the last 180 days or engaged with my Instagram/Facebook pages in the last 365 days.
I uploaded the video and didn’t write any Primary Text. This is the Ad Preview.
I planned to run it for 7 days, but after 3 days it had enough data to answer my question…
Results: How Many People Watched A 60-Minute Black Screen Video Without Sound?
The campaign had 11,327 impressions, of which 11,033 were on Desktop. The reach was 10,043.
Reach is “the number of people who saw your ads at least once.”
“An impression is counted as the number of times an instance of an ad is on screen for the first time. … Impressions aren’t counted if they come from invalid traffic we detect such as from non-human sources (such as a bot).”
Bot detection methods are not perfect but I trust Meta to remove 99.9%+ of bot traffic from the metrics.
We got 739 ThruPlays.
These are easy metrics to give an overview of the campaign. The ones that really interest me are the following…
Primary Metric: How Many Video Plays At 95%?
“The metric counts when the 95% point in your video was played. This may include people who played 95% of your video from the beginning, and may also include people who skipped to the 95% point and played your video from there.”
We had 36 video plays at 95%.
I assume these are not unique plays, but the same person can play to this point or skip multiple times.
Ideally this would be zero, but it’s reasonable to think that, in 10,000 people, 36 could skip around to see what was going on with the video…
I looked at another metric to see if these were skippers or something else…
Are People Skipping Around?
“The metric counts multiple types of clicks on your ad, including certain types of interactions with the ad container, links to other destinations, and links to expanded ad experiences.”
I assume this column includes clicks to skip around the video.
So it’s fine, right? People skipped to the end 36 times.
It would be if it was not for these other metrics…
Video Plays At 25%, 50%, And 75%
Watching 25% of this video means sitting for 15 minutes staring at a black screen. Unlikely.
So, if we assume every play is a skipper, we should see the number of clicks being at least the number of 25% video plays, but I would expect more.
Maybe Clicks (All) doesn’t count mobile taps? No problem, mobile taps are not relevant here as 97%+ of the views came from Desktop.
If people clicked around, and that metric records it, it must be higher. So skippers are off the table.
Average Play Time
“This metric is calculated as the amount of time spent playing your video (including replays for a single impression), divided by the total number of video plays. Because this metric includes replays for a video, the reported average video watch time could be longer than the length of the video.”
According to my previous campaigns, an average play time of 36 seconds is high.
Why We Got These Results?
This article is not to bash Meta Ads.
Making machine learning models behave well at scale is challenging, to put it mildly. I have a deep respect and admiration for the people that build and maintain these systems.
The most likely explanation is people forgetting their browser tabs open or being far from the computer. The video starts to play, but they don’t even notice it, as there is no audio in this case.
It’s easy to jump to the conclusion that it’s all bot traffic, but my experience as a data scientist taught me to be more skeptical and look for more answers.
There was a comment that was probably written by a fake account (a political comment about Brazil’s president in an election year).
This analysis is limited by the information I can get from the Ads Manager and the documentation. It can be an issue with how the metrics are calculated.
I know people can behave in very weird ways online, so I would love for anyone at Meta to point out valid reasons I am unaware of.
All we know with this experiment is that a 60-minute black screen video without audio can get reported video plays at 95% on Facebook.
Having accurate metrics for video campaigns will be even more important going forward, as Meta is betting heavily on short-form videos.
The Biggest Takeaway Is…
When running these campaigns and looking at the results, be aware that many of the plays you are seeing are not people really watching the video.
An idea to estimate how many of these views are real people watching the content is to run a lift test: run a legitimate video in parallel with an unwatchable video like the one here or a video unrelated to the brand (though people will still see you are the advertiser and may watch it if they like your brand).
How To Avoid Invalid Views?
I still think it’s useful to run a campaign for people to watch a video of your brand, but one thing I am noticing is that the higher you go on the Awareness-Consideration-Conversion funnel, the narrower must be your target.
Putting it another way, the easier the “conversion”, the more constraints you should put around ML-driven ad platforms.
I saw this recently when a colleague created a Performance Max campaign where the goal was to generate leads.
The conversion was a click to send a Whatsapp message to his business.
I was a bit surprised that he got real people messaging, as the ML model thinks it reached the goal just by getting clicks. Although he got 400+ leads, they were very low quality, with zero converting into paying customers.
The ideal solution would be to use enhanced conversions and optimize for people that convert down the funnel, but when we can’t do it (like in the case of video views campaigns), we have to find the balance with narrow targeting.
Make sure the people that will see the ad have qualified themselves through other behavior (like Interests, Lookalike audiences, or even both).
It’s very different from purchase campaigns. Purchases are a strong signal, but a needle in the haystack problem for machine learning. So it’s worth widening up the targeting.
A goal like a Thruplay is much easier to get, so we should make it harder for the model to achieve, if we want quality.
What About Other Platforms?
This problem likely happens in any video platform, but it would be interesting to quantify it.
Youtube is the one I am most interested in, as it’s a long video platform, but serves mostly in-stream ads.
It’s an idea for another time.
If you liked this experiment, please share the article. Thanks!
And check this cool article about using marketing mix models for online campaigns.