Understanding Video Recommendation Algorithms: A Complete Guide – ██FR█████ █INTELL███████████

This content originally appeared on Level Up Coding – Medium and was authored by Harish Siva Subramanian

Every time you open YouTube, TikTok, or Netflix, you’re experiencing one of the most sophisticated machine learning systems ever built. These recommendation algorithms don’t just guess what you’ll like — they analyze hundreds of signals to predict exactly what will keep you watching. Let’s understand how they work, step by step.

The Big Picture: What Problem Are We Solving?

Imagine you’re YouTube. You have 500 hours of video uploaded every minute and 2 billion users. How do you decide what to show each person?

You can’t show everything — that’s billions of possibilities. You need to quickly narrow down to a handful of videos that this specific user will love right now.

The Core Challenge: Given millions of videos and one user, predict which video they’re most likely to watch and enjoy.

The Three-Stage Architecture

Modern recommendation systems use a funnel approach with three stages:

Stage 1: Candidate Generation (Millions → Hundreds)

Goal: Quickly filter millions of videos down to a few hundred candidates

How: Use fast, simple algorithms to find potentially relevant videos

Example: You’ve been watching cooking videos. The system pulls all cooking videos, plus videos watched by people similar to you.

Stage 2: Ranking (Hundreds → Dozens)

Goal: Use sophisticated machine learning to predict exactly how much you’ll engage with each video

How: Calculate a precise “engagement score” using hundreds of features

Example: Of those 200 cooking videos, which one will you watch longest? The system predicts you’ll watch 80% of Video A but only 30% of Video B.

Stage 3: Re-ranking (Dozens → Final 10)

Goal: Apply business rules, ensure diversity, and optimize the final list

How: Adjust rankings to include variety, new content, and avoid over-representation

Example: You might love pasta recipes, but showing 10 pasta videos in a row is boring. The system mixes in desserts and appetizers.

Stage 1: Candidate Generation — Finding the Needle in the Haystack

Method 1: Collaborative Filtering — “People Like You Watched This”

Core Idea: If you and another user have watched similar videos in the past, you’ll probably like the same videos in the future.

Real-World Example:

You watched: Python Tutorial, Machine Learning Basics, Data Science 101
Sarah watched: Python Tutorial, Machine Learning Basics, Deep Learning Guide
The system notices you’re 80% similar to Sarah
Since Sarah also watched “Deep Learning Guide” and you haven’t, it recommends it to you

How It Works Mathematically:

Imagine a giant table where rows are users and columns are videos. Each cell contains how much that user engaged with that video (watch time percentage):

| Video A | Video B | Video C | Video D
-----------|---------|---------|---------|--------
You        |   90%   |   70%   |    ?    |    ?
Sarah      |   85%   |   75%   |   80%   |   0%
Mike       |   10%   |   15%   |   85%   |   90%

The algorithm finds patterns:

You and Sarah have similar scores for A and B → You’re probably similar
Sarah watched C (80%) → You’ll probably like C too
Mike has opposite tastes → Ignore his recommendations for you

Technical Detail — Matrix Factorization:

The system discovers hidden “taste dimensions.” Think of them as genres, but more abstract:

Dimension 1: “Educational Tech Content”
Dimension 2: “Entertainment Value”
Dimension 3: “Production Quality”

Your profile: High on Dimension 1 (80%), Medium on Dimension 2 (50%), Low on Dimension 3 (20%)

A video scoring high on Dimension 1 will be recommended to you.

Limitations:

Cold Start Problem: What if you’re a brand new user? No history = no recommendations
Popularity Bias: Popular videos get recommended more, creating a rich-get-richer effect

Method 2: Content-Based Filtering — “More Like What You Watched”

Core Idea: Recommend videos similar in content to what you’ve already enjoyed.

Real-World Example:

You watched: “Italian Pasta Recipes” (100% completion)
The system analyzes: Title has “Italian” + “Pasta”, Category is “Cooking”, Tags are “italian, cooking, pasta, dinner”
It finds similar videos with matching words: “Easy Italian Cooking,” “Pasta Carbonara Recipe,” “Italian Dinner Ideas”

How It Works — TF-IDF (Term Frequency-Inverse Document Frequency):

TF-IDF converts video descriptions into numbers representing “importance” of each word.

Example Calculation:

Video 1: “Python programming tutorial for beginners learn python”

Video 2: “Learn machine learning with python programming”

Video 3: “Italian cooking pasta recipe”

Step 1 — Term Frequency (TF): Count how often words appear

Video 1: “python” appears 2 times out of 7 words = 2/7 = 0.286
Video 2: “python” appears 1 time out of 6 words = 1/6 = 0.167

Step 2 — Inverse Document Frequency (IDF): Rare words are more important

“python” appears in 2 out of 3 videos → Common word → Lower importance
“pasta” appears in 1 out of 3 videos → Rare word → Higher importance

Step 3 — TF-IDF Score:

“python” in Video 1: 0.286 × 0.4 = 0.114 (medium importance)
“pasta” in Video 3: 0.25 × 0.9 = 0.225 (high importance)

Videos with similar TF-IDF patterns are considered similar.

Cosine Similarity — Measuring “Closeness”:

Each video becomes a point in multi-dimensional space. Similar videos are close together.

If Video A is at position [0.5, 0.8, 0.2] and Video B is at [0.6, 0.7, 0.3], they’re close → similar.

Example: You watched “Beginner Python Tutorial”

TF-IDF vector: [python: 0.8, beginner: 0.7, tutorial: 0.6, programming: 0.5]

Candidate videos:

“Advanced Python Programming”: [python: 0.9, advanced: 0.7, programming: 0.8, coding: 0.4] → Similarity: 85%
“Italian Cooking Basics”: [italian: 0.8, cooking: 0.9, recipe: 0.6, food: 0.5] → Similarity: 5%

The system recommends “Advanced Python Programming.”

Advantages:

Works for brand new videos (no viewing history needed)
Transparent — you can explain why a video was recommended

Limitations:

Can’t discover across categories (if you only watch cooking, you’ll never see tech videos)
Stuck in a filter bubble

Combining Both Methods

Modern systems use hybrid approaches:

Example Pipeline:

Get 50 candidates from collaborative filtering (people like you)
Get 50 candidates from content-based filtering (similar to your history)
Remove duplicates → ~80 unique candidates
Pass to Stage 2 for detailed ranking

Stage 2: Ranking — The Brain of the System

Now we have 100–200 candidates. Which should be #1? Which should be #10?

Feature Engineering — Teaching the Algorithm What Matters

The system looks at hundreds of features across three categories:

A. User Features — Who Are You?

1. Watch History Patterns

Example:

You’ve watched 50 videos in the past week
Average watch time: 8 minutes per video
You typically watch 75% of each video before leaving

What this tells the system: You’re an engaged user who prefers mid-length content and usually finishes videos.

2. Time-Based Behavior

Example:

Weekday mornings (7–9 AM): You watch 5-minute news videos
Weekday evenings (7–10 PM): You watch 20-minute educational content
Weekends (2–6 PM): You watch 45-minute documentaries

What this tells the system: Right now it’s 8 PM on Tuesday → Recommend 20-minute educational videos.

3. Category Affinity

Example: Your watch history breakdown:

Technology: 60% of videos
Science: 25% of videos
Cooking: 10% of videos
Sports: 5% of videos

What this tells the system: Heavily weight tech videos, moderate weight to science, minimal weight to cooking.

4. Recent Activity

Example:

Last 24 hours: 10 videos watched
Last week: 35 videos watched
Last month: 80 videos watched

What this tells the system: You’re currently very active → Recommend more content, including new discoveries.

B. Video Features — What Is This Video?

5. Engagement Metrics

Example Video Stats:

Total Views: 100,000
Likes: 8,000 (8% like rate) ← Very good! Average is 3–5%
Comments: 1,200 (1.2% comment rate)
Total Watch Time: 500,000 minutes
Average View Duration: 5 minutes (out of 8-minute video = 62.5% completion rate)

What this tells the system: This is high-quality content that keeps people engaged.

6. Video Age (Freshness)

Example:

Video A: Uploaded 2 hours ago, 1,000 views
Video B: Uploaded 2 years ago, 1,000,000 views

What this tells the system:

Video A is trending rapidly (new but already getting views) → Boost it
Video B is evergreen (old but still popular) → Keep recommending it

7. Video Length

Example:

You’re on mobile at 11 PM
Video A: 45 minutes long
Video B: 8 minutes long

What this tells the system: You probably won’t start a 45-minute video right now → Recommend Video B.

C. User-Video Interaction Features — How Do YOU Relate to THIS Video?

8. Category Match

Example:

Your top category: “Technology” (60% of your views)
Candidate Video A: Category = “Technology”
Candidate Video B: Category = “Sports”

Score:

Video A gets +0.6 category affinity bonus
Video B gets +0.05 category affinity bonus

9. Creator Relationship

Example:

You’re subscribed to “Tech Explained” channel
You’ve watched 15 of their last 20 videos
Candidate video is from “Tech Explained”

What this tells the system: Strong affinity → Major boost in ranking

10. Tag Overlap

Example: Your recent videos had tags: [python, programming, tutorial, beginner, coding, machine-learning]

Candidate Video A tags: [python, advanced, programming, AI] → 2 matching tags Candidate Video B tags: [cooking, recipe, italian, dinner] → 0 matching tags

Video A gets higher score due to tag overlap.

The Machine Learning Model — Putting It All Together

Goal: Predict “engagement score” (0–1) representing how much of the video you’ll watch.

Example Feature Vector for One Prediction:

User Features:
- avg_watch_time: 8.5 minutes
- total_videos_watched: 150
- recent_activity: 10 videos (last 24h)
- num_subscriptions: 12
- hour_of_day: 20 (8 PM)
- day_of_week: 2 (Tuesday)

Video Features:
- video_duration: 12 minutes
- video_age_days: 3
- like_rate: 0.08 (8%)
- comment_rate: 0.012 (1.2%)
- completion_rate: 0.65 (65%)
- total_views: 100000

User-Video Features:
- category_affinity: 0.6 (60% of your videos are this category)
- subscribed_to_creator: 1 (yes)
- tag_overlap: 3 matching tags
- watched_creator_before: 1 (yes)

Machine Learning Model (Gradient Boosting Decision Trees):

The model learns patterns like:

IF category_affinity > 0.5 AND subscribed_to_creator = 1 THEN predicted_engagement = 0.85
IF video_duration > 30 minutes AND hour_of_day > 22 THEN predicted_engagement = 0.25
IF completion_rate > 0.7 AND tag_overlap > 2 THEN predicted_engagement = 0.78

Example Predictions:

Video A (Tech tutorial from creator you follow):

Predicted engagement: 0.87 (87% watch time)
Interpretation: You’ll likely watch 10.4 minutes of this 12-minute video

Video B (Sports highlight from unknown creator):

Predicted engagement: 0.15 (15% watch time)
Interpretation: You’ll probably skip after 1–2 minutes

Ranking: Video A ranked #1, Video B ranked #47

Training the Model — Learning from History

Training Data Example:

Past Interactions:
1. User 123 + Video "Python Basics" → Watched 90% (positive example)
2. User 123 + Video "Advanced Physics" → Watched 5% (negative example)
3. User 123 + Video "Cooking Pasta" → Watched 85% (positive example)

The model learns:

User 123 loves Python content (90% engagement)
User 123 doesn’t like Physics (5% engagement)
User 123 enjoys some cooking content (85% engagement)

After training on millions of examples, the model recognizes patterns and can predict engagement for new user-video pairs.

Stage 3: Re-ranking — The Final Polish

You have videos ranked by predicted engagement. But we need to fix a few issues:

Problem 1: No Diversity

Example Before Re-ranking:

Python Tutorial Part 1 (score: 0.92)
Python Tutorial Part 2 (score: 0.91)
Python Tutorial Part 3 (score: 0.90)
Python Functions Explained (score: 0.89)
Python Lists Tutorial (score: 0.88)

Issue: All Python! Boring and creates filter bubbles.

Re-ranking Solution — Diversity Penalty:

As you select videos for the final list, penalize videos from over-represented categories:

Selection Process:

Select #1: “Python Tutorial Part 1” (Python count: 1)
Select #2: Skip “Python Tutorial Part 2” (penalty for duplicate category), choose “Web Design Basics” instead
Select #3: “Python Tutorial Part 2” (Python count: 2, acceptable now)
Select #4: Skip Python videos (too many), choose “SQL for Beginners”

Result: Mix of Python, Web Design, SQL → More interesting!

Problem 2: No New Content

Example: All top videos are from 6+ months ago (proven popular, safe bets)

Re-ranking Solution — Freshness Bonus:

Give bonus points to recently uploaded videos:

Video uploaded today: +0.15 bonus
Video uploaded this week: +0.10 bonus
Video uploaded this month: +0.05 bonus
Video older than 3 months: +0.00 bonus

Example:

Video A: Base score 0.75, uploaded 6 months ago → Final score: 0.75
Video B: Base score 0.70, uploaded yesterday → Final score: 0.70 + 0.15 = 0.85

Video B moves ahead of Video A!

Problem 3: Popularity Bias

Example:

Video A: 10M views, 5% like rate (500K likes)

Video B: 1K views, 12% like rate (120 likes)

Raw ML model might favor Video A (social proof), but Video B has better engagement rate.

Re-ranking Solution — Exploration vs. Exploitation:

Exploitation (90% of time): Show videos with highest predicted scores (safe bets)
Exploration (10% of time): Show promising videos with fewer views (take a risk)

Example: 1 out of every 10 recommendations is a “wildcard” — a video with <10K views but showing strong early engagement.

Why This Matters:

Gives new creators a chance
Helps you discover hidden gems
Prevents the same popular videos from dominating forever

Key Metrics: How We Measure Success

1. Click-Through Rate (CTR)

Formula: (Clicks ÷ Impressions) × 100

Example:

Video thumbnail shown 10,000 times
Clicked 800 times
CTR = (800 ÷ 10,000) × 100 = 8%

What It Means:

3–5% CTR: Average
8–10% CTR: Great thumbnail/title
15%+ CTR: Viral or highly targeted

Why It Matters: High CTR means the recommendation was appealing enough to click.

2. Average View Duration (AVD)

Formula: Total Watch Time ÷ Total Views

Example:

Video is 10 minutes long
1,000 people watched it
Total watch time: 6,500 minutes
AVD = 6,500 ÷ 1,000 = 6.5 minutes (65% completion)

What It Means:

20–40% AVD: Poor content or misleading title
50–60% AVD: Good content
70%+ AVD: Excellent content that holds attention

Why It Matters: High AVD means the recommendation was actually good, not just clickbait.

3. Session Duration

Definition: How long a user stays on the platform after a recommendation

Example: After clicking recommended Video A:

User watches Video A: 8 minutes
Then watches 3 more videos: 25 minutes
Total session: 33 minutes

After clicking recommended Video B:

User watches Video B: 2 minutes
Closes the app
Total session: 2 minutes

Result: Video A is a much better recommendation! It kept the user engaged longer.

Why It Matters: Platforms care about total time on platform, not just individual video performance.

4. Recommendation Coverage

Formula: (Unique Videos Recommended ÷ Total Videos in Catalog) × 100

Example:

Catalog has 1,000,000 videos
In one day, system recommended 50,000 unique videos
Coverage = (50,000 ÷ 1,000,000) × 100 = 5%

What It Means:

1–3% coverage: Too focused on popular content
5–10% coverage: Good balance
20%+ coverage: Possibly too random

Why It Matters: Low coverage means small creators never get discovered. High coverage means recommendations aren’t personalized enough.

5. User Retention (30-Day)

Formula: (Users Active After 30 Days ÷ New Users) × 100

Example:

10,000 new users signed up
After 30 days, 4,000 are still active
Retention = (4,000 ÷ 10,000) × 100 = 40%

What It Means:

20–30% retention: Poor recommendations, users leave
40–50% retention: Good recommendations
60%+ retention: Excellent recommendations, users hooked

Why It Matters: Good recommendations keep users coming back.

Real-World Example: Your YouTube Session

Let’s walk through exactly what happens when you open YouTube at 8 PM on a Tuesday:

Your Profile (Invisible to You, Known to the Algorithm)

Watch History:

Last 7 days: 25 videos, mostly tech tutorials
Average watch time: 12 minutes per video
Top categories: Technology (70%), Science (20%), Gaming (10%)
Subscribed to: 15 tech channels
Typical viewing time: 7–10 PM on weekdays

The System’s Understanding: “This is a tech enthusiast who prefers educational content in the evening. Active user with strong engagement.”

Stage 1: Candidate Generation (Milliseconds)

Collaborative Filtering generates 100 candidates:

Found users similar to you (also watch tech tutorials)
Those users also watched: “Docker Tutorial,” “System Design Interview,” “AWS Basics”
50 candidates from CF

Content-Based Filtering generates 80 candidates:

You recently watched “Python Flask Tutorial”
Similar videos by tags/content: “Django Tutorial,” “FastAPI Guide,” “Python Web Frameworks”
30 new candidates (50 were duplicates)

Total unique candidates: 80 videos

Stage 2: Ranking (Milliseconds)

Machine learning model scores each video:

Candidate A: “Docker for Beginners”

Category affinity: 0.7 (you watch 70% tech)
Video completion rate: 68% (good engagement)
Video age: 2 days (fresh)
Duration: 15 minutes (matches your typical watch time)
Creator: You’re subscribed ✓
Predicted engagement: 0.88

Candidate B: “Advanced Kubernetes”

Category affinity: 0.7
Video completion rate: 55% (harder content)
Video age: 30 days
Duration: 45 minutes (you rarely watch long videos at night)
Creator: Unknown
Predicted engagement: 0.42

Ranking: Docker video scores much higher and ranks #2. Kubernetes video ranks #34.

Stage 3: Re-ranking (Milliseconds)

Adjustments:

Diversity check: Top 5 were all Docker/containerization videos → Move #6 (different topic) to #4
Freshness boost: #7 was uploaded 3 hours ago and trending → Move to #3
Exploration slot: #10 becomes a wildcard — a small channel with high early engagement

Final Top 10:

“Docker for Beginners” (0.88) — Perfect match for your interests
“VS Code Extensions Every Developer Needs” (0.85) — Different topic for diversity
“GitHub Actions in 10 Minutes” (0.72 + 0.15 freshness = 0.87) — New trending video
“System Design Fundamentals” (0.82) — Moved up for diversity
“Python Testing with Pytest” (0.81) — Content-based recommendation
“Linux Command Line Basics” (0.79) — From similar users
“Building REST APIs” (0.78) — Related to recent watches
“Database Indexing Explained” (0.75) — Different but relevant
“Git Workflow Best Practices” (0.74) — From subscribed channel
“Intro to Rust Programming” (0.45 + exploration) — Wildcard for discovery

You click on #1, watch 85% of it, and the cycle repeats!

Common Challenges and Solutions

Challenge 1: Cold Start Problem

Problem: New user with no watch history. What do you recommend?

Solutions:

Onboarding questions: “What are you interested in? Select 5 categories.”
Popular content: Show trending videos in their selected categories
Geographic defaults: Show popular content in their country/language
Fast learning: After just 2–3 videos, system has enough signal

Example:

New user from US, age 25–34, signs up
System shows: Top 10 trending videos for US viewers age 25–34
User watches “JavaScript Tutorial”
Next recommendation: More programming content (learned from 1 video!)

Challenge 2: Filter Bubble

Problem: User only sees Python content, never discovers they might love woodworking

Solutions:

Forced diversity: 10–20% of recommendations from different categories
Serendipity injection: Occasionally show highly-rated content from random categories
Category exploration rewards: If user engages with off-category content, note it

Example:

You watch 100% tech videos
System shows you “Beginner Woodworking” (exploration)
You watch 90% of it
System notes: “User is open to new categories!” → Recommends more diverse content

Challenge 3: Clickbait vs. Quality

Problem: Clickbait titles get high CTR but low watch time. Should we recommend them?

Solution: Multi-objective optimization — Optimize for BOTH clicks AND watch time

Formula:

Score = 0.3 × CTR + 0.7 × Watch Time

Example:

Video A: 15% CTR, 20% avg watch time → Score = 0.3(0.15) + 0.7(0.20) = 0.185
Video B: 8% CTR, 75% avg watch time → Score = 0.3(0.08) + 0.7(0.75) = 0.549

Video B wins despite lower CTR because it provides actual value!

Thank you for reading my article!!

If you like the article and would like to support me, make sure to:

Clap for the story (50 claps) to help this article be featured
Follow me on Medium
View more content on my medium profile
Follow Me: LinkedIn | GitHub

Understanding Video Recommendation Algorithms: A Complete Guide was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.

This content originally appeared on Level Up Coding – Medium and was authored by Harish Siva Subramanian