June 29, 2026 ยท Tags: mathematics, algorithms, machine learning, recommendation systems
Netflix knows what you want to watch. Spotify finds music you didn't know you'd love. Amazon surfaces products you were about to search for. Behind all of this is a single mathematical idea that has been refined for two decades.
The Core Problem #
Every recommendation system starts with the same setup. You have millions of users and millions of items, and a giant matrix recording who interacted with what. The problem is that this matrix is almost entirely empty. Netflix's famous dataset was 98.9% blank. The task is to fill in those blanks, or more practically, to predict which empty cells a user would rate highly.
The key insight, formalized during the 2006 Netflix Prize, is that this sparse matrix sits near a low-dimensional manifold. A user's taste can be captured by maybe 50 numbers: how much they like action versus drama, how they feel about subtitles, whether they prefer slow burns or fast pacing. Same for items. A movie is a point in that same 50-dimensional space. The predicted rating is just the dot product between the user's vector and the movie's vector.
This is matrix factorization, and it won the Netflix Prize. Simon Funk's version used stochastic gradient descent to learn the factors from only the observed ratings, sidestepping the need for a complete matrix. Koren's SVD++ improved on this by folding in implicit signals: the mere act of browsing a movie, even without rating it, carries information.
From Dot Products to Neural Networks #
Matrix factorization assumes a linear relationship between user and item vectors. But human preferences aren't linear. A person might love both dark comedies and children's films while hating everything in between. Neural collaborative filtering (He et al., 2017) replaced the dot product with a multilayer perceptron, letting the model learn these nonlinear patterns.
The catch, pointed out by Rendle et al. in 2020, is that a well-tuned matrix factorization model often beats the neural version. The architectural novelty doesn't automatically translate to better predictions. This remains one of the field's humbling lessons: simple baselines, properly tuned, are hard to beat.
Graphs Enter the Picture #
The newest major development treats the user-item interaction matrix as a graph. Users and items are nodes; interactions are edges. Graph neural networks like LightGCN (2020) propagate information along these edges, so a user's representation absorbs signals from items they touched, then from other users who touched those items, and so on. This captures transitive patterns that matrix factorization misses entirely.
The math is surprisingly clean. At each layer, a node's embedding becomes the average of its neighbors' embeddings, scaled by degree. After a few layers, each user carries information about their extended neighborhood. The prediction is still a dot product, but now between enriched representations.
What the Big Platforms Actually Do #
YouTube uses a two-stage funnel. The first stage frames recommendation as an extreme classification problem: given everything known about a user, which of millions of videos will they watch next? A sampled softmax makes this tractable. The second stage ranks the candidates using watch time rather than clicks, which discourages clickbait.
Amazon precomputes item-to-item similarity tables offline. The relatedness between two products is measured not by raw co-purchase counts but by a ratio that corrects for popularity bias. This is why you get genuinely relevant "customers also bought" suggestions rather than just bestselling items.
Spotify combines collaborative filtering with audio analysis. For new tracks with no listening history, a convolutional neural network processes the raw audio waveform and maps it into the same latent space as the collaborative embeddings. Cold start solved, at least partially.
TikTok takes the most aggressive approach. Internal research suggests only 30 to 50 percent of its recommendations exploit known preferences; the rest deliberately explore, testing new content categories on users. This keeps the feed from calcifying into a filter bubble.
The Hard Unsolved Problems #
Data sparsity means most user-item pairs have never been observed. The theoretical minimum for reliable matrix recovery is roughly O(nr log n) observed entries, but real systems operate far from these guarantees.
Filter bubbles are self-reinforcing: you click what the system shows, and the system shows what you click. Measuring this effect rigorously is difficult, and the literature is mixed on how severe it is across different platforms.
Fairness is a real problem. Popularity bias creates rich-get-richer dynamics. Exposure bias means that "no interaction" doesn't mean "disliked." It might mean "never shown." Inverse propensity scoring provides a mathematically unbiased correction, but calibrating the propensity estimates is its own challenge.
Why This Matters #
Every scroll through a feed, every "recommended for you" section, every autoplay queue is the output of these mathematical frameworks. The equations haven't changed much since the Netflix Prize era (dot products, regularization, gradient descent), but the scale and stakes are in a different league now. The next frontier isn't better prediction accuracy. It's formalizing diversity, fairness, privacy, and long-term user welfare as concrete optimization targets. The most important equations of the next decade might be the ones that constrain what these systems optimize, not the ones that make them more accurate.