By Shreya S Ramanujam , Yongce Li , Afshine Amidi , Sina Ghiassian , Benjamin Klein . Introduction
Picture this — you’re home after a long day at work and can’t wait to catch up on Bridgerton. You log on to Netflix, search for “Bridgerton” and land on this screen:
If this sounds familiar, you already know what a synopsis is — it’s a short glimpse into what a viewer can expect to see in a show. We see two types of synopses here: a series-level synopsis describing the overarching plot, and multiple episode-level synopses describing each episode. A synopsis plays a very important role for us — it helps viewers decide whether a title is worth watching or not. Every synopsis needs to strike a balance: sharing just enough to give viewers a sense of what to expect, while leaving enough intrigue to draw them in. But of course, we can’t just write anything . “In the fog-drenched streets of Victorian London, Detective Daphne Bridgerton races to unmask a serial killer known only as the elusive Lady Whistledown.” Intriguing? Definitely. But it is most certainly not the premise of Bridgerton. Which leads us to the first (and most important) rule of writing a good synopsis: it must be factual . At Netflix’s scale, with thousands of titles, multiple episodes per season and multiple languages, fact-checking all of these by hand simply doesn’t scale. This is where our factuality service comes in: an automated way to judge whether a synopsis is factual or not. Our current factuality checker feeds a reasoning model everything we know about a title (like metadata and text assets) and asks it to judge synopsis factuality according to a rigorous rubric. There are two main limitations of this approach that we want to optimize:
High cost of long-context reasoning calls: Each decision requires expensive, long-context model calls to reasoning models, which doesn’t scale well when we want to check multiple titles, episodes, languages, and style variants. Misaligned rationales: The model’s explanations often diverge from our annotators’ reasoning, which makes its factuality verdicts less reliable. We want these rationales to be better aligned so that (1) the model can consistently flag nuanced errors, and (2) its explanations can be used downstream to automatically correct issues in the synopsis. Our new approach tackles both problems at once. On the cost side, it delivers a 52. 4% reduction per synopsis. On the quality side, it significantly improves both verdict accuracy and reasoning alignment: Recall–Fail ( “Of all truly non-factual synopses, how many does the system correctly reject?” ) rises from 0. 77 to 0. 86 , and the new factuality service’s explanations now show ~ 82% overlap with human rationales (up from ~ 69% ). In other words, it’s not just cheaper and more accurate — it’s also much better aligned with how our expert annotators actually judge synopses. So, what exactly does factuality mean? Let’s go back to the Bridgerton example. Say you see this synopsis about Bridgerton Season 1:
Synopsis A: “Bridgerton follows Daphne Bridgerton, the eldest daughter of the powerful Bridgerton family as she makes her debut onto Regency London’s competitive marriage market.” It’s easy to judge that this synopsis is factually correct — no wrong facts, and it gives the reader a clear, accurate glimpse of what to expect from the show. Let’s take another example:
Synopsis B: “Set in modern-day London, this gritty crime drama follows Detective Daphne Bridgerton as she hunts a serial killer while hiding her own criminal past.” Another easy one — the plot is so blatantly off the mark that it is definitely not factual. But what about this? Synopsis C: “Bridgerton follows the mysterious, sharp-tongued Lady Whistledown as she exposes the secrets of London’s high society through her scandalous gossip sheet.” Technically, every statement here is true — Lady Whistledown does indeed publish a gossip sheet, and if there’s one thing she is known for, it’s her scathing exposés. So then, is this a factual synopsis? Not quite! While technically true, it paints a misleading picture of the show. A new viewer might reasonably expect it to revolve around Lady Whistledown, rather than Daphne navigating the marriage market. This, too, is a type of non-factuality we want to catch. Lastly, here’s the trickiest of all:
Synopsis D: “When Daphne Bridgerton makes her society debut, she catches the eye of the aloof Duke of Hastings, and the two begin a passionate love affair that sets the whole ton abuzz.” Again, every statement is true, but the phrasing is misleading. A new viewer might assume they fall in love immediately, missing the fact that their relationship begins as a fake, mutually beneficial arrangement. This is another type of nuanced error our system should flag. So now that we have an idea of the kinds of errors we care about, the real question is: how do we build a system that is able to catch both glaring as well as subtle factuality violations? Current Service and Its Limitations
Our current factuality pipeline is a straightforward, holistic LLM call. We feed a reasoning model everything we know about a title (text assets, metadata, and a detailed factuality rubric) and ask it to judge the factuality of a synopsis. We repeat the call several times to reduce output randomness, and if any run flags the synopsis as non-factual, it fails. This approach has two main limitations:
High cost: Each judgment requires multiple (five, to be exact) calls to a large reasoning model over extremely long contexts (100k+ tokens), which does not scale well. Misaligned reasoning: The model can fail synopses for nitpicky reasons (e. g. , “This is a show about wonderful magic” gets failed because the characters never literally call the magic “wonderful”) while overlooking nuanced errors by convincing itself they “don’t really make the synopsis false.” This misalignment makes verdicts less reliable and rationales less useful for downstream correction. Optimizing the Holistic Pipeline
To understand exactly where we should optimize the current holistic pipeline, we study the current pipeline both qualitatively and quantitatively on a custom factuality benchmark that we built. This benchmark comprises synopses hand-labeled by human annotators as factual or non-factual, along with their rationales. This analysis reveals several key insights:
Insight 1: Do we really need five holistic calls? Let’s think back to why we added multiple holistic calls in the first place: to reduce sampling noise. In practice, though, we see an interesting dichotomy:
Blatantly true / false synopses (Bridgerton A, B): All runs agree on the verdict with near-identical reasoning. Nuanced errors (Bridgerton C, D): The model is much more likely to disagree with itself — some runs catch the error, some don’t. This is exactly where additional sampling helps. This suggests a simple optimization: A and B don’t really need all 5 calls. If we can somehow identify these “easy” synopses upfront, we can cut down on the number of calls for them, while still giving harder cases like C and D the extra calls they need. The easiest way to do this is to ask the model to output its confidence alongside its verdict and rationale. We can then separate “easy” (high confidence) synopses from “hard” (low confidence) ones and only send the hard cases through additional checks. Think of it like airport security: most bags go straight through, but those that trigger the alarm (low confidence) get extra screening. Implementing this easy fix makes our pipeline go from the current holistic model to a new “confidence gated” holistic model:
Insight 2: Five identical judges calls give us no new information
With confidence gating, our pipeline only escalates harder synopses to the full five holistic calls…