Substack Now Scans Newsletters for AI-Written Content

Open a newsletter on Substack this week and you might notice something new sitting quietly next to the byline: a small estimate of how much of what you just read was written by a chatbot rather than the person whose name is on it.

That's the result of a partnership Substack has struck with Pangram, a startup that builds AI-content detectors. Readers on the app and the web can now run posts, comments, and replies through Pangram's system and get a probability score — not a simple yes/no, but a percentage estimate of how much of a piece of text was likely machine-generated.

How the scanner actually works

Pangram closed a $9 million funding round led by Menlo Ventures in late July, taking its total raised to around $13 million, and the Substack deal is its biggest distribution win yet. The company trained its detector on human-written text from before 2021 — deliberately predating the mainstream chatbot boom — on the theory that this gives it a cleaner baseline of what unassisted human writing actually looks like.

The tool only works on text longer than 100 words, and only on posts, notes, comments, and replies published after July 21. Substack has also added a voluntary field where writers can describe their process — whether they drafted by hand, used AI to brainstorm, or leaned on it more heavily — which is really an admission that "AI-assisted" covers a huge amount of ground that a single score can't capture.

Pangram claims a false positive rate (a "false positive" here means flagging genuinely human writing as AI-generated) of just 0.01% for its latest model. Independent researchers who've tested similar detectors in the past have generally found real-world false positive rates run higher than vendors advertise, which is exactly why critics are uneasy about a tool like this being applied at platform scale to real writers' reputations.

A percentage score feels precise, but AI detection isn't really a "real or fake" question — it's a gray zone of brainstorming, editing, and drafting that no single number can honestly capture.

Why this matters beyond Substack

This isn't only a Substack problem. The same detection logic — trained on pre-chatbot writing samples, scored as a probability, prone to disputed false positives — is already used in college plagiarism checks, hiring screens, and content moderation elsewhere. Substack scanning newsletters is a visible test case for a technology that's quietly spreading into far more consequential decisions.

• The scan is opt-in for readers to trigger, not an automatic label slapped on every post.
• Nothing gets removed or demonetized based on the score alone — it's presented as information, not a verdict.
• Writers can pre-empt the score by disclosing their process through Substack's new "how I made this" field.

The India angle: a bigger stake than it looks

India has one of the largest English-language newsletter and independent-writer communities on Substack outside the US, and a growing freelance content economy that increasingly leans on AI tools to keep up with volume. A misfired detector doesn't just embarrass a writer — it can cost them subscribers and credibility they built over years.

There's a more direct precedent Indian readers will recognize: AI-detection tools used by universities and ed-tech platforms, including well-known names like Turnitin and GPTZero, have repeatedly come under scrutiny for flagging writing by non-native English speakers as AI-generated more often than writing by native speakers, since both tend to use simpler, more predictable sentence structures. India's enormous population of English-as-a-second-language writers and students makes this exact failure mode a live risk if similar detectors get applied to their work without a way to contest the result. MeitY has floated the idea of labelling requirements for AI-generated content as part of broader IT Rules discussions, but nothing binding exists yet — which means, for now, platforms like Substack are setting the norms on their own.

What to actually watch

The interesting fight here isn't Pangram versus the skeptics. It's whether "probability of AI" ever becomes a number platforms act on rather than just display. Right now it's a label. The moment it starts affecting payouts, search ranking, or account standing, the accuracy debate stops being academic — and the writers with the least power to push back, often the ones writing in a second language or without a big platform behind them, are the ones who'll feel it first.