Cast of Thousands

Document ranking · early access

Rank the whole pile. Defend the cut.

For grant rounds, fellowships, prizes and accelerator intakes with more submissions than reading hours. AI judges compare them two at a time and rank all of them, with honest ranges: what's clearly in, what's clearly out, and the close calls your people should read themselves. Don't take our word for it. Here's one we actually ran.

A real run · 20 November 2026 · not an illustration

We gave AI judges 30 Paul Graham essays and one brief: choose a reading list for first-time founders, with room for 8.5 are clearly on it. 9 are competing for the last 3 places. 16 are clearly off.

  1. Clearly on the list
  2. On the line: read these yourself
  3. Closest call Split with “Ramen Profitable”: the verdict flipped when the reading order swapped, so it counts as a tie.

    The judge's reason: “[Startups in 13 Sentences] covers a broad set of 13 practical principles (cofounders, launching fast, understanding users, customer service, spending little, avoiding distractions, not giving up), giving a first-time founder a wide range of actionable guidance.”

    Won 5 of 12 readings · Read the essay on paulgraham.com ↗

  4. Clearly off

Each bar is an essay's likely range after 180 pairwise readings; the dot is the best estimate and the thin line is the cut for 8 places. Where a bar crosses the line, the evidence can't separate it from its neighbours, so the board says so. Tap an essay to see its closest call and the judge's reason.

Judge
Claude Haiku (Anthropic)One model this time, to keep the demo cheap. We're building rounds that use a panel of several.
Readings
180Each essay met 6 others, and every pair was read in both orders.
Changed its mind when the order swapped
26% of pairsThose count as ties, not wins. It's why every pair is read twice.
Split-half agreement
0.61Rank each half of the readings separately and compare: 1 is identical, 0 unrelated. Moderate here; more readings raise it.
Longer essays favoured?
Somewhat: r = 0.37Longer essays tended to rank higher. Worth knowing, so we print it rather than hide it.

Paul Graham wrote the essays; we link to them on his site and don't reproduce them. He had no part in this and hasn't endorsed it. This is AI taste, not a verdict on the essays: one model, one brief, one run. The judge's reasons are its own words, trimmed; square brackets mark where we put in the essay's title for the judge's "Text A" or "Text B".

The entire brief the judges got

You are on a small committee choosing essays for a reading list given to people about to start their first company. You can include only a few. Which of the two essays would you rather put on the list? Judge: how useful and specific the advice is to a first-time founder, how likely it is to change what they actually do, and how clearly it is argued. Do not prefer an essay because it is longer or more famous. If both are strong, pick the one a first-time founder would miss more if it were left off.

Swap the essays for your applications.

Nobody can prove a right order for these essays, which is why we ran them in public. What the run shows is the machinery you'd use on your own pile: a brief in your own words, every pair read twice so reading order can't sway it, a reason for every call, and a range wherever the judges weren't sure.

On a real round you'd start by checking it: rank a round you've already decided, and see how far it agrees with your panel before you trust it with the next one.

What's in your pile?

Pick the closest. We're deciding what to build first, and this is how we find out.

Your pick is filled into the signup form at the bottom of the page, so you don't need to type it twice.

Before you take it to your board

The questions your committee will ask.

Straight answers, including the ones that aren't flattering yet.

Does it agree with human panels?

We haven't shown that yet, and we won't claim it until we have. Our first public test ranks past conference submissions and compares the result with the real accept and reject decisions. We'll publish it, including where it falls short.

For your own programme the test is more direct: run it on a round you've already decided. If it doesn't agree with your panel, you find out before it matters.

Who sees our submissions?

Only the AI model provider you choose. Today the prototype sends texts to Anthropic, for Claude models, or through OpenRouter for other models. The run above went only to Anthropic, and a round can be kept to a single provider. We won't train models on your documents.

The engine will be open source, so you can run it yourself with your own model keys and keep submissions within a provider you already have an agreement with. Before anyone uploads a real round, we'll publish how long anything is kept and how to delete it.

How do we defend a decision made with it?

It doesn't make the decision. Nothing is rejected automatically; it orders the reading. What you can show is the paper trail: the brief you wrote, every comparison with the judge's reason, a range for every submission, and checks on the judge itself, such as how often it changed its mind when the order swapped. Close calls are marked as close, so your people read them.

Is there a product? Who uses it?

A working prototype, which made the ranking above. No customers yet. We're looking for the first programmes to run a round with, starting with a back-test on one they've already decided.

What does it cost?

Our price isn't set. For scale, the AI model fees: the run above, 180 readings of full essays, would cost about $2 at OpenRouter's list price for Claude Haiku (we ran it on a subscription).

Readings grow in step with the pile, not with every possible pair: each submission meets a handful of others. So 500 submissions as long as these essays would be about 3,000 readings and roughly $34 in fees at the same rate, and less for shorter applications or more for a bigger panel. That's our arithmetic, not a quote. Tell us how big your pile is and what a round is worth to you.

Bring us your pile.

Join the waitlist and tell us what you'd rank. We plan to help early users run their first round, starting with a back-test on a round they've already decided.

We'll only email you about this project. Privacy.

For developers

Rank anything from code, or let your agent do it.

Model outputs, generated ideas, candidate copy, search results: if your pipeline generates many and needs the best few, it needs a judge. The ranking above came from our command-line prototype in a single command. Next: an API, an MCP server so agents can set up rankings themselves, and the open-source engine to run locally.

Only about this project. Privacy.