Paste your college essay into a chatbot and you will probably get a B+ and a compliment. Paste in a weaker essay and you might get nearly the same grade. Ask again tomorrow and the number moves.
We built Kolly's scorer to behave like an instrument instead. This post explains how it works and shows the test we ran to check that it actually does.
The test
We took the same frontier AI model that powers Kolly (the same family behind Claude) and graded 44 essays two ways. One way was a plain chatbot-style prompt: read the essay, give it a score out of 100 and a letter grade. The other way was Kolly's full scoring system. The model and the essays were identical, so any difference between the two reads comes from the method.
The essays were chosen so we would know the right answer in advance. Eighteen were from College Essay Guy's collection of outstanding Common App essays, real essays by real students who got into selective schools, picked as exemplars by one of the most respected voices in admissions. Twelve were weak essays, each one read and confirmed as genuinely weak by a person before any scoring ran. The rest came from our internal calibration set. We also verified that none of the test essays appeared anywhere in Kolly's reference material, so the system could not recognize an essay it had already seen.
What a plain chatbot read looks like
The plain prompt did fine at the easy part. Show it a brilliant essay and a broken one and it puts the brilliant one higher.
The trouble starts where students actually live. It squeezed all eighteen famous essays into a 20 point band, between 70 and 90, which works out to three letter grades. The best of the verified weak essays got a 68. So the gap between a weak draft and a nationally celebrated essay came down to two points. And the number would not hold still: the same weak essay, graded three times with identical settings, came back 45, 48, and 58. A 13 point swing on the same words.
That is the read you are getting when you paste a draft into a general chatbot. It sounds authoritative. It just does not have anything to measure against.
How Kolly scores instead
Four things separate a Kolly read from that. We will describe what they do rather than publish the recipe, for the obvious reason.
Every essay is scored against real accepted essays. Kolly's corpus holds 500+ essays that earned admission to schools like Stanford, Harvard, MIT, and Penn. A scoring run reads your essay in the company of those essays, so a 4 means something fixed: this stands with work that got in. Your percentile is your position among accepted essays, computed from score distributions we rebuild whenever the system changes.
Six dimensions, each tied to evidence. Hook, theme, detail, authenticity, reflection, and writing are scored separately, and each score has to be backed by lines quoted from your actual essay: the values you showed through action, the insights you earned, the moments of real vulnerability. A high reflection score cannot exist without the reflective sentences to prove it.
The system is required to find your weakest dimension. Internal rules force real differentiation between dimensions, because a grader that hands back six nearly identical numbers has not read carefully. In our test, this focus was consistent: Kolly kept pointing at detail and reflection as the places drafts fall short, the same two weaknesses advisors like College Essay Guy name most often. A plain prompt asked the same question scattered its answers almost at random.
Your score stays put. In the test, Kolly's run to run variation stayed within 0.4 on a 5 point scale, roughly a tenth of the chatbot read's swing relative to its scale. And in the product, your score is saved with your exact draft, so the same essay is never quietly re-judged.
What the numbers came out to
Kolly separated the accepted essays from the weak ones with an AUC of 0.98, which means a randomly chosen accepted essay outscored a randomly chosen weak one 98 times out of 100. The famous essays landed where curated exemplars should land: most of them read Strong or Solid on our scale, spread across a range instead of piled at the ceiling, with the strongest at the 81st percentile of accepted essays.
The spread matters more than the praise. A grader that scores everything 90 can never tell you whether your revision worked. Kolly's percentile moves when the essay improves, because it is measured against a fixed reference class rather than a model's mood that day.
The fine print
The test ran on July 14 and 15, 2026, on pinned model versions, with the analysis code written and locked before any scoring started. Sample sizes were modest, 18 accepted and 12 weak essays, so we report the results as ranges rather than decimals carved in stone. We did not test other companies' products, only the method difference on one model, and the full run cost about $15 in model usage.
If you want to see what a read looks like on your own draft, the essay reviewer runs one free, and every score comes back with the percentile, the six dimensions, and the one fix to make first.
