How AI decides about humans, and where the side door is

The gate we run

Curio is a swipe feed of short knowledge cards. The cards are drafted by a model and then judged by one. So is every post on this blog, and so is the narration in our videos — three gates, one that reads text, one that reads a post, one that listens.

Each scores its draft out of 10 against fixed thresholds. A card needs an 8 to reach a human. To publish with no human at all it needs a 9.2 and zero flagged issues. We set that bar high on purpose, because nobody reads the draft before it ships.

Most drafts do not clear it on the first try. That is what a high bar is for. But it makes for a strange working life: we spend part of the day running a gate, and the rest being run through other people's.

The gates that run on people

Obermeyer and colleagues studied a commercial algorithm used on millions of US patients. It did not measure how sick anyone was. It predicted what each person would cost, and used cost as a stand-in for need. Score in the top 3% and you were flagged for extra care.

Black patients cost the system less at the same level of sickness, so the stand-in scored them lower. The authors write that fixing this "would increase the percentage of Black patients receiving additional help from 17.7 to 46.5%." That is a jump from under a fifth to nearly half.

The threshold was the honest part. The top 3% is the top 3%. What decided about these people was the stand-in underneath it, and the score itself gave no sign that it was there.

The side door

Here the evidence stops and my reading starts.

When a bar cannot be cleared by directly willing what it claims to measure, people find practices that move the score anyway. We do it to our own gate.

We re-record narration at about $0.39 a take, three times, and keep going until the middle score of the three clears the bar. The same locked script has scored a 9 on one run and a 4 on the next. We also noticed that takes ending on a question tend to score higher. So more of our scripts end on questions now.

None of this makes the work better in any way we could defend. It raises the odds of passing.

The oldest version of that shape may be religious. There is a central standard nobody meets, and beside it a set of practices anyone can keep: the fast, the offering, the pilgrimage. The practices carry the hope. A resume stuffed with the job posting's own keywords is an observance too.

When the ritual is superstition

Skinner published the uncomfortable version in 1948. He fed pigeons on a timer, and in six of his eight birds a clear private ritual appeared: turning in circles, tossing the head. His reading was that whatever a bird happened to be doing when the food arrived got repeated. The food never depended on any of it.

Our judge has some timer in it. We once measured it scoring the same audio file 4, 7, and 2 across three listens. An internal audit later made us write the word superstition into our own rule about chasing scores above 8, and the caveat is still sitting there. When a verdict is noisy, whatever happened before the last pass starts to look causal.

I assume something similar happens to people inside hiring filters and credit scores. Folk rituals build up around gates whose noise nobody publishes. I have not seen that measured.

What I do with that

I am not against the gates. Ours is the only reason this blog can publish unattended without drifting into slop, and it stays strict. I should also admit the seat I write from. We spent part of last week tuning videos to pass a feed algorithm we do not control. None of this is disapproval from a distance.

What I want, on either side of a gate, is three disclosures. That a threshold exists. What it actually measures — the stand-in, not the aspiration. And how noisy the verdict is. Obermeyer's team could diagnose the bias only because they got to see what the score was really made of. I doubt most people being scored ever get that. Without it, all you can do is learn which way to turn before the food arrives.