SoftSignal

Birthdays are the easiest thing to fake online. For the IEEE Young Professionals Industry Hackathon at the University of Calgary, Jericho Huelar and I took the age-assurance case: decide teen (13 to 17) or adult (23+) for 3,000 accounts using only what people write and how they use the app. No typed birthday, no photos. The output is a ranked likely-teen list for a trust-and-safety team, with one setting they control: the share of adults they accept being wrongly flagged.

The model is a two-level stack in scikit-learn. Level one is a TF-IDF text model over word and character n-grams with a logistic regression on top, with digits masked so an age typed in a post can't be read. Level two is a second logistic regression over that text score plus 16 activity and style columns, trained only on out-of-fold scores so no account is scored by a model that saw its own label. With all 2,100 training labels it catches 92% of teens on the 900 held-out accounts while flagging 17.1% of adults.

The harder part was the loop. The system starts with zero labels, and each round a simulated verifier reveals a small random audit sample. A challenger model refits on those labels and only goes live when it beats the current one on accounts neither has seen. Around the loop sit five Claude agents: a drift watcher, an error analyst, a loop controller, a verify-band triager, and an honesty auditor that checks every number on our slides against the results files. Each agent has a plain-rule fallback, and the console shows on every round whether it ran live, fell back, or was replayed.

In the recorded run the model went live at round seven, taking recall from 79.8% to 88.4% and the false-teen rate from 35.6% to 16.7% on held-out accounts. The honest finding is that the agents did not make the model more accurate. Over five seeds the agent-run loop ended level with the plain-rule loop, so the accuracy comes from the model and the agents earn their place by explaining, adapting, and auditing. The text is real 2004 blog posts, the activity data is synthetic, and the result is a likelihood for requesting verification, never proof of age.

Two details make the scores usable by a human reviewer. Because the second level is a linear model on standardized inputs, every score breaks down exactly into signed per-feature contributions, and the top three become the reason chips shown beside each account. The inputs are an allow-list of 16 activity and style columns asserted in code: age, gender, job, account age, and friend count are forbidden as features, and a test fails if one slips in.

The agents are fenced the same way. Every number in an agent's output has to appear in its input, a recorded output is only replayed when the round's input hash matches, and the controller's proposals are re-checked against the hold floor and promote gate in plain code. My side of the build was the drift watcher, the triager, the auditor, the replay layer, the per-account explanations, and a second console served from Python's standard library http.server next to the Streamlit one.