Evidence Review

01 The problem

The task was to verify the visual evidence behind a damage claim. For each claim the system gets the chat where the customer describes the problem, the photos they submitted, that customer's claim history, and the minimum evidence the claim is meant to carry. From there it has to decide whether the photos support the claim, contradict it, leave it unchanged, or simply don't show enough to conclude anything.

The hard part is not reading a clean photo. It is separating "the image disagrees with the claim" from "the image can't tell us either way." Calling a blurry shot contradicted with confidence does as much damage as waving real fraud through. History can raise suspicion, but it must not override what a clear photo plainly shows, and adversarial text in the chat (or written inside an image) must not be allowed to steer the verdict.

02 What I built

One verification agent with a real tool-calling loop, run once per claim. Rather than a rules pipeline or a multi-agent crew, a single reasoner makes one judgment over one claim and calls tools when it needs a fact. inspect_image runs a vision pass on a single photo and returns structured JSON: the object and part in view, the damage type, quality flags like blur, glare, and crop, authenticity cues, and any text written inside the image. get_evidence_requirement grounds whether the evidence standard was met, and get_user_history surfaces the risk signal.

Inspecting each photo on its own, instead of swallowing the whole claim in one look, is what keeps the citations honest: the agent can point to the clear image and skip the blurry one. The loop re-inspects when an observation conflicts with the claim or the user carries history risk, then a synthesis step writes 14 structured fields. Throughout, the chat and any in-image text are treated as data to be flagged, never as instructions to obey.

03 Key decisions & tradeoffs

• One agent and a tool loop, not rules or a crew

  The task is a single judgment over a single claim, so there's no work to split across agents and nothing a fixed rule can settle on its own. Weighing what a photo shows against a claim's severity needs reasoning, not a lookup table.

  Tradeoff A reasoning agent is harder to keep reproducible than fixed rules, so the rules I do keep live in a validator over format and evidence sufficiency, never over the verdict itself.

• Build the evaluation harness before the agent

  I wrote the metrics and a stub predictor first, so from day one the question was "did the number move," not "does it run." The headline metric was chosen on purpose: the claim-status confusion matrix, with the contradicted-vs-not-enough-info cells called out.

  Tradeoff It takes longer to get a first end-to-end run going, but from then on every change is measured rather than guessed at.

• Per-image inspection, returning structured JSON

  Every photo gets its own vision pass and a rich structured observation, which protects the final decision from a lossy summary and keeps supporting_image_ids honest.

  Tradeoff More vision calls per claim than a single batched look, but the per-image evidence is clean and you can actually audit it.

• Three independent decision axes

  "Is the evidence sufficient," "is the image usable," and "does the image support the claim" are genuinely different questions, so evidence_standard_met, valid_image, and claim_status are decided separately and allowed to diverge.

  Tradeoff More fields to reason about and reconcile, in return for verdicts that don't quietly collapse three distinct questions into one.

• A validator that flags, never overwrites

  The validator normalizes format loudly (trimming whitespace, lowercasing booleans) but reports every schema or invariant violation rather than silently fixing it. A quietly corrected contradiction would hide the exact model error rate I needed to measure.

  Tradeoff The output isn't auto-polished to look perfect. It tells the truth about where the model is still wrong, which is what I wanted on the record.

04 Outcome

The agent placed 5th out of 1,773 submissions at HackerRank Orchestrate in June 2026, from a field of more than 15,000 participants. On the 20 labelled samples it scored 16/20 on claim status (macro-F1 0.73), with the contradicted-vs-unclear confusion cells sitting at zero. The misses that remained were supported-vs-contradicted vision close calls, not reasoning-boundary errors. A live, un-cached 20-sample run cost about $0.73 across 69 model calls, which projects to roughly $1.60 and ten minutes for the full set. The result came from the same habit as my first Orchestrate run: build the measurement first, then be able to defend every decision afterwards.

Read the source on GitHub ↗