Making a golf rules assistant cite rule numbers instead of inventing them
In a domain with a published, numbered rulebook, an ungrounded model answer is unacceptable — fluent wrong golf rulings cost strokes. RuleCaddie retrieves Rules of Golf passages first, answers from that text, attaches citations from the retrieval layer, and declines when the passages do not cover the question. Offline local rules data is not a fallback; it is the substrate.
A confident wrong answer costs a stroke
The reason this domain is unforgiving is that the answers are checkable and the consequences are immediate. If the app says free relief and the correct answer is one penalty stroke, someone signs an incorrect scorecard. There's no ambiguity to hide in afterward.
It's also a domain where wrong answers sound exactly like right ones. Golf rules have a strong internal grammar: relief areas, nearest point of complete relief, one club-length, penalty areas, provisional balls. A model that has absorbed that grammar can generate something perfectly shaped and entirely fabricated, including a rule number that doesn't say what it's claimed to say.
Which is the actual insight. The risk isn't that the model doesn't know golf. It's that the failure mode is invisible to the person least equipped to catch it, since anyone confident enough to detect the error wouldn't have needed to ask.
Ground it in the rulebook that already exists
The Rules of Golf are published by the R&A and the USGA. They're numbered, hierarchical, and stable across a revision cycle. That structure is a gift, and the correct move is to retrieve against it rather than to rely on what a model absorbed during training.
So a question retrieves the relevant rule text first, and the model answers from that text rather than from memory. The prompt is explicit that the retrieved passages are the source, that the answer must be supported by them, and that if they don't cover the question, that's the answer. The model's job shifts from recalling golf rules to reading a short set of passages and explaining what they mean for this specific situation, which is a task it's genuinely good at.
Retrieval also gives the app's knowledge a version. The Rules are revised on a cycle, and when they change we re-ingest rather than hope a model was retrained somewhere along the way. The answer a user gets is traceable to a specific edition of a specific document, which is the property that makes it defensible when someone in the group disagrees with it.
Chunking so the citation travels with the text
The retrieval unit is the rule and its sub-clauses, not a fixed number of characters. That sounds like an implementation detail and it decides whether the citations can be trusted at all.
Chunk by character count and you get passages that start mid-sentence and straddle the boundary between two rules, so the model is reading text belonging to 16.1b and text belonging to 16.1c with nothing marking the seam. Any citation it produces is then a guess about which rule the sentence it relied on came from, and a guess is precisely what we were trying to eliminate. Chunking on the document's own structure means every retrieved passage arrives already labeled with the rule it came from.
The corollary is that the citation shouldn't be something the model writes freely. It's attached to the passage by the retrieval layer and passed through, so the number in the answer is the number of the text that was actually read. A model asked to produce both the explanation and the reference has one more chance to produce a plausible reference. A model handed the reference has none.
Citations are a contract, not decoration
Every substantive answer cites the rule it comes from, and the citation is checkable. Rule 16.1 is a specific thing a user can look up on their phone while standing there, and the value isn't that the number looks authoritative, it's that a skeptical user can verify the answer in fifteen seconds.
That verifiability is what makes the whole thing honest. An assistant that can be checked is a different product from one that must be trusted, and in a domain with a public authoritative source there's no reason to build the second kind. It also changes what a wrong answer costs: a misapplied rule with a citation attached gets caught by the group, while an uncited one gets believed.
It has a useful side effect on the model's behavior too. Requiring a citation for each claim tends to suppress the confident freewheeling answer, because a claim that isn't in the retrieved text has nothing to point at.
Teaching it to decline
Plenty of real questions aren't answerable. Local rules vary by course. Committee decisions override defaults. Some situations genuinely depend on facts the app can't see, like exactly where the ball crossed into a penalty area, and the correct response is a question rather than a ruling.
So the system is built to say it doesn't know, and to say why: this isn't covered by the rules text, this depends on your course's local rules, this needs a detail you haven't given me. That's harder to get than it sounds, because the default behavior of a helpful model is to be helpful, and declining reads as a failure unless you design for it.
The framing that worked was making the boundary about the source rather than about confidence. Answer what the retrieved passages support. When they don't cover it, say so and name what's missing. That's a rule about text, which is checkable, rather than a rule about certainty, which isn't.
The offline database isn't the lesser product
Golf courses have terrible reception. That's not an edge case, it's the median condition on the back nine of a course cut into a hillside, and an app that needs a network to answer a rules question will fail exactly when it's needed.
So RuleCaddie ships a local rules database and a browse-and-search path over it that requires no network and no API key at all. If there's no connectivity, or no key configured, you still get to the rule. You lose the conversational interpretation of your specific situation, which is real, and you keep the authoritative text, which is the part that actually settles arguments.
The order of construction mattered here. We built the offline path first and layered the conversational one on top, which forced the rules data to be structured and searchable on the device rather than existing only as retrieval fodder for a server. Had we built the chat first, the offline mode would have been a stripped-down apology bolted on at the end, and it would have shown.
Questions
- Why can't you just prompt a model with golf rules knowledge?
- Models produce structurally plausible rulings, including fake or misapplied rule numbers. Wrong answers sound like right ones, and the person least able to catch the error is the one asking under time pressure on the course.
- How should citations be produced for a rules assistant?
- Chunk by the document's own rule/sub-clause structure so every passage arrives pre-labeled. Pass the citation through from retrieval rather than letting the model invent numbers. Citations are a contract users can verify on their phone in seconds.
- Why ship an offline rules database alongside the LLM path?
- Golf-course reception is often bad exactly when a ruling is needed. Local browse/search over authoritative text works with no network and no API key. You lose situational interpretation; you keep the text that settles arguments.