AI agent as internal policy Q&A bot — saving 5-10 hrs/week

When a Head of Ops at a 140-person company told me she was getting interrupted 30+ times a day with "what's our PTO carryover policy?" — and her HR manager was getting another 30 — we built her policy Q&A bot in four days. The hardest part wasn't the AI; it was finding which version of the handbook was actually current.

What this agent actually is

A policy Q&A bot is a retrieval-augmented LLM (RAG) wired to a small library of trusted documents — your employee handbook, HR policies, IT/security policy, expense policy, leave/PTO rules, etc. An employee asks in plain language. The agent retrieves the relevant passages and answers, citing the document and section.

Definition: RAG (Retrieval-Augmented Generation) — an architecture where, before the LLM answers, a retrieval step pulls the most relevant passages from your document library into the prompt. The model "reads" your docs, doesn't memorize them.

This is not an HR replacement. It's an interruption-killer for the 80% of questions whose answer is literally written somewhere — but no employee can find it. HR/Ops handles the 20% that's edge-case or sensitive.

Why this is the right "first agent" for SMB

Most SMBs reach for a customer-facing agent first (sales, support) and stumble into hard problems: hallucinations facing real customers, escalation gaps, brand exposure. A policy Q&A bot:

• Faces internal users (lower stakes, faster forgiveness on errors).
• Has a closed knowledge corpus (your docs, finite).
• Has a clear "I don't know" / escalation behavior (route to HR/Ops Slack).
• Doesn't require complex tool-use or multi-step reasoning.

It's the agent equivalent of "training wheels" — high real value, low risk profile, exactly the right starting complexity.

The 1-week build (no enterprise SSO required)

Day 1 — Source docs audit: which version of each policy is actually current?
        Answer in writing. Park outdated versions in an /archive folder.
Day 2 — Pick the platform. SMB-friendly options:
        • Microsoft Copilot Studio (if you have M365)
        • Google NotebookLM Enterprise (if Workspace)
        • OpenAI Custom GPT or Claude Project (small teams, ≤50 users)
        • A managed RAG platform (e.g., Glean, Sana, Coveo) for larger SMBs
Day 3 — Ingest. Upload the cleaned doc set. Configure citations on.
Day 4 — Tune retrieval. Try 20 representative real questions.
        Where it whiffs, fix the source doc (chunking is usually the issue).
Day 5 — Define escalation. "If user asks X, route to HR Slack channel."
        Add the disclaimer: "this is policy info, not legal/HR advice."
Day 6-7 — Soft launch with 1 department. Collect every "wrong answer" report.
          Iterate retrieval. Then expand.

You will not have a perfect bot on day 7. You will have a 70%-good bot on day 7 that becomes 90%-good in 30 days as you fix the questions it misses.

Tuning retrieval — the only step that matters

If your bot is wrong, retrieval is almost always the cause, not the model. Three things to tune:

1. Chunking — how documents get sliced before embedding. Too-large chunks dilute relevance; too-small chunks fragment context. Start with ~500-800 tokens per chunk, with 100-token overlap.
2. Metadata — every chunk should be tagged with document name, section, and effective date. Filter by "current" before retrieval.
3. Question reformulation — when a user asks "can I work from home Friday?", the system should rewrite that into "remote work policy" terms before retrieving. Most modern platforms do this automatically; check it's on.

Definition: Embedding — a numerical representation of a chunk of text that captures its meaning. The retrieval step compares the user's question embedding to all chunk embeddings and pulls the closest matches.

Tool tip (Course for Business): Building this internally is the perfect "ship in week 1" project for the 5-day program. We frame it as Augment, don't replace — the bot doesn't replace HR; it removes the 20-times-a-day interruption that was burning HR's deep work. The Friday cohort lab is where employees actually wire the agent to their team's policies. https://course.aiadvisoryboard.me/business

The escalation question — the difference between useful and dangerous

A naive bot tries to answer everything. A useful one knows what it doesn't know and routes.

Bake three escalation rules into the system prompt:

• "If the question involves a specific person's situation (medical, performance, conflict), do not answer. Route to HR with the question."
• "If retrieval confidence is below threshold, say 'I'm not sure — let me route this to the right person' and post to the HR Slack channel."
• "If the question touches legal or contractual obligations, add: 'this is policy info, not legal advice — verify with HR/Legal.'"

The Klarna walked-back-AI-CSR pattern is the cautionary tale here. Klarna pulled back its AI customer-service agent in 2025 after CSAT dropped — the gap was escalation, not the model. Build the escalation in from day 1.

Team scan (what AI champions report after week 1)

• Adoption: ~60-80% of staff use the bot at least once in week 1 if it's promoted in the all-hands.
• Top question categories: PTO/leave (40%), expense rules (20%), benefits (15%), IT/security (10%), other (15%).
• Saved time, illustrative range: 5-10 hours/week — split between HR (3-5 hrs), Ops/manager interruptions (2-4 hrs), and onboarding new hires (1-2 hrs).
• Biggest source of wrong answers in week 1: outdated documents in the corpus. Fix the docs, not the model.
• Top reason for non-adoption: "I didn't know it existed" — internal launch matters more than the build.
• Wins beyond Q&A: new-hire ramp time drops because the bot answers the "dumb questions" they're embarrassed to ask managers.
• Champion role: triage the bad-answer reports weekly, fix the underlying source doc.
• Surprise win: managers themselves use the bot (they don't remember every policy either).

Micro-case (what changes after 7-14 days)

A 180-employee professional services firm built the bot in a week using their existing M365 tenant + Copilot Studio. By day 10, the HR manager reported a meaningful drop in ad-hoc interruptions — particularly around expense policy and PTO carryover, which were 60% of her inbound. New hires going through week-1 onboarding asked the bot ~15 questions each before going to their manager. By week 4, the firm had a "wrong-answer log" of 11 questions; 9 traced to outdated docs, 2 to retrieval tuning. The bot handled roughly 75% of policy questions without escalation.

Note on this case: This example is illustrative — based on typical patterns we observe with companies of 30-500 employees, not a single named client. Specific numbers are rounded approximations of common ranges, not guarantees.

Tool tip (Course for Business): A policy Q&A bot is the textbook Shoulder-to-Shoulder project for week 1 — every department lead pairs with an AI champion to ingest their policies, tune retrieval, and ship. By Friday, every department has its own slice of the bot working. 6-week program lets the wrong-answer iteration happen in cohort review, not in isolated effort. https://course.aiadvisoryboard.me/business

FAQ

Don't I need enterprise SSO and a data team? No. SMBs successfully ship policy Q&A bots on Microsoft Copilot Studio, Google NotebookLM, or even an OpenAI Custom GPT for small teams. Enterprise SSO is helpful at 200+ employees; below that, your existing M365 or Workspace login is enough.

What if our policies are scattered across SharePoint, Google Drive, and a wiki? Day-1 audit fixes that. Centralize the canonical version of each policy in one location, archive the rest. The bot is only as good as the corpus.

What about HR-sensitive answers (harassment, performance)? Hard escalation rules — never answer those. The bot says "this requires a person, here's how to reach them" and posts the question to the right HR channel.

Will employees trust an AI for policy answers? They trust it more than asking a manager who might be wrong. Citations matter — the bot must cite "Employee Handbook §4.2" so employees can verify. Without citations, trust collapses.

Is this an "AI agent" or just a chatbot? The line is blurry. With retrieval + escalation actions (post to Slack, create a ticket), it's an agent. Call it whatever your culture prefers; the architecture is the same.

What to do this week

If you have HR, Ops, or managers fielding the same 20 questions every week — your first agent should be a policy Q&A bot, and it can be live in a week. The hard part isn't the AI; it's the source-doc audit.

If you want every employee to ship their first AI automation in five days — book a 30-min call and we'll map your team's first week: https://course.aiadvisoryboard.me/business