# AI agent as internal policy Q&A bot — saving 5-10 hrs/week

> How an SMB (no enterprise SSO required) bootstraps an AI agent that answers internal policy questions from its own handbook. RAG, retrieval tuning, escalation — the whole 1-week build.

- Author: Yaroslav Maxymovych (Founder & CEO, AI Advisory Board)
- Published: 2026-05-09
- Updated: 2026-10-07
- Source: https://aiadvisoryboard.me/blog/ai-agent-policy-qa-bot

When a Head of Ops at a 140-person company told me she was getting interrupted 30+ times a day with "what's our PTO carryover policy?" — and her HR manager was getting another 30 — we built her policy Q&A bot in four days. The hardest part wasn't the AI; it was finding which version of the handbook was actually current.

## TL;DR

- An internal policy Q&A bot — an AI agent that answers staff questions from your own handbook — is the highest-ROI first agent for a 30-500-employee SMB.
- Illustrative range: 5-10 hours/week saved across HR, Ops, and managers, mostly in interruption avoidance and faster onboarding.
- You don't need enterprise SSO or a data team to bootstrap one. You need clean source documents, retrieval tuning, and a defined escalation path.

## What this agent actually is

A policy Q&A bot is a retrieval-augmented LLM (RAG) wired to a small library of trusted documents — your employee handbook, HR policies, IT/security policy, expense policy, leave/PTO rules, etc. An employee asks in plain language. The agent retrieves the relevant passages and answers, citing the document and section.

> **Definition:** RAG (Retrieval-Augmented Generation) — an architecture where, before the LLM answers, a retrieval step pulls the most relevant passages from your document library into the prompt. The model "reads" your docs, doesn't memorize them.

This is _not_ an HR replacement. It's an interruption-killer for the 80% of questions whose answer is literally written somewhere — but no employee can find it. HR/Ops handles the 20% that's edge-case or sensitive.

## Why this is the right "first agent" for SMB

Most SMBs reach for a customer-facing agent first (sales, support) and stumble into hard problems: hallucinations facing real customers, escalation gaps, brand exposure. A policy Q&A bot:

- Faces internal users (lower stakes, faster forgiveness on errors).
- Has a closed knowledge corpus (your docs, finite).
- Has a clear "I don't know" / escalation behavior (route to HR/Ops Slack).
- Doesn't require complex tool-use or multi-step reasoning.

It's the agent equivalent of "training wheels" — high real value, low risk profile, exactly the right starting complexity.

## The 1-week build (no enterprise SSO required)

```
Day 1 — Source docs audit: which version of each policy is actually current?
        Answer in writing. Park outdated versions in an /archive folder.
Day 2 — Pick the platform. SMB-friendly options:
        • Microsoft Copilot Studio (if you have M365)
        • Google NotebookLM Enterprise (if Workspace)
        • OpenAI Custom GPT or Claude Project (small teams, ≤50 users)
        • A managed RAG platform (e.g., Glean, Sana, Coveo) for larger SMBs
Day 3 — Ingest. Upload the cleaned doc set. Configure citations on.
Day 4 — Tune retrieval. Try 20 representative real questions.
        Where it whiffs, fix the source doc (chunking is usually the issue).
Day 5 — Define escalation. "If user asks X, route to HR Slack channel."
        Add the disclaimer: "this is policy info, not legal/HR advice."
Day 6-7 — Soft launch with 1 department. Collect every "wrong answer" report.
          Iterate retrieval. Then expand.
```

You will not have a perfect bot on day 7. You will have a 70%-good bot on day 7 that becomes 90%-good in 30 days as you fix the questions it misses.

## Tuning retrieval — the only step that matters

If your bot is wrong, retrieval is almost always the cause, not the model. Three things to tune:

1. **Chunking** — how documents get sliced before embedding. Too-large chunks dilute relevance; too-small chunks fragment context. Start with ~500-800 tokens per chunk, with 100-token overlap.
2. **Metadata** — every chunk should be tagged with document name, section, and effective date. Filter by "current" before retrieval.
3. **Question reformulation** — when a user asks "can I work from home Friday?", the system should rewrite that into "remote work policy" terms before retrieving. Most modern platforms do this automatically; check it's on.

> **Definition:** Embedding — a numerical representation of a chunk of text that captures its meaning. The retrieval step compares the user's question embedding to all chunk embeddings and pulls the closest matches.

> **Tool tip (Course for Business):** Building this internally is the perfect "ship in week 1" project for the 5-day program. We frame it as **Augment, don't replace** — the bot doesn't replace HR; it removes the 20-times-a-day interruption that was burning HR's deep work. The Friday cohort lab is where employees actually wire the agent to their team's policies. https://course.aiadvisoryboard.me/business

## The escalation question — the difference between useful and dangerous

A naive bot tries to answer everything. A useful one knows what it doesn't know and routes.

Bake three escalation rules into the system prompt:

- "If the question involves a specific person's situation (medical, performance, conflict), do not answer. Route to HR with the question."
- "If retrieval confidence is below threshold, say 'I'm not sure — let me route this to the right person' and post to the HR Slack channel."
- "If the question touches legal or contractual obligations, add: 'this is policy info, not legal advice — verify with HR/Legal.'"

The Klarna walked-back-AI-CSR pattern is the cautionary tale here. Klarna pulled back its AI customer-service agent in 2025 after CSAT dropped — the gap was escalation, not the model. Build the escalation in from day 1.

## Team scan (what AI champions report after week 1)

- Adoption: ~60-80% of staff use the bot at least once in week 1 if it's promoted in the all-hands.
- Top question categories: PTO/leave (40%), expense rules (20%), benefits (15%), IT/security (10%), other (15%).
- Saved time, illustrative range: 5-10 hours/week — split between HR (3-5 hrs), Ops/manager interruptions (2-4 hrs), and onboarding new hires (1-2 hrs).
- Biggest source of wrong answers in week 1: outdated documents in the corpus. Fix the docs, not the model.
- Top reason for non-adoption: "I didn't know it existed" — internal launch matters more than the build.
- Wins beyond Q&A: new-hire ramp time drops because the bot answers the "dumb questions" they're embarrassed to ask managers.
- Champion role: triage the bad-answer reports weekly, fix the underlying source doc.
- Surprise win: managers themselves use the bot (they don't remember every policy either).

## Micro-case (what changes after 7-14 days)

A 180-employee professional services firm built the bot in a week using their existing M365 tenant + Copilot Studio. By day 10, the HR manager reported a meaningful drop in ad-hoc interruptions — particularly around expense policy and PTO carryover, which were 60% of her inbound. New hires going through week-1 onboarding asked the bot ~15 questions each before going to their manager. By week 4, the firm had a "wrong-answer log" of 11 questions; 9 traced to outdated docs, 2 to retrieval tuning. The bot handled roughly 75% of policy questions without escalation.

> **Note on this case:** This example is illustrative — based on typical patterns we observe with companies of 30-500 employees, not a single named client. Specific numbers are rounded approximations of common ranges, not guarantees.

> **Tool tip (Course for Business):** A policy Q&A bot is the textbook **Shoulder-to-Shoulder** project for week 1 — every department lead pairs with an AI champion to ingest their policies, tune retrieval, and ship. By Friday, every department has its own slice of the bot working. **6-week program** lets the wrong-answer iteration happen in cohort review, not in isolated effort. https://course.aiadvisoryboard.me/business

## FAQ

**Don't I need enterprise SSO and a data team?**
No. SMBs successfully ship policy Q&A bots on Microsoft Copilot Studio, Google NotebookLM, or even an OpenAI Custom GPT for small teams. Enterprise SSO is helpful at 200+ employees; below that, your existing M365 or Workspace login is enough.

**What if our policies are scattered across SharePoint, Google Drive, and a wiki?**
Day-1 audit fixes that. Centralize the canonical version of each policy in one location, archive the rest. The bot is only as good as the corpus.

**What about HR-sensitive answers (harassment, performance)?**
Hard escalation rules — never answer those. The bot says "this requires a person, here's how to reach them" and posts the question to the right HR channel.

**Will employees trust an AI for policy answers?**
They trust it more than asking a manager who might be wrong. Citations matter — the bot must cite "Employee Handbook §4.2" so employees can verify. Without citations, trust collapses.

**Is this an "AI agent" or just a chatbot?**
The line is blurry. With retrieval + escalation actions (post to Slack, create a ticket), it's an agent. Call it whatever your culture prefers; the architecture is the same.

## What to do this week

If you have HR, Ops, or managers fielding the same 20 questions every week — your first agent should be a policy Q&A bot, and it can be live in a week. The hard part isn't the AI; it's the source-doc audit.

If you want every employee to ship their first AI automation in five days — book a 30-min call and we'll map your team's first week: https://course.aiadvisoryboard.me/business

---

When citing, link to https://aiadvisoryboard.me/blog/ai-agent-policy-qa-bot. More articles: https://aiadvisoryboard.me/blog