MPS School Information Chatbot
AI assistant that answers staff and family questions about Millard Public Schools (Omaha, NE), using only the district’s own published websites and documents.
Project details
- Team project for Millard Public Schools District Office, with a retrieval pipeline and evaluation system. About 98,000 indexed chunks from district and school sites, PDFs, Drive files, and calendars, rebuilt with one command after a refetch.
- Hybrid vector and BM25 search, per-school partitions, LLM reranking, cited answers, and a fact-checking pass. The bot says ‘I don’t know’ instead of guessing.
- A 162-question test set with a 3× LLM judge and cost tracking. An A/B test selected a reranker with the same measured quality, about 9% lower cost, and about 2× faster responses.
Context
A team project for a real client, the Millard Public Schools District Office. District information is spread across 44 district and school sites, PDFs, Google Drive files, Google Sites, and embedded calendars. Staff could not find answers quickly, and a general-purpose chatbot could invent them.
The goal was an assistant that answers from the district’s own sources, cites them, and says ‘I don’t know’ when the sources are silent.
A rebuildable data pipeline
- The crawler covers district and school websites and partner community sites. It automatically skips not-found pages, logins, carts, and browser error pages.
- Web pages become clean Markdown. A frequency-based filter removes navigation, without hard-coded rules for individual layouts.
- Digital and scanned PDFs are converted; a vision model transcribes scanned pages one at a time. The pipeline also handles Google Drive and Docs files, Office documents, and images.
- Google Calendar recurrences expand into actual school-year dates, accounting for holidays and moved or cancelled instances.
- Parsers generate structured staff directories, school cards, policies, and enrollment data. Enrollment numbers are checked against the report’s own totals before being written.
- One command rebuilds everything after a site refetch, with no manual fixes. About 98,000 chunks are stored in ChromaDB with text-embedding-3-large, alongside a BM25 lexical index.
Retrieval & answering
- A classifier detects the school or schools, source restrictions such as ‘only in PDFs’, requests for complete lists, and small talk. Query rewriting creates several search variants.
- Hybrid vector and BM25 search runs separately for each school partition so small schools are not lost. Critical candidates, such as a school’s own directory, bypass the reranker.
- An LLM reranker filters candidates, then the answer is generated with citations. A final pass checks every draft fact against the sources and corrects unsupported claims.
- ‘List all X’ questions use generated directories instead of partial search results. Large lists first ask the user to choose a category.
Honesty by design
Three layers keep the assistant from guessing:
- Retrieval-level refusal when nothing relevant is found.
- A prompt rule against inference beyond the sources.
- A self-check that verifies facts against the sources.
Published schedules are presented as published, never as the actual state of buildings.
Evaluation & cost control
- A custom 162-question test set, scored three times per answer by an LLM judge. Run history and per-question cost are tracked.
- A few questions are spot-checked between full runs to save API budget.
- A custom usage meter wraps every API client and records tokens and dollar cost per turn and per pipeline stage, with prices in a JSON configuration. Unknown costs are recorded as unknown, never as $0.
- An A/B test of an alternative reranker through the OpenRouter Decisions API found the same quality at about 9% lower cost and about 2× faster responses. That reranker was kept.
Current stack: language & testing
- Python 3 for the backend, data pipeline, and evaluation system.
- pytest: about 1,600 tests, with about 110 test files covering the pipeline’s pure functions without network or model calls. Node unit tests cover the panel’s JavaScript.
Data collection & conversion
- Selenium with undetected-chromedriver runs a real Chrome browser for crawling. curl_cffi downloads documents with a Chrome TLS fingerprint.
- BeautifulSoup and lxml parse HTML; markdownify converts it to Markdown.
- pypdf extracts PDF text. PyMuPDF renders scanned pages as images for vision OCR.
- python-docx and openpyxl handle Office documents.
- iCal parsing uses python-dateutil for RRULE expansion and zoneinfo/tzdata for time zones.
Retrieval stack
- ChromaDB is the local, persistent vector store, holding about 98,000 chunks embedded with text-embedding-3-large.
- The BM25 lexical index is written from scratch, without an external BM25 library.
- Hybrid retrieval merges vector and BM25 results, with separate queries for each school partition.
Backend & interface
- Flask with Server-Sent Events streams answers, pipeline stages, and token usage. The panel also includes follow-up suggestion chips and a per-chat usage card.
- The front end uses vanilla JavaScript and marked.js for Markdown, without a framework.
- SQLite stores chat history and the message-review trace. The interface includes an admin panel, an evaluation dashboard, and a Browse section.
Models & their roles
- GPT-6 Luna (OpenAI): the default for answer generation, query classification, reranking, and self-check. Each step’s model and reasoning effort can be configured separately.
- Claude Haiku 5.5 (Anthropic Messages API): an alternative chat model selectable in the panel through an adapter with the same interface.
- text-embedding-3-large (OpenAI): embeddings for retrieval.
- GPT-5.6 Luna: page-by-page vision OCR for scanned PDFs, Drive files, images, and calendar attachments.
- Jev 1.13 through the OpenRouter Decisions API: reranking judge, message-type classifier, and shadow fact-check. It answers one yes/no question per chunk and decides by probability. In the project’s A/B test, the alternative reranker matched chat-model quality at about 9% lower cost and about 2× the speed.
- GPT-5.6 Luna: the evaluation judge, with three votes per answer. Its model is pinned so runs remain comparable when the bot’s model changes.
- A local Qwen model through Ollama: structured extraction of the programs directory, with no API cost.
How the model choices evolved
The project began before GPT-6 Luna was released. Tests later showed a lower cost in this pipeline, leading to its adoption.
I subsequently chose Haiku 5.5 for chat. Claude is more familiar to me, and in my use Haiku ran about twice as fast at a similar cost to Luna. These comparisons describe this project’s workload.