CloudOps Knowledge Assistant.
Operations knowledge. With an evidence trail.
An in-development RAG system for on-call cloud and DevOps questions. Its repository contains ingestion, hybrid retrieval, search-time role filters, grounded-answer checks, a FastAPI service and a Review Bench interface. Live provider evaluation and the AWS demo remain pending.
01 / THE PROBLEM
Evidence must respect access.
On-call engineers need answers that respect access rights, cite the right evidence and abstain when sources are insufficient. This system is being built around those constraints, with retrieval and answer checks that can be inspected.
02 / ARCHITECTURE
How the system moves.
- 01
Ingest
Structure-aware chunks and parent sections are indexed with content hashes for incremental updates.
- 02
Authorize
Role filters enter both vector and BM25 OpenSearch queries before retrieved text reaches the model.
- 03
Retrieve
Titan V2 embeddings and BM25 results can be fused by RRF or weighted scoring, then reranked and expanded to parent context.
- 04
Ground
Generation code checks citations, source conflicts and evidence strength, then abstains when support is insufficient.
- 05
Review
FastAPI returns sources and a retrieval trail; the Next.js Review Bench presents answer, evidence and evaluation surfaces.
03 / ENGINEERING DECISIONS
Why these boundaries matter.
Search-time access
ACLs are applied inside each search query and preserved through context expansion, before LLM input.
Inspect the answer
Citation validation and abstention are built into the answer path; streamed clients receive a final validated response.
Evaluate before claiming
The 315-question dataset and harness exist, but baseline scores and live Bedrock comparisons have not been published.
04 / SOURCE EVIDENCE
What the repository shows.
- Manifest: 222 documents total, including 115 synthetic internal documents and 70 incident files
- Evaluation dataset: 315 questions across 10 categories; measured scores are pending
- API, retrieval, security and Review Bench code are present; Terraform is authored but not applied
Selected files from the public repository, pinned to the revision reviewed for this case study.
- Status and phase record
- Corpus manifest
- Retrieval pipeline
- Security model
- API contract
- Evaluation dataset
- AWS Terraform design
HONEST BOUNDARY
Code and local test paths are present, but real Bedrock and OpenSearch evaluation, Terraform apply, AWS spend and load-test results are pending. No retrieval score, faithfulness, latency, security pass rate or production deployment is claimed.