diff --git a/README.md b/README.md index ce83577..48862a0 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@

- Version + Version Python FastAPI OCI @@ -23,14 +23,15 @@ OCI CIS AI Agent is a self-hosted web application that automates **CIS Oracle Cloud Infrastructure Foundations Benchmark 3.0** compliance checks, powered by **OCI Generative AI** for intelligent analysis and an **MCP (Model Context Protocol)** server architecture for extensible task execution. -The platform combines security compliance scanning, AI-powered chat, infrastructure exploration, and vector-based knowledge storage into a single, containerized solution with Oracle Cloud's official light theme. +The platform combines security compliance scanning, AI-powered chat with **RAG (Retrieval-Augmented Generation)**, infrastructure exploration, and vector-based knowledge storage into a single, containerized solution with Oracle Cloud's official light theme. --- ## Features -### ๐Ÿค– AI Chat Agent +### ๐Ÿค– AI Chat Agent with RAG - **OCI Generative AI** integration via official SDK (`oci.generative_ai_inference`) +- **RAG (Retrieval-Augmented Generation)**: automatically queries ADB vector store for relevant context before generating responses - 12 models across 4 providers: **Cohere** (Command A/R/R+), **Meta** (Llama 4/3.3/3.2/3.1), **Google** (Gemini 2.5), **xAI** (Grok 3/4) - 16 OCI regions supported with auto-generated endpoints - Full parameter control: temperature, max_tokens, top_p, top_k, frequency/presence penalty @@ -63,6 +64,15 @@ The platform combines security compliance scanning, AI-powered chat, infrastruct - Wallet ZIP upload and automatic extraction - Connection testing - Configurable embeddings table name (default: `CIS_EMBEDDINGS`) +- Link to GenAI config for embedding generation via OCI GenAI + +### ๐Ÿงฌ Embeddings Management +- Dedicated tab for managing vector embeddings +- **Embed CIS Reports**: automatically chunk reports by section (IAM, Networking, Compute, etc.) and generate embeddings +- **Upload text files**: upload `.txt` files, automatically chunked by paragraphs +- **OCI GenAI Embeddings**: uses Cohere Embed models (v3.0, multilingual, light) via OCI GenAI `embed_text` API +- Browse, inspect and delete individual embeddings from the ADB vector store +- 4 embedding models supported: Cohere Embed English/Multilingual v3.0 (1024d), Light variants (384d) ### ๐Ÿ” Security - **JWT authentication** with configurable expiry @@ -92,6 +102,9 @@ The platform combines security compliance scanning, AI-powered chat, infrastruct โ”‚ โ”‚ โ”‚ MCP Servers โ”‚โ”€โ”€โ–ถ Tools โ”‚ โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ oracledb โ”‚โ”€โ”€โ–ถ ADB โ”‚ โ”‚ +โ”‚ โ”‚ โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค โ”‚ โ”‚ +โ”‚ โ”‚ โ”‚ RAG Pipeline โ”‚โ”€โ”€โ–ถ Embed + โ”‚ โ”‚ +โ”‚ โ”‚ โ”‚ โ”‚ Search โ”‚ โ”‚ โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ SQLite (agent.db) โ”‚ โ”‚ @@ -208,14 +221,27 @@ Register MCP servers for extended task execution: MCP servers can be **linked to ADB Vector** databases, enabling them to use the vector store as a tool during report execution. -### Step 4 โ€” ADB Vector (Optional) +### Step 4 โ€” ADB Vector + RAG (Optional) -For persistent vector storage of CIS findings and embeddings: +For persistent vector storage and RAG-powered chat: 1. Add DSN (TNS name from tnsnames.ora, e.g., `myatp_high`) 2. Set credentials (username/password) -3. Upload Wallet ZIP (for mTLS) -4. Test the connection +3. Select a **GenAI Config** (for embedding generation via OCI GenAI) +4. Select an **Embedding Model** (Cohere Embed v3.0 recommended) +5. Upload Wallet ZIP (for mTLS) +6. Test the connection +7. Click **Create Table** to initialize the embeddings table in ADB + +### Step 5 โ€” Embeddings (Optional) + +Navigate to the **Embeddings** tab to populate the vector store: + +1. **From CIS Reports**: Select a completed report and generate embeddings (1 per section) +2. **From text files**: Upload `.txt` files for automatic chunking and embedding +3. Browse and manage existing embeddings + +Once embeddings exist, the **chat automatically uses RAG** โ€” it queries the ADB vector store for relevant context before generating responses with the selected GenAI model. --- @@ -241,12 +267,12 @@ Allow group to read buckets in compartment ``` oci-cis-agent/ โ”œโ”€โ”€ backend/ -โ”‚ โ”œโ”€โ”€ app.py # FastAPI application (933 lines) +โ”‚ โ”œโ”€โ”€ app.py # FastAPI application (~1100 lines) โ”‚ โ”œโ”€โ”€ cis_runner.py # CIS Benchmark check executor โ”‚ โ”œโ”€โ”€ Dockerfile # Python 3.12 + OCI CLI โ”‚ โ””โ”€โ”€ requirements.txt # Dependencies โ”œโ”€โ”€ frontend/ -โ”‚ โ””โ”€โ”€ index.html # SPA with Oracle Cloud theme (419 lines) +โ”‚ โ””โ”€โ”€ index.html # SPA with Oracle Cloud theme (~620 lines) โ”œโ”€โ”€ nginx/ โ”‚ โ””โ”€โ”€ default.conf # Reverse proxy config โ”œโ”€โ”€ docker-compose.yml # Orchestration @@ -314,12 +340,22 @@ oci-cis-agent/ | Method | Endpoint | Description | |--------|----------|-------------| -| POST | `/api/adb/config` | Save ADB connection | +| POST | `/api/adb/config` | Save ADB connection (with GenAI config + embedding model) | | GET | `/api/adb/configs` | List ADB connections | | POST | `/api/adb/{id}/upload-wallet` | Upload wallet ZIP | | POST | `/api/adb/test/{id}` | Test ADB connection | +| POST | `/api/adb/{id}/ensure-table` | Create embeddings table in ADB | | DELETE | `/api/adb/configs/{id}` | Delete ADB config | +### Embeddings + +| Method | Endpoint | Description | +|--------|----------|-------------| +| POST | `/api/embeddings/report/{rid}` | Generate embeddings from CIS report (chunked by section) | +| POST | `/api/embeddings/upload` | Upload .txt file and generate embeddings (chunked by paragraphs) | +| GET | `/api/embeddings/{vid}/list` | List embeddings in ADB (paginated) | +| DELETE | `/api/embeddings/{vid}/{doc_id}` | Delete individual embedding | + ### Chat & Reports | Method | Endpoint | Description | @@ -360,6 +396,15 @@ oci-cis-agent/ | xAI | Grok 4 | GENERIC | | xAI | Grok 3 | GENERIC | +### Embedding Models + +| Provider | Model | Dimensions | +|----------|-------|------------| +| Cohere | Embed English v3.0 | 1024 | +| Cohere | Embed Multilingual v3.0 | 1024 | +| Cohere | Embed English Light v3.0 | 384 | +| Cohere | Embed Multilingual Light v3.0 | 384 | + ### GenAI Regions `us-chicago-1` ยท `us-ashburn-1` ยท `us-phoenix-1` ยท `uk-london-1` ยท `eu-frankfurt-1` ยท `ap-tokyo-1` ยท `ap-osaka-1` ยท `sa-saopaulo-1` ยท `ca-toronto-1` ยท `ap-melbourne-1` ยท `ap-mumbai-1` ยท `eu-amsterdam-1` ยท `me-jeddah-1` ยท `ap-singapore-1` ยท `ap-seoul-1` ยท `sa-vinhedo-1` @@ -386,6 +431,7 @@ oci-cis-agent/ | Version | Date | Changes | |---------|------|---------| +| **v1.2** | 2026-03 | RAG pipeline (OCI GenAI embeddings + ADB vector search), dedicated Embeddings tab, CIS report chunking, file upload embedding | | **v1.1** | 2025-02 | OCI SDK GenAI (exact pattern), OCI Account Explorer, MCPโ†”ADB linking, full chat parameters | | **v1.0** | 2025-02 | Initial release: OCI theme, GenAI integration, MCP servers, ADB vector, JWT+MFA+RBAC |