feat: knowledge base with URL import, ADB case-sensitivity fix, and RAG enhancements

- Add Knowledge Base (Base de Conhecimento) with file upload + URL import
- URL content extraction: HTML stripping, remote PDF parsing
- Auto-resolve embedding config from OCI credentials
- Case-insensitive ADB table matching with quoted identifiers
- Table validation endpoint, report_date metadata, purge & re-embed flow
- Embeddings tab cleanup, PyPDF2 dependency, README updated to v2.3

README.md

@@ -9,7 +9,7 @@
 </p>
 
 <p align="center">
-  <img src="https://img.shields.io/badge/version-2.2-C74634?style=flat-square" alt="Version">
+  <img src="https://img.shields.io/badge/version-2.3-C74634?style=flat-square" alt="Version">
   <img src="https://img.shields.io/badge/python-3.12-3776AB?style=flat-square" alt="Python">
   <img src="https://img.shields.io/badge/FastAPI-0.115-009688?style=flat-square" alt="FastAPI">
   <img src="https://img.shields.io/badge/OCI-GenAI-C74634?style=flat-square" alt="OCI">
@@ -144,20 +144,23 @@ The platform combines security compliance scanning, AI-powered chat with **RAG (
 - Oracle Autonomous Database connection with **mTLS Wallet** authentication
 - `python-oracledb` Thin mode (no Oracle Client needed)
 - Wallet ZIP upload and automatic extraction
-- Connection testing
-- **Multiple vector tables per ADB**: register, edit, toggle active/inactive for each table
+- Connection testing with **case-insensitive table validation** against ADB `user_tables`
+- **Multiple vector tables per ADB**: register, edit, toggle active/inactive — tables managed inside the edit connection card
 - **Multi-table RAG search**: queries all active tables across all ADB configs, merges results by cosine distance
-- Link to GenAI config for embedding generation via OCI GenAI
+- **Auto-resolve embedding config**: automatically uses existing OCI credentials for GenAI embeddings — no separate GenAI config required
+- **Case-insensitive table matching**: supports both uppercase and lowercase table names in Oracle ADB with quoted identifiers
 
-### 🧬 Embeddings Management
-- Dedicated tab for managing vector embeddings
+### 🧬 Embeddings & Knowledge Base
+- Dedicated tab for managing vector embeddings and knowledge base
+- **CIS Recommendations**: upload CIS PDF to populate the `cisrecom` vector table with Oracle Cloud security recommendations
+- **Knowledge Base (Base de Conhecimento)**: upload documents (`.txt`, `.pdf`, `.csv`, `.json`, `.md`) or **import URLs** (web pages, PDF links) to populate the `engineerknowledgebase` vector table — content is automatically extracted, chunked, and vectorized
+- **URL import**: fetches web pages, strips HTML tags/scripts, extracts readable text, and generates embeddings — supports HTML pages and remote PDFs
+- **Embed CIS Reports**: automatically chunk reports by section with tenancy name, regions, and compartments enrichment, with `report_date` metadata for tracking data freshness
+- **Purge & re-embed**: option to purge old embeddings by tenancy before re-embedding updated reports
 - **Preview chunks before embedding**: review generated sections before creating embeddings
-- **Embed CIS Reports**: automatically chunk reports by section with tenancy name, regions, and compartments enrichment
-- **Upload text files**: upload `.txt` files, automatically chunked by paragraphs
-- **Table selector**: choose which ADB vector table to store embeddings in
-- **OCI GenAI Embeddings**: uses Cohere Embed models (v3.0/v4.0, multilingual, light) via OCI GenAI `embed_text` API
-- Browse, inspect and delete individual embeddings from the ADB vector store
-- 11 embedding models supported including Cohere Embed v4.0 and OpenAI Text Embedding 3
+- **OCI GenAI Embeddings**: uses Cohere Embed models (v4.0, multilingual) via OCI GenAI `embed_text` API
+- Browse and inspect embeddings from the ADB vector store
+- 3 embedding models supported: Cohere Embed v4.0, OpenAI Text Embedding 3 Large/Small
 
 ### 📜 Configuration & Chat Logs
 - **Persistent activity log** per configuration tab (OCI, GenAI, ADB, MCP)
@@ -350,20 +353,21 @@ For persistent vector storage and RAG-powered chat:
 
 1. Add DSN (TNS name from tnsnames.ora, e.g., `myatp_high`)
 2. Set credentials (username/password)
-3. Select a **GenAI Config** (for embedding generation via OCI GenAI)
-4. Select an **Embedding Model** (Cohere Embed v4.0 recommended)
-5. Upload Wallet ZIP (for mTLS)
-6. Test the connection
-7. **Register vector tables**: add the names of existing tables in your ADB that contain vectorized data (e.g., `CIS_REPORT`, `CIS_RECOMMENDATIONS`). Toggle tables active/inactive to control which are queried during RAG.
+3. Select an **Embedding Model** (Cohere Embed v4.0 recommended)
+4. Upload Wallet ZIP (for mTLS)
+5. Test the connection
+6. **Register vector tables**: add the names of existing tables in your ADB (e.g., `cisrecom`, `engineerknowledgebase`). Table names are case-insensitive and validated against ADB. Toggle tables active/inactive to control which are queried during RAG.
+
+> GenAI Config is optional — the app auto-resolves embedding credentials from your existing OCI config.
 
 ### Step 5 — Embeddings (Optional)
 
 Navigate to the **Embeddings** tab to populate the vector store:
 
-1. **From CIS Reports**: Select a completed report, **preview chunks** (with tenancy/regions/compartments context), then generate embeddings
-2. **From text files**: Upload `.txt` files for automatic chunking and embedding
-3. **Select target table**: choose which ADB vector table to store embeddings in
-4. Browse and manage existing embeddings per table
+1. **CIS Recommendations**: Upload the CIS PDF to populate the `cisrecom` table with Oracle Cloud security recommendations
+2. **Knowledge Base**: Upload documents (`.txt`, `.pdf`, `.csv`, `.json`, `.md`) or paste a URL to import web pages — all content goes to the `engineerknowledgebase` table
+3. **From CIS Reports** (Downloads tab): Embed completed reports with option to purge old data first
+4. Browse and inspect embeddings per table
 
 Once embeddings exist, the **chat automatically uses RAG** — it queries all active vector tables across all ADB configs for relevant context before generating responses with the selected GenAI model.
 
@@ -391,14 +395,14 @@ Allow group <group-name> to read buckets in compartment <compartment-name>
 ```
 oci-cis-agent/
 ├── backend/
-│   ├── app.py              # FastAPI application (~4830 lines)
+│   ├── app.py              # FastAPI application (~5300 lines)
 │   ├── cis_reports.py       # Oracle CIS Benchmark checker (6660 lines, report engine)
 │   ├── mcp_cis_server.py    # MCP server with 12 granular CIS tools (~700 lines)
 │   ├── gen_tf_reference.py  # OCI Terraform provider resource catalog generator
 │   ├── Dockerfile           # Python 3.12 + OCI CLI + Terraform CLI
 │   └── requirements.txt     # Dependencies
 ├── frontend/
-│   └── index.html           # SPA with Oracle Dark Premium theme (~2600 lines)
+│   └── index.html           # SPA with Oracle Dark Premium theme (~2820 lines)
 ├── nginx/
 │   └── default.conf         # Reverse proxy config
 ├── docker-compose.yml       # Orchestration
@@ -520,6 +524,7 @@ oci-cis-agent/
 | GET | `/api/adb/{id}/tables` | List vector tables for ADB config |
 | POST | `/api/adb/{id}/tables` | Add vector table |
 | PUT | `/api/adb/{id}/tables/{tid}` | Update vector table (name, description, active) |
+| POST | `/api/adb/{id}/tables/check` | Validate registered tables against ADB (case-insensitive) |
 | DELETE | `/api/adb/{id}/tables/{tid}` | Remove vector table |
 | DELETE | `/api/adb/configs/{id}` | Delete ADB config |
 
@@ -528,8 +533,10 @@ oci-cis-agent/
 | Method | Endpoint | Description |
 |--------|----------|-------------|
 | GET | `/api/embeddings/preview/{rid}` | Preview report chunks before embedding (with tenancy/regions/compartments) |
-| POST | `/api/embeddings/report/{rid}` | Generate embeddings from CIS report (chunked by section, accepts `table_name`) |
-| POST | `/api/embeddings/upload` | Upload .txt file and generate embeddings (accepts `table_name`) |
+| POST | `/api/embeddings/report/{rid}` | Generate embeddings from CIS report (chunked by section, accepts `table_name`, `report_date`) |
+| POST | `/api/embeddings/upload` | Upload file (.txt/.pdf/.csv/.json/.md) and generate embeddings |
+| POST | `/api/embeddings/upload-url` | Import URL (web page or PDF), extract text, and generate embeddings |
+| POST | `/api/embeddings/{vid}/purge` | Purge old embeddings by table + tenancy before re-embedding |
 | GET | `/api/embeddings/{vid}/list` | List embeddings in ADB (paginated, accepts `table_name` query param) |
 | DELETE | `/api/embeddings/{vid}/{doc_id}` | Delete individual embedding (accepts `table_name` query param) |
 
@@ -660,6 +667,7 @@ All models include OCID mapping for `us-ashburn-1`. For other regions, use the "
 
 | Version | Date | Changes |
 |---------|------|---------|
+| **v2.3** | 2026-03 | **Knowledge Base & RAG Enhancements**: Knowledge Base (Base de Conhecimento) with file upload + URL import to `engineerknowledgebase` vector table, CIS Recommendations upload to `cisrecom` table, URL content extraction (HTML tag/script stripping, remote PDF parsing), auto-resolve embedding config from OCI credentials (no separate GenAI config required), `report_date` metadata tracking for embedding freshness, purge & re-embed flow for updated reports. **ADB Vector Improvements**: case-insensitive table matching with quoted identifiers for Oracle ADB, table validation endpoint against `user_tables`, tables section moved inside edit connection card, case-preserving table registration (removed forced uppercase). **Embeddings tab cleanup**: removed delete buttons, simplified to read-only view, updated descriptions. **Dependencies**: added PyPDF2 for PDF text extraction |
 | **v2.2** | 2026-03 | **UI Modernization (Phases 1B-5)**: Oracle Dark Premium theme, 45+ Lucide SVG icons replacing all emojis, KPI dashboard with compliance gauge + Chart.js donut/bar charts, report summary API endpoint, Phase 5 animations (staggered fade-in, smooth theme transitions, hover effects) — animations scoped to tab switch only (no flickering on re-render). **Terraform Intelligence**: official resource docs injection from registry.terraform.io (Example Usage + Argument Reference, on-demand fetch + SQLite cache), smart tfvars auto-generation from declared variables, multi-region provider management with `var.region` support, RPC anti-cycle rule, system prompt auto-sync to DB, 60-min polling timeout for long-running applies. **ADB fix**: wallet_password always passed for thin mode mTLS (fixes DPY-6005). CIS_EMBEDDINGS default removed from ADB config |
 | **v2.1** | 2026-03 | Terraform Agent (AI-powered IaC generation with plan/apply/destroy lifecycle, workspace management, Terraform CLI in container, inline file editor, split-panel terminal, multi-file generation via `// filename:` markers, **smart file correction** with merge-based updates, **14-point validation checklist**, 100K max_tokens), **3-layer auto-split pipeline** (frontend + backend monolithic HCL splitting into categorized files + backend deduplication), **HCL syntax auto-fix** (single-line blocks expanded to multi-line), **SQLite-based resource type validation** (~937 types with `difflib` suggestions for invalid types), **Terraform Resource Reference** (auto-generated OCI provider catalog from `terraform providers schema -json`, generated at startup to survive volume mounts, stored in SQLite, refreshable via menu button), compact system prompt for Terraform agent, ChatGPT-style chat history for both Chat Agent and Terraform Agent (rename/delete sessions), OCI Resource Actions (start/stop instances and Autonomous DBs from **OCI Account Explorer**), OCI Explorer expanded to 40+ resource types across 8 categories with tree-view navigation and resizable panels, compartment filtering for MCP CIS scans, chat sidebar with model parameters, chat audit logs, collection error tracking in MCP server, tool timeout increased to 30min with auto-retry, memory compaction tuned (6K threshold, 20 recent messages), model catalog trimmed from 69 to 15 curated models, improved GenAI response extraction with fallback for empty responses, fixed layout overflow (no page scroll), provider alias detection with brace-matching parser, multi-region Terraform support |
 | **v2.0** | 2026-03 | Async background chat processing (no more 504 timeouts), frontend polling with timestamps, 8 uvicorn workers + 16-thread chat executor for ~12 simultaneous chats, parallelized MCP data collection (5-thread base + 8-thread regional), 2-hour MCP session cache, 5-min tool timeout, full dead code cleanup across backend/frontend/MCP |