GustavoNogueira/A-Team-Security-Infra-Agent-Engineer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: auto-mapped CIS report embedding, section embed, ADB RAW(16) fix

Browse Source 
- Auto-embed report: maps each CSV to its ADB table (summaryreportcsvvector, identityandaccess, networking, etc.)
- Section embed: new endpoint POST /api/embeddings/report/{rid}/section with per-section button in UI
- Table validation: checks registered ADB tables before embedding, reports missing tables
- Auto-detect embedding dimension: reads table dim and selects correct model (3072→large, 1536→small)
- ADB RAW(16) ID fix: all vector inserts use HEXTORAW() for UUID (fixes ORA-01465)
- Float32 vectors: all inserts use array.array('f') for FLOAT32 compatibility
- Embedding status: real-time progress polling (Embedding X/Y — table: Z)
- Loading per section: spinner only on the section being embedded, not all
- Removed individual file embed buttons (only section + full report)
- Summary CSV chunking: groups by CIS section with tenancy + extract_date metadata
- Findings CSV chunking: each row becomes a document with structured content
- README: documented all 11 required ADB vector tables with descriptions
This commit is contained in:
nogueiraguh
2026-03-21 00:10:50 -03:00
parent 2c5f0f2e1c
commit 73c60212f7
4 changed files with 494 additions and 118 deletions
Download Patch File Download Diff File Expand all files Collapse all files

31
README.md
View File

@@ -394,17 +394,44 @@ For persistent vector storage and RAG-powered chat:
3. Select an **Embedding Model** (Cohere Embed v4.0 recommended)
4. Upload Wallet ZIP (for mTLS)
5. Test the connection
6. **Register vector tables**: add the names of existing tables in your ADB (e.g., `cisrecom`, `engineerknowledgebase`). Table names are case-insensitive and validated against ADB. Toggle tables active/inactive to control which are queried during RAG.
6. **Register vector tables**: add the names of existing tables in your ADB. Table names are case-insensitive and validated against ADB. Toggle tables active/inactive to control which are queried during RAG.
> GenAI Config is optional — the app auto-resolves embedding credentials from your existing OCI config.
#### Required ADB Vector Tables
The following tables must be created in your Autonomous Database for the auto-embedding to work correctly. Each table must have the schema: `ID VARCHAR2(100), TEXT CLOB, EMBEDDING VECTOR, METADATA CLOB`.
**CIS Report Tables** — auto-populated when you click "Embed Report":
| Table Name | Purpose | Source CSVs |
|-----------|---------|-------------|
| `summaryreportcsvvector` | CIS report summary (compliance scores, section totals) | `cis_summary_report.csv` |
| `identityandaccess` | IAM findings (users, policies, MFA, API keys) | `cis_Identity_and_Access_Management_*.csv` |
| `networking` | Network findings (security lists, NSGs, VCNs) | `cis_Networking_*.csv` |
| `computeinstances` | Compute findings (instances, metadata, boot) | `cis_Compute_*.csv` |
| `loggingandmonitoring` | Logging findings (alarms, events, notifications) | `cis_Logging_and_Monitoring_*.csv` |
| `objectstorage` | Object Storage findings (buckets, visibility, encryption) | `cis_Storage_Object_Storage_*.csv` |
| `storageblockvolume` | Block Volume findings (encryption, CMK) | `cis_Storage_Block_Volumes_*.csv` |
| `filestorageservice` | File Storage findings (encryption, CMK) | `cis_Storage_File_Storage_Service_*.csv` |
| `assetmanagement` | Asset Management findings (compartments, tagging) | `cis_Asset_Management_*.csv` |
**Other Tables** — populated manually or via dedicated uploads:
| Table Name | Purpose | How to populate |
|-----------|---------|-----------------|
| `cisrecom` | CIS Benchmark recommendations and best practices | Upload CIS PDF in Embeddings tab |
| `engineerknowledgebase` | General knowledge base (blogs, docs, PDFs) | Upload files or import URLs in Embeddings tab |
> When you click **"Embed Report"** on a completed CIS report, the system automatically maps each CSV to its corresponding table and embeds all findings with tenancy name and extract date for isolation. Progress is shown in real-time.
### Step 5 — Embeddings (Optional)
Navigate to the **Embeddings** tab to populate the vector store:
1. **CIS Recommendations**: Upload the CIS PDF to populate the `cisrecom` table with Oracle Cloud security recommendations
2. **Knowledge Base**: Upload documents (`.txt`, `.pdf`, `.csv`, `.json`, `.md`) or paste a URL to import web pages — all content goes to the `engineerknowledgebase` table
3. **From CIS Reports** (Downloads tab): Embed completed reports with option to purge old data first
3. **From CIS Reports** (Reports tab): Click "Embed Report" to auto-embed all findings CSVs into their mapped tables
4. Browse and inspect embeddings per table
Once embeddings exist, the **chat automatically uses RAG** — it queries all active vector tables across all ADB configs for relevant context before generating responses with the selected GenAI model.