diff --git a/README.md b/README.md index 2511a49..30177c8 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@

- Version + Version Python FastAPI OCI @@ -144,20 +144,23 @@ The platform combines security compliance scanning, AI-powered chat with **RAG ( - Oracle Autonomous Database connection with **mTLS Wallet** authentication - `python-oracledb` Thin mode (no Oracle Client needed) - Wallet ZIP upload and automatic extraction -- Connection testing -- **Multiple vector tables per ADB**: register, edit, toggle active/inactive for each table +- Connection testing with **case-insensitive table validation** against ADB `user_tables` +- **Multiple vector tables per ADB**: register, edit, toggle active/inactive — tables managed inside the edit connection card - **Multi-table RAG search**: queries all active tables across all ADB configs, merges results by cosine distance -- Link to GenAI config for embedding generation via OCI GenAI +- **Auto-resolve embedding config**: automatically uses existing OCI credentials for GenAI embeddings — no separate GenAI config required +- **Case-insensitive table matching**: supports both uppercase and lowercase table names in Oracle ADB with quoted identifiers -### 🧬 Embeddings Management -- Dedicated tab for managing vector embeddings +### 🧬 Embeddings & Knowledge Base +- Dedicated tab for managing vector embeddings and knowledge base +- **CIS Recommendations**: upload CIS PDF to populate the `cisrecom` vector table with Oracle Cloud security recommendations +- **Knowledge Base (Base de Conhecimento)**: upload documents (`.txt`, `.pdf`, `.csv`, `.json`, `.md`) or **import URLs** (web pages, PDF links) to populate the `engineerknowledgebase` vector table — content is automatically extracted, chunked, and vectorized +- **URL import**: fetches web pages, strips HTML tags/scripts, extracts readable text, and generates embeddings — supports HTML pages and remote PDFs +- **Embed CIS Reports**: automatically chunk reports by section with tenancy name, regions, and compartments enrichment, with `report_date` metadata for tracking data freshness +- **Purge & re-embed**: option to purge old embeddings by tenancy before re-embedding updated reports - **Preview chunks before embedding**: review generated sections before creating embeddings -- **Embed CIS Reports**: automatically chunk reports by section with tenancy name, regions, and compartments enrichment -- **Upload text files**: upload `.txt` files, automatically chunked by paragraphs -- **Table selector**: choose which ADB vector table to store embeddings in -- **OCI GenAI Embeddings**: uses Cohere Embed models (v3.0/v4.0, multilingual, light) via OCI GenAI `embed_text` API -- Browse, inspect and delete individual embeddings from the ADB vector store -- 11 embedding models supported including Cohere Embed v4.0 and OpenAI Text Embedding 3 +- **OCI GenAI Embeddings**: uses Cohere Embed models (v4.0, multilingual) via OCI GenAI `embed_text` API +- Browse and inspect embeddings from the ADB vector store +- 3 embedding models supported: Cohere Embed v4.0, OpenAI Text Embedding 3 Large/Small ### 📜 Configuration & Chat Logs - **Persistent activity log** per configuration tab (OCI, GenAI, ADB, MCP) @@ -350,20 +353,21 @@ For persistent vector storage and RAG-powered chat: 1. Add DSN (TNS name from tnsnames.ora, e.g., `myatp_high`) 2. Set credentials (username/password) -3. Select a **GenAI Config** (for embedding generation via OCI GenAI) -4. Select an **Embedding Model** (Cohere Embed v4.0 recommended) -5. Upload Wallet ZIP (for mTLS) -6. Test the connection -7. **Register vector tables**: add the names of existing tables in your ADB that contain vectorized data (e.g., `CIS_REPORT`, `CIS_RECOMMENDATIONS`). Toggle tables active/inactive to control which are queried during RAG. +3. Select an **Embedding Model** (Cohere Embed v4.0 recommended) +4. Upload Wallet ZIP (for mTLS) +5. Test the connection +6. **Register vector tables**: add the names of existing tables in your ADB (e.g., `cisrecom`, `engineerknowledgebase`). Table names are case-insensitive and validated against ADB. Toggle tables active/inactive to control which are queried during RAG. + +> GenAI Config is optional — the app auto-resolves embedding credentials from your existing OCI config. ### Step 5 — Embeddings (Optional) Navigate to the **Embeddings** tab to populate the vector store: -1. **From CIS Reports**: Select a completed report, **preview chunks** (with tenancy/regions/compartments context), then generate embeddings -2. **From text files**: Upload `.txt` files for automatic chunking and embedding -3. **Select target table**: choose which ADB vector table to store embeddings in -4. Browse and manage existing embeddings per table +1. **CIS Recommendations**: Upload the CIS PDF to populate the `cisrecom` table with Oracle Cloud security recommendations +2. **Knowledge Base**: Upload documents (`.txt`, `.pdf`, `.csv`, `.json`, `.md`) or paste a URL to import web pages — all content goes to the `engineerknowledgebase` table +3. **From CIS Reports** (Downloads tab): Embed completed reports with option to purge old data first +4. Browse and inspect embeddings per table Once embeddings exist, the **chat automatically uses RAG** — it queries all active vector tables across all ADB configs for relevant context before generating responses with the selected GenAI model. @@ -391,14 +395,14 @@ Allow group to read buckets in compartment ``` oci-cis-agent/ ├── backend/ -│ ├── app.py # FastAPI application (~4830 lines) +│ ├── app.py # FastAPI application (~5300 lines) │ ├── cis_reports.py # Oracle CIS Benchmark checker (6660 lines, report engine) │ ├── mcp_cis_server.py # MCP server with 12 granular CIS tools (~700 lines) │ ├── gen_tf_reference.py # OCI Terraform provider resource catalog generator │ ├── Dockerfile # Python 3.12 + OCI CLI + Terraform CLI │ └── requirements.txt # Dependencies ├── frontend/ -│ └── index.html # SPA with Oracle Dark Premium theme (~2600 lines) +│ └── index.html # SPA with Oracle Dark Premium theme (~2820 lines) ├── nginx/ │ └── default.conf # Reverse proxy config ├── docker-compose.yml # Orchestration @@ -520,6 +524,7 @@ oci-cis-agent/ | GET | `/api/adb/{id}/tables` | List vector tables for ADB config | | POST | `/api/adb/{id}/tables` | Add vector table | | PUT | `/api/adb/{id}/tables/{tid}` | Update vector table (name, description, active) | +| POST | `/api/adb/{id}/tables/check` | Validate registered tables against ADB (case-insensitive) | | DELETE | `/api/adb/{id}/tables/{tid}` | Remove vector table | | DELETE | `/api/adb/configs/{id}` | Delete ADB config | @@ -528,8 +533,10 @@ oci-cis-agent/ | Method | Endpoint | Description | |--------|----------|-------------| | GET | `/api/embeddings/preview/{rid}` | Preview report chunks before embedding (with tenancy/regions/compartments) | -| POST | `/api/embeddings/report/{rid}` | Generate embeddings from CIS report (chunked by section, accepts `table_name`) | -| POST | `/api/embeddings/upload` | Upload .txt file and generate embeddings (accepts `table_name`) | +| POST | `/api/embeddings/report/{rid}` | Generate embeddings from CIS report (chunked by section, accepts `table_name`, `report_date`) | +| POST | `/api/embeddings/upload` | Upload file (.txt/.pdf/.csv/.json/.md) and generate embeddings | +| POST | `/api/embeddings/upload-url` | Import URL (web page or PDF), extract text, and generate embeddings | +| POST | `/api/embeddings/{vid}/purge` | Purge old embeddings by table + tenancy before re-embedding | | GET | `/api/embeddings/{vid}/list` | List embeddings in ADB (paginated, accepts `table_name` query param) | | DELETE | `/api/embeddings/{vid}/{doc_id}` | Delete individual embedding (accepts `table_name` query param) | @@ -660,6 +667,7 @@ All models include OCID mapping for `us-ashburn-1`. For other regions, use the " | Version | Date | Changes | |---------|------|---------| +| **v2.3** | 2026-03 | **Knowledge Base & RAG Enhancements**: Knowledge Base (Base de Conhecimento) with file upload + URL import to `engineerknowledgebase` vector table, CIS Recommendations upload to `cisrecom` table, URL content extraction (HTML tag/script stripping, remote PDF parsing), auto-resolve embedding config from OCI credentials (no separate GenAI config required), `report_date` metadata tracking for embedding freshness, purge & re-embed flow for updated reports. **ADB Vector Improvements**: case-insensitive table matching with quoted identifiers for Oracle ADB, table validation endpoint against `user_tables`, tables section moved inside edit connection card, case-preserving table registration (removed forced uppercase). **Embeddings tab cleanup**: removed delete buttons, simplified to read-only view, updated descriptions. **Dependencies**: added PyPDF2 for PDF text extraction | | **v2.2** | 2026-03 | **UI Modernization (Phases 1B-5)**: Oracle Dark Premium theme, 45+ Lucide SVG icons replacing all emojis, KPI dashboard with compliance gauge + Chart.js donut/bar charts, report summary API endpoint, Phase 5 animations (staggered fade-in, smooth theme transitions, hover effects) — animations scoped to tab switch only (no flickering on re-render). **Terraform Intelligence**: official resource docs injection from registry.terraform.io (Example Usage + Argument Reference, on-demand fetch + SQLite cache), smart tfvars auto-generation from declared variables, multi-region provider management with `var.region` support, RPC anti-cycle rule, system prompt auto-sync to DB, 60-min polling timeout for long-running applies. **ADB fix**: wallet_password always passed for thin mode mTLS (fixes DPY-6005). CIS_EMBEDDINGS default removed from ADB config | | **v2.1** | 2026-03 | Terraform Agent (AI-powered IaC generation with plan/apply/destroy lifecycle, workspace management, Terraform CLI in container, inline file editor, split-panel terminal, multi-file generation via `// filename:` markers, **smart file correction** with merge-based updates, **14-point validation checklist**, 100K max_tokens), **3-layer auto-split pipeline** (frontend + backend monolithic HCL splitting into categorized files + backend deduplication), **HCL syntax auto-fix** (single-line blocks expanded to multi-line), **SQLite-based resource type validation** (~937 types with `difflib` suggestions for invalid types), **Terraform Resource Reference** (auto-generated OCI provider catalog from `terraform providers schema -json`, generated at startup to survive volume mounts, stored in SQLite, refreshable via menu button), compact system prompt for Terraform agent, ChatGPT-style chat history for both Chat Agent and Terraform Agent (rename/delete sessions), OCI Resource Actions (start/stop instances and Autonomous DBs from **OCI Account Explorer**), OCI Explorer expanded to 40+ resource types across 8 categories with tree-view navigation and resizable panels, compartment filtering for MCP CIS scans, chat sidebar with model parameters, chat audit logs, collection error tracking in MCP server, tool timeout increased to 30min with auto-retry, memory compaction tuned (6K threshold, 20 recent messages), model catalog trimmed from 69 to 15 curated models, improved GenAI response extraction with fallback for empty responses, fixed layout overflow (no page scroll), provider alias detection with brace-matching parser, multi-region Terraform support | | **v2.0** | 2026-03 | Async background chat processing (no more 504 timeouts), frontend polling with timestamps, 8 uvicorn workers + 16-thread chat executor for ~12 simultaneous chats, parallelized MCP data collection (5-thread base + 8-thread regional), 2-hour MCP session cache, 5-min tool timeout, full dead code cleanup across backend/frontend/MCP | diff --git a/backend/app.py b/backend/app.py index 7103438..d48b303 100644 --- a/backend/app.py +++ b/backend/app.py @@ -466,6 +466,16 @@ async def current_user(cred: HTTPAuthorizationCredentials = Depends(security)): if not u or not s: raise HTTPException(401, "Sessão inválida") return dict(u) +def _user_from_token(token: str): + """Resolve user from a raw JWT token string (for query-param auth on downloads).""" + try: p = pyjwt.decode(token, APP_SECRET, algorithms=[JWT_ALG]) + except Exception: raise HTTPException(401, "Token inválido") + with db() as c: + u = c.execute("SELECT * FROM users WHERE id=? AND is_active=1", (p["sub"],)).fetchone() + s = c.execute("SELECT * FROM sessions WHERE id=? AND is_active=1", (p["sid"],)).fetchone() + if not u or not s: raise HTTPException(401, "Sessão inválida") + return dict(u) + def require(*roles): async def dep(u=Depends(current_user)): if u["role"] not in roles: raise HTTPException(403, f"Requer role: {', '.join(roles)}") @@ -929,7 +939,7 @@ async def explore_security_lists(cid: str, compartment_id: str = Query(None), re _, default_comp = _explore_comp(cid) comp = compartment_id or default_comp sls = vn.list_security_lists(comp).data - def _fmt_rule(r, direction): + def _fmt_rule(r, direction, idx): proto_map = {"6":"TCP","17":"UDP","1":"ICMP","all":"ALL"} proto = proto_map.get(r.protocol, r.protocol) src_dst = r.source if direction == "ingress" else r.destination @@ -941,17 +951,98 @@ async def explore_security_lists(cid: str, compartment_id: str = Query(None), re dp = r.udp_options.destination_port_range if dp: ports = f"{dp.min}-{dp.max}" if dp.min != dp.max else str(dp.min) return {"direction":direction,"protocol":proto,"source_dest":src_dst or "","ports":ports, - "stateless":r.is_stateless,"description":getattr(r,'description','') or ""} + "stateless":r.is_stateless,"description":getattr(r,'description','') or "","rule_index":idx} result = [] for s in sls: - ingress = [_fmt_rule(r,"ingress") for r in (s.ingress_security_rules or [])] - egress = [_fmt_rule(r,"egress") for r in (s.egress_security_rules or [])] + ingress = [_fmt_rule(r,"ingress",i) for i,r in enumerate(s.ingress_security_rules or [])] + egress = [_fmt_rule(r,"egress",i) for i,r in enumerate(s.egress_security_rules or [])] result.append({"id":s.id,"display_name":s.display_name,"vcn_id":s.vcn_id,"lifecycle_state":s.lifecycle_state, "ingress_count":len(ingress),"egress_count":len(egress),"rules":ingress+egress}) return result except Exception as e: return {"error": str(e)[:500]} +@app.post("/api/oci/explore/{cid}/security_lists/{sl_id}/rules") +async def add_security_list_rule(cid: str, sl_id: str, req: dict, u=Depends(require("admin", "user"))): + """Add a rule to a Security List (appends to existing rules).""" + try: + import oci + config = _get_oci_config(cid) + region = req.get("region") + if region: config["region"] = region + vn = oci.core.VirtualNetworkClient(config) + sl = vn.get_security_list(sl_id).data + direction = req.get("direction", "ingress").lower() + protocol = str(req.get("protocol", "6")) + source_dest = req.get("source_dest", "0.0.0.0/0") + port_min = req.get("port_min") + port_max = req.get("port_max") + description = req.get("description", "").strip() or None + is_stateless = req.get("is_stateless", False) + tcp_opts = None + udp_opts = None + if protocol == "6" and port_min is not None: + tcp_opts = oci.core.models.TcpOptions( + destination_port_range=oci.core.models.PortRange(min=int(port_min), max=int(port_max or port_min))) + elif protocol == "17" and port_min is not None: + udp_opts = oci.core.models.UdpOptions( + destination_port_range=oci.core.models.PortRange(min=int(port_min), max=int(port_max or port_min))) + ingress_rules = list(sl.ingress_security_rules or []) + egress_rules = list(sl.egress_security_rules or []) + if direction == "ingress": + ingress_rules.append(oci.core.models.IngressSecurityRule( + protocol=protocol, source=source_dest, source_type="CIDR_BLOCK", + is_stateless=is_stateless, description=description, + tcp_options=tcp_opts, udp_options=udp_opts)) + else: + egress_rules.append(oci.core.models.EgressSecurityRule( + protocol=protocol, destination=source_dest, destination_type="CIDR_BLOCK", + is_stateless=is_stateless, description=description, + tcp_options=tcp_opts, udp_options=udp_opts)) + update = oci.core.models.UpdateSecurityListDetails( + ingress_security_rules=ingress_rules, egress_security_rules=egress_rules) + vn.update_security_list(sl_id, update) + _audit(u["id"], u["username"], "seclist_add_rule", sl_id, + f"{direction} {protocol} {source_dest} port={port_min or 'ALL'}") + return {"ok": True, "message": "Regra adicionada com sucesso"} + except Exception as e: + raise HTTPException(400, str(e)[:500]) + +@app.delete("/api/oci/explore/{cid}/security_lists/{sl_id}/rules") +async def remove_security_list_rule(cid: str, sl_id: str, req: dict, u=Depends(require("admin", "user"))): + """Remove a rule from a Security List by index.""" + try: + import oci + config = _get_oci_config(cid) + region = req.get("region") + if region: config["region"] = region + vn = oci.core.VirtualNetworkClient(config) + sl = vn.get_security_list(sl_id).data + direction = req.get("direction", "ingress").lower() + rule_index = req.get("rule_index") + if rule_index is None: raise HTTPException(400, "rule_index obrigatório") + ingress_rules = list(sl.ingress_security_rules or []) + egress_rules = list(sl.egress_security_rules or []) + if direction == "ingress": + if 0 <= rule_index < len(ingress_rules): + removed = ingress_rules.pop(rule_index) + desc = f"ingress idx={rule_index} src={removed.source}" + else: + raise HTTPException(400, "Índice inválido") + else: + if 0 <= rule_index < len(egress_rules): + removed = egress_rules.pop(rule_index) + desc = f"egress idx={rule_index} dst={removed.destination}" + else: + raise HTTPException(400, "Índice inválido") + update = oci.core.models.UpdateSecurityListDetails( + ingress_security_rules=ingress_rules, egress_security_rules=egress_rules) + vn.update_security_list(sl_id, update) + _audit(u["id"], u["username"], "seclist_remove_rule", sl_id, desc) + return {"ok": True, "message": "Regra removida com sucesso"} + except Exception as e: + raise HTTPException(400, str(e)[:500]) + @app.get("/api/oci/explore/{cid}/block_volumes") async def explore_block_volumes(cid: str, compartment_id: str = Query(None), region: str = Query(None), u=Depends(current_user)): try: @@ -1029,11 +1120,97 @@ async def explore_nsgs(cid: str, compartment_id: str = Query(None), region: str _, default_comp = _explore_comp(cid) comp = compartment_id or default_comp nsgs = vn.list_network_security_groups(compartment_id=comp).data - return [{"id":n.id,"display_name":n.display_name,"vcn_id":n.vcn_id,"lifecycle_state":n.lifecycle_state, - "time_created":str(n.time_created)} for n in nsgs] + result = [] + proto_map = {"1": "ICMP", "6": "TCP", "17": "UDP", "all": "ALL"} + for n in nsgs: + rules_data = oci.pagination.list_call_get_all_results( + vn.list_network_security_group_security_rules, n.id).data + rules = [] + for r in rules_data: + ports = "" + if r.tcp_options and r.tcp_options.destination_port_range: + pr = r.tcp_options.destination_port_range + ports = str(pr.min) if pr.min == pr.max else f"{pr.min}-{pr.max}" + elif r.udp_options and r.udp_options.destination_port_range: + pr = r.udp_options.destination_port_range + ports = str(pr.min) if pr.min == pr.max else f"{pr.min}-{pr.max}" + src_dst = "" + src_dst_type = "" + if r.direction == "INGRESS": + src_dst = r.source or "" + src_dst_type = r.source_type or "" + else: + src_dst = r.destination or "" + src_dst_type = r.destination_type or "" + rules.append({ + "id": r.id, "direction": r.direction, + "protocol": proto_map.get(r.protocol, r.protocol), + "protocol_num": r.protocol, + "source_dest": src_dst, "source_dest_type": src_dst_type, + "ports": ports, "is_stateless": r.is_stateless or False, + "description": r.description or "" + }) + result.append({"id": n.id, "display_name": n.display_name, "vcn_id": n.vcn_id, + "lifecycle_state": n.lifecycle_state, "time_created": str(n.time_created), + "rule_count": len(rules), "rules": rules}) + return result except Exception as e: return {"error": str(e)[:500]} +@app.post("/api/oci/explore/{cid}/nsgs/{nsg_id}/rules") +async def add_nsg_rule(cid: str, nsg_id: str, req: dict, u=Depends(require("admin", "user"))): + """Add a security rule to an NSG.""" + try: + import oci + config = _get_oci_config(cid) + region = req.get("region") + if region: config["region"] = region + vn = oci.core.VirtualNetworkClient(config) + direction = req.get("direction", "INGRESS").upper() + protocol = str(req.get("protocol", "6")) + source_dest = req.get("source_dest", "0.0.0.0/0") + source_dest_type = req.get("source_dest_type", "CIDR_BLOCK") + port_min = req.get("port_min") + port_max = req.get("port_max") + description = req.get("description", "") + is_stateless = req.get("is_stateless", False) + rule = {"direction": direction, "protocol": protocol, "is_stateless": is_stateless, "description": description} + if direction == "INGRESS": + rule["source"] = source_dest + rule["source_type"] = source_dest_type + else: + rule["destination"] = source_dest + rule["destination_type"] = source_dest_type + if protocol == "6" and port_min is not None: + rule["tcp_options"] = oci.core.models.TcpOptions( + destination_port_range=oci.core.models.PortRange(min=int(port_min), max=int(port_max or port_min))) + elif protocol == "17" and port_min is not None: + rule["udp_options"] = oci.core.models.UdpOptions( + destination_port_range=oci.core.models.PortRange(min=int(port_min), max=int(port_max or port_min))) + details = oci.core.models.AddNetworkSecurityGroupSecurityRulesDetails( + security_rules=[oci.core.models.AddSecurityRuleDetails(**rule)]) + resp = vn.add_network_security_group_security_rules(nsg_id, details) + _audit(u["id"], u["username"], "nsg_add_rule", nsg_id, f"{direction} {protocol} {source_dest} {port_min or 'ALL'}") + added = [{"id": r.id} for r in (resp.data.security_rules or [])] + return {"ok": True, "message": "Regra adicionada com sucesso", "rules_added": added} + except Exception as e: + raise HTTPException(400, str(e)[:500]) + +@app.delete("/api/oci/explore/{cid}/nsgs/{nsg_id}/rules/{rule_id}") +async def remove_nsg_rule(cid: str, nsg_id: str, rule_id: str, region: str = Query(None), u=Depends(require("admin", "user"))): + """Remove a security rule from an NSG.""" + try: + import oci + config = _get_oci_config(cid) + if region: config["region"] = region + vn = oci.core.VirtualNetworkClient(config) + details = oci.core.models.RemoveNetworkSecurityGroupSecurityRulesDetails(security_rule_ids=[rule_id]) + vn.remove_network_security_group_security_rules(nsg_id, details) + _audit(u["id"], u["username"], "nsg_remove_rule", nsg_id, f"rule_id={rule_id}") + return {"ok": True, "message": "Regra removida com sucesso"} + except Exception as e: + raise HTTPException(400, str(e)[:500]) + @app.get("/api/oci/explore/{cid}/route_tables") async def explore_route_tables(cid: str, compartment_id: str = Query(None), region: str = Query(None), u=Depends(current_user)): try: @@ -2225,44 +2402,92 @@ def _get_adb_connection(cfg: dict): params["wallet_password"] = _dec(cfg["wallet_password_enc"]) if cfg.get("wallet_password_enc") else "" return oracledb.connect(**params) +def _resolve_embed_config(oci_config_id: str = None, genai_cfg: dict = None) -> dict: + """Resolve embedding config from genai_cfg or oci_config. Returns dict with oci_config_id, endpoint, genai_region, compartment_id.""" + if genai_cfg: + return genai_cfg + if oci_config_id: + with db() as c: + # Try linked genai config first + gc = c.execute("SELECT * FROM genai_configs WHERE oci_config_id=? ORDER BY is_default DESC, created_at DESC", + (oci_config_id,)).fetchone() + if gc: return dict(gc) + # Fallback: build from oci_config directly + oc = c.execute("SELECT * FROM oci_configs WHERE id=?", (oci_config_id,)).fetchone() + if oc: + return { + "oci_config_id": oc["id"], + "genai_region": oc["region"], + "endpoint": f"https://inference.generativeai.{oc['region']}.oci.oraclecloud.com", + "compartment_id": oc.get("compartment_id") or oc["tenancy_ocid"], + } + # Last resort: any genai config + with db() as c: + gc = c.execute("SELECT * FROM genai_configs ORDER BY is_default DESC, created_at DESC").fetchone() + if gc: return dict(gc) + # Or any oci config + oc = c.execute("SELECT * FROM oci_configs ORDER BY created_at DESC").fetchone() + if oc: + return { + "oci_config_id": oc["id"], + "genai_region": oc["region"], + "endpoint": f"https://inference.generativeai.{oc['region']}.oci.oraclecloud.com", + "compartment_id": oc.get("compartment_id") or oc["tenancy_ocid"], + } + raise HTTPException(400, "Nenhuma credencial OCI configurada para gerar embeddings.") + def _embed_text(text: str, genai_cfg: dict, embedding_model_id: str) -> list: """Generate embedding using OCI GenAI embed endpoint.""" import oci config_path = str(OCI_DIR / genai_cfg["oci_config_id"] / "config") config = oci.config.from_file(config_path, "DEFAULT") - endpoint = genai_cfg["endpoint"] + endpoint = genai_cfg.get("endpoint") or f"https://inference.generativeai.{genai_cfg.get('genai_region','us-ashburn-1')}.oci.oraclecloud.com" client = oci.generative_ai_inference.GenerativeAiInferenceClient( config=config, service_endpoint=endpoint, retry_strategy=oci.retry.NoneRetryStrategy(), timeout=(10, 120) ) embed_detail = oci.generative_ai_inference.models.EmbedTextDetails() embed_detail.inputs = [text] - # Resolve OCID for embedding model by region emb_info = EMBEDDING_MODELS.get(embedding_model_id, {}) region = genai_cfg.get("genai_region", "") emb_ref = emb_info.get("ocids", {}).get(region) or embedding_model_id embed_detail.serving_mode = oci.generative_ai_inference.models.OnDemandServingMode(model_id=emb_ref) - embed_detail.compartment_id = genai_cfg["compartment_id"] + embed_detail.compartment_id = genai_cfg.get("compartment_id", "") embed_detail.truncate = "NONE" embed_detail.input_type = "SEARCH_QUERY" response = client.embed_text(embed_detail) return response.data.embeddings[0] -def _vector_search(cfg: dict, query_embedding: list, top_k: int = 5, table_name: str = None) -> list: - """Search ADB vector store using cosine similarity. Returns top-K documents.""" +def _vector_search(cfg: dict, query_embedding: list, top_k: int = 5, table_name: str = None, tenancy: str = None) -> list: + """Search ADB vector store using cosine similarity. Returns top-K documents. + If tenancy is provided, filters METADATA JSON to match that tenancy only.""" import array table_name = table_name or cfg.get("table_name", "") conn = _get_adb_connection(cfg) try: cur = conn.cursor() vec = array.array('d', query_embedding) - cur.execute(f""" - SELECT ID, CONTENT, METADATA, SOURCE, - VECTOR_DISTANCE(EMBEDDING, :1, COSINE) AS distance - FROM {table_name} - ORDER BY distance ASC - FETCH FIRST :2 ROWS ONLY - """, [vec, top_k]) + if tenancy: + # Filter by tenancy in METADATA JSON field using LIKE for broad compatibility + # Matches both structured JSON {"tenancy":"X",...} and legacy "tenancy: X, ..." + tenancy_filter = f'%"tenancy":"{tenancy}"%' + tenancy_filter_legacy = f'%tenancy: {tenancy}%' + cur.execute(f""" + SELECT ID, TEXT, METADATA, + VECTOR_DISTANCE(EMBEDDING, :1, COSINE) AS distance + FROM "{table_name}" + WHERE (METADATA LIKE :3 OR METADATA LIKE :4) + ORDER BY distance ASC + FETCH FIRST :2 ROWS ONLY + """, [vec, top_k, tenancy_filter, tenancy_filter_legacy]) + else: + cur.execute(f""" + SELECT ID, TEXT, METADATA, + VECTOR_DISTANCE(EMBEDDING, :1, COSINE) AS distance + FROM "{table_name}" + ORDER BY distance ASC + FETCH FIRST :2 ROWS ONLY + """, [vec, top_k]) results = [] for row in cur: content = row[1] @@ -2270,7 +2495,7 @@ def _vector_search(cfg: dict, query_embedding: list, top_k: int = 5, table_name: content = content.read() results.append({ "id": row[0], "content": content or "", - "metadata": row[2], "source": row[3], "distance": float(row[4]) + "metadata": row[2], "distance": float(row[3]) }) cur.close() return results @@ -2291,15 +2516,15 @@ def _build_rag_context(documents: list) -> str: return "\n\n---\n\n".join(parts) def _get_active_adb_configs(user_id: str) -> list[dict]: - """Get all active ADB vector configs with a linked GenAI config.""" + """Get all active ADB vector configs. GenAI config is resolved automatically if not linked.""" with db() as c: rows = c.execute( - "SELECT * FROM adb_vector_configs WHERE user_id=? AND is_active=1 AND genai_config_id IS NOT NULL ORDER BY created_at DESC", + "SELECT * FROM adb_vector_configs WHERE user_id=? AND is_active=1 ORDER BY created_at DESC", (user_id,) ).fetchall() if not rows: rows = c.execute( - "SELECT * FROM adb_vector_configs WHERE is_active=1 AND genai_config_id IS NOT NULL ORDER BY created_at DESC" + "SELECT * FROM adb_vector_configs WHERE is_active=1 ORDER BY created_at DESC" ).fetchall() return [dict(r) for r in rows] @@ -2623,6 +2848,50 @@ async def test_adb(vid: str, u=Depends(require("admin","user"))): _config_log("adb", vid, cname, "error", "test", msg, u["id"], u["username"]) return {"status":"error","message":msg} +@app.post("/api/adb/{vid}/tables/check") +async def check_adb_tables(vid: str, u=Depends(require("admin","user"))): + """Check which registered tables exist in ADB and list all user tables.""" + with db() as c: + cfg = c.execute("SELECT * FROM adb_vector_configs WHERE id=?", (vid,)).fetchone() + if not cfg: raise HTTPException(404) + registered = _get_tables_for_config(vid, active_only=False) + reg_names_upper = {t["table_name"].upper() for t in registered} + try: + conn = _get_adb_connection(dict(cfg)) + cur = conn.cursor() + cur.execute("SELECT USER FROM DUAL") + current_user_name = cur.fetchone()[0] + cur.execute("SELECT table_name FROM user_tables ORDER BY table_name") + adb_tables = [r[0] for r in cur.fetchall()] + adb_lookup = {t.upper(): t for t in adb_tables} + results = [] + for t in registered: + tname_reg = t["table_name"] + tname_upper = tname_reg.upper() + real_name = adb_lookup.get(tname_upper) + if real_name: + try: + cur.execute(f'SELECT COUNT(*) FROM "{real_name}"') + cnt = cur.fetchone()[0] + case_note = f" (ADB: {real_name})" if real_name != tname_reg else "" + results.append({"table": tname_reg, "adb_name": real_name, "status": "ok", "rows": cnt, "message": f"{cnt} registros{case_note}"}) + except Exception as e: + results.append({"table": tname_reg, "status": "error", "message": str(e)[:200]}) + else: + results.append({"table": tname_reg, "status": "not_found", "message": f"Não existe no schema {current_user_name}"}) + unregistered = [t for t in adb_tables if t.upper() not in reg_names_upper] + cur.close() + conn.close() + return { + "status": "success", + "schema_user": current_user_name, + "adb_tables_total": len(adb_tables), + "registered": results, + "unregistered_in_adb": unregistered[:50] + } + except Exception as e: + return {"status": "error", "message": f"Erro de conexão: {str(e)[:500]}"} + @app.delete("/api/adb/configs/{vid}") async def del_adb(vid: str, u=Depends(require("admin","user"))): with db() as c: c.execute("DELETE FROM adb_vector_configs WHERE id=?", (vid,)) @@ -2668,7 +2937,7 @@ async def update_adb( _config_log("adb", vid, config_name, "success", "save", f"Conexão atualizada: {config_name} ({dsn})", u["id"], u["username"]) return {"id": vid, "config_name": config_name} -_TABLE_NAME_RE = re.compile(r'^[A-Z][A-Z0-9_]{0,127}$') +_TABLE_NAME_RE = re.compile(r'^[A-Za-z][A-Za-z0-9_]{0,127}$') @app.get("/api/adb/{vid}/tables") async def list_adb_tables(vid: str, u=Depends(current_user)): @@ -2679,7 +2948,7 @@ async def list_adb_tables(vid: str, u=Depends(current_user)): @app.post("/api/adb/{vid}/tables") async def add_adb_table(vid: str, req: dict, u=Depends(require("admin","user"))): - table_name = req.get("table_name", "").strip().upper() + table_name = req.get("table_name", "").strip() description = req.get("description", "").strip() if not table_name: raise HTTPException(400, "table_name é obrigatório") if not _TABLE_NAME_RE.match(table_name): raise HTTPException(400, "Nome da tabela inválido. Use letras maiúsculas, números e underscores.") @@ -2712,7 +2981,7 @@ async def update_adb_table(vid: str, tid: str, req: dict, u=Depends(require("adm if not row: raise HTTPException(404, "Table entry not found") sets, vals = [], [] if "table_name" in req: - tn = req["table_name"].strip().upper() + tn = req["table_name"].strip() if not tn: raise HTTPException(400, "table_name é obrigatório") if not _TABLE_NAME_RE.match(tn): raise HTTPException(400, "Nome da tabela inválido") sets.append("table_name=?"); vals.append(tn) @@ -2743,11 +3012,45 @@ async def ingest_documents(vid: str, req: IngestDocReq, bg: BackgroundTasks, u=D bg.add_task(_ingest_documents_task, cfg, dict(gc), req.documents, u["id"], u["username"], table_name=req.table_name) return {"ok": True, "message": f"Ingestão iniciada para {len(req.documents)} documentos", "config_id": vid} -def _ingest_documents_task(cfg: dict, genai_cfg: dict, documents: list, user_id: str, username: str, table_name: str = None): - """Background task: embed and insert documents into ADB via OCI GenAI.""" +def _build_metadata_json(tenancy: str = "", compartments: str = "", section: str = "", + report_date: str = "", extra: dict = None) -> str: + """Build a structured JSON metadata string for vector embeddings.""" + meta = {} + if tenancy: + meta["tenancy"] = tenancy + if compartments: + meta["compartments"] = compartments + if section: + meta["section"] = section + if report_date: + meta["report_date"] = report_date + if extra: + meta.update(extra) + return json.dumps(meta, ensure_ascii=False) if meta else "" + + +def _auto_register_table(adb_config_id: str, table_name: str, description: str = ""): + """Auto-register a table in adb_vector_tables if not already present.""" + if not table_name: + return + with db() as c: + exists = c.execute("SELECT 1 FROM adb_vector_tables WHERE adb_config_id=? AND table_name=? COLLATE NOCASE", + (adb_config_id, table_name)).fetchone() + if not exists: + c.execute("INSERT INTO adb_vector_tables (id, adb_config_id, table_name, description) VALUES (?,?,?,?)", + (str(uuid.uuid4()), adb_config_id, table_name, description)) + log.info(f"Auto-registered table '{table_name}' for ADB config {adb_config_id}") + +def _ingest_documents_task(cfg: dict, genai_cfg: dict, documents: list, user_id: str, username: str, + table_name: str = None, tenancy: str = None, compartments: str = None, + report_date: str = None): + """Background task: embed and insert documents into ADB via OCI GenAI. + Tenancy and compartments are stored in METADATA as structured JSON for filtering.""" import array emb_model = cfg.get("embedding_model_id", "cohere.embed-v4.0") table_name = table_name or cfg.get("table_name", "") + # Auto-register table so it appears in multi-table RAG search + _auto_register_table(cfg["id"], table_name) conn = _get_adb_connection(cfg) try: cur = conn.cursor() @@ -2758,18 +3061,31 @@ def _ingest_documents_task(cfg: dict, genai_cfg: dict, documents: list, user_id: if not content: continue embedding = _embed_text(content, genai_cfg, emb_model) vec = array.array('d', embedding) + # Build structured metadata with tenancy isolation + doc_tenancy = tenancy or doc.get("tenancy", "") + doc_compartments = compartments or doc.get("compartments", "") + metadata = _build_metadata_json( + tenancy=doc_tenancy, + compartments=doc_compartments, + section=doc.get("section", ""), + report_date=report_date or "", + extra={"legacy_metadata": doc.get("metadata", "")} if doc.get("metadata") else None + ) cur.execute(f""" - INSERT INTO {table_name} (ID, CONTENT, EMBEDDING, METADATA, SOURCE) - VALUES (:1, :2, :3, :4, :5) - """, [str(uuid.uuid4()), content, vec, doc.get("metadata", ""), doc.get("source", "manual_upload")]) + INSERT INTO "{table_name}" (ID, TEXT, EMBEDDING, METADATA) + VALUES (:1, :2, :3, :4) + """, [str(uuid.uuid4()), content, vec, metadata]) inserted += 1 except Exception as e: log.error(f"Failed to ingest document: {e}") conn.commit() cur.close() - log.info(f"Ingested {inserted}/{len(documents)} documents into {table_name}") + log.info(f"Ingested {inserted}/{len(documents)} documents into {table_name}" + + (f" (tenancy={tenancy})" if tenancy else "")) _audit(user_id, username, "ingest_documents", cfg["id"], f"{inserted} documents") - _config_log("adb", cfg["id"], cfg.get("config_name"), "success", "ingest", f"{inserted}/{len(documents)} documentos ingeridos em {table_name}", user_id, username) + _config_log("adb", cfg["id"], cfg.get("config_name"), "success", "ingest", + f"{inserted}/{len(documents)} documentos ingeridos em {table_name}" + + (f" (tenancy: {tenancy})" if tenancy else ""), user_id, username) except Exception as e: log.error(f"Ingestion task failed: {e}") _config_log("adb", cfg["id"], cfg.get("config_name"), "error", "ingest", str(e)[:500], user_id, username) @@ -2781,6 +3097,8 @@ def _chunk_report_by_section(report_data: dict) -> list: """Chunk a CIS report into documents grouped by section.""" if isinstance(report_data, str): report_data = json.loads(report_data) + if isinstance(report_data, list): + report_data = {"findings": {str(i): item for i, item in enumerate(report_data)}, "tenancy": "unknown"} findings = report_data.get("findings", {}) tenancy = report_data.get("tenancy", "unknown") generated_at = report_data.get("generated_at", "") @@ -2813,6 +3131,9 @@ def _chunk_report_by_section(report_data: dict) -> list: documents.append({ "content": "\n".join(lines), "source": f"CIS Report - {tenancy} - {generated_at}", + "section": section_name, + "tenancy": tenancy, + "compartments": ", ".join(compartments[:50]), "metadata": f"tenancy: {tenancy}, section: {section_name}, total: {len(checks)}, passed: {passed}, failed: {failed}, review: {review}" }) return documents @@ -2834,18 +3155,20 @@ def _chunk_text_file(text: str, filename: str, chunk_size: int = 1000) -> list: documents.append({"content": current_chunk, "source": filename, "metadata": f"chunk: {chunk_num}"}) return documents -def _get_adb_and_genai(vid: str): - """Load ADB config and its linked GenAI config. Returns (adb_cfg, genai_cfg) or raises.""" +def _get_adb_and_genai(vid: str, oci_config_id: str = None): + """Load ADB config and resolve embed config. + Priority: ADB.genai_config_id → genai by oci_config_id → oci_config directly → any available.""" with db() as c: cfg = c.execute("SELECT * FROM adb_vector_configs WHERE id=?", (vid,)).fetchone() if not cfg: raise HTTPException(404, "ADB config not found") cfg = dict(cfg) - if not cfg.get("genai_config_id"): - raise HTTPException(400, "GenAI config not linked to this ADB connection") - with db() as c: - gc = c.execute("SELECT * FROM genai_configs WHERE id=?", (cfg["genai_config_id"],)).fetchone() - if not gc: raise HTTPException(400, "Linked GenAI config not found") - return cfg, dict(gc) + genai_cfg = None + if cfg.get("genai_config_id"): + with db() as c: + row = c.execute("SELECT * FROM genai_configs WHERE id=?", (cfg["genai_config_id"],)).fetchone() + if row: genai_cfg = dict(row) + gc = _resolve_embed_config(oci_config_id=oci_config_id, genai_cfg=genai_cfg) + return cfg, gc @app.get("/api/embeddings/preview/{rid}") async def preview_report_chunks(rid: str, u=Depends(current_user)): @@ -2861,9 +3184,10 @@ async def preview_report_chunks(rid: str, u=Depends(current_user)): except Exception: raise HTTPException(400, "Invalid report data") documents = _chunk_report_by_section(report_data) - return {"tenancy": report_data.get("tenancy", "unknown"), - "regions": report_data.get("regions", []), - "compartments": report_data.get("compartments", []), + rd = report_data if isinstance(report_data, dict) else {} + return {"tenancy": rd.get("tenancy", "unknown"), + "regions": rd.get("regions", []), + "compartments": rd.get("compartments", []), "total_chunks": len(documents), "chunks": documents} @@ -2872,7 +3196,7 @@ async def embed_report(rid: str, req: dict, bg: BackgroundTasks, u=Depends(requi vid = req.get("adb_config_id") if not vid: raise HTTPException(400, "adb_config_id is required") with db() as c: - r = c.execute("SELECT json_path,tenancy_name FROM reports WHERE id=? AND status='completed'", (rid,)).fetchone() + r = c.execute("SELECT json_path,tenancy_name,config_id FROM reports WHERE id=? AND status='completed'", (rid,)).fetchone() if not r: raise HTTPException(404, "Report not found or not completed") json_path = r["json_path"] if not json_path or not Path(json_path).exists(): @@ -2883,17 +3207,63 @@ async def embed_report(rid: str, req: dict, bg: BackgroundTasks, u=Depends(requi raise HTTPException(400, "Invalid report data") documents = _chunk_report_by_section(report_data) if not documents: raise HTTPException(400, "No sections found in report") - cfg, gc = _get_adb_and_genai(vid) + # Optional section filter — embed only a specific section + section_filter = req.get("section") + if section_filter: + documents = [d for d in documents if d.get("section") == section_filter] + if not documents: raise HTTPException(400, f"Section '{section_filter}' not found in report") + cfg, gc = _get_adb_and_genai(vid, oci_config_id=r.get("config_id")) target_table = req.get("table_name") or None - bg.add_task(_ingest_documents_task, cfg, gc, documents, u["id"], u["username"], table_name=target_table) - _audit(u["id"], u["username"], "embed_report", rid, f"{len(documents)} sections") - return {"ok": True, "message": f"Embedding de {len(documents)} seções iniciado", "sections": len(documents)} + # Extract tenancy and compartments for isolation + tenancy = report_data.get("tenancy", r["tenancy_name"] or "unknown") + report_date = report_data.get("generated_at", "") + compartments_list = report_data.get("compartments", []) + compartments_str = json.dumps(compartments_list) if compartments_list else "" + bg.add_task(_ingest_documents_task, cfg, gc, documents, u["id"], u["username"], + table_name=target_table, tenancy=tenancy, compartments=compartments_str, + report_date=report_date) + label = f"section={section_filter}" if section_filter else f"{len(documents)} sections" + _audit(u["id"], u["username"], "embed_report", rid, f"{label}, tenancy={tenancy}") + return {"ok": True, "message": f"Embedding de {len(documents)} seção(ões) iniciado (tenancy: {tenancy})", "sections": len(documents), "tenancy": tenancy} + +def _extract_pdf_text(file_bytes: bytes) -> str: + """Extract text from a PDF file using PyPDF2 or pdfplumber.""" + try: + import PyPDF2 + import io + reader = PyPDF2.PdfReader(io.BytesIO(file_bytes)) + pages = [] + for page in reader.pages: + text = page.extract_text() + if text: + pages.append(text.strip()) + return "\n\n".join(pages) + except ImportError: + pass + try: + import pdfplumber + import io + pages = [] + with pdfplumber.open(io.BytesIO(file_bytes)) as pdf: + for page in pdf.pages: + text = page.extract_text() + if text: + pages.append(text.strip()) + return "\n\n".join(pages) + except ImportError: + raise HTTPException(400, "PDF support requires PyPDF2 or pdfplumber. Install: pip install PyPDF2") @app.post("/api/embeddings/upload") async def embed_upload(adb_config_id: str = Form(...), table_name: str = Form(""), file: UploadFile = File(...), bg: BackgroundTasks = None, u=Depends(require("admin","user"))): - if not file.filename.lower().endswith(".txt"): - raise HTTPException(400, "Only .txt files are supported") - content = (await file.read()).decode("utf-8", errors="replace") + fname = file.filename.lower() + allowed = ('.txt', '.pdf', '.csv', '.json', '.md') + if not any(fname.endswith(ext) for ext in allowed): + raise HTTPException(400, f"Formatos aceitos: {', '.join(allowed)}") + raw = await file.read() + if fname.endswith('.pdf'): + content = _extract_pdf_text(raw) + else: + content = raw.decode("utf-8", errors="replace") if not content.strip(): raise HTTPException(400, "File is empty") documents = _chunk_text_file(content, file.filename) if not documents: raise HTTPException(400, "No content chunks found") @@ -2903,6 +3273,52 @@ async def embed_upload(adb_config_id: str = Form(...), table_name: str = Form("" _audit(u["id"], u["username"], "embed_upload", file.filename, f"{len(documents)} chunks") return {"ok": True, "message": f"Embedding de {len(documents)} chunks iniciado", "chunks": len(documents), "filename": file.filename} +def _extract_text_from_html(html: str) -> str: + """Extract readable text from HTML, stripping tags and scripts.""" + import re as _re + text = _re.sub(r']*>[\s\S]*?', ' ', html, flags=_re.IGNORECASE) + text = _re.sub(r']*>[\s\S]*?', ' ', text, flags=_re.IGNORECASE) + text = _re.sub(r'<[^>]+>', ' ', text) + text = _re.sub(r'&[a-zA-Z]+;', ' ', text) + text = _re.sub(r'&#\d+;', ' ', text) + text = _re.sub(r'\s+', ' ', text).strip() + return text + +@app.post("/api/embeddings/upload-url") +async def embed_upload_url( + adb_config_id: str = Form(...), + table_name: str = Form(""), + url: str = Form(...), + bg: BackgroundTasks = None, + u=Depends(require("admin", "user")) +): + import requests as req + url = url.strip() + if not url.startswith(("http://", "https://")): + raise HTTPException(400, "URL inválida — deve começar com http:// ou https://") + try: + resp = req.get(url, timeout=30, headers={"User-Agent": "Mozilla/5.0 OCI-CIS-Agent/1.0"}) + resp.raise_for_status() + except Exception as e: + raise HTTPException(400, f"Erro ao acessar URL: {str(e)[:300]}") + ct = resp.headers.get("content-type", "") + if "pdf" in ct: + content = _extract_pdf_text(resp.content) + elif "html" in ct or "text" in ct: + content = _extract_text_from_html(resp.text) + else: + content = resp.text + if not content or not content.strip(): + raise HTTPException(400, "Nenhum conteúdo extraído da URL") + documents = _chunk_text_file(content, url) + if not documents: + raise HTTPException(400, "Nenhum chunk gerado do conteúdo") + cfg, gc = _get_adb_and_genai(adb_config_id) + target_table = table_name.strip() or None + bg.add_task(_ingest_documents_task, cfg, gc, documents, u["id"], u["username"], table_name=target_table) + _audit(u["id"], u["username"], "embed_url", url, f"{len(documents)} chunks") + return {"ok": True, "message": f"Embedding de {len(documents)} chunks iniciado", "chunks": len(documents), "url": url} + @app.get("/api/embeddings/{vid}/list") async def list_embeddings(vid: str, table_name: str = Query(""), limit: int = Query(50), offset: int = Query(0), u=Depends(current_user)): with db() as c: @@ -2913,17 +3329,18 @@ async def list_embeddings(vid: str, table_name: str = Query(""), limit: int = Qu cur = conn.cursor() table_name = table_name.strip() or cfg["table_name"] if not table_name: raise HTTPException(400, "Nenhuma tabela selecionada") - cur.execute(f"SELECT COUNT(*) FROM {table_name}") + cur.execute(f'SELECT COUNT(*) FROM "{table_name}"') total = cur.fetchone()[0] cur.execute(f""" - SELECT ID, SOURCE, METADATA, CREATED_AT FROM {table_name} - ORDER BY CREATED_AT DESC + SELECT ID, METADATA FROM "{table_name}" OFFSET :1 ROWS FETCH NEXT :2 ROWS ONLY """, [offset, limit]) rows = [] for row in cur: - rows.append({"id": row[0], "source": row[1], "metadata": row[2], - "created_at": str(row[3]) if row[3] else None}) + rid = row[0].hex() if isinstance(row[0], bytes) else str(row[0]) + meta = row[1] + if hasattr(meta, 'read'): meta = meta.read() + rows.append({"id": rid, "metadata": meta}) cur.close(); conn.close() return {"total": total, "offset": offset, "limit": limit, "documents": rows} except Exception as e: @@ -2939,13 +3356,42 @@ async def delete_embedding(vid: str, doc_id: str, table_name: str = Query(""), u cur = conn.cursor() table_name = table_name.strip() or cfg["table_name"] if not table_name: raise HTTPException(400, "Nenhuma tabela selecionada") - cur.execute(f"DELETE FROM {table_name} WHERE ID = :1", [doc_id]) + cur.execute(f'DELETE FROM "{table_name}" WHERE ID = :1', [doc_id]) conn.commit() cur.close(); conn.close() return {"ok": True} except Exception as e: raise HTTPException(500, f"Erro ao deletar: {str(e)[:500]}") +@app.post("/api/embeddings/{vid}/purge") +async def purge_embeddings(vid: str, req: dict, u=Depends(require("admin","user"))): + """Delete old embeddings from a table, optionally filtered by tenancy.""" + table_name = req.get("table_name", "").strip() + tenancy = req.get("tenancy", "").strip() + if not table_name: raise HTTPException(400, "table_name é obrigatório") + with db() as c: + cfg = c.execute("SELECT * FROM adb_vector_configs WHERE id=?", (vid,)).fetchone() + if not cfg: raise HTTPException(404) + try: + conn = _get_adb_connection(dict(cfg)) + cur = conn.cursor() + if tenancy: + cur.execute(f'SELECT COUNT(*) FROM "{table_name}" WHERE METADATA LIKE :1', + [f'%"tenancy":"{tenancy}"%']) + count = cur.fetchone()[0] + cur.execute(f'DELETE FROM "{table_name}" WHERE METADATA LIKE :1', + [f'%"tenancy":"{tenancy}"%']) + else: + cur.execute(f'SELECT COUNT(*) FROM "{table_name}"') + count = cur.fetchone()[0] + cur.execute(f'DELETE FROM "{table_name}"') + conn.commit() + cur.close(); conn.close() + _audit(u["id"], u["username"], "purge_embeddings", vid, f"table={table_name}, tenancy={tenancy or 'ALL'}, deleted={count}") + return {"ok": True, "deleted": count, "table": table_name, "tenancy": tenancy or "ALL"} + except Exception as e: + raise HTTPException(500, f"Erro ao limpar: {str(e)[:500]}") + # ── Reports ─────────────────────────────────────────────────────────────────── def _classify_report_file(fname: str) -> str: @@ -3158,7 +3604,9 @@ async def list_report_files(rid: str, u=Depends(current_user)): return [dict(f) for f in files] @app.get("/api/reports/{rid}/files/{fid}/download") -async def download_report_file(rid: str, fid: str, u=Depends(current_user)): +async def download_report_file(rid: str, fid: str, token: str = Query(None), cred: HTTPAuthorizationCredentials = Depends(HTTPBearer(auto_error=False))): + u = _user_from_token(token) if token else (await current_user(cred) if cred else None) + if not u: raise HTTPException(401, "Not authenticated") with db() as c: r = c.execute("SELECT user_id FROM reports WHERE id=?", (rid,)).fetchone() if not r: raise HTTPException(404) @@ -3176,7 +3624,9 @@ async def report_html(rid): return FileResponse(r["html_path"], media_type="text/html") @app.get("/api/reports/{rid}/download") -async def report_dl(rid, fmt: str = Query("json"), u=Depends(current_user)): +async def report_dl(rid, fmt: str = Query("json"), token: str = Query(None), cred: HTTPAuthorizationCredentials = Depends(HTTPBearer(auto_error=False))): + u = _user_from_token(token) if token else (await current_user(cred) if cred else None) + if not u: raise HTTPException(401, "Not authenticated") with db() as c: r=c.execute("SELECT * FROM reports WHERE id=?",(rid,)).fetchone() if not r: raise HTTPException(404) p = r["json_path"] if fmt=="json" else r["html_path"] @@ -3265,23 +3715,37 @@ async def _chat_background(mid: str, sid: str, msg: ChatMsg, user: dict, genai_c history = [{"role":r["role"],"content":r["content"]} for r in prev] # ── RAG: augment with vector context from ALL active ADB configs ── + # Resolve active tenancy for filtered vector search + rag_tenancy = None + active_oci_for_rag = genai_cfg.get("oci_config_id") or (msg.oci_config_id if hasattr(msg, 'oci_config_id') and msg.oci_config_id else None) + if active_oci_for_rag: + with db() as c: + oci_for_rag = c.execute("SELECT tenancy_name FROM oci_configs WHERE id=?", (active_oci_for_rag,)).fetchone() + if oci_for_rag: + rag_tenancy = oci_for_rag["tenancy_name"] + log.info(f"RAG: filtering by tenancy '{rag_tenancy}'") + rag_context = "" adb_cfgs = _get_active_adb_configs(user["id"]) if adb_cfgs: all_documents = [] for adb_cfg in adb_cfgs: try: - with db() as c: - emb_genai = c.execute("SELECT * FROM genai_configs WHERE id=?", (adb_cfg["genai_config_id"],)).fetchone() + genai_linked = None + if adb_cfg.get("genai_config_id"): + with db() as c: + row = c.execute("SELECT * FROM genai_configs WHERE id=?", (adb_cfg["genai_config_id"],)).fetchone() + if row: genai_linked = dict(row) + emb_genai = _resolve_embed_config(oci_config_id=active_oci_for_rag, genai_cfg=genai_linked) if emb_genai: emb_model = adb_cfg.get("embedding_model_id", "cohere.embed-v4.0") - query_embedding = _embed_text(msg.message, dict(emb_genai), emb_model) + query_embedding = _embed_text(msg.message, emb_genai, emb_model) tables = _get_tables_for_config(adb_cfg["id"], active_only=True) if not tables: tables = [{"table_name": adb_cfg.get("table_name", "")}] for tbl in tables: try: - documents = _vector_search(adb_cfg, query_embedding, top_k=5, table_name=tbl["table_name"]) + documents = _vector_search(adb_cfg, query_embedding, top_k=5, table_name=tbl["table_name"], tenancy=rag_tenancy) if documents: for doc in documents: doc["source"] = f"{doc.get('source', 'unknown')} [{tbl['table_name']}]" @@ -3993,7 +4457,9 @@ async def tf_workspace_status(wid: str, u=Depends(current_user)): @app.get("/api/terraform/workspaces/{wid}/download") -async def tf_download(wid: str, u=Depends(current_user)): +async def tf_download(wid: str, token: str = Query(None), cred: HTTPAuthorizationCredentials = Depends(HTTPBearer(auto_error=False))): + u = _user_from_token(token) if token else (await current_user(cred) if cred else None) + if not u: raise HTTPException(401, "Not authenticated") with db() as c: r = c.execute("SELECT tf_code, name FROM terraform_workspaces WHERE id=? AND user_id=?", (wid, u["id"])).fetchone() if not r or not r["tf_code"]: raise HTTPException(404, "No code found") diff --git a/backend/requirements.txt b/backend/requirements.txt index 894174d..926c86c 100644 --- a/backend/requirements.txt +++ b/backend/requirements.txt @@ -7,3 +7,4 @@ oci==2.133.0 oracledb==2.4.1 mcp>=1.0.0 requests>=2.31.0 +PyPDF2>=3.0.0 diff --git a/frontend/index.html b/frontend/index.html index 1873c6a..0a04f9c 100644 --- a/frontend/index.html +++ b/frontend/index.html @@ -526,7 +526,7 @@ const S={user:null,token:null,tab:'chat',msgs:[],sid:null,reports:[],ociCfg:[],g chatParams:{temperature:1,max_tokens:6000,top_p:0.95,top_k:1,frequency_penalty:0,presence_penalty:0},chatPanel:'',chatUseTools:true, chatPrompts:[],editingPrompt:null,trackingReportId:null,ociRegions:{},rptSelRegions:[],rptRegionsOpen:false,rptRegionFilter:'', rptOciOpen:false,rptOciFilter:'',rptOciVal:'',rptRselOpen:false,rptRselFilter:'',rptRselVal:'',ociFormRegOpen:false,ociFormRegFilter:'',ociFormRegVal:'', - rptLevel:2,rptObp:false,rptRaw:false,rptRedact:false,reportFiles:{},dlExpandedRid:null,dlTenancyFilter:'', + rptLevel:2,rptObp:false,rptRaw:false,rptRedact:false,reportFiles:{},dlExpandedRid:null,dlTenancyFilter:'',dlSectionFilter:'', cisVer:null,cisCheckResult:null,cisUpdating:false,rptHistFilter:'',rptKpi:null,rptKpiLoading:false,chatFiles:[], tfMsgs:[],tfSid:null,tfModel:'',tfOci:'',tfRegion:'',tfCompartment:'',tfPlan:[],tfCode:'',tfPanel:'', tfWs:null,tfWsList:[],tfPlanOut:'',tfApplyOut:'',tfDestroyOut:'',tfStatus:'draft',tfRunning:false,tfConfirmDest:false,tfMdOpen:false, @@ -647,7 +647,7 @@ function rChat(){ else if(S.chatModel){const mi=S.models[S.chatModel];curLabel=mi?mi.name:S.chatModel} const isDirect=S.chatModel&&!S.chatModel.startsWith('cfg:'); const ociSel=isDirect?``:''; - const ragBadge=S.adbCfg.some(c=>c.genai_config_id&&c.is_active)?'RAG':''; + const ragBadge=S.adbCfg.some(c=>c.is_active)&&(S.genaiCfg.length||S.ociCfg.length)?'RAG':''; const toolCount=S.mcpSvr.filter(m=>m.is_active&&Array.isArray(m.tools)&&m.tools.length).reduce((a,m)=>a+m.tools.length,0); let sideContent=''; if(S.chatPanel==='config')sideContent=rChatConfig(isDirect,toolCount); @@ -1729,6 +1729,7 @@ async function expLoadRes(){ function renderExpData(d){if(!d)return'

Selecione um compartment na árvore.

'; if(d.error)return`
${d.error}
`;if(!Array.isArray(d)||!d.length)return'

Nenhum recurso encontrado neste compartment.

'; if(S.expResType==='security_lists')return rExpSecLists(d); + if(S.expResType==='nsgs')return rExpNsgs(d); if(S.expResType==='route_tables')return rExpRouteTables(d); const isInst=S.expResType==='instances'; const isAdb=S.expResType==='databases'; @@ -1750,18 +1751,119 @@ function rExpSecLists(d){ const ingress=rules.filter(r=>r.direction==='ingress'); const egress=rules.filter(r=>r.direction==='egress'); const expanded=S._expExpanded&&S._expExpanded[idx]; + const showForm=S._slAddForm===sl.id; return`
${sl.display_name} ${sl.lifecycle_state}
${expanded?'▼':'▶'} ${ingress.length} ingress · ${egress.length} egress
id: ${sl.id}
-${expanded?`
${ingress.length?`
↓ Ingress (${ingress.length})
- -${ingress.map(r=>``).join('')} +${expanded?`
+
+${showForm?`
+
${IC.shield} Nova Regra
+
+
+
+
+
+
+
+
+
+
+
+
+
`:''} +${ingress.length?`
↓ Ingress (${ingress.length})
+
ProtocoloOrigemPortasStatelessDescrição
${r.protocol}${r.source_dest}${r.ports||'ALL'}${r.stateless?'Sim':'Não'}${r.description||'—'}
+${ingress.map(r=>``).join('')}
ProtocoloOrigemPortasStatelessDescrição
${r.protocol}${r.source_dest}${r.ports||'ALL'}${r.stateless?'Sim':'Não'}${r.description||'—'}
`:''} ${egress.length?`
↑ Egress (${egress.length})
- -${egress.map(r=>``).join('')} +
ProtocoloDestinoPortasStatelessDescrição
${r.protocol}${r.source_dest}${r.ports||'ALL'}${r.stateless?'Sim':'Não'}${r.description||'—'}
+${egress.map(r=>``).join('')}
ProtocoloDestinoPortasStatelessDescrição
${r.protocol}${r.source_dest}${r.ports||'ALL'}${r.stateless?'Sim':'Não'}${r.description||'—'}
`:''}
`:''}
`}).join('')}`} +async function slAddRule(slId){ + const dir=document.getElementById('slDir').value; + const proto=document.getElementById('slProto').value; + const cidr=document.getElementById('slCidr').value.trim(); + const portMin=document.getElementById('slPortMin').value; + const portMax=document.getElementById('slPortMax').value; + const desc=document.getElementById('slDesc').value; + const stateless=document.getElementById('slStateless').value==='true'; + if(!cidr){sm('slFormMsg','Informe o CIDR.','e');return} + const region=S.expSelRegions.length===1?S.expSelRegions[0]:null; + const body={direction:dir,protocol:proto,source_dest:cidr,description:desc,is_stateless:stateless,region:region||undefined}; + if(proto!=='all'&&proto!=='1'&&portMin)Object.assign(body,{port_min:parseInt(portMin),port_max:parseInt(portMax||portMin)}); + try{const d=await $api('/oci/explore/'+S.expCfg+'/security_lists/'+slId+'/rules',{method:'POST',body}); + sm('slFormMsg',IC.ok+' '+d.message,'s');S._slAddForm=null;await expLoadRes()}catch(e){sm('slFormMsg',e.message,'e')}} +async function slRemoveRule(slId,direction,ruleIndex){ + if(!confirm('Remover esta regra da Security List?'))return; + const region=S.expSelRegions.length===1?S.expSelRegions[0]:null; + try{await $api('/oci/explore/'+S.expCfg+'/security_lists/'+slId+'/rules',{method:'DELETE',body:{direction,rule_index:ruleIndex,region:region||undefined}});await expLoadRes()}catch(e){alert('Erro: '+e.message)}} +async function slAddMyIp(slId){ + try{const resp=await fetch('https://api.ipify.org?format=json');const data=await resp.json();const myIp=data.ip+'/32'; + document.getElementById('slCidr').value=myIp; + document.getElementById('slDesc').value='Acesso temporário - '+myIp; + sm('slFormMsg',IC.ok+' IP detectado: '+myIp+' — ajuste porta e clique Salvar','s')}catch(e){sm('slFormMsg','Não foi possível detectar o IP: '+e.message,'e')}} +function rExpNsgs(d){ + return`
${d.length} NSG(s)
${d.map((nsg,idx)=>{ + const rules=nsg.rules||[]; + const ingress=rules.filter(r=>r.direction==='INGRESS'); + const egress=rules.filter(r=>r.direction==='EGRESS'); + const expanded=S._expExpanded&&S._expExpanded['nsg'+idx]; + const showForm=S._nsgAddForm===nsg.id; + return`
+
${nsg.display_name} ${nsg.lifecycle_state}
+
${expanded?'▼':'▶'} ${ingress.length} ingress · ${egress.length} egress
+
id: ${nsg.id}
+${expanded?`
+
+${showForm?`
+
${IC.shield} Nova Regra
+
+
+
+
+
+
+
+
+
+
+
+
+
`:''} +${ingress.length?`
↓ Ingress (${ingress.length})
+ +${ingress.map(r=>``).join('')} +
ProtocoloOrigemPortasStatelessDescrição
${r.protocol}${r.source_dest}${r.ports||'ALL'}${r.is_stateless?'Sim':'Não'}${r.description||'—'}
`:''} +${egress.length?`
↑ Egress (${egress.length})
+ +${egress.map(r=>``).join('')} +
ProtocoloDestinoPortasStatelessDescrição
${r.protocol}${r.source_dest}${r.ports||'ALL'}${r.is_stateless?'Sim':'Não'}${r.description||'—'}
`:''}
`:''}
`}).join('')}`} +async function nsgAddRule(nsgId){ + const dir=document.getElementById('nsgDir').value; + const proto=document.getElementById('nsgProto').value; + const cidr=document.getElementById('nsgCidr').value.trim(); + const portMin=document.getElementById('nsgPortMin').value; + const portMax=document.getElementById('nsgPortMax').value; + const desc=document.getElementById('nsgDesc').value; + const stateless=document.getElementById('nsgStateless').value==='true'; + if(!cidr){sm('nsgFormMsg','Informe o CIDR.','e');return} + const region=S.expSelRegions.length===1?S.expSelRegions[0]:null; + const body={direction:dir,protocol:proto,source_dest:cidr,source_dest_type:'CIDR_BLOCK',description:desc,is_stateless:stateless,region:region||undefined}; + if(proto!=='all'&&proto!=='1'&&portMin)Object.assign(body,{port_min:parseInt(portMin),port_max:parseInt(portMax||portMin)}); + try{const d=await $api('/oci/explore/'+S.expCfg+'/nsgs/'+nsgId+'/rules',{method:'POST',body}); + sm('nsgFormMsg',IC.ok+' '+d.message,'s');S._nsgAddForm=null;await expLoadRes()}catch(e){sm('nsgFormMsg',e.message,'e')}} +async function nsgRemoveRule(nsgId,ruleId){ + if(!confirm('Remover esta regra do NSG?'))return; + const region=S.expSelRegions.length===1?S.expSelRegions[0]:null; + const q=region?'?region='+region:''; + try{await $api('/oci/explore/'+S.expCfg+'/nsgs/'+nsgId+'/rules/'+ruleId+q,{method:'DELETE'});await expLoadRes()}catch(e){alert('Erro: '+e.message)}} +async function nsgAddMyIp(nsgId){ + try{const resp=await fetch('https://api.ipify.org?format=json');const data=await resp.json();const myIp=data.ip+'/32'; + document.getElementById('nsgCidr').value=myIp; + document.getElementById('nsgDesc').value='Acesso temporário - '+myIp; + sm('nsgFormMsg',IC.ok+' IP detectado: '+myIp+' — ajuste porta e clique Salvar','s')}catch(e){sm('nsgFormMsg','Não foi possível detectar o IP: '+e.message,'e')}} function rExpRouteTables(d){ return`
${d.length} route table(s)
${d.map((rt,idx)=>{ const routes=rt.routes||[]; @@ -2026,22 +2128,98 @@ ${!filtered.length?'
${IC.folde ${isExp?fHtml:''}
`}).join('')}
`}`} async function toggleFiles(rid){if(S.dlExpandedRid===rid){S.dlExpandedRid=null;R();return} S.dlExpandedRid=rid;if(!S.reportFiles[rid]){try{S.reportFiles[rid]=await $api('/reports/'+rid+'/files')}catch(e){S.reportFiles[rid]=[]}}R()} +function _dlSection(name){ + // Extract CIS section from filename: cis_Networking_2-1.csv → Networking + const m=name.match(/^(?:cis|obp)_([A-Za-z_]+?)_\d/); + if(m)return m[1].replace(/_/g,' '); + if(name.includes('summary'))return 'Summary'; + if(name.includes('error'))return 'Error'; + return 'Other'; +} +const DL_SECTION_TABLE={ + 'Identity and Access Management':'identityandaccess', + 'Networking':'networking', + 'Compute':'computeinstances', + 'Logging and Monitoring':'loggingandmonitoring', + 'Storage Object Storage':'objectstorage', + 'Storage Block Volumes':'storageblockvolume', + 'Storage File Storage Service':'filestorageservice', + 'Asset Management':'assetmanagement', + 'Summary':'summaryreportcsvvector' +}; function renderFileList(rid,files){if(!files.length)return'
Nenhum arquivo encontrado
'; - const groups={};files.forEach(f=>{if(!groups[f.file_category])groups[f.file_category]=[];groups[f.file_category].push(f)}); - const labels={summary:'Summary Reports',cis_finding:'CIS Findings',obp_finding:'OBP Findings',obp_best_practice:'OBP Best Practices',raw_data:'Raw Data',error:'Error Reports',diagram:'Diagrams',consolidated:'Consolidated',other:'Other'}; - const icons={summary:IC.chart,cis_finding:IC.shield,obp_finding:IC.warn,obp_best_practice:IC.ok,raw_data:IC.log,error:IC.err,diagram:IC.barChart,consolidated:IC.file,other:IC.file}; - let h='
'; - for(const[cat,cf] of Object.entries(groups)){ - h+=`
${icons[cat]||IC.file} ${labels[cat]||cat} (${cf.length})
`; - cf.forEach(f=>{ - h+=` + const secIcons={'Identity and Access Management':IC.lock,'Networking':IC.globe,'Compute':IC.server,'Logging and Monitoring':IC.log,'Storage Object Storage':IC.db,'Asset Management':IC.pkg,'Summary':IC.chart,'Error':IC.err,'Other':IC.file}; + // Group files by section + const bySection={};files.forEach(f=>{const s=_dlSection(f.file_name);if(!bySection[s])bySection[s]=[];bySection[s].push(f)}); + const sectionOrder=Object.keys(bySection).sort((a,b)=>{if(a==='Summary')return -1;if(b==='Summary')return 1;if(a==='Other')return 1;if(b==='Other')return -1;return a.localeCompare(b)}); + const hasAdb=S.adbCfg.length>0&&(S.genaiCfg.length>0||S.ociCfg.length>0); + let h='
'; + // Header with total + embed all button + h+=`
`; + h+=`${files.length} arquivo(s) em ${sectionOrder.length} seções`; + if(hasAdb)h+=``; + h+=`
`; + sectionOrder.forEach(sec=>{ + const sf=bySection[sec]; + const icon=secIcons[sec]||IC.file; + // Section header with embed button + h+=`
`; + h+=`
${icon}${sec}(${sf.length})
`; + if(hasAdb)h+=``; + h+=`
`; + // File grid + h+=`
`; + sf.forEach(f=>{ + const ext=f.file_name.split('.').pop().toLowerCase(); + const colors={html:'#e44d26',csv:'#217346',json:'#f5a623',xlsx:'#217346',txt:'#6b7280',pdf:'#dc2626'}; + const ac=colors[ext]||'var(--ac)'; + h+=` ${_dlFileIcon(f.file_name)} -
${f.file_name}
-${_dlFmtSize(f.file_size)} -`}); - } +
${f.file_name}
+
${_dlFmtSize(f.file_size)}
+`}); + h+=`
`; + }); h+='
';return h} -function dlRpt(id,f){window.open(API+'/reports/'+id+'/download?fmt='+f,'_blank')} +async function dlEmbedSection(rid,section){ + if(!S.adbCfg.length){alert('Nenhuma conexão ADB configurada.');return} + if(!S.genaiCfg.length&&!S.ociCfg.length){alert('Nenhuma credencial OCI configurada.');return} + const vid=S.adbCfg.find(c=>c.is_active)?.id||S.adbCfg[0].id; + const cfgName=S.adbCfg.find(c=>c.id===vid)?.config_name||'ADB'; + const report=S.reports.find(r=>r.id===rid); + const reportDate=report?.created_at?new Date(report.created_at).toLocaleDateString('pt-BR'):''; + const tenancy=report?.tenancy_name||''; + const label=section||'todas as seções'; + const tableName=section?DL_SECTION_TABLE[section]:null; + const tableInfo=tableName?`\nTabela: ${tableName}`:'(cada seção → tabela correspondente)'; + const purge=confirm(`Embed "${label}"\nConexão: ${cfgName}${tableInfo}\nReport: ${tenancy} — ${reportDate}\n\nDeseja LIMPAR os dados antigos antes de embedar?\n\nOK = Limpar antigos + embedar novos\nCancelar = Apenas adicionar (sem limpar)`); + // Either way, proceed with embedding. purge=true means clean first. + if(section&&tableName){ + try{ + if(purge)await $api('/embeddings/'+vid+'/purge',{method:'POST',body:{table_name:tableName,tenancy}}); + const d=await $api('/embeddings/report/'+rid,{method:'POST',body:{adb_config_id:vid,section,table_name:tableName}}); + alert(d.message); + }catch(e){alert('Erro: '+e.message)} + }else if(!section){ + const files=S.reportFiles[rid]||[]; + const sections=[...new Set(files.map(f=>_dlSection(f.file_name)))].filter(s=>DL_SECTION_TABLE[s]); + let ok=0,fail=0; + for(const sec of sections){ + const tbl=DL_SECTION_TABLE[sec]; + try{ + if(purge)await $api('/embeddings/'+vid+'/purge',{method:'POST',body:{table_name:tbl,tenancy}}); + await $api('/embeddings/report/'+rid,{method:'POST',body:{adb_config_id:vid,section:sec,table_name:tbl}});ok++; + }catch(e){fail++} + } + alert(`Embedding: ${ok} seções enviadas${fail?' ('+fail+' erros)':''}`); + }else{ + try{ + if(purge)await $api('/embeddings/'+vid+'/purge',{method:'POST',body:{table_name:section,tenancy}}); + const d=await $api('/embeddings/report/'+rid,{method:'POST',body:{adb_config_id:vid,section}}); + alert(d.message); + }catch(e){alert('Erro: '+e.message)} + }} +function dlRpt(id,f){window.open(API+'/reports/'+id+'/download?fmt='+f+'&token='+S.token,'_blank')} async function refreshDl(){S.reports=await $api('/reports');S.reportFiles={};S.dlExpandedRid=null;R()} /* ── Edit helpers ── */ @@ -2315,37 +2493,52 @@ ${S.adbCfg.map(c=>{const em=S.embModels[c.embedding_model_id];const tbls=c.table ${c.username}${tbls.length?tbls.map(t=>''+t.table_name+'').join(' '):'—'} ${c.use_mtls?IC.ok:'—'}${c.wallet_dir?IC.ok:IC.err} ${c.genai_config_id?''+(em?em.name:c.embedding_model_id)+'':'—'} - `}).join('')} + `}).join('')} ${!S.adbCfg.length?'

Nenhuma conexão ADB registrada.

':''}
-
${ea?IC.edit+' Editar Conexão':IC.plus+' Nova Conexão'}
-
${ea?'Editando: '+ea.config_name+'':'Conexão via python-oracledb Thin mode com Wallet (mTLS) para Oracle Autonomous Database. Envie o Wallet ZIP para preencher o DSN automaticamente.'}
-
-
${ea?'Envie novo wallet ou deixe vazio para manter o atual':'Envie o wallet para extrair os DSNs do tnsnames.ora'}
+${ea?`
${IC.edit} Editar Conexão — ${ea.config_name}
+
+
Envie novo wallet ou deixe vazio para manter o atual
Selecione ou digite o DSN. Ex: myatp_high
-
-
-
+
+
+
Credenciais OCI usadas para gerar embeddings via GenAI.
-
+
Modelo OCI GenAI para gerar vetores.
-
-
${ea?'':''}
-${ea?`
${IC.log} Tabelas de Vetores — ${ea.config_name}
-
Tabelas no ADB com dados vetorizados. A busca RAG consultará todas as tabelas ativas.
+
+
+
+
${IC.log} Tabelas de Vetores
+
Registre aqui as tabelas que já existem no ADB. A busca RAG consultará todas as tabelas ativas.
${(ea.tables||[]).length?` ${(ea.tables||[]).map(t=>` - +`).join('')}
TabelaDescriçãoAtivaAções
`:'

Nenhuma tabela registrada.

'}
-
-
- -
`:''} +
+
+ +
` +:`
${IC.plus} Nova Conexão
+
Conexão via python-oracledb Thin mode com Wallet (mTLS) para Oracle Autonomous Database. Envie o Wallet ZIP para preencher o DSN automaticamente.
+
+
Envie o wallet para extrair os DSNs do tnsnames.ora
+
+
Selecione ou digite o DSN. Ex: myatp_high
+
+
+
+
+
Credenciais OCI usadas para gerar embeddings via GenAI.
+
+
Modelo OCI GenAI para gerar vetores.
+
+
`} ${S.adbCfg.length?`
${IC.upload} Atualizar Wallet
@@ -2371,6 +2564,16 @@ async function sAdb(){const btn=document.getElementById('abtn');btn.disabled=tru await $api(url,{method,body:fd,headers:{}});S.editing=null; S.adbCfg=await $api('/adb/configs');sm('am',IC.ok+' Conexão '+(isEdit?'atualizada':'salva')+'!'+(wf?' Wallet incluído.':''),'s');R()}catch(e){sm('am',e.message,'e');btn.disabled=false;btn.textContent=isEdit?'Salvar Alterações':'Salvar Conexão'}} async function tAdb(id){try{const d=await $api('/adb/test/'+id,{method:'POST'});sm('am',d.status==='success'?IC.ok+' '+d.message:IC.err+' '+d.message,d.status==='success'?'s':'e');refreshCLogs('adb')}catch(e){sm('am',e.message,'e')}} +async function checkTables(id){sm('am',' Verificando tabelas no ADB...','i'); + try{const d=await $api('/adb/'+id+'/tables/check',{method:'POST'}); + if(d.status==='error'){sm('am',IC.err+' '+d.message,'e');return} + let html='Schema: '+d.schema_user+' ('+d.adb_tables_total+' tabelas no ADB)

'; + html+=''; + d.registered.forEach(r=>{const st=r.status==='ok'?''+IC.ok+' Existe':r.status==='not_found'?''+IC.err+' Não encontrada':''+IC.err+' Erro'; + html+=''}); + html+='
Tabela RegistradaStatusDetalhes
'+r.table+''+st+''+(r.rows!==undefined?r.rows+' registros':(r.message||''))+'
'; + if(d.unregistered_in_adb&&d.unregistered_in_adb.length){html+='
Tabelas no ADB não registradas ('+d.unregistered_in_adb.length+')
'+d.unregistered_in_adb.join(', ')+'
'} + sm('am',html,'i')}catch(e){sm('am',IC.err+' '+e.message,'e')}} async function dAdb(id){if(!confirm('Excluir conexão?'))return;await $api('/adb/configs/'+id,{method:'DELETE'});S.adbCfg=await $api('/adb/configs');R()} async function uWallet(){const fd=new FormData();const f=document.getElementById('awf2').files[0];if(!f)return sm('wm','Selecione um arquivo wallet ZIP.','e'); fd.append('wallet',f);const id=document.getElementById('awsel').value;if(!id)return sm('wm','Selecione uma conexão ADB.','e'); @@ -2381,7 +2584,7 @@ async function addTable(vid){const name=document.getElementById('newTblName').va try{await $api('/adb/'+vid+'/tables',{method:'POST',body:{table_name:name,description:desc}});S.adbCfg=await $api('/adb/configs');sm('tbm',IC.ok+' Tabela '+name+' registrada!','s');R()}catch(e){sm('tbm',e.message,'e')}} async function saveTable(vid,tid){const name=document.getElementById('tn_'+tid)?.value.trim();const desc=document.getElementById('td_'+tid)?.value.trim(); if(!name)return sm('tbm','Nome da tabela é obrigatório.','e'); - try{await $api('/adb/'+vid+'/tables/'+tid,{method:'PUT',body:{table_name:name.toUpperCase(),description:desc}});S.adbCfg=await $api('/adb/configs');sm('tbm',IC.ok+' Tabela atualizada.','s');R()}catch(e){sm('tbm',e.message,'e')}} + try{await $api('/adb/'+vid+'/tables/'+tid,{method:'PUT',body:{table_name:name,description:desc}});S.adbCfg=await $api('/adb/configs');sm('tbm',IC.ok+' Tabela atualizada.','s');R()}catch(e){sm('tbm',e.message,'e')}} async function toggleTable(vid,tid,val){try{await $api('/adb/'+vid+'/tables/'+tid,{method:'PUT',body:{is_active:val}});S.adbCfg=await $api('/adb/configs');R()}catch(e){sm('tbm',e.message,'e')}} async function removeTable(vid,tid){if(!confirm('Excluir esta tabela do registro?'))return; try{await $api('/adb/'+vid+'/tables/'+tid,{method:'DELETE'});S.adbCfg=await $api('/adb/configs');sm('tbm',IC.ok+' Tabela removida.','s');R()}catch(e){sm('tbm',e.message,'e')}} @@ -2390,40 +2593,47 @@ async function removeTable(vid,tid){if(!confirm('Excluir esta tabela do registro function tblOpts(vid){const cfg=S.adbCfg.find(c=>c.id===vid);if(!cfg||!(cfg.tables||[]).length)return''; return cfg.tables.filter(t=>t.is_active).map(t=>'').join('')} function rEmbeddings(){ - const adbOpts=S.adbCfg.filter(c=>c.genai_config_id).map(c=>'').join(''); + const adbOpts=S.adbCfg.map(c=>'').join(''); const rptOpts=S.reports.filter(r=>r.status==='completed').map(r=>'').join(''); - if(!S.adbCfg.filter(c=>c.genai_config_id).length)return`
${IC.dna}

Nenhuma conexão ADB com GenAI configurada.

Configure em ADB Vector vinculando uma Config GenAI.

`; - const firstVid=S.adbCfg.filter(c=>c.genai_config_id)[0]?.id||''; + if(!S.adbCfg.length)return`
${IC.dna}

Nenhuma conexão ADB configurada.

Configure em ADB Vector.

`; + if(!S.genaiCfg.length&&!S.ociCfg.length)return`
${IC.dna}

Nenhuma credencial OCI configurada.

Configure em Config → OCI.

`; + const firstVid=S.adbCfg.find(c=>c.is_active)?.id||S.adbCfg[0]?.id||''; return`
${IC.log} Embeddings Existentes
Consulte os documentos armazenados nas tabelas de embeddings do ADB.
-
${IC.chart} Embed Relatório CIS
-
Visualize os chunks gerados antes de criar os embeddings. Cada seção do relatório gera um chunk com tenancy, regiões e compartments.
-
-
-
-
-
-
-
${IC.file} Upload de Arquivo
-
Faça upload de um arquivo .txt para gerar embeddings automaticamente (chunking por parágrafos).
-
-
-
-
`} +
${IC.shield} CIS Recommendations
+
Envie a versão mais recente do PDF para manter as recomendações CIS Oracle Cloud atualizadas na base vetorial.
+
+
+
+
${IC.scroll} Base de Conhecimento
+
Envie documentos para alimentar a base de conhecimento da equipe. Os arquivos serão vetorizados e disponibilizados para consulta via chat.
+
+
+
+ + + +
+
+
+ + + +
`} async function loadEmbs(){const vid=document.getElementById('elsel').value;if(!vid)return; const tbl=document.getElementById('eltbl')?.value||''; try{const d=await $api('/embeddings/'+vid+'/list'+(tbl?'?table_name='+encodeURIComponent(tbl):'')); if(!d.documents.length){document.getElementById('emb-list').innerHTML='

Nenhum embedding encontrado.

';return} - document.getElementById('emb-list').innerHTML=''+ - d.documents.map(e=>'').join('')+ - '
IDSourceMetadataDataAções
'+e.id.substring(0,8)+'...'+ - (e.source||'—')+''+(e.metadata||'—')+''+ - (e.created_at||'—')+'
Total: '+d.total+' documentos
'} + document.getElementById('emb-list').innerHTML='
'+ + d.documents.map(e=>{let meta=e.metadata||'—';if(typeof meta==='object')meta=JSON.stringify(meta);if(meta.length>200)meta=meta.substring(0,200)+'…'; + return''+ + ''}).join('')+ + '
IDMetadata
'+e.id.substring(0,12)+''+meta+'
Total: '+d.total+' documentos
'} catch(e){document.getElementById('emb-list').innerHTML='
Erro: '+e.message+'
'}} async function previewChunks(){const rid=document.getElementById('errpt').value; @@ -2455,12 +2665,22 @@ async function embedReport(){const vid=document.getElementById('ersel').value;co const btn=document.getElementById('erbtn');btn.disabled=true;btn.textContent='Gerando...';sm('erm',' Gerando embeddings do relatório...','i'); try{const d=await $api('/embeddings/report/'+rid,{method:'POST',body:{adb_config_id:vid,table_name:tbl||undefined}});sm('erm',IC.ok+' '+d.message,'s');btn.disabled=false;btn.textContent='Gerar Embeddings do Relatório'}catch(e){sm('erm',e.message,'e');btn.disabled=false;btn.textContent='Gerar Embeddings do Relatório'}} +async function embedCisRecom(){const vid=document.getElementById('crsel').value;const f=document.getElementById('crf').files[0]; + if(!vid||!f)return sm('crm','Selecione conexão ADB e arquivo.','e'); + const fd=new FormData();fd.append('file',f);fd.append('adb_config_id',vid);fd.append('table_name','cisrecom'); + const btn=document.getElementById('crbtn');btn.disabled=true;btn.textContent='Enviando...';sm('crm',' Processando PDF e gerando embeddings...','i'); + try{const d=await $api('/embeddings/upload',{method:'POST',body:fd,headers:{}});sm('crm',IC.ok+' '+d.message+' → tabela CISRECOM','s');btn.disabled=false;btn.textContent=IC.upload+' Upload CIS Recommendations'}catch(e){sm('crm',e.message,'e');btn.disabled=false;btn.textContent=IC.upload+' Upload CIS Recommendations'}} async function embedFile(){const vid=document.getElementById('efsel').value;const f=document.getElementById('eff').files[0]; if(!vid||!f)return sm('efm','Selecione conexão ADB e arquivo.','e'); - const tbl=document.getElementById('eftbl')?.value||''; - const fd=new FormData();fd.append('file',f);fd.append('adb_config_id',vid);if(tbl)fd.append('table_name',tbl); + const tbl='engineerknowledgebase'; + const fd=new FormData();fd.append('file',f);fd.append('adb_config_id',vid);fd.append('table_name',tbl); const btn=document.getElementById('efbtn');btn.disabled=true;btn.textContent='Enviando...';sm('efm',' Enviando e gerando embeddings...','i'); - try{const d=await $api('/embeddings/upload',{method:'POST',body:fd,headers:{}});sm('efm',IC.ok+' '+d.message,'s');btn.disabled=false;btn.textContent='Upload e Gerar Embeddings'}catch(e){sm('efm',e.message,'e');btn.disabled=false;btn.textContent='Upload e Gerar Embeddings'}} + try{const d=await $api('/embeddings/upload',{method:'POST',body:fd,headers:{}});sm('efm',IC.ok+' '+d.message,'s');btn.disabled=false;btn.textContent='Upload Arquivo'}catch(e){sm('efm',e.message,'e');btn.disabled=false;btn.textContent='Upload Arquivo'}} +async function embedUrl(){const vid=document.getElementById('efsel').value;const url=document.getElementById('efurl').value.trim(); + if(!vid||!url)return sm('efm','Selecione conexão ADB e informe a URL.','e'); + const fd=new FormData();fd.append('adb_config_id',vid);fd.append('table_name','engineerknowledgebase');fd.append('url',url); + const btn=document.getElementById('efurlbtn');btn.disabled=true;btn.textContent='Importando...';sm('efm',' Buscando conteúdo e gerando embeddings...','i'); + try{const d=await $api('/embeddings/upload-url',{method:'POST',body:fd,headers:{}});sm('efm',IC.ok+' '+d.message,'s');btn.disabled=false;btn.textContent='Importar URL';document.getElementById('efurl').value=''}catch(e){sm('efm',e.message,'e');btn.disabled=false;btn.textContent='Importar URL'}} async function delEmb(vid,docId){if(!confirm('Excluir este embedding?'))return; const tbl=document.getElementById('eltbl')?.value||'';