GustavoNogueira/A-Team-Security-Infra-Agent-Engineer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: auto-split monolithic Terraform files and fix single-line HCL blocks

Browse Source 
- Add 3-layer protection pipeline for model-generated Terraform code:
  1. Frontend auto-split: splits monolithic HCL into multiple files by resource type
  2. Backend auto-split: Python port as last defense before writing to disk
  3. Backend deduplication: removes monolithic leftovers with duplicate resources
- Fix single-line block expansion to handle ALL HCL blocks (nested included):
  ingress_security_rules, egress_security_rules, route_rules, match_criteria, etc.
- Add prompt size logging for terraform agent diagnostics
- Strengthen system prompt: minimum 3 files, final reminder section
This commit is contained in:
nogueiraguh
2026-03-09 00:40:48 -03:00
parent 15a8366f9c
commit 6fb2eecddf
2 changed files with 323 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files

188
backend/app.py
View File

@@ -1949,8 +1949,8 @@ TF_DEFAULT_SYSTEM_PROMPT = """Você é um agente especializado EXCLUSIVAMENTE em
- Referencie o provider alias nos recursos: `provider = oci.secondary`
- NUNCA diga que não pode criar recursos em outra região. Use provider aliases para qualquer região OCI.
### Múltiplos Arquivos (PADRÃO)
- Por PADRÃO, SEMPRE gere o código em MÚLTIPLOS arquivos organizados por função:
### Múltiplos Arquivos (OBRIGATÓRIO)
- SEMPRE gere o código em MÚLTIPLOS arquivos organizados por função, NÃO IMPORTA o tamanho da infraestrutura:
- `variables.tf` — todas as variáveis (incluindo `variable "region"`)
- `networking.tf` — VCN, subnets, gateways, route tables, security lists
- `compute.tf` — instances, instance pools
@@ -1958,9 +1958,10 @@ TF_DEFAULT_SYSTEM_PROMPT = """Você é um agente especializado EXCLUSIVAMENTE em
- `firewall.tf` — network firewalls, policies
- `outputs.tf` — outputs úteis
- Outros arquivos conforme necessário (ex: `storage.tf`, `iam.tf`, `loadbalancer.tf`)
- MÍNIMO OBRIGATÓRIO: gere pelo menos `variables.tf`, um ou mais arquivos de recursos, e `outputs.tf` (mínimo 3 arquivos).
- Nomeie cada bloco com um comentário na primeira linha: `// filename: networking.tf`, `// filename: compute.tf`, etc.
- Cada bloco `hcl` vira um arquivo separado no sistema.
- Somente gere tudo em um único bloco se o usuário pedir EXPLICITAMENTE "um único arquivo" ou a infraestrutura for muito simples (1-2 recursos).
- NUNCA gere tudo em um único bloco ou em apenas 2 arquivos. Somente gere arquivo único se o usuário pedir EXPLICITAMENTE "um único arquivo".
### REGRA CRÍTICA: Unicidade de Recursos
- **NUNCA** declare o mesmo recurso (mesmo tipo + mesmo nome) em mais de um arquivo.
@@ -2091,6 +2092,11 @@ Antes de entregar qualquer código (novo ou corrigido), valide TODOS os itens ab
```
O erro "Invalid single-argument block definition" ocorre quando HCL encontra `bloco { a = 1, b = 2 }` com mais de um argumento. SEMPRE usar formato multi-line.
Isso vale para TODOS os blocos: `ingress_security_rules`, `egress_security_rules`, `route_rules`, `match_criteria`, `tcp_options`, `udp_options`, etc.
### LEMBRETE FINAL (MAIS IMPORTANTE)
- Para NOVAS gerações: SEMPRE gere MÚLTIPLOS arquivos (mínimo 3: variables.tf + recursos + outputs.tf). NUNCA entregue apenas 1 ou 2 arquivos.
- Para CORREÇÕES (quando há "ARQUIVOS TERRAFORM ATUAIS NO WORKSPACE"): gere SOMENTE os arquivos que mudaram.
- Se a mensagem do usuário NÃO contém "ARQUIVOS TERRAFORM ATUAIS NO WORKSPACE", trate como NOVA geração e gere TODOS os arquivos necessários.
"""
# ── Terraform OCI Resource Reference (generated at build time, updatable at runtime) ──
@@ -3267,6 +3273,12 @@ async def _chat_background(mid: str, sid: str, msg: ChatMsg, user: dict, genai_c
ref_compact
log.info(f"Terraform: injected resource reference ({len(ref_compact)} chars)")
# Log total prompt sizes for debugging
if agent_type == "terraform":
sp_len = len(cfg_dict.get("system_prompt", ""))
msg_len = len(msg.message) if hasattr(msg, 'message') else 0
log.info(f"Terraform prompt sizes: system_prompt={sp_len} chars (~{sp_len//4} tokens), user_msg={msg_len} chars (~{msg_len//4} tokens), max_tokens={cfg_dict.get('max_tokens')}")
mcp_tools = []
tool_defs = None
if msg.use_tools:
@@ -3844,22 +3856,178 @@ async def tf_refresh_reference(u=Depends(current_user)):
return result
def _fix_single_line_blocks(content: str) -> str:
"""Expand ANY single-line HCL block with multiple args into multi-line format.
Matches both top-level (variable, resource) AND nested blocks (ingress_security_rules, route_rules, etc.)
Does NOT match map assignments (tags = { ... }) because of the '=' before '{'."""
import re as _re
def _expand(m):
header, body = m.group(1), m.group(2)
# Split by comma first
args = [a.strip() for a in body.split(',') if a.strip()]
if len(args) < 2:
# Try space-separated key=value
args = _re.findall(r'\w+\s*=\s*(?:"[^"]*"|\S+)', body)
if len(args) < 2:
return m.group(0)
indent = _re.match(r'^(\s*)', header).group(1)
return header.rstrip() + ' {\n' + '\n'.join(indent + ' ' + a for a in args) + '\n' + indent + '}'
return _re.sub(
r'^(\s*[a-z]\w*(?:\s+"[^"]*")*)\s*\{\s*(.+?)\s*\}\s*$',
_expand, content, flags=_re.MULTILINE)
def _split_tf_monolith(content: str) -> dict:
"""Split a monolithic HCL string into multiple logical files by resource type.
Returns dict of {filename: content} or None if splitting not needed."""
import re as _re
content = _fix_single_line_blocks(content)
lines = content.split('\n')
top_blocks = []
preamble = []
accum = []
in_block = False
b_depth = 0
cur_header = ''
for line in lines:
stripped = _re.sub(r'"[^"]*"', '', line)
opens = stripped.count('{')
closes = stripped.count('}')
if not in_block:
hdr = _re.match(r'^\s*(variable|output|resource|data|provider|module)\s+"', line) or \
_re.match(r'^\s*(locals|terraform)\s*\{', line)
if hdr:
in_block = True
cur_header = line
accum = preamble + [line]
preamble = []
b_depth = opens - closes
if b_depth <= 0:
b_depth = 0
if b_depth == 0 and opens > 0:
top_blocks.append((cur_header, '\n'.join(accum)))
in_block = False
accum = []
else:
preamble.append(line)
else:
accum.append(line)
b_depth += opens - closes
if b_depth <= 0:
b_depth = 0
top_blocks.append((cur_header, '\n'.join(accum)))
in_block = False
accum = []
cur_header = ''
if accum:
top_blocks.append((cur_header or '', '\n'.join(accum)))
if len(top_blocks) < 3:
return None
cats = {'variables': [], 'outputs': [], 'networking': [], 'compute': [], 'database': [],
'firewall': [], 'loadbalancer': [], 'storage': [], 'iam': [], 'drg': [],
'data': [], 'providers': [], 'other': []}
for header, body in top_blocks:
h = header.strip()
if h.startswith('variable '):
cats['variables'].append(body)
elif h.startswith('output '):
cats['outputs'].append(body)
elif h.startswith('provider '):
cats['providers'].append(body)
elif _re.search(r'resource\s+"oci_core_(drg|remote_peering|drg_route|drg_attachment)', h):
cats['drg'].append(body)
elif _re.search(r'resource\s+"oci_core_(vcn|subnet|internet_gateway|nat_gateway|service_gateway|route_table|security_list|network_security_group|dhcp_options|local_peering_gateway|virtual_circuit)', h):
cats['networking'].append(body)
elif _re.search(r'resource\s+"oci_core_instance', h):
cats['compute'].append(body)
elif _re.search(r'resource\s+"oci_(database|nosql|mysql|psql)', h) or _re.search(r'resource\s+"oci_core_autonomous', h):
cats['database'].append(body)
elif _re.search(r'resource\s+"oci_network_firewall', h):
cats['firewall'].append(body)
elif _re.search(r'resource\s+"oci_(load_balancer|network_load_balancer)', h):
cats['loadbalancer'].append(body)
elif _re.search(r'resource\s+"oci_(objectstorage|file_storage)', h):
cats['storage'].append(body)
elif _re.search(r'resource\s+"oci_identity', h):
cats['iam'].append(body)
elif h.startswith('data '):
cats['data'].append(body)
else:
cats['other'].append(body)
result = {}
mapping = [('variables.tf', 'variables'), ('providers.tf', 'providers'), ('networking.tf', 'networking'),
('drg.tf', 'drg'), ('compute.tf', 'compute'), ('database.tf', 'database'),
('firewall.tf', 'firewall'), ('loadbalancer.tf', 'loadbalancer'), ('storage.tf', 'storage'),
('iam.tf', 'iam'), ('data.tf', 'data'), ('main.tf', 'other'), ('outputs.tf', 'outputs')]
for fname, key in mapping:
if cats[key]:
result[fname] = '\n\n'.join(cats[key])
return result if len(result) >= 3 else None
def _write_tf_files(wdir: Path, tf_code: str):
"""Parse tf_code for '// filename: xxx.tf' markers and write separate files."""
"""Parse tf_code for '// filename: xxx.tf' markers and write separate files.
If only 1-2 large files, auto-split into multiple files by resource type."""
import re as _re
parts = _re.split(r'^//\s*filename:\s*(\S+)\s*$', tf_code, flags=_re.MULTILINE)
files = {}
if len(parts) >= 3:
# parts = [preamble, filename1, content1, filename2, content2, ...]
if parts[0].strip():
(wdir / "main.tf").write_text(parts[0].strip())
files['main.tf'] = parts[0].strip()
for i in range(1, len(parts), 2):
fname = parts[i].strip()
fname = parts[i].strip().replace("/", "_").replace("..", "_")
content = parts[i + 1].strip() if i + 1 < len(parts) else ""
if fname and content:
safe_name = fname.replace("/", "_").replace("..", "_")
(wdir / safe_name).write_text(content)
files[fname] = content
else:
(wdir / "main.tf").write_text(tf_code)
files['main.tf'] = tf_code
# Auto-split: if 1-2 large files, split by resource type
if len(files) <= 2:
all_content = '\n\n'.join(files.values())
if len(all_content) > 2000:
split = _split_tf_monolith(all_content)
if split:
files = split
log.info(f"Backend auto-split monolithic TF into {len(files)} files: {list(files.keys())}")
# Deduplicate: if a resource appears in multiple files, keep only in the split file
# This handles edge cases where monolithic + split files both exist
if len(files) > 2:
resource_locations = {} # "type.name" -> [(filename, line)]
dupes_in = set()
for fname, content in files.items():
for m in _re.finditer(r'^resource\s+"(\S+)"\s+"(\S+)"', content, _re.MULTILINE):
key = f'{m.group(1)}.{m.group(2)}'
resource_locations.setdefault(key, []).append(fname)
# Find files that contain ONLY duplicate resources (= monolithic leftovers)
for key, fnames in resource_locations.items():
if len(fnames) > 1:
# The largest file is likely the monolithic one
largest = max(fnames, key=lambda f: len(files.get(f, '')))
dupes_in.add(largest)
# Remove monolithic files that are fully duplicated
for fname in dupes_in:
if all(
any(f2 != fname and f2 in [fl for fl in resource_locations.get(key, []) if fl != fname])
for key, flist in resource_locations.items()
if fname in flist
) and len(files) - 1 >= 2:
log.info(f"Removing duplicate monolithic file: {fname}")
del files[fname]
# Fix single-line blocks in all files
for fname in files:
files[fname] = _fix_single_line_blocks(files[fname])
# Write files to disk
for fname, content in files.items():
(wdir / fname).write_text(content)
async def _terraform_exec(wid: str, action: str, user: dict):

154
frontend/index.html
View File

@@ -714,11 +714,125 @@ async function loadChatLogs(){
const TF_ICON=`<svg viewBox="0 0 16 16" width="16" height="16"><path d="M5.6 1v4.2l3.6 2.1V3.1L5.6 1zm4.4 2.1v4.2l3.6-2.1V1L10 3.1zM1.4 3.5v4.2l3.6 2.1V5.6L1.4 3.5zM5.6 8.1v4.2L9.2 14.4V10.2L5.6 8.1z" fill="#7b42bc"/></svg>`;
function escHtml(s){return s.replace(/&/g,'&amp;').replace(/</g,'&lt;').replace(/>/g,'&gt;').replace(/"/g,'&quot;')}
function _fixSingleLineBlocks(content){
// Expand ANY single-line HCL block with multiple args into multi-line format.
// Matches both top-level (variable, resource) AND nested blocks (ingress_security_rules, route_rules, etc.)
// Does NOT match map assignments (tags = { ... }) because of the "=" before "{"
// Pattern: identifier ["quoted"]* { key=val, key=val } (no "=" between identifier and "{")
return content.replace(/^(\s*[a-z]\w*(?:\s+"[^"]*")*)\s*\{\s*(.+?)\s*\}\s*$/gm, (match, header, body) => {
// Skip if body doesn't have multiple key=value pairs
let args = body.split(/,\s*/);
if(args.length < 2){
args = [];
const re = /(\w+)\s*=\s*(?:"[^"]*"|\S+)/g;
let m2;
while((m2 = re.exec(body)) !== null) args.push(m2[0]);
}
if(args.length < 2) return match;
const indent = header.match(/^(\s*)/)[1];
return header.trimEnd() + ' {\n' + args.map(a => indent + ' ' + a.trim()).join('\n') + '\n' + indent + '}';
});
}
function _splitTfMonolith(content){
// Split a monolithic HCL file into multiple logical files by parsing top-level blocks.
// Returns array of {name, content} or null if splitting not needed.
content = _fixSingleLineBlocks(content);
const lines=content.split('\n');
const topBlocks=[];
let preamble=[]; // comments/blanks before next block
let accum=[];
let inBlock=false;
let bDepth=0;
let curHeader='';
for(let i=0;i<lines.length;i++){
const line=lines[i];
// Count braces outside of quoted strings (simple heuristic: ignore braces inside "...")
const stripped=line.replace(/"[^"]*"/g,'');
const open=(stripped.match(/\{/g)||[]).length;
const close=(stripped.match(/\}/g)||[]).length;
if(!inBlock){
// Match top-level block declarations only (not attribute assignments like "provider = oci.x")
const hdr=line.match(/^\s*(variable|output|resource|data|provider|module)\s+"/) || line.match(/^\s*(locals|terraform)\s*\{/);
if(hdr){
// Start of a new block — attach preamble (comments above)
inBlock=true;
curHeader=line;
accum=preamble.concat([line]);
preamble=[];
bDepth=open-close;
if(bDepth<=0) bDepth=0;
// Single-line block (opens and closes on same line)
if(bDepth===0 && open>0){
topBlocks.push({header:curHeader,body:accum.join('\n')});
inBlock=false; accum=[];
}
} else {
// Comment, blank, or stray line between blocks — save as preamble for next block
preamble.push(line);
}
} else {
accum.push(line);
bDepth+=open-close;
if(bDepth<=0){
bDepth=0;
topBlocks.push({header:curHeader,body:accum.join('\n')});
inBlock=false; accum=[]; curHeader='';
}
}
}
// Flush remaining
if(accum.length) topBlocks.push({header:curHeader||'',body:accum.join('\n')});
if(topBlocks.length<3) return null;
// Categorize each block by its header
const cats={variables:[],outputs:[],networking:[],compute:[],database:[],
firewall:[],loadbalancer:[],storage:[],iam:[],drg:[],data:[],providers:[],other:[]};
for(const b of topBlocks){
const h=b.header.trim();
if(h.startsWith('variable ')) cats.variables.push(b.body);
else if(h.startsWith('output ')) cats.outputs.push(b.body);
else if(h.startsWith('provider ')) cats.providers.push(b.body);
else if(/resource\s+"oci_core_(drg|remote_peering|drg_route|drg_attachment)/.test(h)) cats.drg.push(b.body);
else if(/resource\s+"oci_core_(vcn|subnet|internet_gateway|nat_gateway|service_gateway|route_table|security_list|network_security_group|dhcp_options|local_peering_gateway|virtual_circuit)/.test(h)) cats.networking.push(b.body);
else if(/resource\s+"oci_core_instance/.test(h)) cats.compute.push(b.body);
else if(/resource\s+"oci_(database|nosql|mysql|psql)/.test(h) || /resource\s+"oci_core_autonomous/.test(h)) cats.database.push(b.body);
else if(/resource\s+"oci_network_firewall/.test(h)) cats.firewall.push(b.body);
else if(/resource\s+"oci_(load_balancer|network_load_balancer)/.test(h)) cats.loadbalancer.push(b.body);
else if(/resource\s+"oci_(objectstorage|file_storage)/.test(h)) cats.storage.push(b.body);
else if(/resource\s+"oci_identity/.test(h)) cats.iam.push(b.body);
else if(h.startsWith('data ')) cats.data.push(b.body);
else cats.other.push(b.body);
}
// Build result files in logical order
const result=[];
const add=(name,arr)=>{if(arr.length) result.push({name,content:arr.join('\n\n')})};
add('variables.tf', cats.variables);
add('providers.tf', cats.providers);
add('networking.tf', cats.networking);
add('drg.tf', cats.drg);
add('compute.tf', cats.compute);
add('database.tf', cats.database);
add('firewall.tf', cats.firewall);
add('loadbalancer.tf', cats.loadbalancer);
add('storage.tf', cats.storage);
add('iam.tf', cats.iam);
add('data.tf', cats.data);
add('main.tf', cats.other);
add('outputs.tf', cats.outputs);
return result.length>=3 ? result : null;
}
function fmTf(t){
// Extract ALL hcl blocks, parse // filename: comments, build separate files
const codeBlocks=[];
let r=t.replace(/```(?:hcl|terraform)\s*\n([\s\S]*?)```/g,(m,code)=>{
const trimmed=code.trim();
const trimmed=_fixSingleLineBlocks(code.trim());
// Check for // filename: comment on first line
const fnMatch=trimmed.match(/^\/\/\s*filename:\s*(\S+)/);
const fileName=fnMatch?fnMatch[1]:'main.tf';
@@ -736,30 +850,52 @@ function fmTf(t){
return '<div class="tf-plan-block"><div class="tf-plan-header">'+TF_ICON+' Resource Plan — '+S.tfPlan.length+' resource(s)</div><div class="tf-plan-list">'+
S.tfPlan.map(x=>'<div class="tf-plan-item"><span class="tf-plan-add">+</span><span class="tf-plan-type">'+escHtml(x.type)+'</span><span class="tf-plan-desc">'+escHtml(x.desc)+'</span></div>').join('')+'</div></div>';
});
// Build files list — merge new blocks with existing files (corrections preserve untouched files)
// Auto-split: if model returned 1-2 large blocks, split into multiple files
// Runs on BOTH first generation and corrections — monolithic responses always get split
let didAutoSplit=false;
if(codeBlocks.length<=2){
const totalContent=codeBlocks.map(b=>b.content).join('\n\n');
if(totalContent.length>2000){
const split=_splitTfMonolith(totalContent);
if(split){
codeBlocks.length=0;
split.forEach(f=>codeBlocks.push(f));
didAutoSplit=true;
// Rebuild HTML with split files
const splitHtml=split.map((f,i)=>
'<div class="tf-code-block"><div class="tf-code-header"><span>'+TF_ICON+' '+escHtml(f.name)+'</span>'+
'<button class="btn" onclick="tfCopyBlock('+i+')">Copy</button></div>'+
'<pre class="tf-pre"><code>'+escHtml(f.content)+'</code></pre></div>'
).join('');
// Replace the original single code block HTML
r=r.replace(/<div class="tf-code-block">[\s\S]*?<\/pre><\/div>/g,'');
r=splitHtml+r;
console.log('[TF] Auto-split monolithic file into '+split.length+' files');
}
}
}
// Build files list
if(codeBlocks.length){
const newMap={};
codeBlocks.forEach(b=>{
if(newMap[b.name])newMap[b.name]+='\n\n'+b.content;
else newMap[b.name]=b.content;
});
// If existing files: merge (update matched, keep unmatched)
if(S.tfFiles.length){
// If auto-split happened, REPLACE all files (monolithic = full regeneration, not partial correction)
// If model returned multiple named files (no auto-split), use merge logic for corrections
if(didAutoSplit || !S.tfFiles.length){
S.tfFiles=Object.entries(newMap).map(([name,content])=>({name,content,size:content.length,type:'hcl'}));
}else{
const merged=[];
const usedNew=new Set();
// Update existing files with new content if name matches
for(const f of S.tfFiles){
if(newMap[f.name]!=null){merged.push({name:f.name,content:newMap[f.name],size:newMap[f.name].length,type:'hcl'});usedNew.add(f.name)}
else{merged.push(f)}
}
// Add any truly new files
for(const[name,content] of Object.entries(newMap)){
if(!usedNew.has(name))merged.push({name,content,size:content.length,type:'hcl'})
}
S.tfFiles=merged;
}else{
// First generation: set all
S.tfFiles=Object.entries(newMap).map(([name,content])=>({name,content,size:content.length,type:'hcl'}));
}
S.tfCode=S.tfFiles.map(f=>'// filename: '+f.name+'\n'+f.content).join('\n\n');
}