clean-up e persistencia na nuvem

This commit is contained in:
2026-05-08 08:29:08 -03:00
parent f58cdb63e2
commit 1b13e119a9
3 changed files with 788 additions and 37 deletions

383
README.md
View File

@@ -89,8 +89,8 @@ O mesmo ocorre para:
Existem 2 arquivos de deployment neste material:
- langfuse_dist.yaml
- final.yaml
- **langfuse_dist.yaml** (ou **langfuse_dist_cloud.yaml** para utilização de storage externo ao clustes Kubernetes)
- **final.yaml**
Ambos os arquivos contém:
- StorageClass
@@ -102,8 +102,15 @@ Ambos os arquivos contém:
Para ambientes produtivos, use o arquivo **langfuse_dist.yaml**. As variáveis de ambiente já estão preparadas para cada componente do langfuse.
Se deseja validar de forma ilustrada e passar por todos os componentes deste material, use as ferramentas mirror_images.sh, create_secrets.sh, update_secret.sh e set_var.sh para montar um arquivo YAML de deployment como tudo ajustado sem necessidade de criar um pipeline DEVOPS para isto.
## Persistência fora do Cluster Kubernetes
O arquivo **langfuse_dist_cloud.yaml** permite que se utilize o PostgreSQL e Redis fora do cluster Kubernetes. Isto permite que se faça a governança destes recursos de forma mais tradicional, além de maior economia na escalabilidade e melhoras em troubleshooting em geral.
Caso use a opção de utilizar os recursos de PostgreSQL e Redis dentro do cluster Kubernetes, existe um pacote adicional neste tutorial chamado **langfuse-k8s-cleanup.zip** onde há mecanismos para o clean-up de forma agendada dentro do próprio cluster Kubernetes.
## PostgreSQL
Responsável por:
@@ -1102,70 +1109,369 @@ Define se ClickHouse opera em cluster distribuído.
---
## LANGFUSE_S3_EVENT_UPLOAD_BUCKET
# Configuração OCI Object Storage para Upload de Eventos no Langfuse
```bash id="vars44"
export LANGFUSE_S3_EVENT_UPLOAD_BUCKET="langfuse-events"
```
## Objetivo
### Objetivo
Estas variáveis configuram o Langfuse para armazenar eventos, traces e arquivos no OCI Object Storage utilizando a API compatível com Amazon S3.
Bucket de eventos.
O Langfuse utiliza esse storage para:
- Persistência de eventos
- Upload de traces grandes
- Armazenamento de payloads
- Redução de carga no PostgreSQL
- Redução de crescimento do ClickHouse
- Armazenamento distribuído e durável
---
# Variáveis de Ambiente
## LANGFUSE_S3_EVENT_UPLOAD_BUCKET
```yaml
- name: LANGFUSE_S3_EVENT_UPLOAD_BUCKET
value: "langfuse-events"
```
Objetivo
Define o bucket do OCI Object Storage onde os eventos serão armazenados.
Observações
-O bucket deve existir previamente.
-O bucket precisa estar na mesma região configurada.
-Pode ser Standard ou Archive (recomendado Standard).
oci os bucket create \
--name langfuse-events \
--compartment-id <COMPARTMENT_OCID>
## LANGFUSE_S3_EVENT_UPLOAD_PREFIX
```yaml
- name: LANGFUSE_S3_EVENT_UPLOAD_PREFIX
value: "events/"
```
Objetivo
Define o prefixo (pasta lógica) dentro do bucket.
Resultado
Os objetos serão gravados em:
langfuse-events/events/
Exemplo de objeto
events/traces/abc123.json
Benefícios
Permite:
- Separar ambientes
- Organizar dados
- Facilitar lifecycle policies
- Facilitar limpeza automática
Exemplos úteis
Produção:
value: "prod/events/"
Homologação:
value: "hml/events/"
Multi-Tenant:
value: "tenant-a/events/"
## LANGFUSE_S3_EVENT_UPLOAD_REGION
```bash id="vars45"
export LANGFUSE_S3_EVENT_UPLOAD_REGION="${OCI_REGION}"
```
- name: LANGFUSE_S3_EVENT_UPLOAD_REGION
value: "${OCI_REGION}"
### Objetivo
Objetivo
Região do bucket.
Define a região OCI do Object Storage.
---
Exemplos
sa-saopaulo-1
us-chicago-1
eu-frankfurt-1
Importância
A região precisa coincidir com:
* bucket criado
* endpoint S3
* credenciais utilizadas
Exemplo
value: "sa-saopaulo-1"
## LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT
```bash id="vars46"
export LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT="https://${OCI_NAMESPACE}.compat.objectstorage.${OCI_REGION}.oraclecloud.com"
```
- name: LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT
value: "https://${OCI_NAMESPACE}.compat.objectstorage.${OCI_REGION}.oraclecloud.com"
### Objetivo
Objetivo
Endpoint S3-compatible OCI.
Define o endpoint S3 compatível da OCI.
---
O OCI Object Storage possui uma API compatível com Amazon S3.
## LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE
O Langfuse usa esse endpoint como se fosse AWS S3.
```bash id="vars47"
export LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE="true"
```
⸻
### Objetivo
Estrutura do endpoint
Ativa integração OCI nativa.
https://<NAMESPACE>.compat.objectstorage.<REGION>.oraclecloud.com
---
Exemplo real
## LANGFUSE_OCI_AUTH_TYPE
https://mycompany.compat.objectstorage.sa-saopaulo-1.oraclecloud.com
```bash id="vars48"
export LANGFUSE_OCI_AUTH_TYPE="instance_principal"
```
### Objetivo
O que é OCI_NAMESPACE
Utiliza Instance Principal em vez de Access Key.
O namespace é um identificador único global do tenancy OCI.
### Benefício
Exemplo
Não precisamos:
xkthfgahad
* access key
* secret key
Como descobrir o namespace
OCI CLI
oci os ns get
Console OCI
Object Storage → Namespace
⸻
## LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID
- name: LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID
value: "${S3_ACCESS_KEY}"
Objetivo
Define o Access Key utilizado para autenticação S3.
Funciona similar ao AWS Access Key ID.
>**Nota:** O par de keys de Custom Keys não pode ser gerado automaticamente pois não é fornecido o Secret. Isso só é possível via Console OCi
Identity & Security
→ Users
→ Seu usuário
→ Customer Secret Keys
→ Generate Secret Key
## LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY
- name: LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY
value: "${S3_SECRET_KEY}"
Objetivo
Define o segredo associado ao Access Key.
É equivalente ao AWS Secret Access Key.
⸻
Observações importantes
* A secret é exibida apenas UMA vez na criação.
* Deve ser armazenada com segurança.
* Idealmente usar OCI Vault.
⸻
Recomendação
Nunca colocar diretamente no YAML.
Preferir:
valueFrom:
secretKeyRef:
name: object-storage-secret
key: S3_SECRET_KEY
## LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE
- name: LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE
value: "true"
Objetivo
Força o cliente S3 a usar Path Style ao invés de Virtual Host Style.
⸻
Diferença entre estilos
Virtual Host Style (AWS padrão)
https://bucket.endpoint/object
Exemplo:
https://langfuse-events.s3.amazonaws.com/file.json
Path Style
https://endpoint/bucket/object
Exemplo:
https://compat.objectstorage.sa-saopaulo-1.oraclecloud.com/langfuse-events/file.json
Por que OCI precisa disso
O endpoint compatível S3 da OCI funciona melhor com:
LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE=true
Sem isso podem ocorrer:
- erro de assinatura
- erro de hostname
- erro 403
- bucket não encontrado
### Fluxo de Funcionamento
Langfuse
↓
S3 Compatible Client
↓
OCI Object Storage Endpoint
↓
Bucket OCI
↓
Persistência de Eventos
Benefícios dessa Arquitetura
Redução de Crescimento do PostgreSQL
Payloads grandes não ficam somente no banco.
⸻
Melhor Escalabilidade
Object Storage escala automaticamente.
⸻
Alta Durabilidade
OCI Object Storage possui alta redundância.
⸻
Menor Custo
Storage é mais barato que expandir PostgreSQL.
⸻
Facilidade de Backup
Object Storage já possui durabilidade nativa.
⸻
Recomendações Enterprise
Usar OCI Vault
Evitar secrets hardcoded.
⸻
Usar Instance Principal
Evitar Access Key fixa.
⸻
Criar Lifecycle Policies
Exemplo:
Apagar eventos após 30 dias
Mover para Archive após 7 dias
### Exemplo de Lifecycle Policy
Object Storage
→ Bucket
→ Lifecycle Policy
### Segurança Recomendada
Bucket privado
Nunca público.
⸻
Network privada
Preferencialmente:
* OKE privado
* Object Storage via Service Gateway
⸻
Policies mínimas
Exemplo:
Allow dynamic-group oke-cluster to manage objects in compartment observability
⸻
Importante
No OCI, esse Access Key pertence a um usuário OCI.
Não é o OCID do usuário.
⸻
Como gerar
OCI Console
---
@@ -1405,6 +1711,9 @@ Ao final:
# Referências
- [Langfuse - OCI Object Storage (Native)](https://langfuse.com/self-hosting/deployment/infrastructure/blobstorage#oci-object-storage)
- [OCI Object Storage](https://docs.oracle.com/en-us/iaas/Content/Object/Concepts/objectstorageoverview.htm)
- [OCI S3 Compatibility API](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm)
- [Langfuse Self Hosting](https://langfuse.com/self-hosting)
# Acknowledgments

Binary file not shown.

View File

@@ -0,0 +1,442 @@
apiVersion: v1
kind: Secret
metadata:
name: langfuse-external-services
namespace: ${K8S_NAMESPACE}
type: Opaque
stringData:
# PostgreSQL OCI
DATABASE_URL: "postgresql://langfuse:${DB_PASSWORD}@10.0.10.15:5432/langfuse"
# Redis OCI
REDIS_HOST: "10.0.20.10"
REDIS_PORT: "6379"
REDIS_AUTH: "${REDIS_AUTH}"
# ClickHouse
CLICKHOUSE_PASSWORD: "${CLICKHOUSE_PASSWORD}"
# Langfuse
NEXTAUTH_SECRET: "${NEXTAUTH_SECRET}"
SALT: "${SALT}"
ENCRYPTION_KEY: "${ENCRYPTION_KEY}"
# OCI Object Storage
S3_ACCESS_KEY: "${S3_ACCESS_KEY}"
S3_SECRET_KEY: "${S3_SECRET_KEY}"
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: ${APP_NAME}-clickhouse
namespace: ${K8S_NAMESPACE}
spec:
serviceName: ${APP_NAME}-clickhouse
replicas: 1
selector:
matchLabels:
app: ${APP_NAME}-clickhouse
template:
metadata:
labels:
app: ${APP_NAME}-clickhouse
spec:
imagePullSecrets:
- name: ocirsecret
containers:
- name: clickhouse
image: ${IMAGE_REPOSITORY}/clickhouse:${CLICKHOUSE_TAG}
ports:
- containerPort: 8123
- containerPort: 9000
resources:
requests:
cpu: "2"
memory: "8Gi"
limits:
cpu: "4"
memory: "16Gi"
volumeMounts:
- name: data
mountPath: /var/lib/clickhouse
env:
- name: CLICKHOUSE_DB
value: "default"
- name: CLICKHOUSE_USER
value: "default"
- name: CLICKHOUSE_PASSWORD
value: "${CLICKHOUSE_PASSWORD}"
- name: CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT
value: "1"
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: oci-bv-high-performance
resources:
requests:
storage: 200Gi
---
apiVersion: v1
kind: Service
metadata:
name: ${APP_NAME}-clickhouse
namespace: ${K8S_NAMESPACE}
spec:
selector:
app: ${APP_NAME}-clickhouse
ports:
- name: http
port: 8123
- name: native
port: 9000
---
# ===== LANGFUSE APPLICATION =====
apiVersion: apps/v1
kind: Deployment
metadata:
name: ${APP_NAME}
namespace: ${K8S_NAMESPACE}
spec:
replicas: 1
selector:
matchLabels:
app: ${APP_NAME}
template:
metadata:
labels:
app: ${APP_NAME}
spec:
imagePullSecrets:
- name: ocirsecret
initContainers:
- name: wait-for-postgres
image: busybox
env:
- name: DB_HOST
value: "10.0.10.15"
command:
- sh
- -c
- |
until nc -z $DB_HOST 5432; do
echo "⏳ Waiting for PostgreSQL..."
sleep 2
done
- name: wait-for-redis
image: busybox
env:
- name: REDIS_HOST
valueFrom:
secretKeyRef:
name: langfuse-external-services
key: REDIS_HOST
command:
- sh
- -c
- |
until nc -z $REDIS_HOST 6379; do
echo "⏳ Waiting for Redis..."
sleep 2
done
- name: wait-for-clickhouse
image: busybox
command:
- sh
- -c
- |
until nc -z ${APP_NAME}-clickhouse 9000; do
echo "⏳ Waiting for ClickHouse..."
sleep 2
done
containers:
- name: ${APP_NAME}
image: ${IMAGE_REPOSITORY}/langfuse:${LANGFUSE_TAG}
ports:
- containerPort: 3000
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: langfuse-external-services
key: DATABASE_URL
- name: NEXTAUTH_SECRET
value: "${NEXTAUTH_SECRET}"
- name: SALT
value: "${SALT}"
- name: ENCRYPTION_KEY
value: "${ENCRYPTION_KEY}"
- name: NEXTAUTH_URL
value: ${NEXTAUTH_URL}
- name: TELEMETRY_ENABLED
value: "false"
- name: CLICKHOUSE_URL
value: "http://${APP_NAME}-clickhouse:8123"
- name: CLICKHOUSE_USER
value: "default"
- name: CLICKHOUSE_PASSWORD
value: "${CLICKHOUSE_PASSWORD}"
- name: CLICKHOUSE_CLUSTER_ENABLED
value: "false"
- name: CLICKHOUSE_MIGRATION_URL
value: "clickhouse://${APP_NAME}-clickhouse:9000"
- name: REDIS_HOST
valueFrom:
secretKeyRef:
name: langfuse-external-services
key: REDIS_HOST
- name: REDIS_AUTH
valueFrom:
secretKeyRef:
name: langfuse-external-services
key: REDIS_AUTH
# ===== OCI OBJECT STORAGE (EVENTS) =====
- name: LANGFUSE_S3_EVENT_UPLOAD_BUCKET
value: "langfuse-events"
- name: LANGFUSE_S3_EVENT_UPLOAD_PREFIX
value: "events/"
- name: LANGFUSE_S3_EVENT_UPLOAD_REGION
value: "${OCI_REGION}"
- name: LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT
value: "https://${OCI_NAMESPACE}.compat.objectstorage.${OCI_REGION}.oraclecloud.com"
- name: LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID
value: "${S3_ACCESS_KEY}"
- name: LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY
value: "${S3_SECRET_KEY}"
- name: LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE
value: "true"
resources:
requests:
cpu: "50m"
memory: "2Gi"
limits:
cpu: "512m"
memory: "3Gi"
livenessProbe:
httpGet:
path: /api/public/health
port: 3000
initialDelaySeconds: 30
periodSeconds: 15
readinessProbe:
httpGet:
path: /api/public/health
port: 3000
initialDelaySeconds: 15
periodSeconds: 10
---
# ===== LANGFUSE WORKER =====
apiVersion: apps/v1
kind: Deployment
metadata:
name: ${APP_NAME}-worker
namespace: ${K8S_NAMESPACE}
spec:
replicas: 1
selector:
matchLabels:
app: ${APP_NAME}-worker
template:
metadata:
labels:
app: ${APP_NAME}-worker
spec:
imagePullSecrets:
- name: ocirsecret
initContainers:
- name: wait-for-postgres
image: busybox
command:
- sh
- -c
- |
until nc -z $DB_HOST 5432; do
echo "⏳ Waiting for PostgreSQL..."
sleep 2
done
- name: wait-for-redis
image: busybox
command:
- sh
- -c
- |
until nc -z $REDIS_HOST 6379; do
echo "⏳ Waiting for Redis..."
sleep 2
done
- name: wait-for-clickhouse
image: busybox
command:
- sh
- -c
- |
until nc -z ${APP_NAME}-clickhouse 9000; do
echo "⏳ Waiting for ClickHouse..."
sleep 2
done
containers:
- name: worker
image: ${IMAGE_REPOSITORY}/langfuse-worker:${LANGFUSE_WORKER_TAG}
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: langfuse-external-services
key: DATABASE_URL
- name: ENCRYPTION_KEY
value: "${ENCRYPTION_KEY}"
- name: CLICKHOUSE_URL
value: "http://${APP_NAME}-clickhouse:8123"
- name: CLICKHOUSE_USER
value: "default"
- name: CLICKHOUSE_PASSWORD
value: "${CLICKHOUSE_PASSWORD}"
- name: CLICKHOUSE_CLUSTER_ENABLED
value: "false"
- name: CLICKHOUSE_MIGRATION_URL
value: "clickhouse://${APP_NAME}-clickhouse:9000"
- name: REDIS_HOST
valueFrom:
secretKeyRef:
name: langfuse-external-services
key: REDIS_HOST
- name: REDIS_PORT
value: "6379"
- name: REDIS_AUTH
valueFrom:
secretKeyRef:
name: langfuse-external-services
key: REDIS_AUTH
# ===== OCI OBJECT STORAGE (EVENTS) =====
- name: LANGFUSE_S3_EVENT_UPLOAD_BUCKET
value: "langfuse-events"
- name: LANGFUSE_S3_EVENT_UPLOAD_PREFIX
value: "events/"
- name: LANGFUSE_S3_EVENT_UPLOAD_REGION
value: "${OCI_REGION}"
- name: LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT
value: "https://${OCI_NAMESPACE}.compat.objectstorage.${OCI_REGION}.oraclecloud.com"
- name: LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID
value: "${S3_ACCESS_KEY}"
- name: LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY
value: "${S3_SECRET_KEY}"
- name: LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE
value: "true"
resources:
requests:
cpu: "50m"
memory: "2Gi"
limits:
cpu: "512m"
memory: "3Gi"
---
# ===== LANGFUSE SERVICE =====
apiVersion: v1
kind: Service
metadata:
name: langfuse-web
namespace: ${K8S_NAMESPACE}
annotations:
service.beta.kubernetes.io/oci-load-balancer-internal: "true"
oci.oraclecloud.com/healthcheck-path: "/api/public/health"
oci.oraclecloud.com/healthcheck-port: "3000"
spec:
type: LoadBalancer
selector:
app: ${APP_NAME}
ports:
- protocol: TCP
port: 80
targetPort: 3000
---
# ===== LANGFUSE HPA - HORIZONTAL POD AUTOSCALER =====
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: ${APP_NAME}-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: ${APP_NAME}
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
---
# ===== WORKER - AUTOSCALER =====
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: langfuse-worker-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: ${APP_NAME}-worker
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
---
apiVersion: v1
kind: Service
metadata:
name: ${APP_NAME}-lb
namespace: ${K8S_NAMESPACE}
annotations:
service.beta.kubernetes.io/oci-load-balancer-shape: "flexible"
oci.oraclecloud.com/healthcheck-path: "/api/public/health"
spec:
type: LoadBalancer
selector:
app: ${APP_NAME}
ports:
- port: 80
targetPort: 3000