diff --git a/README.md b/README.md index a416cb5..8366340 100644 --- a/README.md +++ b/README.md @@ -89,8 +89,8 @@ O mesmo ocorre para: Existem 2 arquivos de deployment neste material: -- langfuse_dist.yaml -- final.yaml +- **langfuse_dist.yaml** (ou **langfuse_dist_cloud.yaml** para utilização de storage externo ao clustes Kubernetes) +- **final.yaml** Ambos os arquivos contém: - StorageClass @@ -102,8 +102,15 @@ Ambos os arquivos contém: Para ambientes produtivos, use o arquivo **langfuse_dist.yaml**. As variáveis de ambiente já estão preparadas para cada componente do langfuse. + Se deseja validar de forma ilustrada e passar por todos os componentes deste material, use as ferramentas mirror_images.sh, create_secrets.sh, update_secret.sh e set_var.sh para montar um arquivo YAML de deployment como tudo ajustado sem necessidade de criar um pipeline DEVOPS para isto. +## Persistência fora do Cluster Kubernetes + +O arquivo **langfuse_dist_cloud.yaml** permite que se utilize o PostgreSQL e Redis fora do cluster Kubernetes. Isto permite que se faça a governança destes recursos de forma mais tradicional, além de maior economia na escalabilidade e melhoras em troubleshooting em geral. + +Caso use a opção de utilizar os recursos de PostgreSQL e Redis dentro do cluster Kubernetes, existe um pacote adicional neste tutorial chamado **langfuse-k8s-cleanup.zip** onde há mecanismos para o clean-up de forma agendada dentro do próprio cluster Kubernetes. + ## PostgreSQL Responsável por: @@ -1102,70 +1109,369 @@ Define se ClickHouse opera em cluster distribuído. --- -## LANGFUSE_S3_EVENT_UPLOAD_BUCKET +# Configuração OCI Object Storage para Upload de Eventos no Langfuse -```bash id="vars44" -export LANGFUSE_S3_EVENT_UPLOAD_BUCKET="langfuse-events" -``` +## Objetivo -### Objetivo +Estas variáveis configuram o Langfuse para armazenar eventos, traces e arquivos no OCI Object Storage utilizando a API compatível com Amazon S3. -Bucket de eventos. +O Langfuse utiliza esse storage para: + +- Persistência de eventos +- Upload de traces grandes +- Armazenamento de payloads +- Redução de carga no PostgreSQL +- Redução de crescimento do ClickHouse +- Armazenamento distribuído e durável --- +# Variáveis de Ambiente + +## LANGFUSE_S3_EVENT_UPLOAD_BUCKET + +```yaml +- name: LANGFUSE_S3_EVENT_UPLOAD_BUCKET + value: "langfuse-events" +``` + +Objetivo + +Define o bucket do OCI Object Storage onde os eventos serão armazenados. + +Observações + +-O bucket deve existir previamente. +-O bucket precisa estar na mesma região configurada. +-Pode ser Standard ou Archive (recomendado Standard). + + oci os bucket create \ + --name langfuse-events \ + --compartment-id + +## LANGFUSE_S3_EVENT_UPLOAD_PREFIX + +```yaml +- name: LANGFUSE_S3_EVENT_UPLOAD_PREFIX + value: "events/" +``` + +Objetivo + +Define o prefixo (pasta lógica) dentro do bucket. + +Resultado + +Os objetos serão gravados em: + + langfuse-events/events/ + +Exemplo de objeto + + events/traces/abc123.json + +Benefícios + +Permite: + +- Separar ambientes +- Organizar dados +- Facilitar lifecycle policies +- Facilitar limpeza automática + +Exemplos úteis + +Produção: + + value: "prod/events/" + +Homologação: + + value: "hml/events/" + +Multi-Tenant: + + value: "tenant-a/events/" + ## LANGFUSE_S3_EVENT_UPLOAD_REGION -```bash id="vars45" -export LANGFUSE_S3_EVENT_UPLOAD_REGION="${OCI_REGION}" -``` + - name: LANGFUSE_S3_EVENT_UPLOAD_REGION + value: "${OCI_REGION}" -### Objetivo +Objetivo -Região do bucket. +Define a região OCI do Object Storage. ---- +Exemplos + + sa-saopaulo-1 + us-chicago-1 + eu-frankfurt-1 + +Importância + +A região precisa coincidir com: + +* bucket criado +* endpoint S3 +* credenciais utilizadas + +Exemplo + + value: "sa-saopaulo-1" ## LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT + + - name: LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT + value: "https://${OCI_NAMESPACE}.compat.objectstorage.${OCI_REGION}.oraclecloud.com" -```bash id="vars46" -export LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT="https://${OCI_NAMESPACE}.compat.objectstorage.${OCI_REGION}.oraclecloud.com" -``` +Objetivo -### Objetivo +Define o endpoint S3 compatível da OCI. -Endpoint S3-compatible OCI. +O OCI Object Storage possui uma API compatível com Amazon S3. ---- +O Langfuse usa esse endpoint como se fosse AWS S3. -## LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE +⸻ -```bash id="vars47" -export LANGFUSE_USE_OCI_NATIVE_OBJECT_STORAGE="true" -``` +Estrutura do endpoint -### Objetivo + https://.compat.objectstorage..oraclecloud.com -Ativa integração OCI nativa. +Exemplo real ---- + https://mycompany.compat.objectstorage.sa-saopaulo-1.oraclecloud.com -## LANGFUSE_OCI_AUTH_TYPE -```bash id="vars48" -export LANGFUSE_OCI_AUTH_TYPE="instance_principal" -``` +O que é OCI_NAMESPACE -### Objetivo +O namespace é um identificador único global do tenancy OCI. -Utiliza Instance Principal em vez de Access Key. +Exemplo -### Benefício + xkthfgahad -Não precisamos: +Como descobrir o namespace -* access key -* secret key +OCI CLI + + oci os ns get + +Console OCI + +Object Storage → Namespace + +⸻ + +## LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID + + - name: LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID + value: "${S3_ACCESS_KEY}" + +Objetivo + +Define o Access Key utilizado para autenticação S3. + +Funciona similar ao AWS Access Key ID. + +>**Nota:** O par de keys de Custom Keys não pode ser gerado automaticamente pois não é fornecido o Secret. Isso só é possível via Console OCi + + Identity & Security + + → Users + + → Seu usuário + + → Customer Secret Keys + + → Generate Secret Key + +## LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY + + - name: LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY + value: "${S3_SECRET_KEY}" + +Objetivo + +Define o segredo associado ao Access Key. + +É equivalente ao AWS Secret Access Key. + +⸻ + +Observações importantes + +* A secret é exibida apenas UMA vez na criação. +* Deve ser armazenada com segurança. +* Idealmente usar OCI Vault. + +⸻ + +Recomendação + +Nunca colocar diretamente no YAML. + +Preferir: + + valueFrom: + secretKeyRef: + name: object-storage-secret + key: S3_SECRET_KEY + +## LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE + + - name: LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE + value: "true" + +Objetivo + +Força o cliente S3 a usar Path Style ao invés de Virtual Host Style. + +⸻ + +Diferença entre estilos + +Virtual Host Style (AWS padrão) + + https://bucket.endpoint/object + +Exemplo: + + https://langfuse-events.s3.amazonaws.com/file.json + +Path Style + + https://endpoint/bucket/object + +Exemplo: + + https://compat.objectstorage.sa-saopaulo-1.oraclecloud.com/langfuse-events/file.json + + +Por que OCI precisa disso + +O endpoint compatível S3 da OCI funciona melhor com: + + LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE=true + + +Sem isso podem ocorrer: + +- erro de assinatura +- erro de hostname +- erro 403 +- bucket não encontrado + +### Fluxo de Funcionamento + + Langfuse + ↓ + S3 Compatible Client + ↓ + OCI Object Storage Endpoint + ↓ + Bucket OCI + ↓ + Persistência de Eventos + +Benefícios dessa Arquitetura + +Redução de Crescimento do PostgreSQL + +Payloads grandes não ficam somente no banco. + +⸻ + +Melhor Escalabilidade + +Object Storage escala automaticamente. + +⸻ + +Alta Durabilidade + +OCI Object Storage possui alta redundância. + +⸻ + +Menor Custo + +Storage é mais barato que expandir PostgreSQL. + +⸻ + +Facilidade de Backup + +Object Storage já possui durabilidade nativa. + +⸻ + +Recomendações Enterprise + +Usar OCI Vault + +Evitar secrets hardcoded. + +⸻ + +Usar Instance Principal + +Evitar Access Key fixa. + +⸻ + +Criar Lifecycle Policies + +Exemplo: + + Apagar eventos após 30 dias + Mover para Archive após 7 dias + +### Exemplo de Lifecycle Policy + + Object Storage + → Bucket + → Lifecycle Policy + +### Segurança Recomendada + +Bucket privado + +Nunca público. + +⸻ + +Network privada + +Preferencialmente: + +* OKE privado +* Object Storage via Service Gateway + +⸻ + +Policies mínimas + +Exemplo: + + Allow dynamic-group oke-cluster to manage objects in compartment observability + + + +⸻ + +Importante + +No OCI, esse Access Key pertence a um usuário OCI. + +Não é o OCID do usuário. + +⸻ + +Como gerar + +OCI Console --- @@ -1405,6 +1711,9 @@ Ao final: # Referências - [Langfuse - OCI Object Storage (Native)](https://langfuse.com/self-hosting/deployment/infrastructure/blobstorage#oci-object-storage) +- [OCI Object Storage](https://docs.oracle.com/en-us/iaas/Content/Object/Concepts/objectstorageoverview.htm) +- [OCI S3 Compatibility API](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) +- [Langfuse Self Hosting](https://langfuse.com/self-hosting) # Acknowledgments diff --git a/src/langfuse-k8s-cleanup.zip b/src/langfuse-k8s-cleanup.zip new file mode 100644 index 0000000..3fb8dbd Binary files /dev/null and b/src/langfuse-k8s-cleanup.zip differ diff --git a/src/langfuse_dist_cloud.yaml b/src/langfuse_dist_cloud.yaml new file mode 100644 index 0000000..cf207df --- /dev/null +++ b/src/langfuse_dist_cloud.yaml @@ -0,0 +1,442 @@ +apiVersion: v1 +kind: Secret +metadata: + name: langfuse-external-services + namespace: ${K8S_NAMESPACE} +type: Opaque +stringData: + # PostgreSQL OCI + DATABASE_URL: "postgresql://langfuse:${DB_PASSWORD}@10.0.10.15:5432/langfuse" + # Redis OCI + REDIS_HOST: "10.0.20.10" + REDIS_PORT: "6379" + REDIS_AUTH: "${REDIS_AUTH}" + # ClickHouse + CLICKHOUSE_PASSWORD: "${CLICKHOUSE_PASSWORD}" + # Langfuse + NEXTAUTH_SECRET: "${NEXTAUTH_SECRET}" + SALT: "${SALT}" + ENCRYPTION_KEY: "${ENCRYPTION_KEY}" + # OCI Object Storage + S3_ACCESS_KEY: "${S3_ACCESS_KEY}" + S3_SECRET_KEY: "${S3_SECRET_KEY}" +--- +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: ${APP_NAME}-clickhouse + namespace: ${K8S_NAMESPACE} +spec: + serviceName: ${APP_NAME}-clickhouse + replicas: 1 + selector: + matchLabels: + app: ${APP_NAME}-clickhouse + template: + metadata: + labels: + app: ${APP_NAME}-clickhouse + spec: + imagePullSecrets: + - name: ocirsecret + containers: + - name: clickhouse + image: ${IMAGE_REPOSITORY}/clickhouse:${CLICKHOUSE_TAG} + ports: + - containerPort: 8123 + - containerPort: 9000 + resources: + requests: + cpu: "2" + memory: "8Gi" + limits: + cpu: "4" + memory: "16Gi" + volumeMounts: + - name: data + mountPath: /var/lib/clickhouse + env: + - name: CLICKHOUSE_DB + value: "default" + - name: CLICKHOUSE_USER + value: "default" + - name: CLICKHOUSE_PASSWORD + value: "${CLICKHOUSE_PASSWORD}" + - name: CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT + value: "1" + volumeClaimTemplates: + - metadata: + name: data + spec: + accessModes: ["ReadWriteOnce"] + storageClassName: oci-bv-high-performance + resources: + requests: + storage: 200Gi +--- +apiVersion: v1 +kind: Service +metadata: + name: ${APP_NAME}-clickhouse + namespace: ${K8S_NAMESPACE} +spec: + selector: + app: ${APP_NAME}-clickhouse + ports: + - name: http + port: 8123 + - name: native + port: 9000 +--- +# ===== LANGFUSE APPLICATION ===== + +apiVersion: apps/v1 +kind: Deployment +metadata: + name: ${APP_NAME} + namespace: ${K8S_NAMESPACE} +spec: + replicas: 1 + selector: + matchLabels: + app: ${APP_NAME} + template: + metadata: + labels: + app: ${APP_NAME} + spec: + imagePullSecrets: + - name: ocirsecret + + initContainers: + + - name: wait-for-postgres + image: busybox + env: + + - name: DB_HOST + + value: "10.0.10.15" + command: + - sh + - -c + - | + until nc -z $DB_HOST 5432; do + echo "⏳ Waiting for PostgreSQL..." + sleep 2 + done + + - name: wait-for-redis + image: busybox + env: + - name: REDIS_HOST + valueFrom: + secretKeyRef: + name: langfuse-external-services + key: REDIS_HOST + command: + - sh + - -c + - | + until nc -z $REDIS_HOST 6379; do + echo "⏳ Waiting for Redis..." + sleep 2 + done + + - name: wait-for-clickhouse + image: busybox + + command: + - sh + - -c + - | + until nc -z ${APP_NAME}-clickhouse 9000; do + echo "⏳ Waiting for ClickHouse..." + sleep 2 + done + containers: + - name: ${APP_NAME} + image: ${IMAGE_REPOSITORY}/langfuse:${LANGFUSE_TAG} + ports: + - containerPort: 3000 + env: + - name: DATABASE_URL + valueFrom: + secretKeyRef: + name: langfuse-external-services + key: DATABASE_URL + + - name: NEXTAUTH_SECRET + value: "${NEXTAUTH_SECRET}" + - name: SALT + value: "${SALT}" + - name: ENCRYPTION_KEY + value: "${ENCRYPTION_KEY}" + - name: NEXTAUTH_URL + value: ${NEXTAUTH_URL} + - name: TELEMETRY_ENABLED + value: "false" + - name: CLICKHOUSE_URL + value: "http://${APP_NAME}-clickhouse:8123" + - name: CLICKHOUSE_USER + value: "default" + - name: CLICKHOUSE_PASSWORD + value: "${CLICKHOUSE_PASSWORD}" + - name: CLICKHOUSE_CLUSTER_ENABLED + value: "false" + - name: CLICKHOUSE_MIGRATION_URL + value: "clickhouse://${APP_NAME}-clickhouse:9000" + + - name: REDIS_HOST + valueFrom: + secretKeyRef: + name: langfuse-external-services + key: REDIS_HOST + + - name: REDIS_AUTH + valueFrom: + secretKeyRef: + name: langfuse-external-services + key: REDIS_AUTH + + # ===== OCI OBJECT STORAGE (EVENTS) ===== + - name: LANGFUSE_S3_EVENT_UPLOAD_BUCKET + value: "langfuse-events" + + - name: LANGFUSE_S3_EVENT_UPLOAD_PREFIX + value: "events/" + + - name: LANGFUSE_S3_EVENT_UPLOAD_REGION + value: "${OCI_REGION}" + + - name: LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT + value: "https://${OCI_NAMESPACE}.compat.objectstorage.${OCI_REGION}.oraclecloud.com" + + - name: LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID + value: "${S3_ACCESS_KEY}" + + - name: LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY + value: "${S3_SECRET_KEY}" + + - name: LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE + value: "true" + resources: + requests: + cpu: "50m" + memory: "2Gi" + limits: + cpu: "512m" + memory: "3Gi" + livenessProbe: + httpGet: + path: /api/public/health + port: 3000 + initialDelaySeconds: 30 + periodSeconds: 15 + readinessProbe: + httpGet: + path: /api/public/health + port: 3000 + initialDelaySeconds: 15 + periodSeconds: 10 + +--- +# ===== LANGFUSE WORKER ===== + +apiVersion: apps/v1 +kind: Deployment +metadata: + name: ${APP_NAME}-worker + namespace: ${K8S_NAMESPACE} +spec: + replicas: 1 + selector: + matchLabels: + app: ${APP_NAME}-worker + template: + metadata: + labels: + app: ${APP_NAME}-worker + spec: + imagePullSecrets: + - name: ocirsecret + + initContainers: + + - name: wait-for-postgres + image: busybox + + command: + - sh + - -c + - | + until nc -z $DB_HOST 5432; do + echo "⏳ Waiting for PostgreSQL..." + sleep 2 + done + + - name: wait-for-redis + image: busybox + + command: + - sh + - -c + - | + until nc -z $REDIS_HOST 6379; do + echo "⏳ Waiting for Redis..." + sleep 2 + done + + - name: wait-for-clickhouse + image: busybox + + command: + - sh + - -c + - | + until nc -z ${APP_NAME}-clickhouse 9000; do + echo "⏳ Waiting for ClickHouse..." + sleep 2 + done + containers: + - name: worker + image: ${IMAGE_REPOSITORY}/langfuse-worker:${LANGFUSE_WORKER_TAG} + env: + - name: DATABASE_URL + valueFrom: + secretKeyRef: + name: langfuse-external-services + key: DATABASE_URL + - name: ENCRYPTION_KEY + value: "${ENCRYPTION_KEY}" + - name: CLICKHOUSE_URL + value: "http://${APP_NAME}-clickhouse:8123" + - name: CLICKHOUSE_USER + value: "default" + - name: CLICKHOUSE_PASSWORD + value: "${CLICKHOUSE_PASSWORD}" + - name: CLICKHOUSE_CLUSTER_ENABLED + value: "false" + - name: CLICKHOUSE_MIGRATION_URL + value: "clickhouse://${APP_NAME}-clickhouse:9000" + + - name: REDIS_HOST + valueFrom: + secretKeyRef: + name: langfuse-external-services + key: REDIS_HOST + + - name: REDIS_PORT + value: "6379" + + - name: REDIS_AUTH + valueFrom: + secretKeyRef: + name: langfuse-external-services + key: REDIS_AUTH + + # ===== OCI OBJECT STORAGE (EVENTS) ===== + - name: LANGFUSE_S3_EVENT_UPLOAD_BUCKET + value: "langfuse-events" + + - name: LANGFUSE_S3_EVENT_UPLOAD_PREFIX + value: "events/" + + - name: LANGFUSE_S3_EVENT_UPLOAD_REGION + value: "${OCI_REGION}" + + - name: LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT + value: "https://${OCI_NAMESPACE}.compat.objectstorage.${OCI_REGION}.oraclecloud.com" + + - name: LANGFUSE_S3_EVENT_UPLOAD_ACCESS_KEY_ID + value: "${S3_ACCESS_KEY}" + + - name: LANGFUSE_S3_EVENT_UPLOAD_SECRET_ACCESS_KEY + value: "${S3_SECRET_KEY}" + + - name: LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE + value: "true" + resources: + requests: + cpu: "50m" + memory: "2Gi" + limits: + cpu: "512m" + memory: "3Gi" +--- +# ===== LANGFUSE SERVICE ===== + +apiVersion: v1 +kind: Service +metadata: + name: langfuse-web + namespace: ${K8S_NAMESPACE} + annotations: + service.beta.kubernetes.io/oci-load-balancer-internal: "true" + oci.oraclecloud.com/healthcheck-path: "/api/public/health" + oci.oraclecloud.com/healthcheck-port: "3000" +spec: + type: LoadBalancer + selector: + app: ${APP_NAME} + ports: + - protocol: TCP + port: 80 + targetPort: 3000 +--- +# ===== LANGFUSE HPA - HORIZONTAL POD AUTOSCALER ===== + +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: ${APP_NAME}-hpa +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: ${APP_NAME} + minReplicas: 2 + maxReplicas: 10 + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: 70 +--- +# ===== WORKER - AUTOSCALER ===== +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: langfuse-worker-hpa +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: ${APP_NAME}-worker + minReplicas: 2 + maxReplicas: 10 + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: 70 +--- +apiVersion: v1 +kind: Service +metadata: + name: ${APP_NAME}-lb + namespace: ${K8S_NAMESPACE} + annotations: + service.beta.kubernetes.io/oci-load-balancer-shape: "flexible" + oci.oraclecloud.com/healthcheck-path: "/api/public/health" +spec: + type: LoadBalancer + selector: + app: ${APP_NAME} + ports: + - port: 80 + targetPort: 3000 \ No newline at end of file