forked from diegoecab/oci-deal-accelerator
Diagram generation: ref-arch-driven procedure + spec validator + KB enrichment
The diagram path now follows a documented standard procedure (lookup the closest Oracle Architecture Center reference → confirm components → author absolute_layout → spec validator → render → visually verify) and ships persistent guardrails so layout regressions can't recur. Persistent procedure changes (apply to all users, all sessions): - tools/diagram_spec_validator.py — geometry checks (CONTAINER_TOO_THIN, CONTAINER_PADDING_VIOLATION, LABEL_OVERFLOW_PARENT) run BEFORE either renderer (drawio + PPTX). Catches the subnet-collapse / label-overflow bugs that the post-render drawio validator missed. - tools/oci_diagram_gen.py + tools/oci_pptx_diagram_gen.py — call the spec validator before emitting any output. Adds mysql / mysql_heatwave type aliases. - tools/archcenter_pattern_lookup.py — scores against cached page descriptions (not just the 1-line summary), supports --queries for multi-fragment composition, and applies synonym expansion via kb/architecture-center/synonyms.yaml so "LB HA cross AD" matches "load balancer high availability availability domain". - kb/architecture-center/synonyms.yaml — canonical synonym table (load balancer, autonomous database, data guard, …) used by the lookup scorer. KB enrichment: - tools/archcenter_description_fetcher.py + 121 cached _description.md under kb/diagram/assets/archcenter-refs/<slug>/. Removes the runtime dependency on docs.oracle.com when authoring specs and feeds the pattern-lookup scorer. - 110+ cached .drawio / .svg / .png references for offline reuse, plus the OCI Toolkit v24.2 import (kb/diagram/assets/oci-toolkit-drawio). Documentation: - docs/skill/output-formats.md — new "Standard diagram-generation procedure (MANDATORY)" + geometry rules + the new validator entry. - SKILL.md option 2 — references the mandatory procedure. - README.md — describes the spec validator, archcenter_pattern_lookup and description fetcher, and updates the KB-health table. Tooling that backs the procedure (cumulative across recent sessions): tools/archcenter_case_runner.py, archcenter_batch_driver.py, archcenter_zip_downloader.py, drawio_visual_validator.py, drawio_fidelity_eval.py, harvest_drawio_icon.py, import_oci_library.py, oci_pptx_diagram_gen.py, oci_pptx_render.py, refresh_pptx_icon_index.py. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,285 @@
|
||||
# Run a quantized GGUF large language model on an Ampere A2 cluster
|
||||
|
||||
- Source: https://docs.oracle.com/en/solutions/run-quantized-gguf-llm-ampere-cluster/index.html
|
||||
- Date: 2025-06
|
||||
- Type: reference-architecture
|
||||
- Services: compute
|
||||
- Tags: ai-ml
|
||||
|
||||
## Summary (catalog)
|
||||
|
||||
CPU-based LLM inference on Ampere A2 shapes using quantized GGUF models. Cost-efficient for moderate throughput requirements. llama.cpp for model serving without GPU dependency.
|
||||
|
||||
## Architecture (fetched from source)
|
||||
|
||||
Architecture
|
||||
|
||||
|
||||
|
||||
|
||||
This Arm-based architecture offers an LLM implementation at exceptional
|
||||
value, running at a fraction of the traditional AI infrastructure cost. Use this
|
||||
architecture for a budget-friendly approach to getting started with AI.
|
||||
|
||||
|
||||
An OCI public load balancer sits at the front, distributing incoming traffic to a
|
||||
pool of compute instances. There is an instance pool of Ampere A2 nodes.
|
||||
Each node is a 2-core, Arm-based compute instance running Ubuntu. The nodes
|
||||
are managed in an OCI instance pool, making it easy to scale horizontally as
|
||||
traffic grows. An internet gateway enables public access to both the load
|
||||
balancer and the backend instances when needed.
|
||||
|
||||
|
||||
Each Ampere A2 compute instance runs Ubuntu 22.04 (Arm), a quantized
|
||||
GPT-Generated Unified Format (GGUF) LLM (like TinyLlama or Phi-2) served
|
||||
locally using llama.cpp, a simple HTML/JS landing page served via NGINX, and
|
||||
a Python-based backend wired into llama-cpp-python that handles prompts from
|
||||
the UI and streams model output back to the page.
|
||||
|
||||
|
||||
Each compute node in the pool is designed to be lightweight yet fully
|
||||
self-sufficient. Upon launch, it bootstraps itself using a cloud-init script
|
||||
that installs and runs everything needed to serve an LLM from scratch. The
|
||||
nodes are configured as follows:
|
||||
|
||||
|
||||
|
||||
- Installs Dependencies : Dependencies such as
|
||||
build-essential, cmake, git, NGINX, and python3-pip are installed
|
||||
automatically. llama-cpp-python is compiled from the source to
|
||||
ensure full ARM64 compatibility.
|
||||
|
||||
|
||||
- Builds : Nodes pull the latest version of llama.cpp
|
||||
from GitHub, and builds it using OpenBLAS for optimized CPU
|
||||
inference, while keeping everything local - no external runtime
|
||||
dependency on GPUs, or inference APIs.
|
||||
|
||||
|
||||
- Downloads the Model : A quantized GGUF model
|
||||
(TinyLlama or similar) is fetched directly from Hugging Face and
|
||||
placed in the models directory.
|
||||
|
||||
|
||||
- Serves a Landing Page : A minimal HTML/JavaScript UI is served via
|
||||
NGINX on port 80. The UI lets users submit prompts and view LLM
|
||||
responses directly from their browser.
|
||||
|
||||
|
||||
- Handles Inference via Python : A small Python backend
|
||||
uses llama-cpp-python to interact with the local model. It exposes a
|
||||
/generate endpoint which the landing page
|
||||
sends a POST request to when a user submits a question.
|
||||
|
||||
|
||||
- Starts on Boot : Everything is wrapped in a
|
||||
systemd service, so inference automatically
|
||||
restarts on instance reboot or failure - no manual touch
|
||||
required.
|
||||
|
||||
|
||||
|
||||
The following diagram illustrates this reference architecture.
|
||||
|
||||
|
||||
Description of the illustration gen-ai-ampere-gguf-llm-arch.png
|
||||
|
||||
gen-ai-ampere-gguf-llm-arch.zip
|
||||
|
||||
|
||||
The architecture has the following components:
|
||||
|
||||
|
||||
|
||||
- Region
|
||||
An OCI
|
||||
region is a localized geographic area that
|
||||
contains one or more data centers, hosting
|
||||
availability domains. Regions are independent of
|
||||
other regions, and vast distances can separate
|
||||
them (across countries or even
|
||||
continents).
|
||||
|
||||
|
||||
|
||||
- Availability domains
|
||||
Availability
|
||||
domains are standalone, independent data centers
|
||||
within a region. The physical resources in each
|
||||
availability domain are isolated from the
|
||||
resources in the other availability domains, which
|
||||
provides fault tolerance. Availability domains
|
||||
don’t share infrastructure such as power or
|
||||
cooling, or the internal availability domain
|
||||
network. So, a failure at one availability domain
|
||||
shouldn't affect the other availability domains in
|
||||
the region.
|
||||
|
||||
|
||||
|
||||
- Fault domains
|
||||
A
|
||||
fault domain is a grouping of hardware and
|
||||
infrastructure within an availability domain. Each
|
||||
availability domain has three fault domains with
|
||||
independent power and hardware. When you
|
||||
distribute resources across multiple fault
|
||||
domains, your applications can tolerate physical
|
||||
server failure, system maintenance, and power
|
||||
failures inside a fault domain.
|
||||
|
||||
|
||||
|
||||
- Virtual cloud network (VCN) and subnets
|
||||
A
|
||||
VCN is a customizable, software-defined network
|
||||
that you set up in an OCI region. Like traditional
|
||||
data center networks, VCNs give you control over
|
||||
your network environment. A VCN can have multiple
|
||||
non-overlapping classless inter-domain routing
|
||||
(CIDR) blocks that you can change after you create
|
||||
the VCN. You can segment a VCN into subnets, which
|
||||
can be scoped to a region or to an availability
|
||||
domain. Each subnet consists of a contiguous range
|
||||
of addresses that don't overlap with the other
|
||||
subnets in the VCN. You can change the size of a
|
||||
subnet after creation. A subnet can be public or
|
||||
private.
|
||||
|
||||
|
||||
|
||||
- Load balancer
|
||||
Oracle Cloud
|
||||
Infrastructure Load Balancing provides automated traffic distribution from a
|
||||
single entry point to multiple servers.
|
||||
|
||||
|
||||
|
||||
|
||||
- Internet
|
||||
gateway
|
||||
An
|
||||
internet gateway allows traffic between the public
|
||||
subnets in a VCN and the public internet.
|
||||
|
||||
|
||||
|
||||
- Instance
|
||||
pool
|
||||
An
|
||||
instance pool is a group of instances within a
|
||||
region that are created from the same instance
|
||||
configuration and managed as a group.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Considerations
|
||||
|
||||
|
||||
|
||||
|
||||
Before implementing this architecture, consider the following.
|
||||
|
||||
|
||||
|
||||
- Ampere A2 compute instance costs
|
||||
Each node runs Ampere A2 with
|
||||
2 OCPUs and 16GB of RAM. The pricing for this OCPU
|
||||
is currently $0.01 per OCPU per hour. The monthly
|
||||
cost works out to $14.40 with 1 node always
|
||||
on.
|
||||
|
||||
|
||||
|
||||
- Load balancer costs
|
||||
A public load balancer (small shape) is
|
||||
priced currently at approximately $0.029 per hour. The monthly cost works out to
|
||||
approximately $21. You can further reduce costs by setting up a custom load
|
||||
balancer on another Ampere instance.
|
||||
|
||||
|
||||
|
||||
- Storage costs
|
||||
Each node stores the OS, llama.cpp, and the model
|
||||
which is approximately 5-6GB. The default boot
|
||||
volume is approximately 50GB. Note the first 200GB
|
||||
per month are free.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Explore More
|
||||
|
||||
|
||||
|
||||
|
||||
Learn more about running a GGUF LLM on an Ampere A2 cluster in Oracle Cloud
|
||||
Infrastructure .
|
||||
|
||||
|
||||
|
||||
Review these additional resources:
|
||||
|
||||
|
||||
|
||||
- Oracle Cloud
|
||||
Infrastructure Documentation
|
||||
|
||||
- Well-architected framework
|
||||
for Oracle Cloud Infrastructure
|
||||
|
||||
- Oracle Cloud Cost Estimator
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Acknowledgments
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
- Author : Badr Tharwat
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Title and Copyright Information
|
||||
|
||||
|
||||
|
||||
|
||||
Run a quantized GGUF large language model on an Ampere A2 cluster
|
||||
|
||||
|
||||
G35514-01
|
||||
|
||||
|
||||
June 2025
|
||||
|
||||
|
||||
Copyright © 2025,
|
||||
|
||||
Oracle and/or its affiliates.
|
||||
File diff suppressed because one or more lines are too long
Binary file not shown.
|
After Width: | Height: | Size: 33 KiB |
File diff suppressed because one or more lines are too long
|
After Width: | Height: | Size: 52 KiB |
Binary file not shown.
@@ -0,0 +1,7 @@
|
||||
The diagram you downloaded is available in these formats:
|
||||
- DRAWIO
|
||||
- SVG
|
||||
|
||||
You can customize them for your organization using the associated tools:
|
||||
- For DRAWIO format, use draw.io for Confluence, online at diagrams.net, or the desktop app. Go to diagrams.net for more information.
|
||||
- For SVG format, use an SVG editor such as Inkscape or Sketsa SVG Editor, which are free and available for Windows, macOS, Linux.
|
||||
Reference in New Issue
Block a user