Platform Monkey ctx

RAG vs. Fine-Tune vs. Prompt Caching: The Architecture Decision Tree (with Real Deployments)

· By
Sources

For Senior Architects, the question is rarely "Can the model do this?" but "How do I inject knowledge without bankrupting the compute budget or introducing stochastic hallucinations?"

Most teams treat RAG and Fine-Tuning as binary choices. In production, they are orthogonal levers. Prompt Caching is the third, often ignored lever that solves the "long-context tax" for stable instructions.

The Architectural Decision Tree


START: I need the model to know [Specific Domain Knowledge]
│
├── Is the data volatile (changes hourly/daily)?
│   └── YES ───▶ [ RAG (Retrieval-Augmented Generation) ]
│               (Fact-grounded, traceable, real-time)
│
├── Is the goal to change "how" the model speaks (style, format, syntax)?
│   └── YES ───▶ [ Fine-Tuning (PEFT/QLoRA) ]
│               (Domain-specialized, low-latency, structural)
│
└── Is there a large, stable prefix (1k+ tokens) reused across requests?
    └── YES ───▶ [ Prompt Caching ]
                (Latency reduction, 90% input cost savings)
    └── NO  ───▶ [ Standard Prompting ]

HYBRID PATHS:
- RAG + Fine-Tuning (RAFT): Fine-tune to recognize RAG-style context + RAG for facts.
- RAG + Caching: Cache the system prompt and tool schemas; RAG for the dynamic context.

1. Prompt Caching: The "Low Hanging Fruit"

Prompt caching is a server-side optimization that preserves the KV (Key-Value) tensors of a prompt prefix. When a subsequent request shares the same prefix, the model skips the prefill stage for those tokens.

Production Mechanics: OpenAI vs. Anthropic

The industry has converged on two TTL (Time-To-Live) tiers. The critical architectural decision is choosing the write-cost vs. read-hit trade-off.


| Provider  | TTL Tier | Write Cost | Read Cost | Min Length | Logic |
|-----------|----------|------------|-----------|------------|---------------------------------------------------|
| OpenAI    | 30m      | 1.25x Input| 0.1x Input| 1,024 tok  | Default; good for bursty agent turns.            |
| OpenAI    | 12h      | 2.5x Input | 0.1x Input| 1,024 tok  | Stable docs; pays if reused >2x in 12h window.   |
| Anthropic | 5m       | 1.25x Input| 0.1x Input| 512-4k tok | High turnover; minimal write penalty.              |
| Anthropic | 1h       | 2.0x Input | 0.1x Input| 512-4k tok | Stable instructions; pays if reused after 5m gap.  |

Implementation Blueprint

To maximize hit rates, you must stabilize the prefix. Volatile data (timestamps, user IDs) must be pushed to the end of the prompt.


import openai

# Use explicit breakpoints to avoid writing volatile suffixes to cache
response = openai.chat.completions.create(
    model="gpt-5.6",
    prompt_cache_options={
        "mode": "explicit",
        "ttl": "12h"
    },
    messages=[
        {
            "role": "developer", 
            "content": [
                {"type": "text", "text": "You are a Senior Platform Engineer... [10k tokens of stable docs]"},
                # Explicit breakpoint: everything before this is cached for 12h
                {"type": "prompt_cache_breakpoint", "mode": "explicit"}
            ]
        },
        {"role": "user", "content": "How do I configure the TGW NAT for this specific VPC?"}
    ]
)

2. RAG: The "Fact Engine"

RAG is a retrieval pipeline that injects external data into the prompt at runtime. It is the only viable pattern for data with high volatility or where a citation is required for auditability.

Advanced Production Architecture

Simple "vector search" fails in production due to the "lost in the middle" phenomenon. A world-class RAG blueprint requires a Hybrid Search + Re-ranking flow.


User Query ──▶ [Query Expansion/HyDE] ──▶ [Hybrid Retriever] ──▶ [Re-Ranker] ──▶ [Context Window] ──▶ LLM
                                           │ (Vector + BM25)       │ (Cross-Encoder)
                                           ▼                       ▼
                                     [Vector DB]             [Top K Scores]


from qdrant_client import QdrantClient
from sentence_transformers import CrossEncoder

client = QdrantClient("localhost", port=6333)
reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')

def retrieve_and_rank(query, top_k=5):
    # 1. Hybrid Search: Combine dense (vector) and sparse (BM25) results
    search_results = client.search(
        collection_name="platform_docs",
        query_vector=embedding_model.encode(query),
        query_text=query, # triggers sparse search if configured
        limit=20 
    )
    
    # 2. Re-ranking: Use a Cross-Encoder to score the top 20 results accurately
    pairs = [[query, res.payload['text']] for res in search_results]
    scores = reranker.predict(pairs)
    
    # Sort by score and return top_k
    ranked_results = [x for _, x in sorted(zip(scores, search_results), reverse=True)]
    return ranked_results[:top_k]

3. Fine-Tuning: The "Style & Domain Engine"

Fine-tuning is not for adding facts; it is for teaching the model a new language, a specific format, or deep domain intuition. For 98% of production use cases, QLoRA (Quantized Low-Rank Adaptation) is the optimal strategy.

The QLoRA Blueprint

QLoRA freezes the base model in 4-bit NF4 quantization and trains only small BF16 adapter matrices. This reduces VRAM requirements by up to 90% while retaining ~95% of full-parameter performance.


from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

# 1. 4-bit Quantization Config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="bfloat16",
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3-70B", 
    quantization_config=bnb_config,
    device_map="auto"
)

# 2. LoRA Adapter Config
lora_config = LoraConfig(
    r=64, 
    lora_alpha=128, 
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"], 
    lora_dropout=0.05, 
    bias="none", 
    task_type="CAUSAL_LM"
)

model = prepare_model_for_kbit_training(model)
model = get_peft_model(model, lora_config)

Production Realities: The Cost & Performance Matrix

Choosing the wrong pattern leads to "The Hallucination Gap" (RAG) or "The Knowledge Drift" (FT). Use the following matrix to calibrate your choice.


| Metric            | Prompt Caching | RAG (Hybrid)        | Fine-Tuning (QLoRA)  |
|-------------------|----------------|---------------------|----------------------|
| **Data Volatility**| High (Implicit)| High (Explicit)      | Low (Static)        |
| **Fact Accuracy**  | Model-dependent| High (with Citations)| Medium (Hallucinates)|
| **Update Cost**   | Zero (Auto)    | Low (DB Update)     | High (Re-train)      |
| **Inference Latency**| -80% TTFT     | +Search Latency     | No overhead (merged) |
| **VRAM Cost**     | Low (Server)   | Medium (Vector DB)   | Low (Adapter)       |
| **Specialization**| Zero           | Context-based       | Deep Structural     |

Operational Toil & Hidden Costs

Who Should Skip This?

Do NOT implement these patterns if:

References & Further Reading