Practical Guide to GenAI on AWS

We are, for the first time, building systems that can read, reason, and act on our behalf at a scale no team of humans ever could — and it is worth pausing, once, to notice how strange and new that is. Then we go back to engineering it.

I've shipped this stack across healthcare, media, insurance, and finance, and the failure pattern repeats regardless of industry: most generative AI pilots die between the demo and production — not because the model underperforms, but because the surrounding infrastructure was never designed for real traffic, real cost controls, or real data governance. A claims workflow, a content pipeline, and a clinical intake form have almost nothing in common on the surface; underneath, they fail the same way, for the same reasons. Deploying GenAI on AWS well means treating the model as one component in a system, not the system itself. This guide walks through the full stack: where the LLM sits, how retrieval grounds it in real data, how agents turn it into automation, what it costs at scale, and how to evaluate whether any of it is actually working.

Bedrock is the fastest path to a managed foundation model without standing up your own inference infrastructure, but the decision that matters most is retrieval: how you ground the model in your organization's actual data. A well-built RAG pipeline on top of OpenSearch or a vector-enabled Aurora instance will outperform a bigger model with no grounding, every time.

Choosing where the LLM sits

Bedrock gives you a menu of foundation models (Claude, Llama, Titan, Mistral) behind one API, with no GPU fleet to manage. That's the right default for most teams. Reach for SageMaker JumpStart or a self-hosted endpoint only when you need a fine-tuned open-weight model, strict data residency the managed service can't guarantee, or throughput patterns that make provisioned capacity cheaper than on-demand tokens. Don't self-host to save money before you've measured — the operational cost of running inference infrastructure usually exceeds the token savings until volume is very high.

Model choice within Bedrock is a routing problem, not a one-time decision — a lesson I've relearned in every industry I've built in. A production system typically runs several models side by side, and the routing logic itself is usually just another small, cheap LLM call classifying the incoming request before it ever reaches the expensive model.

Sending every request to the biggest model is the single most common source of runaway cost — most incoming requests are simple enough for the cheapest tier, and the routing layer's only job is telling them apart before money is spent.

Retrieval-augmented generation, in practice

RAG is three pipelines wearing one name: ingestion, retrieval, and generation. Most RAG quality problems trace back to chunking and retrieval, not the model. Chunk boundaries that split a table or cut a definition in half will quietly degrade every downstream answer, and no amount of prompt engineering fixes that upstream.

On AWS, OpenSearch Service (with its k-NN plugin) and vector-enabled Aurora PostgreSQL (pgvector) are the two realistic default stores. OpenSearch suits high-volume hybrid search with existing text infrastructure; Aurora pgvector suits teams already running Postgres who want one fewer system to operate. Bedrock Knowledge Bases wraps ingestion and retrieval into a managed layer over either, which is worth using until you hit a retrieval-quality ceiling the managed defaults can't clear — re-ranking, custom chunking strategies, or metadata filtering usually buy the next increment of quality.

Freshness is the failure mode teams miss: if ingestion only runs on a nightly batch, an agent will confidently answer with yesterday's data. Wire ingestion to S3 event notifications so a document update triggers re-chunking and re-embedding within minutes, not overnight.

Agentic workflows and automation

The step past "answer a question with retrieved context" is letting the model take actions — call internal APIs, query a database, kick off a downstream job — and decide which action to take based on the request. Bedrock AgentCore is now the current managed path for this (Bedrock Agents Classic stopped onboarding new customers in mid-2026), or a hand-rolled orchestration loop with Step Functions and Lambda — either way the model gets a set of tools with defined schemas and plans a sequence of calls rather than emitting free text. This is where most of the real automation value shows up: routing support tickets, reconciling records across systems, drafting and filing structured reports — tasks that are too varied for a fixed script but too repetitive to have a person do by hand.

Two disciplines separate a reliable agent from a demo that breaks in week two, and I've paid for skipping both. First, keep the action space narrow and explicit — every tool the model can call should have a tight schema, input validation, and a blast radius you've thought through, because an agent will eventually call a tool with malformed or adversarial input. Second, put a human approval step in front of any action with real-world consequences (sending an email, modifying a record, spending money) until the agent's decisions have a track record. In regulated environments — claims adjudication, underwriting, clinical documentation — that approval step isn't optional friction, it's the control your compliance team will ask for anyway. The automation value comes from cutting the work down to a review, not from removing the review entirely.

Step Functions is the underrated piece here: it gives you retries, timeouts, parallel branches, and a visual execution history for free, which turns an agent loop from an opaque chain of LLM calls into something you can debug at 2am. Treat the orchestration layer as infrastructure, not glue code. Common automation targets that fit this pattern well: triaging inbound support tickets by intent and urgency, reconciling mismatched records between two systems of record, extracting structured fields from unstructured documents (invoices, contracts, intake forms), and drafting first-pass responses or reports that a human edits rather than writes from scratch.

Self-checking agents

The cheapest reliability upgrade in an agent loop isn't a bigger model — it's a second, smaller model whose only job is to check the first one's work before it ships. A self-checking agent runs its own output back through a verifier step: did the tool call match the schema, does the answer actually cite retrieved context instead of hallucinating around it, does a structured extraction have all required fields. When the check fails, the loop retries or escalates instead of returning a bad result. I've leaned on this pattern hardest in claims and underwriting work, where a wrong extracted number doesn't just look bad, it becomes a liability.

Keep the verifier cheap and narrow — it doesn't need to be smart, it needs to be a strict, deterministic-as-possible gate: schema validation, a rubric check, a lookup against source data. A verifier that's just "ask a bigger model if this looks right" is better than nothing, but a verifier that checks something concrete (numbers match the source document, required fields are non-null, the cited passage actually contains the claim) catches failures the first model can't see in its own output. Cap the retry count — an agent that keeps failing its own check should escalate to a person, not loop forever burning tokens.

Cost and scale

Token usage scales with usage patterns you can't fully predict at design time, so budget guardrails — usage caps, model tiering, caching repeated queries — need to be built in from day one, not bolted on after the first surprise bill. Prompt caching (supported by several Bedrock models) can cut costs substantially for workloads that reuse a large, stable context — a system prompt, a document, a schema — across many requests, since only the new portion of the prompt is billed at full rate.

Batch inference is the other lever most teams skip: anything that doesn't need a real-time response — bulk classification, nightly summarization, backfilling embeddings — belongs in Bedrock's batch API, priced well below on-demand and without holding up a synchronous request path.

Combine that with CloudWatch-based token and cost dashboards per workload, not just per account, or the first bill spike will arrive with no way to trace it back to a cause.

Evaluation, the part everyone skips

Treat evaluation as a first-class part of the system. Without a repeatable way to measure output quality against real use cases, you can't tell the difference between a model upgrade that helped and one that quietly made things worse. I've watched teams ship a "better" model that quietly tanked a downstream metric no one was watching — an approval rate, a churn signal, a readmission flag — because the eval set never grew past the demo queries. Build a small, versioned test set from real production queries early, and re-run it on every prompt, model, or retrieval change — not just at launch.

For open-ended generation, pair automated checks (does the output contain required fields, pass a schema check, stay under a length bound) with periodic human or LLM-graded review against a rubric, since automated metrics alone consistently miss tone and correctness failures that matter to users. Track a small set of workload-specific metrics over time — task success rate, escalation rate to a human, average latency, cost per resolved task — rather than a single blended "quality score" that hides which part of the system is actually regressing.

The teams that get GenAI into durable production don't have a smarter model — they have a tighter loop between what the model outputs, what actually happened as a result, and what gets fed back into the next prompt or fine-tune. That loop, more than any single AWS service, is the thing worth building first. We spent decades teaching machines to see and to search; now we're teaching them to act, and the responsibility for what they do lands, as it always has, on the people who built the loop.

References

• Amazon Bedrock — what is Bedrock
• Amazon Bedrock Knowledge Bases — how it works
• Amazon OpenSearch Service — k-NN search
• Aurora PostgreSQL — pgvector extension
• Amazon Bedrock AgentCore documentation
• AWS Step Functions — developer guide
• Amazon Bedrock — prompt caching
• Amazon Bedrock — batch inference
• Amazon CloudWatch — overview