Shipping AI features without wrecking performance

An AI feature runs on two clocks at once, and mixing them up is how a good demo turns into a shipped game that stutters. One clock is the frame: 16.6 milliseconds at 60fps, 33 at 30. The other is the request: however long a model takes to answer, past a toy that means hundreds of milliseconds to several seconds. None of what follows is tied to one engine; the math, the tradeoffs, and the threading discipline hold whether the call fires from Unity, Unreal, Godot, or a browser tab.

What budget should an AI feature respect?

Frame time is the budget your render loop and gameplay simulation live inside; miss it and the game visibly stutters. Request latency is the budget a network or inference call lives inside, with no relationship to your frame rate. A cloud model answering in 800 milliseconds has just spent roughly 48 frames at 60fps, and no version of that call fits inside one.

The rule that follows is easy to skip under deadline pressure: nothing that can take longer than a frame runs on the thread that produces frames. NPC dialogue, procedural narrative, AI-assisted asset generation, anything hitting a network, all of it runs async and reports back through a queue the main loop checks instead of a value it waits on. What can stay synchronous is narrow: small on-device classifiers and lookups against a pre-built table, closer to a hash lookup than generation.

What does quantization actually trade away?

A model’s weights start out as 16-bit or 32-bit floats, and quantization rounds them down to fewer bits to shrink the file and its runtime memory. Int8, written q8, uses 8 bits per weight; int4, q4, uses 4. Going from 16-bit to q8 roughly halves the footprint; going to q4 roughly quarters it.

The cost is precision, and it shows up as behavior rather than a spec-sheet number. Q8 reads close to indistinguishable from full precision. Q4 is a bigger cut: usable for short jobs like a barked line, but more prone to repetition and reasoning that falls apart past a couple of steps. Below q4, quality drops off fast enough that it rarely earns its place outside research.

Treat those numbers as a floor, not the full footprint. The KV-cache grows with every turn, and the runtime carries its own overhead, so budget headroom past the weight size alone.

Why doesn’t a 7B model fit a console today?

Current-generation consoles share one memory pool, somewhere around 16GB total, across assets, textures, audio, and the OS reserve. A 7B model at q4 alone claims about 4GB for weights before the KV-cache adds more, taken directly from the budget a game’s assets are supposed to live in.

Compute is the constraint that gets missed. A console’s GPU is already earmarked for the frame you’re chasing 60fps on, and LLM-shaped matrix math competes for that same silicon; consoles carry no separate inference headroom the way a discrete-GPU PC does. That’s why on-device sizes for console and most mobile hardware today sit well under 1B parameters, doing one narrow job, intent classification, a line variant, a templated reaction, rather than open-ended conversation.

PC is the exception, which is why almost every local-LLM demo you’ve seen runs there: a discrete GPU with 12 to 24GB of VRAM holds a 7B to 13B model at q4 at a playable rate. Flagship phones with a dedicated NPU are creeping toward small 1-3B models, but that’s the frontier, not the hardware most players own.

How do you keep an AI call from blocking the main loop?

Never issue a blocking call, network or local inference, on the thread that ticks game logic and submits the frame. A single missed frame is a visible hitch; an inference call is hundreds of frames’ worth of time on that thread. Fire the request from a worker thread, and give the calling code a queue to poll instead of a value to wait on.

Design the waiting state as part of the character. An NPC mid-request should keep idling and animating like every other NPC, while physics, other characters, and the player keep ticking untouched. Engines already carry this discipline for asset streaming and disk I/O; AI inference is one more long-latency operation that doesn’t get to stall the frame.

How do you budget cloud inference cost?

Cloud inference bills per token, both what you send and what it returns, and a conversation’s real cost is the sum across every turn. Most chat APIs are stateless, so each turn resends the system prompt, any injected lore, and the growing history, so cost climbs with length even though the player only typed one more line.

Say a system prompt plus lore snippet runs 350 tokens, history averages 300 by mid-chat, and a capped reply runs 60: roughly 650 input and 60 output tokens per turn. At an illustrative $0.15 per million input tokens and $0.60 per million output tokens, one turn costs about $0.00013, a ten-turn conversation about $0.0013, and ten thousand conversations a day lands around $13, scaling linearly with player count and length.

That scaling is the real argument for the caps and caching a production NPC system needs: a hard token limit on replies, and a cache for the handful of questions every player asks first. Adding LLM-driven dialogue to NPCs covers both. Budget it like a texture memory pool: spend deliberately, and know the cost of every extra thing you let into the conversation.

How should a feature degrade when the network drops?

A network call fails eventually, on someone’s connection, during someone’s session, no matter how good your provider’s uptime looks on a dashboard. Design the failure path before the happy path: when a call times out or errors, the game shows a static, in-character line or a pre-scripted branch, never a blank bubble, a spinner, or a raw error string.

Offline is the same problem at the extreme. With no connection, a cloud-dependent feature should fail into a scripted or on-device fallback immediately, not after a doomed timeout. A guard who looks away like he didn’t hear you reads as a design choice; a stalled loading icon reads as a bug.

Going deeper

Batching groups multiple simultaneous requests into a single forward pass on the same hardware, raising throughput per GPU-second at the cost of a small queueing delay per request. It matters once concurrent NPC conversations run into the dozens or hundreds; below that, waiting for a batch to fill costs more than the gain is worth.

KV-cache reuse skips recomputing attention over a prompt’s unchanging prefix, the system prompt and any injected lore, on every turn. Most cloud APIs handle this server-side, sometimes at a discounted rate for the cached portion; check the provider’s docs for how they price it. Local frameworks expose it directly: llama.cpp, MIT licensed and free for commercial use, keeps a persistent context alive across turns instead of reprocessing history from scratch, cutting latency and compute at the cost of keeping that context resident on a device that may already be tight.

Distillation trains a small, task-specific model on a larger model’s outputs instead of running the larger model at inference time for a job that doesn’t need its full range. A game that only needs to classify what the player is asking, or pick one of six flavor lines, can distill that to a model in the low hundreds of millions of parameters that matches the larger model’s quality on that one task, and fits a console or mobile budget where the larger model never would.

Profiling checklists differ by platform class, and skipping that pass is how a feature that ran fine on a dev machine ships broken on the hardware players own.

Whichever budget you’re fighting, memory, latency, or cost, it’s worth a post in the AI Game Dev Org Discord, where people have hit most of these ceilings already. Several of the shipped features on the showcase were built inside exactly these constraints.