A local AI model fits on disk and still fails when the application loads it. Download size is not the same as peak runtime memory, especially when document processing and the interface are active at the same time.
An on-device AI product needs a measured memory budget and a useful path for devices that cannot support the chosen workload.
Budget for the complete inference workload
Model weights are only part of memory use. Runtime allocations, context, intermediate work, input processing, and other application features can contribute. The exact footprint depends on the model, runtime, hardware backend, and task.
Google's LiteRT-LM overview documents supported runtime capabilities. It does not establish that a particular model and workload fit every device running your Flutter app.
Measure several stages on named devices
For a proposed document assistant, inspect these stages:
| Stage | What to record |
|---|---|
| App idle | Baseline application memory |
| Document import | Extraction and image-processing peak |
| Model load | Runtime and weight allocation |
| Generation | Context-dependent peak and responsiveness |
| Repeated requests | Whether resources are released or accumulate |
Record device, operating-system version, model version, runtime, input size, and measurement method. Avoid comparing unexplained memory numbers from different setups.
Flutter's DevTools memory documentation is useful for Dart and application investigation. Use appropriate native tooling as well when allocations occur outside the Dart heap.
Define a preflight and a runtime fallback
A preflight can check supported hardware and available resources where the platform exposes suitable information. It cannot guarantee future free memory: other applications and the operating system can change conditions after the check.
Handle load failure without destroying the user's document or leaving a permanently busy interface. Offer the supported alternative, such as a smaller model, reduced workload, or a clearly optional cloud path.
Do not silently upload private documents because local inference failed. Any cloud fallback needs the product's explicit data-handling design and user choice.
Avoid overlapping expensive phases unnecessarily
If extraction and inference do not need to run concurrently, sequencing them may reduce peak pressure. Release no-longer-needed page images and temporary objects through the runtime's documented lifecycle.
A smaller model may use fewer resources while answering less reliably on the task. Evaluate answer quality and usability as well as memory. Quantisation and backend changes also require testing rather than a blanket assumption of improvement.
My ChalkLens project provides an offline document-processing context. The offline acceptance guide covers another requirement that a successful model load does not prove.
On-device memory questions
Can we use the model file size as the required RAM?
No. Measure the full runtime workload.
Is a one-time free-memory check enough?
No. Handle resource pressure and runtime failures too.
What should a proof of concept deliver?
Provide named-device measurements, quality checks, and recovery behaviour. Use them to scope supported hardware.