If an app needs to work in a classroom without dependable internet, a cloud-only model changes the product before you write the first prompt. The feature is available only when the connection cooperates.
On-device AI removes that particular dependency, but it introduces a different one: the user's hardware must run the workload well enough. The decision is a trade between operating constraints, not a contest with one universal winner.
What is on-device AI?
On-device inference runs the model on the user's computer or phone. Cloud inference sends a request to a remotely hosted model. An application can combine the two, but the interface should make important differences in data flow and availability clear.
Local inference does not automatically mean the entire app is offline or that no data leaves the device. Downloads, account services, telemetry and optional features can still use a network connection. Inspect the complete workflow before making a privacy or offline claim.
For each proposed feature, list what happens when the network is unavailable. If the answer is that the main task stops, decide whether that is acceptable for the intended user.
Compare the constraints before comparing models
| Question | On-device consideration | Cloud consideration |
|---|---|---|
| Can the user depend on connectivity? | Generation may continue after setup | Remote calls require a usable connection |
| What hardware is available? | Memory and runtime support limit the model | Server capacity is managed remotely |
| Where does the input go? | It can remain local in a fully local path | Data is sent to a service for processing |
| How are models updated? | Download, compatibility and migration work | Provider changes and version management |
| What does usage cost? | Device resources and distribution | Metered requests and hosting charges |
The table is a starting point, not a performance benchmark. A local model can be slower on one machine and adequate on another. Compare the actual task on the minimum hardware you intend to support.
Include setup in the test. A feature that works offline after downloading a model should say that clearly. First-run behavior without a model is a separate experience from later offline use.
Decide what an offline fallback should preserve
A fallback can mean more than running a smaller model. The app might let users read previous results, edit a saved document or queue a task until a connection returns.
For a writing tool, saved drafts may be enough. For classroom teaching, access to the prepared lesson can be essential even when generation is unavailable. Ask what the user must accomplish at the moment the network fails.
A hybrid path can be useful, but avoid quietly sending material to a cloud service after advertising local processing. Make that an explicit option with a clearly described data path.
Why ChalkLens uses a local desktop workflow
I built ChalkLens around textbook PDFs, scans and a teacher's laptop. Its documented workflow extracts source material, generates a lesson kit locally and saves the result for use during teaching.
PDF or image -> PDFKit or native OCR -> Flutter lesson workspace
|
v
Local Gemma inference
|
v
Saved lesson and Present modeThe point of this design is to keep the lesson workflow usable without depending on a remote inference request. It does not establish that generated explanations are always correct or that every computer can run the same model comfortably.
The ChalkLens source documents the model setup and macOS demo path. Google's LiteRT-LM project is a primary reference for the runtime's current capabilities. Verify the model, runtime and hardware combination rather than assuming desktop support from a model name alone.
Questions about local and hosted AI
Is on-device AI always cheaper?
Not necessarily. Consider implementation, device requirements, model distribution and support, alongside any avoided per-request charges. Use the expected workload when comparing costs.
Does local inference make an answer private automatically?
It changes where inference happens. You still need to inspect storage, logs, network features and access to the device before describing the full privacy behavior.
Should an app offer both local and cloud models?
Only when the benefit justifies two operating paths and a clear user choice. Supporting both adds testing and maintenance work.
Write down the task that must still work when the connection fails. Then use the ChalkLens case study as one concrete example of designing around that constraint.