On-Device Embeddings and Cloud Language Models in Flowo

What runs on the phone and what runs in the cloud

Flowo splits AI work by the kind of output it needs. Work that ends in a ranking runs on the phone. Work that ends in new sentences runs on a language model behind our backend.

• On the phone: embedding the text of tasks and notes, then ranking it against a query by cosine similarity.
• In the cloud: the assistant chat, task breakdown, time estimates, sorting Braindump notes into groups and the first plan in onboarding.
• In the cloud, on a separate path: dictation, where audio streams from the phone to a speech service.

There is no generative model on the device. An earlier build carried one, and we removed that path during App Store readiness work. The cloud is now the only place where Flowo generates text.

How the embedder is wired into a Flutter app

The embedder is native code on both platforms, reached from Dart through one method channel with four methods: load, embed, embedBatch and dispose. On iOS a Swift plugin runs a Core ML model with compute units set to all, so Core ML decides per layer which processor runs it, including the Neural Engine. On Android a Kotlin plugin runs a TensorFlow Lite interpreter on two threads and adds the NNAPI delegate on Android 9 and later.

Both plugins accept raw text and tokenize it themselves. Each has its own implementation of the Unigram Viterbi algorithm and reads the same tokenizer.json, so the app carries no third-party tokenizer library. The input is always 128 token ids with a matching attention mask. Longer text is cut to 126 pieces plus a start marker and an end marker.

The two exports differ in one step. The Core ML graph has mean pooling built in and returns one vector. The Android model returns a vector per token, and the Kotlin plugin averages the vectors of real tokens and skips the padding.

Inference stays off the main thread: a serial queue on iOS and a single worker thread on Android. Neither plugin logs the input text or the resulting vector.

Why the model ships inside the app

The model is part of the app bundle, and Flowo never downloads model files at run time. Two things led to that decision. App Store Review Guideline 2.5.2 asks apps to be self-contained. Our earlier design also fetched a 568 MB model after install, and on iPhone that transfer stalled when iOS suspended it in the background.

Bundling costs download size. The Core ML package keeps 32-bit weights and is over 200 MB. The Android file uses int8 weights and is about half that. Flutter builds one asset set for both platforms, so each build removes the other platform's model: the Android build filters out the Core ML package, and an Xcode build phase deletes the Android file. A verifier script then opens the release archive and fails if the wrong model is inside or the right one is missing.

A SHA-256 of each model is recorded when the model is converted. When a release build carries that hash, the native loader hashes the bundled model and refuses to load it on a mismatch.

How a converted model is accepted

A converted model is accepted only after the conversion script has executed it. The script pins the source to one commit of paraphrase-multilingual-MiniLM-L12-v2 and loads weights only from safetensors. Its Python dependencies install from a hash-locked requirements file.

For Android, the script runs the exported file in the LiteRT interpreter and compares the pooled output with PyTorch on a test sentence written in English and Ukrainian. The cosine distance must be 0.002 or less. A candidate that fails is deleted.

Quantization is explicit. Dense weights use dynamic int8 and activations stay in floating point. The three embedding tables get symmetric per-channel int8, and the export stops if the architecture ever has a different number of tables.

What a cloud request contains

A cloud request carries the text the model needs for one job, selected on the phone. An assistant message goes out with the last ten turns of the conversation. Tool results from earlier turns are replaced by a short stub, so a week plan the model read three messages ago is not sent again.

Each assistant request also carries a context object. Among its fields are the current time, today's schedule blocks, up to 15 top-level tasks with their estimates, and Braindump group names with item counts. The body of a note is not in that context.

Notes reach the model through search. When the model asks for notes on a topic, the phone embeds the query and compares it with the stored paragraph vectors in a background isolate. It returns the five best matches by default, each cut to 400 words. Task lookup tries a name match first and falls back to the same embedding search, which returns five tasks.

Other features send less. A breakdown request holds the task title, its estimate, its notes, the number of steps wanted and the app language. Dictation audio goes from the phone to Amazon Transcribe over a presigned WebSocket, so our backend never receives the audio stream. The app ends a dictation session at two minutes and caps the transcript it forwards at 2,500 characters.

Latency and failure behaviour

The two sides fail differently, and the code treats them differently. Embedding needs no network. On iOS the first load compiles the Core ML package once and caches the compiled model under a name that includes the model hash, so an app update with a new model rebuilds it.

Vectors are computed once and stored. A task keeps its vector together with a version marker and the title it was computed from, and the vector is recomputed only when the title changes. A note is split into paragraphs when it is saved, and each paragraph gets a vector in the background. A search then costs one embedding for the query plus one dot product per stored vector.

Cloud calls use a 30 second connect timeout and a 60 second receive timeout. The client does not retry a failed AI request on its own, because a timeout can arrive after the server has already done and counted the work. Sorting notes falls back to rule-based grouping when the cloud call fails, and a dictation session that cannot connect tells the person to type.