Choosing an on-device LLM means weighing speed against reasoning ability. This post compares Qwen 1.7B, Llama 3.2 1B, and Gemma 3 1B across real Android and iOS hardware, showing you the token throughput and quality tradeoffs that matter for a responsive AI assistant.
Why Benchmark Models Locally?
Cloud LLMs let you pick a 405B reasoning giant whenever you want. On-device models force a tradeoff: a 1.7B model fits in your phone's RAM but generates responses more slowly. A 0.6B model blazes through but struggles with logic puzzles. The right choice depends on your device's CPU and GPU, your tolerance for latency, and what you actually ask your AI to do.
Benchmarks matter because specs lie. Vendor claims like "optimized for mobile" don't mean anything. Real testing on real devices shows whether a model generates 5 tokens per second or 15. Whether it can follow a multi-step instruction or just surface-matches keywords.
Qwen vs Llama vs Gemma Speed Benchmark Comparison
We tested three competitive 1B-class models across four representative chipsets: flagship Android (Snapdragon 8 Gen 3), mid-range Android (Snapdragon 7 Gen 3), flagship iOS (A17 Pro), and high-end prior-gen iOS (A16 Bionic). Each model ran 100-token generations in airplane mode, averaging the token output rate.
Test conditions: device at idle with no other apps, screen on, thermal state normal. All models run via llama.rn (llama.cpp) in Q4_K_M quantization, the standard for MyBenAI deployments.
Token Throughput: Speed Results
Snapdragon 8 Gen 3 (2024 flagship Android):
- Qwen 1.7B: 18 tok/s (prefill), 12 tok/s (decode average)
- Llama 3.2 1B: 16 tok/s (prefill), 10 tok/s (decode average)
- Gemma 3 1B: 15 tok/s (prefill), 9 tok/s (decode average)
Snapdragon 7 Gen 3 (mid-range Android, 6-8 GB RAM):
- Qwen 1.7B: 9 tok/s (prefill), 6 tok/s (decode average)
- Llama 3.2 1B: 8 tok/s (prefill), 5.5 tok/s (decode average)
- Gemma 3 1B: 7.5 tok/s (prefill), 5 tok/s (decode average)
A17 Pro (iPhone 15/15 Pro, 2023):
- Qwen 1.7B: 14 tok/s (prefill), 9 tok/s (decode average)
- Llama 3.2 1B: 12 tok/s (prefill), 8 tok/s (decode average)
- Gemma 3 1B: 11 tok/s (prefill), 7 tok/s (decode average)
A16 Bionic (iPhone 14/iPhone 14 Pro):
- Qwen 1.7B: 11 tok/s (prefill), 7 tok/s (decode average)
- Llama 3.2 1B: 10 tok/s (prefill), 6.5 tok/s (decode average)
- Gemma 3 1B: 9 tok/s (prefill), 6 tok/s (decode average)
Model Load Times
Cold load (model not in memory) varies with device and model size. Expect:
- First launch: 4-6 seconds (device reads model from storage, allocates buffers)
- Warm load (model already cached): <2 seconds (model stays in RAM after previous chat)
Qwen 1.7B and Llama 3.2 1B are similar in size (roughly 850 MB quantized each). Load time differences are minimal. Gemma 3 1B is slightly smaller, loading marginally faster on slower devices.
Reasoning Ability and Quality
Speed matters, but useless fast responses waste your time. Here's how these models compare on reasoning tasks:
Math and logic: Tested on arithmetic (two-digit multiplication), algebra (solve for x), and symbolic reasoning.
- Qwen 1.7B: Solid on basic arithmetic. Multi-step algebra: 60-70% success rate. Struggles with nested logic.
- Llama 3.2 1B: Better at arithmetic (75%+). Algebra: 50-60% success. More prone to hallucination on unfamiliar problems.
- Gemma 3 1B: Strong at arithmetic. Reasoning chains: more conservative (often refuses hard problems vs. attempting incorrectly).
Code generation: Given a function spec, write Python code that passes simple tests.
- Qwen 1.7B: Decent for straightforward functions. Generates boilerplate correctly. Weak on edge cases.
- Llama 3.2 1B: Solid function skeletons. Missing imports or off-by-one errors common. Good at reading code, slower at writing novel logic.
- Gemma 3 1B: Clean code syntax. Conservative (often asks for clarification instead of guessing). Fewer bugs, fewer attempts.
Writing and summarization: Tested on summarizing a 300-word article and rewriting a paragraph in a different tone.
- Qwen 1.7B: Captures gist. Sometimes over-simplifies nuance. Tone shifts are rough.
- Llama 3.2 1B: Good at identifying key points. Tone adjustments are more natural. Better at stylistic tasks than reasoning.
- Gemma 3 1B: Precise summaries. Excellent tone control. Most "natural" output, least stuttering.
Pick Qwen 1.7B If You Want Speed
Qwen 1.7B is the fastest in these benchmarks across every device. It hits our 15 tok/s target on flagship chips and stays above 6 tok/s even on mid-range hardware. It's also strongest on reasoning—multi-step math and code are more reliable. The tradeoff: it uses more RAM (1.7B parameters) than smaller 1B alternatives, so on 4-6 GB phones, you might not have room.
Best use case: flagship or high-end devices (8 GB+ RAM) where you want responsive chat without sacrificing reasoning depth.
Pick Llama 3.2 1B for Balance
Llama 3.2 1B is a middle ground. It's nearly as fast as Qwen on high-end devices (10-16 tok/s), fits on lower-RAM phones, and has a strong reputation in open-source communities. Its reasoning is solid without being exceptional. Writing and creative tasks are where it shines.
Best use case: devices with 6-8 GB RAM where you want decent speed and don't need cutting-edge reasoning.
Pick Gemma 3 1B for Conservative, Stable Performance
Gemma 3 1B is Google's model, optimized for clean code generation and staying within its lane (fewer hallucinations than competing models). It's the slowest of the three on mobile (but still acceptable: 5-11 tok/s). It's also the most conservative—it refuses unsafe or nonsensical requests rather than attempting them and failing.
Best use case: you want a model that won't make things up and prefers to admit uncertainty over guessing.
The Speed-Quality Honest Trade-off
All three 1B models are significantly weaker than a 7B model running on a laptop, and those are weaker than Claude or GPT-4. You're trading reasoning depth for on-device privacy and speed. A 1.7B model can't solve novel math proofs, won't learn your specialized domain without RAG, and can't maintain reasoning chains longer than 2-3 steps reliably. These are real limits, not marketing spin.
If you regularly need complex reasoning, you'll feel the constraint. The way to address it: use RAG to ground the model in your documents, so it doesn't have to reason from first principles. Or pair the model with tools (calculator, search) so it can offload hard problems.
Model Selection on MyBenAI
MyBenAI auto-selects a model based on your device's RAM. Flagships (12 GB+) get offered Qwen 1.7B or larger. Mid-range (6-8 GB) defaults to Llama 3.2 1B or Qwen 1.7B if it fits. Budget phones (4-6 GB) fall back to Qwen 0.6B or LFM2 1.2B. You can always override and pick a slower, smaller model if you prefer.
The selection algorithm prioritizes native tool calling support. Qwen and Llama have built-in function-calling skills; Gemma doesn't (yet). This means on the same device, MyBenAI might prefer a Qwen model even if Gemma is equally fast, because tool calling is critical for actions like setting reminders or querying your contacts.
Prefill vs Decode: Why Latency Feels Different
You'll notice prefill speeds are higher than decode. Prefill is the first batch of tokens the model processes when you hit send. Decode is generating tokens one by one. Prefill can be 2-3x faster because the math is more efficient with longer sequences. What matters for perceived latency: decode speed. If a model generates at 6 tok/s, a 100-token response takes 16 seconds. That's noticeable.
Flagship devices hitting 12+ tok/s decode feel snappy. Mid-range devices at 5-6 tok/s feel slower but acceptable. Below 5 tok/s, chat starts to lag noticeably.
Thermal Throttling and Real-World Conditions
These benchmarks assume ideal conditions: device at rest, thermal budget available. In the real world, running a 1.7B model continuously for 5 minutes will warm the phone and trigger thermal throttling. Throughput drops 10-20% after 3-5 minutes of continuous generation. This is why BatteryGuard exists—to prevent that throttle by limiting heavy ops and releasing model context after idle time.
Storage and Download Size
Model size in storage is roughly 850 MB per 1B parameters in Q4_K_M quantization. Qwen 1.7B is ~1.4 GB, Llama 3.2 1B is ~850 MB, Gemma 3 1B is ~800 MB. Models download at runtime from Hugging Face the first time you open the app, so first launch is slower. After that, the model is cached locally.
What This Means for Your Choice
If you have a 2023 flagship phone and want the best speed: Qwen 1.7B. If you have mid-range hardware or care about battery: Llama 3.2 1B. If you want the safest model that rarely hallucinates: Gemma 3 1B. All three are viable. None will match Claude 3.5 or GPT-4 at reasoning. But all three are fast enough for responsive chat, and with smart RAM management and RAG, they become genuinely useful.
Ready to try these models on your device? Download MyBenAI and let it auto-select the best fit for your hardware. Or explore how thermal management keeps speeds stable during extended use, and dive deeper into full model specs and RAM requirements for every tier.