Bonsai 27B: First 27‑Billion‑Parameter LLM Running Fully On‑Device at 3.9 GB

The AI community woke up to a surprise on July 14: PrismML released Bonsai 27B, a 27‑billion‑parameter model that can run on a consumer smartphone. By compressing the model to a 3.9 GB 1‑bit variant and a 5.9 GB ternary variant, the company shattered the long‑standing belief that such large models belong only in data‑center clouds. The release includes full multimodal support, 262K‑token context, and speculative decoding for speed. For developers, this means powerful reasoning, tool‑calling, and vision capabilities can now be embedded directly into mobile apps, opening new privacy‑first and offline use cases.

What happened

PrismML announced two low‑bit variants of Bonsai 27B, both built on the Qwen‑3.6 27B backbone. The ternary version uses {-1,0,+1} weights with FP16 group‑wise scaling, yielding a 5.9 GB footprint that fits on a laptop while preserving 95% of full‑precision performance. The 1‑bit version uses binary {-1,+1} weights, achieving a 3.9 GB size that fits within the memory budget of an iPhone 17 Pro, delivering roughly 90% of the baseline accuracy.

Across a 15‑benchmark suite covering knowledge, reasoning, math, coding, instruction following, tool calling, and vision, the ternary model retained 95% and the 1‑bit model retained 90% of full‑precision scores. Both variants run end‑to‑end in low‑bit mode, including embeddings, attention, MLPs, and the LM head, and ship with a compact 4‑bit vision tower. The models are released under the Apache 2.0 license, enabling immediate experimentation.

Why it matters

Running a 27B‑class model on a phone collapses the latency and privacy gap between cloud inference and edge execution. Developers can now ship AI‑driven features—such as on‑device code assistance, multi‑step planning, and image‑aware assistants—without sending data to remote servers. The reduced memory footprint also lowers power consumption, extending battery life for continuous AI workloads. Moreover, the open‑source license encourages community‑driven optimization and integration into diverse platforms, accelerating the shift toward decentralized AI.

How to think about it

Start by profiling your target device’s memory and compute budget; the 1‑bit variant fits under 4 GB, while the ternary version needs about 6 GB. Use PrismML’s provided GGUF files and the spec‑decode API to maximize token throughput. For latency‑sensitive tasks, benchmark both variants on the actual hardware and decide if the 5‑% accuracy trade‑off of the 1‑bit model is acceptable. Integrate the vision tower only when image input is essential, as it adds a modest 4‑bit overhead. Finally, consider hybrid deployment: run the 1‑bit model on‑device for privacy‑critical steps and fall back to a cloud service for heavy‑weight inference when needed.

FAQ