The Brain in Your Pocket: How On-Device AI is Changing Everything

For years, the most powerful artificial intelligence systems operated exclusively in the cloud, tethered to massive data centers. Every query to ChatGPT or Gemini Pro made a round-trip across the internet. But a quiet revolution is underway: AI is migrating from the cloud to the device in your hand.

Welcome to the world of on-device AI—also called edge AI—where companies like Google and Apple are leading the charge by embedding Large Language Models (LLMs) directly into mobile platforms.

This shift is not just technical—it’s transformational. AI that’s local is faster, more private, and always available. Let’s explore how Google’s Gemini Nano and Apple’s Foundation Models are shaping this future.

Why On-Device AI Matters

Before diving into platform-specific implementations, let’s look at why this change is so impactful:

• Speed & Low Latency: No network round-trips and faster responses.
• Enhanced Privacy: Data stays on your device, reducing exposure risks.
• Offline Use: AI features remain functional even without an internet connection.
• Cost Efficiency: Less reliance on expensive cloud compute infrastructure.

Google’s Approach: Gemini Nano on Chrome and Android

Google is embedding Gemini Nano, the smallest model in the Gemini family, into both Chrome and Android to bring lightweight, performant AI to billions of devices.

What is Gemini Nano?

Gemini Nano is a compact, optimized LLM designed for on-device execution. Despite its size, it supports fast inference with minimal memory usage, making it ideal for tasks like summarization and smart replies—without cloud dependency.

Gemini Nano in Chrome: AI on the Web

Google is integrating Gemini Nano into Chrome to bring AI natively to the web. This empowers developers to build AI features directly into websites—without backend models. It can be called by Javascript directly in webpage or chrome extension.

Example: Convert web page to Markdown

This is a simple Chrome extension leveraging Gemini Nano model in chrome to convert web page to Markdown. Install it using this link: Markdown It.

This extension uses Prompt API. Gemini Nano also offers Writer API, Rewriter API, Translator API, Language detectior AI, Summarizer API, which follow the similar usage pattern.

      let session;
      try {
        // 1. Check if the Prompt API is available and the model is ready.
        if (typeof LanguageModel === 'undefined') {
          throw new Error('The Prompt API (LanguageModel) is not available. Please check your Chrome version.');
        }

        const modelAvailability = await LanguageModel.availability();
        if (modelAvailability === 'unavailable') {
           throw new Error(`AI model is unavailable. Status: ${modelAvailability}`);
        }

        // 2. Create a language model session. This may trigger a download if needed.
        session = await LanguageModel.create();

        // 3. Create a detailed prompt for the AI model.
        const prompt = `Convert the following page content to Markdown format. ...`;

        // 4. Stream the response from the model for better performance.
        const stream = session.promptStreaming(prompt);
        let markdownResponse = '';
        for await (const chunk of stream) {
          markdownResponse += chunk;
        }
      }

Gemini Nano in Android Platform

On Android, Gemini Nano can be used in two ways:

• For summarization, proofreading, rewrite, and image description, ML Kit GenAI APIs offers high level api.
• Google AI Edge SDK allows experimental access for more general purpose usages of this on-device AI capabilities.

Apple’s Vision: Foundation Models on iOS

With iOS 18, Apple unveiled Apple Intelligence, integrating its on-device LLMs via the new FoundationModels framework. Designed around privacy, safety, and capability, this framework enables powerful generative features while keeping data entirely local. Apple opened this FoundationModels to developers in iOS 26. The usage is straightforward like below.

let instructions = "Write a poem"
let session = LanguageModelSession(instructions: instructions)"
let response = try await session.respond(to: prompt)

Futhermore, Foundationmodels from apple, support structured output and tool calling. Generable macro and Tool facilate this usage under the hood, which we will cover in the futuer posts.

Conclusion

The migration of AI to the edge is more than just an engineering feat—it redefines our everyday interaction with technology. By empowering devices with intelligence that’s fast, private, and resilient, companies are putting the future of AI right in our pockets.

Google’s democratized, web-integrated approach and Apple’s privacy-driven, system-native experience both point to the same truth: on-device AI is here—and it’s only getting smarter.

Sharing is caring!