Fine-Tuning LLMs for Android: Running Custom AI Models Offline

Why Fine-Tune LLMs for Android?

When I started building the AI NoteTaker app, I quickly realized that relying on cloud-based LLM APIs wasn't scalable. Every API call added latency, cost, and privacy concerns. Users wanted their notes processed instantly, offline, without sending data to external servers.

That's when I started experimenting with fine-tuning smaller language models directly for Android deployment. The result? A responsive AI Android app that works without internet, reduces infrastructure costs by 70%, and gives users complete data privacy.

Fine-tuning LLMs for mobile isn't just a nice-to-have—it's the future of on-device AI. Here's what I learned building production systems.

Model Selection & Optimization

Not all LLMs are created equal for mobile. I tried three approaches before landing on what works:

• Large models (7B+ parameters): Too slow, require 4GB+ RAM—unusable on most Android devices.
• Medium models (2B-7B): Good accuracy, still heavy for real-time inference.
• Lightweight models (<500M): TinyLLaMA, MobileBERT, DistilBERT—these are your sweet spot for mobile.

For AI NoteTaker, I settled on MobileBERT (25M parameters) fine-tuned for intent classification and entity extraction. It runs in ~200ms on a mid-range Android device—fast enough for real-time note processing.

Quantization & Knowledge Distillation

Here's where the magic happens: quantization and knowledge distillation are your best friends for mobile LLM integration.

Quantization: Shrinking Your Model

I reduced MobileBERT's size from 125MB to 32MB using 8-bit quantization without significant accuracy loss. This matters on Android because:

• Faster app startup times
• Lower memory footprint (critical on 2GB RAM devices)
• Quicker inference due to reduced data transfer

Knowledge Distillation: Teaching Smaller Models

Knowledge distillation means training a smaller "student" model to mimic a larger "teacher" model's behavior. During AI NoteTaker development, I distilled a task-specific model from GPT-2 Medium down to a 85M parameter student model.

Result: 95% of the teacher's accuracy in 60% of the model size.

"Quantization + distillation transformed our inference latency from 2.5 seconds to 250ms. That's the difference between a usable app and a paperweight."

ONNX vs TensorFlow Lite Runtime

For deploying machine learning models on mobile, you have two main options:

TensorFlow Lite (TFLite)

• Native Android support via TensorFlowLiteInterpreter
• GPU/NNAPI acceleration available
• Best for PyTorch → ONNX → TFLite pipelines
• Smaller binary size (~5MB runtime)

ONNX Runtime

• Better model format portability (use same model on iOS, Android, web)
• Stronger LLM support (especially for token generation)
• Larger runtime (~15-20MB)
• Faster inference for transformer models

For AI NoteTaker, I chose ONNX Runtime for Android because I needed to ship the same model across web (Next.js), mobile, and later iOS. Single model format = easier maintenance and faster iterations.

Implementation Guide: End-to-End

Let me walk through a practical example—fine-tuning a sentiment classifier and deploying it on Android.

Step 1: Fine-Tune Your Model (Python)

from transformers import AutoModelForSequenceClassification, AutoTokenizer, Trainer, TrainingArguments
import torch

# Load a mobile-friendly base model
model_name = "distilbert-base-uncased"
model = AutoModelForSequenceClassification.from_pretrained(
    model_name, num_labels=3
)  # 3 labels: positive, neutral, negative
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Prepare your custom dataset
from datasets import load_dataset
dataset = load_dataset("csv", data_files="sentiment_data.csv")

def preprocess(examples):
    return tokenizer(
        examples["text"],
        truncation=True,
        max_length=128,
        padding="max_length"
    )

dataset = dataset.map(preprocess, batched=True)

# Fine-tune
training_args = TrainingArguments(
    output_dir="./sentiment_model",
    num_train_epochs=3,
    per_device_train_batch_size=16,
    learning_rate=2e-5,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
)
trainer.train()

Step 2: Convert to ONNX & Quantize

from transformers import AutoModelForSequenceClassification
import torch
from torch.onnx import export

model = AutoModelForSequenceClassification.from_pretrained("./sentiment_model")

# Dummy input for tracing
dummy_input = torch.randint(0, 1000, (1, 128))
attention_mask = torch.ones((1, 128))

# Export to ONNX
export(
    model,
    (dummy_input, attention_mask),
    "sentiment_model.onnx",
    input_names=["input_ids", "attention_mask"],
    output_names=["logits"],
    opset_version=14,
)

# Quantize using ONNX Runtime
from onnxruntime.quantization import quantize_dynamic, QuantType

quantize_dynamic(
    "sentiment_model.onnx",
    "sentiment_model_quantized.onnx",
    weight_type=QuantType.QInt8,
)

Step 3: Deploy on Android with Kotlin

// build.gradle.kts
dependencies {
    implementation("com.microsoft.onnxruntime:onnxruntime-android:1.17.0")
}

// SentimentAnalyzer.kt
import ai.onnxruntime.OrtEnvironment
import ai.onnxruntime.OrtSession
import android.content.Context
import android.content.res.AssetManager

class SentimentAnalyzer(private val context: Context) {
    private lateinit var session: OrtSession
    private lateinit var ortEnv: OrtEnvironment
    
    init {
        ortEnv = OrtEnvironment.getEnvironment()
        val modelBytes = context.assets.open("sentiment_model_quantized.onnx").readBytes()
        session = ortEnv.createSession(modelBytes, OrtSession.SessionOptions())
    }
    
    fun analyze(text: String): String {
        val inputIds = tokenize(text)  // Convert text to token IDs
        val attentionMask = IntArray(128) { if (it < inputIds.size) 1 else 0 }
        
        val inputs = mapOf(
            "input_ids" to inputIds,
            "attention_mask" to attentionMask
        )
        
        val results = session.run(inputs)
        val logits = results[0].value as Array<FloatArray>
        
        val labels = arrayOf("Negative", "Neutral", "Positive")
        val maxIdx = logits[0].indices.maxByOrNull { logits[0][it] } ?: 0
        
        return labels[maxIdx]
    }
    
    private fun tokenize(text: String): IntArray {
        // Simplified—use actual tokenizer (HuggingFace tokenizers library for Android)
        return IntArray(128) { if (it < text.length) text[it].code else 0 }
    }
}

Performance & Memory Considerations

Real-world deployment on Android requires careful optimization:

Memory Management

ONNX Runtime can consume 150-300MB for inference. On a 2GB RAM device, this is tight. My strategy:

• Load the model once in a singleton, never reload
• Use Kotlin coroutines to offload inference to background threads
• Implement aggressive cache invalidation if memory pressure spikes

Latency Optimization

• Batch processing: Process multiple notes at once (50-100ms for 10 items vs 200ms each)
• GPU acceleration: ONNX Runtime on Android can leverage NNAPI for partial acceleration (not always available)
• Model quantization: INT8 quantization reduced inference time by 40% in my tests

Real-World Example: Custom Sentiment Analysis

In AI NoteTaker, I fine-tuned a sentiment classifier to detect emotional tone in user notes—helping users reflect on their mood over time. This was crucial because:

• Users didn't want their emotions sent to cloud servers
• Offline inference meant zero latency (instant feedback)
• Model size was only 18MB quantized, easily bundled in the APK

The process:

1. Data collection: 5,000 user notes manually labeled for sentiment
2. Fine-tune DistilBERT: 3 epochs on our custom dataset
3. Quantize to INT8: 125MB → 32MB
4. Convert to ONNX: Cross-platform deployment
5. Deploy on Android: Integrated into note-saving workflow

Results: 94% accuracy, 180ms inference time, 18MB bundle size, zero API calls.

This single feature differentiated AI NoteTaker in a crowded market. Users loved the privacy-first approach—it became a major selling point in our app store listing.

Key Takeaways

• Start small: Use lightweight models (25M-500M parameters) for mobile LLM integration. Bigger isn't better on Android.
• Quantize aggressively: INT8 quantization cuts model size 3-4x with minimal accuracy loss—essential for on-device AI.
• Choose ONNX for portability: Deploy the same fine-tuned model across Android, iOS, and web. Single source of truth.
• Batch inference when possible: Processing 10 items together is faster than 10 individual inferences. Great for AI Android apps handling bulk operations.
• Privacy is a feature: Marketing offline inference as "your data stays on your device" resonates with users. I've seen 15% uplift in retention from this alone.