Initial commit: HeliBoard + Gemma 4 on-device AI correction
Integrates LiteRT-LM (Google) with Gemma 4 E2B-it for on-device spell/grammar correction in HeliBoard. After sentence-ending punctuation, the model checks the sentence and shows a correction in the suggestion strip. Tapping it replaces the original text — fully offline, no cloud API. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
nova
@@ -0,0 +1,139 @@
|
||||
# AIKeyboard — HeliBoard + Gemma 4 On-Device AI Correction
|
||||
|
||||
HeliBoard fork that adds on-device spell and grammar correction powered by
|
||||
**Gemma 4 E2B-it** running locally via Google's **LiteRT-LM** SDK.
|
||||
|
||||
After you finish typing a sentence (`.`, `!`, `?`), the model silently checks
|
||||
it and — if it finds an error — shows the corrected sentence in the suggestion
|
||||
strip. Tap it once to replace the original text. Everything runs on-device,
|
||||
no network, no cloud API.
|
||||
|
||||
---
|
||||
|
||||
## How it works
|
||||
|
||||
```
|
||||
User types "guten rag."
|
||||
↓
|
||||
InputLogic detects sentence-ending punctuation
|
||||
↓
|
||||
AiTriggerHook (500 ms debounce) extracts last sentence
|
||||
↓
|
||||
AiCorrectionEngine.correctSentence()
|
||||
→ LiteRT-LM Engine (GPU first, CPU fallback)
|
||||
→ Gemma 4 E2B-it .litertlm model
|
||||
↓
|
||||
AiSuggestionManager.postSuggestion("Guten Tag.", "guten rag.")
|
||||
↓
|
||||
Suggestion strip shows "Guten Tag." in italic/accent color
|
||||
↓
|
||||
User taps → original text replaced with correction
|
||||
```
|
||||
|
||||
### Components
|
||||
|
||||
| File | Role |
|
||||
|------|------|
|
||||
| `app/…/ai/AiCorrectionEngine.kt` | LiteRT-LM wrapper; GPU→CPU fallback |
|
||||
| `app/…/ai/AiTriggerHook.kt` | Sentence detection + debounce |
|
||||
| `app/…/ai/AiSuggestionManager.kt` | Posts corrected text to suggestion strip |
|
||||
| `patches/0001-InputLogic-ai-hook.patch` | Hook in `InputLogic.java` after `commitCodePoint` |
|
||||
| `patches/0002-LatinIME-ai-lifecycle.patch` | AI object lifecycle in `LatinIME.java` |
|
||||
| `patches/0003-SuggestionStrip-ai-style.patch` | Italic + accent color for AI suggestions |
|
||||
|
||||
---
|
||||
|
||||
## Requirements
|
||||
|
||||
- Android device with **ARM64** (arm64-v8a), Android 11+ (API 31)
|
||||
- Android Studio **Ladybug** or newer / Gradle 8+
|
||||
- NDK **28.0.13004108**
|
||||
- **Gemma 4 E2B-it** model in `.litertlm` format (~2.6 GB)
|
||||
Download: <https://huggingface.co/litert-community/gemma-4-E2B-it-litert-lm>
|
||||
|
||||
---
|
||||
|
||||
## Setup
|
||||
|
||||
### 1. Clone and prepare HeliBoard sources
|
||||
|
||||
```bash
|
||||
git clone http://172.17.2.68:3001/nova/AIKeyboard.git
|
||||
cd AIKeyboard
|
||||
bash setup_heliboard.sh
|
||||
```
|
||||
|
||||
`setup_heliboard.sh` clones HeliBoard from GitHub and applies the three AI
|
||||
integration patches automatically.
|
||||
|
||||
### 2. Push the model to the device
|
||||
|
||||
```bash
|
||||
adb push gemma-4-E2B-it.litertlm /data/local/tmp/gemma-4-E2B-it.litertlm
|
||||
adb shell chmod 644 /data/local/tmp/gemma-4-E2B-it.litertlm
|
||||
```
|
||||
|
||||
### 3. Build and install
|
||||
|
||||
```bash
|
||||
./gradlew installDebug
|
||||
```
|
||||
|
||||
Enable **AIKeyboard** as your input method in Android Settings → System →
|
||||
Languages & input → On-screen keyboard.
|
||||
|
||||
---
|
||||
|
||||
## Patch details
|
||||
|
||||
The three patches modify HeliBoard source files in `heliboard/`:
|
||||
|
||||
**0001 — InputLogic hook**
|
||||
After every committed code point, checks `AiTriggerHook.isSentenceEnder()`.
|
||||
On a sentence-ender, reads up to 500 chars before the cursor and calls
|
||||
`onSentenceEndDetected()`.
|
||||
|
||||
**0002 — LatinIME lifecycle**
|
||||
Instantiates `AiCorrectionEngine`, `AiSuggestionManager`, and `AiTriggerHook`
|
||||
in `onCreate()`, cleans them up in `onDestroy()`. Overrides
|
||||
`showAiSuggestion()` to bypass the normal suggestions-enabled gate and
|
||||
intercepts `pickSuggestionManually()` to do a clean sentence replacement
|
||||
(finish composing → delete original → commit corrected text).
|
||||
|
||||
**0003 — SuggestionStrip styling**
|
||||
Detects the `KIND_AI_FLAG` bit (`0x10000`) on a `SuggestedWordInfo` and
|
||||
applies italic style + auto-correct accent color to visually distinguish AI
|
||||
suggestions from normal word predictions.
|
||||
|
||||
---
|
||||
|
||||
## Model placement
|
||||
|
||||
The model must be at `/data/local/tmp/gemma-4-E2B-it.litertlm` on the device.
|
||||
This path is configured in `AiCorrectionEngine.MODEL_PATH`.
|
||||
|
||||
The model file is **not** tracked in this repository (2.6 GB, `.litertlm` is
|
||||
in `.gitignore`).
|
||||
|
||||
---
|
||||
|
||||
## Backend selection
|
||||
|
||||
`AiCorrectionEngine` tries **GPU** (WebGPU/Vulkan via LiteRT-LM) first.
|
||||
If GPU initialization fails, it falls back to **CPU** immediately. If a GPU
|
||||
inference error occurs at runtime, it permanently switches to CPU for the
|
||||
session and recreates the engine.
|
||||
|
||||
Tested on:
|
||||
- Google Pixel 7 (Tensor G2) — CPU backend (OpenCL not available)
|
||||
- Devices with Mali-G710 — GPU via WebGPU/Vulkan
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
The AI integration layer (`app/src/main/java/helium314/keyboard/latin/ai/`)
|
||||
is original work added to this project.
|
||||
|
||||
HeliBoard itself is licensed under **GPL-3.0-only**. See
|
||||
`heliboard/LICENSE` after running `setup_heliboard.sh`.
|
||||
Reference in New Issue
Block a user