Developers building AI applications for Apple devices face a persistent challenge: running sophisticated models locally without sacrificing speed or draining device resources. Mirai tackles this with an inference engine built specifically for Apple Silicon, converting and optimizing models from Hugging Face with a single line of code. The system taps directly into the Neural Engine available across Mac, iPhone, and iPad hardware, utilizing unified memory bandwidth that reaches 273 GB/s on M4 chips.
Performance numbers tell the story. Qwen3-0.6B hits 147 tokens per second on M4 Max hardware. The engine squeezes this speed from Apple's 38 TOPS Neural Engine on M4 devices, making it faster than competing on-device runtimes according to the company's benchmarks. Model quantization maintains quality while shrinking size, and cross-device compatibility means the same optimized model runs on phones, tablets, and computers without reconfiguration.
The offline-first architecture matters for specific scenarios. Translation works without internet connection. Classification and intent detection happen on-device. Real-time text generation processes locally. Summarization and extraction don't ping servers. This setup delivers two benefits developers care about: user privacy through local processing and cost reduction by avoiding cloud API calls for every inference request.
Mirai does not force an all-or-nothing approach. The routing system lets applications split work between device and cloud infrastructure, choosing local processing for speed-sensitive tasks and cloud for heavier workloads. This flexibility addresses the reality that some models remain too large for mobile devices while others run comfortably on modern Apple hardware.
The current version (0.1.18) handles text generation, summarization, extraction, classification, intent detection, and translation. Speech capabilities appear on the roadmap. Speech-to-text, text-to-speech, speech-to-speech, and voice commands are marked as coming soon. Benchmarking tools ship with the engine, letting developers measure actual performance on their target hardware before committing to deployment.
The target user works in development shops building native Apple applications where inference latency matters or where privacy requirements prohibit sending user data to external servers. Companies reducing cloud API runs by shifting inference to user devices also fit the profile. The Hugging Face integration simplifies the path from open-source models to production deployment on Apple hardware.
Real-time performance depends on model size and device capabilities. Smaller models fly on recent hardware. Larger models still benefit from optimization but will not match cloud GPU speeds. The system assumes Apple Silicon, so Intel Mac support doesn't exist.
