Introduction to On-device AI
Overview
Why On-device
- Cost Effective: Reduces recurring costs by minimizing dependency on cloud computing resources.
- Efficient: Faster processing speed and power efficiency by leveraging local computation power.
- Private: Keeps data on the device, enhancing security and protecting user privacy.
- Personalized: Allows continuous model customization without external data transfer or updates.
Device In-the-loop Deployment
- Capture model
- Compile for the target device
- Validate numerics
- Measure Performance
- Deploy
Applications
- Live translation
- Live transcription
- Photo portrait generation
- Photo Al editing
- Semantic search
- Text summarization
- Virtual assistants
- Writing assistance
- Image generation
Semantic Segmentation Models/ Algorithms
- ResNet (Residual Network): Uses residual connections to enable training of very deep networks by allowing gradients to flow through layers without diminishing.
- HRNet (High-Resolution Network): Maintains high-res representations through the network, enabling it to capture fine details for accurate segmentation.
- FANet (Feature Agglomeration Network): Focuses on agglomerating features from different scales, enhancing the model's ability to discern finer details.
- DDRNet (Dual Dynamic Resolution Network): Employs dual-path architectures to balance efficiency and accuracy, facilitating real-time semantic segmentation.
Fuss Free Network (FFNET)
https://arxiv.org/abs/2206.08236
Fuss-Free Network (FFNet): A simple encoder-decoder architecture with a ResNet-like backbone and small multi-scale head.
Performance: Performs on-par or better than complex semantic segmentation architectures such as HRNet, FANet and DDRNets.
Preparing for on-device deployment
On-device deployment key concepts
TensorFlow Lite
- Optimized for Mobile: Specially designed for mobile and embedded devices, ensuring fast, efficient performance.
- Low Latency: Provides fast response times by reducing the computational overhead of model inference.
- Flexibility in Deployment: Supports a wide range of devices from smartphones to loT gadgets, making Al ubiquitous.
- Energy Efficient: It uses less power than traditional models and is ideal for battery-operated devices.
- Hardware Acceleration Accelerated on Qualcomm NPU with the Qualcomm Al Engine delegation.
