Snapseed’s AI Object Brush Transforms Mobile Photo Editing

Mobile photo editing often falters when it comes to selecting objects with precision. Users want to enhance subjects or tweak backgrounds, but traditional tools make the process slow and imprecise especially on small touchscreens. Google’s latest breakthrough, Snapseed: the Object Brush, uses advanced AI to make selecting and editing objects as easy as a swipe.

Intuitive Object Selection via AI

The heart of this innovation is the Interactive Segmenter model. Running entirely on your device, it offers privacy and near-instant responsiveness. With a tap or quick stroke, the model instantly recognizes and selects whole objects like pets, people, or skies in less than 20 milliseconds. 

Its precision means selections hug object edges, allowing for detailed edits. Users can add or subtract areas on the fly, thanks to integrations with MediaPipe and LiteRT’s GPU acceleration.

• Quick, accurate selection through intuitive gestures
• On-device processing ensures privacy and speed
• Refinement tools help perfect every selection

Training Smarter, Not Just Bigger

Delivering high-quality segmentation on mobile required a smart approach. Google began with a powerful "teacher" model trained on 30,000 expertly labeled images covering over 350 object types. While accurate, this model was too large for phones. To solve this, engineers used knowledge distillation, transferring the teacher’s knowledge into a smaller, faster "edge" model optimized for mobile devices.

Additionally, a vast dataset of over 2 million weakly-labeled images simulated countless real-world editing scenarios. This helped the AI learn to recognize a wide variety of objects and respond to diverse user prompts, all while remaining lightweight and fast.

Simulating User Prompts

The model learned from simulated gestures: random taps and scribbles both inside and outside objects. This teaching method enabled it to distinguish what users want to include or exclude, and supports advanced selection features like lasso and box prompts.

Speed and Quality, Without Compromise

A key architectural advance was splitting the process into two sub-models. A robust image encoder runs once to extract deep semantic features. Then, a lean encoder-decoder rapidly processes each user gesture, generating a precise mask in real time. This separation brings the best of both worlds: depth of understanding and interactive speed.

• Deep semantic features from a powerful image encoder
• Lightning-fast editing thanks to a streamlined encoder-decoder
• Efficient 8-bit quantized models for smooth mobile performance

Sharp Selections at High Resolution

To keep edits crisp, the model predicts selection masks at 768x768 pixels, then upsamples to match photo resolution even up to 4K. An edge-preserving joint-bilateral upsampling method ensures selections remain sharp, only applying the final high-res mask after users finish their gestures for a seamless experience.

Expanding Impact Across Google

The Interactive Segmenter already enhances editing in Snapseed and powers the Gallery app on Chromebook Plus 14. Google aims to bring this technology to even more creative tools, setting the stage for a new era of intuitive, AI-driven image editing across its ecosystem.

Conclusion

Snapseed’s Object Brush marries advanced AI with user-friendly design, making professional-grade photo editing possible right on your phone. Fast, accurate, and private, this on-device segmentation tool empowers anyone to realize their creative vision with just a tap or swipe with no desktop required.

Source: research.google/blog/introducing-interactive-on-device-segmentation-in-snapseed/