Introducing Perceptron Mk1.5

Tracking an object through a video

In order for an embodied agent to complete real-world tasks (e.g., inventory tracking, sports analytics), it is often necessary to track unique entities across time. To enable this, we trained Mk1.5 to emit object tracks as timestamped geometries, rather than one-off, per-frame detections.

The result: Mk1.5 leads three of the four video object segmentation benchmarks we measured.

With native object tracking support, Mk1.5 reduces the need for complex re-identification pipelines that sit downstream of most detection models. Whether you’re tracking inventory in a retail environment or players on a sports feed, you can deploy this capability today.

Understanding egocentric video

First-person video understanding enables applications ranging from annotating robotic training data to helping smart glasses interpret and respond to what a wearer sees.

By training specifically on egocentric video we are able to show significant gains on tasks like hand tracking and subtask segmentation: Mk1.5 localizes hands in a first-person frame 50 percent better than the strongest Gemini model we measured.

Adding audio as a new modality

Mk1.5 is natively built to understand audio. We do not claim to be at the frontier of audio-visual understanding in this first release, but our current public capabilities are already strong enough to be useful:

Audio-visual capabilities enable everything from time-aligned transcription to sound-based event detection.

Learning to use tools

Mk1.5 supports arbitrary tools: any function declared in the OpenAI format with a JSON Schema. Once defined, Mk1.5 decides what to call and when.

Mk1.5 remains competitive with other frontier models on multimodal search evaluated across LiveVQA-W, Browsecomp-VL, MMSearch.

Mk1.5 shows a large improvement with tool calls enabled (+36.1 point over no tools on MMSearch). This performance gain is significant because we do not train our models to memorize world knowledge. Instead we train them to learn efficiently and adapt. By demonstrating effective tool use, we show that Mk1.5 is equipped to learn new control surfaces.

In our demo environment, we support web search, page reads, zoom, and reverse image search, enabling Mk1.5 to leverage knowledge that it did not aquire in training:

Tool use can be the difference between a static control model and a physical agent that generalizes to new tasks and functions across many embodiments.

Parallelizing with sub-agents

Beyond making tool calls, Mk1.5 can deploy sub-agents to parallelize and accelerate task completion:

Faster inference while pushing the frontier

Mk1.5 delivers ~2-5x faster end-to-end request completion, while continuing to push the frontier on performance.

And the improvements in inference speed are tangible. This example is shown at 1x speed:

Reasoning in visual space

Mk1.5 maintains frontier capability on video and embodied reasoning.

Coupling advanced visual reasoning with 5x inference speed up enables Mk1.5 to produce more accurate answers (e.g., on segmentation tasks) much faster: