"Train in Python, serve in C++" might as well be Google’s motto when it comes to much of its ML work. Python is easy for data scientists to wield, but keeping its interpreter in the serving pipeline can add overhead and deployment headaches. The industry-backed OpenXLA Project provides tools for compiling machine learning computations into native code for different processors. The latest component of this setup to be open sourced is XSE , short for the XLA Serving Engine. One-shot inference
XSE is not so much suited for large-scale LLM inference (which would be better served by vLLM or SGLang ), but rather for smaller, more sharply task-oriented models, of which there are plenty at Google. "LLM serving gets most of the attention right now, but a significant amount of production inference is still ranking, recommendation, embedding, and vision models. These tend to be short requests with very high [queries-per-second] and tight latency budgets," Nathan Beach , Google director of product management, wrote in a LinkedIn missive Tuesday. At Google, XSE already handles more than six million queries per second across services including YouTube Shorts, Google Ads, and Search. These models aren’t there for a leisurely chat. They rank results, recommend videos, and make other decisions on the fly, often without users noticing…