Zillow builds in-house AI architecture to curb token costs
Zillow is bypassing single-model APIs to build a proprietary AI context layer, a strategy that Glean says can slash token consumption by half and reveals how enterprises must structure data to generate real returns on artificial intelligence.
Zillow has built a proprietary AI architecture to maintain customer context across months-long real estate transactions, bypassing single model APIs in favor of smaller, task-specific systems. The company now runs thousands of Glean agents in production to handle repetitive internal tasks. Speaking at VB Transform 2026, Zillow SVP of Engineering Toby Roberts said the company chose to own its context layer rather than route users through an external chat interface.
This infrastructure decision carries direct implications for corporate AI spending. Arvind Jain, CEO of Glean, argued that enterprises waste significant capital letting individual departments rebuild connections to the same systems. By routing tasks through a centralized gateway and using precomputed context, companies can avoid burning tokens assembling information from scratch.
"Claude is also very slow because the first part of assembling that context actually takes forever," Jain said. By precomputing that context and routing most tasks to smaller, cheaper models instead of defaulting to frontier models, Jain noted the approach can cut token consumption by as much as half.
Zillow's products touch roughly 80% of U.S. real estate transactions, requiring systems that track users from a phone screen to a loan officer over years. Roberts noted Zillow leveraged 20 years of machine learning history behind products like Zestimate to build an in-house harness using fine-tuned models. "This context layer has to live to be able to support you where you are at any given point in your journey," Roberts said.
For investors evaluating corporate AI claims, Roberts offered a strict caveat on return-on-investment metrics. Zillow credits AI adoption with a 40% increase in shipped code, but Roberts warned that number only holds up because the engineering team established DORA metrics baselines years before the AI rollout. "Models by themselves are not enough to bring automation with AI inside your enterprise," Jain said. "You do have to connect it with your enterprise context."
The executives also noted that even with permissions-aware platforms, regulated data requires hard-coded compliance checks. Relying on automatic architecture inheritance is insufficient for a company managing sensitive financial and personal data.
