Databricks Slashes Costs for Domain-Specific AI Agent Evaluation

As generative AI agents become more sophisticated, maintaining high-quality evaluation is critical but costs can spiral quickly with traditional approaches. Databricks is changing the game by introducing a token-based pricing model for MLflow GenAI evaluation, reducing expenses by up to 95% while ensuring that evaluation processes remain accurate and reliable for production environments.

Token-Based Pricing: A Transparent Revolution

Historically, the cost of evaluating AI agents at scale was daunting, particularly for production deployments that require multiple judges and process high volumes of data. Databricks’ new model bills users based on actual token consumption, $0.15 per million input tokens and $0.60 per million output tokens, instead of a fixed price per judge request. 

This approach provides predictable, usage-based billing and has slashed costs dramatically for real-world teams. Teams gain transparency into cost calculation and only pay for what they use

• Old model: $0.0175 per judge request, adding up to $875/day for 10,000 traces with five judges
  
  
• New model: Around $45/day for the same workload, thanks to token-centric billing

Production-Tested Prompts, Now Open Source

Building effective, domain-specific evaluation prompts is often a repetitive and resource-intensive process. Databricks addresses this by open-sourcing a library of industry-tested prompts tailored for sectors like finance, healthcare, technical documentation, and AI safety. 

These prompts, validated against benchmarks such as FinanceBench and HotPotQA, help teams kickstart robust evaluation pipelines without starting from scratch.

You can explore our production-grade prompts here. 

• Prompts are optimized for both accuracy and token efficiency
• Industry benchmarks include finance, multi-hop reasoning, technical docs, and LLM safety
• Available for free use and adaptation via the MLflow GitHub repository

Bring Your Own Judge: Ultimate Flexibility

Some organizations need more control over their evaluation processes, whether for compliance, privacy, or specialized requirements. MLflow now enables users to bring their own large language models (LLMs) (including OpenAI, Anthropic, or custom models) at no extra evaluation cost. This flexibility empowers teams to:

• Meet strict regulatory or privacy standards
• Leverage existing contracts with LLM providers
• Deploy proprietary, fine-tuned models for unique domains
• Maintain full autonomy over evaluation workflows

Scalable, Secure, and Enterprise-Ready

Cost effectiveness is only valuable if it scales securely for enterprise needs. MLflow GenAI evaluation on Databricks integrates with Unity Catalog to manage governance and compliance, uses Delta Lake to store traces for analytics and dashboards, and enables direct monitoring within MLflow. Serverless compute ensures organizations only pay for what they use, eliminating infrastructure overhead and idle resource costs.

• Unity Catalog supports compliance and governance
• Delta Lake enables advanced analytics and data integration
• Serverless compute delivers flexible, pay-as-you-go scalability

Getting Started Is Simple

Databricks’ token-based pricing and open-source prompt libraries are now live for all customers. Existing users are automatically upgraded, while newcomers can leverage quickstart guides and training resources. Open-source users just need to upgrade to MLflow 3.4.0 or later to access the full suite of evaluation prompts.

• Current users: No action required, new pricing is automatic
• New users: Access quickstart guides or agent-building courses
• Open-source: Upgrade MLflow to 3.4.0+ for prompt access

Democratizing GenAI Evaluation

With transparent token-based pricing and open-source, production-proven prompts, Databricks is making it easier and more affordable than ever to build and monitor high-quality, domain-specific AI agents. No matter your industry, you now have the tools to scale robust evaluation without the heavy price tag.

Source: Databricks Blog