“AI observability is essential to really unlock the full potential of these models.”
At Machinecon 2024, Krishna Gade, Founder and CEO of Fiddler, delivered an insightful presentation on the critical importance of AI observability in deploying large language models (LLMs) safely and responsibly. As companies across various industries rush to adopt LLMs, Gade emphasized the need for robust oversight and monitoring to mitigate risks and ensure optimal performance.
Gade began by highlighting the rapid adoption of LLMs across industries. He cited examples of innovative applications, such as a leading car manufacturer using generative AI to simulate experiments, saving physical resources and improving efficiency. He also mentioned major banks implementing generative AI for fraud detection and anti-money laundering efforts, with significant gains reported.
“We are seeing lots of different use cases,” Gade noted. “A lot of innovative ideas are being pursued by industries across different verticals to deploy these large language models.”
The Dark Side of AI Adoption
However, Gade was quick to point out the potential pitfalls of hasty LLM deployment. He referenced several high-profile AI failures that have resulted in brand damage and financial losses for companies. One notable example was the Air Canada chatbot incident, where hallucinated information led to litigation and severe reputational damage.
“We are also seeing the other side of it,” Gade cautioned. “We are seeing AI failures happen that have hurt the brand reputation of these companies, that have made some of these companies incur some financial losses.”
The MOOD Stack
To address these challenges, he introduced the concept of the MOOD stack, an acronym representing the key components of modern AI infrastructure:
- Modeling: The choice of LLMs, including both closed-source and increasingly popular open-source options.
- Orchestration: Systems to combine models, datasets, and business-critical information.
- Observability: Tools to monitor and evaluate LLM applications in real-time.
- Data: The foundation of any AI system, including storage, transformation, and warehousing.
AI Observability: The Cornerstone of Responsible AI
He emphasized that AI observability is crucial for evaluating models, monitoring production performance, and protecting LLM applications. He explained, “AI observability is about being able to measure your model performance in an automated manner – measure accuracy, measure safety, measure the quality of it.”
Key aspects of AI observability, according to Gade, include:
- Evaluation: Pre-production testing of LLM applications.
- Monitoring: Real-time performance tracking and alerting in production environments.
- Protection: Implementing guardrails and prompt defense layers to enhance security.
Trust Models – The Next Frontier in AI Evaluation
Gade introduced the concept of “trust models” – smaller language models designed to evaluate LLMs in real-time. These specialized models can assess various aspects of LLM output, such as hallucination rates, toxicity, and adherence to source context.
“Trust models are basically think about these as last small language models that are essentially evaluating your LLMs,” Gade explained. “They can actually detect toxicity issues in responses, where you can say is this response having inflammatory information or racist content or any brand safety violations.”
The Path Forward – Continuous Monitoring and Customization
Gade concluded by emphasizing the need for continuous monitoring across multiple metrics, including:
- Hallucination metrics: Relevancy, coherency, and faithfulness to source material.
- Safety and security metrics: PII protection, jailbreaking detection, and brand safety.
- Operational metrics: Cost, latency, and other performance indicators.
By implementing robust AI observability practices, companies can create live reports and dashboards to track LLM performance and drive business impact.
“We believe that AI observability is fundamentally very, very important for you to productionize LLMs in a safe and responsible manner,” Gade stated, underlining the critical role of oversight in the future of AI adoption.
As enterprises continue to explore the vast potential of LLMs, Krishna Gade’s insights at Machinecon 2024 serve as a crucial reminder of the importance of responsible AI deployment. By embracing AI observability and implementing comprehensive monitoring solutions, companies can harness the power of LLMs while mitigating risks and ensuring long-term success.
