AI Dev TLV'25

LLMs are evolving from single, monolithic systems into collaborative networks of specialized agents, each with its own role, knowledge, and perspective. This shift from one agent to many opens new frontiers in reasoning, coordination, and reliability. But it also raises difficult questions: How do agents communicate effectively? How do we prevent chaos as autonomy scales? And how do we evaluate intelligence that emerges from interaction rather than isolation?

In this session, I’ll dive into the practical and architectural challenges of building multi-agent systems: from choosing communication protocols and defining shared memory, to orchestrating reasoning loops and managing conflicting goals. Drawing from our experience at monday.com, I’ll share how we’re designing and evaluating cooperative agents for real-world use cases like search, task planning, and contextual copilots, balancing autonomy with control.

We’ll also explore the evolving landscape of multi-agent frameworks and orchestration topologies (hierarchical, peer-to-peer, and protocol-driven), and discuss how concepts like emergent coordination, agent protocols, and evaluation methodologies are reshaping how we build production-grade AI systems.

Join me to explore how multi-agent architectures can transform AI from single-turn assistants into self-organizing, adaptive systems capable of solving real business problems at scale.

Add to Calendarcalendarcalendarcalendarcalendar

Once we were an agent, then an agentic agent, and now a full-blown multi-agent system. When I say 'once,' I mean one year ago. Remember when you could build products that didn't change under your feet every 5 minutes? I do. It was nice.

In this talk, I'll share Qodo's journey through AI's chaos: 'agentifying' our code generation agent, architecting a multi-agent code review system using LangGraph and MCPs, and how we prepare for the inevitable next shift—because in AI, the only constant is change.

Add to Calendarcalendarcalendarcalendarcalendar

What if you could ask your visual data a question, and it showed you exactly what your model missed?

In vehicle inspection, spotting edge cases like a shallow scratch on a tail light can make all the difference. These rare issues often lie in the long tail of the distribution - subtle, underrepresented, and easy to miss. At UVeye, we scan over a million vehicles each month, making it critical to surface those rare examples that impact real-world performance and customer trust.

In this talk, I will share the journey of developing an agentic AI architecture designed to improve deep neural network performance within an algorithm development workflow. I’ll describe the key steps that helped me build a system that performs smart, selective semantic search over massive datasets. It uses an MCP to retrieve metadata via SQL queries and RAG to fetch relevant images and videos, enabling context-aware search across petabyte-scale data. I’ll demonstrate how this approach helped us find missed cases, retrain models, and improve results.

By the end of this talk, you’ll learn how agentic AI can transform passive data into an interactive, searchable source of truth.

Add to Calendarcalendarcalendarcalendarcalendar

As AI-powered code understanding and assistance become mainstream, one architectural debate quietly shapes the developer experience: Should we index the codebase with Retrieval-Augmented Generation (RAG) techniques, or opt for real-time, structural, or non-indexed approaches? This talk dives deep into the pros and cons of codebase indexing for AI-powered tools—drawing from practical experience building our large-scale internal platform (Ask Dev), recent research, and real-world deployments at enterprise scale.

Attendees will leave with a clear understanding of when to reach for RAG, when to avoid it, and the questions every engineering org should ask before embedding their codebase.

Add to Calendarcalendarcalendarcalendarcalendar

Natural Intelligence operates one of the world’s most diversified and scaled PPC programs. This talk presents our GenAI-first approach—spanning planning, architecture, agentic automation, and measurement—to scale while preserving quality across campaign management, creative production, and analytics.

Add to Calendarcalendarcalendarcalendarcalendar

Large language models behave a lot like brains - and sometimes, the smartest way to improve them isn’t through more data or bigger prompts, but through careful “neurosurgery.” In this talk, Oren Yosifon explores practical methods to reshape base models themselves - from pruning and ablation to targeted tuning - to boost performance, reduce cost, and bypass common model limitations. Forget endless prompting: this is about understanding and modifying the model’s inner circuits for speed, precision, and efficiency. If you’ve ever wondered how to make your LLM smaller, smarter, and cheaper, this session is your crash course in practical AI brainwork.

Add to Calendarcalendarcalendarcalendarcalendar

Behind every great AI system lies one thing: high-quality data. In this talk, we’ll walk through the practical process of transforming any website into structured, meaningful context that can power your LLM applications. We’ll explore modern web-scraping workflows - from JavaScript rendering and browser automation to cost-efficient data extraction - and share real-world techniques for handling dynamic sites and anti-bot systems. Expect hands-on insights, not a product pitch.

Add to Calendarcalendarcalendarcalendarcalendar

What happens when you teach a vision-language model to understand beauty edits like your users do? In this talk, we’ll explore how we fine-tuned Qwen2-VL-7B-Instruct to power intelligent, personalized edit suggestions in Facetune. From the challenges of working with proprietary models to the breakthroughs of fine-tuning on real user data, this session reveals how open-source AI can deliver better personalization, lower latency, and greater control—all while scaling to millions of users.

Add to Calendarcalendarcalendarcalendarcalendar

**Most AI apps don’t fail because of bad models—they fail because they stop learning.**

Your first prompt will suck. So will your second. The difference between a flaky demo and a dependable product is what happens next: does the system improve, or does your team keep duct-taping guesswork?

In this talk, we’ll dive into real-world feedback systems that keep agents learning in production. Based on shipping dozens of LLM apps and interviewing 50+ engineering teams, we’ll break down:

- What feedback signals actually matter—and what’s noise
- The anatomy of a high-signal feedback loop: from collection to iteration
- How to turn thumbs, edits, and complaints into structured improvements
- Practical methods to move beyond shallow System-1 prompting into real adaptability

If you’re building agents that interact with real users, this talk will show you how to make them consistently smarter—not just functional.

Add to Calendarcalendarcalendarcalendarcalendar

Slow inference, sky-high GPU bills, users complaining about latency—sound familiar? Most teams mistakenly throw more hardware at LLMs performance problems instead of fine-tuning their vLLM configurations. The truth is, vLLM’s default settings are optimized for impressive demos, not efficient production use. Even a single misconfigured parameter can severely reduce throughput or exhaust your GPU memory.

At AI21, we’ve extensively optimized Jamba’s family models using vLLM. In this session, I’ll share practical insights and hard-won lessons from tuning critical vLLM engine arguments. You’ll gain a deeper understanding of how these configurations directly impact inference efficiency, memory management, GPU utilization, latency, and overall model throughput.

We’ll compare real-world scenarios:

- **Massive Context (256K tokens):** Discover how specific configuration choices dramatically affect large-scale memory allocation and performance.
- **Typical Workload (4K tokens):** Understand how alternate parameter settings can differently optimize performance for smaller contexts.

By exploring memory allocation strategies, batching mechanisms, and the nuances of chunked prefill, attendees will walk away with actionable recommendations ready for immediate implementation. Learn exactly which settings matter most to cut costs and significantly boost performance.

**Who Should Attend:**

ML engineers, software developers, infrastructure specialists, and anyone interested in efficiently deploying large language models in production.

**Prerequisites:**

Optional—Basic familiarity with LLMs and general ML deployment practices. Technical knowledge of inference and GPU optimization is beneficial but not mandatory.

Add to Calendarcalendarcalendarcalendarcalendar

More and more people are using AI agents to get things done - reading content, booking flights, managing tasks - all without ever visiting traditional websites or apps. Instead of full-blown interfaces, what they’re interacting with are bite-sized "applets" built to work inside these agents.

In this talk, we’ll look at what happens when SaaS apps start breaking down into these smaller pieces, how AI agents are becoming the new browsers, understand the technical details, limitations and opportunities, and why this shift is fundamentally changing the way we think of the Web itself.

People are increasingly turning to AI agents for everyday tasks. From reading the news to online shopping, with the help of MCP, agents are poised to become the next generation of browsers. In this emerging landscape, applications must find ways to maintain their identity and UX to remain relevant for their users. MCP-UI offers a solution for apps to deliver rich, interactive UI on top of MCP, unlocking new frontiers in agentic UI.

In this talk, we’ll explore the future of the agentic web:
The agentic web - how AI agents are replacing traditional browsers and reshaping how users interact with content and services
MCP-UI - deep dive into MCP-UI architecture
Real world use cases
Evolving MCP - current limitations and opportunities to extend the spec in the CWG (UI and other fields)
Future vision
Join us to discover how MCP-UI enables the new web, where users can access their favorite apps uniformly through any agent.

Add to Calendarcalendarcalendarcalendarcalendar

More and more people are using AI agents to get things done - reading content, booking flights, managing tasks - all without ever visiting traditional websites or apps. Instead of full-blown interfaces, what they’re interacting with are bite-sized "applets" built to work inside these agents.

In this talk, we’ll look at what happens when SaaS apps start breaking down into these smaller pieces, how AI agents are becoming the new browsers, understand the technical details, limitations and opportunities, and why this shift is fundamentally changing the way we think of the Web itself.

People are increasingly turning to AI agents for everyday tasks. From reading the news to online shopping, with the help of MCP, agents are poised to become the next generation of browsers. In this emerging landscape, applications must find ways to maintain their identity and UX to remain relevant for their users. MCP-UI offers a solution for apps to deliver rich, interactive UI on top of MCP, unlocking new frontiers in agentic UI.

In this talk, we’ll explore the future of the agentic web:
The agentic web - how AI agents are replacing traditional browsers and reshaping how users interact with content and services
MCP-UI - deep dive into MCP-UI architecture
Real world use cases
Evolving MCP - current limitations and opportunities to extend the spec in the CWG (UI and other fields)
Future vision
Join us to discover how MCP-UI enables the new web, where users can access their favorite apps uniformly through any agent.

Add to Calendarcalendarcalendarcalendarcalendar

In 2025, generative AI is no longer an experiment. It is part of production systems, and platform engineering teams are under pressure. How can they let developers move fast with AI while still holding onto the hard-won practices of good platform engineering?

This talk explores that tension through real experiences building internal AI platforms. It begins with the familiar foundations: Kubernetes, containers, and automation. From there, it expands into the new demands of GenAI-Ops, including retrieval-augmented generation (RAG), fine-tuning workflows, and orchestrating autonomous agents. Along the way, we will look at where existing platform patterns still work, where they fall short, and what entirely new responsibilities emerge.

Attendees will leave with a clearer picture of what “AI platform engineering” actually means, why it is not simply data science with a new label, and how to position their platform teams at the center of their company’s AI strategy.

Add to Calendarcalendarcalendarcalendarcalendar

Conversations with LLMs follow the same pragmatic principles that govern human dialogue — common ground, belief systems, speaker authority, and conflict resolution. In human interactions, we constantly resolve contradictions, defer to authority, and restore coherence. Effective LLM interactions require the same disciplined approach.

This talk introduces a pragmatic toolkit for context management, demonstrating how mastering these principles helps explain why an unwanted outcome occurred, how to correct it, and how to prevent it next time. Together, we’ll explore practical techniques for detecting potential conflicts in context, setting clear speaker authority, and defining what is known — and what is not — to the model, resulting in more reliable, coherent, and robust agents.

Add to Calendarcalendarcalendarcalendarcalendar

AI can already write code - but that’s only a fraction of a developer’s job. The harder part is everything else: securing systems, reviewing PRs, analyzing tests, optimizing queries, and debugging production.

In this talk, Shahar Polak shares a real-world case study from ImagenAI on building and deploying AI agents that tackle these exact challenges in production. Learn how agents reduced security review time from hours to minutes, flagged coverage gaps before release, and optimized real database queries.

This session is for engineers who want to move beyond autocomplete and see how AI agents can handle the messy, time-consuming work that makes or breaks production systems.

Add to Calendarcalendarcalendarcalendarcalendar

Developers have gone full-steam from traditional programming to AI code assistants and vibe coding but where does that leave application security? What are the risks of LLMs introducing insecure code and AI applications manipulated through prompt engineering to bypass security controls?

In a series of real-world application hacking demos I’ll demonstrate how developers mistakenly trust LLMs in generative AI code assistants that introduce insecure code and result in vulnerable applications that attackers easily exploit against you. We don’t stop here. We’ll apply adversarial attacks on neural networks in the form of prompt injection payloads to compromise LLMs integrated into AI application workflows and weaponize them for SQL injection and other business logic bypass.

Join me in this session where you’ll learn how security vulnerabilities such as path traversal, prototype pollution, sql injection and other AI security risks are impacting LLMs and how to leverage AI security guardrails to secure GenAI code.

Add to Calendarcalendarcalendarcalendarcalendar

What if your AI could think like a detective before searching? Instead of blindly executing queries with incomplete information, imagine a system that systematically enriches data until it has enough context to hit the exact target.
We enhanced our retrieval system with aggressive data enrichment that identifies and resolves space-narrowing entities like IDs, dates, and identifiers. The system iteratively gathers missing contextual information, then dynamically injects this enriched context back into the decision process, transforming ambiguous queries into precise identifiers.
I’ll show you the implementation patterns and architectural decisions that enable your agent to reliably select the right candidate from thousands of possibilities by leveraging intelligent context gathering.
This session is for developers and researchers facing the soul-crushing reality of delivering accurate results when your data is anything but complete.
*This talk can be adapted to a 20-minute session length.

Add to Calendarcalendarcalendarcalendarcalendar

Unlock the power of AI image generation for production-level features!
In this talk, we’ll cover how to take image generation from personal use to a practical tool for developers in production - from choosing the right model and mastering prompt engineering to ensuring output quality.
We’ll also address production challenges, including performance optimization, implementing fallback strategies, and tailoring model use to fit specific use cases, ensuring consistent delivery of reliable, high-quality, and visually impressive results.

Add to Calendarcalendarcalendarcalendarcalendar

Have you ever shipped a GenAI agent that dazzled in its first conversation, only to discover weeks later that quality decays as new edge-cases pour in? At first, fixing prompts feels easy—write a new instruction, test it, ship it. But as the agent scales, each fix takes longer. Updates require evaluation, testing across dozens of scenarios, and coordination with downstream systems. Multiply that by hundreds of incoming requests, and suddenly your data scientists spend their days doing nothing but prompt surgery.
That was the trap we fell into at Apartment List, a rental marketplace connecting millions of renters with over 6 million verified units. Our conversational agent, Lea, supports this scale by answering neighborhood questions, scheduling tours, and even triggering application workflows. Every reply has operational and financial impact, so accuracy matters. Yet keeping Lea’s prompts fresh was turning into a bottleneck.

Our solution was to reimagine prompt optimization as a semi-automated workflow. We built a feedback engine where taggers, clients, and product managers can flag issues in live conversations, propose fixes, and immediately hand them to our self-improving agents. This model not only checks for contradictions but also runs evaluation for regression checks, merges safe changes on its own, or drafts a diff-style “prompt pull request” for review. Instead of endless manual edits, data scientists step in only where judgment is needed—cutting review time dramatically while keeping quality high.

In this talk, we’ll share how this system works end-to-end: UI patterns for contributors, heuristics for safe merges, approval flows, and rollout metrics. Attendees will leave with a concrete blueprint for accelerating GenAI prompt improvement cycles—reducing the grind on DS teams while sustaining quality at scale.

Add to Calendarcalendarcalendarcalendarcalendar

As multi-agent LLM workflows make their way into production systems, a critical blind spot remains: which agent actually drives the final output? Without this knowledge, debugging, observability, and guardrail placement become guesswork — adding latency, reducing safety, and eroding trust.
In this talk, we introduce CAIR (Counterfactual-based Agent Influence Ranker), recently accepted at EMNLP 2025, and show how it helps developers answer this question at inference time. CAIR runs lightweight counterfactual “what if” scenarios to dynamically rank agent influence. Evaluated on 30 workflows and 230 functionalities in our AAW-Zoo dataset, CAIR consistently outperformed baselines and enabled latency-aware guardrails that reduced inference time by nearly 28% without sacrificing effectiveness.

Attendees will learn:
• How to measure agent influence inside dynamic LLM workflows
• How this improves observability and debugging
• How to apply guardrails selectively for safer, faster systems
• Lessons from applying CAIR to real-world multi-agent setups
This session is for engineers and platform teams building production-ready multi-agent systems who need practical tools to make them interpretable, efficient, and trustworthy.

Add to Calendarcalendarcalendarcalendarcalendar

Building reliable agentic AI systems in production environments presents unique challenges when dealing with massive, noisy datasets. This talk shares hard-won lessons from developing Klaudia, Komodor's AI agent that processes millions of Kubernetes events daily to deliver autonomous troubleshooting with 95%+ accuracy.

The fundamental challenge isn't LLM capability—it's building systems that maintain reliability when 90% of your data is noise. We'll explore why most agentic AI fails in production: hallucinations masquerading as insights, inability to validate reasoning chains, and the brittle nature of RAG systems when dealing with complex, interconnected failure modes.

This session covers practical know-how learned through painful production iterations: how to build validation frameworks that catch LLM errors before they reach users, architectural patterns for constraining problem spaces without losing effectiveness, and techniques for creating evidence-based reasoning that can be audited and improved systematically.

You'll learn specific strategies for LLM validation in high-stakes environments, including confidence scoring systems, multi-agent verification patterns, and iterative investigation loops that prevent runaway reasoning. We'll cover the hard-earned lessons about what works and what spectacularly fails when building trustworthy AI agents that must deliver accurate results rather than plausible-sounding explanations.

Add to Calendarcalendarcalendarcalendarcalendar

Fasten your mental seatbelts—we're rocketing through a rollercoaster ride unraveling the hidden biases of Large Language Models (LLMs). These AI powerhouses, from which we demand superhero-like feats of intellect, often come with their own kryptonite - ingrained biases that could trip us up in the most unexpected ways. From advising women conservatively on financial matters to misdiagnosing medical conditions based on gender, LLMs can perpetuate stereotypes and deepen societal divides.
We’ll delve into heart-racing real-world consequences of these biases and the cutting-edge strategies we’re deploying to tackle them head-on. Explore the landscape of bias types that plague these systems, from stereotypical inferences to discriminatory compliance.
This session packs a punch, aimed at not only highlighting the issues but also spotlighting actionable insights and proven tools to enhance sensitivity and accuracy in our digital interactions. Leading an initiative for bias measurement and mitigation across all generative AI applications at Intuit, I bring firsthand experience and expertise to this critical discussion. Gear up for a session that promises not just to inform but also to ignite a spark for change, empowering you to refine the brains behind the bits.

Add to Calendarcalendarcalendarcalendarcalendar

Everyone is excited about conversational AI. Everyone is implementing their own chatbots, until they have to make a conversation behave in production.
Replacing a rigid, quiz-style signup with a dynamic onboarding chat sounds great, but executing it is far from simple. It requires designing a conversation that adapts dynamically, collects data, run actions, and completes all of this within a reasonable timeframe. The real headache isn’t “adding an LLM”; it’s the engineering of an agent that can make decisions, and acts on them using automatic tool triggering, which includes presenting actual assets during the conversation.
In this talk, I’ll show how we treated onboarding as a conversation-engineering problem and shipped a production agent using our internal Python-based Agents SDK. We'll walk through the core building blocks and key considerations of creating and maintaining a real-world onboarding agent in production. By the end of the talk you will learn how to structure conversation flows that adapt dynamically, and how to engineer your agent to reliably achieve specific conversational goals.

Add to Calendarcalendarcalendarcalendarcalendar

Traditional debugging approaches face significant gaps when confronting the probabilistic nature of generative AI systems.
Remember when you could just pause the code and see exactly what's happening?
This talk explores practical techniques for debugging LLMs, focusing on systematic troubleshooting methodologies and analysis approaches you can create yourself.
We'll discover techniques that expose reasoning failures and explore how enforcing structured outputs creates deterministic interfaces that simplify validation and error detection.
These approaches can transform the often frustrating process of debugging generative AI into a more structured and manageable workflow.
Whether you're new to generative AI development or already building with these tools, you'll walk away with approaches you can use right away to tame the unpredictability of generative AI.

Add to Calendarcalendarcalendarcalendarcalendar

What are agents, and how can they be leveraged to revolutionize GenAI systems? How do we design modular, scalable solutions that align cutting-edge machine learning with practical applications? In this lecture, we will explore the foundational concepts of Agents and demonstrate how Booking.com is building the next-generation GenAI systems to enhance user experiences.
At the heart of a robust GenAI system lies the concept of agents—modular self contained entities capable of executing complex tasks by orchestrating machine learning skills and leveraging specialized tools. These agents are designed to handle diverse use cases, from answering intricate user queries to personalizing itineraries and optimizing Booking.com’s search functionality.
We will share a real in production example A Question-Answering Agent that fuses LLMs, real-time data, and APIs for context-aware answers.
Finally, we’ll show how we evaluate quality in production using LLM-based evaluations, including Judge LLMs for automatic assessment, dialog quality and more.

Add to Calendarcalendarcalendarcalendarcalendar

What are agents, and how can they be leveraged to revolutionize GenAI systems? How do we design modular, scalable solutions that align cutting-edge machine learning with practical applications? In this lecture, we will explore the foundational concepts of Agents and demonstrate how Booking.com is building the next-generation GenAI systems to enhance user experiences.
At the heart of a robust GenAI system lies the concept of agents—modular self contained entities capable of executing complex tasks by orchestrating machine learning skills and leveraging specialized tools. These agents are designed to handle diverse use cases, from answering intricate user queries to personalizing itineraries and optimizing Booking.com’s search functionality.
We will share a real in production example A Question-Answering Agent that fuses LLMs, real-time data, and APIs for context-aware answers.
Finally, we’ll show how we evaluate quality in production using LLM-based evaluations, including Judge LLMs for automatic assessment, dialog quality and more.

Add to Calendarcalendarcalendarcalendarcalendar