Members-Only
Recent Talks & Demos are for members only
You must be an AI Tinkerers active member to view these talks and demos.
Subject-Matter Expert Language Liaison (SMELL): A framework for aligning LLM evaluators to human feedback
Learn about SMELL, a framework for aligning LLM evaluations with human judgment. It uses a four-stage pipeline to create scalable, domain-specific evaluation systems for more reliable AI.
Subject-Matter Expert Language Liaison (SMELL) is a framework designed to improve the evaluation of large language models (LLMs) by combining human expertise with LLM capabilities. It bridges the gap between generic LLM judges and domain-specific human feedback. The framework uses a four-stage pipeline: human data annotation, feedback synthesis, rubric generation, and evaluation. SMELL effectively creates feedback-informed evaluation systems that align LLM outputs with nuanced human judgments, making it scalable and adaptable for a variety of specialized tasks. It’s an innovative approach to automating LLM evaluation, achieving strong performance even with limited labeled data.
SMELL efficiently generates human-aligned LLM judges via synthesized expert feedback.
SMELL generates LLM evaluation rubrics from expert feedback via API synthesis.
Compose Email
Loading recent emails...