Ragas CLI 实战用 judge_alignment 模板度量 LLM-as-judge 与人类评估标准的对齐度【免费下载链接】ragasSupercharge Your LLM Application Evaluations 项目地址: https://gitcode.com/gh_mirrors/ra/ragas导读judge_alignment是 Ragas CLI 提供的官方快速开始模板之一用于量化「LLM 充当评审LLM-as-judge」时其评判结论与人类标注结果之间的一致性。本文将带你从零创建该项目、运行评估并逐行拆解模板源码深入DiscreteMetric、discrete_metric装饰器与experiment实验机制等底层实现帮助你掌握一套可复用的「评审质量度量 → 迭代提示词 → 再度量」闭环方法论。什么是 Judge Alignment 评估在 LLM 应用中越来越多团队用 LLM 替代人工来打分、判通过/失败。但 LLM 评审是否可靠它与人类评审标准到底有多一致judge_alignment模板给出的答案是把「LLM 评审的结论」与「人类标注的结论」逐一对比计算对齐率。该模板的评估对象并不是 RAG 系统本身而是评审这一环节场景Scenario数据集里预先存放已生成的回答pre-existing responses由 LLM 评审进行评判人类标签Human Labels每条回答附带人工标注的 pass/fail 真值ground truthLLM 评审LLM Judge评审依据给定的评分标准grading notes对同一条回答给出 pass/fail 结论对齐指标Alignment Metric逐条比较 LLM 结论与人类结论是否一致一致记为 aligned不一致记为 misaligned最终汇总为对齐率。快速开始五分钟跑通一个对齐评估1. 创建项目ragas quickstart judge_alignment cd judge_alignmentragas quickstart是 Ragas CLI 提供的脚手架命令实现在 src/ragas/cli.py执行后会从ragas_examples模板目录克隆一份完整可运行的示例工程。该命令支持-o/--output-dir指定输出目录例如ragas quickstart judge_alignment -o ./my-project。值得说明的是模板解析机制CLI 会依次尝试「本地已安装的ragas_examples包 → 当前 Ragas 仓库开发目录 → 从上游仓库下载压缩包」并将模板拷贝到目标目录见 src/ragas/cli.py。本仓库中该模板的原始源码位于 examples/ragas_examples/judge_alignment包含evals.py与__init__.py数据集与依赖清单会随模板一并生成。2. 安装依赖模板使用uv管理依赖uv sync3. 设置 API Key评估过程需要调用 LLM 作为评审这里以 OpenAI 为例export OPENAI_API_KEYyour-openai-key模板代码通过load_dotenv()自动加载.env文件因此也可以把 key 写入.env后直接运行。4. 运行评估uv run python evals.py默认运行基线版本baseline judge如需运行改进后的 v2 评审追加--v2参数uv run python evals.py --v2运行结束后终端会打印形如✅ Baseline alignment: 18/20 passed (90.0%)的对齐率汇总实验详情则持久化到experiments/与logs/目录。项目结构解读模板生成的项目结构如下judge_alignment/ ├── README.md # Project documentation ├── pyproject.toml # Project configuration ├── evals.py # Evaluation workflow ├── __init__.py # Python package marker └── evals/ ├── datasets/ # Test datasets ├── experiments/ # Evaluation results └── logs/ # Execution logs其中evals.py是整个评估流程的核心既定义了两种评审提示词对应的指标也定义了数据加载、对齐比较和实验编排逻辑。深入代码evals.py 逐段拆解模板的全部评估逻辑集中在evals.py仓库源码为 examples/ragas_examples/judge_alignment/evals.py。下面按功能拆解。两个评审指标基线 vs 改进版模板定义了两种DiscreteMetric用于对比同一评审任务下不同提示词的对齐表现# Baseline judge (simple prompt) accuracy_metric DiscreteMetric( nameaccuracy, promptCheck if the response contains points mentioned from the grading notes and return pass or fail.\n\nResponse: {response}\nGrading Notes: {grading_notes}, allowed_values[pass, fail], ) # Improved judge (enhanced with abbreviation guide) accuracy_metric_v2 DiscreteMetric( nameaccuracy, promptEvaluate if the response covers ALL the key concepts from the grading notes. Accept semantic equivalents but carefully check for missing concepts. ABBREVIATION GUIDE - decode these correctly: • Financial: valvaluation, post-$post-money, revrevenue, ARR/MRRAnnual/Monthly Recurring Revenue, COGSCost of Goods Sold, OpexOperating Expenses, LTVLifetime Value, CACCustomer Acquisition Cost • Business: mktmarket, reg/regsregulation/regulatory, corp govcorporate governance, integrintegration, SMSales Marketing, RDResearch Development, acqacquisition • Technical: syssystem, elimelimination, IPIntellectual Property, TAMTotal Addressable Market, diffdifferentiation • Metrics: NPSNet Promoter Score, SROISocial Return on Investment, projprojection, certcertification EVALUATION APPROACH: Step 1 - Parse grading notes into distinct concepts: - Separate by commas, semicolons, or line breaks - Each item is a concept that must be verified - Example: *Gross Margin* 40%, CAC, LTV:CAC 3:1 3 concepts Step 2 - For each concept, check if its addressed: - Accept semantic equivalents (e.g., customer acquisition cost CAC) - Accept implicit coverage when its clear (e.g., revenue forecasting covers historical vs forecasted rev) - Be flexible on exact numbers (e.g., around 40% acceptable for 40%) Step 3 - Count missing concepts: - Missing 0 concepts PASS - Missing 1 concepts FAIL (even one genuinely missing concept should fail) - Exception: If a long list (10 items) has 1 very minor detail missing but all major points covered, use judgment CRITICAL RULES: 1. Do NOT require exact wording - market demand mkt demand demand analysis 2. Markers (* or !) mean important, not mandatory exact phrases: - *traction evidence* can be satisfied by discussing metrics, growth, or validation - !unbiased assumptions can be satisfied by discussing assumption methodology 3. Numbers should be mentioned but accept approximations: - $47B to $10B can be $47 billion dropped to around $10 billion - LTV:CAC 3:1 can be LTV to CAC ratio of at least 3 to 1 or 3x or higher 4. FAIL only when concepts are genuinely absent: - If notes mention liquidation prefs, anti-dilution, board seats but response only has board seats → FAIL - If notes mention scalability, tech debt, IP but response never discusses technical risks → FAIL - If notes mention GDPR compliance and response never mentions GDPR or EU regulations → FAIL 5. PASS when ALL concepts present: - All concepts covered, even with different wording → PASS - Concepts addressed implicitly when clearly implied → PASS - Minor phrasing differences → PASS - One or more concepts genuinely absent → FAIL Response: {response} Grading Notes: {grading_notes} Are ALL distinct concepts from the grading notes covered in the response (accepting semantic equivalents and implicit coverage)?, allowed_values[pass, fail], )对比两个提示词可以发现 v2 版的改进思路补充领域缩写词典金融/商业/技术/指标四类、给出「拆解要点 → 逐点核对 → 统计缺失」的显式推理步骤、以及 PASS/FAIL 的判定规则与反例。这正是 LLM-as-judge 提示词工程中「把隐性标准显式化」的典型做法——judge_alignment模板的价值就在于能定量告诉你这类改进到底有没有效果。从底层实现看src/ragas/metrics/discrete.pyDiscreteMetric是SimpleLLMMetric与DiscreteValidator的组合核心参数有两个name指标名称用于结果标识与持久化prompt提示词模板可包含{response}、{grading_notes}等运行时填充的占位符allowed_values允许输出的离散取值集合默认是[pass, fail]。初始化时DiscreteMetric会基于allowed_values自动生成带reason推理过程与value离散结论两个字段的结构化响应模型并通过 instructor 库约束 LLM 的输出必须落在allowed_values内从而保证下游对齐比较的格式稳定。数据加载def load_dataset(csv_path: Optional[Path] None) - Dataset: Load annotated dataset with human judgments. Expected columns: question, grading_notes, response, target (pass/fail) path csv_path or (Path(__file__).resolve().parent / datasets / benchmark_df.csv) df pd.read_csv(path) dataset Dataset(namellm_judge_alignment, backendlocal/csv, root_dir.) for _, row in df.iterrows(): dataset.append({ question: row[question], grading_notes: row[grading_notes], response: row[response], target: str(row[target]).strip().lower(), }) return dataset数据以 CSV 形式存放模板生成后位于evals/datasets/目录每条样本包含四个字段question原始问题grading_notes人类评审使用的评分要点期望覆盖的概念清单response待评审的回答模板中直接使用数据集中预存的回答生产中可替换为 LLM 应用的输出target人类标注的 pass/fail 真值加载时统一转小写以对齐 LLM 输出。对齐指标比较两个离散结论discrete_metric(namejudge_alignment, allowed_values[pass, fail]) def judge_alignment(judge_label: str, human_label: str) - MetricResult: Compare judge decision with human label. judge judge_label.strip().lower() human human_label.strip().lower() if judge human: return MetricResult(valuepass, reasonfJudge{judge}; Human{human}) return MetricResult(valuefail, reasonfJudge{judge}; Human{human})这里的核心是discrete_metric装饰器src/ragas/metrics/discrete.py。与直接实例化DiscreteMetric不同它把一个普通 Python 函数包装成指标函数体负责接收judge_labelLLM 评审结论与human_label人类结论两个参数并返回比较结果装饰器负责把函数返回值规整为MetricResult(value..., reason...)结构。对齐逻辑本身极其简单——LLM 与人类结论相同即passaligned不同即failmisalignedreason字段同时记录两侧的取值方便事后排查具体哪一条不一致、不一致的原因是什么。实验编排Judge → Compare 流水线experiment() async def judge_experiment( row: Dict[str, Any], accuracy_metric: DiscreteMetric, llm, ): Run complete evaluation: Judge → Compare with human. # Step 1: Get response (in production, this is where youd call your LLM app) # For this evaluation, we use pre-existing responses from the dataset app_response row[response] # Step 2: Judge evaluates the response judge_score await accuracy_metric.ascore( questionrow[question], grading_notesrow[grading_notes], responseapp_response, llmllm, ) # Step 3: Compare judge decision with human target alignment judge_alignment.score( judge_labeljudge_score.value, human_labelrow[target] ) return { **row, judge_label: judge_score.value, judge_critique: judge_score.reason, alignment: alignment.value, alignment_reason: alignment.reason, }整个评估是一条三步流水线取回答生产中在此处调用被评测的 LLM 应用模板使用数据集中预存的回答LLM 评审调用accuracy_metric.ascore(...)让评审对回答打分把问题、评分要点和回答一并传入提示词对齐比较将评审结论judge_score.value与人类真值row[target]交给judge_alignment.score(...)比较并把judge_label评审结论、judge_critique评审理由、alignment是否对齐与alignment_reason对齐理由追加到原始样本上返回。experiment()装饰器src/ragas/experiment.py把该函数包装为可对数据集批量执行的实验通过arun(dataset, name...)为数据集中的每一条样本创建异步任务、以asyncio.as_completed并发执行并展示 tqdm 进度条最后把结果落盘到后端默认与数据集同源即本地 CSV/JSONL 目录并返回Experiment视图对象src/ragas/experiment.py。这也是为什么judge_experiment里可以放心使用await——整条流水线天然支持异步并发。两个入口main 与 main_v2async def main(): Example: evaluate judge with baseline prompt. dataset load_dataset() logger.info(fLoaded dataset with {len(dataset)} samples) openai_client AsyncOpenAI(api_keyos.environ.get(OPENAI_API_KEY)) llm llm_factory(gpt-4o-mini, clientopenai_client) logger.info(Running baseline evaluation...) results await judge_experiment.arun( dataset, namejudge_baseline_v1_gpt-4o-mini, accuracy_metricaccuracy_metric, llmllm, ) passed sum(1 for r in results if r[alignment] pass) total len(results) logger.info(f✅ Baseline alignment: {passed}/{total} passed ({passed/total:.1%})) return results async def main_v2(): Evaluate judge with improved v2 prompt. # 结构与 main 相同仅将 accuracy_metric 替换为 accuracy_metric_v2 # 实验名称改为 judge_accuracy_v2_gpt-4o-mini ...两个入口的差异仅在于使用的评审指标与实验命名方便你在同一份代码里分别跑出基线与改进版的对齐率进行对比。模型默认使用gpt-4o-mini通过llm_factory创建来自 src/ragas/llms并复用同一个AsyncOpenAI客户端。if __name__ __main__部分根据命令行参数决定运行哪个版本if __name__ __main__: import asyncio import sys # Run v2 if --v2 flag is passed, otherwise run baseline if len(sys.argv) 1 and sys.argv[1] --v2: asyncio.run(main_v2()) else: asyncio.run(main())结果如何解读对齐率与理由字段运行结束后你会得到两套关键输出对齐率passed / total的百分比。它直接回答「LLM 评审与人类评审有多一致」。基线提示词简单容易漏判、误判v2 提示词显式给出缩写词典与判定规则通常能显著提升对齐率——这正是模板希望你观察到的对比效果逐条结果每条样本新增了judge_label、judge_critique、alignment、alignment_reason四个字段。judge_critique记录了评审的推理过程alignment_reason记录了两侧取值两者结合可以精确定位「人类判 pass、LLM 判 fail」的具体样本及其原因是后续优化提示词的直接素材。MetricResultsrc/ragas/metrics/result.py同时承载value与reasonvalue是可用于程序比较的原始值reason是面向人可读的解释文本并支持traces附加输入输出追踪信息。测试数据说明模板自带的数据集围绕金融/商业场景构造包含预评审的回答pre-evaluated responses人类标注的 pass/fail 标签带期望要点的评分说明grading notes with expected points大量缩写与商业术语如 CAC、LTV、ARR/MRR、COGS、NPS 等用于考验评审能否正确解码专业表达。正是这些缩写术语使得「简单提示词基线评审」极易出现误判把缩写当作缺失概念而误判 fail从而让 v2 改进版的对齐优势一目了然。数据加载逻辑见上文load_dataset期望的 CSV 列为question、grading_notes、response、target。使用场景比较与迭代评审场景一比较评审版本模板天然支持 A/B 对比先后运行uv run python evals.py与uv run python evals.py --v2即可得到两个评审提示词的对齐率判断「提示词改进是否真的拉近了 LLM 与人类的评审标准」。抽象成伪代码即# Test baseline judge results_v1 await run_with_judge(accuracy_metric) # Test improved judge results_v2 await run_with_judge(accuracy_metric_v2) # Compare alignment rates场景二改进评审质量对齐评估的真正价值在于形成迭代闭环识别不一致模式从alignment_reason与judge_critique中找出 LLM 与人类分歧的典型样本归纳是术语误解、要点遗漏还是判定标准过严/过松更新评审提示词把归纳出的规则显式写进提示词如 v2 的缩写词典、Step 1-3 拆解流程、CRITICAL RULES 反例重新评估对齐再次运行实验观察对齐率是否上升重复直至满意持续迭代直到对齐率达到业务可接受的水平。这一闭环同样适用于更广的提示词与模型对比场景若想对比不同提示词的整体表现可参考 Prompt Evaluation 模板若想对比不同 LLM 作为评审的差异可参考 LLM Benchmarking 模板。小结judge_alignment模板把「LLM 评审 vs 人类评审」的一致性度量做成了开箱即用的工程化流程ragas quickstart judge_alignment一条命令即可获得完整项目evals.py用两个DiscreteMetric定义评审提示词用discrete_metric定义对齐比较逻辑用experiment驱动批量并发评估并落盘结果最终的对齐率与逐条理由为你提供量化反馈支撑「发现分歧 → 改进提示词 → 再评估」的评审质量迭代。无论你正在为 RAG 应用、Agent 还是工作流搭建自动化评估先把「评审本身靠不靠谱」用这个模板度量清楚都是值得优先做的一步。【免费下载链接】ragasSupercharge Your LLM Application Evaluations 项目地址: https://gitcode.com/gh_mirrors/ra/ragas创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考 Ragas CLI 实战：用 judge_alignment 模板度量 LLM-as-judge 与人类评估标准的对齐度
Ragas CLI 实战用 judge_alignment 模板度量 LLM-as-judge 与人类评估标准的对齐度【免费下载链接】ragasSupercharge Your LLM Application Evaluations 项目地址: https://gitcode.com/gh_mirrors/ra/ragas导读judge_alignment是 Ragas CLI 提供的官方快速开始模板之一用于量化「LLM 充当评审LLM-as-judge」时其评判结论与人类标注结果之间的一致性。本文将带你从零创建该项目、运行评估并逐行拆解模板源码深入DiscreteMetric、discrete_metric装饰器与experiment实验机制等底层实现帮助你掌握一套可复用的「评审质量度量 → 迭代提示词 → 再度量」闭环方法论。什么是 Judge Alignment 评估在 LLM 应用中越来越多团队用 LLM 替代人工来打分、判通过/失败。但 LLM 评审是否可靠它与人类评审标准到底有多一致judge_alignment模板给出的答案是把「LLM 评审的结论」与「人类标注的结论」逐一对比计算对齐率。该模板的评估对象并不是 RAG 系统本身而是评审这一环节场景Scenario数据集里预先存放已生成的回答pre-existing responses由 LLM 评审进行评判人类标签Human Labels每条回答附带人工标注的 pass/fail 真值ground truthLLM 评审LLM Judge评审依据给定的评分标准grading notes对同一条回答给出 pass/fail 结论对齐指标Alignment Metric逐条比较 LLM 结论与人类结论是否一致一致记为 aligned不一致记为 misaligned最终汇总为对齐率。快速开始五分钟跑通一个对齐评估1. 创建项目ragas quickstart judge_alignment cd judge_alignmentragas quickstart是 Ragas CLI 提供的脚手架命令实现在 src/ragas/cli.py执行后会从ragas_examples模板目录克隆一份完整可运行的示例工程。该命令支持-o/--output-dir指定输出目录例如ragas quickstart judge_alignment -o ./my-project。值得说明的是模板解析机制CLI 会依次尝试「本地已安装的ragas_examples包 → 当前 Ragas 仓库开发目录 → 从上游仓库下载压缩包」并将模板拷贝到目标目录见 src/ragas/cli.py。本仓库中该模板的原始源码位于 examples/ragas_examples/judge_alignment包含evals.py与__init__.py数据集与依赖清单会随模板一并生成。2. 安装依赖模板使用uv管理依赖uv sync3. 设置 API Key评估过程需要调用 LLM 作为评审这里以 OpenAI 为例export OPENAI_API_KEYyour-openai-key模板代码通过load_dotenv()自动加载.env文件因此也可以把 key 写入.env后直接运行。4. 运行评估uv run python evals.py默认运行基线版本baseline judge如需运行改进后的 v2 评审追加--v2参数uv run python evals.py --v2运行结束后终端会打印形如✅ Baseline alignment: 18/20 passed (90.0%)的对齐率汇总实验详情则持久化到experiments/与logs/目录。项目结构解读模板生成的项目结构如下judge_alignment/ ├── README.md # Project documentation ├── pyproject.toml # Project configuration ├── evals.py # Evaluation workflow ├── __init__.py # Python package marker └── evals/ ├── datasets/ # Test datasets ├── experiments/ # Evaluation results └── logs/ # Execution logs其中evals.py是整个评估流程的核心既定义了两种评审提示词对应的指标也定义了数据加载、对齐比较和实验编排逻辑。深入代码evals.py 逐段拆解模板的全部评估逻辑集中在evals.py仓库源码为 examples/ragas_examples/judge_alignment/evals.py。下面按功能拆解。两个评审指标基线 vs 改进版模板定义了两种DiscreteMetric用于对比同一评审任务下不同提示词的对齐表现# Baseline judge (simple prompt) accuracy_metric DiscreteMetric( nameaccuracy, promptCheck if the response contains points mentioned from the grading notes and return pass or fail.\n\nResponse: {response}\nGrading Notes: {grading_notes}, allowed_values[pass, fail], ) # Improved judge (enhanced with abbreviation guide) accuracy_metric_v2 DiscreteMetric( nameaccuracy, promptEvaluate if the response covers ALL the key concepts from the grading notes. Accept semantic equivalents but carefully check for missing concepts. ABBREVIATION GUIDE - decode these correctly: • Financial: valvaluation, post-$post-money, revrevenue, ARR/MRRAnnual/Monthly Recurring Revenue, COGSCost of Goods Sold, OpexOperating Expenses, LTVLifetime Value, CACCustomer Acquisition Cost • Business: mktmarket, reg/regsregulation/regulatory, corp govcorporate governance, integrintegration, SMSales Marketing, RDResearch Development, acqacquisition • Technical: syssystem, elimelimination, IPIntellectual Property, TAMTotal Addressable Market, diffdifferentiation • Metrics: NPSNet Promoter Score, SROISocial Return on Investment, projprojection, certcertification EVALUATION APPROACH: Step 1 - Parse grading notes into distinct concepts: - Separate by commas, semicolons, or line breaks - Each item is a concept that must be verified - Example: *Gross Margin* 40%, CAC, LTV:CAC 3:1 3 concepts Step 2 - For each concept, check if its addressed: - Accept semantic equivalents (e.g., customer acquisition cost CAC) - Accept implicit coverage when its clear (e.g., revenue forecasting covers historical vs forecasted rev) - Be flexible on exact numbers (e.g., around 40% acceptable for 40%) Step 3 - Count missing concepts: - Missing 0 concepts PASS - Missing 1 concepts FAIL (even one genuinely missing concept should fail) - Exception: If a long list (10 items) has 1 very minor detail missing but all major points covered, use judgment CRITICAL RULES: 1. Do NOT require exact wording - market demand mkt demand demand analysis 2. Markers (* or !) mean important, not mandatory exact phrases: - *traction evidence* can be satisfied by discussing metrics, growth, or validation - !unbiased assumptions can be satisfied by discussing assumption methodology 3. Numbers should be mentioned but accept approximations: - $47B to $10B can be $47 billion dropped to around $10 billion - LTV:CAC 3:1 can be LTV to CAC ratio of at least 3 to 1 or 3x or higher 4. FAIL only when concepts are genuinely absent: - If notes mention liquidation prefs, anti-dilution, board seats but response only has board seats → FAIL - If notes mention scalability, tech debt, IP but response never discusses technical risks → FAIL - If notes mention GDPR compliance and response never mentions GDPR or EU regulations → FAIL 5. PASS when ALL concepts present: - All concepts covered, even with different wording → PASS - Concepts addressed implicitly when clearly implied → PASS - Minor phrasing differences → PASS - One or more concepts genuinely absent → FAIL Response: {response} Grading Notes: {grading_notes} Are ALL distinct concepts from the grading notes covered in the response (accepting semantic equivalents and implicit coverage)?, allowed_values[pass, fail], )对比两个提示词可以发现 v2 版的改进思路补充领域缩写词典金融/商业/技术/指标四类、给出「拆解要点 → 逐点核对 → 统计缺失」的显式推理步骤、以及 PASS/FAIL 的判定规则与反例。这正是 LLM-as-judge 提示词工程中「把隐性标准显式化」的典型做法——judge_alignment模板的价值就在于能定量告诉你这类改进到底有没有效果。从底层实现看src/ragas/metrics/discrete.pyDiscreteMetric是SimpleLLMMetric与DiscreteValidator的组合核心参数有两个name指标名称用于结果标识与持久化prompt提示词模板可包含{response}、{grading_notes}等运行时填充的占位符allowed_values允许输出的离散取值集合默认是[pass, fail]。初始化时DiscreteMetric会基于allowed_values自动生成带reason推理过程与value离散结论两个字段的结构化响应模型并通过 instructor 库约束 LLM 的输出必须落在allowed_values内从而保证下游对齐比较的格式稳定。数据加载def load_dataset(csv_path: Optional[Path] None) - Dataset: Load annotated dataset with human judgments. Expected columns: question, grading_notes, response, target (pass/fail) path csv_path or (Path(__file__).resolve().parent / datasets / benchmark_df.csv) df pd.read_csv(path) dataset Dataset(namellm_judge_alignment, backendlocal/csv, root_dir.) for _, row in df.iterrows(): dataset.append({ question: row[question], grading_notes: row[grading_notes], response: row[response], target: str(row[target]).strip().lower(), }) return dataset数据以 CSV 形式存放模板生成后位于evals/datasets/目录每条样本包含四个字段question原始问题grading_notes人类评审使用的评分要点期望覆盖的概念清单response待评审的回答模板中直接使用数据集中预存的回答生产中可替换为 LLM 应用的输出target人类标注的 pass/fail 真值加载时统一转小写以对齐 LLM 输出。对齐指标比较两个离散结论discrete_metric(namejudge_alignment, allowed_values[pass, fail]) def judge_alignment(judge_label: str, human_label: str) - MetricResult: Compare judge decision with human label. judge judge_label.strip().lower() human human_label.strip().lower() if judge human: return MetricResult(valuepass, reasonfJudge{judge}; Human{human}) return MetricResult(valuefail, reasonfJudge{judge}; Human{human})这里的核心是discrete_metric装饰器src/ragas/metrics/discrete.py。与直接实例化DiscreteMetric不同它把一个普通 Python 函数包装成指标函数体负责接收judge_labelLLM 评审结论与human_label人类结论两个参数并返回比较结果装饰器负责把函数返回值规整为MetricResult(value..., reason...)结构。对齐逻辑本身极其简单——LLM 与人类结论相同即passaligned不同即failmisalignedreason字段同时记录两侧的取值方便事后排查具体哪一条不一致、不一致的原因是什么。实验编排Judge → Compare 流水线experiment() async def judge_experiment( row: Dict[str, Any], accuracy_metric: DiscreteMetric, llm, ): Run complete evaluation: Judge → Compare with human. # Step 1: Get response (in production, this is where youd call your LLM app) # For this evaluation, we use pre-existing responses from the dataset app_response row[response] # Step 2: Judge evaluates the response judge_score await accuracy_metric.ascore( questionrow[question], grading_notesrow[grading_notes], responseapp_response, llmllm, ) # Step 3: Compare judge decision with human target alignment judge_alignment.score( judge_labeljudge_score.value, human_labelrow[target] ) return { **row, judge_label: judge_score.value, judge_critique: judge_score.reason, alignment: alignment.value, alignment_reason: alignment.reason, }整个评估是一条三步流水线取回答生产中在此处调用被评测的 LLM 应用模板使用数据集中预存的回答LLM 评审调用accuracy_metric.ascore(...)让评审对回答打分把问题、评分要点和回答一并传入提示词对齐比较将评审结论judge_score.value与人类真值row[target]交给judge_alignment.score(...)比较并把judge_label评审结论、judge_critique评审理由、alignment是否对齐与alignment_reason对齐理由追加到原始样本上返回。experiment()装饰器src/ragas/experiment.py把该函数包装为可对数据集批量执行的实验通过arun(dataset, name...)为数据集中的每一条样本创建异步任务、以asyncio.as_completed并发执行并展示 tqdm 进度条最后把结果落盘到后端默认与数据集同源即本地 CSV/JSONL 目录并返回Experiment视图对象src/ragas/experiment.py。这也是为什么judge_experiment里可以放心使用await——整条流水线天然支持异步并发。两个入口main 与 main_v2async def main(): Example: evaluate judge with baseline prompt. dataset load_dataset() logger.info(fLoaded dataset with {len(dataset)} samples) openai_client AsyncOpenAI(api_keyos.environ.get(OPENAI_API_KEY)) llm llm_factory(gpt-4o-mini, clientopenai_client) logger.info(Running baseline evaluation...) results await judge_experiment.arun( dataset, namejudge_baseline_v1_gpt-4o-mini, accuracy_metricaccuracy_metric, llmllm, ) passed sum(1 for r in results if r[alignment] pass) total len(results) logger.info(f✅ Baseline alignment: {passed}/{total} passed ({passed/total:.1%})) return results async def main_v2(): Evaluate judge with improved v2 prompt. # 结构与 main 相同仅将 accuracy_metric 替换为 accuracy_metric_v2 # 实验名称改为 judge_accuracy_v2_gpt-4o-mini ...两个入口的差异仅在于使用的评审指标与实验命名方便你在同一份代码里分别跑出基线与改进版的对齐率进行对比。模型默认使用gpt-4o-mini通过llm_factory创建来自 src/ragas/llms并复用同一个AsyncOpenAI客户端。if __name__ __main__部分根据命令行参数决定运行哪个版本if __name__ __main__: import asyncio import sys # Run v2 if --v2 flag is passed, otherwise run baseline if len(sys.argv) 1 and sys.argv[1] --v2: asyncio.run(main_v2()) else: asyncio.run(main())结果如何解读对齐率与理由字段运行结束后你会得到两套关键输出对齐率passed / total的百分比。它直接回答「LLM 评审与人类评审有多一致」。基线提示词简单容易漏判、误判v2 提示词显式给出缩写词典与判定规则通常能显著提升对齐率——这正是模板希望你观察到的对比效果逐条结果每条样本新增了judge_label、judge_critique、alignment、alignment_reason四个字段。judge_critique记录了评审的推理过程alignment_reason记录了两侧取值两者结合可以精确定位「人类判 pass、LLM 判 fail」的具体样本及其原因是后续优化提示词的直接素材。MetricResultsrc/ragas/metrics/result.py同时承载value与reasonvalue是可用于程序比较的原始值reason是面向人可读的解释文本并支持traces附加输入输出追踪信息。测试数据说明模板自带的数据集围绕金融/商业场景构造包含预评审的回答pre-evaluated responses人类标注的 pass/fail 标签带期望要点的评分说明grading notes with expected points大量缩写与商业术语如 CAC、LTV、ARR/MRR、COGS、NPS 等用于考验评审能否正确解码专业表达。正是这些缩写术语使得「简单提示词基线评审」极易出现误判把缩写当作缺失概念而误判 fail从而让 v2 改进版的对齐优势一目了然。数据加载逻辑见上文load_dataset期望的 CSV 列为question、grading_notes、response、target。使用场景比较与迭代评审场景一比较评审版本模板天然支持 A/B 对比先后运行uv run python evals.py与uv run python evals.py --v2即可得到两个评审提示词的对齐率判断「提示词改进是否真的拉近了 LLM 与人类的评审标准」。抽象成伪代码即# Test baseline judge results_v1 await run_with_judge(accuracy_metric) # Test improved judge results_v2 await run_with_judge(accuracy_metric_v2) # Compare alignment rates场景二改进评审质量对齐评估的真正价值在于形成迭代闭环识别不一致模式从alignment_reason与judge_critique中找出 LLM 与人类分歧的典型样本归纳是术语误解、要点遗漏还是判定标准过严/过松更新评审提示词把归纳出的规则显式写进提示词如 v2 的缩写词典、Step 1-3 拆解流程、CRITICAL RULES 反例重新评估对齐再次运行实验观察对齐率是否上升重复直至满意持续迭代直到对齐率达到业务可接受的水平。这一闭环同样适用于更广的提示词与模型对比场景若想对比不同提示词的整体表现可参考 Prompt Evaluation 模板若想对比不同 LLM 作为评审的差异可参考 LLM Benchmarking 模板。小结judge_alignment模板把「LLM 评审 vs 人类评审」的一致性度量做成了开箱即用的工程化流程ragas quickstart judge_alignment一条命令即可获得完整项目evals.py用两个DiscreteMetric定义评审提示词用discrete_metric定义对齐比较逻辑用experiment驱动批量并发评估并落盘结果最终的对齐率与逐条理由为你提供量化反馈支撑「发现分歧 → 改进提示词 → 再评估」的评审质量迭代。无论你正在为 RAG 应用、Agent 还是工作流搭建自动化评估先把「评审本身靠不靠谱」用这个模板度量清楚都是值得优先做的一步。【免费下载链接】ragasSupercharge Your LLM Application Evaluations 项目地址: https://gitcode.com/gh_mirrors/ra/ragas创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考