Refhttps://github.com/vllm-project/guidellm这是一个比较专业的LLM性能评测工具里面工程实现也比较优雅技术深度非常深。其他评测工具参考sglang/VLLM性能评测: bench_serving工具安装pip install guidellm --upgrade -i https://pypi.tuna.tsinghua.edu.cn/simplegit clone源码安装pip uninstall -y guidellm git clone https://github.com/vllm-project/guidellm.git cd guidellm/ pip install -e ./docker imageshttps://github.com/vllm-project/guidellm/pkgs/container/guidellm使用样例guidellm run \ --config chat \ --backend kindopenai_http,targethttp://localhost:30000,modelQwen3-30B-A3B \ --profile kindconcurrent,streams20 \ --constraint kindmax_requests,count320 \ --tokenizer kindhf_auto,model/data0/models/Qwen3-30B-A3B-Thinking-2507-FP8 \ --data {kind:synthetic_text,prefix_buckets:[{bucket_weight:100,prefix_count:1,prefix_tokens:11428}],prompt_tokens:860,prompt_tokens_stdev:500,output_tokens:512,output_tokens_stdev:100} \ --seed kindstatic,value1Key parameters:--profile kindtype: Defines the traffic pattern —synchronous,concurrent,throughput,constant,poisson, orsweep--profile kindconstant,rate10: Forconstant/poisson, set requests per second in the profile config; forconcurrent, usestreams; forthroughput, usemax_concurrency--constraint kindmax_duration,secondssecondsor--constraint kindmax_requests,countcount: Limit each strategy by time or request countprefix_count1只生成一个共享前缀所有请求重复使用。prefix_tokens8192共享前缀长度约 8192 tokens。prompt_tokens8192不包含前缀的独立 prompt 长度。多轮对话评测https://github.com/vllm-project/guidellm/blob/main/docs/guides/multiturn.mdKey parameters:- turns3 : Number of turns per conversation- prompt_tokens200 : Input tokens per turn- output_tokens100 : Output tokens per turn- prefix_tokens100 : (Optional) Add a system promptWith System Prompts (Prefixes)guidellm benchmark run \ --target http://localhost:8000 \ --model meta-llama/Llama-3.1-8B-Instruct \ --request-format /v1/chat/completions \ --profile constant \ --rate 2.0 \ --max-requests 100 \ --data prompt_tokens150,output_tokens75,turns4,prefix_tokens100设置共享前缀guidellm benchmark run \ --target http://localhost:30000 \ --model models/DeepSeek-V3.2 \ --request-format /v1/chat/completions \ --data prompt_tokens1024,output_tokens1,turns1,prefix_count1,prefix_tokens1024,prompt_tokens_stdev1000,output_tokens_stdev1 \ --rate-type concurrent \ --random-seed 1 \ --rate 30 \ --max-requests 30guidellm benchmark \ --target http://x.x.x.x \ --model deepseek-v3.1 \ --processor /path/DeepSeek-V3.1-Terminus \ --request-type chat_completions \ --backend-args {validate_backend: false} \ --data prompt_tokens2048,output_tokens512,prompt_tokens_stdev200,output_tokens_stdev100 \ --rate-type concurrent \ --rate 100 \ --max-requests 128 # --data-sampler random \ # can reference OpenAIHTTPBackend args --backend-kwargs {api_key: sk-...} \ --backend-kwargs {http2: false, timeout: 120} \guidellm benchmark \ --target http://localhost:30000 \ --model DeepSeek-V3.1-Terminus \ --processor model/DeepSeek-V3.1-Terminus \ --processor-args {trust_remote_code: true} \ --data prompt_tokens2048,output_tokens512,prompt_tokens_stdev200,output_tokens_stdev100 \ --rate-type poisson \ --rate 1 \ --max-requests 1536guidellm benchmark \ --target http://localhost:30000 \ --model DeepSeek-V3.1-Terminus \ --processor model/DeepSeek-V3.1-Terminus \ --processor-args {trust_remote_code: true} \ --data prompt_tokens2048,output_tokens512,prompt_tokens_stdev200,output_tokens_stdev50 \ --rate-type concurrent \ --rate 2560 \ --max-requests 3200 # --random-seed 1设置前缀长度prefix_tokens_max结果样例ℹ Request Latency Statistics (Completed Requests) |||||||||||||| | Benchmark | Request Latency ||| TTFT ||| ITL ||| TPOT ||| | Strategy | Sec ||| ms ||| ms ||| ms ||| | | Mean | Mdn | p99 | Mean | Mdn | p99 | Mean | Mdn | p99 | Mean | Mdn | p99 | |-----------|------|------|------|--------|--------|--------|------|------|------|------|------|------| | poisson | 11.2 | 11.4 | 12.9 | 2136.1 | 2256.4 | 2326.5 | xxx | xxx | xxx | xxx | xxx |xxx | |||||||||||||| ℹ Server Throughput Statistics |||||||||||| | Benchmark | Requests |||| Input Tokens || Output Tokens || Total Tokens || | Strategy | Per Sec || Concurrency || Per Sec || Per Sec || Per Sec || | | Mdn | Mean | Mdn | Mean | Mdn | Mean | Mdn | Mean | Mdn | Mean | |-----------|-----|------|-------|------|--------|---------|--------|-------|-------|--------| | poisson | 0.2 | 0.8 | 12.0 | 9.0 | 5435.5 | 25898.4 | 418.1 | 546.6 | 421.6 | 2687.4 | ||||||||||||注意与国内的evalscope对比性能时guidellm的ITL evalscope的TPOT。但是guidellm的TPOT ! evalscope的ITL。注意最新版这块计算交换了。当前0.4.0默认ITL/TTFT输出median和P95没有找到配置方法。要改成mean, mdn, p99可以自行修改代码vim /usr/local/lib/python3.12/dist-packages/guidellm/benchmark/outputs/console.py 100_get_stat_type_name_val和add_stats默认参数def add_stats( self, xxx types: Sequence[StatTypesAlias] (mean, median, p99), ): xxx classmethod def _get_stat_type_name_val( cls, stat_type: StatTypesAlias, stats: DistributionSummary | None ) - tuple[str, float | None]: if stat_type mean: return Mean, stats.mean if stats else None elif stat_type median: return Mdn, stats.median if stats else None elif stat_type p95: return p95, stats.percentiles.p95 if stats else None elif stat_type p99: return p99, stats.percentiles.p99 if stats else None else: raise ValueError(fUnsupported stat type: {stat_type})以及print_server_throughput_table等调用add_stats所设置的参数。自定义数据集创建jsonl数据集文件例如{prompt: Hello, how are you?, output_tokens_count: 5, additional_column: foo} {prompt: What is your name?, output_tokens_count: 3, additional_column: baz}Key Fieldsprompt(required): The main text/input. GuideLLM also recognizes common aliases:instruction,input,inputs,question,context,text,content, orbodyoutput_tokens_count(optional): Expected output token count. Aliases:output_tokens,completion_tokensprompt_tokens_count(optional): Input token count. Aliases:prompt_tokens,input_tokensAdditional custom columns can be included and referenced as neededUsing Your Custom JSONL Datasetguidellm benchmark \ --target http://localhost:30000 \ --data custom_dataset.jsonl \ --backend-args {validate_backend: false} \ --request-type chat_completions \ --rate-type concurrent \ --rate 10 \ --max-requests 10 # --processor model_path \ # --processor-args {trust_remote_code: true} \ # --data-sampler shuffle \参数设置参数设置方法命令行参数设置以及环境变量例如参考docs\guides\configuration.mdexport GUIDELLM__OPENAI__API_KEYyour-api-keyGUIDELLM__REQUEST_TIMEOUT等等。重要参数https://github.com/vllm-project/guidellm/blob/main/README.md评测方法rate-type通常采用并发模式或者poisson模式。然后针对性设置--rate参数--rate:Benchmark rate(s) to test. Meaning depends on profile: sweepnumber of benchmarks, concurrentconcurrent requests, async/constant/poissonrequests per second.poisson模式除了rate看上去还可以通过环境变量设置max_concurrency。--request-type可以选择评测类型和端口例如/v1/completions 还是 /v1/chat/completionsGenerativeRequestType Literal[text_completions,chat_completions,audio_transcriptions,audio_translations,]To benchmark the text completions endpoint ( /v1/completions ) instead of the default chat completions endpoint ( /v1/chat/completions ), you need to use the --request-type text_completions CLI option.代码逻辑参数定义和运行入口src\guidellm\__main__.py调用src\guidellm\benchmark\entrypoints.py定义的benchmark_generative_text()async def benchmark_generative_text( args: BenchmarkGenerativeTextArgs, progress: GenerativeConsoleBenchmarkerProgress | None None, console: Console | None None, **constraints: dict[str, ConstraintInitializer | Any], ) - tuple[GenerativeBenchmarksReport, dict[str, Any]]:backend, model await resolve_backend()创建评测backend当前默认为注册名为openai_http的OpenAIHTTPBackendmodel为评测的模型id例如DeepSeekprocessor await resolve_processor(processorargs.processor, modelmodel, consoleconsole)args.processor: Tokenizer pathrequest_loader await resolve_request_loader( dataargs.data, modelmodel, data_argsargs.data_args, data_samplesargs.data_samples, processorprocessor, processor_argsargs.processor_args,profile await resolve_profile( profileargs.profile, rateargs.rate, random_seedargs.random_seed, constraintsconstraints, max_secondsargs.max_seconds, max_requestsargs.max_requests,benchmarker Benchmarker()核心评测调用async for benchmark in benchmarker.run( benchmark_classargs.benchmark_cls, requestsrequest_loader, backendbackend, profileprofile, environmentNonDistributedEnvironment(), dataargs.data, progressprogress, sample_requestsargs.sample_requests, warmupargs.warmup, cooldownargs.cooldown, prefer_response_metricsargs.prefer_response_metrics, ): if benchmark: report.benchmarks.append(benchmark)数据收集GenerativeBenchmark.compile()GenerativeMetrics.compile()time_per_output_token_msStatusDistributionSummary.from_values( value_typesrequest_types, values[req.time_per_output_token_ms or 0.0 for req in requests], ), inter_token_latency_msStatusDistributionSummary.from_values( value_typesrequest_types, values[req.inter_token_latency_ms or 0.0 for req in requests], ),TTFT/ITL/TOPT等计算逻辑class GenerativeRequestStats(StandardBaseDict): def request_latency(self) - float | None: End-to-end request processing latency in seconds. :return: Duration from request start to completion, or None if unavailable. return self.info.timings.request_end - self.info.timings.request_start def time_to_first_token_ms(self) - float | None: Time to first token generation in milliseconds. :return: Latency from request start to first token, or None if unavailable. return 1000 * ( self.info.timings.first_iteration - self.info.timings.request_start ) def time_per_output_token_ms(self) - float | None: Average time per output token in milliseconds. Includes time for first token and all subsequent tokens. :return: Average milliseconds per output token, or None if unavailable. return ( 1000 * (self.info.timings.last_iteration - self.info.timings.request_start) / self.output_metrics.total_tokens ) def inter_token_latency_ms(self) - float | None: Average inter-token latency in milliseconds. Measures time between token generations, excluding first token. :return: Average milliseconds between tokens, or None if unavailable. return ( 1000 * (self.info.timings.last_iteration - self.info.timings.first_iteration) / (self.output_metrics.total_tokens - 1) ) computed_field # type: ignore[misc] property def tokens_per_second(self) - float | None: Overall token throughput including prompt and output tokens. :return: Total tokens per second, or None if unavailable. if not (latency : self.request_latency) or self.total_tokens is None: return None return self.total_tokens / latency computed_field # type: ignore[misc] property def output_tokens_per_second(self) - float | None: Output token generation throughput. :return: Output tokens per second, or None if unavailable. return self.output_tokens / latency computed_field # type: ignore[misc] property def output_tokens_per_iteration(self) - float | None: Average output tokens generated per iteration. :return: Output tokens per iteration, or None if unavailable. return self.output_tokens / self.info.timings.iterations结果生成和打印output_format_results {} for key, output in output_formats.items(): output_result await output.finalize(report) output_format_results[key] output_result # print to consoleGenerativeBenchmarkerOutput.register(console) class GenerativeBenchmarkerConsole(GenerativeBenchmarkerOutput): async def finalize(self, report: GenerativeBenchmarksReport) - str: Print the complete benchmark report to the console. :param report: The completed benchmark report. :return: self._print_benchmarks_metadata(report.benchmarks) self._print_benchmarks_info(report.benchmarks) self._print_benchmarks_stats(report.benchmarks)Benchmarker.run()strategies_generator profile.strategies_generator() strategy, constraints next(strategies_generator) scheduler: Scheduler[RequestT, ResponseT] Scheduler() while strategy is not None: async for ( response,request,request_info,scheduler_state, ) in scheduler.run( requestsrequests, backendbackend, strategystrategy, startup_durationwarmup if warmup and warmup 1 else 0.0, envenvironment, **constraints or {}, ): try: benchmark_class.update_estimate( args, estimated_state, response, request, request_info, scheduler_state, )strategies_generator profile.strategies_generator()strategy, constraints next(strategies_generator)创建通过while创建多个并发的benchmarkScheduler不同评测方案possion, concurrency等设置ProfileSynchronousProfile: synchronousConcurrentProfile:concurrentThroughputProfile: throughputAsyncProfile: [async, constant, poisson]Backend - OpenAIHTTPBackendprocess_startupvalidateprocess_shutdownavailable_modelsresolveresponse_handler self._resolve_response_handler(request_typerequest.request_type)src\guidellm\backends\response_handlers.py