大数据数据分析后端【免费下载链接】datafusionApache DataFusion SQL Query Engine项目地址https://gitcode.com/gh_mirrors/datafu/datafusion点击查看免费下载本篇指南聚焦datafusion-cliDataFusion 的命令行交互工具内置的四个查询函数——parquet_metadata、metadata_cache、statistics_cache与list_files_cache。它们默认不被 DataFusion SQL 引擎包含仅在 CLI 启动时注入用于透视 Parquet 文件物理结构、监控ListingTable背后的文件元数据缓存与统计信息缓存。读完本篇你将掌握这四个函数的调用语法、返回列语义以及它们在仓库源码中的底层实现与相关运行时配置。CLI 专属函数是什么datafusion-cli是 DataFusion 的交互式命令行工具用于对本地或远程如 S3的 CSV、Parquet、JSON、Arrow、Avro 数据文件执行 SQL。除了继承 SQL 引擎的全部能力它还在启动阶段注册了一批引擎默认不包含的表函数Table Function / UDTF。这些注册动作发生在 datafusion-cli/src/main.rs// register parquet_metadata table function to get metadata from parquet files ctx.register_udtf(parquet_metadata, Arc::new(ParquetMetadataFunc {})); // register metadata_cache table function to get the contents of the file metadata cache ctx.register_udtf( metadata_cache, Arc::new(MetadataCacheFunc::new( ctx.task_ctx().runtime_env().cache_manager.clone(), )), ); // register statistics_cache table function ... ctx.register_udtf( statistics_cache, Arc::new(StatisticsCacheFunc::new( ctx.task_ctx().runtime_env().cache_manager.clone(), )), ); ctx.register_udtf( list_files_cache, Arc::new(ListFilesCacheFunc::new( ctx.task_ctx().runtime_env().cache_manager.clone(), )), );可以看到parquet_metadata与其余三个缓存函数有本质分工差异前者直接读取一个 Parquet 文件的页脚元数据并物化成内存表后三者则通过CacheManager读取当前会话运行时环境中的缓存条目快照。它们都实现了TableFunctionImpl源码位于 datafusion-cli/src/functions.rs因此返回的都是可被 SQL 投影、过滤、聚合的普通表。由于这四个函数是普通表函数你可以在大多数可以使用表引用的地方使用它们——FROM子句、JOIN、子查询等场景均可。parquet_metadata透视 Parquet 文件的物理结构parquet_metadata用于检查一个 Parquet 文件的详细元数据包括统计信息、各类大小、页偏移等。这对于理解 Parquet 文件的物理布局Row Group、Column Chunk、页结构非常有帮助。基本用法以下查询查看hits.parquet中WatchID列的元数据SELECT path_in_schema, row_group_id, row_group_num_rows, stats_min, stats_max, total_compressed_size FROM parquet_metadata(hits.parquet) WHERE path_in_schema WatchID LIMIT 3; ------------------------------------------------------------------------------------------------------------------- | path_in_schema | row_group_id | row_group_num_rows | stats_min | stats_max | total_compressed_size | ------------------------------------------------------------------------------------------------------------------- | WatchID | 0 | 450560 | 4611687214012840539 | 9223369186199968220 | 3883759 | | WatchID | 1 | 612174 | 4611689135232456464 | 9223371478009085789 | 5176803 | | WatchID | 2 | 344064 | 4611692774829951781 | 9223363791697310021 | 3031680 | ------------------------------------------------------------------------------------------------------------------- 3 rows in set. Query took 0.053 seconds.返回列说明该函数返回的表中每一行对应文件中的一个 Column Chunk列块包含以下列column_namedata_typeDescriptionfilenameUtf8Name of the filerow_group_idInt64Row group index the column chunk belongs torow_group_num_rowsInt64Count of rows stored in the row grouprow_group_num_columnsInt64Total number of columns in the row group (same for all row groups)row_group_bytesInt64Number of bytes used to store the row group (not including metadata)column_idInt64ID of the columnfile_offsetInt64Offset within the file that this column chunks data beginsnum_valuesInt64Total number of values in this column chunkpath_in_schemaUtf8Path (column name) of the column chunk in the schematypeUtf8Parquet data type of the column chunkstats_minUtf8The minimum value for this column chunk, if stored in the statistics, cast to a stringstats_maxUtf8The maximum value for this column chunk, if stored in the statistics, cast to a stringstats_null_countInt64Number of null values in this column chunk, if stored in the statisticsstats_distinct_countInt64Number of distinct values in this column chunk, if stored in the statisticsstats_min_valueUtf8Same asstats_minstats_max_valueUtf8Same asstats_maxcompressionUtf8Block level compression (e.g.SNAPPY) used for this column chunkencodingsUtf8All block level encodings (e.g.[PLAIN_DICTIONARY, PLAIN, RLE]) used for this column chunkindex_page_offsetInt64Offset in the file of thepage index页索引if anydictionary_page_offsetInt64Offset in the file of the dictionary page, if anydata_page_offsetInt64Offset in the file of the first data page, if anytotal_compressed_sizeInt64Number of bytes the column chunks data after encoding and compression (what is stored in the file)total_uncompressed_sizeInt64Number of bytes the column chunks data after encoding关于stats_min/stats_max的更多信息例如统计信息仅在写入端启用了统计收集时才存在以及各类偏移字段的精确含义请参阅 Parquet 官方格式文档。源码实现一次真实的元数据扫描parquet_metadata的参数解析在 datafusion-cli/src/functions.rs 中完成它接受字符串字面量单引号parquet_metadata(x.parquet)或列引用形式双引号parquet_metadata(x.parquet)否则返回parquet_metadata requires string argument as its input的计划错误。核心流程是使用parquetcrate 的SerializedFileReader打开文件、读取元数据然后双重循环遍历每个 Row Group 的每个 Column Chunk见 functions.rsfor (rg_idx, row_group) in metadata.row_groups().iter().enumerate() { for (col_idx, column) in row_group.columns().iter().enumerate() { // 收集 filename、row_group_id、num_values、file_offset、 // path_in_schema、compression、encodings、各页偏移、压缩前后大小…… } }值得关注的是统计信息的转换逻辑convert_parquet_statisticsfunctions.rs它对Statistics::Boolean / Int32 / Int64 / Int96 / Float / Double / ByteArray / FixedLenByteArray分别处理其中ByteArray与FixedLenByteArray在 ParquetConvertedType::UTF8时优先按 UTF-8 字符串输出否则输出字节的字符串表示stats_min/stats_max与stats_min_value/stats_max_value实际来自同一对转换结果。若 Column Chunk 没有统计信息这些列均为 NULL。最终生成的 RecordBatch 通过MemorySourceConfig包装为内存表提供扫描functions.rs因此该函数的扫描本身是一次性物化的内存读取。metadata_cache查看文件元数据缓存metadata_cache展示 DataFusion 中ListingTable实现使用的默认 File Metadata Cache文件元数据缓存。当扫描包含大量文件的目录时这个缓存用于加速从文件中读取元数据。使用前提与示例首先通过CREATE EXTERNAL TABLE创建一个表语法详见 CREATE EXTERNAL TABLE create external table hits stored as parquet location s3://clickhouse-public-datasets/hits_compatible/athena_partitioned/;然后查询metadata_cache函数即可查看缓存内容 select * from metadata_cache(); --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | path | file_modified | file_size_bytes | e_tag | version | metadata_size_bytes | hits | extra | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | hits_compatible/athena_partitioned/hits_61.parquet | 2022-07-03T15:40:34 | 117270944 | 5db11cad1ca0d80d748fc92c914b010a-6 | NULL | 212949 | 0 | page_indexfalse | | hits_compatible/athena_partitioned/hits_32.parquet | 2022-07-03T15:37:17 | 94506004 | 2f7db49a9fe242179590b615b94a39d2-5 | NULL | 278157 | 0 | page_indexfalse | | hits_compatible/athena_partitioned/hits_40.parquet | 2022-07-03T15:38:07 | 142508647 | 9e5852b45a469d5a05bf270a286eab8a-8 | NULL | 212917 | 0 | page_indexfalse | | hits_compatible/athena_partitioned/hits_93.parquet | 2022-07-03T15:44:07 | 127987774 | 751100bf0dac7d489b9836abf3108b99-7 | NULL | 278318 | 0 | page_indexfalse | | . | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------由于metadata_cache是普通表函数你可以在大多数使用表引用的地方使用它。例如计算缓存条目消耗的总内存 select sum(metadata_size_bytes) from metadata_cache(); ------------------------------------------- | sum(metadata_cache().metadata_size_bytes) | ------------------------------------------- | 22972345 | -------------------------------------------返回列说明column_namedata_typeDescriptionpathUtf8File path relative to the object store / filesystem rootfile_modifiedTimestampLast modified time of the filefile_size_bytesUInt64Size of the file in bytese_tagUtf8Entity Tag (ETag) of the file if availableversionUtf8Version of the file if available (for object stores that support versioning)metadata_size_bytesUInt64Size of the cached metadata in memory (not its thrift encoded form)hitsUInt64Number of times the cached metadata has been accessedextraUtf8Extra information about the cached metadata (e.g., if page index information is included)源码实现要点metadata_cache不接受任何参数functions.rs 会拒绝非空参数列表。实现上它遍历CacheManager的get_file_metadata_cache().list_entries()对每个条目输出ObjectMetalast_modified、size、e_tag、version、缓存内存占用size_bytes与访问次数hits并把file_metadata.extra_info()中形如page_indexfalse的键值对拼进extra列functions.rs。extra_info中的信息如页索引是否已加载正是理解该缓存够不够用的关键例如page_indexfalse表示该文件元数据缓存中不含页索引信息谓词下推命中页索引时可能仍需额外读取。statistics_cache查看文件统计信息缓存与metadata_cache类似statistics_cache展示ListingTable使用的 File Statistics Cache文件统计信息缓存内容。注意前置条件统计信息要被收集必须启用配置项datafusion.execution.collect_statistics在 datafusion/common/src/config.rs 中定义默认值为true。该配置控制ListingTable扫描时是否收集并缓存文件的表级统计信息行数、字节数、排序信息等。查询示例 select * from statistics_cache(); ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | path | file_modified | file_size_bytes | e_tag | version | num_rows | num_columns | table_size_bytes | statistics_size_bytes | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | .../hits.parquet | 2022-06-25T22:22:22 | 14779976446 | 0-5e24d1ee16380-370f48 | NULL | Exact(99997497) | 105 | Exact(36445943240) | 0 | ------------------------------------------------------------------------------------------------------------------------------------------------------------------返回列说明column_namedata_typeDescriptionpathUtf8File path relative to the object store / filesystem rootfile_modifiedTimestampLast modified time of the filefile_size_bytesUInt64Size of the file in bytese_tagUtf8Entity Tag (ETag) of the file if availableversionUtf8Version of the file if available (for object stores that support versioning)num_rowsUtf8Number of rows in the tablenum_columnsUInt64Number of columns in the tabletable_size_bytesUtf8Size of the table, in byteshitsUInt64Number of times the cached file statistics has been accessedstatistics_size_bytesUInt64Size of the cached statistics in memory源码实现要点注意num_rows与table_size_bytes的数据类型是Utf8字符串因为内部保存的是Statistics枚举Exact(...)/Estimated(...)源码中直接以entry.value.statistics.num_rows.to_string()输出functions.rs例如示例中的Exact(99997497)表示行数是精确统计而非估算值。statistics_size_bytes则是调用heap_size(mut DFHeapSizeCtx::default())计算的堆上内存占用。当datafusion.execution.collect_statistics未开启时get_file_statistic_cache()返回的缓存可能为空查询结果即为空表。list_files_cache查看文件列表缓存list_files_cache展示ListingTable使用的ListFilesCache内容。创建ListingTable时DataFusion 会列出表位置下的文件并把结果缓存在ListFilesCache中后续针对同一张表的查询可以复用该缓存信息无需重新执行对象存储的 list 操作。缓存条目按表table作用域隔离。使用示例ListFilesCache的 TTL 可以通过运行时配置调整 set datafusion.runtime.list_files_cache_ttl 30s; create external table overturemaps stored as parquet location s3://overturemaps-us-west-2/release/2025-12-17.0/themebase/typeinfrastructure; 0 row(s) fetched. select table, path, metadata_size_bytes, expires_in, unnest(metadata_list)[file_size_bytes] as file_size_bytes, unnest(metadata_list)[e_tag] as e_tag from list_files_cache() limit 10; ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | table | path | metadata_size_bytes | expires_in | file_size_bytes | e_tag | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | overturemaps | release/2025-12-17.0/themebase/typeinfrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 999055952 | 35fc8fbe8400960b54c66fbb408c48e8-60 | | overturemaps | release/2025-12-17.0/themebase/typeinfrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 975592768 | 8a16e10b722681cdc00242564b502965-59 | | overturemaps | release/2025-12-17.0/themebase/typeinfrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 1082925747 | 24cd13ddb5e0e438952d2499f5dabe06-65 | | overturemaps | release/2025-12-17.0/themebase/typeinfrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 1008425557 | 37663e31c7c64d4ef355882bcd47e361-61 | | overturemaps | release/2025-12-17.0/themebase/typeinfrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 1065561905 | 4e7c50d2d1b3c5ed7b82b4898f5ac332-64 | | overturemaps | release/2025-12-17.0/themebase/typeinfrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 1045655427 | 8fff7e6a72d375eba668727c55d4f103-63 | | overturemaps | release/2025-12-17.0/themebase/typeinfrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 1086822683 | b67167d8022d778936c330a52a5f1922-65 | | overturemaps | release/2025-12-17.0/themebase/typeinfrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 1016732378 | 6d70857a0473ed9ed3fc6e149814168b-61 | | overturemaps | release/2025-12-17.0/themebase/typeinfrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 991363784 | c9cafb42fcbb413f851691c895dd7c2b-60 | | overturemaps | release/2025-12-17.0/themebase/typeinfrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 1032469715 | 7540252d0d67158297a67038a3365e0f-62 | -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------示例中通过unnest(metadata_list)[file_size_bytes]把嵌套的metadata_list列展开成行再以 map 索引方式提取每个文件的file_size_bytes与e_tag。返回列说明column_namedata_typeDescriptiontableUtf8Name of the tablepathUtf8File path relative to the object store / filesystem rootmetadata_size_bytesUInt64Size of the cached metadata in memory (not its thrift encoded form)expires_inDuration(ms)Last modified time of the filehitsUInt64Number of times the cached metadata has been accessedmetadata_listList(Struct)List of metadatas, one for each file under the path.其中metadata_list中每个 metadata 结构体包含以下字段示例值{ file_path: release/2025-12-17.0/themebase/typeinfrastructure/part-00000-d556e455-e0c5-4940-b367-daff3287a952-c000.zstd.parquet, file_modified: 2025-12-17T22:20:29, file_size_bytes: 999055952, e_tag: 35fc8fbe8400960b54c66fbb408c48e8-60, version: null }源码实现要点list_files_cache的 schema 定义见 functions.rsexpires_in采用Duration(ms)而非Timestamp因为ListFilesEntry.expires字段类型是Instant无法直接得到秒数时间戳源码注释也明确说明了这一设计取舍metadata_list则是List(Struct)嵌套结构由StructArray与GenericListArray组装而成functions.rs。expires_in的计算方式是t.duration_since(now).as_millis()functions.rs即从当前时刻到过期时刻的剩余毫秒数——这正好解释了示例中25.264 secs这类倒计时语义。与缓存相关的运行时配置metadata_cache、statistics_cache、list_files_cache三个函数本身是只读的仪表盘真正决定缓存行为的配置位于运行时环境与执行配置中配置项说明默认值来源datafusion.execution.collect_statistics是否收集并缓存文件统计信息statistics_cache有内容的前提datafusion/common/src/config.rs默认truedatafusion.runtime.list_files_cache_ttlListFilesCache条目过期时间可用set datafusion.runtime.list_files_cache_ttl 30s调整datafusion/execution/src/cache/cache_manager.rs 中DEFAULT_LIST_FILES_CACHE_TTL None即默认无限期datafusion.runtime.list_files_cache_limitListFilesCache内存上限DEFAULT_LIST_FILES_CACHE_MEMORY_LIMIT 1MiBcache_manager.rsdatafusion.runtime.metadata_cache_limitFile Metadata Cache 内存上限DEFAULT_METADATA_CACHE_LIMIT 50MiBcache_manager.rsdatafusion.runtime.file_statistics_cache_limitFile Statistics Cache 内存上限DEFAULT_FILE_STATISTICS_MEMORY_LIMIT 20MiBcache_manager.rs这些运行时变量的解析在 datafusion/core/src/execution/context/mod.rs 中完成list_files_cache_ttl会被解析为Duration并通过RuntimeEnvBuilder::with_object_list_cache_ttl生效list_files_cache_limit、metadata_cache_limit、file_statistics_cache_limit则作为容量限制传入对应缓存构造器reset命令可将它们恢复为默认值context/mod.rs。从源码结构看File Statistics Cache 内部保存的CachedFileMetadata同时包含ObjectMeta用于校验文件 size/last_modified 是否变化与统计/排序信息命中后还需通过is_valid_for(current_meta, current_schema_fingerprint)校验有效性见 cache_manager.rs而ListFilesCache以表的 base path 为稳定缓存键缓存值为CachedFileList文件列表 时间戳分区过滤在取回缓存后通过files_matching_prefix完成cache_manager.rs。理解了这些底层语义就能更准确地解读三个缓存函数输出的hits、metadata_size_bytes、expires_in等指标。小结四个 CLI 专属函数覆盖了从单个文件物理结构到会话级缓存运行状态的诊断链条parquet_metadata(file.parquet)—— 单文件级逐 Column Chunk 输出行组、统计、压缩、页偏移等 23 个字段metadata_cache()—— 表级列出 File Metadata Cache 中每个文件的元数据与内存占用statistics_cache()—— 表级列出文件统计缓存需开启datafusion.execution.collect_statisticslist_files_cache()—— 表级列出ListFilesCache中每个表路径的文件清单、TTL 倒计时与嵌套元数据。它们既可以配合WHERE、LIMIT做定向诊断如按path_in_schema过滤查看某列、用sum(metadata_size_bytes)估算缓存内存也可以与unnest等函数组合展开嵌套结构。结合本文给出的源码路径你可以进一步在 datafusion-cli/src/functions.rs 中追踪每个函数的完整实现或在 datafusion/execution/src/cache/cache_manager.rs 中研究三类缓存的容量限制、TTL 与淘汰机制。赞分享大数据数据分析后端【免费下载链接】datafusionApache DataFusion SQL Query Engine项目地址https://gitcode.com/gh_mirrors/datafu/datafusion点击查看免费下载相关推荐mise implode 完全卸载指南安全移除 CLI、工具数据与缓存mise implode 完全卸载指南安全移除 CLI、工具数据与缓存 mise implode 是 mise 内置的一键卸载命令用于彻底移除 mise开发工具CLInds-bootstrap vs 传统模拟器为何选择原生运行解决方案nds bootstrap vs 传统模拟器为何选择原生运行解决方案 nds bootstrap是一款强大的原生运行解决方案能够直接在NDS/DSi主机上嵌入式固件DataFusion Spark 兼容函数测试套件实战spark/.slt 文件编写规范与 datafusion-cli --spark 使用指南DataFusion Spark 兼容函数测试套件实战spark/.slt 文件编写规范与 datafusion cli spark 使用指南 导读 Apac大数据数据分析后端上一篇gemma-4-e4b-it-qat-OptiQ-4bit多模态应用开发图像文本处理全解析下一篇Rufus免费制作启动U盘完整指南GPT/MBR分区方案与UEFI/BIOS参数详解创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考 DataFusion CLI 专属函数完全指南：parquet_metadata 与三大文件缓存自省工具
大数据数据分析后端【免费下载链接】datafusionApache DataFusion SQL Query Engine项目地址https://gitcode.com/gh_mirrors/datafu/datafusion点击查看免费下载本篇指南聚焦datafusion-cliDataFusion 的命令行交互工具内置的四个查询函数——parquet_metadata、metadata_cache、statistics_cache与list_files_cache。它们默认不被 DataFusion SQL 引擎包含仅在 CLI 启动时注入用于透视 Parquet 文件物理结构、监控ListingTable背后的文件元数据缓存与统计信息缓存。读完本篇你将掌握这四个函数的调用语法、返回列语义以及它们在仓库源码中的底层实现与相关运行时配置。CLI 专属函数是什么datafusion-cli是 DataFusion 的交互式命令行工具用于对本地或远程如 S3的 CSV、Parquet、JSON、Arrow、Avro 数据文件执行 SQL。除了继承 SQL 引擎的全部能力它还在启动阶段注册了一批引擎默认不包含的表函数Table Function / UDTF。这些注册动作发生在 datafusion-cli/src/main.rs// register parquet_metadata table function to get metadata from parquet files ctx.register_udtf(parquet_metadata, Arc::new(ParquetMetadataFunc {})); // register metadata_cache table function to get the contents of the file metadata cache ctx.register_udtf( metadata_cache, Arc::new(MetadataCacheFunc::new( ctx.task_ctx().runtime_env().cache_manager.clone(), )), ); // register statistics_cache table function ... ctx.register_udtf( statistics_cache, Arc::new(StatisticsCacheFunc::new( ctx.task_ctx().runtime_env().cache_manager.clone(), )), ); ctx.register_udtf( list_files_cache, Arc::new(ListFilesCacheFunc::new( ctx.task_ctx().runtime_env().cache_manager.clone(), )), );可以看到parquet_metadata与其余三个缓存函数有本质分工差异前者直接读取一个 Parquet 文件的页脚元数据并物化成内存表后三者则通过CacheManager读取当前会话运行时环境中的缓存条目快照。它们都实现了TableFunctionImpl源码位于 datafusion-cli/src/functions.rs因此返回的都是可被 SQL 投影、过滤、聚合的普通表。由于这四个函数是普通表函数你可以在大多数可以使用表引用的地方使用它们——FROM子句、JOIN、子查询等场景均可。parquet_metadata透视 Parquet 文件的物理结构parquet_metadata用于检查一个 Parquet 文件的详细元数据包括统计信息、各类大小、页偏移等。这对于理解 Parquet 文件的物理布局Row Group、Column Chunk、页结构非常有帮助。基本用法以下查询查看hits.parquet中WatchID列的元数据SELECT path_in_schema, row_group_id, row_group_num_rows, stats_min, stats_max, total_compressed_size FROM parquet_metadata(hits.parquet) WHERE path_in_schema WatchID LIMIT 3; ------------------------------------------------------------------------------------------------------------------- | path_in_schema | row_group_id | row_group_num_rows | stats_min | stats_max | total_compressed_size | ------------------------------------------------------------------------------------------------------------------- | WatchID | 0 | 450560 | 4611687214012840539 | 9223369186199968220 | 3883759 | | WatchID | 1 | 612174 | 4611689135232456464 | 9223371478009085789 | 5176803 | | WatchID | 2 | 344064 | 4611692774829951781 | 9223363791697310021 | 3031680 | ------------------------------------------------------------------------------------------------------------------- 3 rows in set. Query took 0.053 seconds.返回列说明该函数返回的表中每一行对应文件中的一个 Column Chunk列块包含以下列column_namedata_typeDescriptionfilenameUtf8Name of the filerow_group_idInt64Row group index the column chunk belongs torow_group_num_rowsInt64Count of rows stored in the row grouprow_group_num_columnsInt64Total number of columns in the row group (same for all row groups)row_group_bytesInt64Number of bytes used to store the row group (not including metadata)column_idInt64ID of the columnfile_offsetInt64Offset within the file that this column chunks data beginsnum_valuesInt64Total number of values in this column chunkpath_in_schemaUtf8Path (column name) of the column chunk in the schematypeUtf8Parquet data type of the column chunkstats_minUtf8The minimum value for this column chunk, if stored in the statistics, cast to a stringstats_maxUtf8The maximum value for this column chunk, if stored in the statistics, cast to a stringstats_null_countInt64Number of null values in this column chunk, if stored in the statisticsstats_distinct_countInt64Number of distinct values in this column chunk, if stored in the statisticsstats_min_valueUtf8Same asstats_minstats_max_valueUtf8Same asstats_maxcompressionUtf8Block level compression (e.g.SNAPPY) used for this column chunkencodingsUtf8All block level encodings (e.g.[PLAIN_DICTIONARY, PLAIN, RLE]) used for this column chunkindex_page_offsetInt64Offset in the file of thepage index页索引if anydictionary_page_offsetInt64Offset in the file of the dictionary page, if anydata_page_offsetInt64Offset in the file of the first data page, if anytotal_compressed_sizeInt64Number of bytes the column chunks data after encoding and compression (what is stored in the file)total_uncompressed_sizeInt64Number of bytes the column chunks data after encoding关于stats_min/stats_max的更多信息例如统计信息仅在写入端启用了统计收集时才存在以及各类偏移字段的精确含义请参阅 Parquet 官方格式文档。源码实现一次真实的元数据扫描parquet_metadata的参数解析在 datafusion-cli/src/functions.rs 中完成它接受字符串字面量单引号parquet_metadata(x.parquet)或列引用形式双引号parquet_metadata(x.parquet)否则返回parquet_metadata requires string argument as its input的计划错误。核心流程是使用parquetcrate 的SerializedFileReader打开文件、读取元数据然后双重循环遍历每个 Row Group 的每个 Column Chunk见 functions.rsfor (rg_idx, row_group) in metadata.row_groups().iter().enumerate() { for (col_idx, column) in row_group.columns().iter().enumerate() { // 收集 filename、row_group_id、num_values、file_offset、 // path_in_schema、compression、encodings、各页偏移、压缩前后大小…… } }值得关注的是统计信息的转换逻辑convert_parquet_statisticsfunctions.rs它对Statistics::Boolean / Int32 / Int64 / Int96 / Float / Double / ByteArray / FixedLenByteArray分别处理其中ByteArray与FixedLenByteArray在 ParquetConvertedType::UTF8时优先按 UTF-8 字符串输出否则输出字节的字符串表示stats_min/stats_max与stats_min_value/stats_max_value实际来自同一对转换结果。若 Column Chunk 没有统计信息这些列均为 NULL。最终生成的 RecordBatch 通过MemorySourceConfig包装为内存表提供扫描functions.rs因此该函数的扫描本身是一次性物化的内存读取。metadata_cache查看文件元数据缓存metadata_cache展示 DataFusion 中ListingTable实现使用的默认 File Metadata Cache文件元数据缓存。当扫描包含大量文件的目录时这个缓存用于加速从文件中读取元数据。使用前提与示例首先通过CREATE EXTERNAL TABLE创建一个表语法详见 CREATE EXTERNAL TABLE create external table hits stored as parquet location s3://clickhouse-public-datasets/hits_compatible/athena_partitioned/;然后查询metadata_cache函数即可查看缓存内容 select * from metadata_cache(); --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | path | file_modified | file_size_bytes | e_tag | version | metadata_size_bytes | hits | extra | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | hits_compatible/athena_partitioned/hits_61.parquet | 2022-07-03T15:40:34 | 117270944 | 5db11cad1ca0d80d748fc92c914b010a-6 | NULL | 212949 | 0 | page_indexfalse | | hits_compatible/athena_partitioned/hits_32.parquet | 2022-07-03T15:37:17 | 94506004 | 2f7db49a9fe242179590b615b94a39d2-5 | NULL | 278157 | 0 | page_indexfalse | | hits_compatible/athena_partitioned/hits_40.parquet | 2022-07-03T15:38:07 | 142508647 | 9e5852b45a469d5a05bf270a286eab8a-8 | NULL | 212917 | 0 | page_indexfalse | | hits_compatible/athena_partitioned/hits_93.parquet | 2022-07-03T15:44:07 | 127987774 | 751100bf0dac7d489b9836abf3108b99-7 | NULL | 278318 | 0 | page_indexfalse | | . | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------由于metadata_cache是普通表函数你可以在大多数使用表引用的地方使用它。例如计算缓存条目消耗的总内存 select sum(metadata_size_bytes) from metadata_cache(); ------------------------------------------- | sum(metadata_cache().metadata_size_bytes) | ------------------------------------------- | 22972345 | -------------------------------------------返回列说明column_namedata_typeDescriptionpathUtf8File path relative to the object store / filesystem rootfile_modifiedTimestampLast modified time of the filefile_size_bytesUInt64Size of the file in bytese_tagUtf8Entity Tag (ETag) of the file if availableversionUtf8Version of the file if available (for object stores that support versioning)metadata_size_bytesUInt64Size of the cached metadata in memory (not its thrift encoded form)hitsUInt64Number of times the cached metadata has been accessedextraUtf8Extra information about the cached metadata (e.g., if page index information is included)源码实现要点metadata_cache不接受任何参数functions.rs 会拒绝非空参数列表。实现上它遍历CacheManager的get_file_metadata_cache().list_entries()对每个条目输出ObjectMetalast_modified、size、e_tag、version、缓存内存占用size_bytes与访问次数hits并把file_metadata.extra_info()中形如page_indexfalse的键值对拼进extra列functions.rs。extra_info中的信息如页索引是否已加载正是理解该缓存够不够用的关键例如page_indexfalse表示该文件元数据缓存中不含页索引信息谓词下推命中页索引时可能仍需额外读取。statistics_cache查看文件统计信息缓存与metadata_cache类似statistics_cache展示ListingTable使用的 File Statistics Cache文件统计信息缓存内容。注意前置条件统计信息要被收集必须启用配置项datafusion.execution.collect_statistics在 datafusion/common/src/config.rs 中定义默认值为true。该配置控制ListingTable扫描时是否收集并缓存文件的表级统计信息行数、字节数、排序信息等。查询示例 select * from statistics_cache(); ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | path | file_modified | file_size_bytes | e_tag | version | num_rows | num_columns | table_size_bytes | statistics_size_bytes | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | .../hits.parquet | 2022-06-25T22:22:22 | 14779976446 | 0-5e24d1ee16380-370f48 | NULL | Exact(99997497) | 105 | Exact(36445943240) | 0 | ------------------------------------------------------------------------------------------------------------------------------------------------------------------返回列说明column_namedata_typeDescriptionpathUtf8File path relative to the object store / filesystem rootfile_modifiedTimestampLast modified time of the filefile_size_bytesUInt64Size of the file in bytese_tagUtf8Entity Tag (ETag) of the file if availableversionUtf8Version of the file if available (for object stores that support versioning)num_rowsUtf8Number of rows in the tablenum_columnsUInt64Number of columns in the tabletable_size_bytesUtf8Size of the table, in byteshitsUInt64Number of times the cached file statistics has been accessedstatistics_size_bytesUInt64Size of the cached statistics in memory源码实现要点注意num_rows与table_size_bytes的数据类型是Utf8字符串因为内部保存的是Statistics枚举Exact(...)/Estimated(...)源码中直接以entry.value.statistics.num_rows.to_string()输出functions.rs例如示例中的Exact(99997497)表示行数是精确统计而非估算值。statistics_size_bytes则是调用heap_size(mut DFHeapSizeCtx::default())计算的堆上内存占用。当datafusion.execution.collect_statistics未开启时get_file_statistic_cache()返回的缓存可能为空查询结果即为空表。list_files_cache查看文件列表缓存list_files_cache展示ListingTable使用的ListFilesCache内容。创建ListingTable时DataFusion 会列出表位置下的文件并把结果缓存在ListFilesCache中后续针对同一张表的查询可以复用该缓存信息无需重新执行对象存储的 list 操作。缓存条目按表table作用域隔离。使用示例ListFilesCache的 TTL 可以通过运行时配置调整 set datafusion.runtime.list_files_cache_ttl 30s; create external table overturemaps stored as parquet location s3://overturemaps-us-west-2/release/2025-12-17.0/themebase/typeinfrastructure; 0 row(s) fetched. select table, path, metadata_size_bytes, expires_in, unnest(metadata_list)[file_size_bytes] as file_size_bytes, unnest(metadata_list)[e_tag] as e_tag from list_files_cache() limit 10; ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | table | path | metadata_size_bytes | expires_in | file_size_bytes | e_tag | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | overturemaps | release/2025-12-17.0/themebase/typeinfrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 999055952 | 35fc8fbe8400960b54c66fbb408c48e8-60 | | overturemaps | release/2025-12-17.0/themebase/typeinfrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 975592768 | 8a16e10b722681cdc00242564b502965-59 | | overturemaps | release/2025-12-17.0/themebase/typeinfrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 1082925747 | 24cd13ddb5e0e438952d2499f5dabe06-65 | | overturemaps | release/2025-12-17.0/themebase/typeinfrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 1008425557 | 37663e31c7c64d4ef355882bcd47e361-61 | | overturemaps | release/2025-12-17.0/themebase/typeinfrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 1065561905 | 4e7c50d2d1b3c5ed7b82b4898f5ac332-64 | | overturemaps | release/2025-12-17.0/themebase/typeinfrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 1045655427 | 8fff7e6a72d375eba668727c55d4f103-63 | | overturemaps | release/2025-12-17.0/themebase/typeinfrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 1086822683 | b67167d8022d778936c330a52a5f1922-65 | | overturemaps | release/2025-12-17.0/themebase/typeinfrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 1016732378 | 6d70857a0473ed9ed3fc6e149814168b-61 | | overturemaps | release/2025-12-17.0/themebase/typeinfrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 991363784 | c9cafb42fcbb413f851691c895dd7c2b-60 | | overturemaps | release/2025-12-17.0/themebase/typeinfrastructure | 2750 | 0 days 0 hours 0 mins 25.264 secs | 1032469715 | 7540252d0d67158297a67038a3365e0f-62 | -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------示例中通过unnest(metadata_list)[file_size_bytes]把嵌套的metadata_list列展开成行再以 map 索引方式提取每个文件的file_size_bytes与e_tag。返回列说明column_namedata_typeDescriptiontableUtf8Name of the tablepathUtf8File path relative to the object store / filesystem rootmetadata_size_bytesUInt64Size of the cached metadata in memory (not its thrift encoded form)expires_inDuration(ms)Last modified time of the filehitsUInt64Number of times the cached metadata has been accessedmetadata_listList(Struct)List of metadatas, one for each file under the path.其中metadata_list中每个 metadata 结构体包含以下字段示例值{ file_path: release/2025-12-17.0/themebase/typeinfrastructure/part-00000-d556e455-e0c5-4940-b367-daff3287a952-c000.zstd.parquet, file_modified: 2025-12-17T22:20:29, file_size_bytes: 999055952, e_tag: 35fc8fbe8400960b54c66fbb408c48e8-60, version: null }源码实现要点list_files_cache的 schema 定义见 functions.rsexpires_in采用Duration(ms)而非Timestamp因为ListFilesEntry.expires字段类型是Instant无法直接得到秒数时间戳源码注释也明确说明了这一设计取舍metadata_list则是List(Struct)嵌套结构由StructArray与GenericListArray组装而成functions.rs。expires_in的计算方式是t.duration_since(now).as_millis()functions.rs即从当前时刻到过期时刻的剩余毫秒数——这正好解释了示例中25.264 secs这类倒计时语义。与缓存相关的运行时配置metadata_cache、statistics_cache、list_files_cache三个函数本身是只读的仪表盘真正决定缓存行为的配置位于运行时环境与执行配置中配置项说明默认值来源datafusion.execution.collect_statistics是否收集并缓存文件统计信息statistics_cache有内容的前提datafusion/common/src/config.rs默认truedatafusion.runtime.list_files_cache_ttlListFilesCache条目过期时间可用set datafusion.runtime.list_files_cache_ttl 30s调整datafusion/execution/src/cache/cache_manager.rs 中DEFAULT_LIST_FILES_CACHE_TTL None即默认无限期datafusion.runtime.list_files_cache_limitListFilesCache内存上限DEFAULT_LIST_FILES_CACHE_MEMORY_LIMIT 1MiBcache_manager.rsdatafusion.runtime.metadata_cache_limitFile Metadata Cache 内存上限DEFAULT_METADATA_CACHE_LIMIT 50MiBcache_manager.rsdatafusion.runtime.file_statistics_cache_limitFile Statistics Cache 内存上限DEFAULT_FILE_STATISTICS_MEMORY_LIMIT 20MiBcache_manager.rs这些运行时变量的解析在 datafusion/core/src/execution/context/mod.rs 中完成list_files_cache_ttl会被解析为Duration并通过RuntimeEnvBuilder::with_object_list_cache_ttl生效list_files_cache_limit、metadata_cache_limit、file_statistics_cache_limit则作为容量限制传入对应缓存构造器reset命令可将它们恢复为默认值context/mod.rs。从源码结构看File Statistics Cache 内部保存的CachedFileMetadata同时包含ObjectMeta用于校验文件 size/last_modified 是否变化与统计/排序信息命中后还需通过is_valid_for(current_meta, current_schema_fingerprint)校验有效性见 cache_manager.rs而ListFilesCache以表的 base path 为稳定缓存键缓存值为CachedFileList文件列表 时间戳分区过滤在取回缓存后通过files_matching_prefix完成cache_manager.rs。理解了这些底层语义就能更准确地解读三个缓存函数输出的hits、metadata_size_bytes、expires_in等指标。小结四个 CLI 专属函数覆盖了从单个文件物理结构到会话级缓存运行状态的诊断链条parquet_metadata(file.parquet)—— 单文件级逐 Column Chunk 输出行组、统计、压缩、页偏移等 23 个字段metadata_cache()—— 表级列出 File Metadata Cache 中每个文件的元数据与内存占用statistics_cache()—— 表级列出文件统计缓存需开启datafusion.execution.collect_statisticslist_files_cache()—— 表级列出ListFilesCache中每个表路径的文件清单、TTL 倒计时与嵌套元数据。它们既可以配合WHERE、LIMIT做定向诊断如按path_in_schema过滤查看某列、用sum(metadata_size_bytes)估算缓存内存也可以与unnest等函数组合展开嵌套结构。结合本文给出的源码路径你可以进一步在 datafusion-cli/src/functions.rs 中追踪每个函数的完整实现或在 datafusion/execution/src/cache/cache_manager.rs 中研究三类缓存的容量限制、TTL 与淘汰机制。赞分享大数据数据分析后端【免费下载链接】datafusionApache DataFusion SQL Query Engine项目地址https://gitcode.com/gh_mirrors/datafu/datafusion点击查看免费下载相关推荐mise implode 完全卸载指南安全移除 CLI、工具数据与缓存mise implode 完全卸载指南安全移除 CLI、工具数据与缓存 mise implode 是 mise 内置的一键卸载命令用于彻底移除 mise开发工具CLInds-bootstrap vs 传统模拟器为何选择原生运行解决方案nds bootstrap vs 传统模拟器为何选择原生运行解决方案 nds bootstrap是一款强大的原生运行解决方案能够直接在NDS/DSi主机上嵌入式固件DataFusion Spark 兼容函数测试套件实战spark/.slt 文件编写规范与 datafusion-cli --spark 使用指南DataFusion Spark 兼容函数测试套件实战spark/.slt 文件编写规范与 datafusion cli spark 使用指南 导读 Apac大数据数据分析后端上一篇gemma-4-e4b-it-qat-OptiQ-4bit多模态应用开发图像文本处理全解析下一篇Rufus免费制作启动U盘完整指南GPT/MBR分区方案与UEFI/BIOS参数详解创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考