GraphRAG 输入与分块指南支持的数据格式、统一 documents Schema 与元数据前置Field Prepending【免费下载链接】graphragA modular graph-based Retrieval-Augmented Generation (RAG) system项目地址: https://gitcode.com/GitHub_Trending/gr/graphrag导读输入数据如何被读取、标准化和切分直接决定了 GraphRAG 图索引的质量与可追溯性。本文以 docs/index/inputs.md 为主线系统讲解 GraphRAG 支持的多种输入格式纯文本、CSV、JSON、JSONL、Parquet、MarkItDown、统一的documentsDataFrame 列结构、自备 DataFrame 的 Bring-Your-Own 接入方式以及分块Chunking阶段通过prepend_metadata将文档元数据重复复制到每个文本块中的配置方法。读完你将能够为自己的语料选择最合适的输入格式、正确配置settings.yaml的input/chunking段、并让每个文本块携带文档级共享信息以提升下游实体提取与检索效果。输入在整个索引流水线中的位置在 GraphRAG 的默认索引流程中详见 docs/index/default_dataflow.md输入处理发生在最前端的Phase 1: Compose TextUnits原始文件先被读取为documents再按配置被切分为更小的TextUnit文本块后续的实体/关系提取、社区检测、社区报告等阶段全部基于这些文本块展开。Documents -- Chunk -- Text Units因此如何把各种形态的源文件变成一张结构统一的documents表是全流程的第一道关卡也是本文的主题。输入加载在索引流水线中由两个核心 workflow 完成load_input_documents负责将存储中的文件读入并解析成统一格式实现见 load_input_documents.pycreate_base_text_units负责把documents切分为text_units并将prepend_metadata配置作用于切分过程实现见 create_base_text_units.py。统一 Documents 模式Input Loading and Schema无论采用哪种输入格式所有文件都会在 GraphRAG 内部被加载为一个documentsDataFrame每个文档占一行并遵循如下共享列结构名称类型说明idstr文档 ID。由文本内容哈希生成保证跨多次运行结果稳定增量更新场景尤为关键。textstr文档完整文本内容。titlestr文档名称。部分格式允许通过配置指定使用哪一列作为标题。creation_datestr文档创建日期以 ISO8601 字符串表示。该值从源文件系统的时间属性中采集。raw_datadict结构化输入的原始源行/对象可选。可被分块阶段的元数据前置metadata prepending使用。该TextDocument数据结构的仓库实现位于 text_document.py其中TextDocument.get(field)支持标准字段id、title、text、creation_date直接取属性其余字段名则会到raw_data中按字段名甚至支持点号嵌套字段查找——这正是下文字段前置能引用结构化源数据任意列的实现基础creation_date由存储层通过storage.get_creation_date(path)从文件系统中采集见 text.py 中对纯文本文件的用法id默认使用 SHA-512 对文本内容进行哈希生成纯文本读取器直接调用gen_sha512_hash({text: text}, [text])见 text.py 与 hashing.py这也是文档强调ID 基于文本内容哈希以跨运行稳定的源码依据。流水线运行结束后最终的 documents 表会以 parquet 形式落盘其最终表结构可参考 docs/index/outputs.md 中关于 final documents 表的说明。输入加载链路可注入的 InputReader 架构所有输入格式的解析都统一收敛到同一个架构InputReader基类 InputReaderFactory工厂。理解这一点有助于你判断遇到不支持的格式怎么办。InputReader 抽象基类input_reader.py 定义了InputReader构造函数接收storage文件来源、file_pattern文件匹配正则与encoding默认utf-8它实现了异步迭代协议__aiter__/_iterate_files用self._storage.find(re.compile(self._file_pattern))找到所有匹配文件逐个调用抽象方法read_file(path)把单个文件解析成list[TextDocument]单个文件解析失败不会中断整体流程——代码会打印 warning 并跳过该文件继续处理具有较高的健壮性。InputReaderFactory 与注册机制input_reader_factory.py 中的create_input_reader(config, storage)依据config.type分发到对应的 Reader 实现input.type配置值Reader 实现默认文件匹配正则text默认TextFileReader.*\.txt$csvCSVFileReader.*\.csv$jsonJSONFileReader—jsonlJSONLinesFileReader—markitdownMarkItDownFileReader—parquetParquetFileReader—同时仓库还导出了register_input_reader(type, initializer)公共注册函数开发者可以把自己实现的 Reader 类按标准 Factory 模式注册进去见 input_reader_factory.py这部分即文档中Custom File Handling所述能力。配置模型说明输入相关配置由 graph_rag_config.py 中的input: InputConfig承载其字段type、encoding、file_pattern、id_column、title_column、text_column定义在 input_config.py并且InputConfig开启了extraallow以便自定义 Reader 增加私有字段。支持的输入格式详解Formats下面逐类说明每种格式的加载语义与可配置列映射。若你的数据形态不在其中文档建议两条路一是自行实现InputReader注册进工厂二是写脚本把数据先转换为以下某一种格式。纯文本Plain Text以.txt结尾的文本文件默认input.type即text。加载规则整文件内容作为text字段title始终为文件名每个文件产出一行TextDocumentraw_data为空None。从源码可以看到纯文本读取器把文件名作为 title、把文件内容哈希作为 id、从文件系统采集创建时间见 text.py。CSV逗号分隔以.csv结尾的文件加载语义采用csv.DictReader逐行解析csv.pyCSV 的每一行被当作一篇独立文档输入目录下若有多个 CSV 文件会被拼接成一个统一的documentsDataFrame可通过input配置块指定text_column与title_column让结构化内容中的指定列充当正文与标题若未配置title使用文件名、text_column默认取名为text的列若数据中存在id列则直接使用否则按前文规则基于文本内容哈希生成 ID。需要留意的是CSV 正文单元格内容通常需要做引号转义以符合 CSV 规范正文中含有换行或逗号的字段务必正确包裹。JSON以.json结尾、内容符合 JSON 规范 的结构化文件。加载语义使用 Python 标准库的json.loads解析文件内容必须是合法 JSON文件根部可以是单个对象也可以是对象数组两种形态都会被自动识别处理多个文件会被拼接成最终表text_column、title_column配置会应用到每个加载对象的属性上。JSON LinesJSONL以.jsonl结尾的文件每行一个独立 JSON 对象。空行与纯空白行会被跳过。Parquet以.parquet结尾的列式文件作为结构化行加载与 CSV/JSON 一样支持text_column、title_column与id_column配置。MarkItDown通过 MarkItDown 转换器读取文档如 PDF、Office 文档等非纯文本格式即input.type: markitdown。在 input_reader_factory.py 中对应MarkItDownFileReader。仓库配套的输入解析测试用例可进一步参考 tests/unit/indexing/input 目录下的test_csv_loader.py、test_json_loader.py、test_jsonl_loader.py、test_parquet_loader.py、test_markitdown_loader.py、test_text_loader.py等文件。Bring-Your-Own DataFrame跳过解析直接喂表如果你已有现成的 pandas DataFrame或数据存放在仓库内置格式之外的位置可以直接调用索引 API 传入自己的表完全绕过上述输入加载/解析环节。GraphRAG 的索引 APIbuild_index提供了input_documents: pd.DataFrame | None参数见 api/index.py其文档字符串明确说明其作用是 Override document loading and parsing and supply your own dataframe of documents to index覆盖文档加载与解析流程提供自定义文档表用于索引。前提约束你传入的 DataFrame 必须符合前文给出的documents统一 Schemaid、text、title、creation_date、raw_data。满足该前提下后面所有分块行为与走内置读取器完全一致。元数据前置Field Prepending的背景动机为什么要在分块时把元数据反复复制进每个文本块文档给出了一个典型场景你在索引一批新闻文章每篇文章以标题headline和作者author开头然后是正文。默认分块按配置的chunk_size从头到尾均匀切分——即前n个 token 进入第一个文本块再下一个n个进入第二块依此类推。这意味着位于文档开头的导语/头部信息不会出现在每个文本块中只会留在第一个块里。当后续阶段对这些文本块进行实体提取或摘要时中间与末尾的块就会缺少这篇文章是谁、标题是什么这类全局共享信息。为避免该问题GraphRAG 提供配置把这类重复性内容复制到每个文本单元中。分块配置与字段前置配置input 配置块声明可引用字段如上文所述结构化输入源的字段保存在raw_data中可在分块前置阶段按字段名直接引用。id、title、text、creation_date这些标准字段即使是非结构化文本也可以被引用。chunking 配置块控制前置行为prepend_metadata用于指示导入器把选中的文档字段复制到每个文本块的开头值以key: value的键值对形式写入每对一行若未配置该设置元数据在切分时默认被忽略对于结构化输入raw_data对象内含源文件中出现的其余字段对 CSV 而言即除被用作text/title/id之外的所有列。配置模型ChunkingConfig中该字段被定义为prepend_metadata: list[str] | None见 chunking_config.py即一组待前置字段名。从实现看create_base_text_unitsworkflow 会把config.chunking.prepend_metadata传入分块函数并在切分前对每个文档执行document.collect(prepend_metadata)收集字段再用add_metadata(metadata..., line_delimiter.\n)将其序列化为键值对文本写入每个块见 create_base_text_units.py 与 transformers.py。关于小节命名的一致性提醒settings.yaml中承载分块配置的顶级小节为chunking对应GraphRagConfig.chunking见 graph_rag_config.py由graphrag init生成的 settings 文件也使用该键名本文转录的早期版本文档示例中曾写作chunks请以你所使用版本实际生成的 settings.yaml 为准。一个最小示例给每个块带上title与tag假设源文件software.csv内容如下text,title,tag My first program,Hello World,tutorial An early space shooter game,Space Invaders,arcade配置chunking: prepend_metadata: [title,tag]则加载出的documentsDataFrame 为idtitletextcreation_dateraw_data(generated from text)Hello WorldMy first program(create date of software.csv){ text: My first program, title: Hello World, tag: tutorial }(generated from text)Space InvadersAn early space shooter game(create date of software.csv){ text: An early space shooter game, title: Space Invaders, tag: arcade }分块时title与tag两个字段会被拼接到每个文本块起始处使模型始终能看到该软件条目的标题与分类标签。三种格式的分块完整示例下文三个示例使用单词数模拟 token 以便演示。需要说明的是LLM 的 token 并不等同于单词实际切分以所选编码模型的 token 数为准token 概念与计数方式参考。纯文本文件前置文件名title两篇独立新闻文本文件。文件US to lift most federal COVID-19 vaccine mandates.txt内容WASHINGTON (AP) The Biden administration will end most of the last remaining federal COVID-19 vaccine requirements next week when the national public health emergency for the coronavirus ends, the White House said Monday. Vaccine requirements for federal workers and federal contractors, as well as foreign air travelers to the U.S., will end May 11. The government is also beginning the process of lifting shot requirements for Head Start educators, healthcare workers, and noncitizens at U.S. land borders. The requirements are among the last vestiges of some of the more coercive measures taken by the federal government to promote vaccination as the deadly virus raged, and their end marks the latest display of how President Joe Bidens administration is moving to treat COVID-19 as a routine, endemic illness. While I believe that these vaccine mandates had a tremendous beneficial impact, we are now at a point where we think that it makes a lot of sense to pull these requirements down, White House COVID-19 coordinator Dr. Ashish Jha told The Associated Press on Monday.文件NY lawmakers begin debating budget 1 month after due date.txt内容ALBANY, N.Y. (AP) New York lawmakers began voting Monday on a $229 billion state budget due a month ago that would raise the minimum wage, crack down on illicit pot shops and ban gas stoves and furnaces in new buildings. Negotiations among Gov. Kathy Hochul and her fellow Democrats in control of the Legislature dragged on past the April 1 budget deadline, largely because of disagreements over changes to the bail law and other policy proposals included in the spending plan. Floor debates on some budget bills began Monday. State Senate Majority Leader Andrea Stewart-Cousins said she expected voting to be wrapped up Tuesday for a budget she said contains significant wins for New Yorkers. I would have liked to have done this sooner. I think we would all agree to that, Cousins told reporters before voting began. This has been a very policy-laden budget and a lot of the policies had to parsed through. Hochul was able to push through a change to the bail law that will eliminate the standard that requires judges to prescribe the least restrictive means to ensure defendants return to court. Hochul said judges needed the extra discretion. Some liberal lawmakers argued that it would undercut the sweeping bail reforms approved in 2019 and result in more people with low incomes and people of color in pretrial detention. Here are some other policy provisions that will be included in the budget, according to state officials. The minimum wage would be raised to $17 in New York City and some of its suburbs and $16 in the rest of the state by 2026. Thats up from $15 in the city and $14.20 upstate.settings.yamlinput: type: text metadata: [title] chunking: size: 100 overlap: 0 prepend_metadata: trueDocuments DataFrameidtitletextcreation_datemetadata(generated from text)US to lift most federal COVID-19 vaccine mandates.txt(full content of text file)(create date of article txt file){ title: US to lift most federal COVID-19 vaccine mandates.txt }(generated from text)NY lawmakers begin debating budget 1 month after due date.txt(full content of text file)(create date of article txt file){ title: NY lawmakers begin debating budget 1 month after due date.txt }Raw Text Chunkscontentlengthtitle: US to lift most federal COVID-19 vaccine mandates.txtWASHINGTON (AP) The Biden administration will end most of the last remaining federal COVID-19 vaccine requirements next week when the national public health emergency for the coronavirus ends, the White House said Monday. Vaccine requirements for federal workers and federal contractors, as well as foreign air travelers to the U.S., will end May 11. The government is also beginning the process of lifting shot requirements for Head Start educators, healthcare workers, and noncitizens at U.S. land borders. The requirements are among the last vestiges of some of the more coercive measures taken by the federal government to promote vaccination as109title: US to lift most federal COVID-19 vaccine mandates.txtthe deadly virus raged, and their end marks the latest display of how President Joe Bidens administration is moving to treat COVID-19 as a routine, endemic illness. While I believe that these vaccine mandates had a tremendous beneficial impact, we are now at a point where we think that it makes a lot of sense to pull these requirements down, White House COVID-19 coordinator Dr. Ashish Jha told The Associated Press on Monday.82title: NY lawmakers begin debating budget 1 month after due date.txtALBANY, N.Y. (AP) New York lawmakers began voting Monday on a $229 billion state budget due a month ago that would raise the minimum wage, crack down on illicit pot shops and ban gas stoves and furnaces in new buildings. Negotiations among Gov. Kathy Hochul and her fellow Democrats in control of the Legislature dragged on past the April 1 budget deadline, largely because of disagreements over changes to the bail law and other policy proposals included in the spending plan. Floor debates on some budget bills began Monday. State Senate Majority Leader Andrea Stewart-Cousins said she expected voting to111title: NY lawmakers begin debating budget 1 month after due date.txtbe wrapped up Tuesday for a budget she said contains significant wins for New Yorkers. I would have liked to have done this sooner. I think we would all agree to that, Cousins told reporters before voting began. This has been a very policy-laden budget and a lot of the policies had to parsed through. Hochul was able to push through a change to the bail law that will eliminate the standard that requires judges to prescribe the least restrictive means to ensure defendants return to court. Hochul said judges needed the extra discretion. Some liberal lawmakers argued that it111title: NY lawmakers begin debating budget 1 month after due date.txtwould undercut the sweeping bail reforms approved in 2019 and result in more people with low incomes and people of color in pretrial detention. Here are some other policy provisions that will be included in the budget, according to state officials. The minimum wage would be raised to $17 in New York City and some of its suburbs and $16 in the rest of the state by 2026. Thats up from $15 in the city and $14.20 upstate.89该示例要点两篇输入文档共被切分成5 个输出文本块第一篇 2 块第二篇 3 块每篇文档的文件名title被前置到该文档所有文本块的开头前置的 title 不计入 chunk sizelength只统计正文部分的词数每篇文档的最后一个文本块通常小于 chunk size因为它只包含剩余的最后若干 token。CSV借助text_column/title_column做列映射同一篇文章作为 CSV 的两行注意为便于阅读正文未按实际 CSV 规范转义。文件articles.csv内容headline,article US to lift most federal COVID-19 vaccine mandates,WASHINGTON (AP) The Biden administration will end most of the last remaining federal COVID-19 vaccine requirements next week when the national public health emergency for the coronavirus ends, the White House said Monday. Vaccine requirements for federal workers and federal contractors, as well as foreign air travelers to the U.S., will end May 11. The government is also beginning the process of lifting shot requirements for Head Start educators, healthcare workers, and noncitizens at U.S. land borders. The requirements are among the last vestiges of some of the more coercive measures taken by the federal government to promote vaccination as the deadly virus raged, and their end marks the latest display of how President Joe Bidens administration is moving to treat COVID-19 as a routine, endemic illness. While I believe that these vaccine mandates had a tremendous beneficial impact, we are now at a point where we think that it makes a lot of sense to pull these requirements down, White House COVID-19 coordinator Dr. Ashish Jha told The Associated Press on Monday. NY lawmakers begin debating budget 1 month after due date,ALBANY, N.Y. (AP) New York lawmakers began voting Monday on a $229 billion state budget due a month ago that would raise the minimum wage, crack down on illicit pot shops and ban gas stoves and furnaces in new buildings. Negotiations among Gov. Kathy Hochul and her fellow Democrats in control of the Legislature dragged on past the April 1 budget deadline, largely because of disagreements over changes to the bail law and other policy proposals included in the spending plan. Floor debates on some budget bills began Monday. State Senate Majority Leader Andrea Stewart-Cousins said she expected voting to be wrapped up Tuesday for a budget she said contains significant wins for New Yorkers. I would have liked to have done this sooner. I think we would all agree to that, Cousins told reporters before voting began. This has been a very policy-laden budget and a lot of the policies had to parsed through. Hochul was able to push through a change to the bail law that will eliminate the standard that requires judges to prescribe the least restrictive means to ensure defendants return to court. Hochul said judges needed the extra discretion. Some liberal lawmakers argued that it would undercut the sweeping bail reforms approved in 2019 and result in more people with low incomes and people of color in pretrial detention. Here are some other policy provisions that will be included in the budget, according to state officials. The minimum wage would be raised to $17 in New York City and some of its suburbs and $16 in the rest of the state by 2026. Thats up from $15 in the city and $14.20 upstate.CSV 场景可把headline配置为title_column、article配置为text_column原理与下文 JSON 示例一致这里不再赘述切分细节。JSON列映射 overlap 重叠演示本示例为两篇文章各建一个 JSON 文件只配置读取字段不向前置块追加元数据。文件article1.json内容{ headline: US to lift most federal COVID-19 vaccine mandates, content: WASHINGTON (AP) The Biden administration will end most of the last remaining federal COVID-19 vaccine requirements next week when the national public health emergency for the coronavirus ends, the White House said Monday. Vaccine requirements for federal workers and federal contractors, as well as foreign air travelers to the U.S., will end May 11. The government is also beginning the process of lifting shot requirements for Head Start educators, healthcare workers, and noncitizens at U.S. land borders. The requirements are among the last vestiges of some of the more coercive measures taken by the federal government to promote vaccination as the deadly virus raged, and their end marks the latest display of how President Joe Bidens administration is moving to treat COVID-19 as a routine, endemic illness. \While I believe that these vaccine mandates had a tremendous beneficial impact, we are now at a point where we think that it makes a lot of sense to pull these requirements down,\ White House COVID-19 coordinator Dr. Ashish Jha told The Associated Press on Monday. }文件article2.json内容{ headline: NY lawmakers begin debating budget 1 month after due date, content: ALBANY, N.Y. (AP) New York lawmakers began voting Monday on a $229 billion state budget due a month ago that would raise the minimum wage, crack down on illicit pot shops and ban gas stoves and furnaces in new buildings. Negotiations among Gov. Kathy Hochul and her fellow Democrats in control of the Legislature dragged on past the April 1 budget deadline, largely because of disagreements over changes to the bail law and other policy proposals included in the spending plan. Floor debates on some budget bills began Monday. State Senate Majority Leader Andrea Stewart-Cousins said she expected voting to be wrapped up Tuesday for a budget she said contains \significant wins\ for New Yorkers. \I would have liked to have done this sooner. I think we would all agree to that,\ Cousins told reporters before voting began. \This has been a very policy-laden budget and a lot of the policies had to parsed through.\ Hochul was able to push through a change to the bail law that will eliminate the standard that requires judges to prescribe the \least restrictive\ means to ensure defendants return to court. Hochul said judges needed the extra discretion. Some liberal lawmakers argued that it would undercut the sweeping bail reforms approved in 2019 and result in more people with low incomes and people of color in pretrial detention. Here are some other policy provisions that will be included in the budget, according to state officials. The minimum wage would be raised to $17 in New York City and some of its suburbs and $16 in the rest of the state by 2026. Thats up from $15 in the city and $14.20 upstate. }settings.yamlinput: type: json title_column: headline text_column: content chunking: size: 100 overlap: 10Documents DataFrameidtitletextcreation_datemetadata(generated from text)US to lift most federal COVID-19 vaccine mandates(article column content)(create date of article1.json){ }(generated from text)NY lawmakers begin debating budget 1 month after due date(article column content)(create date of article2.json){ }Raw Text ChunkscontentlengthWASHINGTON (AP) The Biden administration will end most of the last remaining federal COVID-19 vaccine requirements next week when the national public health emergency for the coronavirus ends, the White House said Monday. Vaccine requirements for federal workers and federal contractors, as well as foreign air travelers to the U.S., will end May 11. The government is also beginning the process of lifting shot requirements for Head Start educators, healthcare workers, and noncitizens at U.S. land borders. The requirements are among the last vestiges of some of the more coercive measures taken by the federal government to promote vaccination as100measures taken by the federal government to promote vaccination as the deadly virus raged, and their end marks the latest display of how President Joe Bidens administration is moving to treat COVID-19 as a routine, endemic illness. While I believe that these vaccine mandates had a tremendous beneficial impact, we are now at a point where we think that it makes a lot of sense to pull these requirements down, White House COVID-19 coordinator Dr. Ashish Jha told The Associated Press on Monday.83ALBANY, N.Y. (AP) New York lawmakers began voting Monday on a $229 billion state budget due a month ago that would raise the minimum wage, crack down on illicit pot shops and ban gas stoves and furnaces in new buildings. Negotiations among Gov. Kathy Hochul and her fellow Democrats in control of the Legislature dragged on past the April 1 budget deadline, largely because of disagreements over changes to the bail law and other policy proposals included in the spending plan. Floor debates on some budget bills began Monday. State Senate Majority Leader Andrea Stewart-Cousins said she expected voting to100Senate Majority Leader Andrea Stewart-Cousins said she expected voting to be wrapped up Tuesday for a budget she said contains significant wins for New Yorkers. I would have liked to have done this sooner. I think we would all agree to that, Cousins told reporters before voting began. This has been a very policy-laden budget and a lot of the policies had to parsed through. Hochul was able to push through a change to the bail law that will eliminate the standard that requires judges to prescribe the least restrictive means to ensure defendants return to court. Hochul said judges100means to ensure defendants return to court. Hochul said judges needed the extra discretion. Some liberal lawmakers argued that it would undercut the sweeping bail reforms approved in 2019 and result in more people with low incomes and people of color in pretrial detention. Here are some other policy provisions that will be included in the budget, according to state officials. The minimum wage would be raised to $17 in New York City and some of its suburbs and $16 in the rest of the state by 2026. Thats up from $15 in the city and $14.20 upstate.98该示例要点两篇 JSON 文档同样被切分成 5 个输出文本块本次没有做元数据前置因此每个块的内容紧贴配置的 chunk size100仅每篇文档的最后一个块不足 100配置了overlap: 10可以观察到相邻文本块之间共享了末尾 10 个 token——例如块 1 结尾的 as 与块 2 开头重叠。overlap 用于降低因硬切分而把语义割裂在块边界处的风险。分块参数chunk_size 与 overlap 的取舍chunking.sizechunk_size按 token 计数的块大小。分块是必须的因为文档内容往往超过所用语言模型的上下文窗口。默认流程中该值默认配置为 1200 token见 docs/index/default_dataflow.md更大的块会带来更低的提取保真度与更不具参考价值的上下文文本但通常能显著加快处理速度需要按语料与成本权衡。chunking.overlap相邻文本块之间共享的 token 数用于缓解切点处的语义断裂。分块相关的单元测试可参考 tests/unit/chunking如test_chunker.py、test_prepend_metadata.pyworkflow 层面的测试在 test_create_base_text_units.py。本文配套可继续深入阅读的仓库资料输入 Schema 的运行时载体 text_document.py各格式 Reader 实现packages/graphrag-input/graphrag_inputtext.py、csv.py、json.py、jsonl.py、parquet.py、markitdown.pyReader 注册与工厂分发input_reader_factory.py输入加载与分块 workflowload_input_documents.py 与 create_base_text_units.py分块元数据变换工具transformers.py自定义组件架构说明docs/index/architecture.md默认数据流整体概览docs/index/default_dataflow.md完整配置总览docs/index/overview.md 与 docs/config/overview.md最后再次强调一条贯穿全文的原则文档 ID 由文本内容哈希生成SHA-512目的是跨运行稳定当使用 CSV/JSON 等结构化格式时请优先保留或显式指定稳定id列这直接影响增量索引与溯源在多次重建间的可靠性。【免费下载链接】graphragA modular graph-based Retrieval-Augmented Generation (RAG) system项目地址: https://gitcode.com/GitHub_Trending/gr/graphrag创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考 GraphRAG 输入与分块指南：支持的数据格式、统一 documents Schema 与元数据前置（Field Prepending）
GraphRAG 输入与分块指南支持的数据格式、统一 documents Schema 与元数据前置Field Prepending【免费下载链接】graphragA modular graph-based Retrieval-Augmented Generation (RAG) system项目地址: https://gitcode.com/GitHub_Trending/gr/graphrag导读输入数据如何被读取、标准化和切分直接决定了 GraphRAG 图索引的质量与可追溯性。本文以 docs/index/inputs.md 为主线系统讲解 GraphRAG 支持的多种输入格式纯文本、CSV、JSON、JSONL、Parquet、MarkItDown、统一的documentsDataFrame 列结构、自备 DataFrame 的 Bring-Your-Own 接入方式以及分块Chunking阶段通过prepend_metadata将文档元数据重复复制到每个文本块中的配置方法。读完你将能够为自己的语料选择最合适的输入格式、正确配置settings.yaml的input/chunking段、并让每个文本块携带文档级共享信息以提升下游实体提取与检索效果。输入在整个索引流水线中的位置在 GraphRAG 的默认索引流程中详见 docs/index/default_dataflow.md输入处理发生在最前端的Phase 1: Compose TextUnits原始文件先被读取为documents再按配置被切分为更小的TextUnit文本块后续的实体/关系提取、社区检测、社区报告等阶段全部基于这些文本块展开。Documents -- Chunk -- Text Units因此如何把各种形态的源文件变成一张结构统一的documents表是全流程的第一道关卡也是本文的主题。输入加载在索引流水线中由两个核心 workflow 完成load_input_documents负责将存储中的文件读入并解析成统一格式实现见 load_input_documents.pycreate_base_text_units负责把documents切分为text_units并将prepend_metadata配置作用于切分过程实现见 create_base_text_units.py。统一 Documents 模式Input Loading and Schema无论采用哪种输入格式所有文件都会在 GraphRAG 内部被加载为一个documentsDataFrame每个文档占一行并遵循如下共享列结构名称类型说明idstr文档 ID。由文本内容哈希生成保证跨多次运行结果稳定增量更新场景尤为关键。textstr文档完整文本内容。titlestr文档名称。部分格式允许通过配置指定使用哪一列作为标题。creation_datestr文档创建日期以 ISO8601 字符串表示。该值从源文件系统的时间属性中采集。raw_datadict结构化输入的原始源行/对象可选。可被分块阶段的元数据前置metadata prepending使用。该TextDocument数据结构的仓库实现位于 text_document.py其中TextDocument.get(field)支持标准字段id、title、text、creation_date直接取属性其余字段名则会到raw_data中按字段名甚至支持点号嵌套字段查找——这正是下文字段前置能引用结构化源数据任意列的实现基础creation_date由存储层通过storage.get_creation_date(path)从文件系统中采集见 text.py 中对纯文本文件的用法id默认使用 SHA-512 对文本内容进行哈希生成纯文本读取器直接调用gen_sha512_hash({text: text}, [text])见 text.py 与 hashing.py这也是文档强调ID 基于文本内容哈希以跨运行稳定的源码依据。流水线运行结束后最终的 documents 表会以 parquet 形式落盘其最终表结构可参考 docs/index/outputs.md 中关于 final documents 表的说明。输入加载链路可注入的 InputReader 架构所有输入格式的解析都统一收敛到同一个架构InputReader基类 InputReaderFactory工厂。理解这一点有助于你判断遇到不支持的格式怎么办。InputReader 抽象基类input_reader.py 定义了InputReader构造函数接收storage文件来源、file_pattern文件匹配正则与encoding默认utf-8它实现了异步迭代协议__aiter__/_iterate_files用self._storage.find(re.compile(self._file_pattern))找到所有匹配文件逐个调用抽象方法read_file(path)把单个文件解析成list[TextDocument]单个文件解析失败不会中断整体流程——代码会打印 warning 并跳过该文件继续处理具有较高的健壮性。InputReaderFactory 与注册机制input_reader_factory.py 中的create_input_reader(config, storage)依据config.type分发到对应的 Reader 实现input.type配置值Reader 实现默认文件匹配正则text默认TextFileReader.*\.txt$csvCSVFileReader.*\.csv$jsonJSONFileReader—jsonlJSONLinesFileReader—markitdownMarkItDownFileReader—parquetParquetFileReader—同时仓库还导出了register_input_reader(type, initializer)公共注册函数开发者可以把自己实现的 Reader 类按标准 Factory 模式注册进去见 input_reader_factory.py这部分即文档中Custom File Handling所述能力。配置模型说明输入相关配置由 graph_rag_config.py 中的input: InputConfig承载其字段type、encoding、file_pattern、id_column、title_column、text_column定义在 input_config.py并且InputConfig开启了extraallow以便自定义 Reader 增加私有字段。支持的输入格式详解Formats下面逐类说明每种格式的加载语义与可配置列映射。若你的数据形态不在其中文档建议两条路一是自行实现InputReader注册进工厂二是写脚本把数据先转换为以下某一种格式。纯文本Plain Text以.txt结尾的文本文件默认input.type即text。加载规则整文件内容作为text字段title始终为文件名每个文件产出一行TextDocumentraw_data为空None。从源码可以看到纯文本读取器把文件名作为 title、把文件内容哈希作为 id、从文件系统采集创建时间见 text.py。CSV逗号分隔以.csv结尾的文件加载语义采用csv.DictReader逐行解析csv.pyCSV 的每一行被当作一篇独立文档输入目录下若有多个 CSV 文件会被拼接成一个统一的documentsDataFrame可通过input配置块指定text_column与title_column让结构化内容中的指定列充当正文与标题若未配置title使用文件名、text_column默认取名为text的列若数据中存在id列则直接使用否则按前文规则基于文本内容哈希生成 ID。需要留意的是CSV 正文单元格内容通常需要做引号转义以符合 CSV 规范正文中含有换行或逗号的字段务必正确包裹。JSON以.json结尾、内容符合 JSON 规范 的结构化文件。加载语义使用 Python 标准库的json.loads解析文件内容必须是合法 JSON文件根部可以是单个对象也可以是对象数组两种形态都会被自动识别处理多个文件会被拼接成最终表text_column、title_column配置会应用到每个加载对象的属性上。JSON LinesJSONL以.jsonl结尾的文件每行一个独立 JSON 对象。空行与纯空白行会被跳过。Parquet以.parquet结尾的列式文件作为结构化行加载与 CSV/JSON 一样支持text_column、title_column与id_column配置。MarkItDown通过 MarkItDown 转换器读取文档如 PDF、Office 文档等非纯文本格式即input.type: markitdown。在 input_reader_factory.py 中对应MarkItDownFileReader。仓库配套的输入解析测试用例可进一步参考 tests/unit/indexing/input 目录下的test_csv_loader.py、test_json_loader.py、test_jsonl_loader.py、test_parquet_loader.py、test_markitdown_loader.py、test_text_loader.py等文件。Bring-Your-Own DataFrame跳过解析直接喂表如果你已有现成的 pandas DataFrame或数据存放在仓库内置格式之外的位置可以直接调用索引 API 传入自己的表完全绕过上述输入加载/解析环节。GraphRAG 的索引 APIbuild_index提供了input_documents: pd.DataFrame | None参数见 api/index.py其文档字符串明确说明其作用是 Override document loading and parsing and supply your own dataframe of documents to index覆盖文档加载与解析流程提供自定义文档表用于索引。前提约束你传入的 DataFrame 必须符合前文给出的documents统一 Schemaid、text、title、creation_date、raw_data。满足该前提下后面所有分块行为与走内置读取器完全一致。元数据前置Field Prepending的背景动机为什么要在分块时把元数据反复复制进每个文本块文档给出了一个典型场景你在索引一批新闻文章每篇文章以标题headline和作者author开头然后是正文。默认分块按配置的chunk_size从头到尾均匀切分——即前n个 token 进入第一个文本块再下一个n个进入第二块依此类推。这意味着位于文档开头的导语/头部信息不会出现在每个文本块中只会留在第一个块里。当后续阶段对这些文本块进行实体提取或摘要时中间与末尾的块就会缺少这篇文章是谁、标题是什么这类全局共享信息。为避免该问题GraphRAG 提供配置把这类重复性内容复制到每个文本单元中。分块配置与字段前置配置input 配置块声明可引用字段如上文所述结构化输入源的字段保存在raw_data中可在分块前置阶段按字段名直接引用。id、title、text、creation_date这些标准字段即使是非结构化文本也可以被引用。chunking 配置块控制前置行为prepend_metadata用于指示导入器把选中的文档字段复制到每个文本块的开头值以key: value的键值对形式写入每对一行若未配置该设置元数据在切分时默认被忽略对于结构化输入raw_data对象内含源文件中出现的其余字段对 CSV 而言即除被用作text/title/id之外的所有列。配置模型ChunkingConfig中该字段被定义为prepend_metadata: list[str] | None见 chunking_config.py即一组待前置字段名。从实现看create_base_text_unitsworkflow 会把config.chunking.prepend_metadata传入分块函数并在切分前对每个文档执行document.collect(prepend_metadata)收集字段再用add_metadata(metadata..., line_delimiter.\n)将其序列化为键值对文本写入每个块见 create_base_text_units.py 与 transformers.py。关于小节命名的一致性提醒settings.yaml中承载分块配置的顶级小节为chunking对应GraphRagConfig.chunking见 graph_rag_config.py由graphrag init生成的 settings 文件也使用该键名本文转录的早期版本文档示例中曾写作chunks请以你所使用版本实际生成的 settings.yaml 为准。一个最小示例给每个块带上title与tag假设源文件software.csv内容如下text,title,tag My first program,Hello World,tutorial An early space shooter game,Space Invaders,arcade配置chunking: prepend_metadata: [title,tag]则加载出的documentsDataFrame 为idtitletextcreation_dateraw_data(generated from text)Hello WorldMy first program(create date of software.csv){ text: My first program, title: Hello World, tag: tutorial }(generated from text)Space InvadersAn early space shooter game(create date of software.csv){ text: An early space shooter game, title: Space Invaders, tag: arcade }分块时title与tag两个字段会被拼接到每个文本块起始处使模型始终能看到该软件条目的标题与分类标签。三种格式的分块完整示例下文三个示例使用单词数模拟 token 以便演示。需要说明的是LLM 的 token 并不等同于单词实际切分以所选编码模型的 token 数为准token 概念与计数方式参考。纯文本文件前置文件名title两篇独立新闻文本文件。文件US to lift most federal COVID-19 vaccine mandates.txt内容WASHINGTON (AP) The Biden administration will end most of the last remaining federal COVID-19 vaccine requirements next week when the national public health emergency for the coronavirus ends, the White House said Monday. Vaccine requirements for federal workers and federal contractors, as well as foreign air travelers to the U.S., will end May 11. The government is also beginning the process of lifting shot requirements for Head Start educators, healthcare workers, and noncitizens at U.S. land borders. The requirements are among the last vestiges of some of the more coercive measures taken by the federal government to promote vaccination as the deadly virus raged, and their end marks the latest display of how President Joe Bidens administration is moving to treat COVID-19 as a routine, endemic illness. While I believe that these vaccine mandates had a tremendous beneficial impact, we are now at a point where we think that it makes a lot of sense to pull these requirements down, White House COVID-19 coordinator Dr. Ashish Jha told The Associated Press on Monday.文件NY lawmakers begin debating budget 1 month after due date.txt内容ALBANY, N.Y. (AP) New York lawmakers began voting Monday on a $229 billion state budget due a month ago that would raise the minimum wage, crack down on illicit pot shops and ban gas stoves and furnaces in new buildings. Negotiations among Gov. Kathy Hochul and her fellow Democrats in control of the Legislature dragged on past the April 1 budget deadline, largely because of disagreements over changes to the bail law and other policy proposals included in the spending plan. Floor debates on some budget bills began Monday. State Senate Majority Leader Andrea Stewart-Cousins said she expected voting to be wrapped up Tuesday for a budget she said contains significant wins for New Yorkers. I would have liked to have done this sooner. I think we would all agree to that, Cousins told reporters before voting began. This has been a very policy-laden budget and a lot of the policies had to parsed through. Hochul was able to push through a change to the bail law that will eliminate the standard that requires judges to prescribe the least restrictive means to ensure defendants return to court. Hochul said judges needed the extra discretion. Some liberal lawmakers argued that it would undercut the sweeping bail reforms approved in 2019 and result in more people with low incomes and people of color in pretrial detention. Here are some other policy provisions that will be included in the budget, according to state officials. The minimum wage would be raised to $17 in New York City and some of its suburbs and $16 in the rest of the state by 2026. Thats up from $15 in the city and $14.20 upstate.settings.yamlinput: type: text metadata: [title] chunking: size: 100 overlap: 0 prepend_metadata: trueDocuments DataFrameidtitletextcreation_datemetadata(generated from text)US to lift most federal COVID-19 vaccine mandates.txt(full content of text file)(create date of article txt file){ title: US to lift most federal COVID-19 vaccine mandates.txt }(generated from text)NY lawmakers begin debating budget 1 month after due date.txt(full content of text file)(create date of article txt file){ title: NY lawmakers begin debating budget 1 month after due date.txt }Raw Text Chunkscontentlengthtitle: US to lift most federal COVID-19 vaccine mandates.txtWASHINGTON (AP) The Biden administration will end most of the last remaining federal COVID-19 vaccine requirements next week when the national public health emergency for the coronavirus ends, the White House said Monday. Vaccine requirements for federal workers and federal contractors, as well as foreign air travelers to the U.S., will end May 11. The government is also beginning the process of lifting shot requirements for Head Start educators, healthcare workers, and noncitizens at U.S. land borders. The requirements are among the last vestiges of some of the more coercive measures taken by the federal government to promote vaccination as109title: US to lift most federal COVID-19 vaccine mandates.txtthe deadly virus raged, and their end marks the latest display of how President Joe Bidens administration is moving to treat COVID-19 as a routine, endemic illness. While I believe that these vaccine mandates had a tremendous beneficial impact, we are now at a point where we think that it makes a lot of sense to pull these requirements down, White House COVID-19 coordinator Dr. Ashish Jha told The Associated Press on Monday.82title: NY lawmakers begin debating budget 1 month after due date.txtALBANY, N.Y. (AP) New York lawmakers began voting Monday on a $229 billion state budget due a month ago that would raise the minimum wage, crack down on illicit pot shops and ban gas stoves and furnaces in new buildings. Negotiations among Gov. Kathy Hochul and her fellow Democrats in control of the Legislature dragged on past the April 1 budget deadline, largely because of disagreements over changes to the bail law and other policy proposals included in the spending plan. Floor debates on some budget bills began Monday. State Senate Majority Leader Andrea Stewart-Cousins said she expected voting to111title: NY lawmakers begin debating budget 1 month after due date.txtbe wrapped up Tuesday for a budget she said contains significant wins for New Yorkers. I would have liked to have done this sooner. I think we would all agree to that, Cousins told reporters before voting began. This has been a very policy-laden budget and a lot of the policies had to parsed through. Hochul was able to push through a change to the bail law that will eliminate the standard that requires judges to prescribe the least restrictive means to ensure defendants return to court. Hochul said judges needed the extra discretion. Some liberal lawmakers argued that it111title: NY lawmakers begin debating budget 1 month after due date.txtwould undercut the sweeping bail reforms approved in 2019 and result in more people with low incomes and people of color in pretrial detention. Here are some other policy provisions that will be included in the budget, according to state officials. The minimum wage would be raised to $17 in New York City and some of its suburbs and $16 in the rest of the state by 2026. Thats up from $15 in the city and $14.20 upstate.89该示例要点两篇输入文档共被切分成5 个输出文本块第一篇 2 块第二篇 3 块每篇文档的文件名title被前置到该文档所有文本块的开头前置的 title 不计入 chunk sizelength只统计正文部分的词数每篇文档的最后一个文本块通常小于 chunk size因为它只包含剩余的最后若干 token。CSV借助text_column/title_column做列映射同一篇文章作为 CSV 的两行注意为便于阅读正文未按实际 CSV 规范转义。文件articles.csv内容headline,article US to lift most federal COVID-19 vaccine mandates,WASHINGTON (AP) The Biden administration will end most of the last remaining federal COVID-19 vaccine requirements next week when the national public health emergency for the coronavirus ends, the White House said Monday. Vaccine requirements for federal workers and federal contractors, as well as foreign air travelers to the U.S., will end May 11. The government is also beginning the process of lifting shot requirements for Head Start educators, healthcare workers, and noncitizens at U.S. land borders. The requirements are among the last vestiges of some of the more coercive measures taken by the federal government to promote vaccination as the deadly virus raged, and their end marks the latest display of how President Joe Bidens administration is moving to treat COVID-19 as a routine, endemic illness. While I believe that these vaccine mandates had a tremendous beneficial impact, we are now at a point where we think that it makes a lot of sense to pull these requirements down, White House COVID-19 coordinator Dr. Ashish Jha told The Associated Press on Monday. NY lawmakers begin debating budget 1 month after due date,ALBANY, N.Y. (AP) New York lawmakers began voting Monday on a $229 billion state budget due a month ago that would raise the minimum wage, crack down on illicit pot shops and ban gas stoves and furnaces in new buildings. Negotiations among Gov. Kathy Hochul and her fellow Democrats in control of the Legislature dragged on past the April 1 budget deadline, largely because of disagreements over changes to the bail law and other policy proposals included in the spending plan. Floor debates on some budget bills began Monday. State Senate Majority Leader Andrea Stewart-Cousins said she expected voting to be wrapped up Tuesday for a budget she said contains significant wins for New Yorkers. I would have liked to have done this sooner. I think we would all agree to that, Cousins told reporters before voting began. This has been a very policy-laden budget and a lot of the policies had to parsed through. Hochul was able to push through a change to the bail law that will eliminate the standard that requires judges to prescribe the least restrictive means to ensure defendants return to court. Hochul said judges needed the extra discretion. Some liberal lawmakers argued that it would undercut the sweeping bail reforms approved in 2019 and result in more people with low incomes and people of color in pretrial detention. Here are some other policy provisions that will be included in the budget, according to state officials. The minimum wage would be raised to $17 in New York City and some of its suburbs and $16 in the rest of the state by 2026. Thats up from $15 in the city and $14.20 upstate.CSV 场景可把headline配置为title_column、article配置为text_column原理与下文 JSON 示例一致这里不再赘述切分细节。JSON列映射 overlap 重叠演示本示例为两篇文章各建一个 JSON 文件只配置读取字段不向前置块追加元数据。文件article1.json内容{ headline: US to lift most federal COVID-19 vaccine mandates, content: WASHINGTON (AP) The Biden administration will end most of the last remaining federal COVID-19 vaccine requirements next week when the national public health emergency for the coronavirus ends, the White House said Monday. Vaccine requirements for federal workers and federal contractors, as well as foreign air travelers to the U.S., will end May 11. The government is also beginning the process of lifting shot requirements for Head Start educators, healthcare workers, and noncitizens at U.S. land borders. The requirements are among the last vestiges of some of the more coercive measures taken by the federal government to promote vaccination as the deadly virus raged, and their end marks the latest display of how President Joe Bidens administration is moving to treat COVID-19 as a routine, endemic illness. \While I believe that these vaccine mandates had a tremendous beneficial impact, we are now at a point where we think that it makes a lot of sense to pull these requirements down,\ White House COVID-19 coordinator Dr. Ashish Jha told The Associated Press on Monday. }文件article2.json内容{ headline: NY lawmakers begin debating budget 1 month after due date, content: ALBANY, N.Y. (AP) New York lawmakers began voting Monday on a $229 billion state budget due a month ago that would raise the minimum wage, crack down on illicit pot shops and ban gas stoves and furnaces in new buildings. Negotiations among Gov. Kathy Hochul and her fellow Democrats in control of the Legislature dragged on past the April 1 budget deadline, largely because of disagreements over changes to the bail law and other policy proposals included in the spending plan. Floor debates on some budget bills began Monday. State Senate Majority Leader Andrea Stewart-Cousins said she expected voting to be wrapped up Tuesday for a budget she said contains \significant wins\ for New Yorkers. \I would have liked to have done this sooner. I think we would all agree to that,\ Cousins told reporters before voting began. \This has been a very policy-laden budget and a lot of the policies had to parsed through.\ Hochul was able to push through a change to the bail law that will eliminate the standard that requires judges to prescribe the \least restrictive\ means to ensure defendants return to court. Hochul said judges needed the extra discretion. Some liberal lawmakers argued that it would undercut the sweeping bail reforms approved in 2019 and result in more people with low incomes and people of color in pretrial detention. Here are some other policy provisions that will be included in the budget, according to state officials. The minimum wage would be raised to $17 in New York City and some of its suburbs and $16 in the rest of the state by 2026. Thats up from $15 in the city and $14.20 upstate. }settings.yamlinput: type: json title_column: headline text_column: content chunking: size: 100 overlap: 10Documents DataFrameidtitletextcreation_datemetadata(generated from text)US to lift most federal COVID-19 vaccine mandates(article column content)(create date of article1.json){ }(generated from text)NY lawmakers begin debating budget 1 month after due date(article column content)(create date of article2.json){ }Raw Text ChunkscontentlengthWASHINGTON (AP) The Biden administration will end most of the last remaining federal COVID-19 vaccine requirements next week when the national public health emergency for the coronavirus ends, the White House said Monday. Vaccine requirements for federal workers and federal contractors, as well as foreign air travelers to the U.S., will end May 11. The government is also beginning the process of lifting shot requirements for Head Start educators, healthcare workers, and noncitizens at U.S. land borders. The requirements are among the last vestiges of some of the more coercive measures taken by the federal government to promote vaccination as100measures taken by the federal government to promote vaccination as the deadly virus raged, and their end marks the latest display of how President Joe Bidens administration is moving to treat COVID-19 as a routine, endemic illness. While I believe that these vaccine mandates had a tremendous beneficial impact, we are now at a point where we think that it makes a lot of sense to pull these requirements down, White House COVID-19 coordinator Dr. Ashish Jha told The Associated Press on Monday.83ALBANY, N.Y. (AP) New York lawmakers began voting Monday on a $229 billion state budget due a month ago that would raise the minimum wage, crack down on illicit pot shops and ban gas stoves and furnaces in new buildings. Negotiations among Gov. Kathy Hochul and her fellow Democrats in control of the Legislature dragged on past the April 1 budget deadline, largely because of disagreements over changes to the bail law and other policy proposals included in the spending plan. Floor debates on some budget bills began Monday. State Senate Majority Leader Andrea Stewart-Cousins said she expected voting to100Senate Majority Leader Andrea Stewart-Cousins said she expected voting to be wrapped up Tuesday for a budget she said contains significant wins for New Yorkers. I would have liked to have done this sooner. I think we would all agree to that, Cousins told reporters before voting began. This has been a very policy-laden budget and a lot of the policies had to parsed through. Hochul was able to push through a change to the bail law that will eliminate the standard that requires judges to prescribe the least restrictive means to ensure defendants return to court. Hochul said judges100means to ensure defendants return to court. Hochul said judges needed the extra discretion. Some liberal lawmakers argued that it would undercut the sweeping bail reforms approved in 2019 and result in more people with low incomes and people of color in pretrial detention. Here are some other policy provisions that will be included in the budget, according to state officials. The minimum wage would be raised to $17 in New York City and some of its suburbs and $16 in the rest of the state by 2026. Thats up from $15 in the city and $14.20 upstate.98该示例要点两篇 JSON 文档同样被切分成 5 个输出文本块本次没有做元数据前置因此每个块的内容紧贴配置的 chunk size100仅每篇文档的最后一个块不足 100配置了overlap: 10可以观察到相邻文本块之间共享了末尾 10 个 token——例如块 1 结尾的 as 与块 2 开头重叠。overlap 用于降低因硬切分而把语义割裂在块边界处的风险。分块参数chunk_size 与 overlap 的取舍chunking.sizechunk_size按 token 计数的块大小。分块是必须的因为文档内容往往超过所用语言模型的上下文窗口。默认流程中该值默认配置为 1200 token见 docs/index/default_dataflow.md更大的块会带来更低的提取保真度与更不具参考价值的上下文文本但通常能显著加快处理速度需要按语料与成本权衡。chunking.overlap相邻文本块之间共享的 token 数用于缓解切点处的语义断裂。分块相关的单元测试可参考 tests/unit/chunking如test_chunker.py、test_prepend_metadata.pyworkflow 层面的测试在 test_create_base_text_units.py。本文配套可继续深入阅读的仓库资料输入 Schema 的运行时载体 text_document.py各格式 Reader 实现packages/graphrag-input/graphrag_inputtext.py、csv.py、json.py、jsonl.py、parquet.py、markitdown.pyReader 注册与工厂分发input_reader_factory.py输入加载与分块 workflowload_input_documents.py 与 create_base_text_units.py分块元数据变换工具transformers.py自定义组件架构说明docs/index/architecture.md默认数据流整体概览docs/index/default_dataflow.md完整配置总览docs/index/overview.md 与 docs/config/overview.md最后再次强调一条贯穿全文的原则文档 ID 由文本内容哈希生成SHA-512目的是跨运行稳定当使用 CSV/JSON 等结构化格式时请优先保留或显式指定稳定id列这直接影响增量索引与溯源在多次重建间的可靠性。【免费下载链接】graphragA modular graph-based Retrieval-Augmented Generation (RAG) system项目地址: https://gitcode.com/GitHub_Trending/gr/graphrag创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考