通过Analyzer进行分词Anailzer的组成elasticsearch 的内置分词器Anailzer的组成character filters : 针对原始文本处理例如去除htmltokenizer 按照规切分为单词token filter 将切分的单词进行加工小写删除 stopwords 增加同义词elasticsearch 的内置分词器Standard Analyzer - 默认分词器按词切分小写处理Simple Analyzer - 按照非字母切分符号被过滤小写处理Stop Analyzer - 小写处理停用词过滤the aisWhitespace Analyzer - 按照空格切分不转小写Keyword Analyzer - 不分词直接将输入当作输出Patter Analyzer - 正则表达式默认\W非自负分隔Language - 提供了30多种 常见语言的分词器Customer Analyzer - 自定义分词器ICU Analyzer - 需要安装plugin Elasticsearch-plugin install analysis-icu查看不同的analyzer的效果standardGET _analyze { analyzer: standard, text: 2 running Quick brown-foxes leap over lazy dogs in the summer evening. }simpeGET _analyze { analyzer: simple, text: 2 running Quick brown-foxes leap over lazy dogs in the summer evening. }stopGET _analyze { analyzer: stop, text: 2 running Quick brown-foxes leap over lazy dogs in the summer evening. }whitespaceGET _analyze { analyzer: whitespace, text: 2 running Quick brown-foxes leap over lazy dogs in the summer evening. }keywordGET _analyze { analyzer: keyword, text: 2 running Quick brown-foxes leap over lazy dogs in the summer evening. }patternGET _analyze { analyzer: pattern, text: 2 running Quick brown-foxes leap over lazy dogs in the summer evening. }englishGET _analyze { analyzer: english, text: 2 running Quick brown-foxes leap over lazy dogs in the summer evening. }icu_analyzerPOST _analyze { analyzer: icu_analyzer, text: 他说的确实在理” }standardPOST _analyze { analyzer: standard, text: 他说的确实在理” }更多的中文分词器IK支持自定义词库支持热更新分词字典https://github.com/medcl/elasticsearch-analysis-ikTHULC清华大学自然语言处理和社会人文计算实验室对一套中文分词器https://github.com/microbun/elasticsearch-thulac-plugin
通过Analyzer进行分词
发布时间：2026/7/28 13:51:23
通过Analyzer进行分词Anailzer的组成elasticsearch 的内置分词器Anailzer的组成character filters : 针对原始文本处理例如去除htmltokenizer 按照规切分为单词token filter 将切分的单词进行加工小写删除 stopwords 增加同义词elasticsearch 的内置分词器Standard Analyzer - 默认分词器按词切分小写处理Simple Analyzer - 按照非字母切分符号被过滤小写处理Stop Analyzer - 小写处理停用词过滤the aisWhitespace Analyzer - 按照空格切分不转小写Keyword Analyzer - 不分词直接将输入当作输出Patter Analyzer - 正则表达式默认\W非自负分隔Language - 提供了30多种 常见语言的分词器Customer Analyzer - 自定义分词器ICU Analyzer - 需要安装plugin Elasticsearch-plugin install analysis-icu查看不同的analyzer的效果standardGET _analyze { analyzer: standard, text: 2 running Quick brown-foxes leap over lazy dogs in the summer evening. }simpeGET _analyze { analyzer: simple, text: 2 running Quick brown-foxes leap over lazy dogs in the summer evening. }stopGET _analyze { analyzer: stop, text: 2 running Quick brown-foxes leap over lazy dogs in the summer evening. }whitespaceGET _analyze { analyzer: whitespace, text: 2 running Quick brown-foxes leap over lazy dogs in the summer evening. }keywordGET _analyze { analyzer: keyword, text: 2 running Quick brown-foxes leap over lazy dogs in the summer evening. }patternGET _analyze { analyzer: pattern, text: 2 running Quick brown-foxes leap over lazy dogs in the summer evening. }englishGET _analyze { analyzer: english, text: 2 running Quick brown-foxes leap over lazy dogs in the summer evening. }icu_analyzerPOST _analyze { analyzer: icu_analyzer, text: 他说的确实在理” }standardPOST _analyze { analyzer: standard, text: 他说的确实在理” }更多的中文分词器IK支持自定义词库支持热更新分词字典https://github.com/medcl/elasticsearch-analysis-ikTHULC清华大学自然语言处理和社会人文计算实验室对一套中文分词器https://github.com/microbun/elasticsearch-thulac-plugin
)
