Package com.hw.langchain.text.splitter

Class TextSplitter

java.lang.Object

com.hw.langchain.text.splitter.TextSplitter

All Implemented Interfaces:

BaseDocumentTransformer

Direct Known Subclasses:

CharacterTextSplitter, RecursiveCharacterTextSplitter

public abstract class TextSplitter
extends Object
implements BaseDocumentTransformer

Interface for splitting text into chunks.

Author:

HamaWhite

Field Summary

Fields

Modifier and Type	Field	Description
protected boolean	addStartIndex	If `true`, includes chunk's start index in metadata
protected int	chunkOverlap	Overlap in characters between chunks.
protected int	chunkSize	Maximum size of chunks to return.
protected boolean	keepSeparator	Whether or not to keep the separator in the chunks.
protected Function<String,Integer>	lengthFunction	Function that measures the length of given chunks.

Constructor Summary

Constructors

Constructor	Description
TextSplitter()

Method Summary

Modifier and Type	Method	Description
List<Document>	createDocuments(List<String> texts, List<Map<String,Object>> metadatas)	Create documents from a list of texts.
protected List<String>	mergeSplits(List<String> splits, String separator)	We now want to combine these smaller pieces into medium size chunks to send to the LLM.
List<Document>	splitDocuments(List<Document> documents)	Split documents.
abstract List<String>	splitText(String text)	Split text into multiple components.
List<Document>	transformDocuments(List<Document> documents, Map<String,Object> kwargs)	Transform sequence of documents by splitting them.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

chunkSize

protected int chunkSize

Maximum size of chunks to return.

chunkOverlap

protected int chunkOverlap

Overlap in characters between chunks.

lengthFunction

protected Function<String,Integer> lengthFunction

Function that measures the length of given chunks.

keepSeparator

protected boolean keepSeparator

Whether or not to keep the separator in the chunks.

addStartIndex

protected boolean addStartIndex

If `true`, includes chunk's start index in metadata

Constructor Details

TextSplitter

public TextSplitter()

Method Details

splitText

public abstract List<String> splitText(String text)

Split text into multiple components.

createDocuments

public List<Document> createDocuments(List<String> texts,
 List<Map<String,Object>> metadatas)

Create documents from a list of texts.

splitDocuments

public List<Document> splitDocuments(List<Document> documents)

Split documents.

mergeSplits

protected List<String> mergeSplits(List<String> splits,
 String separator)

We now want to combine these smaller pieces into medium size chunks to send to the LLM.

transformDocuments

public List<Document> transformDocuments(List<Document> documents,
 Map<String,Object> kwargs)

Transform sequence of documents by splitting them.

Specified by:

transformDocuments in interface BaseDocumentTransformer