Processors API¶

Purpose¶

The processors module provides the text-processing, natural-language processing, document parsing, and geometry-aware PDF extraction services used by the Foo application after document ingestion and before analysis, tokenization, embedding, or persistence.

The module supports:

Text normalization and cleanup
HTML, XML, Markdown, image-reference, and encoding removal
Sentence, paragraph, page, word, and token segmentation
Vocabulary and frequency-distribution generation
TF-IDF vector creation
Sentence-transformer encoding and semantic search
NLTK tokenization, stemming, lemmatization, part-of-speech tagging, and named-entity recognition
Microsoft Word document processing
Geometry-aware PDF text extraction
Repeated PDF header and footer detection
PDF artifact cleanup, spacing repair, and hyphenation reconstruction
PDF table extraction and export to CSV or Excel
Centralized validation and exception logging through boogr.Error and boogr.Logger

Module Constants¶

The module defines reusable character and pattern collections used by the processing classes:

Constant	Purpose
`DELIMITERS`	Sentence and clause delimiters used during text segmentation and cleanup.
`SYMBOLS`	Configured symbols that can be removed while preserving selected sentence delimiters.
`ASCII_LETTERS`	Set containing uppercase and lowercase ASCII letters.
`DIGITS`	Set containing decimal digits.
`PUNCTUATION`	Set containing characters from `string.punctuation`.
`WHITESPACE`	Set containing supported whitespace characters.
`CONTROL_CHARACTERS`	Set containing ASCII control characters and the delete character.
`NUMERALS`	Regular-expression pattern used to identify Roman numerals.

Processing Architecture¶

The processing hierarchy is organized around the Processor base class.

Processor
├── TextParser
├── NltkParser
├── WordParser
└── PdfParser

Processor initializes shared runtime state used by the specialized parser classes, including token collections, page buffers, chunk collections, vocabulary stores, encoding objects, lemmatizers, stemmers, and frequency distributions.

Text Processing¶

The TextParser class provides the general-purpose text preprocessing functionality used by Foo.

Cleanup Operations¶

Method	Purpose
`collapse_whitespace()`	Replaces contiguous whitespace with one ordinary space.
`remove_punctuation()`	Removes non-terminal punctuation while preserving sentence-ending punctuation.
`reduce_repeats()`	Reduces excessive punctuation and symbol repetitions caused by OCR or extraction.
`normalize_text()`	Converts alphabetic content to lowercase.
`remove_errors()`	Repairs mojibake and removes invalid extraction characters.
`remove_fragments()`	Removes isolated extraction debris and unsupported private-use glyphs.
`remove_symbols()`	Removes configured symbols while preserving selected sentence delimiters.
`remove_html()`	Extracts visible text from HTML content.
`remove_xml()`	Extracts visible text from XML content.
`remove_markdown()`	Removes Markdown syntax while preserving visible content.
`remove_stopwords()`	Removes standalone English stopwords.
`remove_encodings()`	Decodes HTML entities and normalizes Unicode artifacts.
`remove_headers()`	Removes repeated headers and footers from fixed-length logical pages.
`remove_numbers()`	Removes decimal digit sequences.
`remove_numerals()`	Removes qualifying uppercase Roman numerals.
`remove_images()`	Removes Markdown, HTML, data-URI, and standalone image references.

Segmentation Operations¶

Method	Purpose
`split_sentences()`	Segments text into sentences using NLTK.
`split_pages()`	Splits text into form-feed-delimited or fixed-line logical pages.
`split_paragraphs()`	Splits text files at blank-line paragraph boundaries.
`chunk_files()`	Splits files into sentence-based chunks.
`chunk_data()`	Builds fixed-size word chunks from a text file.
`chunk_datasets()`	Cleans and chunks multiple source files into Excel datasets.
`convert_jsonl()`	Converts text files into chunked JSONL-style output.

Tokenization and Vectorization¶

Method	Purpose
`tiktokenize()`	Encodes text into TikToken token identifiers.
`create_frequency_distribution()`	Creates a token-frequency DataFrame.
`create_vocabulary()`	Creates a vocabulary Series from token frequencies.
`create_wordbag()`	Creates a bag-of-words DataFrame.
`create_vectors()`	Creates TF-IDF vectors while preserving source token order.
`encode_sentences()`	Encodes supplied tokens with a sentence-transformer model.
`semantic_search()`	Returns the most similar tokens for a semantic query.

Text Processing Example¶

from processors import TextParser


parser = TextParser()

source_text = """
Foo   processes extracted text, removes encoding artifacts,
and prepares content for downstream analysis.
"""

collapsed_text = parser.collapse_whitespace( source_text )
normalized_text = parser.normalize_text( collapsed_text )
cleaned_text = parser.remove_encodings( normalized_text )
sentences = parser.split_sentences( cleaned_text )

NLTK Processing¶

The NltkParser class provides NLTK-backed lexical and grammatical processing.

Supported Operations¶

Method	Purpose
`initialize_resources()`	Verifies and downloads required NLTK resources.
`word_tokenizer()`	Tokenizes lowercase text into words.
`sentence_tokenizer()`	Tokenizes lowercase text into sentences.
`word_stemmer()`	Applies Porter stemming to word tokens.
`word_lemmatizer()`	Applies WordNet lemmatization to word tokens.
`pos_tagger()`	Assigns part-of-speech tags to tokens.
`named_entity_recognition()`	Extracts named entities and their NLTK labels.
`chunk_words()`	Groups word tokens into fixed-size chunks.
`chunk_sentences()`	Groups sentence tokens into fixed-size chunks.

NLTK Example¶

from processors import NltkParser


parser = NltkParser()

text = "The Environmental Protection Agency manages federal environmental programs."

tokens = parser.word_tokenizer( text )
tagged_tokens = parser.pos_tagger( text )
named_entities = parser.named_entity_recognition( text )

Word Document Processing¶

The WordParser class provides Word document extraction and lexical analysis.

The class maintains document text, paragraphs, sentences, cleaned sentences, vocabulary, and frequency-distribution state.

Supported Operations¶

Method	Purpose
`extract_text()`	Extracts text for the selected document page.
`split_sentences()`	Segments extracted page text into sentences.
`clean_sentences()`	Removes unsupported characters and normalizes sentence text.
`create_vocabulary()`	Creates a stopword-filtered vocabulary.
`compute_frequency_distribution()`	Calculates word frequencies across cleaned sentences.
`summarize()`	Prints document, sentence, vocabulary, and frequency statistics.

PDF Processing¶

The PdfParser class provides geometry-aware PDF extraction and cleanup.

Rather than relying solely on linear PDF text extraction, the parser evaluates the location of each text block on the page. Blocks are classified as header, body, or footer content according to configurable page-height ratios.

Geometry-Aware Extraction¶

extract_pages() reads PDF text blocks and records:

Page number
Block index
Horizontal and vertical coordinates
Block midpoint
Header, body, or footer zone
Extracted text
Drop status

The resulting page dictionaries can be processed by remove_repeats() and reconstructed by rebuild_pages().

Header and Footer Removal¶

remove_repeats() normalizes header and footer candidates and marks repeated content for removal. The method also recognizes common page-number formats, including:

Numeric page labels
Roman-numeral page labels
Page n
Page n of n
p. n

PDF Cleanup Operations¶

Method	Purpose
`geometric_extract()`	Runs geometry-aware page extraction and rebuilds the resulting text.
`extract_pages()`	Extracts positioned text blocks from PDF pages.
`remove_repeats()`	Marks repeated headers, footers, and page labels for removal.
`clean_artifacts()`	Removes parser markers, image markers, file paths, and PDF structural debris.
`repair_spacing()`	Repairs malformed punctuation and letter-spaced text.
`rejoin_hyphenation()`	Rejoins line-break and embedded word hyphenation.
`rebuild_pages()`	Reconstructs cleaned page text from retained blocks.
`extract_lines()`	Returns nonempty lines from geometry-aware extraction.
`extract_text()`	Returns rebuilt text from the selected PDF pages.
`extract_tables()`	Extracts PDF tables into pandas DataFrames.

Export Operations¶

Method	Purpose
`export_csv()`	Writes each extracted table to a separate CSV file.
`export_text()`	Writes extracted lines to a UTF-8 text file.
`export_excel()`	Writes extracted tables to separate Excel worksheets.

Geometry-Aware PDF Example¶

from processors import PdfParser


parser = PdfParser(
    headers=False,
    size=10,
    tables=True,
)

pages = parser.extract_pages(
    path="documents/report.pdf",
    header_ratio=0.08,
    footer_ratio=0.08,
)

cleaned_pages = parser.remove_repeats(
    pages=pages,
    minimum_repeats=3,
)

text = parser.rebuild_pages(
    pages=cleaned_pages,
    preserve_page_breaks=True,
)

text = parser.clean_artifacts( text )
text = parser.repair_spacing( text )
text = parser.rejoin_hyphenation( text )

Table Extraction Example¶

from processors import PdfParser


parser = PdfParser()

df_tables = parser.extract_tables(
    path="documents/report.pdf",
)

if df_tables:
    parser.export_excel(
        tables=df_tables,
        path="output/report_tables.xlsx",
    )

Error Handling¶

Processing methods validate required values through throw_if().

Wrapped failures are converted to the project Error type, populated with module, class, and method metadata, written through Logger, and re-raised.

except Exception as e:
    exception = Error( e )
    exception.module = 'processors'
    exception.cause = 'TextParser'
    exception.method = 'remove_html( self, text: str ) -> str'
    Logger( ).write( exception )
    raise exception

API Reference¶

processors ¶

    Assembly:                Foo
    Filename:                processors.py
    Author:                  Terry D. Eppler
    Created:                 05-31-2022

    Last Modified By:        Terry D. Eppler
    Last Modified On:        05-01-2025

           Boo is a df analysis tool that integrates various Generative AI, Text-Processing, and
           Machine-Learning algorithms for federal analysts.
           Copyright ©  2022  Terry Eppler

   Permission is hereby granted, free of charge, to any person obtaining a copy
   of this software and associated documentation files (the “Software”),
   to deal in the Software without restriction,
   including without limitation the rights to use,
   copy, modify, merge, publish, distribute, sublicense,
   and/or sell copies of the Software,
   and to permit persons to whom the Software is furnished to do so,
   subject to the following conditions:

   The above copyright notice and this permission notice shall be included in all
   copies or substantial portions of the Software.

   THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
   INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
   FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT.
   IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
   DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
   ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
   DEALINGS IN THE SOFTWARE.

   You can contact me at:  terryeppler@gmail.com or eppler.terry@epa.gov

Provides text-processing, parsing, tokenization, NLP, and PDF cleanup utilities for Chonky. Purpose: Defines the processing classes used after document ingestion and before embedding or vector persistence. The module supports text normalization, formatting cleanup, tokenization, lemmatization, stemming, vocabulary construction, frequency analysis, Word document parsing, and geometry-aware PDF text reconstruction. Existing wrapped exception handlers record validation and processing failures through the application logger before re-raising the project Error wrapper.

Classes¶

Processor ¶

Processor()

Processor processing component.

Purpose

Initializes shared state used by text, Word, NLTK, and PDF parser subclasses, including token caches, line/page buffers, vocabulary stores, NLP helpers, and cleaned-text fields used across the processing workflow.

Attributes:

Name	Type	Description
`lemmatizer`	`Optional[WordNetLemmatizer]`	WordNet lemmatizer used to reduce tokens to their canonical lexical forms.
`stemmer`	`Optional[PorterStemmer]`	Porter stemmer used to reduce tokens to word stems.
`file_path`	`Optional[str]`	Path of the file currently being loaded, parsed, cleaned, or exported.
`normalized`	`Optional[str]`	Text produced by the active normalization operation.
`lemmatized`	`Optional[str]`	Text produced after lemmatization.
`tokenized`	`Optional[str]`	Text or serialized token content produced by tokenization.
`encoding`	`Optional[Encoding]`	TikToken encoding used to convert text into model-compatible token identifiers.
`nlp`	`Optional[Language]`	Optional spaCy language pipeline used for linguistic processing.
`parts_of_speech`	`Optional[List[Tuple[str, str]]]`	Token and part-of-speech tag pairs produced by linguistic analysis.
`embedddings`	`Optional[List[ndarray]]`	Numeric embedding vectors generated for tokens, sentences, or document content.
`chunk_size`	`Optional[int]`	Number of tokens, words, sentences, or records assigned to each generated chunk.
`corrected`	`Optional[str]`	Text produced after spelling, extraction, or formatting corrections.
`raw_input`	`Optional[str]`	Unmodified text supplied to the current processing operation.
`raw_html`	`Optional[str]`	Original HTML content retained before markup removal.
`raw_pages`	`Optional[List[str]]`	Unprocessed page text extracted from a source document.
`lines`	`Optional[List[str]]`	Ordered text lines loaded from or extracted from the active source.
`tokens`	`Optional[List[str]]`	Ordered tokens produced from the active text.
`files`	`Optional[List[str]]`	File names or paths associated with a batch-processing operation.
`pages`	`Optional[List[str]]`	Logical pages or page structures extracted from the active document.
`paragraphs`	`Optional[List[str]]`	Paragraphs extracted from the active text or document.
`ids`	`Optional[List[int]]`	Numeric token, record, page, or chunk identifiers created during processing.
`stop_words`	`Optional[set]`	Stopword vocabulary used to exclude common terms from lexical processing.
`vocabulary`	`Optional[set]`	Unique terms produced from the active token collection or corpus.
`corpus`	`Optional[DataFrame]`	Tabular corpus containing text, tokens, frequencies, or other processed records.
`removed`	`Optional[List[str]]`	Text fragments, symbols, tokens, or artifacts removed during cleanup.
`frequency_distribution`	`Optional[DataFrame]`	Mapping or tabular representation of token frequencies.

TextParser ¶

TextParser()

Bases: Processor

TextParser processing component.

Purpose

Provides text cleanup, normalization, tokenization, chunking, vocabulary, frequency-distribution, and semantic-preparation utilities used by the Text Processing, Analysis, and Tokenization tabs.

Attributes:

Name	Type	Description
`lowercase`	`Optional[str]`	Runtime state used by `TextParser` during processing operations.
`cleaned_text`	`Optional[str]`	Runtime state used by `TextParser` during processing operations.
`cleaned_lines`	`Optional[List[str]]`	Runtime state used by `TextParser` during processing operations.
`cleaned_tokens`	`Optional[List[str]]`	Runtime state used by `TextParser` during processing operations.
`cleaned_pages`	`Optional[List[str]]`	Runtime state used by `TextParser` during processing operations.
`cleaned_html`	`Optional[str]`	Runtime state used by `TextParser` during processing operations.
`conditional_distribution`	`Optional[DataFrame]`	Runtime state used by `TextParser` during processing operations.
`PUNCTUATION`	`Optional[Set[str]]`	Runtime state used by `TextParser` during processing operations.
`CONTROL_CHARACTERS`	`Optional[Set[str]]`	Runtime state used by `TextParser` during processing operations.
`DELIMITERS`	`Optional[Set[str]]`	Runtime state used by `TextParser` during processing operations.
`DIGITS`	`Optional[Set[str]]`	Runtime state used by `TextParser` during processing operations.
`SYMBOLS`	`Optional[Set[str]]`	Runtime state used by `TextParser` during processing operations.
`NUMERALS`	`Optional[str]`	Runtime state used by `TextParser` during processing operations.

Purpose

Initializes parser state, reusable helper objects, and runtime caches used by later processing methods.

Methods:¶

load_text ¶

load_text(filepath: str) -> str | None

Load text.

Purpose

Executes the load text operation for the TextParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`filepath`	`str`	Filepath value used by the processing operation. Expected type: `str`.	required

Returns:

Type	Description
`str \| None`	str \| None: Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

collapse_whitespace ¶

collapse_whitespace(text: str) -> str | None

Collapse whitespace.

Purpose

Converts each contiguous sequence of whitespace characters into one ordinary space while preserving capitalization, punctuation, numbers, symbols, and content order.

Parameters:

Name	Type	Description	Default
`text`	`str`	Source text containing whitespace to collapse.	required

Returns:

Type	Description
`str \| None`	str \| None: Text containing one ordinary space between adjacent content.

Raises:

Type	Description
`Error`	Raised when validation or whitespace collapsing fails.

remove_punctuation ¶

remove_punctuation(text: str) -> str

Remove punctuation.

Purpose

Removes non-terminal punctuation while preserving sentence-ending periods, question marks, exclamation points, capitalization, numbers, and readable token boundaries.

Parameters:

Name	Type	Description	Default
`text`	`str`	Source text containing punctuation to remove.	required

Returns:

Name	Type	Description
`str`	`str`	Text with non-terminal punctuation removed.

Raises:

Type	Description
`Error`	Raised when validation or punctuation removal fails.

reduce_repeats ¶

reduce_repeats(text: str) -> str

Reduce repeated artifacts.

Purpose

Reduces excessive repeated punctuation and symbol runs commonly produced by OCR or document extraction while preserving ellipses, paired dashes, capitalization, and text order.

Parameters:

Name	Type	Description	Default
`text`	`str`	Source text containing repeated punctuation or symbol artifacts.	required

Returns:

Name	Type	Description
`str`	`str`	Text with excessive repeated artifacts reduced.

Raises:

Type	Description
`Error`	Raised when validation or repeat reduction fails.

normalize_text ¶

normalize_text(text: str) -> str | None

Normalize text case.

Purpose

Converts alphabetic text to lowercase while preserving punctuation, numbers, whitespace, document structure, and all non-case-related content.

Parameters:

Name	Type	Description	Default
`text`	`str`	Source text to convert to lowercase.	required

Returns:

Type	Description
`str \| None`	str \| None: Lowercase representation of the supplied text.

Raises:

Type	Description
`Error`	Raised when validation or lowercase normalization fails.

remove_errors ¶

remove_errors(text: str) -> str

Remove extraction errors.

Purpose

Repairs recognized sequences and removes replacement characters, null bytes, invisible formatting artifacts, surrogate characters, private-use extraction glyphs, and unsupported control characters while preserving valid text, punctuation, numbers, line boundaries, tabs, and document order.

Parameters:

Name	Type	Description	Default
`text`	`str`	Source text containing extraction or encoding artifacts.	required

Returns:

Name	Type	Description
`str`	`str`	Text with recognized extraction artifacts removed or repaired.

Raises:

Type	Description
`Error`	Raised when validation or extraction-error removal fails.

remove_fragments ¶

remove_fragments(text: str) -> str

Remove malformed fragments.

Purpose

Removes isolated extraction debris and unsupported private-use glyphs while preserving valid words, acronyms, identifiers, legal symbols, punctuation, numbers, whitespace, and document order.

Parameters:

Name	Type	Description	Default
`text`	`str`	Source text containing possible extraction fragments.	required

Returns:

Name	Type	Description
`str`	`str`	Text with isolated malformed extraction fragments removed.

Raises:

Type	Description
`Error`	Raised when validation or fragment removal fails.

remove_symbols ¶

remove_symbols(text: str) -> str | None

Remove configured symbols.

Purpose

Removes configured non-sentence symbols while preserving capitalization, numbers, sentence delimiters, document order, and readable token boundaries.

Parameters:

Name	Type	Description	Default
`text`	`str`	Source text containing configured symbols to remove.	required

Returns:

Type	Description
`str \| None`	str \| None: Text with configured symbols removed.

Raises:

Type	Description
`Error`	Raised when validation or symbol removal fails.

remove_html ¶

remove_html(text: str) -> str

Remove HTML markup.

Purpose

Removes HTML elements and non-visible content while preserving visible text, inline word boundaries, paragraph boundaries, capitalization, punctuation, numbers, URLs, legal citations, and document order.

Parameters:

Name	Type	Description	Default
`text`	`str`	Source text containing HTML markup or entities.	required

Returns:

Name	Type	Description
`str`	`str`	Visible text extracted from the supplied HTML content.

Raises:

Type	Description
`Error`	Raised when validation or HTML processing fails.

remove_xml ¶

remove_xml(text: str) -> str

Remove XML markup.

Purpose

Removes XML declarations, comments, processing instructions, and elements while preserving visible text, inline word boundaries, existing line boundaries, capitalization, punctuation, numbers, URLs, legal citations, and document order.

Parameters:

Name	Type	Description	Default
`text`	`str`	Source text containing XML markup.	required

Returns:

Name	Type	Description
`str`	`str`	Visible text extracted from the supplied XML content.

Raises:

Type	Description
`Error`	Raised when validation or XML processing fails.

remove_markdown ¶

remove_markdown(text: str) -> str | None

Remove Markdown syntax.

Purpose

Removes Markdown formatting syntax while preserving visible link labels, image alternative text, headings, list content, code content, capitalization, hyphenated terms, and paragraph structure.

Parameters:

Name	Type	Description	Default
`text`	`str`	Source text containing Markdown syntax.	required

Returns:

Type	Description
`str \| None`	str \| None: Readable text with Markdown formatting syntax removed.

Raises:

Type	Description
`Error`	Raised when validation or Markdown processing fails.

remove_stopwords ¶

remove_stopwords(text: str) -> str

Remove English stopwords.

Purpose

Removes standalone English stopwords while preserving URLs, punctuation, numbers, line boundaries, paragraph boundaries, capitalization, citations, and document order.

Parameters:

Name	Type	Description	Default
`text`	`str`	Source text containing English stopwords.	required

Returns:

Name	Type	Description
`str`	`str`	Text with standalone English stopwords removed.

Raises:

Type	Description
`Error`	Raised when validation or stopword removal fails.

remove_encodings ¶

remove_encodings(text: str) -> str

Remove encoding artifacts.

Purpose

Decodes HTML entities, normalizes compatible Unicode characters, converts Unicode spacing characters to ordinary spaces, and removes control, formatting, surrogate, private-use, replacement, and soft-hyphen artifacts while preserving readable text, capitalization, punctuation, numbers, tabs, line boundaries, URLs, and document order.

Parameters:

Name	Type	Description	Default
`text`	`str`	Source text containing encoded entities or Unicode artifacts.	required

Returns:

Name	Type	Description
`str`	`str`	Text containing normalized Unicode characters without unsafe encoding artifacts.

Raises:

Type	Description
`Error`	Raised when validation or encoding normalization fails.

remove_headers ¶

remove_headers(
    filepath: str,
    lines: int = 50,
    headers: int = 3,
    footers: int = 3,
) -> str

Remove repeated headers and footers.

Purpose

Removes exact header and footer line sequences repeated across multiple fixed-length text pages while preserving unique page content, line boundaries, page order, and single-page documents.

Parameters:

Name	Type	Description	Default
`filepath`	`str`	Path to the UTF-8 text file to process.	required
`lines`	`int`	Maximum number of lines assigned to each logical page.	`50`
`headers`	`int`	Number of leading lines evaluated as a repeated header.	`3`
`footers`	`int`	Number of trailing lines evaluated as a repeated footer.	`3`

Returns:

Name	Type	Description
`str`	`str`	Text with qualifying repeated headers and footers removed.

Raises:

Type	Description
`Error`	Raised when validation, file access, or header and footer removal fails.

remove_numbers ¶

remove_numbers(text: str) -> str | None

Remove numeric digits.

Purpose

Removes contiguous decimal digit sequences while preserving capitalization, punctuation, surrounding text, and readable token boundaries.

Parameters:

Name	Type	Description	Default
`text`	`str`	Source text containing numeric digits to remove.	required

Returns:

Type	Description
`str \| None`	str \| None: Text with decimal digit sequences removed.

Raises:

Type	Description
`Error`	Raised when validation or numeric-digit removal fails.

remove_numerals ¶

remove_numerals(text: str) -> str

Remove Roman numerals.

Purpose

Removes standalone uppercase Roman numerals containing at least two characters while preserving single-letter words, lowercase text, ordinary numbers, punctuation, whitespace, citations, and document order.

Parameters:

Name	Type	Description	Default
`text`	`str`	Source text containing Roman numerals to remove.	required

Returns:

Name	Type	Description
`str`	`str`	Text with qualifying standalone Roman numerals removed.

Raises:

Type	Description
`Error`	Raised when validation or Roman-numeral removal fails.

remove_images ¶

remove_images(text: str) -> str

Remove textual image references.

Purpose

Removes Markdown image declarations, HTML image elements, data-image references, and standalone image URLs while preserving surrounding text, capitalization, document order, and non-image links.

Parameters:

Name	Type	Description	Default
`text`	`str`	Source text containing textual image references.	required

Returns:

Name	Type	Description
`str`	`str`	Text with supported image references removed.

Raises:

Type	Description
`Error`	Raised when validation or image-reference removal fails.

tiktokenize ¶

tiktokenize(
    text: str, encoding: str = "cl100k_base"
) -> DataFrame

Tokenize text with a TikToken encoding.

Purpose

Encodes text into model-compatible token identifiers without altering capitalization, punctuation, numbers, whitespace, symbols, or content order.

Parameters:

Name	Type	Description	Default
`text`	`str`	Source text to tokenize.	required
`encoding`	`str`	TikToken encoding name used to generate token identifiers.	`'cl100k_base'`

Returns:

Name	Type	Description
`DataFrame`	`DataFrame`	Token identifiers in source order.

Raises:

Type	Description
`Error`	Raised when validation, encoding resolution, or tokenization fails.

split_sentences ¶

split_sentences(text: str) -> List[str]

Split text into sentences.

Purpose

Segments text into sentences without altering capitalization, punctuation, numbers, symbols, whitespace within sentences, legal citations, or content order.

Parameters:

Name	Type	Description	Default
`text`	`str`	Source text to segment into sentences.	required

Returns:

Type	Description
`List[str]`	List[ str ]: Sentences in source order.

Raises:

Type	Description
`Error`	Raised when validation or sentence tokenization fails.

split_pages ¶

split_pages(filepath: str, num: int = 50) -> List[str]

Split text into logical pages.

Purpose

Splits UTF-8 text into form-feed-delimited pages when page delimiters are available, or fixed-length line groups otherwise, while resetting parser state for each call.

Parameters:

Name	Type	Description	Default
`filepath`	`str`	Path to the UTF-8 text file to split.	required
`num`	`int`	Maximum number of lines assigned to each fallback logical page.	`50`

Returns:

Type	Description
`List[str]`	List[ str ]: Nonempty logical pages in source order.

Raises:

Type	Description
`Error`	Raised when validation, file access, or page splitting fails.

split_paragraphs ¶

split_paragraphs(filepath: str) -> DataFrame

Split text into paragraphs.

Purpose

Splits a text file at blank-line paragraph boundaries while preserving internal sentence spacing, punctuation, capitalization, numbers, symbols, and paragraph order.

Parameters:

Name	Type	Description	Default
`filepath`	`str`	Path to the text file to split into paragraphs.	required

Returns:

Name	Type	Description
`DataFrame`	`DataFrame`	Nonempty paragraphs in source order.

Raises:

Type	Description
`Error`	Raised when validation, file access, decoding, or paragraph splitting fails.

create_frequency_distribution ¶

create_frequency_distribution(
    tokens: List[str],
) -> DataFrame | None

Create frequency distribution.

Purpose

Executes the create frequency distribution operation for the TextParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`tokens`	`List[str]`	Tokens value used by the processing operation. Expected type: `List[ str ]`.	required

Returns:

Type	Description
`DataFrame \| None`	DataFrame \| None: Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

create_vocabulary ¶

create_vocabulary(tokens: List[str]) -> Series | None

Create vocabulary.

Purpose

Executes the create vocabulary operation for the TextParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`tokens`	`List[str]`	Tokens value used by the processing operation. Expected type: `List[ str ]`.	required

Returns:

Type	Description
`Series \| None`	Series \| None: Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

create_wordbag ¶

create_wordbag(tokens: List[str]) -> DataFrame | None

Create wordbag.

Purpose

Executes the create wordbag operation for the TextParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`tokens`	`List[str]`	Tokens value used by the processing operation. Expected type: `List[ str ]`.	required

Returns:

Type	Description
`DataFrame \| None`	DataFrame \| None: Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

create_vectors ¶

create_vectors(tokens: List[str]) -> DataFrame

Create TF-IDF vectors.

Purpose

Creates one TF-IDF feature vector for each supplied token while preserving token capitalization, punctuation, symbols, duplicates, and source order.

Parameters:

Name	Type	Description	Default
`tokens`	`List[str]`	Tokens used to construct the TF-IDF feature matrix.	required

Returns:

Name	Type	Description
`DataFrame`	`DataFrame`	TF-IDF vectors indexed by the source tokens.

Raises:

Type	Description
`Error`	Raised when validation or TF-IDF vector creation fails.

clean_file ¶

clean_file(filepath: str) -> str | None

Clean file.

Purpose

Executes the clean file operation for the TextParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`filepath`	`str`	Filepath value used by the processing operation. Expected type: `str`.	required

Returns:

Type	Description
`str \| None`	str \| None: Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

clean_files ¶

clean_files(source: str, destination: str) -> None

Clean files.

Purpose

Executes the clean files operation for the TextParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`source`	`str`	Source value used by the processing operation. Expected type: `str`.	required
`destination`	`str`	Destination value used by the processing operation. Expected type: `str`.	required

Returns:

Name	Type	Description
`None`	`None`	Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

chunk_files ¶

chunk_files(source: str, destination: str) -> None

Chunk files.

Purpose

Executes the chunk files operation for the TextParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`source`	`str`	Source value used by the processing operation. Expected type: `str`.	required
`destination`	`str`	Destination value used by the processing operation. Expected type: `str`.	required

Returns:

Name	Type	Description
`None`	`None`	Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

chunk_data ¶

chunk_data(
    filepath: str, size: int = 10
) -> DataFrame | None

Chunk data.

Purpose

Executes the chunk data operation for the TextParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`filepath`	`str`	Filepath value used by the processing operation. Expected type: `str`.	required
`size`	`int`	Size value used by the processing operation. Expected type: `int`. Defaults to `10`.	`10`

Returns:

Type	Description
`DataFrame \| None`	DataFrame \| None: Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

chunk_datasets ¶

chunk_datasets(
    source: str, destination: str, size: int = 10
) -> DataFrame

Chunk datasets.

Purpose

Executes the chunk datasets operation for the TextParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`source`	`str`	Source value used by the processing operation. Expected type: `str`.	required
`destination`	`str`	Destination value used by the processing operation. Expected type: `str`.	required
`size`	`int`	Size value used by the processing operation. Defaults to `10`.	`10`

Returns:

Name	Type	Description
`DataFrame`	`DataFrame`	Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

convert_jsonl ¶

convert_jsonl(
    source: str, destination: str, size: int = 10
) -> None

Convert jsonl.

Purpose

Executes the convert jsonl operation for the TextParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`source`	`str`	Source value used by the processing operation. Expected type: `str`.	required
`destination`	`str`	Destination value used by the processing operation. Expected type: `str`.	required
`size`	`int`	Size value used by the processing operation. Expected type: `int`. Defaults to `10`.	`10`

Returns:

Name	Type	Description
`None`	`None`	Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

encode_sentences ¶

encode_sentences(
    tokens: List[str], model: str = "all-MiniLM-L6-v2"
) -> Tuple[List[str], np.ndarray]

Encode sentences.

Purpose

Executes the encode sentences operation for the TextParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`tokens`	`List[str]`	Tokens value used by the processing operation. Expected type: `List[ str ]`.	required
`model`	`str`	Model value used by the processing operation. Expected type: `str`. Defaults to `'all-MiniLM-L6-v2'`.	`'all-MiniLM-L6-v2'`

Returns:

Type	Description
`Tuple[List[str], ndarray]`	Tuple[ List[ str ], np.ndarray ]: Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

semantic_search ¶

semantic_search(
    query: str,
    tokens: List[str],
    embeddings: ndarray,
    model: SentenceTransformer,
    top: int = 5,
) -> List[tuple[str, float]]

Semantic search.

Purpose

Executes the semantic search operation for the TextParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`query`	`str`	Query value used by the processing operation. Expected type: `str`.	required
`tokens`	`List[str]`	Tokens value used by the processing operation. Expected type: `List[ str ]`.	required
`embeddings`	`ndarray`	Embeddings value used by the processing operation. Expected type: `np.ndarray`.	required
`model`	`SentenceTransformer`	Model value used by the processing operation. Expected type: `SentenceTransformer`.	required
`top`	`int`	Top value used by the processing operation. Expected type: `int`. Defaults to `5`.	`5`

Returns:

Type	Description
`List[tuple[str, float]]`	List[ tuple[ str, float ] ]: Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

NltkParser ¶

NltkParser()

Bases: Processor

NltkParser processing component.

Purpose

Provides NLTK-backed parsing utilities for corpus tokenization, stemming, lemmatization, part-of-speech tagging, named-entity handling, and lexical diagnostics used by Chonky analysis workflows.

Attributes:

Name	Type	Description
`word_tokens`	`Optional[List[str]]`	Runtime state used by `NltkParser` during processing operations.
`sentence_tokens`	`Optional[List[str]]`	Runtime state used by `NltkParser` during processing operations.
`stemmed_tokens`	`Optional[List[str]]`	Runtime state used by `NltkParser` during processing operations.
`lemmatized_tokens`	`Optional[List[str]]`	Runtime state used by `NltkParser` during processing.
`tagged_tokens`	`Optional[List[Tuple[str, str]]]`	Runtime state used by `NltkParser` during processing operations.
`named_entities`	`Optional[List[Tuple[str, str]]]`	Runtime state used by `NltkParser` during processing operations.

Purpose

Initializes parser state, reusable helper objects, and runtime caches used by later processing methods.

Methods:¶

initialize_resources ¶

initialize_resources() -> None

Initialize resources.

Purpose

Executes the initialize resources operation for the NltkParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Returns:

Name	Type	Description
`None`	`None`	Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

word_tokenizer ¶

word_tokenizer(text: str) -> List[str] | None

Word tokenizer.

Purpose

Executes the word tokenizer operation for the NltkParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`text`	`str`	Text value used by the processing operation. Expected type: `str`.	required

Returns:

Type	Description
`List[str] \| None`	List[ str ] \| None: Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

sentence_tokenizer ¶

sentence_tokenizer(text: str) -> List[str] | None

Sentence tokenizer.

Purpose

Executes the sentence tokenizer operation for the NltkParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`text`	`str`	Text value used by the processing operation. Expected type: `str`.	required

Returns:

Type	Description
`List[str] \| None`	List[ str ] \| None: Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

word_stemmer ¶

word_stemmer(text: str) -> List[str] | None

Word stemmer.

Purpose

Executes the word stemmer operation for the NltkParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`text`	`str`	Text value used by the processing operation. Expected type: `str`.	required

Returns:

Type	Description
`List[str] \| None`	List[ str ] \| None: Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

word_lemmatizer ¶

word_lemmatizer(text: str) -> List[str] | None

Word lemmatizer.

Purpose

Executes the word lemmatizer operation for the NltkParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`text`	`str`	Text value used by the processing operation. Expected type: `str`.	required

Returns:

Type	Description
`List[str] \| None`	List[ str ] \| None: Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

pos_tagger ¶

pos_tagger(text: str) -> List[Tuple[str, str]] | None

Pos tagger.

Purpose

Executes the pos tagger operation for the NltkParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`text`	`str`	Text value used by the processing operation. Expected type: `str`.	required

Returns:

Type	Description
`List[Tuple[str, str]] \| None`	List[ Tuple[ str, str ] ] \| None: Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

named_entity_recognition ¶

named_entity_recognition(
    text: str,
) -> List[Tuple[str, str]] | None

Named entity recognition.

Purpose

Executes the named entity recognition operation for the NltkParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`text`	`str`	Text value used by the processing operation. Expected type: `str`.	required

Returns:

Type	Description
`List[Tuple[str, str]] \| None`	List[ Tuple[ str, str ] ] \| None: Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

chunk_words ¶

chunk_words(text: str, size: int = 5) -> DataFrame | None

Chunk words.

Purpose

Executes the chunk words operation for the NltkParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`text`	`str`	Text value used by the processing operation. Expected type: `str`.	required
`size`	`int`	Size value used by the processing operation. Expected type: `int`. Defaults to `5`.	`5`

Returns:

Type	Description
`DataFrame \| None`	DataFrame \| None: Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

chunk_sentences ¶

chunk_sentences(
    text: str, size: int = 15
) -> DataFrame | None

Chunk sentences.

Purpose

Executes the chunk sentences operation for the NltkParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`text`	`str`	Text value used by the processing operation. Expected type: `str`.	required
`size`	`int`	Size value used by the processing operation. Expected type: `int`. Defaults to `15`.	`15`

Returns:

Type	Description
`DataFrame \| None`	DataFrame \| None: Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

WordParser ¶

WordParser(filepath: str)

Bases: Processor

WordParser processing component.

Purpose

Provides Microsoft Word document parsing utilities that extract paragraphs, tables, metadata, and cleaned text for downstream processing and analysis.

Attributes:

Name	Type	Description
`sentences`	`Optional[List[str]]`	Runtime state used by `WordParser` during processing operations.
`cleaned_sentences`	`Optional[List[str]]`	Runtime state used by `WordParser` during processing operations.
`document`	`Optional[Document]`	Runtime state used by `WordParser` during processing operations.
`raw_text`	`Optional[str]`	Runtime state used by `WordParser` during processing operations.
`paragraphs`	`Optional[List[str]]`	Runtime state used by `WordParser` during processing operations.
`file_path`	`Optional[str]`	Runtime state used by `WordParser` during processing operations.
`vocabulary`	`Optional[set]`	Runtime state used by `WordParser` during processing operations.
`document`	`Optional[Document]`	Runtime state used by `WordParser` during processing operations.

Purpose

Initializes parser state, reusable helper objects, and runtime caches used by later processing methods.

Parameters:

Name	Type	Description	Default
`filepath`	`str`	Filepath value used by the processing operation. Expected type: `str`.	required

Methods:¶

extract_text ¶

extract_text(num: int = 1) -> str | None

Extract text.

Purpose

Executes the extract text operation for the WordParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`num`	`int`	Num value used by the processing operation. Defaults to `1`.	`1`

Returns:

Type	Description
`str \| None`	str \| None: Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

split_sentences ¶

split_sentences() -> List[str] | None

Split sentences.

Purpose

Executes the split sentences operation for the WordParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Returns:

Type	Description
`List[str] \| None`	List[ str ] \| None: Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

clean_sentences ¶

clean_sentences() -> List[str] | None

Clean sentences.

Purpose

Executes the clean sentences operation for the WordParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Returns:

Type	Description
`List[str] \| None`	List[ str ] \| None: Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

create_vocabulary ¶

create_vocabulary() -> set | None

Create vocabulary.

Purpose

Executes the create vocabulary operation for the WordParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Returns:

Type	Description
`set \| None`	set \| None: Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

compute_frequency_distribution ¶

compute_frequency_distribution() -> Dict[str, int] | None

Compute frequency distribution.

Purpose

Executes the compute frequency distribution operation for the WordParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Returns:

Type	Description
`Dict[str, int] \| None`	Dict[ str, int ] \| None: Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

summarize ¶

summarize() -> List[str] | None

Summarize.

Purpose

Executes the summarize operation for the WordParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Returns:

Type	Description
`List[str] \| None`	List[ str ] \| None: Result produced by the processing operation.

PdfParser ¶

PdfParser(
    headers: bool = False,
    size: int = 10,
    tables: bool = True,
)

Bases: Processor

PdfParser processing component.

Purpose

Provides geometry-aware PDF parsing and cleanup utilities that extract page blocks, remove repeating headers and footers, repair line spacing, normalize artifacts, and rebuild PDF text for downstream processing.

Attributes:

Name	Type	Description
`strip_headers`	`Optional[bool]`	Runtime state used by `PdfParser` during processing operations.
`minimum_length`	`Optional[int]`	Runtime state used by `PdfParser` during processing operations.
`extract_tables_enabled`	`Optional[bool]`	Runtime state used by `PdfParser` during processing operations.
`extracted_lines`	`Optional[List]`	Runtime state used by `PdfParser` during processing operations.
`extracted_tables`	`Optional[List]`	Runtime state used by `PdfParser` during processing operations.
`extracted_pages`	`Optional[List]`	Runtime state used by `PdfParser` during processing operations.

Purpose

Initializes parser state, reusable helper objects, and runtime caches used by later processing methods.

Parameters:

Name	Type	Description	Default
`headers`	`bool`	Headers value used by the processing operation. Expected type: `bool`. Defaults to `False`.	`False`
`size`	`int`	Size value used by the processing operation. Expected type: `int`. Defaults to `10`.	`10`
`tables`	`bool`	Tables value used by the processing operation. Expected type: `bool`. Defaults to `True`.	`True`

Methods:¶

geometric_extract ¶

geometric_extract(
    path: str,
    count: Optional[int] = None,
    header_ratio: float = 0.08,
    footer_ratio: float = 0.08,
    preserve_page_breaks: bool = False,
) -> str | None

Geometric extract.

Purpose

Executes the geometric extract operation for the PdfParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`path`	`str`	Path value used by the processing operation. Expected type: `str`.	required
`count`	`Optional[int]`	Count value used by the processing operation. Defaults to `None`.	`None`
`header_ratio`	`float`	Header ratio value used by the processing operation. Defaults to `0.08`.	`0.08`
`footer_ratio`	`float`	Footer ratio value used by the processing operation. Defaults to `0.08`.	`0.08`
`preserve_page_breaks`	`bool`	Preserve page breaks value used by the processing operation.	`False`

Returns:

Type	Description
`str \| None`	str \| None: Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

extract_pages ¶

extract_pages(
    path: str,
    count: Optional[int] = None,
    header_ratio: float = 0.08,
    footer_ratio: float = 0.08,
) -> List[dict] | None

Extract pages.

Purpose

Executes the extract pages operation for the PdfParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`path`	`str`	Path value used by the processing operation. Expected type: `str`.	required
`count`	`Optional[int]`	Count value used by the processing operation. Expected type: `Optional[ int ]`. Defaults to `None`.	`None`
`header_ratio`	`float`	Header ratio value used by the processing operation. Expected type: `float`. Defaults to `0.08`.	`0.08`
`footer_ratio`	`float`	Footer ratio value used by the processing operation. Expected type: `float`. Defaults to `0.08`.	`0.08`

Returns:

Type	Description
`List[dict] \| None`	List[ dict ] \| None: Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

remove_repeats ¶

remove_repeats(
    pages: List[Dict], minimum_repeats: int = 3
) -> List[Dict] | None

Remove repeats.

Purpose

Executes the remove repeats operation for the PdfParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`pages`	`List[Dict]`	Pages value used by the processing operation. Expected type: `List[ dict ]`.	required
`minimum_repeats`	`int`	Minimum repeats value used by the processing operation. Expected type: `int`. Defaults to `3`.	`3`

Returns:

Type	Description
`List[Dict] \| None`	List[ dict ] \| None: Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

clean_artifacts ¶

clean_artifacts(text: str) -> str

Clean artifacts.

Purpose

Executes the clean artifacts operation for the PdfParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`text`	`str`	Text value used by the processing operation. Expected type: `str`.	required

Returns:

Name	Type	Description
`str`	`str`	Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

repair_spacing ¶

repair_spacing(text: str) -> str

Repair spacing.

Purpose

Repairs malformed punctuation spacing and extracted letter-spaced text while preserving valid URL schemes, legal citations, capitalization, line boundaries, and document order.

Parameters:

Name	Type	Description	Default
`text`	`str`	Source text containing spacing artifacts produced during PDF extraction.	required

Returns:

Name	Type	Description
`str`	`str`	Text with supported spacing artifacts repaired.

Raises:

Type	Description
`Error`	Raised when validation or spacing repair fails.

rejoin_hyphenation ¶

rejoin_hyphenation(
    text: str, repair_embedded: bool = True
) -> str

Rejoin hyphenation.

Purpose

Executes the rejoin hyphenation operation for the PdfParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`text`	`str`	Text value used by the processing operation. Expected type: `str`.	required
`repair_embedded`	`bool`	Repair embedded value used by the processing operation. Expected type: `bool`. Defaults to `True`.	`True`

Returns:

Name	Type	Description
`str`	`str`	Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

rebuild_pages ¶

rebuild_pages(
    pages: List[dict], preserve_page_breaks: bool = False
) -> str

Rebuild pages.

Purpose

Executes the rebuild pages operation for the PdfParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`pages`	`List[dict]`	Pages value used by the processing operation. Expected type: `List[ dict ]`.	required
`preserve_page_breaks`	`bool`	Preserve page breaks value used by the processing operation. Expected type: `bool`. Defaults to `False`.	`False`

Returns:

Name	Type	Description
`str`	`str`	Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

extract_lines ¶

extract_lines(
    path: str, count: Optional[int] = None
) -> List[str] | None

Extract lines.

Purpose

Executes the extract lines operation for the PdfParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`path`	`str`	Path value used by the processing operation. Expected type: `str`.	required
`count`	`Optional[int]`	Count value used by the processing operation. Expected type: `Optional[ int ]`. Defaults to `None`.	`None`

Returns:

Type	Description
`List[str] \| None`	List[ str ] \| None: Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

extract_text ¶

extract_text(
    path: str, count: Optional[int] = None
) -> str | None

Extract text.

Purpose

Executes the extract text operation for the PdfParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`path`	`str`	Path value used by the processing operation. Expected type: `str`.	required
`count`	`Optional[int]`	Count value used by the processing operation. Expected type: `Optional[ int ]`. Defaults to `None`.	`None`

Returns:

Type	Description
`str \| None`	str \| None: Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

extract_tables ¶

extract_tables(
    path: str, count: Optional[int] = None
) -> List[pd.DataFrame] | None

Extract tables.

Purpose

Executes the extract tables operation for the PdfParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`path`	`str`	Path value used by the processing operation. Expected type: `str`.	required
`count`	`Optional[int]`	Count value used by the processing operation. Expected type: `Optional[ int ]`. Defaults to `None`.	`None`

Returns:

Type	Description
`List[DataFrame] \| None`	List[ pd.DataFrame ] \| None: Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

export_csv ¶

export_csv(tables: List[DataFrame], filename: str) -> None

Export csv.

Purpose

Executes the export csv operation for the PdfParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`tables`	`List[DataFrame]`	Tables value used by the processing operation. Expected type: `List[ pd.DataFrame ]`.	required
`filename`	`str`	Filename value used by the processing operation. Expected type: `str`.	required

Returns:

Name	Type	Description
`None`	`None`	Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

export_text ¶

export_text(lines: List[str], path: str) -> None

Export text.

Purpose

Executes the export text operation for the PdfParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`lines`	`List[str]`	Lines value used by the processing operation. Expected type: `List[ str ]`.	required
`path`	`str`	Path value used by the processing operation. Expected type: `str`.	required

Returns:

Name	Type	Description
`None`	`None`	Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.

export_excel ¶

export_excel(tables: List[DataFrame], path: str) -> None

Export excel.

Purpose

Executes the export excel operation for the PdfParser workflow, updating instance state where required and returning the processed result used by downstream loading, processing, analysis, tokenization, or embedding steps.

Parameters:

Name	Type	Description	Default
`tables`	`List[DataFrame]`	Tables value used by the processing operation. Expected type: `List[ pd.DataFrame ]`.	required
`path`	`str`	Path value used by the processing operation. Expected type: `str`.	required

Returns:

Name	Type	Description
`None`	`None`	Result produced by the processing operation.

Raises:

Type	Description
`Error`	Raised when validation or processing fails.