hojichar
HojiChar: The Text Processing Pipeline
Official docs: https://hojichar.github.io/HojiChar/hojichar.html
Features
- HojiChar provides a way to combine multiple arbitrary text processing tasks into a streamlined pipeline.
- The sequence of operations can be described declaratively, ensuring portability.
- HojiChar allows users to gather detailed statistical information from large amounts of text during processing.
- It enables management of any Python text processing tasks, providing a Command Line Interface (CLI) capable of parallel processing.
Background and what is for HojiChar
Text preprocessing is far from a one-size-fits-all process. Depending on the data source and the specific task at hand, various steps including normalization, noise removal, and filtering may be necessary. Not all texts require the same level of preprocessing. For instance, relatively clean texts may only need minimal filtering, while "dirtier" sources like Common Crawl data often require more thorough processing. As a result, the preprocessing profile has to be tailored to each specific domain.
Many preprocessing operations can be viewed as filters, taking string as input, applying a transformation, and outputting the processed string. Even though these operations might seem straightforward individually, managing them in a multi-layered, efficient manner can be challenging.
Inspired by torchvision.transforms and iver56/audiomentations, HojiChar addresses these challenges. It enables users to define each text processing step as a class inheriting from hojichar.Filter and use hojichar.Compose to chain them together into a single filter. By writing out the Compose recipe as a profile, the preprocessing process for a specific domain's text can be made portable. Moreover, Compose automatically logs various metrics for each filter, such as byte changes, processing time, and number of rejected texts. This allows users to assess the validity of each operation and consider trade-offs between computation time and performance.
While there are other text normalization tools available, most are designed to perform a specific set of operations. Text preprocessing, despite its importance in the LLM era, is often considered a mundane task compared to machine learning or artificial intelligence tasks. As a result, many existing solutions can be ad hoc, poorly maintained, or inadequately tested. Recognizing these issues, we developed HojiChar as a robust tool for configuring text preprocessing.
For large text corpora, HojiChar also provides near-duplicate detection through GenerateDedupLSH. In our benchmark, HojiChar 0.18.0 generates MinHash LSH keys over 18× faster than Hugging Face DataTrove. The chart compares hash generation throughput with other implementations.
Install
Python 3.10 or later is required. CI tests Python 3.10 through 3.15 on Linux and macOS. Python 3.9 is no longer supported.
pip install hojichar
If you want to use the additional filters, install the package with the following command:
pip install 'hojichar[all]'
If you want to use AsyncChatAPI filter, install the package with the following command:
pip install 'hojichar[openai]'
If you want to use the near-deduplication (using MinHash LSH algorithm) filter, install the package with the following command:
pip install 'hojichar[dedup]'
On Python 3.15, the all and dedup extras currently build Fugashi from source.
Install MeCab first: brew install mecab on macOS, or
sudo apt-get install libmecab-dev on Debian/Ubuntu.
This prerequisite is temporary until compatible Fugashi wheels are available;
issue #109 tracks its removal.
Defining a Compose Object
The Compose class in HojiChar allows you to create a sequence of text processing filters.
from hojichar import Compose, document_filters
cleaner = Compose([
document_filters.JSONLoader(key="text"),
document_filters.AcceptJapanese(),
document_filters.DocumentLengthFilter(min_doc_len=0,max_doc_len=1000),
document_filters.ExampleHojiChar(),
document_filters.JSONDumper()
])
When a Compose object is called, it accepts a string and returns the processed string.
>>> cleaner('{"text": "こんにちは、"}')
{"text": "こんにちは、<hojichar>"}
The filter pipeline above accomplishes the following steps:
- Extracts the value from the
'text'key in the JSON object. - Discards the string if it's not in Japanese.
- Rejects any text shorter than 0 characters or longer than 1000 characters.
- Appends
<hojichar>to the string. - Outputs the processed string as JSON with the key "text".
The filters used in the pipeline are predefined filters found in hojichar.filters.
While HojiChar provides some fundamental text processing filters and plans to add more in the future, users can also define their custom filters.
Working with Metadata
JSONL corpora often store useful metadata alongside the text itself. JSONLoader and JSONDumper make it easy to keep that information with the document through the entire pipeline:
JSONLoaderautomatically merges anyextrasdictionary present in the input JSON intoDocument.extras. When you passextra_keys=["url", "title"], those fields are copied as well, so metadata survives even if new extras are added later in the pipeline.- Filters can freely append to
document.extras(for example, to record trace information or filter outcomes). JSONDumper(export_extras=True)writes the currentDocument.extrasback into the output JSON, making round-tripping metadata straightforward.
from hojichar import Compose, document_filters
pipeline = Compose(
[
document_filters.JSONLoader(extra_keys=["url"]),
document_filters.DocumentNormalizer(),
document_filters.JSONDumper(export_extras=True),
]
)
>>> pipeline('{"text": " Hello ", "extras": {"source": "blog"}, "url": "https://example.com"}')
{"text": "Hello", "extras": {"source": "blog", "url": "https://example.com"}}
In this example, the loader keeps both the embedded extras and the url field, later filters can enrich document.extras, and the dumper emits everything for downstream consumers.
User-defined Filters
A filter composing a Compose object is a class that inherits the Filter class and implements the text processing within the apply function.
from hojichar.core.filter_interface import Filter
class YourFilter(Filter):
def apply(self, document):
text = document.text
"""
Write your text transformation...
"""
document.text = text
return document
The apply method accepts a hojichar.Document type as an argument and returns it after the transformations. The Document is a class that encapsulates a string.
The Document class can have additional metadata via the extras attribute. This allows you to associate values with the document that can be utilized in subsequent filters. Reject documents
- The
hojichar.Documenthas anis_rejectedattribute. If a filter sets this flag toTrue,Composewill discard the document during processing.
Definition of __init__ for custom filter
When creating a user-defined class and applying a custom constructor, make sure to initialize the parent class.
class YourFilter(Filter):
def __init__(self, your_param, *args, **kwargs) -> None:
super().__init__(*args, **kwargs)
self.your_param = your_param
def apply(self, document):
text = document.text
text = process(text, self.your_param)
document.text = text
return document
This is because The Filter class implicitly has several arguments, one of which is p.
cleaner = Compose([
document_filters.JSONLoader(key="text"),
document_filters.AcceptJapanese(p=0.5),
document_filters.JSONDumper()
])
The p argument passed to the document_filters.AcceptJapanese constructor determines the probability of applying the filter; with a probability of 1-p, it acts as an identity function. This behavior is defined in the parent class hojichar.Filter.
Batch and Stream Processing with apply_batch and apply_stream
The Filter and Compose classes support efficient batch and stream processing through the apply_batch and apply_stream methods.
apply_batch
- The
apply_batchmethod processes a list ofDocumentobjects in one go. By default, it applies theapplymethod to each document individually. - Users can override
apply_batchin custom filters for optimized batch operations.class YourBatchFilter(Filter): def apply_batch(self, documents: Sequence[Document]) -> list[Document]: # Implement your batch processing logic here return documents
apply_stream
The apply_stream method processes an iterable (e.g., generator) of Document objects, ideal for large datasets or stream-based processing. If the use_batch flag is set to True in a Filter's constructor, its apply_batch implementation will be utilized during stream processing.
Example Usage:
stream = (Document(f"text {i}") for i in range(10000))
processed_stream = cleaner.apply_stream(stream)
for doc in processed_stream:
print(doc.text)
This allows HojiChar to efficiently process massive corpora while maintaining low memory consumption.
Additional Notes on Compose
- Even though the behavior of a
Composeobject when called is a text-in, text-out function,Composeitself also inherits from theFilterclass. Therefore, applying theapplymethod to aComposeobject results inhojihcar.Documentclass being used as input and output. Composeclass behaves like a Filter. If you add a Compose object as one of the filters in the constructor of Compose, the filter will be unfolded recursively.
HojiChar running asynchronously
- HojiChar supports asynchronous processing of text data using the
AsyncComposeclass. This allows you to build pipelines that can handle out-of-CPU processing, such as making API calls. You can define async versions of filter using the
AsyncFilterclass.from hojichar import AsyncFilter class YourAsyncFilter(AsyncFilter): async def apply(self, document): text = document.text # Perform asynchronous processing here document.text = text return documentThe
AsyncComposeclass accepts bothFilterandAsyncFilterobjects, allowing you to mix synchronous and asynchronous filters in a single pipeline.- For filters that use the default per-document
apply_batch,AsyncFilter.apply_streamkeeps at mostbatch_sizedocuments in flight and replenishes the window as each document finishes. Results preserve input order by default; setordered=Falseto yield completed documents immediately when input order is not required. - Filters that override
apply_batchcontinue to process complete batches, so implementations backed by a real batch API retain their batching behavior.
Synchronous tools can consume an AsyncCompose pipeline through imap_apply. The bridge owns
one background event loop and shuts down the pipeline after iteration by default. Use the returned
iterator as a context manager when the consumer may stop early.
with async_pipeline.imap_apply(input_doc_iter) as output_docs:
for document in output_docs:
print(document.text)
Example
Nowadays, text processing is enhanced by the intelligence of LLMs.
This example demonstrates how to use the AsyncChatAPI filter to process text data with OpenAI compatible APIs. This filter allows you to build high throughput of "Chain of LLMs" easily.
import os
from hojichar import AsyncCompose
from hojichar.filters.document_filters import JSONLoader, JSONDumper
from hojichar.utils.async_handlers import write_stream_to_file
async_pipeline = AsyncCompose(
[
JSONLoader(input_key="text"),
AsyncChatAPI(
model_id="gpt-4o",
openai_endpoint_url="https://api.openai.com/v1",
openai_api_key=os.getenv("OPENAI_API_KEY"),
max_concurrent_requests=128,
output_key="llm_output",
message_generator=lambda doc: [{"role": "user", "content": doc.text[:1000]}],
),
JSONDumper(export_extras=True),
]
)
with open("input.jsonl") as f:
async with async_pipeline:
async_output_stream = (str(doc) async for doc in async_pipeline.apply_stream(f))
await write_stream_to_file(async_output_stream, "output.jsonl", chunk_size=128) # Write async-iterable to file efficiently
- You can use this filter by installing
'hojichar[openai]' - The filter works with OpenAI compatible APIs, like the endpoint hosted by vLLM. It's useful for text-augumentation tasks.
- The AsyncChatAPI works 1K req/sec with optimized vLLM server. (We reccomend to use
uvloopto get better throughput.)
- The AsyncChatAPI works 1K req/sec with optimized vLLM server. (We reccomend to use
Get Metrics of processing
HojiChar tracks detailed statistics at both the filter and pipeline levels, helping you monitor and debug your processing pipeline.
Each Filter (including Compose) maintains a Statistics object containing information such as input size, output size, discarded document count, and processing time.
Example: Getting Statistics from a Compose Object
stats = cleaner.get_total_statistics_map()
print(stats)
[{'cumulative_time_ns': 337250,
'diff_bytes': 10,
'diff_chars': 10,
'discard_num': 0,
'input_bytes': 45,
'input_chars': 23,
'input_num': 1,
'name': 'Total',
'output_bytes': 55,
'output_chars': 33,
'output_num': 1},
{'cumulative_time_ns': 80209,
'diff_bytes': -12,
'diff_chars': -12,
'discard_num': 0,
'input_bytes': 45,
'input_chars': 23,
'input_num': 1,
'name': '0-JSONLoader',
'output_bytes': 33,
'output_chars': 11,
'output_num': 1},
{'cumulative_time_ns': 17500,
'diff_bytes': 0,
'diff_chars': 0,
'discard_num': 0,
'input_bytes': 33,
'input_chars': 11,
'input_num': 1,
'name': '1-AcceptJapanese',
'output_bytes': 33,
'output_chars': 11,
'output_num': 1},
{'cumulative_time_ns': 8125,
'diff_bytes': 0,
'diff_chars': 0,
'discard_num': 0,
'input_bytes': 33,
'input_chars': 11,
'input_num': 1,
'name': '2-DocumentLengthFilter',
'output_bytes': 33,
'output_chars': 11,
'output_num': 1},
{'cumulative_time_ns': 6042,
'diff_bytes': 10,
'diff_chars': 10,
'discard_num': 0,
'input_bytes': 33,
'input_chars': 11,
'input_num': 1,
'name': '3-ExampleHojiChar',
'output_bytes': 43,
'output_chars': 21,
'output_num': 1},
{'cumulative_time_ns': 81125,
'diff_bytes': 12,
'diff_chars': 12,
'discard_num': 0,
'input_bytes': 43,
'input_chars': 21,
'input_num': 1,
'name': '4-JSONDumper',
'output_bytes': 55,
'output_chars': 33,
'output_num': 1}]
- Use
get_statistics()to get the raw Statistics object for any filter. - Use
get_total_statistics()to get a list of statistics for all filters in a Compose pipeline. - Use
get_total_statistics_map()to retrieve the statistics as a list of dicts.
These tools allow granular monitoring of how each filter contributes to data reduction, rejection, or transformation.
Parallel application of Compose
The hojichar.Parallel class allows for the application of Compose to an iterable of Document concurrently. This class empowers users to process vast collections of documents by harnessing the power of multiple CPU cores.
Example usage of Parallel class to proces a very large JSON Lines file concurrently.
import hojichar
input_file = "your_text.jsonl"
input_doc_iter = (hojichar.Document(line) for line in open(input_file))
cleaner = hojichar.Compose([
hojichar.document_filters.JSONLoader(),
hojichar.document_filters.DocumentNormalizer(),
# Insert your filters
hojichar.document_filters.JSONDumper(),
])
with hojichar.Parallel(cleaner, num_jobs=10) as pfilter:
out_doc_iter = pfilter.imap_apply(input_doc_iter)
with open("your_processed_text.jsonl", "w") as fp:
for doc in out_doc_iter:
fp.write(doc.text + "\n")
- Always use the
Parallelclass within awithstatement. Parallel.imap_apply(doc_iter)processes an iterator ofDocumentand returns an iterator of the processed documents.- By default, processed documents are yielded as soon as they complete. Pass
ordered=TruetoParallelto yield them in the same order as the input documents. - If no start method is configured,
Parallelusesforkso workers can inherit unpicklable filters. SetHOJICHAR_MP_START_METHODto explicitly choose another context. - For additional options and details about the
Parallelclass, please refer to the official documentation.
CLI tool and preprocessing profile
- HojiChar provides CLI tools for text preprocess pipeline.
User defines a series of preprocessing into a python file as profile.
Example:
cat <your_text.jsonl> | hojichar -p your_preprocessing_profile.py -o your_text_preprocessed.jsonlhojichar --helpusage: hojichar [-h] --profile <profile.py> [--args ARGS [ARGS ...]] [--output OUTPUT] [--input INPUT] [--dump-stats <path to stats.json>] [--exit-on-error] [--all] [--jobs JOBS] options: -h, --help show this help message and exit --profile <profile.py>, -p <profile.py> Path to a Python file that implements your custom filter. --args ARGS [ARGS ...] Pass additional arguments to the profile. Use it like `--args arg1 arg2` etc. The arguments should be space-separated. --output OUTPUT, -o OUTPUT Specifies the path for the output file. Defaults to standard output. --input INPUT, -i INPUT Specifies the path for the input file. Defaults to standard input. If set this path, the progress bar is enabled. --dump-stats <path to stats.json> Dump statistics to file. If the file exists, it will be appended. --exit-on-error Exit if an exception occurs during filtering. Useful for debugging custom filters. --all A flag that specifies whether to include discarded samples. This is useful when inspecting discarded samples. --jobs JOBS, -j JOBS The number ob parallel jobs. By default, the nuber of the CPU core.
Definition of Profile
- HojiChar CLI receives a series of preprocessing as a profile.
- The preprocessing profile is provided as a Python file. Two patterns of the file are allowed.
- hojichar.utils.load_compose.load_compose() loads these profile.
FILTER profile
hojichar.Composemust be defined asFILTERvariable.Example.
import json from hojichar import Compose, Filter from hojichar.filters.document_filters import ExampleHojiChar, JSONLoader class JSONDumper(Filter): def apply(self, document): text = document.text document.text = json.dumps({"text": text}, ensure_ascii=False) return document # FILTER must define Compose object. FILTER = Compose( [ JSONLoader(), ExampleHojiChar(), JSONDumper(), ] )- Pass the texts to the filter you have defined using a pipe as follows.
cat <your_file> | hojichar -p example_profile.py
- Pass the texts to the filter you have defined using a pipe as follows.
hojichar.utils.load_compose.load_filter_from_file()loads this type of profile.
FACTORY profile
- A callable function that returns
hojichar.Composemust be defined asFACTORYvariable. - The callable can receive arguments. In this way, parameters can be passed to the profile.
- Some kinds of value are not preferred to static. For example, random seeds and some flags modify the behavior of a filter, etc
FACTORYprovides a mechanism to pass those values as arguments to the preprocessing.
Example.
import json from hojichar import Compose, Filter from hojichar.filters.document_filters import JSONLoader class AddSomething(Filter): # Concat some value after every document. def __init__(self, something: str, *args, **kwargs) -> None: self.something = something def apply(self, document): text = document.text + self.something document.text = text return document class JSONDumper(Filter): def apply(self, document): text = document.text document.text = json.dumps({"text": text}, ensure_ascii=False) return document def callback(something): return Compose( [ JSONLoader(), AddSomething(something), JSONDumper(), ] ) # FACTORY must be callable which returns Compose object. FACTORY = callbackUsing
FACTORYprofile with arguments in CLI.cat <your_file> | hojichar -p example_profile.py --args arg1 arg2hojichar.utils.load_compose.load_parametrized_filter_from_file()orload_factory_from_fileloads this type of profile.
For Developers
Installing from the Source Directory
To install the package, execute the following commands:
git clone https://github.com/HojiChar/HojiChar.git
cd HojiChar
uv sync --all-extras
Testing
Some filters incorporate doctests. You can run these tests with the command:
pytest --doctest-modules .
This command should be executed from the root of the project.
Code style
- HojiChar requires type hints for all code. Type checking is performed in continuous integration (CI) in addition to the pytest tests.
- HojiChar code is subject to inspection and formatting by the
ruffLinter. For configuration details, please refer topyproject.toml. You can perform linting and formatting from the root of the project using the following commands:
Linting
uvx ruff check .
Formatting
uvx ruff format .
Building the Documentation
We use Pdoc for building the documentation. You can build the documentation using the following command:
pdoc -o docs hojichar
Run this command from the project root.
In practice, the process of building the documentation is automated by CI. When a Pull Request is merged into the main branch, the documentation is built in the docs/ directory of the docs branch. This directory is then deployed to the official documentation site by GitHub Pages.
Creating a Source Tarball
To create a source tarball, for instance, for packaging or distribution, run the following command:
uv build
The tarball will be created in the dist directory. This command will compile the source code, and the resulting tarball can be installed with no additional dependencies other than the Python standard library.
Creating a Release and Uploading it to PyPI
Versions uploaded to PyPI are identified by git tags. The __version__ variable in __init__.py or the version entry in pyproject.toml are ignored. The uv-dynamic-versioning plugin is used to implement this process.
The steps to push to PyPI are as follows, although in actuality, the process is automated by CI when a GitHub release is created from the tag.
git checkout v0.1.2
uv build
uv publish --index testpypi --token ${PYPI_TOKEN}
The actual task for the manager is to apply the appropriate tag to the commit to be released and to create the release from GitHub:
git tag -a v0.1.2 -m "Version 0.1.2"
git push origin v0.1.2
1""" 2.. include:: ../README.md 3""" 4 5from .core.async_composition import AsyncCompose, AsyncFilterAdapter 6from .core.async_filter_interface import AsyncFilter 7from .core.composition import Compose 8from .core.filter_interface import Filter, TokenFilter 9from .core.inspection import StatsContainer 10from .core.models import Document, Token 11from .core.parallel import Parallel 12from .filters import ( 13 deduplication, 14 document_filters, 15 language_identification, 16 token_filters, 17 tokenization, 18) 19 20__version__ = "0.0.0" # Replaced by uv-dynamic-versioning when deploying 21 22__all__ = [ 23 "core", 24 "filters", 25 "utils", 26 "Compose", 27 "Filter", 28 "TokenFilter", 29 "Document", 30 "Token", 31 "Parallel", 32 "StatsContainer", 33 "deduplication", 34 "document_filters", 35 "language_identification", 36 "token_filters", 37 "tokenization", 38 "AsyncCompose", 39 "AsyncFilterAdapter", 40 "AsyncFilter", 41]
16class Compose(Filter): 17 def __init__( 18 self, 19 filters: list[Filter | TokenFilter], 20 random_state: int | np.random.Generator | None = None, 21 *args: Any, 22 **kwargs: Any, 23 ) -> None: 24 """ 25 Compose a filter from pre-defined filter-objects. 26 Filter which has `skip_rejected` flag ignores a document which has `is_rejected` flag. 27 By doing so, Compose avoid applying filters that do not affect the output. 28 29 Parameters 30 ---------- 31 filters : list[Filter | TokenFilter] 32 Filter instances which apply to the corpus. 33 34 random_state : int | np.random.Generator | None, optional 35 Default = None 36 Seed for applying filters randomly. 37 `random_state` must be int or np.random.Generator instance. 38 """ 39 super().__init__(random_state=random_state, *args, **kwargs) 40 self.set_filters(filters) 41 self.logger = logging.getLogger(f"{self.__module__}.{self.__class__.__name__}") 42 43 self._statistics.name = "Total" 44 45 def set_filters(self, filters: list[Filter | TokenFilter]) -> None: 46 """ 47 Set the filter to a Compose object. The filter is expanded if the 48 list of filters in the argument contains a filter bound by Compose. 49 50 Args: 51 filters (list[Filter | TokenFilter]): Target filters 52 """ 53 self.filters: list[Filter | TokenFilter] = [] 54 55 filter_idx = 0 56 for f in filters: 57 if isinstance(f, Compose): 58 for sub in f.filters: 59 sub._set_rng_if_not_initialized(self._rng) 60 name = f"{filter_idx}-{sub.__class__.__name__}" 61 sub.name = name 62 sub._statistics.name = name 63 self.filters.append(sub) 64 filter_idx += 1 65 else: 66 f._set_rng_if_not_initialized(self._rng) 67 name = f"{filter_idx}-{f.__class__.__name__}" 68 f.name = name 69 f._statistics.name = name 70 self.filters.append(f) 71 filter_idx += 1 72 73 def __call__(self, text: str, **kwargs: Any) -> str: 74 """ 75 Apply the composed filter to a text and return the processed text. 76 If the document is rejected, return an empty string. 77 """ 78 document = Document(text, **kwargs) 79 document = self.apply(document) 80 if document.is_rejected: 81 return "" 82 else: 83 return document.text 84 85 def apply(self, document: Document) -> Document: 86 """ 87 Apply the composed filter to a document and return the processed document. 88 """ 89 stat = get_doc_info(document) 90 for i, filt in enumerate(self.filters): 91 document = filt._apply(document) 92 new_stat = get_doc_info(document) 93 self._statistics.update_by_diff(stat, new_stat) 94 return document 95 96 def apply_batch(self, batch: Sequence[Document]) -> list[Document]: 97 """ 98 Apply the composed filter to a batch of documents and return the processed documents. 99 The `apply_batch` method implemented in sub-filters is called in order. 100 """ 101 102 stats = [get_doc_info(doc) for doc in batch] 103 for i, filt in enumerate(self.filters): 104 batch = filt._apply_batch(batch) 105 batch = self._finalize_batch(batch, stats) 106 return list(batch) 107 108 def apply_stream(self, stream: Iterable[Document]) -> Iterable[Document]: 109 """ 110 Apply the composed filter to a stream of documents and return the processed documents. 111 The `apply_stream` method implemented in sub-filters is called in order. 112 113 114 In a sub-filter, if `apply_batch` is overridden and implemented, you need to set `use_batch` 115 to True at that filter to utilize that implementation. Otherwise, the 116 method implemented in `apply` will be applied to the stream. 117 """ 118 stream = self._count_input_stats(stream) 119 for i, filt in enumerate(self.filters): 120 stream = filt.apply_stream(stream) 121 122 for doc in stream: 123 in_stat = doc._get_initial_stats() 124 if in_stat is None: 125 in_stat = get_doc_info(doc) 126 self.logger.debug( 127 "Initial stats missing for document during stream aggregation; " 128 "using current stats as fallback" 129 ) 130 out_stat = get_doc_info(doc) 131 132 self._statistics.update_by_diff(in_stat, out_stat) 133 doc._clear_initial_stats() 134 yield doc 135 136 def _count_input_stats(self, stream: Iterable[Document]) -> Iterable[Document]: 137 for doc in stream: 138 doc._set_initial_stats(get_doc_info(doc)) 139 yield doc 140 141 def get_total_statistics(self) -> list[Statistics]: 142 """ 143 Get the statistics of the Compose object and sub filters. 144 145 The statistics of the Compose class are stored in an object with the name "Total", 146 and sub-filters's are stored with names in the format {filter_index}-{filter class name}. 147 """ 148 stats = [] 149 stats.append(self.get_statistics()) 150 for i, filt in enumerate(self.filters): 151 stats.append(filt.get_statistics()) 152 return stats 153 154 def get_total_statistics_map(self) -> list[dict[str, Any]]: 155 """ 156 Get the statistics of the Compose object and sub filters as a list of dictionaries. 157 """ 158 stats = self.get_total_statistics() 159 return [stat.to_dict() for stat in stats] 160 161 def shutdown(self) -> None: 162 for f in self.filters: 163 f.shutdown() 164 165 super().shutdown() 166 167 @property 168 def statistics(self) -> dict: 169 """ 170 Deprecated 171 172 Get the statistics of the Compose object and sub filters. 173 174 This property is retained for compatibility with previous versions. 175 Please use `get_total_statistics` or `get_total_statistics_map` instead. 176 """ 177 return inspection.statistics_obj_adapter( # type: ignore 178 self.get_total_statistics() 179 ).get_human_readable_values() 180 181 @property 182 def statistics_obj(self) -> inspection.StatsContainer: 183 """ 184 Deprecated 185 186 Get the statistics of the Compose object and sub filters. 187 This method returns a StatsContainer object which contains the statistics 188 of the Compose object and sub filters. 189 190 This property is retained for compatibility with previous versions. 191 Please use `get_total_statistics` or `get_total_statistics_map` instead. 192 """ 193 return inspection.statistics_obj_adapter(self.get_total_statistics()) # type: ignore 194 195 @deprecated_since("1.0.0", "get_total_statistics") 196 def summary(self, format: str = "print") -> None: 197 info = [ 198 { 199 "layer": i, 200 "name": filt.name, 201 "doc": filt.__doc__, 202 } 203 for i, filt in enumerate(self.filters) 204 ] 205 206 def to_json(filter_info: dict) -> dict: 207 filter_info["doc"] = "".join(d.strip() for d in filter_info["doc"].split("\n")) 208 return filter_info 209 210 if format == "json": 211 print(json.dumps(list(map(to_json, info)), ensure_ascii=False, indent="\t")) 212 if format == "print": 213 for layer in info: 214 print(f"[{layer['layer']}] {layer['name']}") 215 pprint.pprint(layer["doc"])
Base class for all filters. Document-level filters must inherit from this class.
The definition of text processing is in apply method.
If you define a new filter, override the method.
When this class is called, apply the filter from string to string.
With context manager, you can use the filter as follows:
with YourFilter(p=0.5) as filt:
text = filt("This is a sample text.")
17 def __init__( 18 self, 19 filters: list[Filter | TokenFilter], 20 random_state: int | np.random.Generator | None = None, 21 *args: Any, 22 **kwargs: Any, 23 ) -> None: 24 """ 25 Compose a filter from pre-defined filter-objects. 26 Filter which has `skip_rejected` flag ignores a document which has `is_rejected` flag. 27 By doing so, Compose avoid applying filters that do not affect the output. 28 29 Parameters 30 ---------- 31 filters : list[Filter | TokenFilter] 32 Filter instances which apply to the corpus. 33 34 random_state : int | np.random.Generator | None, optional 35 Default = None 36 Seed for applying filters randomly. 37 `random_state` must be int or np.random.Generator instance. 38 """ 39 super().__init__(random_state=random_state, *args, **kwargs) 40 self.set_filters(filters) 41 self.logger = logging.getLogger(f"{self.__module__}.{self.__class__.__name__}") 42 43 self._statistics.name = "Total"
Compose a filter from pre-defined filter-objects.
Filter which has skip_rejected flag ignores a document which has is_rejected flag.
By doing so, Compose avoid applying filters that do not affect the output.
Parameters
filters : list[Filter | TokenFilter] Filter instances which apply to the corpus.
random_state : int | np.random.Generator | None, optional
Default = None
Seed for applying filters randomly.
random_state must be int or np.random.Generator instance.
45 def set_filters(self, filters: list[Filter | TokenFilter]) -> None: 46 """ 47 Set the filter to a Compose object. The filter is expanded if the 48 list of filters in the argument contains a filter bound by Compose. 49 50 Args: 51 filters (list[Filter | TokenFilter]): Target filters 52 """ 53 self.filters: list[Filter | TokenFilter] = [] 54 55 filter_idx = 0 56 for f in filters: 57 if isinstance(f, Compose): 58 for sub in f.filters: 59 sub._set_rng_if_not_initialized(self._rng) 60 name = f"{filter_idx}-{sub.__class__.__name__}" 61 sub.name = name 62 sub._statistics.name = name 63 self.filters.append(sub) 64 filter_idx += 1 65 else: 66 f._set_rng_if_not_initialized(self._rng) 67 name = f"{filter_idx}-{f.__class__.__name__}" 68 f.name = name 69 f._statistics.name = name 70 self.filters.append(f) 71 filter_idx += 1
Set the filter to a Compose object. The filter is expanded if the list of filters in the argument contains a filter bound by Compose.
Args: filters (list[Filter | TokenFilter]): Target filters
85 def apply(self, document: Document) -> Document: 86 """ 87 Apply the composed filter to a document and return the processed document. 88 """ 89 stat = get_doc_info(document) 90 for i, filt in enumerate(self.filters): 91 document = filt._apply(document) 92 new_stat = get_doc_info(document) 93 self._statistics.update_by_diff(stat, new_stat) 94 return document
Apply the composed filter to a document and return the processed document.
96 def apply_batch(self, batch: Sequence[Document]) -> list[Document]: 97 """ 98 Apply the composed filter to a batch of documents and return the processed documents. 99 The `apply_batch` method implemented in sub-filters is called in order. 100 """ 101 102 stats = [get_doc_info(doc) for doc in batch] 103 for i, filt in enumerate(self.filters): 104 batch = filt._apply_batch(batch) 105 batch = self._finalize_batch(batch, stats) 106 return list(batch)
Apply the composed filter to a batch of documents and return the processed documents.
The apply_batch method implemented in sub-filters is called in order.
108 def apply_stream(self, stream: Iterable[Document]) -> Iterable[Document]: 109 """ 110 Apply the composed filter to a stream of documents and return the processed documents. 111 The `apply_stream` method implemented in sub-filters is called in order. 112 113 114 In a sub-filter, if `apply_batch` is overridden and implemented, you need to set `use_batch` 115 to True at that filter to utilize that implementation. Otherwise, the 116 method implemented in `apply` will be applied to the stream. 117 """ 118 stream = self._count_input_stats(stream) 119 for i, filt in enumerate(self.filters): 120 stream = filt.apply_stream(stream) 121 122 for doc in stream: 123 in_stat = doc._get_initial_stats() 124 if in_stat is None: 125 in_stat = get_doc_info(doc) 126 self.logger.debug( 127 "Initial stats missing for document during stream aggregation; " 128 "using current stats as fallback" 129 ) 130 out_stat = get_doc_info(doc) 131 132 self._statistics.update_by_diff(in_stat, out_stat) 133 doc._clear_initial_stats() 134 yield doc
Apply the composed filter to a stream of documents and return the processed documents.
The apply_stream method implemented in sub-filters is called in order.
In a sub-filter, if apply_batch is overridden and implemented, you need to set use_batch
to True at that filter to utilize that implementation. Otherwise, the
method implemented in apply will be applied to the stream.
141 def get_total_statistics(self) -> list[Statistics]: 142 """ 143 Get the statistics of the Compose object and sub filters. 144 145 The statistics of the Compose class are stored in an object with the name "Total", 146 and sub-filters's are stored with names in the format {filter_index}-{filter class name}. 147 """ 148 stats = [] 149 stats.append(self.get_statistics()) 150 for i, filt in enumerate(self.filters): 151 stats.append(filt.get_statistics()) 152 return stats
Get the statistics of the Compose object and sub filters.
The statistics of the Compose class are stored in an object with the name "Total", and sub-filters's are stored with names in the format {filter_index}-{filter class name}.
154 def get_total_statistics_map(self) -> list[dict[str, Any]]: 155 """ 156 Get the statistics of the Compose object and sub filters as a list of dictionaries. 157 """ 158 stats = self.get_total_statistics() 159 return [stat.to_dict() for stat in stats]
Get the statistics of the Compose object and sub filters as a list of dictionaries.
161 def shutdown(self) -> None: 162 for f in self.filters: 163 f.shutdown() 164 165 super().shutdown()
This method is called when the filter is no longer needed. You can override this method to release resources or perform cleanup tasks.
Deprecated
Get the statistics of the Compose object and sub filters.
This property is retained for compatibility with previous versions.
Please use get_total_statistics or get_total_statistics_map instead.
Deprecated
Get the statistics of the Compose object and sub filters. This method returns a StatsContainer object which contains the statistics of the Compose object and sub filters.
This property is retained for compatibility with previous versions.
Please use get_total_statistics or get_total_statistics_map instead.
195 @deprecated_since("1.0.0", "get_total_statistics") 196 def summary(self, format: str = "print") -> None: 197 info = [ 198 { 199 "layer": i, 200 "name": filt.name, 201 "doc": filt.__doc__, 202 } 203 for i, filt in enumerate(self.filters) 204 ] 205 206 def to_json(filter_info: dict) -> dict: 207 filter_info["doc"] = "".join(d.strip() for d in filter_info["doc"].split("\n")) 208 return filter_info 209 210 if format == "json": 211 print(json.dumps(list(map(to_json, info)), ensure_ascii=False, indent="\t")) 212 if format == "print": 213 for layer in info: 214 print(f"[{layer['layer']}] {layer['name']}") 215 pprint.pprint(layer["doc"])
Inherited Members
23class Filter(ABC): 24 """ 25 Base class for all filters. 26 Document-level filters must inherit from this class. 27 28 The definition of text processing is in `apply` method. 29 If you define a new filter, override the method. 30 31 When this class is called, apply the filter from string to string. 32 33 With context manager, you can use the filter as follows: 34 ```python 35 with YourFilter(p=0.5) as filt: 36 text = filt("This is a sample text.") 37 ``` 38 39 """ 40 41 def __init__( 42 self, 43 p: float = 1.0, 44 skip_rejected: bool = True, 45 *args: Any, 46 random_state: int | np.random.Generator | None = None, 47 use_batch: bool = False, 48 batch_size: int = 128, 49 **kwargs: Any, 50 ) -> None: 51 """ 52 Initialize the filter. 53 Parameters 54 ---------- 55 p : float 56 The probability of applying the filter. 57 If `p` is 1, the filter will always be applied. 58 skip_rejected : bool 59 If `True`, the filter will skip documents that are already rejected. 60 If you want to apply the filter to all documents (e.g., postprocess), set this to `False`. 61 random_state : int | np.random.Generator | None 62 Seed for the random number generator. 63 If `None`, a new random number generator will be created. 64 If `None`, and use in the `Compose` class, the random state is shared with the `Compose` object. 65 use_batch : bool 66 If `True`, the filter will process documents in batches in the `apply_stream` method. 67 batch_size : int 68 The size of the batch to process documents in the `apply_stream` method. 69 kwargs : Any 70 Additional keyword arguments to pass to the filter. 71 """ 72 self.name = self.__class__.__name__ 73 self.logger = logging.getLogger(f"{self.__module__}.{self.__class__.__name__}") 74 assert 0 <= p <= 1 75 self.p = p 76 self.__init_rng(random_state) 77 self.skip_rejected = skip_rejected 78 self.use_batch = use_batch 79 self.batch_size = batch_size 80 81 self._statistics: Statistics = Statistics() 82 83 @abstractmethod 84 def apply(self, document: Document) -> Document: 85 """ 86 Definition of filter behavior. 87 88 The document must have a protocol `TextContent`, 89 and mostly used hojichar.Document class. 90 91 In this method, the filter will modify `document.text` or 92 `document.extras` and set `document.is_rejected = True` to discard the document. 93 94 Parameters 95 ---------- 96 document : Document 97 Input document 98 99 Returns 100 ------- 101 Document 102 Processed Document 103 """ 104 105 @deprecated_since(version="1.0.0", alternative="apply") 106 def apply_filter(self, document: Document) -> Document: 107 document = self.apply(document) 108 return document 109 110 def _check_skip(self, document: Document) -> bool: 111 """ 112 Check if the document should be skipped by this filter. 113 If `skip_rejected` is set to `True`, this method will return `True` 114 if the document is already rejected. 115 If `p` is less than 1, this method will return `True` with a probability of `1 - p`. 116 """ 117 skip = self.skip_rejected and document.is_rejected 118 if skip: 119 return True 120 if self.p < 1: 121 if self._rng.random() > self.p: 122 return True 123 return False 124 125 def _apply(self, document: Document) -> Document: 126 """ 127 Apply the filter to a single document. 128 This method 129 - checks if the document should be skipped 130 - counts and logging the statistics 131 - logging the reason for rejection if the document is rejected 132 133 This method may be used in `apply` method of `Compose` class. 134 """ 135 136 stats = get_doc_info(document) 137 138 if not self._check_skip(document): 139 document = self.apply(document) 140 141 new_stats = get_doc_info(document) 142 self._statistics.update_by_diff(stats, new_stats) 143 144 if not stats["is_rejected"] and new_stats["is_rejected"]: 145 document.reject_reason = self.get_jsonable_vars() 146 147 return document 148 149 def apply_batch(self, batch: Sequence[Document]) -> list[Document]: 150 """ 151 Apply the filter to a batch of documents. 152 You can override this method if you want to 153 apply the filter to a batch of documents at once. 154 155 This method may be used in `apply_batch` method of `Compose` class. 156 157 Parameters 158 ---------- 159 documents : Sequence[Document] 160 List-like object of input documents 161 162 Returns 163 ------- 164 list[Document] 165 List of processed documents 166 """ 167 return [self.apply(document) for document in batch] 168 169 def _apply_batch(self, batch: Sequence[Document]) -> list[Document]: 170 """ 171 Apply the filter to a batch of documents. 172 This method 173 - checks if the documents should be skipped 174 - counts and logs the statistics 175 - logs the reason for rejection if any document is rejected 176 """ 177 skip = False 178 if self.p < 1: 179 skip = self._rng.random() > self.p 180 181 stats = [get_doc_info(document=doc) for doc in batch] 182 if not skip: 183 batch = self.apply_batch(batch) 184 batch = self._finalize_batch(batch, stats) 185 return batch 186 187 def apply_stream(self, stream: Iterable[Document]) -> Iterable[Document]: 188 """ 189 Apply the filter to a stream of documents. 190 This method is used when you want to process documents one by one. 191 If `use_batch` is set to `True` in the constructor, 192 this method will process documents in batches using the `apply_batch` method. 193 194 Even if an exception occurs during processing, the process will continue, and the following actions will be taken: 195 - Set the `is_rejected` flag of the document to `True` 196 - Set the error details in `reject_reason` 197 - Increment the `errors` count in the statistics retrievable via `get_statistics` 198 199 Parameters 200 ---------- 201 stream : Iterable[Document] 202 Stream of input documents 203 204 Returns 205 ------- 206 Iterable[Document] 207 Stream of processed documents 208 """ 209 210 if not self.use_batch: 211 for document in stream: 212 yield self._try_process(document, self._apply) 213 else: 214 batch: list[Document] = [] 215 for document in stream: 216 if self._check_skip(document): 217 yield document 218 continue 219 220 batch.append(document) 221 if len(batch) >= self.batch_size: 222 stats = [get_doc_info(doc) for doc in batch] 223 batch = self._try_process(batch, self.apply_batch) 224 batch = self._finalize_batch(batch, stats) 225 yield from batch 226 batch.clear() 227 if batch: 228 stats = [get_doc_info(doc) for doc in batch] 229 batch = self._try_process(batch, self.apply_batch) 230 batch = self._finalize_batch(batch, stats) 231 yield from batch 232 233 def _try_process(self, target: T, func: Callable[[T], T]) -> T: 234 try: 235 return func(target) 236 except Exception as e: 237 if isinstance(target, Document): 238 msg = f"{e!r} occurs while processing {self.name} with {target!r}" 239 target.is_rejected = True 240 target.reject_reason = {"error": msg} 241 self._statistics.errors += 1 242 self.logger.error(msg, exc_info=True) 243 return target 244 if isinstance(target, list): 245 msg = f"{e!r} occurs while batch processing {self.name}" 246 self.logger.error(msg, exc_info=True) 247 for doc in target: 248 doc.is_rejected = True 249 doc.reject_reason = {"error": msg} 250 self._statistics.errors += len(target) 251 return target 252 else: 253 raise e 254 255 def __call__(self, text: str, **kwargs: Any) -> str: 256 document = Document(text, **kwargs) 257 document = self._apply(document) 258 return document.text 259 260 def get_statistics(self) -> Statistics: 261 """ 262 Get the statistics of this filter. 263 This method returns the statistics of the filter, 264 which includes the number of processed documents, discarded documents, and other statistics. 265 """ 266 return self._statistics 267 268 def get_statistics_map(self) -> dict[str, Any]: 269 """ 270 Get the statistics of this filter as a dictionary. 271 """ 272 return self._statistics.to_dict() 273 274 def shutdown(self) -> None: 275 """ 276 This method is called when the filter is no longer needed. 277 You can override this method to release resources or perform cleanup tasks. 278 """ 279 pass 280 281 def __enter__(self) -> "Filter": 282 return self 283 284 def __exit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None: 285 """ 286 This method is called when the filter is used in a context manager. 287 It calls the `shutdown` method to release resources or perform cleanup tasks. 288 """ 289 self.shutdown() 290 291 def get_jsonable_vars(self, exclude_keys: set[str] | None = None) -> dict[str, Any]: 292 """ 293 Get the member variable of this filter. 294 Eligible variables are primitive types; [bool, int, float, str, None], 295 and the name of the variable not starts with the underscore; `_`. 296 """ 297 if exclude_keys is None: 298 exclude_keys = set() 299 return { 300 k: v 301 for k, v in vars(self).items() 302 if (_is_jsonable(v) and (k not in exclude_keys) and (not k.startswith("_"))) 303 } 304 305 def _finalize_batch( 306 self: "Filter", 307 batch: Sequence[Document], 308 old_stats: list[dict[str, Any]] = [], 309 ) -> list[Document]: 310 new_stats = [get_doc_info(doc) for doc in batch] 311 for old, new, doc in zip(old_stats, new_stats, batch): 312 self._statistics.update_by_diff(old, new) 313 if not old["is_rejected"] and new["is_rejected"]: 314 doc.reject_reason = self.get_jsonable_vars() 315 return list(batch) 316 317 def __init_rng(self, random_state: int | np.random.Generator | None) -> None: 318 self._owns_rng = True 319 if random_state is None: 320 self._rng = np.random.default_rng() 321 self._owns_rng = False 322 elif isinstance(random_state, int): 323 self._rng = np.random.default_rng(random_state) 324 elif isinstance(random_state, np.random.Generator): 325 self._rng = random_state 326 327 def _set_rng_if_not_initialized(self, rng: np.random.Generator) -> None: 328 """ 329 Set the random number generator for this filter if it is not already initialized. 330 This method is called by Compose class. 331 """ 332 if not self._owns_rng: 333 self._rng = rng
Base class for all filters. Document-level filters must inherit from this class.
The definition of text processing is in apply method.
If you define a new filter, override the method.
When this class is called, apply the filter from string to string.
With context manager, you can use the filter as follows:
with YourFilter(p=0.5) as filt:
text = filt("This is a sample text.")
41 def __init__( 42 self, 43 p: float = 1.0, 44 skip_rejected: bool = True, 45 *args: Any, 46 random_state: int | np.random.Generator | None = None, 47 use_batch: bool = False, 48 batch_size: int = 128, 49 **kwargs: Any, 50 ) -> None: 51 """ 52 Initialize the filter. 53 Parameters 54 ---------- 55 p : float 56 The probability of applying the filter. 57 If `p` is 1, the filter will always be applied. 58 skip_rejected : bool 59 If `True`, the filter will skip documents that are already rejected. 60 If you want to apply the filter to all documents (e.g., postprocess), set this to `False`. 61 random_state : int | np.random.Generator | None 62 Seed for the random number generator. 63 If `None`, a new random number generator will be created. 64 If `None`, and use in the `Compose` class, the random state is shared with the `Compose` object. 65 use_batch : bool 66 If `True`, the filter will process documents in batches in the `apply_stream` method. 67 batch_size : int 68 The size of the batch to process documents in the `apply_stream` method. 69 kwargs : Any 70 Additional keyword arguments to pass to the filter. 71 """ 72 self.name = self.__class__.__name__ 73 self.logger = logging.getLogger(f"{self.__module__}.{self.__class__.__name__}") 74 assert 0 <= p <= 1 75 self.p = p 76 self.__init_rng(random_state) 77 self.skip_rejected = skip_rejected 78 self.use_batch = use_batch 79 self.batch_size = batch_size 80 81 self._statistics: Statistics = Statistics()
Initialize the filter.
Parameters
p : float
The probability of applying the filter.
If p is 1, the filter will always be applied.
skip_rejected : bool
If True, the filter will skip documents that are already rejected.
If you want to apply the filter to all documents (e.g., postprocess), set this to False.
random_state : int | np.random.Generator | None
Seed for the random number generator.
If None, a new random number generator will be created.
If None, and use in the Compose class, the random state is shared with the Compose object.
use_batch : bool
If True, the filter will process documents in batches in the apply_stream method.
batch_size : int
The size of the batch to process documents in the apply_stream method.
kwargs : Any
Additional keyword arguments to pass to the filter.
83 @abstractmethod 84 def apply(self, document: Document) -> Document: 85 """ 86 Definition of filter behavior. 87 88 The document must have a protocol `TextContent`, 89 and mostly used hojichar.Document class. 90 91 In this method, the filter will modify `document.text` or 92 `document.extras` and set `document.is_rejected = True` to discard the document. 93 94 Parameters 95 ---------- 96 document : Document 97 Input document 98 99 Returns 100 ------- 101 Document 102 Processed Document 103 """
Definition of filter behavior.
The document must have a protocol TextContent,
and mostly used hojichar.Document class.
In this method, the filter will modify document.text or
document.extras and set document.is_rejected = True to discard the document.
Parameters
document : Document Input document
Returns
Document Processed Document
149 def apply_batch(self, batch: Sequence[Document]) -> list[Document]: 150 """ 151 Apply the filter to a batch of documents. 152 You can override this method if you want to 153 apply the filter to a batch of documents at once. 154 155 This method may be used in `apply_batch` method of `Compose` class. 156 157 Parameters 158 ---------- 159 documents : Sequence[Document] 160 List-like object of input documents 161 162 Returns 163 ------- 164 list[Document] 165 List of processed documents 166 """ 167 return [self.apply(document) for document in batch]
Apply the filter to a batch of documents. You can override this method if you want to apply the filter to a batch of documents at once.
This method may be used in apply_batch method of Compose class.
Parameters
documents : Sequence[Document] List-like object of input documents
Returns
list[Document] List of processed documents
187 def apply_stream(self, stream: Iterable[Document]) -> Iterable[Document]: 188 """ 189 Apply the filter to a stream of documents. 190 This method is used when you want to process documents one by one. 191 If `use_batch` is set to `True` in the constructor, 192 this method will process documents in batches using the `apply_batch` method. 193 194 Even if an exception occurs during processing, the process will continue, and the following actions will be taken: 195 - Set the `is_rejected` flag of the document to `True` 196 - Set the error details in `reject_reason` 197 - Increment the `errors` count in the statistics retrievable via `get_statistics` 198 199 Parameters 200 ---------- 201 stream : Iterable[Document] 202 Stream of input documents 203 204 Returns 205 ------- 206 Iterable[Document] 207 Stream of processed documents 208 """ 209 210 if not self.use_batch: 211 for document in stream: 212 yield self._try_process(document, self._apply) 213 else: 214 batch: list[Document] = [] 215 for document in stream: 216 if self._check_skip(document): 217 yield document 218 continue 219 220 batch.append(document) 221 if len(batch) >= self.batch_size: 222 stats = [get_doc_info(doc) for doc in batch] 223 batch = self._try_process(batch, self.apply_batch) 224 batch = self._finalize_batch(batch, stats) 225 yield from batch 226 batch.clear() 227 if batch: 228 stats = [get_doc_info(doc) for doc in batch] 229 batch = self._try_process(batch, self.apply_batch) 230 batch = self._finalize_batch(batch, stats) 231 yield from batch
Apply the filter to a stream of documents.
This method is used when you want to process documents one by one.
If use_batch is set to True in the constructor,
this method will process documents in batches using the apply_batch method.
Even if an exception occurs during processing, the process will continue, and the following actions will be taken:
- Set the
is_rejectedflag of the document toTrue - Set the error details in
reject_reason - Increment the
errorscount in the statistics retrievable viaget_statistics
Parameters
stream : Iterable[Document] Stream of input documents
Returns
Iterable[Document] Stream of processed documents
260 def get_statistics(self) -> Statistics: 261 """ 262 Get the statistics of this filter. 263 This method returns the statistics of the filter, 264 which includes the number of processed documents, discarded documents, and other statistics. 265 """ 266 return self._statistics
Get the statistics of this filter. This method returns the statistics of the filter, which includes the number of processed documents, discarded documents, and other statistics.
268 def get_statistics_map(self) -> dict[str, Any]: 269 """ 270 Get the statistics of this filter as a dictionary. 271 """ 272 return self._statistics.to_dict()
Get the statistics of this filter as a dictionary.
274 def shutdown(self) -> None: 275 """ 276 This method is called when the filter is no longer needed. 277 You can override this method to release resources or perform cleanup tasks. 278 """ 279 pass
This method is called when the filter is no longer needed. You can override this method to release resources or perform cleanup tasks.
291 def get_jsonable_vars(self, exclude_keys: set[str] | None = None) -> dict[str, Any]: 292 """ 293 Get the member variable of this filter. 294 Eligible variables are primitive types; [bool, int, float, str, None], 295 and the name of the variable not starts with the underscore; `_`. 296 """ 297 if exclude_keys is None: 298 exclude_keys = set() 299 return { 300 k: v 301 for k, v in vars(self).items() 302 if (_is_jsonable(v) and (k not in exclude_keys) and (not k.startswith("_"))) 303 }
Get the member variable of this filter.
Eligible variables are primitive types; [bool, int, float, str, None],
and the name of the variable not starts with the underscore; _.
336@deprecated_since(version="1.0.0", alternative="Filter") 337class TokenFilter(Filter, ABC): 338 """ 339 Base class for token-level filters. 340 341 Token filters, which shuld be implemented in hojichar/filters/token_filters.py, 342 must inherit from this class. 343 """ 344 345 def __init__( 346 self, p: float = 1, skip_rejected: bool = True, *args: Any, **kwargs: Any 347 ) -> None: 348 self.name = self.__class__.__name__ 349 self.logger = logging.getLogger("hojichar.token_filters." + self.name) 350 assert 0 <= p <= 1 351 self.p = p 352 self.skip_rejected = skip_rejected 353 354 def apply(self, token: Token) -> Token: # type: ignore 355 raise NotImplementedError(f"{self.__class__.__name__}.apply method is not defined") 356 return token 357 358 def apply_filter(self, document: Document) -> Document: 359 document.tokens = [self.apply(token) for token in document.tokens if not token.is_rejected] 360 return document 361 362 def __call__(self, text: str) -> str: # type: ignore 363 token = Token(text) 364 token = self.apply(token) 365 return token.text 366 367 def _apply(self, document: Document) -> Document: 368 """ 369 Apply the token filter to a single document. 370 This method checks if the document should be skipped. 371 """ 372 if self.skip_rejected and document.is_rejected: 373 return document 374 return self.apply_filter(document) 375 376 def get_jsonable_vars(self, exclude_keys: set[str] | None = None) -> dict: 377 """ 378 Get the member variable of this filter. 379 Eligible variables are primitive types; [bool, int, float, str, None], 380 and the name of the variable not starts with the underscore; `_`. 381 """ 382 if exclude_keys is None: 383 exclude_keys = set() 384 return { 385 k: v 386 for k, v in vars(self).items() 387 if (_is_jsonable(v) and (k not in exclude_keys) and (not k.startswith("_"))) 388 }
Base class for token-level filters.
Token filters, which shuld be implemented in hojichar/filters/token_filters.py, must inherit from this class.
345 def __init__( 346 self, p: float = 1, skip_rejected: bool = True, *args: Any, **kwargs: Any 347 ) -> None: 348 self.name = self.__class__.__name__ 349 self.logger = logging.getLogger("hojichar.token_filters." + self.name) 350 assert 0 <= p <= 1 351 self.p = p 352 self.skip_rejected = skip_rejected
Initialize the filter.
Parameters
p : float
The probability of applying the filter.
If p is 1, the filter will always be applied.
skip_rejected : bool
If True, the filter will skip documents that are already rejected.
If you want to apply the filter to all documents (e.g., postprocess), set this to False.
random_state : int | np.random.Generator | None
Seed for the random number generator.
If None, a new random number generator will be created.
If None, and use in the Compose class, the random state is shared with the Compose object.
use_batch : bool
If True, the filter will process documents in batches in the apply_stream method.
batch_size : int
The size of the batch to process documents in the apply_stream method.
kwargs : Any
Additional keyword arguments to pass to the filter.
354 def apply(self, token: Token) -> Token: # type: ignore 355 raise NotImplementedError(f"{self.__class__.__name__}.apply method is not defined") 356 return token
Definition of filter behavior.
The document must have a protocol TextContent,
and mostly used hojichar.Document class.
In this method, the filter will modify document.text or
document.extras and set document.is_rejected = True to discard the document.
Parameters
document : Document Input document
Returns
Document Processed Document
376 def get_jsonable_vars(self, exclude_keys: set[str] | None = None) -> dict: 377 """ 378 Get the member variable of this filter. 379 Eligible variables are primitive types; [bool, int, float, str, None], 380 and the name of the variable not starts with the underscore; `_`. 381 """ 382 if exclude_keys is None: 383 exclude_keys = set() 384 return { 385 k: v 386 for k, v in vars(self).items() 387 if (_is_jsonable(v) and (k not in exclude_keys) and (not k.startswith("_"))) 388 }
Get the member variable of this filter.
Eligible variables are primitive types; [bool, int, float, str, None],
and the name of the variable not starts with the underscore; _.
Inherited Members
24class Document: 25 """ 26 Document class represents a text document with metadata. 27 It contains the text of the document, a flag indicating whether it is rejected, 28 and additional metadata stored in the `extras` dictionary. 29 30 The `tokens` attribute will be deprecated in future versions, 31 and users are encouraged to use the `extras` dictionary to store token-related information. 32 33 Attributes: 34 text (str): The text content of the document. 35 is_rejected (bool): A flag indicating whether the document is rejected. 36 extras (dict[str, Any]): A dictionary to store additional metadata about the document. 37 reject_reason (dict[str, Any]): A dictionary to store the reason for rejection. The 38 filter class and the member name and value will logged at the filter is logged here. 39 initial_stats (dict[str, Any] | None): Internal copy of the document statistics 40 captured before the pipeline mutates the document. 41 42 Next attributes will be deprecated in future versions: 43 tokens (list[Token]): A list of tokens extracted from the document. 44 """ 45 46 def __init__( 47 self, 48 text: str, 49 is_rejected: bool = False, 50 tokens: list[Token] | None = None, 51 extras: dict[str, Any] | None = None, 52 ) -> None: 53 self.text = text 54 self.__original = text 55 self.is_rejected = is_rejected 56 if tokens is None: 57 self.tokens: list[Token] = [] 58 else: 59 self.tokens = tokens 60 61 if extras is None: 62 self.extras: dict[str, Any] = {} 63 else: 64 self.extras = extras 65 66 self.reject_reason: dict[str, Any] = {} 67 self._initial_stats: dict[str, Any] | None = None 68 if "__init_stats" in self.extras: 69 self._initial_stats = self.extras.pop("__init_stats") 70 71 @property 72 def original(self) -> str: 73 return self.__original 74 75 def _set_initial_stats(self, stats: dict[str, Any]) -> None: 76 """ 77 Store the document statistics captured before the pipeline modifies the document. 78 Internal API: not intended for filter implementations. 79 """ 80 self._initial_stats = stats 81 82 def _get_initial_stats(self) -> dict[str, Any] | None: 83 """ 84 Retrieve the stored initial statistics of the document, if available. 85 Internal API: not intended for filter implementations. 86 """ 87 return self._initial_stats 88 89 def _clear_initial_stats(self) -> None: 90 """ 91 Remove the stored initial statistics. Useful after the stats are consumed. 92 Internal API: not intended for filter implementations. 93 """ 94 self._initial_stats = None 95 96 @deprecated_since("1.0.0") 97 def set_tokens(self, tokens: list[str]) -> None: 98 self.tokens = [Token(token) for token in tokens] 99 100 @deprecated_since("1.0.0") 101 def get_tokens(self) -> list[str]: 102 return [token.text for token in self.tokens] 103 104 def __str__(self) -> str: 105 return self.text 106 107 def __repr__(self) -> str: 108 return ( 109 f"Document(text={self.text!r}, is_rejected={self.is_rejected}, extras={self.extras})" # noqa 110 )
Document class represents a text document with metadata.
It contains the text of the document, a flag indicating whether it is rejected,
and additional metadata stored in the extras dictionary.
The tokens attribute will be deprecated in future versions,
and users are encouraged to use the extras dictionary to store token-related information.
Attributes: text (str): The text content of the document. is_rejected (bool): A flag indicating whether the document is rejected. extras (dict[str, Any]): A dictionary to store additional metadata about the document. reject_reason (dict[str, Any]): A dictionary to store the reason for rejection. The filter class and the member name and value will logged at the filter is logged here. initial_stats (dict[str, Any] | None): Internal copy of the document statistics captured before the pipeline mutates the document.
Next attributes will be deprecated in future versions: tokens (list[Token]): A list of tokens extracted from the document.
46 def __init__( 47 self, 48 text: str, 49 is_rejected: bool = False, 50 tokens: list[Token] | None = None, 51 extras: dict[str, Any] | None = None, 52 ) -> None: 53 self.text = text 54 self.__original = text 55 self.is_rejected = is_rejected 56 if tokens is None: 57 self.tokens: list[Token] = [] 58 else: 59 self.tokens = tokens 60 61 if extras is None: 62 self.extras: dict[str, Any] = {} 63 else: 64 self.extras = extras 65 66 self.reject_reason: dict[str, Any] = {} 67 self._initial_stats: dict[str, Any] | None = None 68 if "__init_stats" in self.extras: 69 self._initial_stats = self.extras.pop("__init_stats")
9@deprecated_since("0.1.0", "Document") 10class Token: 11 def __init__(self, text: str, is_rejected: bool = False) -> None: 12 self.text = text 13 self.__original = text 14 self.is_rejected = is_rejected 15 16 @property 17 def original(self) -> str: 18 return self.__original 19 20 def __str__(self) -> str: 21 return self.text
131class Parallel: 132 """ 133 The Parallel class provides a way to apply a hojichar.Compose filter 134 to an iterator of documents in a parallel manner using a specified 135 number of worker processes. This class should be used as a context 136 manager with a 'with' statement. 137 138 When no global start method has been configured, Parallel uses ``fork`` on 139 platforms which support it so unpicklable filters can be inherited by 140 workers. Set ``HOJICHAR_MP_START_METHOD`` to explicitly choose ``fork``, 141 ``spawn``, ``forkserver``, or ``default``. Non-fork contexts require the 142 Compose object and its filters to be picklable. 143 144 Example: 145 146 doc_iter = (hojichar.Document(d) for d in open("my_text.txt")) 147 with Parallel(my_filter, num_jobs=8) as pfilter: 148 for doc in pfilter.imap_apply(doc_iter): 149 pass # Process the filtered document as needed. 150 """ 151 152 def __init__( 153 self, 154 filter: hojichar.Compose, 155 num_jobs: int | None = None, 156 ignore_errors: bool = False, 157 ordered: bool = False, 158 max_in_flight: int | None = None, 159 ): 160 """ 161 Initializes a new instance of the Parallel class. 162 163 Args: 164 filter (hojichar.Compose): A composed filter object that specifies the 165 processing operations to apply to each document in parallel. 166 A copy of the filter is made within a 'with' statement. When the 'with' 167 block terminates,the statistical information obtained through `filter.statistics` 168 or`filter.statistics_obj` is replaced with the total value of the statistical 169 information processed within the 'with' block. 170 171 num_jobs (int | None, optional): The number of worker processes to use. 172 If None, then the number returned by os.cpu_count() is used. Defaults to None. 173 ignore_errors (bool, optional): If set to True, any exceptions thrown during 174 the processing of a document will be caught and logged, but will not 175 stop the processing of further documents. If set to False, the first 176 exception thrown will terminate the entire parallel processing operation. 177 Defaults to False. 178 ordered (bool, optional): If set to True, processed documents are yielded in 179 the same order as the input documents. If set to False, documents are 180 yielded as soon as their processing completes. Defaults to False. 181 max_in_flight (int | None, optional): Strict upper bound on the number of 182 documents drawn from the input iterator whose results have not yet been 183 handed back to the caller. A yielded document keeps its permit until the 184 caller requests the next one, so with `max_in_flight=1` the pool holds a 185 single document end to end (no pipelining). Without a bound, Pool's 186 task-handler thread drains the input as fast as the worker pipe accepts, 187 so a producer that outruns the filters can buffer a large number of 188 documents. If None, no explicit bound is applied. Defaults to None. 189 """ 190 if max_in_flight is not None and max_in_flight < 1: 191 raise ValueError("max_in_flight must be at least 1") 192 self.filter = filter 193 self.num_jobs = num_jobs 194 self.ignore_errors = ignore_errors 195 self.ordered = ordered 196 self.max_in_flight = max_in_flight 197 198 self._pool: Pool | None = None 199 self._pid_stats: dict[int, list[Statistics]] | None = None 200 self._gates: list[_InFlightGate] = [] 201 202 def __enter__(self) -> Parallel: 203 context = _get_parallel_context() 204 self._pool = context.Pool( 205 processes=self.num_jobs, 206 initializer=_init_worker, 207 initargs=(self.filter, self.ignore_errors), 208 ) 209 self._pid_stats = dict() 210 self._gates = [] 211 return self 212 213 def imap_apply(self, docs: Iterator[hojichar.Document]) -> Iterator[hojichar.Document]: 214 """ 215 Takes an iterator of Documents and applies the Compose filter to 216 each Document in a parallel manner. This is a generator method 217 that yields processed Documents. 218 219 Args: 220 docs (Iterator[hojichar.Document]): An iterator of Documents to be processed. 221 222 Raises: 223 RuntimeError: If the Parallel instance is not properly initialized. This 224 generally happens when the method is called outside of a 'with' statement. 225 Exception: If any exceptions are raised within the worker processes. 226 227 Yields: 228 Iterator[hojichar.Document]: An iterator that yields processed Documents. 229 """ 230 if self._pool is None or self._pid_stats is None: 231 raise RuntimeError( 232 "Parallel instance not properly initialized. Use within a 'with' statement." 233 ) 234 gate: _InFlightGate | None = None 235 if self.max_in_flight is not None: 236 gate = _InFlightGate(self.max_in_flight) 237 self._gates.append(gate) 238 docs = gate.feed(docs) 239 try: 240 results = ( 241 self._pool.imap(_worker, docs) 242 if self.ordered 243 else self._pool.imap_unordered(_worker, docs) 244 ) 245 for doc, pid, stat, err_msg in results: 246 self._pid_stats[pid] = stat 247 if err_msg is not None: 248 logger.error(f"Error in worker {pid}: {err_msg}") 249 # The permit is returned only once the caller has taken the 250 # document: releasing before the yield would let the feeder 251 # momentarily draw max_in_flight + 1 documents. On an abnormal 252 # exit (close/throw at the yield point) the gate is stopped 253 # *before* the permit is returned, so a feeder woken by the 254 # release always observes the stop and cannot draw again. 255 try: 256 yield doc 257 except BaseException: 258 if gate is not None: 259 gate.stop() 260 gate.release() 261 raise 262 else: 263 if gate is not None: 264 gate.release() 265 except Exception: 266 self.__exit__(None, None, None) 267 raise 268 finally: 269 if gate is not None: 270 gate.stop() 271 272 def __exit__(self, exc_type, exc_value, traceback) -> None: # type: ignore 273 # Feeders blocked on a max_in_flight gate must exit before the pool is 274 # joined, or shutdown would wait on them forever. 275 for gate in self._gates: 276 gate.stop() 277 if self._pool: 278 self._pool.terminate() 279 self._pool.join() 280 if self._pid_stats: 281 total_stats = functools.reduce( 282 lambda x, y: Statistics.add_list_of_stats(x, y), self._pid_stats.values() 283 ) 284 self.filter._statistics.update(Statistics.get_filter("Total", total_stats)) 285 for stat in total_stats: 286 for filt in self.filter.filters: 287 if stat.name == filt.name: 288 filt._statistics.update(stat) 289 break 290 291 def get_total_statistics(self) -> list[Statistics]: 292 """ 293 Returns a statistics object of the total statistical 294 values processed within the Parallel block. 295 296 Returns: 297 StatsContainer: Statistics object 298 """ 299 if self._pid_stats: 300 total_stats = functools.reduce( 301 lambda x, y: Statistics.add_list_of_stats(x, y), self._pid_stats.values() 302 ) 303 return total_stats 304 else: 305 return [] 306 307 def get_total_statistics_map(self) -> list[dict]: 308 return [stat.to_dict() for stat in self.get_total_statistics()] 309 310 @property 311 def statistics_obj(self) -> inspection.StatsContainer: 312 """ 313 Returns the statistics object of the Parallel instance. 314 This is a StatsContainer object which contains the statistics 315 of the Parallel instance and sub filters. 316 317 Returns: 318 StatsContainer: Statistics object 319 """ 320 return inspection.statistics_obj_adapter(self.get_total_statistics()) # type: ignore
The Parallel class provides a way to apply a hojichar.Compose filter to an iterator of documents in a parallel manner using a specified number of worker processes. This class should be used as a context manager with a 'with' statement.
When no global start method has been configured, Parallel uses fork on
platforms which support it so unpicklable filters can be inherited by
workers. Set HOJICHAR_MP_START_METHOD to explicitly choose fork,
spawn, forkserver, or default. Non-fork contexts require the
Compose object and its filters to be picklable.
Example:
doc_iter = (hojichar.Document(d) for d in open("my_text.txt")) with Parallel(my_filter, num_jobs=8) as pfilter: for doc in pfilter.imap_apply(doc_iter): pass # Process the filtered document as needed.
152 def __init__( 153 self, 154 filter: hojichar.Compose, 155 num_jobs: int | None = None, 156 ignore_errors: bool = False, 157 ordered: bool = False, 158 max_in_flight: int | None = None, 159 ): 160 """ 161 Initializes a new instance of the Parallel class. 162 163 Args: 164 filter (hojichar.Compose): A composed filter object that specifies the 165 processing operations to apply to each document in parallel. 166 A copy of the filter is made within a 'with' statement. When the 'with' 167 block terminates,the statistical information obtained through `filter.statistics` 168 or`filter.statistics_obj` is replaced with the total value of the statistical 169 information processed within the 'with' block. 170 171 num_jobs (int | None, optional): The number of worker processes to use. 172 If None, then the number returned by os.cpu_count() is used. Defaults to None. 173 ignore_errors (bool, optional): If set to True, any exceptions thrown during 174 the processing of a document will be caught and logged, but will not 175 stop the processing of further documents. If set to False, the first 176 exception thrown will terminate the entire parallel processing operation. 177 Defaults to False. 178 ordered (bool, optional): If set to True, processed documents are yielded in 179 the same order as the input documents. If set to False, documents are 180 yielded as soon as their processing completes. Defaults to False. 181 max_in_flight (int | None, optional): Strict upper bound on the number of 182 documents drawn from the input iterator whose results have not yet been 183 handed back to the caller. A yielded document keeps its permit until the 184 caller requests the next one, so with `max_in_flight=1` the pool holds a 185 single document end to end (no pipelining). Without a bound, Pool's 186 task-handler thread drains the input as fast as the worker pipe accepts, 187 so a producer that outruns the filters can buffer a large number of 188 documents. If None, no explicit bound is applied. Defaults to None. 189 """ 190 if max_in_flight is not None and max_in_flight < 1: 191 raise ValueError("max_in_flight must be at least 1") 192 self.filter = filter 193 self.num_jobs = num_jobs 194 self.ignore_errors = ignore_errors 195 self.ordered = ordered 196 self.max_in_flight = max_in_flight 197 198 self._pool: Pool | None = None 199 self._pid_stats: dict[int, list[Statistics]] | None = None 200 self._gates: list[_InFlightGate] = []
Initializes a new instance of the Parallel class.
Args:
filter (hojichar.Compose): A composed filter object that specifies the
processing operations to apply to each document in parallel.
A copy of the filter is made within a 'with' statement. When the 'with'
block terminates,the statistical information obtained through filter.statistics
orfilter.statistics_obj is replaced with the total value of the statistical
information processed within the 'with' block.
num_jobs (int | None, optional): The number of worker processes to use.
If None, then the number returned by os.cpu_count() is used. Defaults to None.
ignore_errors (bool, optional): If set to True, any exceptions thrown during
the processing of a document will be caught and logged, but will not
stop the processing of further documents. If set to False, the first
exception thrown will terminate the entire parallel processing operation.
Defaults to False.
ordered (bool, optional): If set to True, processed documents are yielded in
the same order as the input documents. If set to False, documents are
yielded as soon as their processing completes. Defaults to False.
max_in_flight (int | None, optional): Strict upper bound on the number of
documents drawn from the input iterator whose results have not yet been
handed back to the caller. A yielded document keeps its permit until the
caller requests the next one, so with `max_in_flight=1` the pool holds a
single document end to end (no pipelining). Without a bound, Pool's
task-handler thread drains the input as fast as the worker pipe accepts,
so a producer that outruns the filters can buffer a large number of
documents. If None, no explicit bound is applied. Defaults to None.
213 def imap_apply(self, docs: Iterator[hojichar.Document]) -> Iterator[hojichar.Document]: 214 """ 215 Takes an iterator of Documents and applies the Compose filter to 216 each Document in a parallel manner. This is a generator method 217 that yields processed Documents. 218 219 Args: 220 docs (Iterator[hojichar.Document]): An iterator of Documents to be processed. 221 222 Raises: 223 RuntimeError: If the Parallel instance is not properly initialized. This 224 generally happens when the method is called outside of a 'with' statement. 225 Exception: If any exceptions are raised within the worker processes. 226 227 Yields: 228 Iterator[hojichar.Document]: An iterator that yields processed Documents. 229 """ 230 if self._pool is None or self._pid_stats is None: 231 raise RuntimeError( 232 "Parallel instance not properly initialized. Use within a 'with' statement." 233 ) 234 gate: _InFlightGate | None = None 235 if self.max_in_flight is not None: 236 gate = _InFlightGate(self.max_in_flight) 237 self._gates.append(gate) 238 docs = gate.feed(docs) 239 try: 240 results = ( 241 self._pool.imap(_worker, docs) 242 if self.ordered 243 else self._pool.imap_unordered(_worker, docs) 244 ) 245 for doc, pid, stat, err_msg in results: 246 self._pid_stats[pid] = stat 247 if err_msg is not None: 248 logger.error(f"Error in worker {pid}: {err_msg}") 249 # The permit is returned only once the caller has taken the 250 # document: releasing before the yield would let the feeder 251 # momentarily draw max_in_flight + 1 documents. On an abnormal 252 # exit (close/throw at the yield point) the gate is stopped 253 # *before* the permit is returned, so a feeder woken by the 254 # release always observes the stop and cannot draw again. 255 try: 256 yield doc 257 except BaseException: 258 if gate is not None: 259 gate.stop() 260 gate.release() 261 raise 262 else: 263 if gate is not None: 264 gate.release() 265 except Exception: 266 self.__exit__(None, None, None) 267 raise 268 finally: 269 if gate is not None: 270 gate.stop()
Takes an iterator of Documents and applies the Compose filter to each Document in a parallel manner. This is a generator method that yields processed Documents.
Args: docs (Iterator[hojichar.Document]): An iterator of Documents to be processed.
Raises: RuntimeError: If the Parallel instance is not properly initialized. This generally happens when the method is called outside of a 'with' statement. Exception: If any exceptions are raised within the worker processes.
Yields: Iterator[hojichar.Document]: An iterator that yields processed Documents.
291 def get_total_statistics(self) -> list[Statistics]: 292 """ 293 Returns a statistics object of the total statistical 294 values processed within the Parallel block. 295 296 Returns: 297 StatsContainer: Statistics object 298 """ 299 if self._pid_stats: 300 total_stats = functools.reduce( 301 lambda x, y: Statistics.add_list_of_stats(x, y), self._pid_stats.values() 302 ) 303 return total_stats 304 else: 305 return []
Returns a statistics object of the total statistical values processed within the Parallel block.
Returns: StatsContainer: Statistics object
Returns the statistics object of the Parallel instance. This is a StatsContainer object which contains the statistics of the Parallel instance and sub filters.
Returns: StatsContainer: Statistics object
116@dataclasses.dataclass 117class StatsContainer: 118 total_info: DocStatistics 119 layers_info: dict[str, FilterStatistics] # Key of the dict is filter name. 120 121 def __add__(self, other: StatsContainer) -> StatsContainer: 122 assert self.layers_info.keys() == other.layers_info.keys(), "Layer names must match" 123 return StatsContainer( 124 self.total_info + other.total_info, 125 {k: v + other.layers_info[k] for k, v in self.layers_info.items()}, 126 ) 127 128 def get_human_readable_values(self) -> dict: 129 return { 130 "total_info": self.total_info.get_human_readable_values(), 131 "layers_info": [ 132 layer.get_human_readable_values() for layer in self.layers_info.values() 133 ], 134 } 135 136 def reset(self) -> StatsContainer: 137 self.total_info.reset 138 for layer in self.layers_info.values(): 139 layer.reset() 140 return self
78class AsyncCompose(AsyncFilter): 79 def __init__( 80 self, 81 filters: list[AsyncFilter | Filter], 82 random_state: int | np.random.Generator | None = None, 83 executor: ThreadPoolExecutor | None = None, 84 *args: Any, 85 **kwargs: Any, 86 ): 87 super().__init__(random_state=random_state, *args, **kwargs) 88 self.logger = logging.getLogger(f"{self.__module__}.{self.__class__.__name__}") 89 self._statistics.name = "Total" 90 self._has_external_executor = executor is not None 91 self._executor = executor or ThreadPoolExecutor() 92 self.set_filters(filters) 93 94 def set_filters(self, filters: list[AsyncFilter | Filter]) -> None: 95 self.filters: list[AsyncFilter] = [] 96 filter_idx = 0 97 for f in filters: 98 if isinstance(f, (AsyncCompose, Compose)): 99 for sub in f.filters: 100 name = f"{filter_idx}-{sub.__class__.__name__}" 101 if isinstance(sub, Filter): 102 name = f"{filter_idx}-{sub.__class__.__name__}" 103 sub = AsyncFilterAdapter(sub, executor=self._executor) 104 105 sub._set_rng_if_not_initialized(self._rng) 106 sub.name = name 107 sub._statistics.name = name 108 self.filters.append(sub) 109 filter_idx += 1 110 else: 111 name = f"{filter_idx}-{f.__class__.__name__}" 112 if isinstance(f, Filter): 113 name = f"{filter_idx}-{f.__class__.__name__}" 114 f = AsyncFilterAdapter(f, executor=self._executor) 115 f._set_rng_if_not_initialized(self._rng) 116 f.name = name 117 f._statistics.name = name 118 self.filters.append(f) 119 filter_idx += 1 120 121 async def apply(self, document: Document) -> Document: 122 stat = get_doc_info(document) 123 for filter_idx, filt in enumerate(self.filters): 124 document = await filt._apply(document) 125 new_stat = get_doc_info(document) 126 async with self._stats_lock: 127 self._statistics.update_by_diff(stat, new_stat) 128 return document 129 130 async def apply_batch(self, batch: Sequence[Document]) -> list[Document]: 131 stats = [get_doc_info(doc) for doc in batch] 132 for i, filt in enumerate(self.filters): 133 batch = await filt._apply_batch(batch) 134 batch = await self._finalize_batch(batch, stats) 135 return list(batch) 136 137 async def apply_stream( 138 self, 139 stream: AsyncIterable[Document] | Iterable[Document], 140 ) -> AsyncGenerator[Document, None]: 141 async_stream = handle_stream_as_async(stream, chunk_size=1000, executor=self._executor) 142 async_stream = self._count_input_stats(async_stream) 143 144 for i, filt in enumerate(self.filters): 145 async_stream = filt.apply_stream(async_stream) 146 147 async for doc in async_stream: 148 in_stat = doc._get_initial_stats() 149 if in_stat is None: 150 in_stat = get_doc_info(doc) 151 self.logger.debug( 152 "Initial stats missing for document during async stream aggregation; " 153 "using current stats as fallback" 154 ) 155 out_stat = get_doc_info(doc) 156 async with self._stats_lock: 157 self._statistics.update_by_diff(in_stat, out_stat) 158 doc._clear_initial_stats() 159 yield doc 160 161 def imap_apply( 162 self, 163 stream: AsyncIterable[Document] | Iterable[Document], 164 *, 165 buffer_size: int = 128, 166 shutdown: bool = True, 167 ) -> AsyncToSyncIterator[Document]: 168 """Synchronously consume this asynchronous pipeline. 169 170 The returned iterator runs the pipeline on a dedicated background event loop. It is 171 intended for one-shot use and shuts down the pipeline after consumption by default. 172 Use the iterator as a context manager if iteration may stop early. 173 """ 174 175 finalizer = self.shutdown if shutdown else None 176 return handle_async_stream_as_sync( 177 self.apply_stream(stream), 178 buffer_size=buffer_size, 179 finalizer=finalizer, 180 ) 181 182 async def _count_input_stats( 183 self, async_stream: AsyncIterable[Document] 184 ) -> AsyncGenerator[Document, None]: 185 async for doc in async_stream: 186 doc._set_initial_stats(get_doc_info(doc)) 187 yield doc 188 189 def get_total_statistics(self) -> list[Statistics]: 190 """ 191 Get the statistics of the Compose object and sub filters. 192 193 The statistics of the Compose class are stored in an object with the name "Total", 194 and sub-filters's are stored with names in the format {filter_index}-{filter class name}. 195 """ 196 stats = [] 197 stats.append(self.get_statistics()) 198 for i, filt in enumerate(self.filters): 199 stats.append(filt.get_statistics()) 200 return stats 201 202 def get_total_statistics_map(self) -> list[dict[str, Any]]: 203 """ 204 Get the statistics of the Compose object and sub filters as a list of dictionaries. 205 """ 206 stats = self.get_total_statistics() 207 return [stat.to_dict() for stat in stats] 208 209 @property 210 def statistics(self) -> dict: 211 """ 212 Deprecated 213 214 Get the statistics of the Compose object and sub filters. 215 216 This property is retained for compatibility with previous versions. 217 Please use `get_total_statistics` or `get_total_statistics_map` instead. 218 """ 219 return inspection.statistics_obj_adapter( # type: ignore 220 self.get_total_statistics() 221 ).get_human_readable_values() 222 223 @property 224 def statistics_obj(self) -> inspection.StatsContainer: 225 """ 226 Deprecated 227 228 Get the statistics of the AsyncCompose object and sub filters. 229 This method returns a StatsContainer object which contains the statistics 230 of the AsyncCompose object and sub filters. 231 232 This property is retained for compatibility with previous versions. 233 Please use `get_total_statistics` or `get_total_statistics_map` instead. 234 """ 235 return inspection.statistics_obj_adapter(self.get_total_statistics()) # type: ignore 236 237 async def shutdown(self) -> None: 238 for filt in self.filters: 239 await filt.shutdown() 240 if not self._has_external_executor: 241 self._executor.shutdown() 242 243 async def __aenter__(self) -> "AsyncCompose": 244 return self 245 246 async def __aexit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None: 247 await self.shutdown() 248 if exc_type is not None: 249 raise exc_value
Helper class that provides a standard way to create an ABC using inheritance.
79 def __init__( 80 self, 81 filters: list[AsyncFilter | Filter], 82 random_state: int | np.random.Generator | None = None, 83 executor: ThreadPoolExecutor | None = None, 84 *args: Any, 85 **kwargs: Any, 86 ): 87 super().__init__(random_state=random_state, *args, **kwargs) 88 self.logger = logging.getLogger(f"{self.__module__}.{self.__class__.__name__}") 89 self._statistics.name = "Total" 90 self._has_external_executor = executor is not None 91 self._executor = executor or ThreadPoolExecutor() 92 self.set_filters(filters)
Base class for asynchronous filters.
Parameters
p : float
The probability of applying the filter.
If p is 1, the filter will always be applied.
skip_rejected : bool
If True, the filter will skip documents that are already rejected.
If you want to apply the filter to all documents (e.g., postprocess), set this to False.
random_state : int | np.random.Generator | None
Seed for the random number generator.
If None is specified, the random number generator managed by the Compose class will be used.
use_batch : bool
If True, the filter will process documents in batches in the apply_stream method.
batch_size : int
The size of the batch to process documents in the apply_stream method.
When apply_batch is not overridden, this is also the maximum number of
in-flight document tasks used by the sliding-window scheduler. Ordered
processing applies backpressure after at most two windows of started but
not yet yielded documents.
ordered : bool
If True, apply_stream yields documents in input order. If False, filters
using the default apply_batch implementation yield documents as soon as their
processing completes.
94 def set_filters(self, filters: list[AsyncFilter | Filter]) -> None: 95 self.filters: list[AsyncFilter] = [] 96 filter_idx = 0 97 for f in filters: 98 if isinstance(f, (AsyncCompose, Compose)): 99 for sub in f.filters: 100 name = f"{filter_idx}-{sub.__class__.__name__}" 101 if isinstance(sub, Filter): 102 name = f"{filter_idx}-{sub.__class__.__name__}" 103 sub = AsyncFilterAdapter(sub, executor=self._executor) 104 105 sub._set_rng_if_not_initialized(self._rng) 106 sub.name = name 107 sub._statistics.name = name 108 self.filters.append(sub) 109 filter_idx += 1 110 else: 111 name = f"{filter_idx}-{f.__class__.__name__}" 112 if isinstance(f, Filter): 113 name = f"{filter_idx}-{f.__class__.__name__}" 114 f = AsyncFilterAdapter(f, executor=self._executor) 115 f._set_rng_if_not_initialized(self._rng) 116 f.name = name 117 f._statistics.name = name 118 self.filters.append(f) 119 filter_idx += 1
121 async def apply(self, document: Document) -> Document: 122 stat = get_doc_info(document) 123 for filter_idx, filt in enumerate(self.filters): 124 document = await filt._apply(document) 125 new_stat = get_doc_info(document) 126 async with self._stats_lock: 127 self._statistics.update_by_diff(stat, new_stat) 128 return document
Definition of async filter behavior.
In this method, the filter will modify document.text or
document.extras and set document.is_rejected = True to discard the document.
Parameters
document : Document Input document
Returns
Document Processed Document
130 async def apply_batch(self, batch: Sequence[Document]) -> list[Document]: 131 stats = [get_doc_info(doc) for doc in batch] 132 for i, filt in enumerate(self.filters): 133 batch = await filt._apply_batch(batch) 134 batch = await self._finalize_batch(batch, stats) 135 return list(batch)
Apply the filter to a Sequence of documents.
By default, the processing implemented in apply is executed asynchronously and concurrently.
If the filter processing can be optimized for batch processing, override this method.
137 async def apply_stream( 138 self, 139 stream: AsyncIterable[Document] | Iterable[Document], 140 ) -> AsyncGenerator[Document, None]: 141 async_stream = handle_stream_as_async(stream, chunk_size=1000, executor=self._executor) 142 async_stream = self._count_input_stats(async_stream) 143 144 for i, filt in enumerate(self.filters): 145 async_stream = filt.apply_stream(async_stream) 146 147 async for doc in async_stream: 148 in_stat = doc._get_initial_stats() 149 if in_stat is None: 150 in_stat = get_doc_info(doc) 151 self.logger.debug( 152 "Initial stats missing for document during async stream aggregation; " 153 "using current stats as fallback" 154 ) 155 out_stat = get_doc_info(doc) 156 async with self._stats_lock: 157 self._statistics.update_by_diff(in_stat, out_stat) 158 doc._clear_initial_stats() 159 yield doc
Apply the filter to a stream of documents (Iterable or AsyncIterable).
If use_batch is set to True at initialization, the filter will process documents in batches.
If the stream is not asynchronous, use handle_stream_as_async to convert it to an asynchronous stream.
Even if an exception occurs during processing, the process will continue, and the following actions will be taken:
- Set the
is_rejectedflag of the document toTrue - Set the error details in
reject_reason - Increment the
errorscount in the statistics retrievable viaget_statistics
161 def imap_apply( 162 self, 163 stream: AsyncIterable[Document] | Iterable[Document], 164 *, 165 buffer_size: int = 128, 166 shutdown: bool = True, 167 ) -> AsyncToSyncIterator[Document]: 168 """Synchronously consume this asynchronous pipeline. 169 170 The returned iterator runs the pipeline on a dedicated background event loop. It is 171 intended for one-shot use and shuts down the pipeline after consumption by default. 172 Use the iterator as a context manager if iteration may stop early. 173 """ 174 175 finalizer = self.shutdown if shutdown else None 176 return handle_async_stream_as_sync( 177 self.apply_stream(stream), 178 buffer_size=buffer_size, 179 finalizer=finalizer, 180 )
Synchronously consume this asynchronous pipeline.
The returned iterator runs the pipeline on a dedicated background event loop. It is intended for one-shot use and shuts down the pipeline after consumption by default. Use the iterator as a context manager if iteration may stop early.
189 def get_total_statistics(self) -> list[Statistics]: 190 """ 191 Get the statistics of the Compose object and sub filters. 192 193 The statistics of the Compose class are stored in an object with the name "Total", 194 and sub-filters's are stored with names in the format {filter_index}-{filter class name}. 195 """ 196 stats = [] 197 stats.append(self.get_statistics()) 198 for i, filt in enumerate(self.filters): 199 stats.append(filt.get_statistics()) 200 return stats
Get the statistics of the Compose object and sub filters.
The statistics of the Compose class are stored in an object with the name "Total", and sub-filters's are stored with names in the format {filter_index}-{filter class name}.
202 def get_total_statistics_map(self) -> list[dict[str, Any]]: 203 """ 204 Get the statistics of the Compose object and sub filters as a list of dictionaries. 205 """ 206 stats = self.get_total_statistics() 207 return [stat.to_dict() for stat in stats]
Get the statistics of the Compose object and sub filters as a list of dictionaries.
Deprecated
Get the statistics of the Compose object and sub filters.
This property is retained for compatibility with previous versions.
Please use get_total_statistics or get_total_statistics_map instead.
Deprecated
Get the statistics of the AsyncCompose object and sub filters. This method returns a StatsContainer object which contains the statistics of the AsyncCompose object and sub filters.
This property is retained for compatibility with previous versions.
Please use get_total_statistics or get_total_statistics_map instead.
237 async def shutdown(self) -> None: 238 for filt in self.filters: 239 await filt.shutdown() 240 if not self._has_external_executor: 241 self._executor.shutdown()
You can override this method to release resources or perform cleanup tasks.
Inherited Members
24class AsyncFilterAdapter(AsyncFilter): 25 """ 26 Adapter class for executing hojichar.Filter asynchronously. 27 """ 28 29 def __init__( 30 self, 31 sync_filter: Filter, 32 *args: Any, 33 executor: Executor | None = None, 34 use_batch: bool = True, 35 **kwargs: Any, 36 ): 37 """ 38 Adapter class for executing hojichar.Filter asynchronously. 39 Used to incorporate Filter into AsyncCompose. 40 41 To reduce the overhead of asynchronous context switching, 42 use_batch is set to True by default to process in batches 43 in apply_stream, regardless of the sync_filter's use_batch setting. 44 45 If performing CPU-bound and heavy processing, you can specify an executor 46 to offload the processing to the executor. However, due to Python's GIL 47 constraints, using ThreadPoolExecutor will not parallelize CPU-bound 48 processing, and the entire process will be locked. 49 50 By using ProcessPoolExecutor as the executor, it may be possible to 51 parallelize CPU-bound processing. However, for parallelizing CPU-bound 52 processing, it is recommended to use the hojichar.Parallel class to 53 parallelize synchronous Compose pipeline. 54 """ 55 super().__init__(*args, use_batch=use_batch, **kwargs) 56 self.sync_filter = sync_filter 57 self._has_external_executor = executor is not None 58 self._executor = executor or ThreadPoolExecutor() 59 self.batch_size = sync_filter.batch_size 60 61 async def apply(self, document: Document) -> Document: 62 loop = asyncio.get_running_loop() 63 return await loop.run_in_executor(self._executor, self.sync_filter.apply, document) 64 65 async def apply_batch(self, batch: Sequence[Document]) -> list[Document]: 66 loop = asyncio.get_running_loop() 67 return await loop.run_in_executor( 68 self._executor, 69 lambda: self.sync_filter.apply_batch(batch), 70 ) 71 72 async def shutdown(self) -> None: 73 self.sync_filter.shutdown() 74 if not self._has_external_executor: 75 self._executor.shutdown()
Adapter class for executing hojichar.Filter asynchronously.
29 def __init__( 30 self, 31 sync_filter: Filter, 32 *args: Any, 33 executor: Executor | None = None, 34 use_batch: bool = True, 35 **kwargs: Any, 36 ): 37 """ 38 Adapter class for executing hojichar.Filter asynchronously. 39 Used to incorporate Filter into AsyncCompose. 40 41 To reduce the overhead of asynchronous context switching, 42 use_batch is set to True by default to process in batches 43 in apply_stream, regardless of the sync_filter's use_batch setting. 44 45 If performing CPU-bound and heavy processing, you can specify an executor 46 to offload the processing to the executor. However, due to Python's GIL 47 constraints, using ThreadPoolExecutor will not parallelize CPU-bound 48 processing, and the entire process will be locked. 49 50 By using ProcessPoolExecutor as the executor, it may be possible to 51 parallelize CPU-bound processing. However, for parallelizing CPU-bound 52 processing, it is recommended to use the hojichar.Parallel class to 53 parallelize synchronous Compose pipeline. 54 """ 55 super().__init__(*args, use_batch=use_batch, **kwargs) 56 self.sync_filter = sync_filter 57 self._has_external_executor = executor is not None 58 self._executor = executor or ThreadPoolExecutor() 59 self.batch_size = sync_filter.batch_size
Adapter class for executing hojichar.Filter asynchronously. Used to incorporate Filter into AsyncCompose.
To reduce the overhead of asynchronous context switching, use_batch is set to True by default to process in batches in apply_stream, regardless of the sync_filter's use_batch setting.
If performing CPU-bound and heavy processing, you can specify an executor to offload the processing to the executor. However, due to Python's GIL constraints, using ThreadPoolExecutor will not parallelize CPU-bound processing, and the entire process will be locked.
By using ProcessPoolExecutor as the executor, it may be possible to parallelize CPU-bound processing. However, for parallelizing CPU-bound processing, it is recommended to use the hojichar.Parallel class to parallelize synchronous Compose pipeline.
61 async def apply(self, document: Document) -> Document: 62 loop = asyncio.get_running_loop() 63 return await loop.run_in_executor(self._executor, self.sync_filter.apply, document)
Definition of async filter behavior.
In this method, the filter will modify document.text or
document.extras and set document.is_rejected = True to discard the document.
Parameters
document : Document Input document
Returns
Document Processed Document
65 async def apply_batch(self, batch: Sequence[Document]) -> list[Document]: 66 loop = asyncio.get_running_loop() 67 return await loop.run_in_executor( 68 self._executor, 69 lambda: self.sync_filter.apply_batch(batch), 70 )
Apply the filter to a Sequence of documents.
By default, the processing implemented in apply is executed asynchronously and concurrently.
If the filter processing can be optimized for batch processing, override this method.
72 async def shutdown(self) -> None: 73 self.sync_filter.shutdown() 74 if not self._has_external_executor: 75 self._executor.shutdown()
You can override this method to release resources or perform cleanup tasks.
Inherited Members
26class AsyncFilter(ABC): 27 def __init__( 28 self, 29 *args: Any, 30 p: float = 1.0, 31 skip_rejected: bool = True, 32 random_state: int | np.random.Generator | None = None, 33 use_batch: bool = True, 34 batch_size: int = 128, 35 ordered: bool = True, 36 **kwargs: Any, 37 ): 38 """ 39 Base class for asynchronous filters. 40 41 Parameters 42 ---------- 43 p : float 44 The probability of applying the filter. 45 If `p` is 1, the filter will always be applied. 46 skip_rejected : bool 47 If `True`, the filter will skip documents that are already rejected. 48 If you want to apply the filter to all documents (e.g., postprocess), set this to `False`. 49 random_state : int | np.random.Generator | None 50 Seed for the random number generator. 51 If `None` is specified, the random number generator managed by the Compose class will be used. 52 use_batch : bool 53 If `True`, the filter will process documents in batches in the `apply_stream` method. 54 batch_size : int 55 The size of the batch to process documents in the `apply_stream` method. 56 When `apply_batch` is not overridden, this is also the maximum number of 57 in-flight document tasks used by the sliding-window scheduler. Ordered 58 processing applies backpressure after at most two windows of started but 59 not yet yielded documents. 60 ordered : bool 61 If `True`, `apply_stream` yields documents in input order. If `False`, filters 62 using the default `apply_batch` implementation yield documents as soon as their 63 processing completes. 64 """ 65 if batch_size < 1: 66 raise ValueError("batch_size must be at least 1") 67 68 self.name = self.__class__.__name__ 69 self.logger = logging.getLogger(f"{self.__module__}.{self.__class__.__name__}") 70 assert 0 <= p <= 1 71 self.p = p 72 self.__init_rng(random_state) 73 self.skip_rejected = skip_rejected 74 self.use_batch = use_batch 75 self.batch_size = batch_size 76 self.ordered = ordered 77 78 self._statistics = Statistics(name=self.name) 79 self._stats_lock = asyncio.Lock() 80 81 @abstractmethod 82 async def apply(self, document: Document) -> Document: 83 """ 84 Definition of async filter behavior. 85 86 In this method, the filter will modify `document.text` or 87 `document.extras` and set `document.is_rejected = True` to discard the document. 88 89 Parameters 90 ---------- 91 document : Document 92 Input document 93 94 Returns 95 ------- 96 Document 97 Processed Document 98 """ 99 pass 100 101 def _check_skip(self, document: Document) -> bool: 102 """ 103 Check if the document should be skipped by this filter. 104 If `skip_rejected` is set to `True`, this method will return `True` 105 if the document is already rejected. 106 If `p` is less than 1, this method will return `True` with a probability of `1 - p`. 107 """ 108 skip = self.skip_rejected and document.is_rejected 109 if skip: 110 return True 111 if self.p < 1: 112 if self._rng.random() > self.p: 113 return True 114 return False 115 116 async def _apply(self, document: Document) -> Document: 117 stats = get_doc_info(document) 118 if not self._check_skip(document): 119 document = await self.apply(document) 120 new_stats = get_doc_info(document) 121 async with self._stats_lock: 122 self._statistics.update_by_diff(stats, new_stats) 123 124 if not stats["is_rejected"] and new_stats["is_rejected"]: 125 document.reject_reason = self.get_jsonable_vars() 126 return document 127 128 async def apply_batch(self, batch: Sequence[Document]) -> list[Document]: 129 """ 130 Apply the filter to a Sequence of documents. 131 By default, the processing implemented in `apply` is executed asynchronously and concurrently. 132 If the filter processing can be optimized for batch processing, override this method. 133 """ 134 tasks = [self.apply(doc) for doc in batch] 135 return await asyncio.gather(*tasks) 136 137 async def _apply_batch(self, batch: Sequence[Document]) -> list[Document]: 138 skip = False 139 if self.p < 1: 140 skip = self._rng.random() > self.p 141 142 stats = [get_doc_info(doc) for doc in batch] 143 if not skip: 144 batch = await self.apply_batch(batch) 145 batch = await self._finalize_batch(batch, stats) 146 return list(batch) 147 148 async def apply_stream( 149 self, 150 stream: Iterable[Document] | AsyncIterable[Document], 151 ) -> AsyncGenerator[Document, None]: 152 """ 153 Apply the filter to a stream of documents (Iterable or AsyncIterable). 154 If use_batch is set to `True` at initialization, the filter will process documents in batches. 155 If the stream is not asynchronous, use handle_stream_as_async to convert it to an asynchronous stream. 156 157 Even if an exception occurs during processing, the process will continue, and the following actions will be taken: 158 - Set the `is_rejected` flag of the document to `True` 159 - Set the error details in `reject_reason` 160 - Increment the `errors` count in the statistics retrievable via `get_statistics` 161 """ 162 async_stream: AsyncIterable[Document] = handle_stream_as_async(stream) 163 164 if not self.use_batch: 165 async for doc in async_stream: 166 yield await self._try_process(doc, self._apply) 167 elif type(self).apply_batch is AsyncFilter.apply_batch: 168 async for doc in self._apply_stream_concurrently(async_stream): 169 yield doc 170 else: 171 batch: list[Document] = [] 172 async for doc in async_stream: 173 if self._check_skip(doc): 174 yield doc 175 continue 176 177 batch.append(doc) 178 # Batch size reached, apply batch 179 if len(batch) >= self.batch_size: 180 stats = [get_doc_info(doc) for doc in batch] 181 batch = await self._try_process(batch, self.apply_batch) 182 batch = await self._finalize_batch(batch, stats) 183 for out in batch: 184 yield out 185 batch.clear() 186 187 # Flush remaining documents in the batch 188 if batch: 189 stats = [get_doc_info(doc) for doc in batch] 190 batch = await self._try_process(batch, self.apply_batch) 191 batch = await self._finalize_batch(batch, stats) 192 for out in batch: 193 yield out 194 195 async def _apply_stream_concurrently( 196 self, 197 stream: AsyncIterable[Document], 198 ) -> AsyncGenerator[Document, None]: 199 """Apply documents with a bounded sliding window. 200 201 This path is used when a filter relies on the default per-document `apply_batch` 202 implementation. Filters that override `apply_batch` retain true batch processing. 203 """ 204 async_iterator = stream.__aiter__() 205 206 async def next_document() -> Document: 207 return await async_iterator.__anext__() 208 209 pending: dict[asyncio.Task[Document], int] = {} 210 completed_results: dict[int, Document] = {} 211 next_input_index = 0 212 next_output_index = 0 213 source_task: asyncio.Task[Document] | None = None 214 source_exhausted = False 215 216 # Keep one window of completed out-of-order results in addition to the active 217 # window. This absorbs ordinary latency variation without allowing a stalled 218 # early document to make the scheduler consume an unbounded input stream. 219 ordered_backlog_limit = self.batch_size * 2 220 221 def schedule_next_document_if_possible() -> None: 222 nonlocal source_task 223 224 if source_exhausted or source_task is not None: 225 return 226 if len(pending) >= self.batch_size: 227 return 228 if ( 229 self.ordered 230 and len(pending) + len(completed_results) >= ordered_backlog_limit 231 ): 232 return 233 source_task = asyncio.create_task(next_document()) 234 235 schedule_next_document_if_possible() 236 try: 237 while source_task is not None or pending: 238 wait_for = set(pending) 239 if source_task is not None: 240 wait_for.add(source_task) 241 done, _ = await asyncio.wait( 242 wait_for, 243 return_when=asyncio.FIRST_COMPLETED, 244 ) 245 246 if source_task is not None and source_task in done: 247 completed_source_task = source_task 248 source_task = None 249 try: 250 document = completed_source_task.result() 251 except StopAsyncIteration: 252 source_exhausted = True 253 else: 254 skip = self._check_skip(document) 255 processing_task = asyncio.create_task( 256 self._process_stream_document(document, skip=skip) 257 ) 258 pending[processing_task] = next_input_index 259 next_input_index += 1 260 261 completed_tasks = done.intersection(pending) 262 newly_completed: list[Document] = [] 263 for task in completed_tasks: 264 input_index = pending.pop(task) 265 result = task.result() 266 if self.ordered: 267 completed_results[input_index] = result 268 else: 269 newly_completed.append(result) 270 271 schedule_next_document_if_possible() 272 273 if self.ordered: 274 while next_output_index in completed_results: 275 yield completed_results.pop(next_output_index) 276 next_output_index += 1 277 else: 278 for result in newly_completed: 279 yield result 280 281 # Yielding ordered results may be what frees backlog capacity. 282 schedule_next_document_if_possible() 283 finally: 284 remaining = list(pending) 285 if source_task is not None: 286 remaining.append(source_task) 287 await self._cancel_tasks(remaining) 288 289 async def _process_stream_document(self, document: Document, *, skip: bool) -> Document: 290 if skip: 291 return document 292 293 stats = [get_doc_info(document)] 294 document = await self._try_process(document, self.apply) 295 return (await self._finalize_batch([document], stats))[0] 296 297 @staticmethod 298 async def _cancel_tasks(tasks: Iterable[asyncio.Task[Any]]) -> None: 299 tasks = list(tasks) 300 for task in tasks: 301 if not task.done(): 302 task.cancel() 303 if tasks: 304 await asyncio.gather(*tasks, return_exceptions=True) 305 306 async def _try_process(self, target: T, func: Callable[[T], Awaitable[T]]) -> T: 307 try: 308 return await func(target) 309 except Exception as e: 310 if isinstance(target, Document): 311 msg = f"{e!r} occurs while processing {self.name} with {target!r}" 312 self.logger.error(msg, exc_info=True) 313 target.is_rejected = True 314 target.reject_reason = {"error": msg} 315 async with self._stats_lock: 316 self._statistics.errors += 1 317 return target 318 if isinstance(target, list): 319 msg = f"{e!r} occurs while batch processing {self.name}" 320 self.logger.error(msg, exc_info=True) 321 for doc in target: 322 doc.is_rejected = True 323 doc.reject_reason = {"error": msg} 324 async with self._stats_lock: 325 self._statistics.errors += len(target) 326 return target 327 raise e 328 329 async def __call__(self, text: str) -> str: 330 document = Document(text=text) 331 return (await self._apply(document)).text 332 333 def get_statistics(self) -> Statistics: 334 """ 335 Get the statistics of this filter. 336 Returns: 337 Statistics: The statistics of this filter. 338 """ 339 return self._statistics 340 341 def get_statistics_map(self) -> dict[str, Statistics]: 342 """ 343 Get the statistics of this filter as a dictionary. 344 """ 345 return self._statistics.to_dict() 346 347 async def shutdown(self) -> None: 348 """ 349 You can override this method to release resources or perform cleanup tasks. 350 """ 351 pass 352 353 async def __aenter__(self) -> "AsyncFilter": 354 return self 355 356 async def __aexit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None: 357 await self.shutdown() 358 359 def get_jsonable_vars(self, exclude_keys: set[str] | None = None) -> dict[str, Any]: 360 """ 361 Get the member variable of this filter. 362 Eligible variables are primitive types; [bool, int, float, str, None], 363 and the name of the variable not starts with the underscore; `_`. 364 """ 365 if exclude_keys is None: 366 exclude_keys = set() 367 return { 368 k: v 369 for k, v in vars(self).items() 370 if (_is_jsonable(v) and (k not in exclude_keys) and (not k.startswith("_"))) 371 } 372 373 async def _finalize_batch( 374 self, 375 batch: Sequence[Document], 376 old_stats: list[dict[str, Any]], 377 ) -> list[Document]: 378 new_stats = [get_doc_info(doc) for doc in batch] 379 for old, new, doc in zip(old_stats, new_stats, batch): 380 async with self._stats_lock: 381 self._statistics.update_by_diff(old, new) 382 if not old["is_rejected"] and new["is_rejected"] and "error" not in doc.reject_reason: 383 doc.reject_reason = self.get_jsonable_vars() 384 return list(batch) 385 386 def __init_rng(self, random_state: int | np.random.Generator | None) -> None: 387 self._owns_rng = True 388 if random_state is None: 389 self._rng = np.random.default_rng() 390 self._owns_rng = False 391 elif isinstance(random_state, int): 392 self._rng = np.random.default_rng(random_state) 393 elif isinstance(random_state, np.random.Generator): 394 self._rng = random_state 395 else: 396 raise TypeError( 397 f"random_state must be int or np.random.Generator, not {type(random_state)}" 398 ) 399 400 def _set_rng_if_not_initialized(self, rng: np.random.Generator) -> None: 401 """ 402 Set the random number generator for this filter if it is not already initialized. 403 This method is called by Compose class. 404 """ 405 if not self._owns_rng: 406 self._rng = rng
Helper class that provides a standard way to create an ABC using inheritance.
27 def __init__( 28 self, 29 *args: Any, 30 p: float = 1.0, 31 skip_rejected: bool = True, 32 random_state: int | np.random.Generator | None = None, 33 use_batch: bool = True, 34 batch_size: int = 128, 35 ordered: bool = True, 36 **kwargs: Any, 37 ): 38 """ 39 Base class for asynchronous filters. 40 41 Parameters 42 ---------- 43 p : float 44 The probability of applying the filter. 45 If `p` is 1, the filter will always be applied. 46 skip_rejected : bool 47 If `True`, the filter will skip documents that are already rejected. 48 If you want to apply the filter to all documents (e.g., postprocess), set this to `False`. 49 random_state : int | np.random.Generator | None 50 Seed for the random number generator. 51 If `None` is specified, the random number generator managed by the Compose class will be used. 52 use_batch : bool 53 If `True`, the filter will process documents in batches in the `apply_stream` method. 54 batch_size : int 55 The size of the batch to process documents in the `apply_stream` method. 56 When `apply_batch` is not overridden, this is also the maximum number of 57 in-flight document tasks used by the sliding-window scheduler. Ordered 58 processing applies backpressure after at most two windows of started but 59 not yet yielded documents. 60 ordered : bool 61 If `True`, `apply_stream` yields documents in input order. If `False`, filters 62 using the default `apply_batch` implementation yield documents as soon as their 63 processing completes. 64 """ 65 if batch_size < 1: 66 raise ValueError("batch_size must be at least 1") 67 68 self.name = self.__class__.__name__ 69 self.logger = logging.getLogger(f"{self.__module__}.{self.__class__.__name__}") 70 assert 0 <= p <= 1 71 self.p = p 72 self.__init_rng(random_state) 73 self.skip_rejected = skip_rejected 74 self.use_batch = use_batch 75 self.batch_size = batch_size 76 self.ordered = ordered 77 78 self._statistics = Statistics(name=self.name) 79 self._stats_lock = asyncio.Lock()
Base class for asynchronous filters.
Parameters
p : float
The probability of applying the filter.
If p is 1, the filter will always be applied.
skip_rejected : bool
If True, the filter will skip documents that are already rejected.
If you want to apply the filter to all documents (e.g., postprocess), set this to False.
random_state : int | np.random.Generator | None
Seed for the random number generator.
If None is specified, the random number generator managed by the Compose class will be used.
use_batch : bool
If True, the filter will process documents in batches in the apply_stream method.
batch_size : int
The size of the batch to process documents in the apply_stream method.
When apply_batch is not overridden, this is also the maximum number of
in-flight document tasks used by the sliding-window scheduler. Ordered
processing applies backpressure after at most two windows of started but
not yet yielded documents.
ordered : bool
If True, apply_stream yields documents in input order. If False, filters
using the default apply_batch implementation yield documents as soon as their
processing completes.
81 @abstractmethod 82 async def apply(self, document: Document) -> Document: 83 """ 84 Definition of async filter behavior. 85 86 In this method, the filter will modify `document.text` or 87 `document.extras` and set `document.is_rejected = True` to discard the document. 88 89 Parameters 90 ---------- 91 document : Document 92 Input document 93 94 Returns 95 ------- 96 Document 97 Processed Document 98 """ 99 pass
Definition of async filter behavior.
In this method, the filter will modify document.text or
document.extras and set document.is_rejected = True to discard the document.
Parameters
document : Document Input document
Returns
Document Processed Document
128 async def apply_batch(self, batch: Sequence[Document]) -> list[Document]: 129 """ 130 Apply the filter to a Sequence of documents. 131 By default, the processing implemented in `apply` is executed asynchronously and concurrently. 132 If the filter processing can be optimized for batch processing, override this method. 133 """ 134 tasks = [self.apply(doc) for doc in batch] 135 return await asyncio.gather(*tasks)
Apply the filter to a Sequence of documents.
By default, the processing implemented in apply is executed asynchronously and concurrently.
If the filter processing can be optimized for batch processing, override this method.
148 async def apply_stream( 149 self, 150 stream: Iterable[Document] | AsyncIterable[Document], 151 ) -> AsyncGenerator[Document, None]: 152 """ 153 Apply the filter to a stream of documents (Iterable or AsyncIterable). 154 If use_batch is set to `True` at initialization, the filter will process documents in batches. 155 If the stream is not asynchronous, use handle_stream_as_async to convert it to an asynchronous stream. 156 157 Even if an exception occurs during processing, the process will continue, and the following actions will be taken: 158 - Set the `is_rejected` flag of the document to `True` 159 - Set the error details in `reject_reason` 160 - Increment the `errors` count in the statistics retrievable via `get_statistics` 161 """ 162 async_stream: AsyncIterable[Document] = handle_stream_as_async(stream) 163 164 if not self.use_batch: 165 async for doc in async_stream: 166 yield await self._try_process(doc, self._apply) 167 elif type(self).apply_batch is AsyncFilter.apply_batch: 168 async for doc in self._apply_stream_concurrently(async_stream): 169 yield doc 170 else: 171 batch: list[Document] = [] 172 async for doc in async_stream: 173 if self._check_skip(doc): 174 yield doc 175 continue 176 177 batch.append(doc) 178 # Batch size reached, apply batch 179 if len(batch) >= self.batch_size: 180 stats = [get_doc_info(doc) for doc in batch] 181 batch = await self._try_process(batch, self.apply_batch) 182 batch = await self._finalize_batch(batch, stats) 183 for out in batch: 184 yield out 185 batch.clear() 186 187 # Flush remaining documents in the batch 188 if batch: 189 stats = [get_doc_info(doc) for doc in batch] 190 batch = await self._try_process(batch, self.apply_batch) 191 batch = await self._finalize_batch(batch, stats) 192 for out in batch: 193 yield out
Apply the filter to a stream of documents (Iterable or AsyncIterable).
If use_batch is set to True at initialization, the filter will process documents in batches.
If the stream is not asynchronous, use handle_stream_as_async to convert it to an asynchronous stream.
Even if an exception occurs during processing, the process will continue, and the following actions will be taken:
- Set the
is_rejectedflag of the document toTrue - Set the error details in
reject_reason - Increment the
errorscount in the statistics retrievable viaget_statistics
333 def get_statistics(self) -> Statistics: 334 """ 335 Get the statistics of this filter. 336 Returns: 337 Statistics: The statistics of this filter. 338 """ 339 return self._statistics
Get the statistics of this filter. Returns: Statistics: The statistics of this filter.
341 def get_statistics_map(self) -> dict[str, Statistics]: 342 """ 343 Get the statistics of this filter as a dictionary. 344 """ 345 return self._statistics.to_dict()
Get the statistics of this filter as a dictionary.
347 async def shutdown(self) -> None: 348 """ 349 You can override this method to release resources or perform cleanup tasks. 350 """ 351 pass
You can override this method to release resources or perform cleanup tasks.
359 def get_jsonable_vars(self, exclude_keys: set[str] | None = None) -> dict[str, Any]: 360 """ 361 Get the member variable of this filter. 362 Eligible variables are primitive types; [bool, int, float, str, None], 363 and the name of the variable not starts with the underscore; `_`. 364 """ 365 if exclude_keys is None: 366 exclude_keys = set() 367 return { 368 k: v 369 for k, v in vars(self).items() 370 if (_is_jsonable(v) and (k not in exclude_keys) and (not k.startswith("_"))) 371 }
Get the member variable of this filter.
Eligible variables are primitive types; [bool, int, float, str, None],
and the name of the variable not starts with the underscore; _.