hojichar

HojiChar: The Text Processing Pipeline

PyPI version Python Versions CI wowkflow codecov PyPI Downloads

Official docs: https://hojichar.github.io/HojiChar/hojichar.html

Features

  • HojiChar provides a way to combine multiple arbitrary text processing tasks into a streamlined pipeline.
  • The sequence of operations can be described declaratively, ensuring portability.
  • HojiChar allows users to gather detailed statistical information from large amounts of text during processing.
  • It enables management of any Python text processing tasks, providing a Command Line Interface (CLI) capable of parallel processing.

Background and what is for HojiChar

Text preprocessing is far from a one-size-fits-all process. Depending on the data source and the specific task at hand, various steps including normalization, noise removal, and filtering may be necessary. Not all texts require the same level of preprocessing. For instance, relatively clean texts may only need minimal filtering, while "dirtier" sources like Common Crawl data often require more thorough processing. As a result, the preprocessing profile has to be tailored to each specific domain.

Many preprocessing operations can be viewed as filters, taking string as input, applying a transformation, and outputting the processed string. Even though these operations might seem straightforward individually, managing them in a multi-layered, efficient manner can be challenging.

Inspired by torchvision.transforms and iver56/audiomentations, HojiChar addresses these challenges. It enables users to define each text processing step as a class inheriting from hojichar.Filter and use hojichar.Compose to chain them together into a single filter. By writing out the Compose recipe as a profile, the preprocessing process for a specific domain's text can be made portable. Moreover, Compose automatically logs various metrics for each filter, such as byte changes, processing time, and number of rejected texts. This allows users to assess the validity of each operation and consider trade-offs between computation time and performance.

While there are other text normalization tools available, most are designed to perform a specific set of operations. Text preprocessing, despite its importance in the LLM era, is often considered a mundane task compared to machine learning or artificial intelligence tasks. As a result, many existing solutions can be ad hoc, poorly maintained, or inadequately tested. Recognizing these issues, we developed HojiChar as a robust tool for configuring text preprocessing.

For large text corpora, HojiChar also provides near-duplicate detection through GenerateDedupLSH. In our benchmark, HojiChar 0.18.0 generates MinHash LSH keys over 18× faster than Hugging Face DataTrove. The chart compares hash generation throughput with other implementations.

LSH hash generation throughput, with Datatrove as 1x

Install

Python 3.10 or later is required. CI tests Python 3.10 through 3.15 on Linux and macOS. Python 3.9 is no longer supported.

pip install hojichar

If you want to use the additional filters, install the package with the following command:

pip install 'hojichar[all]'

If you want to use AsyncChatAPI filter, install the package with the following command:

pip install 'hojichar[openai]'

If you want to use the near-deduplication (using MinHash LSH algorithm) filter, install the package with the following command:

pip install 'hojichar[dedup]'

On Python 3.15, the all and dedup extras currently build Fugashi from source. Install MeCab first: brew install mecab on macOS, or sudo apt-get install libmecab-dev on Debian/Ubuntu. This prerequisite is temporary until compatible Fugashi wheels are available; issue #109 tracks its removal.

Defining a Compose Object

The Compose class in HojiChar allows you to create a sequence of text processing filters.

from hojichar import Compose, document_filters

cleaner = Compose([
    document_filters.JSONLoader(key="text"),
    document_filters.AcceptJapanese(),
    document_filters.DocumentLengthFilter(min_doc_len=0,max_doc_len=1000),
    document_filters.ExampleHojiChar(),
    document_filters.JSONDumper()
])

When a Compose object is called, it accepts a string and returns the processed string.

>>> cleaner('{"text": "こんにちは、"}')
{"text": "こんにちは、<hojichar>"}

The filter pipeline above accomplishes the following steps:

  1. Extracts the value from the 'text' key in the JSON object.
  2. Discards the string if it's not in Japanese.
  3. Rejects any text shorter than 0 characters or longer than 1000 characters.
  4. Appends <hojichar> to the string.
  5. Outputs the processed string as JSON with the key "text".

The filters used in the pipeline are predefined filters found in hojichar.filters.

While HojiChar provides some fundamental text processing filters and plans to add more in the future, users can also define their custom filters.

Working with Metadata

JSONL corpora often store useful metadata alongside the text itself. JSONLoader and JSONDumper make it easy to keep that information with the document through the entire pipeline:

  • JSONLoader automatically merges any extras dictionary present in the input JSON into Document.extras. When you pass extra_keys=["url", "title"], those fields are copied as well, so metadata survives even if new extras are added later in the pipeline.
  • Filters can freely append to document.extras (for example, to record trace information or filter outcomes).
  • JSONDumper(export_extras=True) writes the current Document.extras back into the output JSON, making round-tripping metadata straightforward.
from hojichar import Compose, document_filters

pipeline = Compose(
    [
        document_filters.JSONLoader(extra_keys=["url"]),
        document_filters.DocumentNormalizer(),
        document_filters.JSONDumper(export_extras=True),
    ]
)

>>> pipeline('{"text": " Hello ", "extras": {"source": "blog"}, "url": "https://example.com"}')
{"text": "Hello", "extras": {"source": "blog", "url": "https://example.com"}}

In this example, the loader keeps both the embedded extras and the url field, later filters can enrich document.extras, and the dumper emits everything for downstream consumers.

User-defined Filters

A filter composing a Compose object is a class that inherits the Filter class and implements the text processing within the apply function.

from hojichar.core.filter_interface import Filter

class YourFilter(Filter):
    def apply(self, document):
        text = document.text
        """
        Write your text transformation...
        """
        document.text = text
        return document

The apply method accepts a hojichar.Document type as an argument and returns it after the transformations. The Document is a class that encapsulates a string.

The Document class can have additional metadata via the extras attribute. This allows you to associate values with the document that can be utilized in subsequent filters. Reject documents

  • The hojichar.Document has an is_rejected attribute. If a filter sets this flag to True, Compose will discard the document during processing.

Definition of __init__ for custom filter

When creating a user-defined class and applying a custom constructor, make sure to initialize the parent class.

class YourFilter(Filter):
    def __init__(self, your_param, *args, **kwargs) -> None:
        super().__init__(*args, **kwargs)
        self.your_param = your_param

    def apply(self, document):
        text = document.text
        text = process(text, self.your_param)
        document.text = text
        return document

This is because The Filter class implicitly has several arguments, one of which is p.

cleaner = Compose([
    document_filters.JSONLoader(key="text"),
    document_filters.AcceptJapanese(p=0.5),
    document_filters.JSONDumper()
])

The p argument passed to the document_filters.AcceptJapanese constructor determines the probability of applying the filter; with a probability of 1-p, it acts as an identity function. This behavior is defined in the parent class hojichar.Filter.

Batch and Stream Processing with apply_batch and apply_stream

The Filter and Compose classes support efficient batch and stream processing through the apply_batch and apply_stream methods.

apply_batch

  • The apply_batch method processes a list of Document objects in one go. By default, it applies the apply method to each document individually.
  • Users can override apply_batch in custom filters for optimized batch operations.
    class YourBatchFilter(Filter):
        def apply_batch(self, documents: Sequence[Document]) -> list[Document]:
            # Implement your batch processing logic here
            return documents
    

apply_stream

The apply_stream method processes an iterable (e.g., generator) of Document objects, ideal for large datasets or stream-based processing. If the use_batch flag is set to True in a Filter's constructor, its apply_batch implementation will be utilized during stream processing.

Example Usage:

stream = (Document(f"text {i}") for i in range(10000))
processed_stream = cleaner.apply_stream(stream)

for doc in processed_stream:
    print(doc.text)

This allows HojiChar to efficiently process massive corpora while maintaining low memory consumption.

Additional Notes on Compose

  • Even though the behavior of a Compose object when called is a text-in, text-out function, Compose itself also inherits from the Filter class. Therefore, applying the apply method to a Compose object results in hojihcar.Document class being used as input and output.
  • Compose class behaves like a Filter. If you add a Compose object as one of the filters in the constructor of Compose, the filter will be unfolded recursively.

HojiChar running asynchronously

  • HojiChar supports asynchronous processing of text data using the AsyncCompose class. This allows you to build pipelines that can handle out-of-CPU processing, such as making API calls.
  • You can define async versions of filter using the AsyncFilter class.

    from hojichar import AsyncFilter
    
    class YourAsyncFilter(AsyncFilter):
        async def apply(self, document):
            text = document.text
            # Perform asynchronous processing here
            document.text = text
            return document
    
  • The AsyncCompose class accepts both Filter and AsyncFilter objects, allowing you to mix synchronous and asynchronous filters in a single pipeline.

  • For filters that use the default per-document apply_batch, AsyncFilter.apply_stream keeps at most batch_size documents in flight and replenishes the window as each document finishes. Results preserve input order by default; set ordered=False to yield completed documents immediately when input order is not required.
  • Filters that override apply_batch continue to process complete batches, so implementations backed by a real batch API retain their batching behavior.

Synchronous tools can consume an AsyncCompose pipeline through imap_apply. The bridge owns one background event loop and shuts down the pipeline after iteration by default. Use the returned iterator as a context manager when the consumer may stop early.

with async_pipeline.imap_apply(input_doc_iter) as output_docs:
    for document in output_docs:
        print(document.text)

Example

Nowadays, text processing is enhanced by the intelligence of LLMs.

This example demonstrates how to use the AsyncChatAPI filter to process text data with OpenAI compatible APIs. This filter allows you to build high throughput of "Chain of LLMs" easily.

import os

from hojichar import AsyncCompose
from hojichar.filters.document_filters import JSONLoader, JSONDumper
from hojichar.utils.async_handlers import write_stream_to_file


async_pipeline = AsyncCompose(
    [
        JSONLoader(input_key="text"),
        AsyncChatAPI(
            model_id="gpt-4o",
            openai_endpoint_url="https://api.openai.com/v1", 
            openai_api_key=os.getenv("OPENAI_API_KEY"),
            max_concurrent_requests=128,
            output_key="llm_output",
            message_generator=lambda doc: [{"role": "user", "content": doc.text[:1000]}],
        ),
        JSONDumper(export_extras=True),
    ]
)

with open("input.jsonl") as f:
    async with async_pipeline:
        async_output_stream = (str(doc) async for doc in async_pipeline.apply_stream(f))
        await write_stream_to_file(async_output_stream, "output.jsonl", chunk_size=128) # Write async-iterable to file efficiently
  • You can use this filter by installing 'hojichar[openai]'
  • The filter works with OpenAI compatible APIs, like the endpoint hosted by vLLM. It's useful for text-augumentation tasks.
    • The AsyncChatAPI works 1K req/sec with optimized vLLM server. (We reccomend to use uvloop to get better throughput.)

Get Metrics of processing

HojiChar tracks detailed statistics at both the filter and pipeline levels, helping you monitor and debug your processing pipeline.

Each Filter (including Compose) maintains a Statistics object containing information such as input size, output size, discarded document count, and processing time.

Example: Getting Statistics from a Compose Object

stats = cleaner.get_total_statistics_map()
print(stats)
[{'cumulative_time_ns': 337250,
  'diff_bytes': 10,
  'diff_chars': 10,
  'discard_num': 0,
  'input_bytes': 45,
  'input_chars': 23,
  'input_num': 1,
  'name': 'Total',
  'output_bytes': 55,
  'output_chars': 33,
  'output_num': 1},
 {'cumulative_time_ns': 80209,
  'diff_bytes': -12,
  'diff_chars': -12,
  'discard_num': 0,
  'input_bytes': 45,
  'input_chars': 23,
  'input_num': 1,
  'name': '0-JSONLoader',
  'output_bytes': 33,
  'output_chars': 11,
  'output_num': 1},
 {'cumulative_time_ns': 17500,
  'diff_bytes': 0,
  'diff_chars': 0,
  'discard_num': 0,
  'input_bytes': 33,
  'input_chars': 11,
  'input_num': 1,
  'name': '1-AcceptJapanese',
  'output_bytes': 33,
  'output_chars': 11,
  'output_num': 1},
 {'cumulative_time_ns': 8125,
  'diff_bytes': 0,
  'diff_chars': 0,
  'discard_num': 0,
  'input_bytes': 33,
  'input_chars': 11,
  'input_num': 1,
  'name': '2-DocumentLengthFilter',
  'output_bytes': 33,
  'output_chars': 11,
  'output_num': 1},
 {'cumulative_time_ns': 6042,
  'diff_bytes': 10,
  'diff_chars': 10,
  'discard_num': 0,
  'input_bytes': 33,
  'input_chars': 11,
  'input_num': 1,
  'name': '3-ExampleHojiChar',
  'output_bytes': 43,
  'output_chars': 21,
  'output_num': 1},
 {'cumulative_time_ns': 81125,
  'diff_bytes': 12,
  'diff_chars': 12,
  'discard_num': 0,
  'input_bytes': 43,
  'input_chars': 21,
  'input_num': 1,
  'name': '4-JSONDumper',
  'output_bytes': 55,
  'output_chars': 33,
  'output_num': 1}]
  • Use get_statistics() to get the raw Statistics object for any filter.
  • Use get_total_statistics() to get a list of statistics for all filters in a Compose pipeline.
  • Use get_total_statistics_map() to retrieve the statistics as a list of dicts.

These tools allow granular monitoring of how each filter contributes to data reduction, rejection, or transformation.

Parallel application of Compose

The hojichar.Parallel class allows for the application of Compose to an iterable of Document concurrently. This class empowers users to process vast collections of documents by harnessing the power of multiple CPU cores.

Example usage of Parallel class to proces a very large JSON Lines file concurrently.

import hojichar

input_file = "your_text.jsonl"
input_doc_iter = (hojichar.Document(line) for line in open(input_file))

cleaner = hojichar.Compose([
    hojichar.document_filters.JSONLoader(),
    hojichar.document_filters.DocumentNormalizer(),
    # Insert your filters
    hojichar.document_filters.JSONDumper(),
])

with hojichar.Parallel(cleaner, num_jobs=10) as pfilter:
    out_doc_iter = pfilter.imap_apply(input_doc_iter)
    with open("your_processed_text.jsonl", "w") as fp:
        for doc in out_doc_iter:
            fp.write(doc.text + "\n")
  • Always use the Parallel class within a with statement.
  • Parallel.imap_apply(doc_iter) processes an iterator of Document and returns an iterator of the processed documents.
  • By default, processed documents are yielded as soon as they complete. Pass ordered=True to Parallel to yield them in the same order as the input documents.
  • If no start method is configured, Parallel uses fork so workers can inherit unpicklable filters. Set HOJICHAR_MP_START_METHOD to explicitly choose another context.
  • For additional options and details about the Parallel class, please refer to the official documentation.

CLI tool and preprocessing profile

  • HojiChar provides CLI tools for text preprocess pipeline.
  • User defines a series of preprocessing into a python file as profile.

  • Example:

    cat <your_text.jsonl> | hojichar -p your_preprocessing_profile.py -o your_text_preprocessed.jsonl
    
  • hojichar --help

    usage: hojichar [-h] --profile <profile.py> [--args ARGS [ARGS ...]] [--output OUTPUT] [--input INPUT] [--dump-stats <path to stats.json>] [--exit-on-error] [--all] [--jobs JOBS]
    
    options:
    -h, --help            show this help message and exit
    --profile <profile.py>, -p <profile.py>
                            Path to a Python file that implements your custom filter.
    --args ARGS [ARGS ...]
                            Pass additional arguments to the profile. Use it like `--args arg1 arg2` etc. The arguments should be space-separated.
    --output OUTPUT, -o OUTPUT
                            Specifies the path for the output file. Defaults to standard output.
    --input INPUT, -i INPUT
                            Specifies the path for the input file. Defaults to standard input. If set this path, the progress bar is enabled.
    --dump-stats <path to stats.json>
                            Dump statistics to file. If the file exists, it will be appended.
    --exit-on-error       Exit if an exception occurs during filtering. Useful for debugging custom filters.
    --all                 A flag that specifies whether to include discarded samples. This is useful when inspecting discarded samples.
    --jobs JOBS, -j JOBS  The number ob parallel jobs. By default, the nuber of the CPU core.
    

Definition of Profile

  • HojiChar CLI receives a series of preprocessing as a profile.
  • The preprocessing profile is provided as a Python file. Two patterns of the file are allowed.
  • hojichar.utils.load_compose.load_compose() loads these profile.

FILTER profile

  • hojichar.Compose must be defined as FILTER variable.
  • Example.

    import json
    
    from hojichar import Compose, Filter
    from hojichar.filters.document_filters import ExampleHojiChar, JSONLoader
    
    
    class JSONDumper(Filter):
        def apply(self, document):
            text = document.text
            document.text = json.dumps({"text": text}, ensure_ascii=False)
            return document
    
    # FILTER must define Compose object.
    FILTER = Compose(
        [
            JSONLoader(),
            ExampleHojiChar(),
            JSONDumper(),
        ]
    )
    
    • Pass the texts to the filter you have defined using a pipe as follows.
      cat <your_file> | hojichar -p example_profile.py
      
  • hojichar.utils.load_compose.load_filter_from_file() loads this type of profile.

FACTORY profile

  • A callable function that returns hojichar.Compose must be defined as FACTORY variable.
  • The callable can receive arguments. In this way, parameters can be passed to the profile.
    • Some kinds of value are not preferred to static. For example, random seeds and some flags modify the behavior of a filter, etc
    • FACTORY provides a mechanism to pass those values as arguments to the preprocessing.
  • Example.

    import json
    
    from hojichar import Compose, Filter
    from hojichar.filters.document_filters import JSONLoader
    
    
    class AddSomething(Filter): #  Concat some value after every document.
      def __init__(self, something: str, *args, **kwargs) -> None:
          self.something = something
    
      def apply(self, document):
          text = document.text + self.something
          document.text = text
          return document
    
    class JSONDumper(Filter):
      def apply(self, document):
          text = document.text
          document.text = json.dumps({"text": text}, ensure_ascii=False)
          return document
    
    
    def callback(something):
      return Compose(
          [
              JSONLoader(),
              AddSomething(something),
              JSONDumper(),
          ]
      )
    
    # FACTORY must be callable which returns Compose object.
    FACTORY = callback
    
  • Using FACTORY profile with arguments in CLI.

    cat <your_file> | hojichar -p example_profile.py --args arg1 arg2
    
  • hojichar.utils.load_compose.load_parametrized_filter_from_file() or load_factory_from_file loads this type of profile.

For Developers

Installing from the Source Directory

To install the package, execute the following commands:

git clone https://github.com/HojiChar/HojiChar.git
cd HojiChar
uv sync --all-extras

Testing

Some filters incorporate doctests. You can run these tests with the command:

pytest --doctest-modules .

This command should be executed from the root of the project.

Code style

  • HojiChar requires type hints for all code. Type checking is performed in continuous integration (CI) in addition to the pytest tests.
  • HojiChar code is subject to inspection and formatting by the ruff Linter. For configuration details, please refer to pyproject.toml. You can perform linting and formatting from the root of the project using the following commands:

Linting

uvx ruff check .

Formatting

uvx ruff format .

Building the Documentation

We use Pdoc for building the documentation. You can build the documentation using the following command:

pdoc -o docs hojichar

Run this command from the project root.

In practice, the process of building the documentation is automated by CI. When a Pull Request is merged into the main branch, the documentation is built in the docs/ directory of the docs branch. This directory is then deployed to the official documentation site by GitHub Pages.

Creating a Source Tarball

To create a source tarball, for instance, for packaging or distribution, run the following command:

uv build

The tarball will be created in the dist directory. This command will compile the source code, and the resulting tarball can be installed with no additional dependencies other than the Python standard library.

Creating a Release and Uploading it to PyPI

Versions uploaded to PyPI are identified by git tags. The __version__ variable in __init__.py or the version entry in pyproject.toml are ignored. The uv-dynamic-versioning plugin is used to implement this process.

The steps to push to PyPI are as follows, although in actuality, the process is automated by CI when a GitHub release is created from the tag.

git checkout v0.1.2
uv build
uv publish --index testpypi --token ${PYPI_TOKEN}

The actual task for the manager is to apply the appropriate tag to the commit to be released and to create the release from GitHub:

git tag -a v0.1.2 -m "Version 0.1.2"
git push origin v0.1.2
 1"""
 2.. include:: ../README.md
 3"""
 4
 5from .core.async_composition import AsyncCompose, AsyncFilterAdapter
 6from .core.async_filter_interface import AsyncFilter
 7from .core.composition import Compose
 8from .core.filter_interface import Filter, TokenFilter
 9from .core.inspection import StatsContainer
10from .core.models import Document, Token
11from .core.parallel import Parallel
12from .filters import (
13    deduplication,
14    document_filters,
15    language_identification,
16    token_filters,
17    tokenization,
18)
19
20__version__ = "0.0.0"  # Replaced by uv-dynamic-versioning when deploying
21
22__all__ = [
23    "core",
24    "filters",
25    "utils",
26    "Compose",
27    "Filter",
28    "TokenFilter",
29    "Document",
30    "Token",
31    "Parallel",
32    "StatsContainer",
33    "deduplication",
34    "document_filters",
35    "language_identification",
36    "token_filters",
37    "tokenization",
38    "AsyncCompose",
39    "AsyncFilterAdapter",
40    "AsyncFilter",
41]
class Compose(hojichar.Filter):
 16class Compose(Filter):
 17    def __init__(
 18        self,
 19        filters: list[Filter | TokenFilter],
 20        random_state: int | np.random.Generator | None = None,
 21        *args: Any,
 22        **kwargs: Any,
 23    ) -> None:
 24        """
 25        Compose a filter from pre-defined filter-objects.
 26        Filter which has `skip_rejected` flag ignores a document which has `is_rejected` flag.
 27        By doing so, Compose avoid applying filters that do not affect the output.
 28
 29        Parameters
 30        ----------
 31        filters : list[Filter | TokenFilter]
 32            Filter instances which apply to the corpus.
 33
 34        random_state : int | np.random.Generator | None, optional
 35            Default = None
 36            Seed for applying filters randomly.
 37            `random_state` must be int or np.random.Generator instance.
 38        """
 39        super().__init__(random_state=random_state, *args, **kwargs)
 40        self.set_filters(filters)
 41        self.logger = logging.getLogger(f"{self.__module__}.{self.__class__.__name__}")
 42
 43        self._statistics.name = "Total"
 44
 45    def set_filters(self, filters: list[Filter | TokenFilter]) -> None:
 46        """
 47        Set the filter to a Compose object. The filter is expanded if the
 48        list of filters in the argument contains a filter bound by Compose.
 49
 50        Args:
 51            filters (list[Filter | TokenFilter]): Target filters
 52        """
 53        self.filters: list[Filter | TokenFilter] = []
 54
 55        filter_idx = 0
 56        for f in filters:
 57            if isinstance(f, Compose):
 58                for sub in f.filters:
 59                    sub._set_rng_if_not_initialized(self._rng)
 60                    name = f"{filter_idx}-{sub.__class__.__name__}"
 61                    sub.name = name
 62                    sub._statistics.name = name
 63                    self.filters.append(sub)
 64                    filter_idx += 1
 65            else:
 66                f._set_rng_if_not_initialized(self._rng)
 67                name = f"{filter_idx}-{f.__class__.__name__}"
 68                f.name = name
 69                f._statistics.name = name
 70                self.filters.append(f)
 71                filter_idx += 1
 72
 73    def __call__(self, text: str, **kwargs: Any) -> str:
 74        """
 75        Apply the composed filter to a text and return the processed text.
 76        If the document is rejected, return an empty string.
 77        """
 78        document = Document(text, **kwargs)
 79        document = self.apply(document)
 80        if document.is_rejected:
 81            return ""
 82        else:
 83            return document.text
 84
 85    def apply(self, document: Document) -> Document:
 86        """
 87        Apply the composed filter to a document and return the processed document.
 88        """
 89        stat = get_doc_info(document)
 90        for i, filt in enumerate(self.filters):
 91            document = filt._apply(document)
 92        new_stat = get_doc_info(document)
 93        self._statistics.update_by_diff(stat, new_stat)
 94        return document
 95
 96    def apply_batch(self, batch: Sequence[Document]) -> list[Document]:
 97        """
 98        Apply the composed filter to a batch of documents and return the processed documents.
 99        The `apply_batch` method implemented in sub-filters is called in order.
100        """
101
102        stats = [get_doc_info(doc) for doc in batch]
103        for i, filt in enumerate(self.filters):
104            batch = filt._apply_batch(batch)
105        batch = self._finalize_batch(batch, stats)
106        return list(batch)
107
108    def apply_stream(self, stream: Iterable[Document]) -> Iterable[Document]:
109        """
110        Apply the composed filter to a stream of documents and return the processed documents.
111        The `apply_stream` method implemented in sub-filters is called in order.
112
113
114        In a sub-filter, if `apply_batch` is overridden and implemented, you need to set `use_batch`
115        to True at that filter to utilize that implementation. Otherwise, the
116        method implemented in `apply` will be applied to the stream.
117        """
118        stream = self._count_input_stats(stream)
119        for i, filt in enumerate(self.filters):
120            stream = filt.apply_stream(stream)
121
122        for doc in stream:
123            in_stat = doc._get_initial_stats()
124            if in_stat is None:
125                in_stat = get_doc_info(doc)
126                self.logger.debug(
127                    "Initial stats missing for document during stream aggregation; "
128                    "using current stats as fallback"
129                )
130            out_stat = get_doc_info(doc)
131
132            self._statistics.update_by_diff(in_stat, out_stat)
133            doc._clear_initial_stats()
134            yield doc
135
136    def _count_input_stats(self, stream: Iterable[Document]) -> Iterable[Document]:
137        for doc in stream:
138            doc._set_initial_stats(get_doc_info(doc))
139            yield doc
140
141    def get_total_statistics(self) -> list[Statistics]:
142        """
143        Get the statistics of the Compose object and sub filters.
144
145        The statistics of the Compose class are stored in an object with the name "Total",
146        and sub-filters's are stored with names in the format {filter_index}-{filter class name}.
147        """
148        stats = []
149        stats.append(self.get_statistics())
150        for i, filt in enumerate(self.filters):
151            stats.append(filt.get_statistics())
152        return stats
153
154    def get_total_statistics_map(self) -> list[dict[str, Any]]:
155        """
156        Get the statistics of the Compose object and sub filters as a list of dictionaries.
157        """
158        stats = self.get_total_statistics()
159        return [stat.to_dict() for stat in stats]
160
161    def shutdown(self) -> None:
162        for f in self.filters:
163            f.shutdown()
164
165        super().shutdown()
166
167    @property
168    def statistics(self) -> dict:
169        """
170        Deprecated
171
172        Get the statistics of the Compose object and sub filters.
173
174        This property is retained for compatibility with previous versions.
175        Please use `get_total_statistics` or `get_total_statistics_map` instead.
176        """
177        return inspection.statistics_obj_adapter(  # type: ignore
178            self.get_total_statistics()
179        ).get_human_readable_values()
180
181    @property
182    def statistics_obj(self) -> inspection.StatsContainer:
183        """
184        Deprecated
185
186        Get the statistics of the Compose object and sub filters.
187        This method returns a StatsContainer object which contains the statistics
188        of the Compose object and sub filters.
189
190        This property is retained for compatibility with previous versions.
191        Please use `get_total_statistics` or `get_total_statistics_map` instead.
192        """
193        return inspection.statistics_obj_adapter(self.get_total_statistics())  # type: ignore
194
195    @deprecated_since("1.0.0", "get_total_statistics")
196    def summary(self, format: str = "print") -> None:
197        info = [
198            {
199                "layer": i,
200                "name": filt.name,
201                "doc": filt.__doc__,
202            }
203            for i, filt in enumerate(self.filters)
204        ]
205
206        def to_json(filter_info: dict) -> dict:
207            filter_info["doc"] = "".join(d.strip() for d in filter_info["doc"].split("\n"))
208            return filter_info
209
210        if format == "json":
211            print(json.dumps(list(map(to_json, info)), ensure_ascii=False, indent="\t"))
212        if format == "print":
213            for layer in info:
214                print(f"[{layer['layer']}] {layer['name']}")
215                pprint.pprint(layer["doc"])

Base class for all filters. Document-level filters must inherit from this class.

The definition of text processing is in apply method. If you define a new filter, override the method.

When this class is called, apply the filter from string to string.

With context manager, you can use the filter as follows:

with YourFilter(p=0.5) as filt:
    text = filt("This is a sample text.")
Compose( filters: list[hojichar.Filter | hojichar.TokenFilter], random_state: int | numpy.random._generator.Generator | None = None, *args: Any, **kwargs: Any)
17    def __init__(
18        self,
19        filters: list[Filter | TokenFilter],
20        random_state: int | np.random.Generator | None = None,
21        *args: Any,
22        **kwargs: Any,
23    ) -> None:
24        """
25        Compose a filter from pre-defined filter-objects.
26        Filter which has `skip_rejected` flag ignores a document which has `is_rejected` flag.
27        By doing so, Compose avoid applying filters that do not affect the output.
28
29        Parameters
30        ----------
31        filters : list[Filter | TokenFilter]
32            Filter instances which apply to the corpus.
33
34        random_state : int | np.random.Generator | None, optional
35            Default = None
36            Seed for applying filters randomly.
37            `random_state` must be int or np.random.Generator instance.
38        """
39        super().__init__(random_state=random_state, *args, **kwargs)
40        self.set_filters(filters)
41        self.logger = logging.getLogger(f"{self.__module__}.{self.__class__.__name__}")
42
43        self._statistics.name = "Total"

Compose a filter from pre-defined filter-objects. Filter which has skip_rejected flag ignores a document which has is_rejected flag. By doing so, Compose avoid applying filters that do not affect the output.

Parameters

filters : list[Filter | TokenFilter] Filter instances which apply to the corpus.

random_state : int | np.random.Generator | None, optional Default = None Seed for applying filters randomly. random_state must be int or np.random.Generator instance.

def set_filters( self, filters: list[hojichar.Filter | hojichar.TokenFilter]) -> None:
45    def set_filters(self, filters: list[Filter | TokenFilter]) -> None:
46        """
47        Set the filter to a Compose object. The filter is expanded if the
48        list of filters in the argument contains a filter bound by Compose.
49
50        Args:
51            filters (list[Filter | TokenFilter]): Target filters
52        """
53        self.filters: list[Filter | TokenFilter] = []
54
55        filter_idx = 0
56        for f in filters:
57            if isinstance(f, Compose):
58                for sub in f.filters:
59                    sub._set_rng_if_not_initialized(self._rng)
60                    name = f"{filter_idx}-{sub.__class__.__name__}"
61                    sub.name = name
62                    sub._statistics.name = name
63                    self.filters.append(sub)
64                    filter_idx += 1
65            else:
66                f._set_rng_if_not_initialized(self._rng)
67                name = f"{filter_idx}-{f.__class__.__name__}"
68                f.name = name
69                f._statistics.name = name
70                self.filters.append(f)
71                filter_idx += 1

Set the filter to a Compose object. The filter is expanded if the list of filters in the argument contains a filter bound by Compose.

Args: filters (list[Filter | TokenFilter]): Target filters

def apply( self, document: hojichar.Document) -> hojichar.Document:
85    def apply(self, document: Document) -> Document:
86        """
87        Apply the composed filter to a document and return the processed document.
88        """
89        stat = get_doc_info(document)
90        for i, filt in enumerate(self.filters):
91            document = filt._apply(document)
92        new_stat = get_doc_info(document)
93        self._statistics.update_by_diff(stat, new_stat)
94        return document

Apply the composed filter to a document and return the processed document.

def apply_batch( self, batch: collections.abc.Sequence[hojichar.Document]) -> list[hojichar.Document]:
 96    def apply_batch(self, batch: Sequence[Document]) -> list[Document]:
 97        """
 98        Apply the composed filter to a batch of documents and return the processed documents.
 99        The `apply_batch` method implemented in sub-filters is called in order.
100        """
101
102        stats = [get_doc_info(doc) for doc in batch]
103        for i, filt in enumerate(self.filters):
104            batch = filt._apply_batch(batch)
105        batch = self._finalize_batch(batch, stats)
106        return list(batch)

Apply the composed filter to a batch of documents and return the processed documents. The apply_batch method implemented in sub-filters is called in order.

def apply_stream( self, stream: collections.abc.Iterable[hojichar.Document]) -> collections.abc.Iterable[hojichar.Document]:
108    def apply_stream(self, stream: Iterable[Document]) -> Iterable[Document]:
109        """
110        Apply the composed filter to a stream of documents and return the processed documents.
111        The `apply_stream` method implemented in sub-filters is called in order.
112
113
114        In a sub-filter, if `apply_batch` is overridden and implemented, you need to set `use_batch`
115        to True at that filter to utilize that implementation. Otherwise, the
116        method implemented in `apply` will be applied to the stream.
117        """
118        stream = self._count_input_stats(stream)
119        for i, filt in enumerate(self.filters):
120            stream = filt.apply_stream(stream)
121
122        for doc in stream:
123            in_stat = doc._get_initial_stats()
124            if in_stat is None:
125                in_stat = get_doc_info(doc)
126                self.logger.debug(
127                    "Initial stats missing for document during stream aggregation; "
128                    "using current stats as fallback"
129                )
130            out_stat = get_doc_info(doc)
131
132            self._statistics.update_by_diff(in_stat, out_stat)
133            doc._clear_initial_stats()
134            yield doc

Apply the composed filter to a stream of documents and return the processed documents. The apply_stream method implemented in sub-filters is called in order.

In a sub-filter, if apply_batch is overridden and implemented, you need to set use_batch to True at that filter to utilize that implementation. Otherwise, the method implemented in apply will be applied to the stream.

def get_total_statistics(self) -> list[hojichar.core.models.Statistics]:
141    def get_total_statistics(self) -> list[Statistics]:
142        """
143        Get the statistics of the Compose object and sub filters.
144
145        The statistics of the Compose class are stored in an object with the name "Total",
146        and sub-filters's are stored with names in the format {filter_index}-{filter class name}.
147        """
148        stats = []
149        stats.append(self.get_statistics())
150        for i, filt in enumerate(self.filters):
151            stats.append(filt.get_statistics())
152        return stats

Get the statistics of the Compose object and sub filters.

The statistics of the Compose class are stored in an object with the name "Total", and sub-filters's are stored with names in the format {filter_index}-{filter class name}.

def get_total_statistics_map(self) -> list[dict[str, typing.Any]]:
154    def get_total_statistics_map(self) -> list[dict[str, Any]]:
155        """
156        Get the statistics of the Compose object and sub filters as a list of dictionaries.
157        """
158        stats = self.get_total_statistics()
159        return [stat.to_dict() for stat in stats]

Get the statistics of the Compose object and sub filters as a list of dictionaries.

def shutdown(self) -> None:
161    def shutdown(self) -> None:
162        for f in self.filters:
163            f.shutdown()
164
165        super().shutdown()

This method is called when the filter is no longer needed. You can override this method to release resources or perform cleanup tasks.

statistics: dict

Deprecated

Get the statistics of the Compose object and sub filters.

This property is retained for compatibility with previous versions. Please use get_total_statistics or get_total_statistics_map instead.

statistics_obj: hojichar.StatsContainer

Deprecated

Get the statistics of the Compose object and sub filters. This method returns a StatsContainer object which contains the statistics of the Compose object and sub filters.

This property is retained for compatibility with previous versions. Please use get_total_statistics or get_total_statistics_map instead.

@deprecated_since('1.0.0', 'get_total_statistics')
def summary(self, format: str = 'print') -> None:
195    @deprecated_since("1.0.0", "get_total_statistics")
196    def summary(self, format: str = "print") -> None:
197        info = [
198            {
199                "layer": i,
200                "name": filt.name,
201                "doc": filt.__doc__,
202            }
203            for i, filt in enumerate(self.filters)
204        ]
205
206        def to_json(filter_info: dict) -> dict:
207            filter_info["doc"] = "".join(d.strip() for d in filter_info["doc"].split("\n"))
208            return filter_info
209
210        if format == "json":
211            print(json.dumps(list(map(to_json, info)), ensure_ascii=False, indent="\t"))
212        if format == "print":
213            for layer in info:
214                print(f"[{layer['layer']}] {layer['name']}")
215                pprint.pprint(layer["doc"])
class Filter(abc.ABC):
 23class Filter(ABC):
 24    """
 25    Base class for all filters.
 26    Document-level filters must inherit from this class.
 27
 28    The definition of text processing is in `apply` method.
 29    If you define a new filter, override the method.
 30
 31    When this class is called, apply the filter from string to string.
 32
 33    With context manager, you can use the filter as follows:
 34    ```python
 35    with YourFilter(p=0.5) as filt:
 36        text = filt("This is a sample text.")
 37    ```
 38
 39    """
 40
 41    def __init__(
 42        self,
 43        p: float = 1.0,
 44        skip_rejected: bool = True,
 45        *args: Any,
 46        random_state: int | np.random.Generator | None = None,
 47        use_batch: bool = False,
 48        batch_size: int = 128,
 49        **kwargs: Any,
 50    ) -> None:
 51        """
 52        Initialize the filter.
 53        Parameters
 54        ----------
 55        p : float
 56            The probability of applying the filter.
 57            If `p` is 1, the filter will always be applied.
 58        skip_rejected : bool
 59            If `True`, the filter will skip documents that are already rejected.
 60            If you want to apply the filter to all documents (e.g., postprocess), set this to `False`.
 61        random_state : int | np.random.Generator | None
 62            Seed for the random number generator.
 63            If `None`, a new random number generator will be created.
 64            If `None`, and use in the `Compose` class, the random state is shared with the `Compose` object.
 65        use_batch : bool
 66            If `True`, the filter will process documents in batches in the `apply_stream` method.
 67        batch_size : int
 68            The size of the batch to process documents in the `apply_stream` method.
 69        kwargs : Any
 70            Additional keyword arguments to pass to the filter.
 71        """
 72        self.name = self.__class__.__name__
 73        self.logger = logging.getLogger(f"{self.__module__}.{self.__class__.__name__}")
 74        assert 0 <= p <= 1
 75        self.p = p
 76        self.__init_rng(random_state)
 77        self.skip_rejected = skip_rejected
 78        self.use_batch = use_batch
 79        self.batch_size = batch_size
 80
 81        self._statistics: Statistics = Statistics()
 82
 83    @abstractmethod
 84    def apply(self, document: Document) -> Document:
 85        """
 86        Definition of filter behavior.
 87
 88        The document must have a protocol `TextContent`,
 89        and mostly used hojichar.Document class.
 90
 91        In this method, the filter will modify `document.text` or
 92        `document.extras` and set `document.is_rejected = True` to discard the document.
 93
 94        Parameters
 95        ----------
 96        document : Document
 97            Input document
 98
 99        Returns
100        -------
101        Document
102            Processed Document
103        """
104
105    @deprecated_since(version="1.0.0", alternative="apply")
106    def apply_filter(self, document: Document) -> Document:
107        document = self.apply(document)
108        return document
109
110    def _check_skip(self, document: Document) -> bool:
111        """
112        Check if the document should be skipped by this filter.
113        If `skip_rejected` is set to `True`, this method will return `True`
114        if the document is already rejected.
115        If `p` is less than 1, this method will return `True` with a probability of `1 - p`.
116        """
117        skip = self.skip_rejected and document.is_rejected
118        if skip:
119            return True
120        if self.p < 1:
121            if self._rng.random() > self.p:
122                return True
123        return False
124
125    def _apply(self, document: Document) -> Document:
126        """
127        Apply the filter to a single document.
128        This method
129          - checks if the document should be skipped
130          - counts and logging the statistics
131          - logging the reason for rejection if the document is rejected
132
133        This method may be used in `apply` method of `Compose` class.
134        """
135
136        stats = get_doc_info(document)
137
138        if not self._check_skip(document):
139            document = self.apply(document)
140
141        new_stats = get_doc_info(document)
142        self._statistics.update_by_diff(stats, new_stats)
143
144        if not stats["is_rejected"] and new_stats["is_rejected"]:
145            document.reject_reason = self.get_jsonable_vars()
146
147        return document
148
149    def apply_batch(self, batch: Sequence[Document]) -> list[Document]:
150        """
151        Apply the filter to a batch of documents.
152        You can override this method if you want to
153        apply the filter to a batch of documents at once.
154
155        This method may be used in `apply_batch` method of `Compose` class.
156
157        Parameters
158        ----------
159        documents : Sequence[Document]
160            List-like object of input documents
161
162        Returns
163        -------
164        list[Document]
165            List of processed documents
166        """
167        return [self.apply(document) for document in batch]
168
169    def _apply_batch(self, batch: Sequence[Document]) -> list[Document]:
170        """
171        Apply the filter to a batch of documents.
172        This method
173        - checks if the documents should be skipped
174        - counts and logs the statistics
175        - logs the reason for rejection if any document is rejected
176        """
177        skip = False
178        if self.p < 1:
179            skip = self._rng.random() > self.p
180
181        stats = [get_doc_info(document=doc) for doc in batch]
182        if not skip:
183            batch = self.apply_batch(batch)
184        batch = self._finalize_batch(batch, stats)
185        return batch
186
187    def apply_stream(self, stream: Iterable[Document]) -> Iterable[Document]:
188        """
189        Apply the filter to a stream of documents.
190        This method is used when you want to process documents one by one.
191        If `use_batch` is set to `True` in the constructor,
192        this method will process documents in batches using the `apply_batch` method.
193
194        Even if an exception occurs during processing, the process will continue, and the following actions will be taken:
195        - Set the `is_rejected` flag of the document to `True`
196        - Set the error details in `reject_reason`
197        - Increment the `errors` count in the statistics retrievable via `get_statistics`
198
199        Parameters
200        ----------
201        stream : Iterable[Document]
202            Stream of input documents
203
204        Returns
205        -------
206        Iterable[Document]
207            Stream of processed documents
208        """
209
210        if not self.use_batch:
211            for document in stream:
212                yield self._try_process(document, self._apply)
213        else:
214            batch: list[Document] = []
215            for document in stream:
216                if self._check_skip(document):
217                    yield document
218                    continue
219
220                batch.append(document)
221                if len(batch) >= self.batch_size:
222                    stats = [get_doc_info(doc) for doc in batch]
223                    batch = self._try_process(batch, self.apply_batch)
224                    batch = self._finalize_batch(batch, stats)
225                    yield from batch
226                    batch.clear()
227            if batch:
228                stats = [get_doc_info(doc) for doc in batch]
229                batch = self._try_process(batch, self.apply_batch)
230                batch = self._finalize_batch(batch, stats)
231                yield from batch
232
233    def _try_process(self, target: T, func: Callable[[T], T]) -> T:
234        try:
235            return func(target)
236        except Exception as e:
237            if isinstance(target, Document):
238                msg = f"{e!r} occurs while processing {self.name} with {target!r}"
239                target.is_rejected = True
240                target.reject_reason = {"error": msg}
241                self._statistics.errors += 1
242                self.logger.error(msg, exc_info=True)
243                return target
244            if isinstance(target, list):
245                msg = f"{e!r} occurs while batch processing {self.name}"
246                self.logger.error(msg, exc_info=True)
247                for doc in target:
248                    doc.is_rejected = True
249                    doc.reject_reason = {"error": msg}
250                self._statistics.errors += len(target)
251                return target
252            else:
253                raise e
254
255    def __call__(self, text: str, **kwargs: Any) -> str:
256        document = Document(text, **kwargs)
257        document = self._apply(document)
258        return document.text
259
260    def get_statistics(self) -> Statistics:
261        """
262        Get the statistics of this filter.
263        This method returns the statistics of the filter,
264        which includes the number of processed documents, discarded documents, and other statistics.
265        """
266        return self._statistics
267
268    def get_statistics_map(self) -> dict[str, Any]:
269        """
270        Get the statistics of this filter as a dictionary.
271        """
272        return self._statistics.to_dict()
273
274    def shutdown(self) -> None:
275        """
276        This method is called when the filter is no longer needed.
277        You can override this method to release resources or perform cleanup tasks.
278        """
279        pass
280
281    def __enter__(self) -> "Filter":
282        return self
283
284    def __exit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None:
285        """
286        This method is called when the filter is used in a context manager.
287        It calls the `shutdown` method to release resources or perform cleanup tasks.
288        """
289        self.shutdown()
290
291    def get_jsonable_vars(self, exclude_keys: set[str] | None = None) -> dict[str, Any]:
292        """
293        Get the member variable of this filter.
294        Eligible variables are primitive types; [bool, int, float, str, None],
295        and the name of the variable not starts with the underscore; `_`.
296        """
297        if exclude_keys is None:
298            exclude_keys = set()
299        return {
300            k: v
301            for k, v in vars(self).items()
302            if (_is_jsonable(v) and (k not in exclude_keys) and (not k.startswith("_")))
303        }
304
305    def _finalize_batch(
306        self: "Filter",
307        batch: Sequence[Document],
308        old_stats: list[dict[str, Any]] = [],
309    ) -> list[Document]:
310        new_stats = [get_doc_info(doc) for doc in batch]
311        for old, new, doc in zip(old_stats, new_stats, batch):
312            self._statistics.update_by_diff(old, new)
313            if not old["is_rejected"] and new["is_rejected"]:
314                doc.reject_reason = self.get_jsonable_vars()
315        return list(batch)
316
317    def __init_rng(self, random_state: int | np.random.Generator | None) -> None:
318        self._owns_rng = True
319        if random_state is None:
320            self._rng = np.random.default_rng()
321            self._owns_rng = False
322        elif isinstance(random_state, int):
323            self._rng = np.random.default_rng(random_state)
324        elif isinstance(random_state, np.random.Generator):
325            self._rng = random_state
326
327    def _set_rng_if_not_initialized(self, rng: np.random.Generator) -> None:
328        """
329        Set the random number generator for this filter if it is not already initialized.
330        This method is called by Compose class.
331        """
332        if not self._owns_rng:
333            self._rng = rng

Base class for all filters. Document-level filters must inherit from this class.

The definition of text processing is in apply method. If you define a new filter, override the method.

When this class is called, apply the filter from string to string.

With context manager, you can use the filter as follows:

with YourFilter(p=0.5) as filt:
    text = filt("This is a sample text.")
Filter( p: float = 1.0, skip_rejected: bool = True, *args: Any, random_state: int | numpy.random._generator.Generator | None = None, use_batch: bool = False, batch_size: int = 128, **kwargs: Any)
41    def __init__(
42        self,
43        p: float = 1.0,
44        skip_rejected: bool = True,
45        *args: Any,
46        random_state: int | np.random.Generator | None = None,
47        use_batch: bool = False,
48        batch_size: int = 128,
49        **kwargs: Any,
50    ) -> None:
51        """
52        Initialize the filter.
53        Parameters
54        ----------
55        p : float
56            The probability of applying the filter.
57            If `p` is 1, the filter will always be applied.
58        skip_rejected : bool
59            If `True`, the filter will skip documents that are already rejected.
60            If you want to apply the filter to all documents (e.g., postprocess), set this to `False`.
61        random_state : int | np.random.Generator | None
62            Seed for the random number generator.
63            If `None`, a new random number generator will be created.
64            If `None`, and use in the `Compose` class, the random state is shared with the `Compose` object.
65        use_batch : bool
66            If `True`, the filter will process documents in batches in the `apply_stream` method.
67        batch_size : int
68            The size of the batch to process documents in the `apply_stream` method.
69        kwargs : Any
70            Additional keyword arguments to pass to the filter.
71        """
72        self.name = self.__class__.__name__
73        self.logger = logging.getLogger(f"{self.__module__}.{self.__class__.__name__}")
74        assert 0 <= p <= 1
75        self.p = p
76        self.__init_rng(random_state)
77        self.skip_rejected = skip_rejected
78        self.use_batch = use_batch
79        self.batch_size = batch_size
80
81        self._statistics: Statistics = Statistics()

Initialize the filter.

Parameters

p : float The probability of applying the filter. If p is 1, the filter will always be applied. skip_rejected : bool If True, the filter will skip documents that are already rejected. If you want to apply the filter to all documents (e.g., postprocess), set this to False. random_state : int | np.random.Generator | None Seed for the random number generator. If None, a new random number generator will be created. If None, and use in the Compose class, the random state is shared with the Compose object. use_batch : bool If True, the filter will process documents in batches in the apply_stream method. batch_size : int The size of the batch to process documents in the apply_stream method. kwargs : Any Additional keyword arguments to pass to the filter.

@abstractmethod
def apply( self, document: hojichar.Document) -> hojichar.Document:
 83    @abstractmethod
 84    def apply(self, document: Document) -> Document:
 85        """
 86        Definition of filter behavior.
 87
 88        The document must have a protocol `TextContent`,
 89        and mostly used hojichar.Document class.
 90
 91        In this method, the filter will modify `document.text` or
 92        `document.extras` and set `document.is_rejected = True` to discard the document.
 93
 94        Parameters
 95        ----------
 96        document : Document
 97            Input document
 98
 99        Returns
100        -------
101        Document
102            Processed Document
103        """

Definition of filter behavior.

The document must have a protocol TextContent, and mostly used hojichar.Document class.

In this method, the filter will modify document.text or document.extras and set document.is_rejected = True to discard the document.

Parameters

document : Document Input document

Returns

Document Processed Document

@deprecated_since(version='1.0.0', alternative='apply')
def apply_filter( self, document: hojichar.Document) -> hojichar.Document:
105    @deprecated_since(version="1.0.0", alternative="apply")
106    def apply_filter(self, document: Document) -> Document:
107        document = self.apply(document)
108        return document
def apply_batch( self, batch: collections.abc.Sequence[hojichar.Document]) -> list[hojichar.Document]:
149    def apply_batch(self, batch: Sequence[Document]) -> list[Document]:
150        """
151        Apply the filter to a batch of documents.
152        You can override this method if you want to
153        apply the filter to a batch of documents at once.
154
155        This method may be used in `apply_batch` method of `Compose` class.
156
157        Parameters
158        ----------
159        documents : Sequence[Document]
160            List-like object of input documents
161
162        Returns
163        -------
164        list[Document]
165            List of processed documents
166        """
167        return [self.apply(document) for document in batch]

Apply the filter to a batch of documents. You can override this method if you want to apply the filter to a batch of documents at once.

This method may be used in apply_batch method of Compose class.

Parameters

documents : Sequence[Document] List-like object of input documents

Returns

list[Document] List of processed documents

def apply_stream( self, stream: collections.abc.Iterable[hojichar.Document]) -> collections.abc.Iterable[hojichar.Document]:
187    def apply_stream(self, stream: Iterable[Document]) -> Iterable[Document]:
188        """
189        Apply the filter to a stream of documents.
190        This method is used when you want to process documents one by one.
191        If `use_batch` is set to `True` in the constructor,
192        this method will process documents in batches using the `apply_batch` method.
193
194        Even if an exception occurs during processing, the process will continue, and the following actions will be taken:
195        - Set the `is_rejected` flag of the document to `True`
196        - Set the error details in `reject_reason`
197        - Increment the `errors` count in the statistics retrievable via `get_statistics`
198
199        Parameters
200        ----------
201        stream : Iterable[Document]
202            Stream of input documents
203
204        Returns
205        -------
206        Iterable[Document]
207            Stream of processed documents
208        """
209
210        if not self.use_batch:
211            for document in stream:
212                yield self._try_process(document, self._apply)
213        else:
214            batch: list[Document] = []
215            for document in stream:
216                if self._check_skip(document):
217                    yield document
218                    continue
219
220                batch.append(document)
221                if len(batch) >= self.batch_size:
222                    stats = [get_doc_info(doc) for doc in batch]
223                    batch = self._try_process(batch, self.apply_batch)
224                    batch = self._finalize_batch(batch, stats)
225                    yield from batch
226                    batch.clear()
227            if batch:
228                stats = [get_doc_info(doc) for doc in batch]
229                batch = self._try_process(batch, self.apply_batch)
230                batch = self._finalize_batch(batch, stats)
231                yield from batch

Apply the filter to a stream of documents. This method is used when you want to process documents one by one. If use_batch is set to True in the constructor, this method will process documents in batches using the apply_batch method.

Even if an exception occurs during processing, the process will continue, and the following actions will be taken:

  • Set the is_rejected flag of the document to True
  • Set the error details in reject_reason
  • Increment the errors count in the statistics retrievable via get_statistics

Parameters

stream : Iterable[Document] Stream of input documents

Returns

Iterable[Document] Stream of processed documents

def get_statistics(self) -> hojichar.core.models.Statistics:
260    def get_statistics(self) -> Statistics:
261        """
262        Get the statistics of this filter.
263        This method returns the statistics of the filter,
264        which includes the number of processed documents, discarded documents, and other statistics.
265        """
266        return self._statistics

Get the statistics of this filter. This method returns the statistics of the filter, which includes the number of processed documents, discarded documents, and other statistics.

def get_statistics_map(self) -> dict[str, typing.Any]:
268    def get_statistics_map(self) -> dict[str, Any]:
269        """
270        Get the statistics of this filter as a dictionary.
271        """
272        return self._statistics.to_dict()

Get the statistics of this filter as a dictionary.

def shutdown(self) -> None:
274    def shutdown(self) -> None:
275        """
276        This method is called when the filter is no longer needed.
277        You can override this method to release resources or perform cleanup tasks.
278        """
279        pass

This method is called when the filter is no longer needed. You can override this method to release resources or perform cleanup tasks.

def get_jsonable_vars(self, exclude_keys: set[str] | None = None) -> dict[str, typing.Any]:
291    def get_jsonable_vars(self, exclude_keys: set[str] | None = None) -> dict[str, Any]:
292        """
293        Get the member variable of this filter.
294        Eligible variables are primitive types; [bool, int, float, str, None],
295        and the name of the variable not starts with the underscore; `_`.
296        """
297        if exclude_keys is None:
298            exclude_keys = set()
299        return {
300            k: v
301            for k, v in vars(self).items()
302            if (_is_jsonable(v) and (k not in exclude_keys) and (not k.startswith("_")))
303        }

Get the member variable of this filter. Eligible variables are primitive types; [bool, int, float, str, None], and the name of the variable not starts with the underscore; _.

@deprecated_since(version='1.0.0', alternative='Filter')
class TokenFilter(hojichar.Filter, abc.ABC):
336@deprecated_since(version="1.0.0", alternative="Filter")
337class TokenFilter(Filter, ABC):
338    """
339    Base class for token-level filters.
340
341    Token filters, which shuld be implemented in hojichar/filters/token_filters.py,
342    must inherit from this class.
343    """
344
345    def __init__(
346        self, p: float = 1, skip_rejected: bool = True, *args: Any, **kwargs: Any
347    ) -> None:
348        self.name = self.__class__.__name__
349        self.logger = logging.getLogger("hojichar.token_filters." + self.name)
350        assert 0 <= p <= 1
351        self.p = p
352        self.skip_rejected = skip_rejected
353
354    def apply(self, token: Token) -> Token:  # type: ignore
355        raise NotImplementedError(f"{self.__class__.__name__}.apply method is not defined")
356        return token
357
358    def apply_filter(self, document: Document) -> Document:
359        document.tokens = [self.apply(token) for token in document.tokens if not token.is_rejected]
360        return document
361
362    def __call__(self, text: str) -> str:  # type: ignore
363        token = Token(text)
364        token = self.apply(token)
365        return token.text
366
367    def _apply(self, document: Document) -> Document:
368        """
369        Apply the token filter to a single document.
370        This method checks if the document should be skipped.
371        """
372        if self.skip_rejected and document.is_rejected:
373            return document
374        return self.apply_filter(document)
375
376    def get_jsonable_vars(self, exclude_keys: set[str] | None = None) -> dict:
377        """
378        Get the member variable of this filter.
379        Eligible variables are primitive types; [bool, int, float, str, None],
380        and the name of the variable not starts with the underscore; `_`.
381        """
382        if exclude_keys is None:
383            exclude_keys = set()
384        return {
385            k: v
386            for k, v in vars(self).items()
387            if (_is_jsonable(v) and (k not in exclude_keys) and (not k.startswith("_")))
388        }

Base class for token-level filters.

Token filters, which shuld be implemented in hojichar/filters/token_filters.py, must inherit from this class.

TokenFilter(p: float = 1, skip_rejected: bool = True, *args: Any, **kwargs: Any)
345    def __init__(
346        self, p: float = 1, skip_rejected: bool = True, *args: Any, **kwargs: Any
347    ) -> None:
348        self.name = self.__class__.__name__
349        self.logger = logging.getLogger("hojichar.token_filters." + self.name)
350        assert 0 <= p <= 1
351        self.p = p
352        self.skip_rejected = skip_rejected

Initialize the filter.

Parameters

p : float The probability of applying the filter. If p is 1, the filter will always be applied. skip_rejected : bool If True, the filter will skip documents that are already rejected. If you want to apply the filter to all documents (e.g., postprocess), set this to False. random_state : int | np.random.Generator | None Seed for the random number generator. If None, a new random number generator will be created. If None, and use in the Compose class, the random state is shared with the Compose object. use_batch : bool If True, the filter will process documents in batches in the apply_stream method. batch_size : int The size of the batch to process documents in the apply_stream method. kwargs : Any Additional keyword arguments to pass to the filter.

def apply(self, token: hojichar.Token) -> hojichar.Token:
354    def apply(self, token: Token) -> Token:  # type: ignore
355        raise NotImplementedError(f"{self.__class__.__name__}.apply method is not defined")
356        return token

Definition of filter behavior.

The document must have a protocol TextContent, and mostly used hojichar.Document class.

In this method, the filter will modify document.text or document.extras and set document.is_rejected = True to discard the document.

Parameters

document : Document Input document

Returns

Document Processed Document

def apply_filter( self, document: hojichar.Document) -> hojichar.Document:
358    def apply_filter(self, document: Document) -> Document:
359        document.tokens = [self.apply(token) for token in document.tokens if not token.is_rejected]
360        return document
def get_jsonable_vars(self, exclude_keys: set[str] | None = None) -> dict:
376    def get_jsonable_vars(self, exclude_keys: set[str] | None = None) -> dict:
377        """
378        Get the member variable of this filter.
379        Eligible variables are primitive types; [bool, int, float, str, None],
380        and the name of the variable not starts with the underscore; `_`.
381        """
382        if exclude_keys is None:
383            exclude_keys = set()
384        return {
385            k: v
386            for k, v in vars(self).items()
387            if (_is_jsonable(v) and (k not in exclude_keys) and (not k.startswith("_")))
388        }

Get the member variable of this filter. Eligible variables are primitive types; [bool, int, float, str, None], and the name of the variable not starts with the underscore; _.

class Document:
 24class Document:
 25    """
 26    Document class represents a text document with metadata.
 27    It contains the text of the document, a flag indicating whether it is rejected,
 28     and additional metadata stored in the `extras` dictionary.
 29
 30    The `tokens` attribute will be deprecated in future versions,
 31    and users are encouraged to use the `extras` dictionary to store token-related information.
 32
 33    Attributes:
 34        text (str): The text content of the document.
 35        is_rejected (bool): A flag indicating whether the document is rejected.
 36        extras (dict[str, Any]): A dictionary to store additional metadata about the document.
 37        reject_reason (dict[str, Any]): A dictionary to store the reason for rejection. The
 38          filter class and the member name and value will logged at the filter is logged here.
 39        initial_stats (dict[str, Any] | None): Internal copy of the document statistics
 40          captured before the pipeline mutates the document.
 41
 42    Next attributes will be deprecated in future versions:
 43        tokens (list[Token]): A list of tokens extracted from the document.
 44    """
 45
 46    def __init__(
 47        self,
 48        text: str,
 49        is_rejected: bool = False,
 50        tokens: list[Token] | None = None,
 51        extras: dict[str, Any] | None = None,
 52    ) -> None:
 53        self.text = text
 54        self.__original = text
 55        self.is_rejected = is_rejected
 56        if tokens is None:
 57            self.tokens: list[Token] = []
 58        else:
 59            self.tokens = tokens
 60
 61        if extras is None:
 62            self.extras: dict[str, Any] = {}
 63        else:
 64            self.extras = extras
 65
 66        self.reject_reason: dict[str, Any] = {}
 67        self._initial_stats: dict[str, Any] | None = None
 68        if "__init_stats" in self.extras:
 69            self._initial_stats = self.extras.pop("__init_stats")
 70
 71    @property
 72    def original(self) -> str:
 73        return self.__original
 74
 75    def _set_initial_stats(self, stats: dict[str, Any]) -> None:
 76        """
 77        Store the document statistics captured before the pipeline modifies the document.
 78        Internal API: not intended for filter implementations.
 79        """
 80        self._initial_stats = stats
 81
 82    def _get_initial_stats(self) -> dict[str, Any] | None:
 83        """
 84        Retrieve the stored initial statistics of the document, if available.
 85        Internal API: not intended for filter implementations.
 86        """
 87        return self._initial_stats
 88
 89    def _clear_initial_stats(self) -> None:
 90        """
 91        Remove the stored initial statistics. Useful after the stats are consumed.
 92        Internal API: not intended for filter implementations.
 93        """
 94        self._initial_stats = None
 95
 96    @deprecated_since("1.0.0")
 97    def set_tokens(self, tokens: list[str]) -> None:
 98        self.tokens = [Token(token) for token in tokens]
 99
100    @deprecated_since("1.0.0")
101    def get_tokens(self) -> list[str]:
102        return [token.text for token in self.tokens]
103
104    def __str__(self) -> str:
105        return self.text
106
107    def __repr__(self) -> str:
108        return (
109            f"Document(text={self.text!r}, is_rejected={self.is_rejected}, extras={self.extras})"  # noqa
110        )

Document class represents a text document with metadata. It contains the text of the document, a flag indicating whether it is rejected, and additional metadata stored in the extras dictionary.

The tokens attribute will be deprecated in future versions, and users are encouraged to use the extras dictionary to store token-related information.

Attributes: text (str): The text content of the document. is_rejected (bool): A flag indicating whether the document is rejected. extras (dict[str, Any]): A dictionary to store additional metadata about the document. reject_reason (dict[str, Any]): A dictionary to store the reason for rejection. The filter class and the member name and value will logged at the filter is logged here. initial_stats (dict[str, Any] | None): Internal copy of the document statistics captured before the pipeline mutates the document.

Next attributes will be deprecated in future versions: tokens (list[Token]): A list of tokens extracted from the document.

Document( text: str, is_rejected: bool = False, tokens: list[hojichar.Token] | None = None, extras: dict[str, typing.Any] | None = None)
46    def __init__(
47        self,
48        text: str,
49        is_rejected: bool = False,
50        tokens: list[Token] | None = None,
51        extras: dict[str, Any] | None = None,
52    ) -> None:
53        self.text = text
54        self.__original = text
55        self.is_rejected = is_rejected
56        if tokens is None:
57            self.tokens: list[Token] = []
58        else:
59            self.tokens = tokens
60
61        if extras is None:
62            self.extras: dict[str, Any] = {}
63        else:
64            self.extras = extras
65
66        self.reject_reason: dict[str, Any] = {}
67        self._initial_stats: dict[str, Any] | None = None
68        if "__init_stats" in self.extras:
69            self._initial_stats = self.extras.pop("__init_stats")
@deprecated_since('1.0.0')
def set_tokens(self, tokens: list[str]) -> None:
96    @deprecated_since("1.0.0")
97    def set_tokens(self, tokens: list[str]) -> None:
98        self.tokens = [Token(token) for token in tokens]
@deprecated_since('1.0.0')
def get_tokens(self) -> list[str]:
100    @deprecated_since("1.0.0")
101    def get_tokens(self) -> list[str]:
102        return [token.text for token in self.tokens]
@deprecated_since('0.1.0', 'Document')
class Token:
 9@deprecated_since("0.1.0", "Document")
10class Token:
11    def __init__(self, text: str, is_rejected: bool = False) -> None:
12        self.text = text
13        self.__original = text
14        self.is_rejected = is_rejected
15
16    @property
17    def original(self) -> str:
18        return self.__original
19
20    def __str__(self) -> str:
21        return self.text
Token(text: str, is_rejected: bool = False)
11    def __init__(self, text: str, is_rejected: bool = False) -> None:
12        self.text = text
13        self.__original = text
14        self.is_rejected = is_rejected
class Parallel:
131class Parallel:
132    """
133    The Parallel class provides a way to apply a hojichar.Compose filter
134    to an iterator of documents in a parallel manner using a specified
135    number of worker processes. This class should be used as a context
136    manager with a 'with' statement.
137
138    When no global start method has been configured, Parallel uses ``fork`` on
139    platforms which support it so unpicklable filters can be inherited by
140    workers. Set ``HOJICHAR_MP_START_METHOD`` to explicitly choose ``fork``,
141    ``spawn``, ``forkserver``, or ``default``. Non-fork contexts require the
142    Compose object and its filters to be picklable.
143
144    Example:
145
146    doc_iter = (hojichar.Document(d) for d in open("my_text.txt"))
147    with Parallel(my_filter, num_jobs=8) as pfilter:
148        for doc in pfilter.imap_apply(doc_iter):
149            pass  # Process the filtered document as needed.
150    """
151
152    def __init__(
153        self,
154        filter: hojichar.Compose,
155        num_jobs: int | None = None,
156        ignore_errors: bool = False,
157        ordered: bool = False,
158        max_in_flight: int | None = None,
159    ):
160        """
161        Initializes a new instance of the Parallel class.
162
163        Args:
164            filter (hojichar.Compose): A composed filter object that specifies the
165                processing operations to apply to each document in parallel.
166                A copy of the filter is made within a 'with' statement. When the 'with'
167                block terminates,the statistical information obtained through `filter.statistics`
168                or`filter.statistics_obj` is replaced with the total value of the statistical
169                information processed within the 'with' block.
170
171            num_jobs (int | None, optional): The number of worker processes to use.
172                If None, then the number returned by os.cpu_count() is used. Defaults to None.
173            ignore_errors (bool, optional): If set to True, any exceptions thrown during
174                the processing of a document will be caught and logged, but will not
175                stop the processing of further documents. If set to False, the first
176                exception thrown will terminate the entire parallel processing operation.
177                Defaults to False.
178            ordered (bool, optional): If set to True, processed documents are yielded in
179                the same order as the input documents. If set to False, documents are
180                yielded as soon as their processing completes. Defaults to False.
181            max_in_flight (int | None, optional): Strict upper bound on the number of
182                documents drawn from the input iterator whose results have not yet been
183                handed back to the caller. A yielded document keeps its permit until the
184                caller requests the next one, so with `max_in_flight=1` the pool holds a
185                single document end to end (no pipelining). Without a bound, Pool's
186                task-handler thread drains the input as fast as the worker pipe accepts,
187                so a producer that outruns the filters can buffer a large number of
188                documents. If None, no explicit bound is applied. Defaults to None.
189        """
190        if max_in_flight is not None and max_in_flight < 1:
191            raise ValueError("max_in_flight must be at least 1")
192        self.filter = filter
193        self.num_jobs = num_jobs
194        self.ignore_errors = ignore_errors
195        self.ordered = ordered
196        self.max_in_flight = max_in_flight
197
198        self._pool: Pool | None = None
199        self._pid_stats: dict[int, list[Statistics]] | None = None
200        self._gates: list[_InFlightGate] = []
201
202    def __enter__(self) -> Parallel:
203        context = _get_parallel_context()
204        self._pool = context.Pool(
205            processes=self.num_jobs,
206            initializer=_init_worker,
207            initargs=(self.filter, self.ignore_errors),
208        )
209        self._pid_stats = dict()
210        self._gates = []
211        return self
212
213    def imap_apply(self, docs: Iterator[hojichar.Document]) -> Iterator[hojichar.Document]:
214        """
215        Takes an iterator of Documents and applies the Compose filter to
216        each Document in a parallel manner. This is a generator method
217        that yields processed Documents.
218
219        Args:
220            docs (Iterator[hojichar.Document]): An iterator of Documents to be processed.
221
222        Raises:
223            RuntimeError: If the Parallel instance is not properly initialized. This
224                generally happens when the method is called outside of a 'with' statement.
225            Exception: If any exceptions are raised within the worker processes.
226
227        Yields:
228            Iterator[hojichar.Document]: An iterator that yields processed Documents.
229        """
230        if self._pool is None or self._pid_stats is None:
231            raise RuntimeError(
232                "Parallel instance not properly initialized. Use within a 'with' statement."
233            )
234        gate: _InFlightGate | None = None
235        if self.max_in_flight is not None:
236            gate = _InFlightGate(self.max_in_flight)
237            self._gates.append(gate)
238            docs = gate.feed(docs)
239        try:
240            results = (
241                self._pool.imap(_worker, docs)
242                if self.ordered
243                else self._pool.imap_unordered(_worker, docs)
244            )
245            for doc, pid, stat, err_msg in results:
246                self._pid_stats[pid] = stat
247                if err_msg is not None:
248                    logger.error(f"Error in worker {pid}: {err_msg}")
249                # The permit is returned only once the caller has taken the
250                # document: releasing before the yield would let the feeder
251                # momentarily draw max_in_flight + 1 documents. On an abnormal
252                # exit (close/throw at the yield point) the gate is stopped
253                # *before* the permit is returned, so a feeder woken by the
254                # release always observes the stop and cannot draw again.
255                try:
256                    yield doc
257                except BaseException:
258                    if gate is not None:
259                        gate.stop()
260                        gate.release()
261                    raise
262                else:
263                    if gate is not None:
264                        gate.release()
265        except Exception:
266            self.__exit__(None, None, None)
267            raise
268        finally:
269            if gate is not None:
270                gate.stop()
271
272    def __exit__(self, exc_type, exc_value, traceback) -> None:  # type: ignore
273        # Feeders blocked on a max_in_flight gate must exit before the pool is
274        # joined, or shutdown would wait on them forever.
275        for gate in self._gates:
276            gate.stop()
277        if self._pool:
278            self._pool.terminate()
279            self._pool.join()
280        if self._pid_stats:
281            total_stats = functools.reduce(
282                lambda x, y: Statistics.add_list_of_stats(x, y), self._pid_stats.values()
283            )
284            self.filter._statistics.update(Statistics.get_filter("Total", total_stats))
285            for stat in total_stats:
286                for filt in self.filter.filters:
287                    if stat.name == filt.name:
288                        filt._statistics.update(stat)
289                        break
290
291    def get_total_statistics(self) -> list[Statistics]:
292        """
293        Returns a statistics object of the total statistical
294        values processed within the Parallel block.
295
296        Returns:
297            StatsContainer: Statistics object
298        """
299        if self._pid_stats:
300            total_stats = functools.reduce(
301                lambda x, y: Statistics.add_list_of_stats(x, y), self._pid_stats.values()
302            )
303            return total_stats
304        else:
305            return []
306
307    def get_total_statistics_map(self) -> list[dict]:
308        return [stat.to_dict() for stat in self.get_total_statistics()]
309
310    @property
311    def statistics_obj(self) -> inspection.StatsContainer:
312        """
313        Returns the statistics object of the Parallel instance.
314        This is a StatsContainer object which contains the statistics
315        of the Parallel instance and sub filters.
316
317        Returns:
318            StatsContainer: Statistics object
319        """
320        return inspection.statistics_obj_adapter(self.get_total_statistics())  # type: ignore

The Parallel class provides a way to apply a hojichar.Compose filter to an iterator of documents in a parallel manner using a specified number of worker processes. This class should be used as a context manager with a 'with' statement.

When no global start method has been configured, Parallel uses fork on platforms which support it so unpicklable filters can be inherited by workers. Set HOJICHAR_MP_START_METHOD to explicitly choose fork, spawn, forkserver, or default. Non-fork contexts require the Compose object and its filters to be picklable.

Example:

doc_iter = (hojichar.Document(d) for d in open("my_text.txt")) with Parallel(my_filter, num_jobs=8) as pfilter: for doc in pfilter.imap_apply(doc_iter): pass # Process the filtered document as needed.

Parallel( filter: hojichar.Compose, num_jobs: int | None = None, ignore_errors: bool = False, ordered: bool = False, max_in_flight: int | None = None)
152    def __init__(
153        self,
154        filter: hojichar.Compose,
155        num_jobs: int | None = None,
156        ignore_errors: bool = False,
157        ordered: bool = False,
158        max_in_flight: int | None = None,
159    ):
160        """
161        Initializes a new instance of the Parallel class.
162
163        Args:
164            filter (hojichar.Compose): A composed filter object that specifies the
165                processing operations to apply to each document in parallel.
166                A copy of the filter is made within a 'with' statement. When the 'with'
167                block terminates,the statistical information obtained through `filter.statistics`
168                or`filter.statistics_obj` is replaced with the total value of the statistical
169                information processed within the 'with' block.
170
171            num_jobs (int | None, optional): The number of worker processes to use.
172                If None, then the number returned by os.cpu_count() is used. Defaults to None.
173            ignore_errors (bool, optional): If set to True, any exceptions thrown during
174                the processing of a document will be caught and logged, but will not
175                stop the processing of further documents. If set to False, the first
176                exception thrown will terminate the entire parallel processing operation.
177                Defaults to False.
178            ordered (bool, optional): If set to True, processed documents are yielded in
179                the same order as the input documents. If set to False, documents are
180                yielded as soon as their processing completes. Defaults to False.
181            max_in_flight (int | None, optional): Strict upper bound on the number of
182                documents drawn from the input iterator whose results have not yet been
183                handed back to the caller. A yielded document keeps its permit until the
184                caller requests the next one, so with `max_in_flight=1` the pool holds a
185                single document end to end (no pipelining). Without a bound, Pool's
186                task-handler thread drains the input as fast as the worker pipe accepts,
187                so a producer that outruns the filters can buffer a large number of
188                documents. If None, no explicit bound is applied. Defaults to None.
189        """
190        if max_in_flight is not None and max_in_flight < 1:
191            raise ValueError("max_in_flight must be at least 1")
192        self.filter = filter
193        self.num_jobs = num_jobs
194        self.ignore_errors = ignore_errors
195        self.ordered = ordered
196        self.max_in_flight = max_in_flight
197
198        self._pool: Pool | None = None
199        self._pid_stats: dict[int, list[Statistics]] | None = None
200        self._gates: list[_InFlightGate] = []

Initializes a new instance of the Parallel class.

Args: filter (hojichar.Compose): A composed filter object that specifies the processing operations to apply to each document in parallel. A copy of the filter is made within a 'with' statement. When the 'with' block terminates,the statistical information obtained through filter.statistics orfilter.statistics_obj is replaced with the total value of the statistical information processed within the 'with' block.

num_jobs (int | None, optional): The number of worker processes to use.
    If None, then the number returned by os.cpu_count() is used. Defaults to None.
ignore_errors (bool, optional): If set to True, any exceptions thrown during
    the processing of a document will be caught and logged, but will not
    stop the processing of further documents. If set to False, the first
    exception thrown will terminate the entire parallel processing operation.
    Defaults to False.
ordered (bool, optional): If set to True, processed documents are yielded in
    the same order as the input documents. If set to False, documents are
    yielded as soon as their processing completes. Defaults to False.
max_in_flight (int | None, optional): Strict upper bound on the number of
    documents drawn from the input iterator whose results have not yet been
    handed back to the caller. A yielded document keeps its permit until the
    caller requests the next one, so with `max_in_flight=1` the pool holds a
    single document end to end (no pipelining). Without a bound, Pool's
    task-handler thread drains the input as fast as the worker pipe accepts,
    so a producer that outruns the filters can buffer a large number of
    documents. If None, no explicit bound is applied. Defaults to None.
def imap_apply( self, docs: collections.abc.Iterator[hojichar.Document]) -> collections.abc.Iterator[hojichar.Document]:
213    def imap_apply(self, docs: Iterator[hojichar.Document]) -> Iterator[hojichar.Document]:
214        """
215        Takes an iterator of Documents and applies the Compose filter to
216        each Document in a parallel manner. This is a generator method
217        that yields processed Documents.
218
219        Args:
220            docs (Iterator[hojichar.Document]): An iterator of Documents to be processed.
221
222        Raises:
223            RuntimeError: If the Parallel instance is not properly initialized. This
224                generally happens when the method is called outside of a 'with' statement.
225            Exception: If any exceptions are raised within the worker processes.
226
227        Yields:
228            Iterator[hojichar.Document]: An iterator that yields processed Documents.
229        """
230        if self._pool is None or self._pid_stats is None:
231            raise RuntimeError(
232                "Parallel instance not properly initialized. Use within a 'with' statement."
233            )
234        gate: _InFlightGate | None = None
235        if self.max_in_flight is not None:
236            gate = _InFlightGate(self.max_in_flight)
237            self._gates.append(gate)
238            docs = gate.feed(docs)
239        try:
240            results = (
241                self._pool.imap(_worker, docs)
242                if self.ordered
243                else self._pool.imap_unordered(_worker, docs)
244            )
245            for doc, pid, stat, err_msg in results:
246                self._pid_stats[pid] = stat
247                if err_msg is not None:
248                    logger.error(f"Error in worker {pid}: {err_msg}")
249                # The permit is returned only once the caller has taken the
250                # document: releasing before the yield would let the feeder
251                # momentarily draw max_in_flight + 1 documents. On an abnormal
252                # exit (close/throw at the yield point) the gate is stopped
253                # *before* the permit is returned, so a feeder woken by the
254                # release always observes the stop and cannot draw again.
255                try:
256                    yield doc
257                except BaseException:
258                    if gate is not None:
259                        gate.stop()
260                        gate.release()
261                    raise
262                else:
263                    if gate is not None:
264                        gate.release()
265        except Exception:
266            self.__exit__(None, None, None)
267            raise
268        finally:
269            if gate is not None:
270                gate.stop()

Takes an iterator of Documents and applies the Compose filter to each Document in a parallel manner. This is a generator method that yields processed Documents.

Args: docs (Iterator[hojichar.Document]): An iterator of Documents to be processed.

Raises: RuntimeError: If the Parallel instance is not properly initialized. This generally happens when the method is called outside of a 'with' statement. Exception: If any exceptions are raised within the worker processes.

Yields: Iterator[hojichar.Document]: An iterator that yields processed Documents.

def get_total_statistics(self) -> list[hojichar.core.models.Statistics]:
291    def get_total_statistics(self) -> list[Statistics]:
292        """
293        Returns a statistics object of the total statistical
294        values processed within the Parallel block.
295
296        Returns:
297            StatsContainer: Statistics object
298        """
299        if self._pid_stats:
300            total_stats = functools.reduce(
301                lambda x, y: Statistics.add_list_of_stats(x, y), self._pid_stats.values()
302            )
303            return total_stats
304        else:
305            return []

Returns a statistics object of the total statistical values processed within the Parallel block.

Returns: StatsContainer: Statistics object

def get_total_statistics_map(self) -> list[dict]:
307    def get_total_statistics_map(self) -> list[dict]:
308        return [stat.to_dict() for stat in self.get_total_statistics()]
statistics_obj: hojichar.StatsContainer

Returns the statistics object of the Parallel instance. This is a StatsContainer object which contains the statistics of the Parallel instance and sub filters.

Returns: StatsContainer: Statistics object

@dataclasses.dataclass
class StatsContainer:
116@dataclasses.dataclass
117class StatsContainer:
118    total_info: DocStatistics
119    layers_info: dict[str, FilterStatistics]  # Key of the dict is filter name.
120
121    def __add__(self, other: StatsContainer) -> StatsContainer:
122        assert self.layers_info.keys() == other.layers_info.keys(), "Layer names must match"
123        return StatsContainer(
124            self.total_info + other.total_info,
125            {k: v + other.layers_info[k] for k, v in self.layers_info.items()},
126        )
127
128    def get_human_readable_values(self) -> dict:
129        return {
130            "total_info": self.total_info.get_human_readable_values(),
131            "layers_info": [
132                layer.get_human_readable_values() for layer in self.layers_info.values()
133            ],
134        }
135
136    def reset(self) -> StatsContainer:
137        self.total_info.reset
138        for layer in self.layers_info.values():
139            layer.reset()
140        return self
StatsContainer( total_info: hojichar.core.inspection.DocStatistics, layers_info: dict[str, hojichar.core.inspection.FilterStatistics])
def get_human_readable_values(self) -> dict:
128    def get_human_readable_values(self) -> dict:
129        return {
130            "total_info": self.total_info.get_human_readable_values(),
131            "layers_info": [
132                layer.get_human_readable_values() for layer in self.layers_info.values()
133            ],
134        }
def reset(self) -> hojichar.StatsContainer:
136    def reset(self) -> StatsContainer:
137        self.total_info.reset
138        for layer in self.layers_info.values():
139            layer.reset()
140        return self
class AsyncCompose(hojichar.AsyncFilter):
 78class AsyncCompose(AsyncFilter):
 79    def __init__(
 80        self,
 81        filters: list[AsyncFilter | Filter],
 82        random_state: int | np.random.Generator | None = None,
 83        executor: ThreadPoolExecutor | None = None,
 84        *args: Any,
 85        **kwargs: Any,
 86    ):
 87        super().__init__(random_state=random_state, *args, **kwargs)
 88        self.logger = logging.getLogger(f"{self.__module__}.{self.__class__.__name__}")
 89        self._statistics.name = "Total"
 90        self._has_external_executor = executor is not None
 91        self._executor = executor or ThreadPoolExecutor()
 92        self.set_filters(filters)
 93
 94    def set_filters(self, filters: list[AsyncFilter | Filter]) -> None:
 95        self.filters: list[AsyncFilter] = []
 96        filter_idx = 0
 97        for f in filters:
 98            if isinstance(f, (AsyncCompose, Compose)):
 99                for sub in f.filters:
100                    name = f"{filter_idx}-{sub.__class__.__name__}"
101                    if isinstance(sub, Filter):
102                        name = f"{filter_idx}-{sub.__class__.__name__}"
103                        sub = AsyncFilterAdapter(sub, executor=self._executor)
104
105                    sub._set_rng_if_not_initialized(self._rng)
106                    sub.name = name
107                    sub._statistics.name = name
108                    self.filters.append(sub)
109                    filter_idx += 1
110            else:
111                name = f"{filter_idx}-{f.__class__.__name__}"
112                if isinstance(f, Filter):
113                    name = f"{filter_idx}-{f.__class__.__name__}"
114                    f = AsyncFilterAdapter(f, executor=self._executor)
115                f._set_rng_if_not_initialized(self._rng)
116                f.name = name
117                f._statistics.name = name
118                self.filters.append(f)
119                filter_idx += 1
120
121    async def apply(self, document: Document) -> Document:
122        stat = get_doc_info(document)
123        for filter_idx, filt in enumerate(self.filters):
124            document = await filt._apply(document)
125        new_stat = get_doc_info(document)
126        async with self._stats_lock:
127            self._statistics.update_by_diff(stat, new_stat)
128        return document
129
130    async def apply_batch(self, batch: Sequence[Document]) -> list[Document]:
131        stats = [get_doc_info(doc) for doc in batch]
132        for i, filt in enumerate(self.filters):
133            batch = await filt._apply_batch(batch)
134        batch = await self._finalize_batch(batch, stats)
135        return list(batch)
136
137    async def apply_stream(
138        self,
139        stream: AsyncIterable[Document] | Iterable[Document],
140    ) -> AsyncGenerator[Document, None]:
141        async_stream = handle_stream_as_async(stream, chunk_size=1000, executor=self._executor)
142        async_stream = self._count_input_stats(async_stream)
143
144        for i, filt in enumerate(self.filters):
145            async_stream = filt.apply_stream(async_stream)
146
147        async for doc in async_stream:
148            in_stat = doc._get_initial_stats()
149            if in_stat is None:
150                in_stat = get_doc_info(doc)
151                self.logger.debug(
152                    "Initial stats missing for document during async stream aggregation; "
153                    "using current stats as fallback"
154                )
155            out_stat = get_doc_info(doc)
156            async with self._stats_lock:
157                self._statistics.update_by_diff(in_stat, out_stat)
158            doc._clear_initial_stats()
159            yield doc
160
161    def imap_apply(
162        self,
163        stream: AsyncIterable[Document] | Iterable[Document],
164        *,
165        buffer_size: int = 128,
166        shutdown: bool = True,
167    ) -> AsyncToSyncIterator[Document]:
168        """Synchronously consume this asynchronous pipeline.
169
170        The returned iterator runs the pipeline on a dedicated background event loop. It is
171        intended for one-shot use and shuts down the pipeline after consumption by default.
172        Use the iterator as a context manager if iteration may stop early.
173        """
174
175        finalizer = self.shutdown if shutdown else None
176        return handle_async_stream_as_sync(
177            self.apply_stream(stream),
178            buffer_size=buffer_size,
179            finalizer=finalizer,
180        )
181
182    async def _count_input_stats(
183        self, async_stream: AsyncIterable[Document]
184    ) -> AsyncGenerator[Document, None]:
185        async for doc in async_stream:
186            doc._set_initial_stats(get_doc_info(doc))
187            yield doc
188
189    def get_total_statistics(self) -> list[Statistics]:
190        """
191        Get the statistics of the Compose object and sub filters.
192
193        The statistics of the Compose class are stored in an object with the name "Total",
194        and sub-filters's are stored with names in the format {filter_index}-{filter class name}.
195        """
196        stats = []
197        stats.append(self.get_statistics())
198        for i, filt in enumerate(self.filters):
199            stats.append(filt.get_statistics())
200        return stats
201
202    def get_total_statistics_map(self) -> list[dict[str, Any]]:
203        """
204        Get the statistics of the Compose object and sub filters as a list of dictionaries.
205        """
206        stats = self.get_total_statistics()
207        return [stat.to_dict() for stat in stats]
208
209    @property
210    def statistics(self) -> dict:
211        """
212        Deprecated
213
214        Get the statistics of the Compose object and sub filters.
215
216        This property is retained for compatibility with previous versions.
217        Please use `get_total_statistics` or `get_total_statistics_map` instead.
218        """
219        return inspection.statistics_obj_adapter(  # type: ignore
220            self.get_total_statistics()
221        ).get_human_readable_values()
222
223    @property
224    def statistics_obj(self) -> inspection.StatsContainer:
225        """
226        Deprecated
227
228        Get the statistics of the AsyncCompose object and sub filters.
229        This method returns a StatsContainer object which contains the statistics
230        of the AsyncCompose object and sub filters.
231
232        This property is retained for compatibility with previous versions.
233        Please use `get_total_statistics` or `get_total_statistics_map` instead.
234        """
235        return inspection.statistics_obj_adapter(self.get_total_statistics())  # type: ignore
236
237    async def shutdown(self) -> None:
238        for filt in self.filters:
239            await filt.shutdown()
240        if not self._has_external_executor:
241            self._executor.shutdown()
242
243    async def __aenter__(self) -> "AsyncCompose":
244        return self
245
246    async def __aexit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None:
247        await self.shutdown()
248        if exc_type is not None:
249            raise exc_value

Helper class that provides a standard way to create an ABC using inheritance.

AsyncCompose( filters: list[hojichar.AsyncFilter | hojichar.Filter], random_state: int | numpy.random._generator.Generator | None = None, executor: concurrent.futures.thread.ThreadPoolExecutor | None = None, *args: Any, **kwargs: Any)
79    def __init__(
80        self,
81        filters: list[AsyncFilter | Filter],
82        random_state: int | np.random.Generator | None = None,
83        executor: ThreadPoolExecutor | None = None,
84        *args: Any,
85        **kwargs: Any,
86    ):
87        super().__init__(random_state=random_state, *args, **kwargs)
88        self.logger = logging.getLogger(f"{self.__module__}.{self.__class__.__name__}")
89        self._statistics.name = "Total"
90        self._has_external_executor = executor is not None
91        self._executor = executor or ThreadPoolExecutor()
92        self.set_filters(filters)

Base class for asynchronous filters.

Parameters

p : float The probability of applying the filter. If p is 1, the filter will always be applied. skip_rejected : bool If True, the filter will skip documents that are already rejected. If you want to apply the filter to all documents (e.g., postprocess), set this to False. random_state : int | np.random.Generator | None Seed for the random number generator. If None is specified, the random number generator managed by the Compose class will be used. use_batch : bool If True, the filter will process documents in batches in the apply_stream method. batch_size : int The size of the batch to process documents in the apply_stream method. When apply_batch is not overridden, this is also the maximum number of in-flight document tasks used by the sliding-window scheduler. Ordered processing applies backpressure after at most two windows of started but not yet yielded documents. ordered : bool If True, apply_stream yields documents in input order. If False, filters using the default apply_batch implementation yield documents as soon as their processing completes.

def set_filters( self, filters: list[hojichar.AsyncFilter | hojichar.Filter]) -> None:
 94    def set_filters(self, filters: list[AsyncFilter | Filter]) -> None:
 95        self.filters: list[AsyncFilter] = []
 96        filter_idx = 0
 97        for f in filters:
 98            if isinstance(f, (AsyncCompose, Compose)):
 99                for sub in f.filters:
100                    name = f"{filter_idx}-{sub.__class__.__name__}"
101                    if isinstance(sub, Filter):
102                        name = f"{filter_idx}-{sub.__class__.__name__}"
103                        sub = AsyncFilterAdapter(sub, executor=self._executor)
104
105                    sub._set_rng_if_not_initialized(self._rng)
106                    sub.name = name
107                    sub._statistics.name = name
108                    self.filters.append(sub)
109                    filter_idx += 1
110            else:
111                name = f"{filter_idx}-{f.__class__.__name__}"
112                if isinstance(f, Filter):
113                    name = f"{filter_idx}-{f.__class__.__name__}"
114                    f = AsyncFilterAdapter(f, executor=self._executor)
115                f._set_rng_if_not_initialized(self._rng)
116                f.name = name
117                f._statistics.name = name
118                self.filters.append(f)
119                filter_idx += 1
async def apply( self, document: hojichar.Document) -> hojichar.Document:
121    async def apply(self, document: Document) -> Document:
122        stat = get_doc_info(document)
123        for filter_idx, filt in enumerate(self.filters):
124            document = await filt._apply(document)
125        new_stat = get_doc_info(document)
126        async with self._stats_lock:
127            self._statistics.update_by_diff(stat, new_stat)
128        return document

Definition of async filter behavior.

In this method, the filter will modify document.text or document.extras and set document.is_rejected = True to discard the document.

Parameters

document : Document Input document

Returns

Document Processed Document

async def apply_batch( self, batch: collections.abc.Sequence[hojichar.Document]) -> list[hojichar.Document]:
130    async def apply_batch(self, batch: Sequence[Document]) -> list[Document]:
131        stats = [get_doc_info(doc) for doc in batch]
132        for i, filt in enumerate(self.filters):
133            batch = await filt._apply_batch(batch)
134        batch = await self._finalize_batch(batch, stats)
135        return list(batch)

Apply the filter to a Sequence of documents. By default, the processing implemented in apply is executed asynchronously and concurrently. If the filter processing can be optimized for batch processing, override this method.

async def apply_stream( self, stream: collections.abc.AsyncIterable[hojichar.Document] | collections.abc.Iterable[hojichar.Document]) -> collections.abc.AsyncGenerator[hojichar.Document, None]:
137    async def apply_stream(
138        self,
139        stream: AsyncIterable[Document] | Iterable[Document],
140    ) -> AsyncGenerator[Document, None]:
141        async_stream = handle_stream_as_async(stream, chunk_size=1000, executor=self._executor)
142        async_stream = self._count_input_stats(async_stream)
143
144        for i, filt in enumerate(self.filters):
145            async_stream = filt.apply_stream(async_stream)
146
147        async for doc in async_stream:
148            in_stat = doc._get_initial_stats()
149            if in_stat is None:
150                in_stat = get_doc_info(doc)
151                self.logger.debug(
152                    "Initial stats missing for document during async stream aggregation; "
153                    "using current stats as fallback"
154                )
155            out_stat = get_doc_info(doc)
156            async with self._stats_lock:
157                self._statistics.update_by_diff(in_stat, out_stat)
158            doc._clear_initial_stats()
159            yield doc

Apply the filter to a stream of documents (Iterable or AsyncIterable). If use_batch is set to True at initialization, the filter will process documents in batches. If the stream is not asynchronous, use handle_stream_as_async to convert it to an asynchronous stream.

Even if an exception occurs during processing, the process will continue, and the following actions will be taken:

  • Set the is_rejected flag of the document to True
  • Set the error details in reject_reason
  • Increment the errors count in the statistics retrievable via get_statistics
def imap_apply( self, stream: collections.abc.AsyncIterable[hojichar.Document] | collections.abc.Iterable[hojichar.Document], *, buffer_size: int = 128, shutdown: bool = True) -> hojichar.utils.async_handlers.AsyncToSyncIterator[hojichar.Document]:
161    def imap_apply(
162        self,
163        stream: AsyncIterable[Document] | Iterable[Document],
164        *,
165        buffer_size: int = 128,
166        shutdown: bool = True,
167    ) -> AsyncToSyncIterator[Document]:
168        """Synchronously consume this asynchronous pipeline.
169
170        The returned iterator runs the pipeline on a dedicated background event loop. It is
171        intended for one-shot use and shuts down the pipeline after consumption by default.
172        Use the iterator as a context manager if iteration may stop early.
173        """
174
175        finalizer = self.shutdown if shutdown else None
176        return handle_async_stream_as_sync(
177            self.apply_stream(stream),
178            buffer_size=buffer_size,
179            finalizer=finalizer,
180        )

Synchronously consume this asynchronous pipeline.

The returned iterator runs the pipeline on a dedicated background event loop. It is intended for one-shot use and shuts down the pipeline after consumption by default. Use the iterator as a context manager if iteration may stop early.

def get_total_statistics(self) -> list[hojichar.core.models.Statistics]:
189    def get_total_statistics(self) -> list[Statistics]:
190        """
191        Get the statistics of the Compose object and sub filters.
192
193        The statistics of the Compose class are stored in an object with the name "Total",
194        and sub-filters's are stored with names in the format {filter_index}-{filter class name}.
195        """
196        stats = []
197        stats.append(self.get_statistics())
198        for i, filt in enumerate(self.filters):
199            stats.append(filt.get_statistics())
200        return stats

Get the statistics of the Compose object and sub filters.

The statistics of the Compose class are stored in an object with the name "Total", and sub-filters's are stored with names in the format {filter_index}-{filter class name}.

def get_total_statistics_map(self) -> list[dict[str, typing.Any]]:
202    def get_total_statistics_map(self) -> list[dict[str, Any]]:
203        """
204        Get the statistics of the Compose object and sub filters as a list of dictionaries.
205        """
206        stats = self.get_total_statistics()
207        return [stat.to_dict() for stat in stats]

Get the statistics of the Compose object and sub filters as a list of dictionaries.

statistics: dict

Deprecated

Get the statistics of the Compose object and sub filters.

This property is retained for compatibility with previous versions. Please use get_total_statistics or get_total_statistics_map instead.

statistics_obj: hojichar.StatsContainer

Deprecated

Get the statistics of the AsyncCompose object and sub filters. This method returns a StatsContainer object which contains the statistics of the AsyncCompose object and sub filters.

This property is retained for compatibility with previous versions. Please use get_total_statistics or get_total_statistics_map instead.

async def shutdown(self) -> None:
237    async def shutdown(self) -> None:
238        for filt in self.filters:
239            await filt.shutdown()
240        if not self._has_external_executor:
241            self._executor.shutdown()

You can override this method to release resources or perform cleanup tasks.

class AsyncFilterAdapter(hojichar.AsyncFilter):
24class AsyncFilterAdapter(AsyncFilter):
25    """
26    Adapter class for executing hojichar.Filter asynchronously.
27    """
28
29    def __init__(
30        self,
31        sync_filter: Filter,
32        *args: Any,
33        executor: Executor | None = None,
34        use_batch: bool = True,
35        **kwargs: Any,
36    ):
37        """
38        Adapter class for executing hojichar.Filter asynchronously.
39        Used to incorporate Filter into AsyncCompose.
40
41        To reduce the overhead of asynchronous context switching,
42        use_batch is set to True by default to process in batches
43        in apply_stream, regardless of the sync_filter's use_batch setting.
44
45        If performing CPU-bound and heavy processing, you can specify an executor
46        to offload the processing to the executor. However, due to Python's GIL
47        constraints, using ThreadPoolExecutor will not parallelize CPU-bound
48        processing, and the entire process will be locked.
49
50        By using ProcessPoolExecutor as the executor, it may be possible to
51        parallelize CPU-bound processing. However, for parallelizing CPU-bound
52        processing, it is recommended to use the hojichar.Parallel class to
53        parallelize synchronous Compose pipeline.
54        """
55        super().__init__(*args, use_batch=use_batch, **kwargs)
56        self.sync_filter = sync_filter
57        self._has_external_executor = executor is not None
58        self._executor = executor or ThreadPoolExecutor()
59        self.batch_size = sync_filter.batch_size
60
61    async def apply(self, document: Document) -> Document:
62        loop = asyncio.get_running_loop()
63        return await loop.run_in_executor(self._executor, self.sync_filter.apply, document)
64
65    async def apply_batch(self, batch: Sequence[Document]) -> list[Document]:
66        loop = asyncio.get_running_loop()
67        return await loop.run_in_executor(
68            self._executor,
69            lambda: self.sync_filter.apply_batch(batch),
70        )
71
72    async def shutdown(self) -> None:
73        self.sync_filter.shutdown()
74        if not self._has_external_executor:
75            self._executor.shutdown()

Adapter class for executing hojichar.Filter asynchronously.

AsyncFilterAdapter( sync_filter: hojichar.Filter, *args: Any, executor: concurrent.futures._base.Executor | None = None, use_batch: bool = True, **kwargs: Any)
29    def __init__(
30        self,
31        sync_filter: Filter,
32        *args: Any,
33        executor: Executor | None = None,
34        use_batch: bool = True,
35        **kwargs: Any,
36    ):
37        """
38        Adapter class for executing hojichar.Filter asynchronously.
39        Used to incorporate Filter into AsyncCompose.
40
41        To reduce the overhead of asynchronous context switching,
42        use_batch is set to True by default to process in batches
43        in apply_stream, regardless of the sync_filter's use_batch setting.
44
45        If performing CPU-bound and heavy processing, you can specify an executor
46        to offload the processing to the executor. However, due to Python's GIL
47        constraints, using ThreadPoolExecutor will not parallelize CPU-bound
48        processing, and the entire process will be locked.
49
50        By using ProcessPoolExecutor as the executor, it may be possible to
51        parallelize CPU-bound processing. However, for parallelizing CPU-bound
52        processing, it is recommended to use the hojichar.Parallel class to
53        parallelize synchronous Compose pipeline.
54        """
55        super().__init__(*args, use_batch=use_batch, **kwargs)
56        self.sync_filter = sync_filter
57        self._has_external_executor = executor is not None
58        self._executor = executor or ThreadPoolExecutor()
59        self.batch_size = sync_filter.batch_size

Adapter class for executing hojichar.Filter asynchronously. Used to incorporate Filter into AsyncCompose.

To reduce the overhead of asynchronous context switching, use_batch is set to True by default to process in batches in apply_stream, regardless of the sync_filter's use_batch setting.

If performing CPU-bound and heavy processing, you can specify an executor to offload the processing to the executor. However, due to Python's GIL constraints, using ThreadPoolExecutor will not parallelize CPU-bound processing, and the entire process will be locked.

By using ProcessPoolExecutor as the executor, it may be possible to parallelize CPU-bound processing. However, for parallelizing CPU-bound processing, it is recommended to use the hojichar.Parallel class to parallelize synchronous Compose pipeline.

async def apply( self, document: hojichar.Document) -> hojichar.Document:
61    async def apply(self, document: Document) -> Document:
62        loop = asyncio.get_running_loop()
63        return await loop.run_in_executor(self._executor, self.sync_filter.apply, document)

Definition of async filter behavior.

In this method, the filter will modify document.text or document.extras and set document.is_rejected = True to discard the document.

Parameters

document : Document Input document

Returns

Document Processed Document

async def apply_batch( self, batch: collections.abc.Sequence[hojichar.Document]) -> list[hojichar.Document]:
65    async def apply_batch(self, batch: Sequence[Document]) -> list[Document]:
66        loop = asyncio.get_running_loop()
67        return await loop.run_in_executor(
68            self._executor,
69            lambda: self.sync_filter.apply_batch(batch),
70        )

Apply the filter to a Sequence of documents. By default, the processing implemented in apply is executed asynchronously and concurrently. If the filter processing can be optimized for batch processing, override this method.

async def shutdown(self) -> None:
72    async def shutdown(self) -> None:
73        self.sync_filter.shutdown()
74        if not self._has_external_executor:
75            self._executor.shutdown()

You can override this method to release resources or perform cleanup tasks.

class AsyncFilter(abc.ABC):
 26class AsyncFilter(ABC):
 27    def __init__(
 28        self,
 29        *args: Any,
 30        p: float = 1.0,
 31        skip_rejected: bool = True,
 32        random_state: int | np.random.Generator | None = None,
 33        use_batch: bool = True,
 34        batch_size: int = 128,
 35        ordered: bool = True,
 36        **kwargs: Any,
 37    ):
 38        """
 39        Base class for asynchronous filters.
 40
 41        Parameters
 42        ----------
 43        p : float
 44            The probability of applying the filter.
 45            If `p` is 1, the filter will always be applied.
 46        skip_rejected : bool
 47            If `True`, the filter will skip documents that are already rejected.
 48            If you want to apply the filter to all documents (e.g., postprocess), set this to `False`.
 49        random_state : int | np.random.Generator | None
 50            Seed for the random number generator.
 51            If `None` is specified, the random number generator managed by the Compose class will be used.
 52        use_batch : bool
 53            If `True`, the filter will process documents in batches in the `apply_stream` method.
 54        batch_size : int
 55            The size of the batch to process documents in the `apply_stream` method.
 56            When `apply_batch` is not overridden, this is also the maximum number of
 57            in-flight document tasks used by the sliding-window scheduler. Ordered
 58            processing applies backpressure after at most two windows of started but
 59            not yet yielded documents.
 60        ordered : bool
 61            If `True`, `apply_stream` yields documents in input order. If `False`, filters
 62            using the default `apply_batch` implementation yield documents as soon as their
 63            processing completes.
 64        """
 65        if batch_size < 1:
 66            raise ValueError("batch_size must be at least 1")
 67
 68        self.name = self.__class__.__name__
 69        self.logger = logging.getLogger(f"{self.__module__}.{self.__class__.__name__}")
 70        assert 0 <= p <= 1
 71        self.p = p
 72        self.__init_rng(random_state)
 73        self.skip_rejected = skip_rejected
 74        self.use_batch = use_batch
 75        self.batch_size = batch_size
 76        self.ordered = ordered
 77
 78        self._statistics = Statistics(name=self.name)
 79        self._stats_lock = asyncio.Lock()
 80
 81    @abstractmethod
 82    async def apply(self, document: Document) -> Document:
 83        """
 84        Definition of async filter behavior.
 85
 86        In this method, the filter will modify `document.text` or
 87        `document.extras` and set `document.is_rejected = True` to discard the document.
 88
 89        Parameters
 90        ----------
 91        document : Document
 92            Input document
 93
 94        Returns
 95        -------
 96        Document
 97            Processed Document
 98        """
 99        pass
100
101    def _check_skip(self, document: Document) -> bool:
102        """
103        Check if the document should be skipped by this filter.
104        If `skip_rejected` is set to `True`, this method will return `True`
105        if the document is already rejected.
106        If `p` is less than 1, this method will return `True` with a probability of `1 - p`.
107        """
108        skip = self.skip_rejected and document.is_rejected
109        if skip:
110            return True
111        if self.p < 1:
112            if self._rng.random() > self.p:
113                return True
114        return False
115
116    async def _apply(self, document: Document) -> Document:
117        stats = get_doc_info(document)
118        if not self._check_skip(document):
119            document = await self.apply(document)
120        new_stats = get_doc_info(document)
121        async with self._stats_lock:
122            self._statistics.update_by_diff(stats, new_stats)
123
124        if not stats["is_rejected"] and new_stats["is_rejected"]:
125            document.reject_reason = self.get_jsonable_vars()
126        return document
127
128    async def apply_batch(self, batch: Sequence[Document]) -> list[Document]:
129        """
130        Apply the filter to a Sequence of documents.
131        By default, the processing implemented in `apply` is executed asynchronously and concurrently.
132        If the filter processing can be optimized for batch processing, override this method.
133        """
134        tasks = [self.apply(doc) for doc in batch]
135        return await asyncio.gather(*tasks)
136
137    async def _apply_batch(self, batch: Sequence[Document]) -> list[Document]:
138        skip = False
139        if self.p < 1:
140            skip = self._rng.random() > self.p
141
142        stats = [get_doc_info(doc) for doc in batch]
143        if not skip:
144            batch = await self.apply_batch(batch)
145        batch = await self._finalize_batch(batch, stats)
146        return list(batch)
147
148    async def apply_stream(
149        self,
150        stream: Iterable[Document] | AsyncIterable[Document],
151    ) -> AsyncGenerator[Document, None]:
152        """
153        Apply the filter to a stream of documents (Iterable or AsyncIterable).
154        If use_batch is set to `True` at initialization, the filter will process documents in batches.
155        If the stream is not asynchronous, use handle_stream_as_async to convert it to an asynchronous stream.
156
157        Even if an exception occurs during processing, the process will continue, and the following actions will be taken:
158        - Set the `is_rejected` flag of the document to `True`
159        - Set the error details in `reject_reason`
160        - Increment the `errors` count in the statistics retrievable via `get_statistics`
161        """
162        async_stream: AsyncIterable[Document] = handle_stream_as_async(stream)
163
164        if not self.use_batch:
165            async for doc in async_stream:
166                yield await self._try_process(doc, self._apply)
167        elif type(self).apply_batch is AsyncFilter.apply_batch:
168            async for doc in self._apply_stream_concurrently(async_stream):
169                yield doc
170        else:
171            batch: list[Document] = []
172            async for doc in async_stream:
173                if self._check_skip(doc):
174                    yield doc
175                    continue
176
177                batch.append(doc)
178                # Batch size reached, apply batch
179                if len(batch) >= self.batch_size:
180                    stats = [get_doc_info(doc) for doc in batch]
181                    batch = await self._try_process(batch, self.apply_batch)
182                    batch = await self._finalize_batch(batch, stats)
183                    for out in batch:
184                        yield out
185                    batch.clear()
186
187            # Flush remaining documents in the batch
188            if batch:
189                stats = [get_doc_info(doc) for doc in batch]
190                batch = await self._try_process(batch, self.apply_batch)
191                batch = await self._finalize_batch(batch, stats)
192                for out in batch:
193                    yield out
194
195    async def _apply_stream_concurrently(
196        self,
197        stream: AsyncIterable[Document],
198    ) -> AsyncGenerator[Document, None]:
199        """Apply documents with a bounded sliding window.
200
201        This path is used when a filter relies on the default per-document `apply_batch`
202        implementation. Filters that override `apply_batch` retain true batch processing.
203        """
204        async_iterator = stream.__aiter__()
205
206        async def next_document() -> Document:
207            return await async_iterator.__anext__()
208
209        pending: dict[asyncio.Task[Document], int] = {}
210        completed_results: dict[int, Document] = {}
211        next_input_index = 0
212        next_output_index = 0
213        source_task: asyncio.Task[Document] | None = None
214        source_exhausted = False
215
216        # Keep one window of completed out-of-order results in addition to the active
217        # window. This absorbs ordinary latency variation without allowing a stalled
218        # early document to make the scheduler consume an unbounded input stream.
219        ordered_backlog_limit = self.batch_size * 2
220
221        def schedule_next_document_if_possible() -> None:
222            nonlocal source_task
223
224            if source_exhausted or source_task is not None:
225                return
226            if len(pending) >= self.batch_size:
227                return
228            if (
229                self.ordered
230                and len(pending) + len(completed_results) >= ordered_backlog_limit
231            ):
232                return
233            source_task = asyncio.create_task(next_document())
234
235        schedule_next_document_if_possible()
236        try:
237            while source_task is not None or pending:
238                wait_for = set(pending)
239                if source_task is not None:
240                    wait_for.add(source_task)
241                done, _ = await asyncio.wait(
242                    wait_for,
243                    return_when=asyncio.FIRST_COMPLETED,
244                )
245
246                if source_task is not None and source_task in done:
247                    completed_source_task = source_task
248                    source_task = None
249                    try:
250                        document = completed_source_task.result()
251                    except StopAsyncIteration:
252                        source_exhausted = True
253                    else:
254                        skip = self._check_skip(document)
255                        processing_task = asyncio.create_task(
256                            self._process_stream_document(document, skip=skip)
257                        )
258                        pending[processing_task] = next_input_index
259                        next_input_index += 1
260
261                completed_tasks = done.intersection(pending)
262                newly_completed: list[Document] = []
263                for task in completed_tasks:
264                    input_index = pending.pop(task)
265                    result = task.result()
266                    if self.ordered:
267                        completed_results[input_index] = result
268                    else:
269                        newly_completed.append(result)
270
271                schedule_next_document_if_possible()
272
273                if self.ordered:
274                    while next_output_index in completed_results:
275                        yield completed_results.pop(next_output_index)
276                        next_output_index += 1
277                else:
278                    for result in newly_completed:
279                        yield result
280
281                # Yielding ordered results may be what frees backlog capacity.
282                schedule_next_document_if_possible()
283        finally:
284            remaining = list(pending)
285            if source_task is not None:
286                remaining.append(source_task)
287            await self._cancel_tasks(remaining)
288
289    async def _process_stream_document(self, document: Document, *, skip: bool) -> Document:
290        if skip:
291            return document
292
293        stats = [get_doc_info(document)]
294        document = await self._try_process(document, self.apply)
295        return (await self._finalize_batch([document], stats))[0]
296
297    @staticmethod
298    async def _cancel_tasks(tasks: Iterable[asyncio.Task[Any]]) -> None:
299        tasks = list(tasks)
300        for task in tasks:
301            if not task.done():
302                task.cancel()
303        if tasks:
304            await asyncio.gather(*tasks, return_exceptions=True)
305
306    async def _try_process(self, target: T, func: Callable[[T], Awaitable[T]]) -> T:
307        try:
308            return await func(target)
309        except Exception as e:
310            if isinstance(target, Document):
311                msg = f"{e!r} occurs while processing {self.name} with {target!r}"
312                self.logger.error(msg, exc_info=True)
313                target.is_rejected = True
314                target.reject_reason = {"error": msg}
315                async with self._stats_lock:
316                    self._statistics.errors += 1
317                return target
318            if isinstance(target, list):
319                msg = f"{e!r} occurs while batch processing {self.name}"
320                self.logger.error(msg, exc_info=True)
321                for doc in target:
322                    doc.is_rejected = True
323                    doc.reject_reason = {"error": msg}
324                async with self._stats_lock:
325                    self._statistics.errors += len(target)
326                return target
327            raise e
328
329    async def __call__(self, text: str) -> str:
330        document = Document(text=text)
331        return (await self._apply(document)).text
332
333    def get_statistics(self) -> Statistics:
334        """
335        Get the statistics of this filter.
336        Returns:
337            Statistics: The statistics of this filter.
338        """
339        return self._statistics
340
341    def get_statistics_map(self) -> dict[str, Statistics]:
342        """
343        Get the statistics of this filter as a dictionary.
344        """
345        return self._statistics.to_dict()
346
347    async def shutdown(self) -> None:
348        """
349        You can override this method to release resources or perform cleanup tasks.
350        """
351        pass
352
353    async def __aenter__(self) -> "AsyncFilter":
354        return self
355
356    async def __aexit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None:
357        await self.shutdown()
358
359    def get_jsonable_vars(self, exclude_keys: set[str] | None = None) -> dict[str, Any]:
360        """
361        Get the member variable of this filter.
362        Eligible variables are primitive types; [bool, int, float, str, None],
363        and the name of the variable not starts with the underscore; `_`.
364        """
365        if exclude_keys is None:
366            exclude_keys = set()
367        return {
368            k: v
369            for k, v in vars(self).items()
370            if (_is_jsonable(v) and (k not in exclude_keys) and (not k.startswith("_")))
371        }
372
373    async def _finalize_batch(
374        self,
375        batch: Sequence[Document],
376        old_stats: list[dict[str, Any]],
377    ) -> list[Document]:
378        new_stats = [get_doc_info(doc) for doc in batch]
379        for old, new, doc in zip(old_stats, new_stats, batch):
380            async with self._stats_lock:
381                self._statistics.update_by_diff(old, new)
382            if not old["is_rejected"] and new["is_rejected"] and "error" not in doc.reject_reason:
383                doc.reject_reason = self.get_jsonable_vars()
384        return list(batch)
385
386    def __init_rng(self, random_state: int | np.random.Generator | None) -> None:
387        self._owns_rng = True
388        if random_state is None:
389            self._rng = np.random.default_rng()
390            self._owns_rng = False
391        elif isinstance(random_state, int):
392            self._rng = np.random.default_rng(random_state)
393        elif isinstance(random_state, np.random.Generator):
394            self._rng = random_state
395        else:
396            raise TypeError(
397                f"random_state must be int or np.random.Generator, not {type(random_state)}"
398            )
399
400    def _set_rng_if_not_initialized(self, rng: np.random.Generator) -> None:
401        """
402        Set the random number generator for this filter if it is not already initialized.
403        This method is called by Compose class.
404        """
405        if not self._owns_rng:
406            self._rng = rng

Helper class that provides a standard way to create an ABC using inheritance.

AsyncFilter( *args: Any, p: float = 1.0, skip_rejected: bool = True, random_state: int | numpy.random._generator.Generator | None = None, use_batch: bool = True, batch_size: int = 128, ordered: bool = True, **kwargs: Any)
27    def __init__(
28        self,
29        *args: Any,
30        p: float = 1.0,
31        skip_rejected: bool = True,
32        random_state: int | np.random.Generator | None = None,
33        use_batch: bool = True,
34        batch_size: int = 128,
35        ordered: bool = True,
36        **kwargs: Any,
37    ):
38        """
39        Base class for asynchronous filters.
40
41        Parameters
42        ----------
43        p : float
44            The probability of applying the filter.
45            If `p` is 1, the filter will always be applied.
46        skip_rejected : bool
47            If `True`, the filter will skip documents that are already rejected.
48            If you want to apply the filter to all documents (e.g., postprocess), set this to `False`.
49        random_state : int | np.random.Generator | None
50            Seed for the random number generator.
51            If `None` is specified, the random number generator managed by the Compose class will be used.
52        use_batch : bool
53            If `True`, the filter will process documents in batches in the `apply_stream` method.
54        batch_size : int
55            The size of the batch to process documents in the `apply_stream` method.
56            When `apply_batch` is not overridden, this is also the maximum number of
57            in-flight document tasks used by the sliding-window scheduler. Ordered
58            processing applies backpressure after at most two windows of started but
59            not yet yielded documents.
60        ordered : bool
61            If `True`, `apply_stream` yields documents in input order. If `False`, filters
62            using the default `apply_batch` implementation yield documents as soon as their
63            processing completes.
64        """
65        if batch_size < 1:
66            raise ValueError("batch_size must be at least 1")
67
68        self.name = self.__class__.__name__
69        self.logger = logging.getLogger(f"{self.__module__}.{self.__class__.__name__}")
70        assert 0 <= p <= 1
71        self.p = p
72        self.__init_rng(random_state)
73        self.skip_rejected = skip_rejected
74        self.use_batch = use_batch
75        self.batch_size = batch_size
76        self.ordered = ordered
77
78        self._statistics = Statistics(name=self.name)
79        self._stats_lock = asyncio.Lock()

Base class for asynchronous filters.

Parameters

p : float The probability of applying the filter. If p is 1, the filter will always be applied. skip_rejected : bool If True, the filter will skip documents that are already rejected. If you want to apply the filter to all documents (e.g., postprocess), set this to False. random_state : int | np.random.Generator | None Seed for the random number generator. If None is specified, the random number generator managed by the Compose class will be used. use_batch : bool If True, the filter will process documents in batches in the apply_stream method. batch_size : int The size of the batch to process documents in the apply_stream method. When apply_batch is not overridden, this is also the maximum number of in-flight document tasks used by the sliding-window scheduler. Ordered processing applies backpressure after at most two windows of started but not yet yielded documents. ordered : bool If True, apply_stream yields documents in input order. If False, filters using the default apply_batch implementation yield documents as soon as their processing completes.

@abstractmethod
async def apply( self, document: hojichar.Document) -> hojichar.Document:
81    @abstractmethod
82    async def apply(self, document: Document) -> Document:
83        """
84        Definition of async filter behavior.
85
86        In this method, the filter will modify `document.text` or
87        `document.extras` and set `document.is_rejected = True` to discard the document.
88
89        Parameters
90        ----------
91        document : Document
92            Input document
93
94        Returns
95        -------
96        Document
97            Processed Document
98        """
99        pass

Definition of async filter behavior.

In this method, the filter will modify document.text or document.extras and set document.is_rejected = True to discard the document.

Parameters

document : Document Input document

Returns

Document Processed Document

async def apply_batch( self, batch: collections.abc.Sequence[hojichar.Document]) -> list[hojichar.Document]:
128    async def apply_batch(self, batch: Sequence[Document]) -> list[Document]:
129        """
130        Apply the filter to a Sequence of documents.
131        By default, the processing implemented in `apply` is executed asynchronously and concurrently.
132        If the filter processing can be optimized for batch processing, override this method.
133        """
134        tasks = [self.apply(doc) for doc in batch]
135        return await asyncio.gather(*tasks)

Apply the filter to a Sequence of documents. By default, the processing implemented in apply is executed asynchronously and concurrently. If the filter processing can be optimized for batch processing, override this method.

async def apply_stream( self, stream: collections.abc.Iterable[hojichar.Document] | collections.abc.AsyncIterable[hojichar.Document]) -> collections.abc.AsyncGenerator[hojichar.Document, None]:
148    async def apply_stream(
149        self,
150        stream: Iterable[Document] | AsyncIterable[Document],
151    ) -> AsyncGenerator[Document, None]:
152        """
153        Apply the filter to a stream of documents (Iterable or AsyncIterable).
154        If use_batch is set to `True` at initialization, the filter will process documents in batches.
155        If the stream is not asynchronous, use handle_stream_as_async to convert it to an asynchronous stream.
156
157        Even if an exception occurs during processing, the process will continue, and the following actions will be taken:
158        - Set the `is_rejected` flag of the document to `True`
159        - Set the error details in `reject_reason`
160        - Increment the `errors` count in the statistics retrievable via `get_statistics`
161        """
162        async_stream: AsyncIterable[Document] = handle_stream_as_async(stream)
163
164        if not self.use_batch:
165            async for doc in async_stream:
166                yield await self._try_process(doc, self._apply)
167        elif type(self).apply_batch is AsyncFilter.apply_batch:
168            async for doc in self._apply_stream_concurrently(async_stream):
169                yield doc
170        else:
171            batch: list[Document] = []
172            async for doc in async_stream:
173                if self._check_skip(doc):
174                    yield doc
175                    continue
176
177                batch.append(doc)
178                # Batch size reached, apply batch
179                if len(batch) >= self.batch_size:
180                    stats = [get_doc_info(doc) for doc in batch]
181                    batch = await self._try_process(batch, self.apply_batch)
182                    batch = await self._finalize_batch(batch, stats)
183                    for out in batch:
184                        yield out
185                    batch.clear()
186
187            # Flush remaining documents in the batch
188            if batch:
189                stats = [get_doc_info(doc) for doc in batch]
190                batch = await self._try_process(batch, self.apply_batch)
191                batch = await self._finalize_batch(batch, stats)
192                for out in batch:
193                    yield out

Apply the filter to a stream of documents (Iterable or AsyncIterable). If use_batch is set to True at initialization, the filter will process documents in batches. If the stream is not asynchronous, use handle_stream_as_async to convert it to an asynchronous stream.

Even if an exception occurs during processing, the process will continue, and the following actions will be taken:

  • Set the is_rejected flag of the document to True
  • Set the error details in reject_reason
  • Increment the errors count in the statistics retrievable via get_statistics
def get_statistics(self) -> hojichar.core.models.Statistics:
333    def get_statistics(self) -> Statistics:
334        """
335        Get the statistics of this filter.
336        Returns:
337            Statistics: The statistics of this filter.
338        """
339        return self._statistics

Get the statistics of this filter. Returns: Statistics: The statistics of this filter.

def get_statistics_map(self) -> dict[str, hojichar.core.models.Statistics]:
341    def get_statistics_map(self) -> dict[str, Statistics]:
342        """
343        Get the statistics of this filter as a dictionary.
344        """
345        return self._statistics.to_dict()

Get the statistics of this filter as a dictionary.

async def shutdown(self) -> None:
347    async def shutdown(self) -> None:
348        """
349        You can override this method to release resources or perform cleanup tasks.
350        """
351        pass

You can override this method to release resources or perform cleanup tasks.

def get_jsonable_vars(self, exclude_keys: set[str] | None = None) -> dict[str, typing.Any]:
359    def get_jsonable_vars(self, exclude_keys: set[str] | None = None) -> dict[str, Any]:
360        """
361        Get the member variable of this filter.
362        Eligible variables are primitive types; [bool, int, float, str, None],
363        and the name of the variable not starts with the underscore; `_`.
364        """
365        if exclude_keys is None:
366            exclude_keys = set()
367        return {
368            k: v
369            for k, v in vars(self).items()
370            if (_is_jsonable(v) and (k not in exclude_keys) and (not k.startswith("_")))
371        }

Get the member variable of this filter. Eligible variables are primitive types; [bool, int, float, str, None], and the name of the variable not starts with the underscore; _.