tokenizer_t Struct Reference

Represents a handle to a Tokenizer, capable of recognizing meaningful Token substrings in text. More...

#include <hyperon.h>

Public Attributes
const struct RustTokenizer *	tokenizer

Detailed Description

Represents a handle to a Tokenizer, capable of recognizing meaningful Token substrings in text.

Note

tokenizer_t handles must be freed with tokenizer_free()

Member Data Documentation

tokenizer

const struct RustTokenizer* tokenizer_t::tokenizer

Internal. Should not be accessed directly

---

The documentation for this struct was generated from the following file:

• hyperonc-install/include/hyperonc/hyperon/hyperon.h