Search results
12 packages found
Sort by: Default
- Default
- Most downloaded this week
- Most downloaded this month
- Most dependents
- Recently published
Multilingual tokenizer that automatically tags each token with its type
published version 5.3.0, 3 years ago17 dependents licensed under $MIT
56,941
Port from Apache Lucene
published version 5.3.3, 10 years ago0 dependents licensed under $ISC
82
Lexer generator from RegExp spec
published version 0.1.10, 5 years ago0 dependents licensed under $MIT
73
NFA Tokenizer
published version 0.1.8, 8 years ago1 dependents licensed under $MIT
47
Lexical tokenizer with grammar classes ready to go.
published version 1.0.10, 4 years ago0 dependents licensed under $ISC
34
Mathematical expression solver / Reverse Polish Notation calculator for NodeJS
published version 1.0.4, 4 years ago0 dependents licensed under $MIT
30
A text tokenizing library that handles strings, by tokenizing them into arrays, depending on intended format, like sentences, sub sentences, paragraphs, words...
- NLP tokenizer
- Tokenizer
- Sentence tokenizer
- Text tokenizer
- Node.js tokenizer
- Flexible text tokenizer
- CommonJS text tokenizer
- Paragraph tokenizer
- Sub sentence tokenizer
- Stable tokenizer
- Easy tokenizer
- Light text tokenizer
published version 1.2.0, a year ago0 dependents licensed under $MIT
34
A fast bleu score calculator
published version 0.1.4, a month ago0 dependents licensed under $MIT
34
module for nksoft tokens
published version 1.1.2, 5 years ago0 dependents licensed under $ISC
28
RWKV / gpt-NeoX / Pythia, 0-dep tokenizer library, for nodejs
published version 1.0.5, 2 years ago1 dependents licensed under $MIT
23
A simple lexer for javascript
published version 1.0.2, 5 years ago1 dependents licensed under $GPL-3.0-or-later
13
A super simple basic parser / tokenizer for easier processing of various configuration files
published version 1.1.0, a year ago0 dependents licensed under $MIT
11