Skip to content

Corpus Utilities

Utilities to tidy large Tamil corpora before training / RAG.

1) Punctuation normalization

from tamil_utils.corpus import normalize_punct

s = ' “இது”  ஒரு  சோதனை …  சரி  !  இது  இரண்டாம்  ? '
print(normalize_punct(s))
# -> "இது" ஒரு சோதனை ... சரி! இது இரண்டாம்?

What it does

  • Curly quotes → straight quotes
  • Ellipsis … → ... (kept atomic)
  • Collapses redundant whitespace
  • No space before . ! ? : ; and closers
  • Exactly one space after punctuation (unless end of line)

2) Stable de-duplication

from tamil_utils.corpus import dedup_lines

lines = ["தமிழ் NLP\n", "தமிழ் nlp\n", "  தமிழ் NLP\n", "Tamil nlp\n"]
print(dedup_lines(lines))
# -> ['தமிழ் NLP\n', 'Tamil nlp\n']

Rules & options

  • First occurrence wins
  • casefold=True, strip=True by default (affects Latin; preserves originals)
  • Call as dedup_lines(lines, casefold=False, strip=False) to disable either behavior

3) Length filters (chars & tokens)

from tamil_utils.corpus import filter_by_length

data = ["இது", "இது ஒரு", "இது ஒரு சோதனை", "சரி!"]
out = list(filter_by_length(data, min_tokens=2, max_tokens=3))
print(out)
# -> ['இது ஒரு', 'இது ஒரு சோதனை']

Notes

  • Tokenization uses tamil_utils.tokens after NFC normalize

  • Combine character and token bounds as needed:

  • min_chars, max_chars

  • min_tokens, max_tokens

4) Sentence windows for RAG

from tamil_utils.corpus import window_sents

txt = "இது ஒன்று. இது இரண்டு? சரி! முடிந்தது."
print(window_sents(txt, k=2, stride=1))
# -> ['இது ஒன்று. இது இரண்டு?', 'இது இரண்டு? சரி!', 'சரி! முடிந்தது.']

Why

  • Joins k sentences per window with a sliding stride
  • Great for chunking long docs for retrieval pipelines

Tips

  • Use normalize_punct → sents → window_sents for clean, chunked inputs
  • For PowerShell piping, prefer UTF-8 files or run python -X utf8