From ca3c5ee6c62b7eec5035d30241acbb50f4e5043f Mon Sep 17 00:00:00 2001 From: Vinta Chen Date: Fri, 2 Oct 2026 14:34:16 +0800 Subject: [PATCH] docs: rewrite Text Processing category intro The old intro recommended Sqids, which the 2026-10-02 audit removed from the README section. Co-Authored-By: Claude --- .../data/category_intros/text-processing.md | 44 ++++++++++--------- 1 file changed, 23 insertions(+), 21 deletions(-) diff --git a/website/data/category_intros/text-processing.md b/website/data/category_intros/text-processing.md index 70743995..efa8a35a 100644 --- a/website/data/category_intros/text-processing.md +++ b/website/data/category_intros/text-processing.md @@ -1,42 +1,44 @@ -Need a Python text processing library? charset-normalizer reads text in unknown encodings, RapidFuzz does fuzzy matching, and pyparsing builds parsers. +Mojibake like "café" is ftfy's to fix, and charset-normalizer detects encodings. For fuzzy matching, add RapidFuzz to your Python text processing libraries. How to choose: +- Text decoded with the wrong encoding: ftfy - Bytes in an unknown encoding: charset-normalizer -- Fuzzy matching against many strings: RapidFuzz -- Parsers for your own grammar: pyparsing, or parsy for small languages -- Text garbled by a wrong decode (mojibake): ftfy -- Diffs, or close matches without a dependency: difflib +- Fuzzy matching against a long list of strings: RapidFuzz +- Diffs, or close matches with nothing to install: difflib - ASCII-art banners: pyfiglet -- Translations, plus localized dates and numbers: Babel +- Translations, plus dates and numbers in each user's locale: Babel - Syntax highlighting: Pygments +- A grammar written in Python code: pyparsing, or parsy for small languages - Splitting and formatting SQL: sqlparse - Phone numbers: phonenumbers -- URL slugs: python-slugify, or Unidecode for ASCII transliteration alone -- Short public IDs: shortuuid, or Sqids to encode integer IDs +- URL slugs: python-slugify, or Unidecode for plain ASCII +- Short IDs that users will read: shortuuid -Decode with the encoding you know before you guess one. ftfy's docs say to [assume UTF-8](https://ftfy.readthedocs.io/en/latest/avoid.html#assume-utf-8) until proven otherwise, and charset-normalizer's FAQ calls detection [the last resort](https://charset-normalizer.readthedocs.io/en/latest/community/faq.html#should-i-bother-using-detection). For bytes you have no clue about, charset-normalizer is the detector [requests installs](https://github.com/psf/requests/blob/main/pyproject.toml). Call `from_bytes()` or `from_path()`, then `.best()` to [get the most probable result](https://charset-normalizer.readthedocs.io/en/latest/user/advanced_search.html). Its `detect()` is [backward compatible with chardet's](https://charset-normalizer.readthedocs.io/en/latest/user/getstarted.html), so switching is one import. If you stay on chardet, read large files and streams with [`UniversalDetector`](https://chardet.readthedocs.io/en/latest/usage.html). +ftfy [fixes Unicode that's broken in various ways](https://ftfy.readthedocs.io/en/latest/): bad Unicode goes in, good Unicode comes out. It's [a mojibake detector and fixer, not an encoding detector](https://ftfy.readthedocs.io/en/latest/detect.html#is-ftfy-an-encoding-detector), so give it text you've already tried to decode, never bytes. The function you'll call most is [`ftfy.fix_text()`](https://ftfy.readthedocs.io/en/latest/explain.html), which applies every fix it can. ftfy aims never to change text that was decoded correctly. -ftfy fixes text that was already decoded wrong, and it [doesn't take bytes](https://ftfy.readthedocs.io/en/latest/detect.html), so it's no replacement for a detector. Call `ftfy.fix_text()`, [the function you'll use most](https://ftfy.readthedocs.io/en/latest/explain.html#ftfy.fix_text). Every fix is on by default, so [check which ones fit your text](https://ftfy.readthedocs.io/en/latest/config.html). +charset-normalizer [reads text from an unknown charset encoding](https://github.com/jawah/charset_normalizer). Call `from_bytes(data).best()`, or `from_path(path).best()` for a file, and [`str()` of the result is your decoded text](https://charset-normalizer.readthedocs.io/en/latest/user/getstarted.html). chardet does the job differently: charset-normalizer decodes candidate encodings and scores the text, while chardet [scores raw bytes against per-language models](https://chardet.readthedocs.io/en/latest/faq.html#how-is-chardet-different-from-charset-normalizer). -RapidFuzz is a fast string matching library with a C++ core. To compare one string with a list, use the process module's `process.extractOne()` or `process.cdist()`, which is [faster than calling the scorers yourself](https://github.com/rapidfuzz/RapidFuzz). It doesn't lowercase or strip punctuation for you: pass `processor=utils.default_process` when case and punctuation shouldn't count. +RapidFuzz [does fuzzy string matching with various string metrics](https://rapidfuzz.github.io/RapidFuzz/), mostly in C++, with a pure Python fallback for every algorithm. To compare one string against a list, [use the `process` module](https://github.com/rapidfuzz/RapidFuzz#process), like `process.extractOne()`. -difflib ships with Python, so it works where you can't add a dependency. `get_close_matches()` returns the [good enough matches](https://docs.python.org/3/library/difflib.html#difflib.get_close_matches) for a word, and its diffs come out in unified, context, or HTML format. +difflib comes with Python and [compares sequences](https://docs.python.org/3/library/difflib.html), mostly lines of text, producing unified, context, and HTML diffs like the ones `diff` and `git diff` print. For a "did you mean" suggestion, [`get_close_matches()`](https://docs.python.org/3/library/difflib.html#difflib.get_close_matches) returns the best "good enough" matches for a word. -pyparsing lets you [build the grammar in Python code](https://github.com/pyparsing/pyparsing) instead of regular expressions. Since a grammar can accept invalid input, its docs say to use it on [input you assume is well-formatted](https://pyparsing-docs.readthedocs.io/en/latest/HowToUsePyparsing.html#usage-notes). Its [best practices](https://github.com/pyparsing/pyparsing/blob/master/pyparsing/ai/best_practices.md) say to write the grammar out in BNF first. When the grammar is recursive, call `enable_packrat()` right after the import, and give results names to the fields you read back. +pyfiglet is [a full port of FIGlet into pure Python](https://github.com/pwaller/pyfiglet): it renders text in ASCII-art fonts, as in `pyfiglet.figlet_format("text", font="slant")`. -parsy combines small parsers into bigger ones. Its docs say it [excels at small languages](https://parsy.readthedocs.io/en/latest/overview.html) and is easy to read, but to look elsewhere when you need speed or good error messages. For more complex parsers, the [`@generate` decorator](https://parsy.readthedocs.io/en/latest/ref/generating.html) is both more readable and more powerful. +Babel [does two jobs](https://babel.pocoo.org/en/latest/intro.html): it builds gettext message catalogs without the GNU gettext tools for common tasks, and it formats dates, numbers, and locale names from CLDR data. Run catalogs through the `pybabel` command: [`extract` messages, `init` a catalog per language, `update` it, and `compile` it](https://babel.pocoo.org/en/latest/cmdline.html). -Babel does two jobs: gettext message catalogs, and [CLDR locale data](https://babel.pocoo.org/en/latest/intro.html) for localized dates, numbers, and names. Run the catalog steps with the `pybabel` command: [extract, init, update, and compile](https://babel.pocoo.org/en/latest/cmdline.html). Keep times in UTC, and [convert to the user's time zone](https://babel.pocoo.org/en/latest/dates.html#time-zone-support) only for input and display. +Pygments is [a generic syntax highlighter](https://github.com/pygments/pygments) for code hosting, forums, wikis, and other apps that show source code. If you highlight code your users submit, its docs recommend you [stop the Pygments process after a short timeout](https://pygments.org/docs/security/#security-considerations). -Pygments turns code into HTML, LaTeX, ANSI, and more, as a command-line tool or a library. For HTML, it writes CSS classes instead of inline styles, so [generate the stylesheet](https://pygments.org/docs/quickstart/#example) with `HtmlFormatter().get_style_defs()`. Pick the lexer by name or file name, and [guess it](https://pygments.org/docs/quickstart/#guessing-lexers) only when you don't know the language. Pygments [doesn't guarantee how long it runs](https://pygments.org/docs/security/), so on user input, run it with a short timeout and cap how many run at once. +pyparsing [builds a grammar directly in Python code](https://github.com/pyparsing/pyparsing) from a library of classes, instead of lex/yacc or regular expressions. It handles whitespace, quoted strings, and embedded comments for you. Its best practices start with [writing a BNF](https://github.com/pyparsing/pyparsing/wiki/Best-Practices) before any code. [Use results names](https://pyparsing-docs.readthedocs.io/en/latest/HowToUsePyparsing.html#usage-notes) to get tokens by field name rather than by position. parsy [combines small parsers into larger ones](https://github.com/python-parsy/parsy) and gives you building blocks only. Its docs say it [excels at easy-to-read parsers for relatively small languages](https://parsy.readthedocs.io/en/latest/overview.html#other-python-projects). -sqlparse is a [non-validating SQL parser](https://sqlparse.readthedocs.io/en/latest/): it splits scripts into statements, formats them, and walks their tokens, without assuming a SQL dialect. Use `split()`, `format()`, and `parse()`. On SQL from untrusted sources, [keep its grouping limits](https://sqlparse.readthedocs.io/en/latest/api.html#security-and-performance-considerations) as they are. +sqlparse is [a non-validating SQL parser](https://github.com/andialbrecht/sqlparse): it splits scripts into statements, formats them, and walks their token tree, whatever the dialect. Its [three module-level functions](https://github.com/andialbrecht/sqlparse#usage), `split()`, `format()`, and `parse()`, cover most needs. -phonenumbers is a Python port of Google's libphonenumber. Pass `parse()` the region the number was dialed from, unless it's in E.164 format. Then [check it's possible and valid](https://github.com/daviddrysdale/python-phonenumbers) with `is_possible_number()` and `is_valid_number()`. +phonenumbers is [a Python port of Google's libphonenumber](https://github.com/daviddrysdale/python-phonenumbers). Parse a number together with [the country it's dialed from](https://github.com/daviddrysdale/python-phonenumbers#example-usage), unless it's in E.164 format. Then check it with `is_possible_number()` or `is_valid_number()`, and print it with `format_number()`. -python-slugify makes URL slugs, and Unidecode turns Unicode text into ASCII. Their licenses differ. [Unidecode is GPL](https://github.com/avian2/unidecode). python-slugify's own code is MIT, and by default it runs on text-unidecode, which offers the Artistic license or GPL. But python-slugify [switches to Unidecode](https://github.com/un33k/python-slugify) whenever it's installed. +python-slugify [makes Unicode-aware slugs](https://github.com/un33k/python-slugify) with your choice of transliteration backend: by default, Unidecode if it's installed, otherwise text-unidecode. Its own code is MIT, while [Unidecode is GPL](https://github.com/un33k/python-slugify#licensing), so check that license before you install Unidecode next to it. -shortuuid turns UUIDs into [short IDs for users to see](https://github.com/skorokithakis/shortuuid), and leaves out look-alike characters like l, 1, I, O, and 0. Sqids turns database keys and other integers into short IDs. Anyone can [decode them back into numbers](https://sqids.org/faq#not-recommended), so keep them away from sensitive data and user IDs. To check an ID is the canonical one, [re-encode the decoded numbers](https://sqids.org/faq#valid-ids) and compare. +Unidecode makes [lossy ASCII transliterations of Unicode text](https://github.com/avian2/unidecode), close to what someone on a US keyboard would type. Use it for ASCII machine identifiers. Its README says it's best kept off strings your users see. -Store what these libraries generate, or pin their versions, since slugs and IDs can change between releases. python-slugify says to [pin the package and its backend](https://github.com/un33k/python-slugify), and Unidecode says to [store each slug once or lock the version](https://github.com/avian2/unidecode). Sqids says to [pass your own blocklist](https://sqids.org/faq#future-blocklist), even one identical to the default. +shortuuid [generates concise, unambiguous, URL-safe UUIDs](https://github.com/skorokithakis/shortuuid) for IDs users will see: it takes UUIDs from Python's `uuid` module, writes them in base57, and drops look-alike characters such as l, 1, I, O, and 0. Call `shortuuid.uuid()` for a new ID. + +Decode with the encoding you were told whenever there is one. charset-normalizer's FAQ calls detection [a last resort](https://charset-normalizer.readthedocs.io/en/latest/community/faq.html#should-i-bother-using-detection), and ftfy's docs say to [assume UTF-8](https://ftfy.readthedocs.io/en/latest/avoid.html#assume-utf-8) unless you have a specific reason to believe otherwise.