Mirrors_Framework/awesome-python
1
0
Fork 0
mirror of https://github.com/vinta/awesome-python.git synced 2026-10-20 12:53:33 +08:00
Code Issues Packages Projects Releases Wiki Activity

docs: rewrite Text Processing category intro

Browse Source 
The old intro recommended Sqids, which the 2026-10-02 audit removed from the README section.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Vinta Chen
2026-10-02 14:34:16 +08:00
co-authored by Claude
parent c3df3b6a9d
commit ca3c5ee6c6
1 changed files with 23 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
website/data/category_intros/text-processing.md
+23 -21
View File
@@ -1,42 +1,44 @@
Need a Python text processing library? charset-normalizer reads text in unknown encodings, RapidFuzz does fuzzy matching, and pyparsing builds parsers.
Mojibake like "café" is ftfy's to fix, and charset-normalizer detects encodings. For fuzzy matching, add RapidFuzz to your Python text processing libraries.
How to choose:
- Text decoded with the wrong encoding: ftfy
- Bytes in an unknown encoding: charset-normalizer
- Fuzzy matching against many strings: RapidFuzz
- Parsers for your own grammar: pyparsing, or parsy for small languages
- Text garbled by a wrong decode (mojibake): ftfy
- Diffs, or close matches without a dependency: difflib
- Fuzzy matching against a long list of strings: RapidFuzz
- Diffs, or close matches with nothing to install: difflib
- ASCII-art banners: pyfiglet
- Translations, plus localized dates and numbers: Babel
- Translations, plus dates and numbers in each user's locale: Babel
- Syntax highlighting: Pygments
- A grammar written in Python code: pyparsing, or parsy for small languages
- Splitting and formatting SQL: sqlparse
- Phone numbers: phonenumbers
- URL slugs: python-slugify, or Unidecode for ASCII transliteration alone
- Short public IDs: shortuuid, or Sqids to encode integer IDs
- URL slugs: python-slugify, or Unidecode for plain ASCII
- Short IDs that users will read: shortuuid
Decode with the encoding you know before you guess one. ftfy's docs say to [assume UTF-8](https://ftfy.readthedocs.io/en/latest/avoid.html#assume-utf-8) until proven otherwise, and charset-normalizer's FAQ calls detection [the last resort](https://charset-normalizer.readthedocs.io/en/latest/community/faq.html#should-i-bother-using-detection). For bytes you have no clue about, charset-normalizer is the detector [requests installs](https://github.com/psf/requests/blob/main/pyproject.toml). Call `from_bytes()` or `from_path()`, then `.best()` to [get the most probable result](https://charset-normalizer.readthedocs.io/en/latest/user/advanced_search.html). Its `detect()` is [backward compatible with chardet's](https://charset-normalizer.readthedocs.io/en/latest/user/getstarted.html), so switching is one import. If you stay on chardet, read large files and streams with [`UniversalDetector`](https://chardet.readthedocs.io/en/latest/usage.html).
ftfy [fixes Unicode that's broken in various ways](https://ftfy.readthedocs.io/en/latest/): bad Unicode goes in, good Unicode comes out. It's [a mojibake detector and fixer, not an encoding detector](https://ftfy.readthedocs.io/en/latest/detect.html#is-ftfy-an-encoding-detector), so give it text you've already tried to decode, never bytes. The function you'll call most is [`ftfy.fix_text()`](https://ftfy.readthedocs.io/en/latest/explain.html), which applies every fix it can. ftfy aims never to change text that was decoded correctly.
ftfy fixes text that was already decoded wrong, and it [doesn't take bytes](https://ftfy.readthedocs.io/en/latest/detect.html), so it's no replacement for a detector. Call `ftfy.fix_text()`, [the function you'll use most](https://ftfy.readthedocs.io/en/latest/explain.html#ftfy.fix_text). Every fix is on by default, so [check which ones fit your text](https://ftfy.readthedocs.io/en/latest/config.html).
charset-normalizer [reads text from an unknown charset encoding](https://github.com/jawah/charset_normalizer). Call `from_bytes(data).best()`, or `from_path(path).best()` for a file, and [`str()` of the result is your decoded text](https://charset-normalizer.readthedocs.io/en/latest/user/getstarted.html). chardet does the job differently: charset-normalizer decodes candidate encodings and scores the text, while chardet [scores raw bytes against per-language models](https://chardet.readthedocs.io/en/latest/faq.html#how-is-chardet-different-from-charset-normalizer).
RapidFuzz is a fast string matching library with a C++ core. To compare one string with a list, use the process module's `process.extractOne()` or `process.cdist()`, which is [faster than calling the scorers yourself](https://github.com/rapidfuzz/RapidFuzz). It doesn't lowercase or strip punctuation for you: pass `processor=utils.default_process` when case and punctuation shouldn't count.
RapidFuzz [does fuzzy string matching with various string metrics](https://rapidfuzz.github.io/RapidFuzz/), mostly in C++, with a pure Python fallback for every algorithm. To compare one string against a list, [use the `process` module](https://github.com/rapidfuzz/RapidFuzz#process), like `process.extractOne()`.
difflib ships with Python, so it works where you can't add a dependency. `get_close_matches()` returns the [good enough matches](https://docs.python.org/3/library/difflib.html#difflib.get_close_matches) for a word, and its diffs come out in unified, context, or HTML format.
difflib comes with Python and [compares sequences](https://docs.python.org/3/library/difflib.html), mostly lines of text, producing unified, context, and HTML diffs like the ones `diff` and `git diff` print. For a "did you mean" suggestion, [`get_close_matches()`](https://docs.python.org/3/library/difflib.html#difflib.get_close_matches) returns the best "good enough" matches for a word.
pyparsing lets you [build the grammar in Python code](https://github.com/pyparsing/pyparsing) instead of regular expressions. Since a grammar can accept invalid input, its docs say to use it on [input you assume is well-formatted](https://pyparsing-docs.readthedocs.io/en/latest/HowToUsePyparsing.html#usage-notes). Its [best practices](https://github.com/pyparsing/pyparsing/blob/master/pyparsing/ai/best_practices.md) say to write the grammar out in BNF first. When the grammar is recursive, call `enable_packrat()` right after the import, and give results names to the fields you read back.
pyfiglet is [a full port of FIGlet into pure Python](https://github.com/pwaller/pyfiglet): it renders text in ASCII-art fonts, as in `pyfiglet.figlet_format("text", font="slant")`.
parsy combines small parsers into bigger ones. Its docs say it [excels at small languages](https://parsy.readthedocs.io/en/latest/overview.html) and is easy to read, but to look elsewhere when you need speed or good error messages. For more complex parsers, the [`@generate` decorator](https://parsy.readthedocs.io/en/latest/ref/generating.html) is both more readable and more powerful.
Babel [does two jobs](https://babel.pocoo.org/en/latest/intro.html): it builds gettext message catalogs without the GNU gettext tools for common tasks, and it formats dates, numbers, and locale names from CLDR data. Run catalogs through the `pybabel` command: [`extract` messages, `init` a catalog per language, `update` it, and `compile` it](https://babel.pocoo.org/en/latest/cmdline.html).
Babel does two jobs: gettext message catalogs, and [CLDR locale data](https://babel.pocoo.org/en/latest/intro.html) for localized dates, numbers, and names. Run the catalog steps with the `pybabel` command: [extract, init, update, and compile](https://babel.pocoo.org/en/latest/cmdline.html). Keep times in UTC, and [convert to the user's time zone](https://babel.pocoo.org/en/latest/dates.html#time-zone-support) only for input and display.
Pygments is [a generic syntax highlighter](https://github.com/pygments/pygments) for code hosting, forums, wikis, and other apps that show source code. If you highlight code your users submit, its docs recommend you [stop the Pygments process after a short timeout](https://pygments.org/docs/security/#security-considerations).
Pygments turns code into HTML, LaTeX, ANSI, and more, as a command-line tool or a library. For HTML, it writes CSS classes instead of inline styles, so [generate the stylesheet](https://pygments.org/docs/quickstart/#example) with `HtmlFormatter().get_style_defs()`. Pick the lexer by name or file name, and [guess it](https://pygments.org/docs/quickstart/#guessing-lexers) only when you don't know the language. Pygments [doesn't guarantee how long it runs](https://pygments.org/docs/security/), so on user input, run it with a short timeout and cap how many run at once.
pyparsing [builds a grammar directly in Python code](https://github.com/pyparsing/pyparsing) from a library of classes, instead of lex/yacc or regular expressions. It handles whitespace, quoted strings, and embedded comments for you. Its best practices start with [writing a BNF](https://github.com/pyparsing/pyparsing/wiki/Best-Practices) before any code. [Use results names](https://pyparsing-docs.readthedocs.io/en/latest/HowToUsePyparsing.html#usage-notes) to get tokens by field name rather than by position. parsy [combines small parsers into larger ones](https://github.com/python-parsy/parsy) and gives you building blocks only. Its docs say it [excels at easy-to-read parsers for relatively small languages](https://parsy.readthedocs.io/en/latest/overview.html#other-python-projects).
sqlparse is a [non-validating SQL parser](https://sqlparse.readthedocs.io/en/latest/): it splits scripts into statements, formats them, and walks their tokens, without assuming a SQL dialect. Use `split()`, `format()`, and `parse()`. On SQL from untrusted sources, [keep its grouping limits](https://sqlparse.readthedocs.io/en/latest/api.html#security-and-performance-considerations) as they are.
sqlparse is [a non-validating SQL parser](https://github.com/andialbrecht/sqlparse): it splits scripts into statements, formats them, and walks their token tree, whatever the dialect. Its [three module-level functions](https://github.com/andialbrecht/sqlparse#usage), `split()`, `format()`, and `parse()`, cover most needs.
phonenumbers is a Python port of Google's libphonenumber. Pass `parse()` the region the number was dialed from, unless it's in E.164 format. Then [check it's possible and valid](https://github.com/daviddrysdale/python-phonenumbers) with `is_possible_number()` and `is_valid_number()`.
phonenumbers is [a Python port of Google's libphonenumber](https://github.com/daviddrysdale/python-phonenumbers). Parse a number together with [the country it's dialed from](https://github.com/daviddrysdale/python-phonenumbers#example-usage), unless it's in E.164 format. Then check it with `is_possible_number()` or `is_valid_number()`, and print it with `format_number()`.
python-slugify makes URL slugs, and Unidecode turns Unicode text into ASCII. Their licenses differ. [Unidecode is GPL](https://github.com/avian2/unidecode). python-slugify's own code is MIT, and by default it runs on text-unidecode, which offers the Artistic license or GPL. But python-slugify [switches to Unidecode](https://github.com/un33k/python-slugify) whenever it's installed.
python-slugify [makes Unicode-aware slugs](https://github.com/un33k/python-slugify) with your choice of transliteration backend: by default, Unidecode if it's installed, otherwise text-unidecode. Its own code is MIT, while [Unidecode is GPL](https://github.com/un33k/python-slugify#licensing), so check that license before you install Unidecode next to it.
shortuuid turns UUIDs into [short IDs for users to see](https://github.com/skorokithakis/shortuuid), and leaves out look-alike characters like l, 1, I, O, and 0. Sqids turns database keys and other integers into short IDs. Anyone can [decode them back into numbers](https://sqids.org/faq#not-recommended), so keep them away from sensitive data and user IDs. To check an ID is the canonical one, [re-encode the decoded numbers](https://sqids.org/faq#valid-ids) and compare.
Unidecode makes [lossy ASCII transliterations of Unicode text](https://github.com/avian2/unidecode), close to what someone on a US keyboard would type. Use it for ASCII machine identifiers. Its README says it's best kept off strings your users see.
Store what these libraries generate, or pin their versions, since slugs and IDs can change between releases. python-slugify says to [pin the package and its backend](https://github.com/un33k/python-slugify), and Unidecode says to [store each slug once or lock the version](https://github.com/avian2/unidecode). Sqids says to [pass your own blocklist](https://sqids.org/faq#future-blocklist), even one identical to the default.
shortuuid [generates concise, unambiguous, URL-safe UUIDs](https://github.com/skorokithakis/shortuuid) for IDs users will see: it takes UUIDs from Python's `uuid` module, writes them in base57, and drops look-alike characters such as l, 1, I, O, and 0. Call `shortuuid.uuid()` for a new ID.
Decode with the encoding you were told whenever there is one. charset-normalizer's FAQ calls detection [a last resort](https://charset-normalizer.readthedocs.io/en/latest/community/faq.html#should-i-bother-using-detection), and ftfy's docs say to [assume UTF-8](https://ftfy.readthedocs.io/en/latest/avoid.html#assume-utf-8) unless you have a specific reason to believe otherwise.