Skip to content

Optimize name table in unicodedata by excluding names derived by rule NR2 #144882

Closed
Closed
Optimize name table in unicodedata by excluding names derived by rule NR2#144882
Labels
3.15bugs and security fixesextension-modulesC modules in the Modules dirperformancePerformance or resource usagetopic-unicodetype-featureA feature request or enhancement

Description

@serhiy-storchaka
serhiy-storchaka
opened on Feb 16, 2026

For most ideographs, the Name property value is derived by concatenating a script-specific prefix string to the code point, expressed in uppercase hexadecimal, with the usual 4- to 6-digit convention (see rule NR2 in chapter 4.8.1 of Unicode 17.0.0 spec).

Thus, names for Hangul syllables and most Han and Tangut ideographic characters are not explicitly listed in UnicodeData.txt. They are generated algorithmically in unicodedata. See #80667. But ideographic characters for scripts other than Han and Tangut, as well as Egyptian hieroglyphs, have their names listed explicitly in UnicodeData.txt, even when their names are derived by rule NR2. We can reduce the name table if exclude names derived by rule NR2 and generate them using existing code.

Linked PRs

Activity

  1. serhiy-storchaka
    added
    performancePerformance or resource usage
    topic-unicode
    3.15bugs and security fixes
    on Feb 16, 2026
  2. serhiy-storchaka
    added a commit that references this issue on Feb 16, 2026

    pythongh-144882: Optimize name tables in unicodedata by excluding nam…

    50980ce

  3. serhiy-storchaka commented on Feb 16, 2026

    @serhiy-storchaka
    serhiy-storchaka
    on Feb 16, 2026
    MemberAuthor

    Before:

    size of dawg [KiB] 165.27
packed_name_dawg: 169237 bytes
dawg_pos_to_codepoint: 165648 bytes
Size of original table: 2228224 bytes
Best: 4352+52224 bins at shift 8; 108800 bytes
dawg_codepoint_to_pos_index1: 4352 bytes
dawg_codepoint_to_pos_index2: 104448 bytes

    After:

    size of dawg [KiB] 164.75
packed_name_dawg: 168706 bytes
dawg_pos_to_codepoint: 142144 bytes
Size of original table: 2228224 bytes
Best: 4352+46336 bins at shift 8; 97024 bytes
dawg_codepoint_to_pos_index1: 4352 bytes
dawg_codepoint_to_pos_index2: 92672 bytes
  4. StanFromIreland
    added
    type-featureA feature request or enhancement
    extension-modulesC modules in the Modules dir
    on Feb 16, 2026
  5. serhiy-storchaka
    added a commit that references this issue on Feb 18, 2026

    gh-144882: Optimize name tables in unicodedata by excluding names der…

    767b4d0
  6. serhiy-storchaka
    closed this as completedon Feb 18, 2026
  7. brijkapadia
    added a commit that references this issue on Feb 28, 2026

    pythongh-144882: Optimize name tables in unicodedata by excluding nam…

    75d1eaa
  8. ljfp
    added a commit that references this issue on Apr 25, 2026

    pythongh-144882: Optimize name tables in unicodedata by excluding nam…

    70c8837
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.15bugs and security fixesextension-modulesC modules in the Modules dirperformancePerformance or resource usagetopic-unicodetype-featureA feature request or enhancement

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions