Skip to content

compileall with workers would is faster with larger chunksize #117205

Closed
Closed
compileall with workers would is faster with larger chunksize#117205
Labels
performancePerformance or resource usagestdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

Description

@hauntsaninja
hauntsaninja
opened on Mar 25, 2024

Feature or enhancement

Proposal:

Currently we do a map with the default chunksize=1 over files:

cpython/Lib/compileall.py

Line 109 in 3726cb0

results = executor.map(partial(compile_file,

A quick local benchmark suggests that chunksize=4 would be 20% faster

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Linked PRs

Activity

  1. hauntsaninja
    added
    type-featureA feature request or enhancement
    on Mar 25, 2024
  2. Eclips4
    added
    performancePerformance or resource usage
    stdlibStandard Library Python modules in the Lib/ directory
    on Mar 25, 2024
  3. hauntsaninja
    added a commit that references this issue on Apr 3, 2024

    gh-117205: Increase chunksize when compiling pyc in parallel (#117206)

    b4fe02f

  4. JelleZijlstra commented on Apr 4, 2024

    @JelleZijlstra
    JelleZijlstra
    on Apr 4, 2024
    Member

    Looks like this has been done, thanks!

  5. JelleZijlstra
    closed this as completedon Apr 4, 2024
  6. diegorusso
    added a commit that references this issue on Apr 17, 2024

    pythongh-117205: Increase chunksize when compiling pyc in parallel (p…

    aef97cb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performancePerformance or resource usagestdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions