Too many open files [errno=24]

[bdurairaji2-lims]

Bug Report

Describe the bug

We are experiencing a "Too many open files" error with the Fluent Bit Tail input when monitoring a large number of log files on Windows Server.

Our application is used by approximately 100 users, and approximately 5,000–10,000 audit log files can be generated per day.

Fluent Bit monitors these files using the Tail input and sends the records to OpenSearch.

We have observed that Fluent Bit works normally while the number of files is below approximately 8,200. When the number of files increases beyond approximately 8,200, Fluent Bit starts reporting a "Too many open files" error.

If we manually remove some files and reduce the number of files below approximately 8,200, Fluent Bit starts working normally again.

The files from previous days have already been completely processed and their records have been successfully sent to OpenSearch. However, the files remain on disk because they are required for audit/retention purposes.

We would like to understand how Fluent Bit manages files that have reached EOF and have already been successfully processed.

To Reproduce

• Rubular link if applicable: Not applicable

• Example log message if applicable:

[input:tail:tail.90] cannot open E:\LIMSAudit\AuditTextFilePath\solidmedia-478-2026-09-08.json
[2026/09/16 12:33:57.394] [error] [D:\a\fluent-bit\fluent-bit\plugins\in_tail\tail_file.c:1348 errno=24] Too many open files

• Steps to reproduce the problem:

1. Configure Fluent Bit Tail input to monitor a directory containing JSON audit log files.
2. Configure the following settings:

@SET tail_ignore_older=5d
@SET tail_refresh_interval=15

3. Generate approximately 5,000 files on Day 1.
4. Allow Fluent Bit to completely process the files and confirm that the records have been successfully delivered to OpenSearch.
5. Keep the Day-1 files in the audit directory; do not delete them.
6. On Day 2, generate another approximately 5,000 files.
7. The directory now contains approximately 10,000 files.
8. When the number of files reaches approximately 8,200, Fluent Bit starts reporting "Too many open files".
9. Remove some files so that the number of files falls below approximately 8,200.
10. Fluent Bit starts processing normally again.

We have reproduced the behavior based on the number of files present in the directory.

Expected behavior

We expect Fluent Bit to continue processing newly created log files even when a large number of previously processed files remain in the directory.

Once a file has:

• been completely read by Fluent Bit,
• reached EOF, and
• had all its records successfully delivered to OpenSearch,

we would expect Fluent Bit to release unnecessary OS file handles/resources associated with that completed file while retaining the required file state/offset information.

The physical audit files must remain on disk for audit and retention purposes.

We expect Fluent Bit to support a directory containing a large number of completed files without failing with "Too many open files", provided the files are no longer being actively written.

Screenshots

If applicable, we can provide screenshots showing:

• Fluent Bit error
• Number of files in the directory
• Fluent Bit process handle count
• CPU/memory usage

Your Environment

• Version used: Fluent Bit 5.0.6
• Configuration:

@SET tail_ignore_older=5d
@SET tail_refresh_interval=15

Additional Tail configuration:

[INPUT]
    Name              tail
    Path              ${audit_base_folder}\specimen-*.json
    Tag               specimen
    Parser            json
    Ignore_Older      ${tail_ignore_older}
    Read_From_Head   True
    Refresh_Interval  ${tail_refresh_interval}
    Read_Newly_Discovered_Files_From_Head True
    DB                <configured DB/state file>
    storage.type      filesystem

• Environment name and version: Standalone Fluent Bit installation

• Server type and version: Windows Server 2025

• Operating System and version: Windows Server 2025

• Filters and plugins:

  • Tail input
  • JSON parser
  • OpenSearch output
  • Filesystem storage

Additional context

Our LIMS application generates a large number of small audit JSON files. Approximately 5,000–10,000 files can be generated per day.

The files are not continuously modified after they are created. Fluent Bit processes the files and sends their records to OpenSearch. The files must remain on disk for audit and retention requirements.

The important observation is that the issue appears to correlate with the number of files:

• ~5,000 files → Fluent Bit works normally.
• ~8,000 files → Fluent Bit continues to work.
• ~8,200+ files → "Too many open files" occurs.
• Removing files and reducing the count below ~8,200 → Fluent Bit resumes normal operation.

Concern regarding Ignore_Older

We have configured:

Ignore_Older=5d

However, we do not want to use Ignore_Older as the solution because our requirement is to ensure that all audit files are eventually processed and delivered to OpenSearch.

For example, assume Fluent Bit encounters the "Too many open files" error and remains unavailable for more than 5 days.

During this period, new audit files continue to be generated and remain on disk.

If Fluent Bit resumes after 10 days, files that were not successfully processed may now be older than the configured Ignore_Older=5d period.

Our concern is that these files could then be ignored based on their age, even though their records were never successfully delivered to OpenSearch.

For example:

Day 1
  File created
      ↓
Fluent Bit unable to process file

Day 2–10
  File remains on disk
  Fluent Bit remains unavailable

Day 11
  Fluent Bit resumes
      ↓
  File is older than 5 days
      ↓
  Ignore_Older=5d
      ↓
  File may be ignored
      ↓
  Audit record may not reach OpenSearch

Therefore, reducing the Ignore_Older period or relying on Ignore_Older is not an acceptable solution for our use case.

We need a solution that allows Fluent Bit to:

1. Continue to retain the physical audit files on disk.
2. Release unnecessary OS file handles/resources for files that have reached EOF and have been successfully processed.
3. Retain the required file offset/state information.
4. Continue processing files that have not yet been successfully processed.
5. Avoid skipping unprocessed files simply because they have become older than a configured age.
6. Reliably support a large number of audit files in the directory.

Questions for Fluent Bit Team

Could you please clarify:

1. After a Tail input file reaches EOF and all records have been successfully delivered to OpenSearch, does Fluent Bit continue to maintain an OS file handle for that file?

2. Does Fluent Bit continue to actively monitor/track previously completed files while they remain within the Ignore_Older period?

3. Does the Tail input maintain an open file descriptor/handle for every discovered file, or is there another mechanism for managing a large number of files?

4. Is there a known limitation or issue with the Tail input on Windows when monitoring approximately 8,000+ files?

5. Is the approximately 8,200-file threshold related to a Fluent Bit limitation, Windows limitation, or the Tail input's file-handle management?

6. Is there a recommended configuration that allows Fluent Bit to release OS file handles for completed files while retaining their offset/state information?

7. Can Fluent Bit distinguish between:

   • files that have been successfully processed and delivered, and
   • files that have not yet been successfully processed,

   so that unprocessed files are not skipped based solely on their age?

8. What is the recommended Fluent Bit architecture/configuration for processing 5,000–10,000 new files per day while retaining previously processed files on disk?

9. Would using multiple Tail inputs/directories, or another Fluent Bit configuration, be recommended for this workload?

10. Is there any specific Fluent Bit 5.0.6 configuration or known issue related to file-handle management that we should consider?

Our primary requirement is data reliability. We need to retain the audit files on disk and ensure that all files are eventually processed and delivered to OpenSearch, even if Fluent Bit is unavailable for an extended period.

We would appreciate your recommended solution for handling this workload without deleting files and without relying on Ignore_Older as a workaround.

[cosmo0920]

4. Is there a known limitation or issue with the Tail input on Windows when monitoring approximately 8,000+ files?

There's a limitation for this. This is because the number of limit of opening files at the same time is capped with 8192 files which stands for low I/O Level (_open, _read): Supports up to 8,192 files concurrently through the C run-time library.

ref: https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/setmaxstdio?view=msvc-170#remarks

[bdurairaji2-lims]

Hi @cosmo0920,

Thank you for the clarification. We understand that this limitation is related to the Windows C runtime I/O level, which supports a maximum of 8,192 simultaneously open files. That is clear.

However, we have a few questions regarding Fluent Bit's Tail input behaviour and a possible solution for our audit logging scenario.

1. File handle management after reading a file

When Fluent Bit reads a file and reaches EOF, does it keep the file handle open for continuous monitoring, or does it close the file handle and retain only the file offset information?

If the file handle remains open after the file has been completely read, does Fluent Bit continue to keep all previously processed files open?

If so, is there any reason the file handle cannot be closed while retaining the file offset in the database, allowing Fluent Bit to resume reading from the correct position if the file receives new data?

2. Understanding the Ignore_Older configuration

We understand that Fluent Bit provides the Ignore_Older configuration option to ignore older files or records based on the configured time threshold.

For example, if we configure:

Ignore_Older: 5d

Files or records older than five days would be ignored, depending on how this option is applied.

However, if more than 8,192 files are opened within that five-day period, wouldn't we still encounter the same Too many open files (errno=24) error?

In other words, Ignore_Older may help limit the files being processed, but it may not completely resolve the Windows file-handle limitation when a large number of files fall within the configured time period.

3. Possible new configuration option to close previously read files

Would it be possible to introduce a new configuration option in the Tail input, such as:

Close_Open_Files_After: 1d

The expected behaviour would be:

Fluent Bit should read files regardless of their age, ensuring that older audit files are not skipped.
After a file has been completely read and has remained inactive for the configured duration, Fluent Bit should close its file handle.
The file offset and processing state should be retained in the database so that Fluent Bit can resume reading from the correct position if the file receives new data.
Files should be reopened when necessary, without losing or duplicating audit records.

For example, with Close_Open_Files_After 1d, Fluent Bit could close file handles for files that have been completely read and have remained inactive for more than one day, while retaining their processing state.

This would allow us to retain all audit files for our required retention period without requiring Fluent Bit to keep every file handle open simultaneously.

Could you please let us know whether this behaviour is already supported by Fluent Bit, or whether introducing a configuration option like Close_Open_Files_After would be a feasible enhancement to address this limitation on Windows?

Thank you for your time and guidance.

As we are new to Fluent Bit, we would appreciate it if someone could provide a technical solution or guidance for this issue.

Regards,
Babu

[bdurairaji2-lims]

Hi Team,

I have implemented code changes to close files that are no longer in use.

What I implemented

Introduced a new tail option, Close_Open_Files_After, which monitors open files and closes those that have not been used for the configured interval.

Behavior:

Files are read fully regardless of age (old audit files are never skipped).
A periodic collector (in_tail_collect_close_files) closes the handle of any file that is fully read and has had no new data for the configured duration.
The resume offset is persisted (SQLite DB when enabled, plus an in-memory dormant_file_sizes map), and the fd is released.
On the next path scan, a dormant file is only re-opened if its size grew beyond the stored offset — so idle files don't consume a handle. When reopened, it resumes from the exact offset (no loss, no duplicates). Truncation is detected and handled.

Files changed:

tail_config.h — close_open_files_after, coll_fd_close_files, dormant_file_sizes
tail_config.c — create/destroy the dormant hash table
tail_scan.c / tail_scan.h — register/unregister/fetch dormant offsets
tail_scan_glob.c / tail_scan_win32.c — size-based re-pickup on scan
tail_file.c / tail_file.h — flb_tail_file_make_dormant()
tail.c — collector, registration, pause/resume, config-map entry
Guards: only plain files that are cleanly at EOF go dormant (compressed/.gz, buffered partial lines, and rotated files are skipped for safety).

Could you please validate the code and provide your suggestions?

If the implementation is correct, will this fix be available in the next release so that we can use it in our application?

close_open_files_after_changes.zip