Skip to content

Commit a80bf53

Browse files
phph
committed
Initial doc move
1 parent c9fe9af commit a80bf53

File tree

2 files changed

+256
-1
lines changed

2 files changed

+256
-1
lines changed

‎docs/index.asciidoc

+255
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,255 @@
1+
:plugin: file
2+
:type: input
3+
4+
///////////////////////////////////////////
5+
START - GENERATED VARIABLES, DO NOT EDIT!
6+
///////////////////////////////////////////
7+
:version: %VERSION%
8+
:release_date: %RELEASE_DATE%
9+
:changelog_url: %CHANGELOG_URL%
10+
:include_path: ../../../logstash/docs/include
11+
///////////////////////////////////////////
12+
END - GENERATED VARIABLES, DO NOT EDIT!
13+
///////////////////////////////////////////
14+
15+
[id="plugins-{type}-{plugin}"]
16+
17+
=== File
18+
19+
include::{include_path}/plugin_header.asciidoc[]
20+
21+
==== Description
22+
23+
Stream events from files, normally by tailing them in a manner
24+
similar to `tail -0F` but optionally reading them from the
25+
beginning.
26+
27+
By default, each event is assumed to be one line and a line is
28+
taken to be the text before a newline character.
29+
Normally, logging will add a newline to the end of each line written.
30+
If you would like to join multiple log lines into one event,
31+
you'll want to use the multiline codec or filter.
32+
33+
The plugin aims to track changing files and emit new content as it's
34+
appended to each file. It's not well-suited for reading a file from
35+
beginning to end and storing all of it in a single event (not even
36+
with the multiline codec or filter).
37+
38+
==== Reading from remote network volumes
39+
40+
The file input is not tested on remote filesystems such as NFS, Samba, s3fs-fuse, etc. These
41+
remote filesystems typically have behaviors that are very different from local filesystems and
42+
are therefore unlikely to work correctly when used with the file input.
43+
44+
==== Tracking of current position in watched files
45+
46+
The plugin keeps track of the current position in each file by
47+
recording it in a separate file named sincedb. This makes it
48+
possible to stop and restart Logstash and have it pick up where it
49+
left off without missing the lines that were added to the file while
50+
Logstash was stopped.
51+
52+
By default, the sincedb file is placed in the home directory of the
53+
user running Logstash with a filename based on the filename patterns
54+
being watched (i.e. the `path` option). Thus, changing the filename
55+
patterns will result in a new sincedb file being used and any
56+
existing current position state will be lost. If you change your
57+
patterns with any frequency it might make sense to explicitly choose
58+
a sincedb path with the `sincedb_path` option.
59+
60+
A different `sincedb_path` must be used for each input. Using the same
61+
path will cause issues. The read checkpoints for each input must be
62+
stored in a different path so the information does not override.
63+
64+
Sincedb files are text files with four columns:
65+
66+
. The inode number (or equivalent).
67+
. The major device number of the file system (or equivalent).
68+
. The minor device number of the file system (or equivalent).
69+
. The current byte offset within the file.
70+
71+
On non-Windows systems you can obtain the inode number of a file
72+
with e.g. `ls -li`.
73+
74+
==== File rotation
75+
76+
File rotation is detected and handled by this input, regardless of
77+
whether the file is rotated via a rename or a copy operation. To
78+
support programs that write to the rotated file for some time after
79+
the rotation has taken place, include both the original filename and
80+
the rotated filename (e.g. /var/log/syslog and /var/log/syslog.1) in
81+
the filename patterns to watch (the `path` option). Note that the
82+
rotated filename will be treated as a new file so if
83+
`start_position` is set to 'beginning' the rotated file will be
84+
reprocessed.
85+
86+
With the default value of `start_position` ('end') any messages
87+
written to the end of the file between the last read operation prior
88+
to the rotation and its reopening under the new name (an interval
89+
determined by the `stat_interval` and `discover_interval` options)
90+
will not get picked up.
91+
92+
[id="plugins-{type}s-{plugin}-options"]
93+
==== File Input Configuration Options
94+
95+
This plugin supports the following configuration options plus the <<plugins-{type}s-common-options>> described later.
96+
97+
[cols="<,<,<",options="header",]
98+
|=======================================================================
99+
|Setting |Input type|Required
100+
| <<plugins-{type}s-{plugin}-close_older>> |<<number,number>>|No
101+
| <<plugins-{type}s-{plugin}-delimiter>> |<<string,string>>|No
102+
| <<plugins-{type}s-{plugin}-discover_interval>> |<<number,number>>|No
103+
| <<plugins-{type}s-{plugin}-exclude>> |<<array,array>>|No
104+
| <<plugins-{type}s-{plugin}-ignore_older>> |<<number,number>>|No
105+
| <<plugins-{type}s-{plugin}-max_open_files>> |<<number,number>>|No
106+
| <<plugins-{type}s-{plugin}-path>> |<<array,array>>|Yes
107+
| <<plugins-{type}s-{plugin}-sincedb_path>> |<<string,string>>|No
108+
| <<plugins-{type}s-{plugin}-sincedb_write_interval>> |<<number,number>>|No
109+
| <<plugins-{type}s-{plugin}-start_position>> |<<string,string>>, one of `["beginning", "end"]`|No
110+
| <<plugins-{type}s-{plugin}-stat_interval>> |<<number,number>>|No
111+
|=======================================================================
112+
113+
Also see <<plugins-{type}s-common-options>> for a list of options supported by all
114+
input plugins.
115+
116+
&nbsp;
117+
118+
[id="plugins-{type}s-{plugin}-close_older"]
119+
===== `close_older`
120+
121+
* Value type is <<number,number>>
122+
* Default value is `3600`
123+
124+
The file input closes any files that were last read the specified
125+
timespan in seconds ago.
126+
This has different implications depending on if a file is being tailed or
127+
read. If tailing, and there is a large time gap in incoming data the file
128+
can be closed (allowing other files to be opened) but will be queued for
129+
reopening when new data is detected. If reading, the file will be closed
130+
after closed_older seconds from when the last bytes were read.
131+
The default is 1 hour
132+
133+
[id="plugins-{type}s-{plugin}-delimiter"]
134+
===== `delimiter`
135+
136+
* Value type is <<string,string>>
137+
* Default value is `"\n"`
138+
139+
set the new line delimiter, defaults to "\n"
140+
141+
[id="plugins-{type}s-{plugin}-discover_interval"]
142+
===== `discover_interval`
143+
144+
* Value type is <<number,number>>
145+
* Default value is `15`
146+
147+
How often (in seconds) we expand the filename patterns in the
148+
`path` option to discover new files to watch.
149+
150+
[id="plugins-{type}s-{plugin}-exclude"]
151+
===== `exclude`
152+
153+
* Value type is <<array,array>>
154+
* There is no default value for this setting.
155+
156+
Exclusions (matched against the filename, not full path). Filename
157+
patterns are valid here, too. For example, if you have
158+
[source,ruby]
159+
path => "/var/log/*"
160+
161+
You might want to exclude gzipped files:
162+
[source,ruby]
163+
exclude => "*.gz"
164+
165+
[id="plugins-{type}s-{plugin}-ignore_older"]
166+
===== `ignore_older`
167+
168+
* Value type is <<number,number>>
169+
* There is no default value for this setting.
170+
171+
When the file input discovers a file that was last modified
172+
before the specified timespan in seconds, the file is ignored.
173+
After it's discovery, if an ignored file is modified it is no
174+
longer ignored and any new data is read. By default, this option is
175+
disabled. Note this unit is in seconds.
176+
177+
[id="plugins-{type}s-{plugin}-max_open_files"]
178+
===== `max_open_files`
179+
180+
* Value type is <<number,number>>
181+
* There is no default value for this setting.
182+
183+
What is the maximum number of file_handles that this input consumes
184+
at any one time. Use close_older to close some files if you need to
185+
process more files than this number. This should not be set to the
186+
maximum the OS can do because file handles are needed for other
187+
LS plugins and OS processes.
188+
The default of 4095 is set in filewatch.
189+
190+
[id="plugins-{type}s-{plugin}-path"]
191+
===== `path`
192+
193+
* This is a required setting.
194+
* Value type is <<array,array>>
195+
* There is no default value for this setting.
196+
197+
The path(s) to the file(s) to use as an input.
198+
You can use filename patterns here, such as `/var/log/*.log`.
199+
If you use a pattern like `/var/log/**/*.log`, a recursive search
200+
of `/var/log` will be done for all `*.log` files.
201+
Paths must be absolute and cannot be relative.
202+
203+
You may also configure multiple paths. See an example
204+
on the <<array,Logstash configuration page>>.
205+
206+
[id="plugins-{type}s-{plugin}-sincedb_path"]
207+
===== `sincedb_path`
208+
209+
* Value type is <<string,string>>
210+
* There is no default value for this setting.
211+
212+
Path of the sincedb database file (keeps track of the current
213+
position of monitored log files) that will be written to disk.
214+
The default will write sincedb files to `<path.data>/plugins/inputs/file`
215+
NOTE: it must be a file path and not a directory path
216+
217+
[id="plugins-{type}s-{plugin}-sincedb_write_interval"]
218+
===== `sincedb_write_interval`
219+
220+
* Value type is <<number,number>>
221+
* Default value is `15`
222+
223+
How often (in seconds) to write a since database with the current position of
224+
monitored log files.
225+
226+
[id="plugins-{type}s-{plugin}-start_position"]
227+
===== `start_position`
228+
229+
* Value can be any of: `beginning`, `end`
230+
* Default value is `"end"`
231+
232+
Choose where Logstash starts initially reading files: at the beginning or
233+
at the end. The default behavior treats files like live streams and thus
234+
starts at the end. If you have old data you want to import, set this
235+
to 'beginning'.
236+
237+
This option only modifies "first contact" situations where a file
238+
is new and not seen before, i.e. files that don't have a current
239+
position recorded in a sincedb file read by Logstash. If a file
240+
has already been seen before, this option has no effect and the
241+
position recorded in the sincedb file will be used.
242+
243+
[id="plugins-{type}s-{plugin}-stat_interval"]
244+
===== `stat_interval`
245+
246+
* Value type is <<number,number>>
247+
* Default value is `1`
248+
249+
How often (in seconds) we stat files to see if they have been modified.
250+
Increasing this interval will decrease the number of system calls we make,
251+
but increase the time to detect new log lines.
252+
253+
254+
255+
include::{include_path}/{type}.asciidoc[]

‎logstash-input-file.gemspec

+1-1
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ Gem::Specification.new do |s|
1111
s.require_paths = ["lib"]
1212

1313
# Files
14-
s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
14+
s.files = Dir["lib/**/*","spec/**/*","*.gemspec","*.md","CONTRIBUTORS","Gemfile","LICENSE","NOTICE.TXT", "vendor/jar-dependencies/**/*.jar", "vendor/jar-dependencies/**/*.rb", "VERSION", "docs/**/*"]
1515

1616
# Tests
1717
s.test_files = s.files.grep(%r{^(test|spec|features)/})

0 commit comments

Comments
 (0)