FluentD
- Unified Logging with JSON
- reliable forwarding
- pluggable architecture
Two way to setup fluentd:
td-agent
fluent-gem
Directories
# for macOs
/Library/LaunchDaemons/td-agent.plist
/etc/td-agent
/opt/td-agent
/var/log/td-agent
Fluent-gem
# install fluent-gem
gem install fluentd --no-ri --no-rdoc
fluentd --setup ./fluent
# start
fluentd -c ./fluent/fluent.conf -vv & # run in background and verbose
fluentd -c ./fluent/fluent.conf -p PROJECT_NAME/lib/fluent/plugin # run using PROJECT_NAME plugins
# stop
pkill -f fluentd
Config
Location
# td-agent
sudo vi /etc/td-agent/td-agent.conf
# gem
sudo fluentd --setup /etc/fluent
sudo vi /etc/fluent/fluent.conf
Config:
<source>
@type tail
tag "mylog"
path /Users/imantung/mylog
pos_file /Users/imantung/mylog.pos
<parse>
@type none
</parse>
</source>
<match barito>
@type stdout
</match>
Config directive
source
: where all the data come frommatch
: Tell fluentd what to do!filter
: Event processing pipeline- Set system wide configuration: the
system
directive - Group filter and output: the
label
directive - Re-use your config: the
@include
directive
Types of plugins
- Input in
source
- Parser in
source
- Output in
match
- Formatter in
match
- Filter in
filter
- Buffer used by output plugins
Input Plugins
in_tail
similar withtail -F
<source> @type tail path /var/log/httpd-access.log pos_file /var/log/td-agent/httpd-access.log.pos tag apache.access format apache2 </source>
in_forward
listens to a TCP socket to receive the event stream or UDP socket to receive heartbeat. (assume only used within private network)<source> @type forward port 24224 bind 0.0.0.0 </source>
in_secure_forward
SSL with authenticationin_udp
accept UDP payloadin_tcp
accept TCP payloadin_http
accept HTTP payloadin_unix
retrieve records from the Unix Domain Socketin_syslog
retrieve records via the syslog protocol on UDP or TCPin_exec
executes external programs to receive or pull event logsin_scribe
facebook scribein_multiprocess
use multiple CPU cores by spawning multiple child processes (able to handle very big transaction per day)in_dummy
generate dummy event<source> @type dummy dummy {"hello":"world"} </source>
Output Plugins
Non-buffered –> immediately
out_copy
copies events to multiple outputsout_null
write nothingout_roundrobin
distributes events to multiple outputs using a round-robin algorithmout_stdout
Buffered –> queued (parameter)
out_exec_filter
(1) executes an external program using an event as input and (2) reads a new event from the program outputout_forward
forwards events to other fluentd nodesout_secure_forward
SSL with authenticationout_mongo
into mongoout_mongo_replset
using mongo ReplicaSetout_splunk
send data to a Splunk HTTP Event Collector
Time Sliced –> queued with time as key (parameters)
out_exec
passes events to an external program.out_file
into fileout_s3
into amazon s3out_webhdfs
into HDFS
Writing custom plugin
Override method:
class SomeInput < Input
Fluent::Plugin.register_input('NAME', self)
def configure(conf)
def start
def shutdown
class SomeOutput < BufferedOutput
Fluent::Plugin.register_output('NAME', self)
def configure(conf)
def start
def shutdown
def format(tag, time, record)
def write(chunk)
class SomeOutput < Output
Fluent::Plugin.register_output('NAME', self)
def configure(conf)
def start
def shutdown
def emit(tag, es, chain)
class PassThruFilter < Filter
Fluent::Plugin.register_filter('passthru', self)
def configure(conf)
def start
def shutdown
def filter(tag, time, record)