fluent-logger-golang (fluent client-go API)
目前使用此fluent client-go API，將資料傳入Fluentd,
會使用其Async模式來避免後續的latency event被block住, 當Fluentd server有問題時:// Use "Async" to enable asynchronous I/O (connect and write)
// for sending events to Fluentd without blocking
setting.FluentdLogger.Logger, errF = fluent.New(
fluent.Config{Async: true, FluentHost: fluenthost, FluentPort: intFluentPort})
if errF != nil {
fmt.Println("[ERROR]:", errF)
setting.FluentdLogger.Enabled = false
}P.S: When Fluentd server has a problem, the events which are going to send will be buffered on the memory. The default size is 8192BufferLimit
The example of using fluent-logger-golang to send data:Sets the number of events buffered on the memory.
tag := "apm.latency"
var data = map[string]string{
"mytimestamp": strconv.FormatInt(time.Now().Unix(), 10),
"mydata": "hoge",
"myjob": "apm",
"myvalue": "55.55",
}
error := logger.Post(tag, data)FluentdFluentd
Fluentd安裝，請參考官網
https://docs.fluentd.org/installation/install-by-deb
Fluentd操作
# 啟動/關閉/查看服務
sudo systemctl start td-agent.service
sudo systemctl stop td-agent.service
sudo systemctl status td-agent.service
sudo systemctl restart td-agent.service
#修改Fluentd設定
vi /etc/td-agent/td-agent.conf
#查看Fluentd logs
cat /var/log/td-agent/td-agent.logInfluxDB 2.X
2.0 以上的InfluxDB 會需要token，Client端才有權限讀寫，其中一種方式查出Token是用InfluxDB自己的Web http://<your ip address>:8086，登入後點選"Data"
例如:
選擇GO
可以看到 token的值
// You can generate a Token from the "Tokens Tab" in the UI const token = "Iiq0TiIpL9lXn2GATwh3WeZBkLq-SEul6C0yrKLjq4T4WZ9b0BKVAsFeNs8q0Is93SMbhF0l63s4DwJja4MSbw=="
Fluent::Plugin::InfluxDB (This plugin is for using with InfluxDB 2.x 目前使用此Plugin)
This repository contains the reference Fluentd plugin for the InfluxDB 2.0.Fluentd需要此Plugin來對應寫資料到InfluxDB2
我目前用的: The configuration of /etc/td-agent/td-agent.conf
#<match apm.**>
# @type stdout
#</match>
<match apm.**>
@type copy
<store>
@type influxdb2
url <https://localhost:8086>
token Iiq0TiIpL9lXn2GATwh3WeZBkLq-SEul6C0yrKLjq4TXXXX
use_ssl false
bucket apm
org com
time_precision s
tag_keys ["mytimestamp","mydata"]
field_keys ["myvalue"]
</store>
<store>
@type stdout
</store>
</match>The result in InfluxDB
安裝此plugin:
sudo td-agent-gem install fluent-plugin-influxdb-v2
sudo td-agent-gem uninstall fluent-plugin-influxdb-v2influxdb-plugin-fluent (只適用於InfluxDB 1.x instances)
fluent-plugin-influxdb is a buffered output plugin for fluentd and influxDB.
Configuration Example
<match apm.**>
@type influxdb
host localhost
port 8086
dbname apm
user danny
password xxxxxxxxx
use_ssl false
time_precision s
tag_keys ["timestamp", "data"]
sequence_tag _seq
</match>安裝 plugin:
sudo td-agent-gem install fluent-plugin-influxdb
sudo td-agent-gem uninstall fluent-plugin-influxdbReference
遇過的Error Message
[warn]: #0 failed to flush the buffer. retry_time=7 next_retry_seconds=2021-04-22 14:28:33 14429497340994114529/137438953472000000000 +0800 chunk="5c089be4bdfb9a28fb4b5ac58228a90 7" error_class=InfluxDB2::InfluxError error="failure writing points to database: partial write: points beyond retention policy dropped=1" 2021-04-22 14:27:22 +0800 [warn]: #0 suppressed same stacktrace
https://stackoverflow.com/questions/54359348/unable-to-insert-data-in-influxdb
