如何用Logstash 收集 CSV 日志？

使用系统时间作为日志时间上传



日志样例




使用系统时间作为日志时间上传



日志样例

1. [backcolor=transparent]10.116[backcolor=transparent].[backcolor=transparent]14.201[backcolor=transparent],-,[backcolor=transparent]2[backcolor=transparent]/[backcolor=transparent]25[backcolor=transparent]/[backcolor=transparent]2016[backcolor=transparent],[backcolor=transparent]11[backcolor=transparent]:[backcolor=transparent]53[backcolor=transparent]:[backcolor=transparent]17[backcolor=transparent],[backcolor=transparent]W3SVC7[backcolor=transparent],[backcolor=transparent]2132[backcolor=transparent],[backcolor=transparent]200[backcolor=transparent],[backcolor=transparent]0[backcolor=transparent],[backcolor=transparent]GET[backcolor=transparent],[backcolor=transparent]project[backcolor=transparent]/[backcolor=transparent]shenzhen[backcolor=transparent]-[backcolor=transparent]test[backcolor=transparent]/[backcolor=transparent]logstore[backcolor=transparent]/[backcolor=transparent]logstash[backcolor=transparent]/[backcolor=transparent]detail[backcolor=transparent],[backcolor=transparent]C[backcolor=transparent]:[backcolor=transparent]\test\csv\test_csv[backcolor=transparent].[backcolor=transparent]log

1. [backcolor=transparent]input [backcolor=transparent]{
2. [backcolor=transparent]  file [backcolor=transparent]{
3. [backcolor=transparent]    type [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent]"csv_log_1"
4. [backcolor=transparent]    path [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent][[backcolor=transparent]"C:/test/csv/*.log"[backcolor=transparent]]
5. [backcolor=transparent]    start_position [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent]"beginning"
6. [backcolor=transparent]  [backcolor=transparent]}
7. [backcolor=transparent]}
9. [backcolor=transparent]filter [backcolor=transparent]{
10. [backcolor=transparent]  [backcolor=transparent]if[backcolor=transparent] [backcolor=transparent][[backcolor=transparent]type[backcolor=transparent]][backcolor=transparent] [backcolor=transparent]==[backcolor=transparent] [backcolor=transparent]"csv_log_1"[backcolor=transparent] [backcolor=transparent]{
11. [backcolor=transparent]  csv [backcolor=transparent]{
12. [backcolor=transparent]    separator [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent]","
13. [backcolor=transparent]    columns [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent][[backcolor=transparent]"ip"[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]"a"[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]"date"[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]"time"[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]"b"[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]"latency"[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]"status"[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]"size"[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]"method"[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]"url"[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]"file"[backcolor=transparent]]
14. [backcolor=transparent]  [backcolor=transparent]}[backcolor=transparent]
15. [backcolor=transparent]  [backcolor=transparent]}
16. [backcolor=transparent]}
18. [backcolor=transparent]output [backcolor=transparent]{
19. [backcolor=transparent]  [backcolor=transparent]if[backcolor=transparent] [backcolor=transparent][[backcolor=transparent]type[backcolor=transparent]][backcolor=transparent] [backcolor=transparent]==[backcolor=transparent] [backcolor=transparent]"csv_log_1"[backcolor=transparent] [backcolor=transparent]{
20. [backcolor=transparent]  logservice [backcolor=transparent]{
21. [backcolor=transparent]        codec [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent]"json"
22. [backcolor=transparent]        endpoint [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent]"***"
23. [backcolor=transparent]        project [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent]"***"
24. [backcolor=transparent]        logstore [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent]"***"
25. [backcolor=transparent]        topic [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent]""
26. [backcolor=transparent]        source [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent]""
27. [backcolor=transparent]        access_key_id [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent]"***"
28. [backcolor=transparent]        access_key_secret [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent]"***"
29. [backcolor=transparent]        max_send_retry [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent]10
30. [backcolor=transparent]    [backcolor=transparent]}
31. [backcolor=transparent]    [backcolor=transparent]}
32. [backcolor=transparent]}

[backcolor=transparent]注意：

• 配置文件格式必须以 UTF-8 无 BOM 格式编码，可以下载 notepad++ 修改文件编码格式。
• path 填写文件路径时请使用 UNIX 模式的分隔符，如：C:/test/multiline/*.log，否则无法支持模糊匹配。
• type 字段需要统一修改并在该文件内保持一致，如果单台机器存在多个Logstash配置文件，需要保证各配置type 字段唯一，否则会导致数据处理的错乱。

1. [backcolor=transparent]10.116[backcolor=transparent].[backcolor=transparent]14.201[backcolor=transparent],-,[backcolor=transparent]Feb[backcolor=transparent] [backcolor=transparent]25[backcolor=transparent] [backcolor=transparent]2016[backcolor=transparent] [backcolor=transparent]14[backcolor=transparent]:[backcolor=transparent]03[backcolor=transparent]:[backcolor=transparent]44[backcolor=transparent],[backcolor=transparent]W3SVC7[backcolor=transparent],[backcolor=transparent]1332[backcolor=transparent],[backcolor=transparent]200[backcolor=transparent],[backcolor=transparent]0[backcolor=transparent],[backcolor=transparent]GET[backcolor=transparent],[backcolor=transparent]project[backcolor=transparent]/[backcolor=transparent]shenzhen[backcolor=transparent]-[backcolor=transparent]test[backcolor=transparent]/[backcolor=transparent]logstore[backcolor=transparent]/[backcolor=transparent]logstash[backcolor=transparent]/[backcolor=transparent]detail[backcolor=transparent],[backcolor=transparent]C[backcolor=transparent]:[backcolor=transparent]\test\csv\test_csv_withtime[backcolor=transparent].[backcolor=transparent]log

1. [backcolor=transparent]input [backcolor=transparent]{
2. [backcolor=transparent]  file [backcolor=transparent]{
3. [backcolor=transparent]    type [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent]"csv_log_2"
4. [backcolor=transparent]    path [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent][[backcolor=transparent]"C:/test/csv_withtime/*.log"[backcolor=transparent]]
5. [backcolor=transparent]    start_position [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent]"beginning"
6. [backcolor=transparent]  [backcolor=transparent]}
7. [backcolor=transparent]}
9. [backcolor=transparent]filter [backcolor=transparent]{
10. [backcolor=transparent]  [backcolor=transparent]if[backcolor=transparent] [backcolor=transparent][[backcolor=transparent]type[backcolor=transparent]][backcolor=transparent] [backcolor=transparent]==[backcolor=transparent] [backcolor=transparent]"csv_log_2"[backcolor=transparent] [backcolor=transparent]{
11. [backcolor=transparent]  csv [backcolor=transparent]{
12. [backcolor=transparent]    separator [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent]","
13. [backcolor=transparent]    columns [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent][[backcolor=transparent]"ip"[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]"a"[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]"datetime"[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]"b"[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]"latency"[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]"status"[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]"size"[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]"method"[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]"url"[backcolor=transparent],[backcolor=transparent] [backcolor=transparent]"file"[backcolor=transparent]]
14. [backcolor=transparent]  [backcolor=transparent]}[backcolor=transparent]
15. [backcolor=transparent]  date [backcolor=transparent]{
16. [backcolor=transparent]    match [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent][[backcolor=transparent] [backcolor=transparent]"datetime"[backcolor=transparent] [backcolor=transparent],[backcolor=transparent] [backcolor=transparent]"MMM dd YYYY HH:mm:ss"[backcolor=transparent] [backcolor=transparent]]
17. [backcolor=transparent]  [backcolor=transparent]}
18. [backcolor=transparent]  [backcolor=transparent]}
19. [backcolor=transparent]}
21. [backcolor=transparent]output [backcolor=transparent]{
22. [backcolor=transparent]  [backcolor=transparent]if[backcolor=transparent] [backcolor=transparent][[backcolor=transparent]type[backcolor=transparent]][backcolor=transparent] [backcolor=transparent]==[backcolor=transparent] [backcolor=transparent]"csv_log_2"[backcolor=transparent] [backcolor=transparent]{
23. [backcolor=transparent]  logservice [backcolor=transparent]{
24. [backcolor=transparent]        codec [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent]"json"
25. [backcolor=transparent]        endpoint [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent]"***"
26. [backcolor=transparent]        project [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent]"***"
27. [backcolor=transparent]        logstore [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent]"***"
28. [backcolor=transparent]        topic [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent]""
29. [backcolor=transparent]        source [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent]""
30. [backcolor=transparent]        access_key_id [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent]"***"
31. [backcolor=transparent]        access_key_secret [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent]"***"
32. [backcolor=transparent]        max_send_retry [backcolor=transparent]=>[backcolor=transparent] [backcolor=transparent]10
33. [backcolor=transparent]    [backcolor=transparent]}
34. [backcolor=transparent]    [backcolor=transparent]}
35. [backcolor=transparent]}

[backcolor=transparent]注意：

• 配置文件格式必须以 UTF-8 无 BOM 格式编码，可以下载 notepad++ 修改文件编码格式。
• path 填写文件路径时请使用 UNIX 模式的分隔符，如：C:/test/multiline/*.log，否则无法支持模糊匹配。
• type 字段需要统一修改并在该文件内保持一致，如果单台机器存在多个Logstash配置文件，需要保证各配置type 字段唯一，否则会导致数据处理的错乱。