此示例演示如何使用 Logstash 将 JSON 第二层级的数据解析到第一层级（JSON 根）。

例如，原始 JSON 数据格式如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
{
    "distinct_id":"u24a21a5e262debf",
    "time":"1679912055733",
    "time_offset":"0",
    "type":"track",
    "event":"ExposureView",
    "properties":{
        "$app_version":"8.3",
        "$wifi":true,
        "$ip":"39.144.27.121",
        "$province":"河南",
        "$city":"郑州",
        "$screen_width":390,
        "$screen_height":844,
        "$os":"iOS",
        "$manufacturer":"Apple",
        "$model":"iPhone14,5",
        "$os_version":"16.3.1",
        "$latitude":"34.740911",
        "$longitude":"113.665855",
        "product_id":"0vYqgWyuBBwDA8u8eQ7tqNjWOC5",
        "product_name":"进口香蕉牛奶200ml*12盒",
        "product_classify":"食品饮料/饮料冲调/牛奶",
        "product_price":309
    },
    "uuid":"85552fad9e1348ef9b355fa656789bfc"
}

现需要将 properties 节点下的数据解析到 JSON 的根下，期望结果如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
{
    "distinct_id":"u24a21a5e262debf",
    "time":"1679912055733",
    "time_offset":"0",
    "type":"track",
    "event":"ExposureView",
    "$app_version":"8.3",
    "$wifi":true,
    "$ip":"39.144.27.121",
    "$province":"河南",
    "$city":"郑州",
    "$screen_width":390,
    "$screen_height":844,
    "$os":"iOS",
    "$manufacturer":"Apple",
    "$model":"iPhone14,5",
    "$os_version":"16.3.1",
    "$latitude":"34.740911",
    "$longitude":"113.665855",
    "product_id":"0vYqgWyuBBwDA8u8eQ7tqNjWOC5",
    "product_name":"进口香蕉牛奶200ml*12盒",
    "product_classify":"食品饮料/饮料冲调/牛奶",
    "product_price":309,
    "uuid":"85552fad9e1348ef9b355fa656789bfc"
}

在本示例中，假设要处理的数据在 /mnt/e/log/ios.log 文件中，那么 Logstash 处理文件 logstash-test.conf 配置如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
input {
		file {
				path => ["/mnt/e/log/ios.log"]
				type => "log"
		}
}

filter {
    json {
        source => "message"
    }

    mutate {
        add_field => {"@tmp" => "%{properties}"}
    }

    json {
        source => "@tmp"
        remove_field => ["@tmp", "properties"]
    }
}

output {
		stdout {
				codec => rubydebug { metadata => true }
		}
}

使用 mutate 将 properties 字段内容先赋给 @tmp 字段，再使用 json 解析 @tmp 字段，然后移除不需要的字段 @tmp 和 properties。

指定配置文件，运行 Logstash，本示例中 logstash 命令位于 /usr/share/logstash/bin 目录下：

1
/usr/share/logstash/bin/logstash -f /opt/logstash-test/logstash-test.conf

使用 echo 命令往文件 /mnt/e/log/ios.log 写入一行记录：

1
echo '{"distinct_id":"u24a21a5e262debf","time":"1679912055733","time_offset":"0","type":"track","event":"ExposureView","properties":{"$app_version":"8.3","$wifi":true,"$ip":"39.144.27.121","$province":"河南","$city":"郑州","$screen_width":390,"$screen_height":844,"$os":"iOS","$manufacturer":"Apple","$model":"iPhone14,5","$os_version":"16.3.1","$latitude":"34.740911","$longitude":"113.665855","product_id":"0vYqgWyuBBwDA8u8eQ7tqNjWOC5","product_name":"进口香蕉牛奶200ml*12盒","product_classify":"食品饮料/饮料冲调/牛奶","product_price":309},"uuid":"85552fad9e1348ef9b355fa656789bfc"}' >> /mnt/e/log/ios.log

可以看到，在终端打印的结果如下，properties 字段的数据已经被解析到 JSON 的根下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
{
            "@version" => "1",
           "$province" => "河南",
           "$latitude" => "34.740911",
      "$screen_height" => 844,
       "$manufacturer" => "Apple",
                "host" => {
        "name" => "airoo1Oi"
    },
        "product_name" => "进口香蕉牛奶200ml*12盒",
                 "$os" => "iOS",
          "@timestamp" => 2023-03-28T09:24:25.482409200Z,
         "distinct_id" => "u24a21a5e262debf",
          "product_id" => "0vYqgWyuBBwDA8u8eQ7tqNjWOC5",
        "$app_version" => "8.3",
                "uuid" => "85552fad9e1348ef9b355fa656789bfc",
       "$screen_width" => 390,
         "$os_version" => "16.3.1",
               "$city" => "郑州",
         "time_offset" => "0",
               "$wifi" => true,
          "$longitude" => "113.665855",
                "type" => "track",
                "time" => "1679912055733",
              "$model" => "iPhone14,5",
             "message" => "{\"distinct_id\":\"u24a21a5e262debf\",\"time\":\"1679912055733\",\"time_offset\":\"0\",\"type\":\"track\",\"event\":\"ExposureView\",\"properties\":{\"$app_version\":\"8.3\",\"$wifi\":true,\"$ip\":\"39.144.27.121\",\"$province\":\"河南\",\"$city\":\"郑州\",\"$screen_width\":390,\"$screen_height\":844,\"$os\":\"iOS\",\"$manufacturer\":\"Apple\",\"$model\":\"iPhone14,5\",\"$os_version\":\"16.3.1\",\"$latitude\":\"34.740911\",\"$longitude\":\"113.665855\",\"product_id\":\"0vYqgWyuBBwDA8u8eQ7tqNjWOC5\",\"product_name\":\"进口香蕉牛奶200ml*12盒\",\"product_classify\":\"食品饮料/饮料冲调/牛奶\",\"product_price\":309},\"uuid\":\"85552fad9e1348ef9b355fa656789bfc\"}",
       "product_price" => 309,
               "event" => "ExposureView",
    "product_classify" => "食品饮料/饮料冲调/牛奶",
                 "log" => {
        "file" => {
            "path" => "/mnt/e/log/ios.log"
        }
    },
                 "$ip" => "39.144.27.121"
}

如果不需要 message，@version，log，host，@metadata，@timestamp 等字段，可以把这些字段添加到 json 解析过滤器上的 remove_field 配置项，如：

1
2
3
4
json {
        source => "@tmp"
        remove_field => ["@tmp", "properties", "message", "@version", "log", "host", "@metadata", "@timestamp"]
    }

修改完之后重新运行 logstash，再往日志文件写一条日志数据，最终打印的结果如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
{
        "$app_version" => "8.3",
       "$screen_width" => 390,
       "$manufacturer" => "Apple",
                "type" => "track",
                 "$ip" => "39.144.27.121",
           "$province" => "河南",
                 "$os" => "iOS",
           "$latitude" => "34.740911",
               "event" => "ExposureView",
          "product_id" => "0vYqgWyuBBwDA8u8eQ7tqNjWOC5",
               "$wifi" => true,
              "$model" => "iPhone14,5",
       "product_price" => 309,
                "time" => "1679912055733",
    "product_classify" => "食品饮料/饮料冲调/牛奶",
        "product_name" => "进口香蕉牛奶200ml*12盒",
         "distinct_id" => "u24a21a5e262debf",
               "$city" => "郑州",
         "time_offset" => "0",
         "$os_version" => "16.3.1",
      "$screen_height" => 844,
          "$longitude" => "113.665855",
                "uuid" => "85552fad9e1348ef9b355fa656789bfc"
}

（END）