[Cilium] Hubble Source Code Study

 

Source Code Study:

https://github.com/cilium/cilium/blob/master/bpf/bpf_lxc.c#L1223

int handle_policy(struct __ctx_buff *ctx)

| 

-> tail_ipv4_policy()

    |

    -> ipv4_policy()

        |

        V

        https://github.com/cilium/cilium/blob/master/bpf/lib/policy_log.h#L46

        send_policy_verdict_notify()

在此function放入 struct policy_verdict_notify 格式化的資料

        |

    -> ctx_event_output()

   (這function是被remap來自skb_event_output()

   會將event送到perf's ring buffer in Kernel)

  

#Cilium 有 perf reader 可以從 Kernel 讀 perf's ring buffer 然後解開msg

https://github.com/cilium/ebpf/blob/master/perf/reader.go

#另外一個會送send_trace_notify()的來源 (有很多bpf的點都會送)

https://github.com/cilium/cilium/blob/master/bpf/bpf_network.c

int from_network(struct __ctx_buff *ctx)

   |

   V

   https://github.com/cilium/cilium/blob/master/bpf/lib/trace.h

   send_trace_notify()

   這個function會發送packet tracing events是很重要的flow來源

#上面兩個例子的notify type 是 CILIUM_NOTIFY_POLICY_VERDICT & CILIUM_NOTIFY_TRACE

#到了Hubble就會是Flow (Layer3/4)主要來源

case monitorAPI.MessageTypeDrop,

monitorAPI.MessageTypeTrace,

monitorAPI.MessageTypePolicyVerdict:

return p.l34.Decode(payload, decoded)

case monitorAPI.MessageTypeAccessLog:

return p.l7.Decode(payload, decoded)

 

BPF datapath events 定義在這

https://github.com/cilium/cilium/blob/master/pkg/monitor/api/types.go

#跟container有關的send_trace_notify()的來源

https://github.com/cilium/cilium/blob/master/bpf/bpf_lxc.c

tail_handle_ipv4()

| 

-> handle_ipv4_from_lxc()

   |

   V

   https://github.com/cilium/cilium/blob/master/bpf/lib/trace.h

   send_trace_notify()

測試:

kubectl get pods -n kube-system

kubectl exec -n kube-system ds/cilium -- cilium status

hubble observe --verdict DROPPED --pod XXX

hubble observe --since 3m --protocol http --output=json | \

jq -sr 'map(select(.l7.type=="RESPONSE" and .l7.http.method=="GET")) |

group_by(.l7.http.url) |

map({url: .[0].l7.http.url, max_latency_ms: ((map(.l7.latency_ns | tonumber) | max) / 1000000 }) |

sort_by(.max_latency_ms)'

hubble observe --since 3m --protocol tcp --output=json | \

jq -sr 'map(select(.l4.TCP.destination_port==8080)) |

group_by(.l4.TCP.flags)'

hubble observe --since=10m -t l7 -j \

| jq 'select(.l7.dns.rcode==3) | .destination.namespace + "/" + .destination.pod_name' \

| sort | uniq -c | sort -r

hubble-relay serve --peer-service=unix:///var/run/cilium/hubble.sock --listen-address=:4245

curl \

--silent \

--unix-socket /var/run/cilium/hubble.sock \

--request GET \

http://localhost/ \

| jq .

hubble serve --listen-client-urls=0.0.0.0:50051 --listen-client-urls=unix:///var/run/hubble.sock --metrics-server :6943 --metric=dns:query --metric=drop --metric=tcp --metric=flow --