BCC programming 筆記

安裝 BCC 使用 Source Code

For Bionic (18.04 LTS)

Install related libraries and packages

$ sudo apt-get -y install bison build-essential cmake flex git libedit-dev \

libllvm6.0 llvm-6.0-dev libclang-6.0-dev python zlib1g-dev libelf-dev \

iperf3 luajit libluajit-5.1-dev netperf linux-headers-$(uname -r)

Compile BCC source code and install

$ git clone --recursive https://github.com/iovisor/bcc.git

$ git submodule update --recursive

# Or $ git pull --recurse-submodules

$ mkdir bcc/build; cd bcc/build

$ cmake .. -DCMAKE_INSTALL_PREFIX=/usr -DPYTHON_CMD="python python3.8"

$ make -j `nproc`
$ sudo make install 

Quick Start Guide

A Docker container is provided for user to try out bcc.

From your host shell:

docker run -it --rm \
  --privileged \
  -v /lib/modules:/lib/modules:ro \
  -v /usr/src:/usr/src:ro \
  -v /etc/localtime:/etc/localtime:ro \
  --workdir /usr/share/bcc/tools \
  zlim/bcc

常用Linux Commands筆記

 Use rsync on my Windows 10 D:\SourceCode to sync folder from Linux's ~/SourceCode

rsync -r liudanny@<your IP address>:/home/liudanny/SourceCode /cygdrive/d/

List all users with their UID

awk -F: '{printf "%s:%s\n",$1,$3}' /etc/passwd

kubectl command completion in ~/.bashrc

# enable programmable completion features (you don't need to enable

# this, if it's already enabled in /etc/bash.bashrc and /etc/profile

# sources /etc/bash.bashrc).

if ! shopt -oq posix; then

if [ -f /usr/share/bash-completion/bash_completion ]; then

. /usr/share/bash-completion/bash_completion

elif [ -f /etc/bash_completion ]; then

. /etc/bash_completion

fi

fi

source <(kubectl completion bash)

Netfilter 研究筆記

 netfilter-like kernel module to get source and destination address

https://stackoverflow.com/questions/29656012/netfilter-like-kernel-module-to-get-source-and-destination-address

USDT Python Tracing 範例

 Reference for USDT Tracepoint for Python

https://github.com/paulross/dtrace-py

https://www.collabora.com/news-and-blog/blog/2019/05/14/an-ebpf-overview-part-5-tracing-user-processes/

https://github.com/iovisor/bcc/pull/698

Install build tools and python prerequisites

sudo apt install systemtap-sdt-dev

sudo apt install build-essential libssl-dev zlib1g-dev libncurses5-dev libncursesw5-dev libreadline-dev libsqlite3-dev libgdbm-dev libdb5.3-dev libbz2-dev libexpat1-dev liblzma-dev tk-dev libffi-dev

Download and extract python

wget https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tar.xz

tar xf Python-3.7.0.tar.xz

cd Python-3.7.0

curl -o Python-3.7.0.tgz https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tgz

tar -xzf Python-3.7.0.tgz

cd Python-3.7.0

BPFTrace 範例

BPFTrace

Install and compile

sudo apt-get update

sudo apt-get install -y bison cmake flex g++ git libelf-dev zlib1g-dev libfl-dev systemtap-sdt-dev binutils-dev

sudo apt-get install -y llvm-7-dev llvm-7-runtime libclang-7-dev clang-7

git clone https://github.com/iovisor/bpftrace

mkdir bpftrace/build; cd bpftrace/build;

cmake -DCMAKE_BUILD_TYPE=Release ..

make -j8

make install 

TCP/UDP Tracing

sudo bpftrace -e "$(cat udp_sendmsg2.bt)"

sudo bpftrace -e "$(cat udpconnect.bt)"

sudo python trace.py -I 'net/sock.h' 'udp_sendmsg(struct sock *sk) "daddr: %s, dport: %d", sk->sk_daddr, sk->sk_dport'

sudo bpftrace -e 'k:tcp_sendmsg,k:tcp_recvmsg {@[func,pid,tid,comm,args] = count();}'

sudo bpftrace -e 'k:tcp_sendmsg,k:tcp_recvmsg { @[func, pid, tid, comm] = count(); }'

sudo tcpdump -i any 'dst host 10.217.1.80 or dst host 10.217.1.50 or dst host 10.103.221.119 or dst host 10.110.176.166'

Golang 設定與常用指令說明

 Golang設定

加入下列於 ~/.profile

#Golang path

export PATH=$PATH:/usr/local/go/bin

export GOPATH=$HOME/go

export PATH=$PATH:$GOPATH/bin

---

常用的指令與使用說明

初始化 Module

這個會在資料夾底下產生一個初始的 go.mod 檔案

go mod init <module>

• 在 build go 專案的時候，會根據 module name 的最後一路路徑 myproject 當作輸出檔名，會產生 myproject.exe 的檔案
• 在 import 自己專案 package 的路徑的時候，github.com/mileslin/myproject 是根目錄，例如我新增一個 pg1 的 package ，則我在 import 的時候要輸入 import "github.com/mileslin/myproject/pg1"。

取得套件

go get -u github.com/gorilla/mux (-u 是說明，取得套件的時候不要從 cache 取得

#go get 的時候指定版本

go get github.com/gorilla/mux@v1.6.1

除了會新增一個 go.sum 檔案之外呢， go.mod 也會記錄你下載的套件，如下範例

module github.com/mileslin/myproject

go 1.13

require github.com/gorilla/mux v1.7.3 // indirect

// indirect 的意思是說，此套件還沒被用到。 所以只要在程式碼中使用此到件，並且 build 專案後，// indirect 就會消失

最後提一下，下載的套件會被放在 GOPATH/pkg/mod 底下。 (可以使用 go env 指令查看 GOPATH 路徑)

列出套件與套件版本

go list 取得可以使用的套件列表

$ go list all 會列出所有套件，包含 GOROOT 套件

$ go list -m all 只列出 module 列出的套件(包含自己的 package)

$ go list -m -versions github.com/gorilla/mux 列出此套件的所有版本

檢查套件使用情形

列出使套件在哪裡有被用到

$ go mod why github.com/gorilla/mux

Module download

這個會將 go.mod 定義的 library 都下載下來，但基本上 go build, go test 也會自動下載 library 並新增到 go.mod 上，

並會產出 go.sum 來確保 library 是否有更新。可以說能達到一樣的效果

go mod download

如果你忘了 import ，則會出現以下的錯誤

// github.com/streadway/amqp

go build -o ./bin/rabbitmq ./cmd

# rabbitmq/cmd

cmd/main.go:16:15: undefined: amqp

cmd/main.go:41:4: undefined: amqp

Module tidy

移除在 go.mod 不需要的 library

go mod tidy

---

go test

go test -timeout 30s ./probe/endpoint/... -run "TestSpyNoProcesses|TestSpyWithProcesses"

go test -timeout 30s ./probe/endpoint/...

go test -v -run=TestNewWithAssert ./example18-write-testing-and-doc/...

inspektor gadget 常用commands

 編譯與安裝kubectl-gadget

git clone https://github.com/kinvolk/inspektor-gadget.git

cd inspektor-gadget

make kubectl-gadget-linux-amd64

sudo cp kubectl-gadget-linux-amd64 /usr/local/bin/kubectl-gadget

佈署Gadgets on K8S 使用kubectl gadget

kubectl gadget deploy | kubectl apply -f -

or

#指定image

kubectl gadget deploy --image=kinvolk/gadget:fix63 | kubectl apply -f -

# make build & get image kinvolk/gadget:alban-fix63

kubectl gadget deploy --image=kinvolk/gadget:alban-fix63 | kubectl apply -f -

查詢在每個Node上的 gadget's pod name

kubectl get pod -n kube-system -l k8s-app=gadget -o=jsonpath='{.items[*].metadata.name}'

kubectl exec -it <gadget-xxxx> -n kube-system -- bash

Run BCC commands

$ kubectl gadget tcpconnect -A

$ kubectl gadget tcptracer -A

---

Traceloop

查看traceloop 清單

kubectl gadget traceloop list

NODE NAMESPACE PODNAME PODUID INDEX TRACEID CONTAINERID STATUS

ubuntu1804-k8s1 kube-system hubble-bdqn9 851a4185 0 0000b7ec718351b6 b2d36529 created 6 days ago

ubuntu1804-k8s1 default deathstar-657477f57d-crf8w bac896ec 0 0000b7f1f2b1d065 941b6b07 created 6 days ago

ubuntu1804-k8s2 kube-system coredns-6955765f44-nqr56 024f3577 0 0000541b48333690 62d1fade created 6 days ago

ubuntu1804-k8s2 kube-system coredns-6955765f44-pbwt5 00be4172 0 0000541b483869c5 59c000af created 6 days ago

ubuntu1804-k8s2 kube-system hubble-ui-7b767c4654-j7fs7 50ab917c 0 0000541b77264f34 dc9d5823 created 6 days ago

ubuntu1804-k8s2 default helloworld-golang cdb85f1e 1 0000eea8600e2b9a 82dfe69f created 4 days ago

ubuntu1804-k8s2 default helloworld-python 45139ac8 0 0000eea7d9821155 9147da4f created 4 days ago

ubuntu1804-k8s2 default deathstar-657477f57d-pflwg 63bd1b5d 0 00005421e8b30d05 fa336b14 created 6 days ago

ubuntu1804-k8s2 default helloworld-golang cdb85f1e 0 0000eeb0142b9961 b07b2652 created 4 days ago

traceloop 相關 commands

kubectl gadget traceloop pod default helloworld-golang 1

kubectl gadget traceloop show 0004ac1f39174f40

kubectl gadget traceloop pod sock-shop front-end-85c54f4fb-wftx6 0

# 須進入到gadget pod內才能使用

curl --silent --unix-socket /run/traceloop.socket http://localhost/dump-by-traceid?traceid=0000b7ec718351b6

---

Network Policy

Blog 介紹Network Policy Advisor

https://kinvolk.io/blog/2020/03/writing-kubernetes-network-policies-with-inspektor-gadgets-network-policy-advisor/

執行 network-policy monitor command

kubectl gadget network-policy monitor --namespaces default --output ./networktrace.log

製造traffic by curl

#For instance:

kubectl exec helloworld-python -- curl -s -XGET http://helloworld-golang.default.svc.cluster.local:8001/

kubectl exec xwing -- curl -s -XGET http://helloworld-python.default.svc.cluster.local:8080/

產生network-policy

kubectl gadget network-policy report --input ./networktrace.log > network-policy-default.yaml

Prometheus & Grafana & eBPF exporter 設定

 Prometheus

https://k2r2bai.com/2018/06/10/cncf/prometheus/

編輯 prometheus.yml

- job_name: "eBPF"

static_configs:

- targets: ['localhost:9435']

執行Prometheus, Grafana and ebpf_exporter

~/git/ebpf_exporter$ sudo ~/SourceCode/golang/bin/ebpf_exporter --config.file=examples/bio.yaml

cd ~/Prometheus/prometheus-2.16.0.linux-amd64

./prometheus --config.file=prometheus.yml

cd ~/Downloads/grafana-6.0.0

./bin/grafana-server web

# admin/bbyanlfe

#http://140.96.27.24:3000/dashboard/

eBPF exporter 使用說明

build eBPF exporter and run it

cd ~/SourceCode/go_SourceCode/ebpf_exporter

go build cmd/ebpf_exporter/main.go

sudo ./main --config.file=examples/timers.yaml --debbug

#http://140.96.27.24:9435/metrics

visual studio code to run ebpf_exporter with sudo

sudo code --user-data-dir=/home/liudanny/

### launch.json ###

{

// Use IntelliSense to learn about possible attributes.

// Hover to view descriptions of existing attributes.

// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387

"version": "0.2.0",

"configurations": [

{

"name": "Launch",

"type": "go",

"request": "launch",

"mode": "auto",

"program": "${workspaceFolder}/cmd/ebpf_exporter/main.go",

"env": {},

"args": ["--config.file=${workspaceFolder}/examples/timers.yaml"]

}

]

}