Thursday, September 16, 2021

BCC programming 筆記

安裝 BCC 使用 Source Code
For Bionic (18.04 LTS)

Install related libraries and packages
$ sudo apt-get -y install bison build-essential cmake flex git libedit-dev \
libllvm6.0 llvm-6.0-dev libclang-6.0-dev python zlib1g-dev libelf-dev \
iperf3 luajit libluajit-5.1-dev netperf linux-headers-$(uname -r)

Compile BCC source code and install
$ git clone --recursive https://github.com/iovisor/bcc.git
$ git submodule update --recursive
# Or $ git pull --recurse-submodules

$ mkdir bcc/build; cd bcc/build
$ cmake .. -DCMAKE_INSTALL_PREFIX=/usr -DPYTHON_CMD="python python3.8"
$ make -j `nproc`
$ sudo make install 


Quick Start Guide

A Docker container is provided for user to try out bcc.

From your host shell:

docker run -it --rm \
  --privileged \
  -v /lib/modules:/lib/modules:ro \
  -v /usr/src:/usr/src:ro \
  -v /etc/localtime:/etc/localtime:ro \
  --workdir /usr/share/bcc/tools \
  zlim/bcc

常用Linux Commands筆記

 Use rsync on my Windows 10 D:\SourceCode to sync folder from Linux's ~/SourceCode

rsync -r liudanny@<your IP address>:/home/liudanny/SourceCode /cygdrive/d/

List all users with their UID
awk -F: '{printf "%s:%s\n",$1,$3}' /etc/passwd

kubectl command completion in ~/.bashrc
# enable programmable completion features (you don't need to enable
# this, if it's already enabled in /etc/bash.bashrc and /etc/profile
# sources /etc/bash.bashrc).
if ! shopt -oq posix; then
if [ -f /usr/share/bash-completion/bash_completion ]; then
. /usr/share/bash-completion/bash_completion
elif [ -f /etc/bash_completion ]; then
. /etc/bash_completion
fi
fi

source <(kubectl completion bash)

Netfilter 研究筆記

 netfilter-like kernel module to get source and destination address

USDT Python Tracing 範例

 Reference for USDT Tracepoint for Python

https://github.com/paulross/dtrace-py
https://www.collabora.com/news-and-blog/blog/2019/05/14/an-ebpf-overview-part-5-tracing-user-processes/
https://github.com/iovisor/bcc/pull/698

Install build tools and python prerequisites
sudo apt install systemtap-sdt-dev

sudo apt install build-essential libssl-dev zlib1g-dev libncurses5-dev libncursesw5-dev libreadline-dev libsqlite3-dev libgdbm-dev libdb5.3-dev libbz2-dev libexpat1-dev liblzma-dev tk-dev libffi-dev

Download and extract python
wget https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tar.xz
tar xf Python-3.7.0.tar.xz
cd Python-3.7.0
curl -o Python-3.7.0.tgz https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tgz
tar -xzf Python-3.7.0.tgz
cd Python-3.7.0

BPFTrace 範例

BPFTrace
Install and compile
sudo apt-get update
sudo apt-get install -y bison cmake flex g++ git libelf-dev zlib1g-dev libfl-dev systemtap-sdt-dev binutils-dev
sudo apt-get install -y llvm-7-dev llvm-7-runtime libclang-7-dev clang-7
git clone https://github.com/iovisor/bpftrace
mkdir bpftrace/build; cd bpftrace/build;
cmake -DCMAKE_BUILD_TYPE=Release ..
make -j8

make install 

TCP/UDP Tracing

sudo bpftrace -e "$(cat udp_sendmsg2.bt)"
sudo bpftrace -e "$(cat udpconnect.bt)"
sudo python trace.py -I 'net/sock.h' 'udp_sendmsg(struct sock *sk) "daddr: %s, dport: %d", sk->sk_daddr, sk->sk_dport'

sudo bpftrace -e 'k:tcp_sendmsg,k:tcp_recvmsg {@[func,pid,tid,comm,args] = count();}'
sudo bpftrace -e 'k:tcp_sendmsg,k:tcp_recvmsg { @[func, pid, tid, comm] = count(); }'
sudo tcpdump -i any 'dst host 10.217.1.80 or dst host 10.217.1.50 or dst host 10.103.221.119 or dst host 10.110.176.166'

Golang 設定與常用指令說明

 Golang設定

加入下列於 ~/.profile
#Golang path
export PATH=$PATH:/usr/local/go/bin
export GOPATH=$HOME/go
export PATH=$PATH:$GOPATH/bin


常用的指令與使用說明
初始化 Module
這個會在資料夾底下產生一個初始的 go.mod 檔案
go mod init <module>
  • 在 build go 專案的時候,會根據 module name 的最後一路路徑 myproject 當作輸出檔名,會產生 myproject.exe 的檔案
  • 在 import 自己專案 package 的路徑的時候,github.com/mileslin/myproject 是根目錄,例如我新增一個 pg1 的 package ,則我在 import 的時候要輸入 import "github.com/mileslin/myproject/pg1"。

取得套件
go get -u github.com/gorilla/mux (-u 是說明,取得套件的時候不要從 cache 取得

#go get 的時候指定版本
go get github.com/gorilla/mux@v1.6.1
除了會新增一個 go.sum 檔案之外呢, go.mod 也會記錄你下載的套件,如下範例
module github.com/mileslin/myproject
go 1.13
require github.com/gorilla/mux v1.7.3 // indirect
// indirect 的意思是說,此套件還沒被用到。 所以只要在程式碼中使用此到件,並且 build 專案後,// indirect 就會消失
最後提一下,下載的套件會被放在 GOPATH/pkg/mod 底下。 (可以使用 go env 指令查看 GOPATH 路徑)

列出套件與套件版本
go list 取得可以使用的套件列表
$ go list all 會列出所有套件,包含 GOROOT 套件
$ go list -m all 只列出 module 列出的套件(包含自己的 package)
$ go list -m -versions github.com/gorilla/mux 列出此套件的所有版本

檢查套件使用情形
列出使套件在哪裡有被用到
$ go mod why github.com/gorilla/mux

Module download
這個會將 go.mod 定義的 library 都下載下來,但基本上 go build, go test 也會自動下載 library 並新增到 go.mod 上,
並會產出 go.sum 來確保 library 是否有更新。可以說能達到一樣的效果
go mod download

如果你忘了 import ,則會出現以下的錯誤
// github.com/streadway/amqp
go build -o ./bin/rabbitmq ./cmd
# rabbitmq/cmd
cmd/main.go:16:15: undefined: amqp
cmd/main.go:41:4: undefined: amqp

Module tidy
移除在 go.mod 不需要的 library
go mod tidy


go test
go test -timeout 30s ./probe/endpoint/... -run "TestSpyNoProcesses|TestSpyWithProcesses"
go test -timeout 30s ./probe/endpoint/...
go test -v -run=TestNewWithAssert ./example18-write-testing-and-doc/...

inspektor gadget 常用commands

 編譯與安裝kubectl-gadget

git clone https://github.com/kinvolk/inspektor-gadget.git
cd inspektor-gadget
make kubectl-gadget-linux-amd64
sudo cp kubectl-gadget-linux-amd64 /usr/local/bin/kubectl-gadget

佈署Gadgets on K8S 使用kubectl gadget
kubectl gadget deploy | kubectl apply -f -

or
#指定image
kubectl gadget deploy --image=kinvolk/gadget:fix63 | kubectl apply -f -
# make build & get image kinvolk/gadget:alban-fix63
kubectl gadget deploy --image=kinvolk/gadget:alban-fix63 | kubectl apply -f -

查詢在每個Node上的 gadget's pod name
kubectl get pod -n kube-system -l k8s-app=gadget -o=jsonpath='{.items[*].metadata.name}'
kubectl exec -it <gadget-xxxx> -n kube-system -- bash

Run BCC commands
$ kubectl gadget tcpconnect -A
$ kubectl gadget tcptracer -A


Traceloop
查看traceloop 清單
kubectl gadget traceloop list
NODE NAMESPACE PODNAME PODUID INDEX TRACEID CONTAINERID STATUS
ubuntu1804-k8s1 kube-system hubble-bdqn9 851a4185 0 0000b7ec718351b6 b2d36529 created 6 days ago
ubuntu1804-k8s1 default deathstar-657477f57d-crf8w bac896ec 0 0000b7f1f2b1d065 941b6b07 created 6 days ago
ubuntu1804-k8s2 kube-system coredns-6955765f44-nqr56 024f3577 0 0000541b48333690 62d1fade created 6 days ago
ubuntu1804-k8s2 kube-system coredns-6955765f44-pbwt5 00be4172 0 0000541b483869c5 59c000af created 6 days ago
ubuntu1804-k8s2 kube-system hubble-ui-7b767c4654-j7fs7 50ab917c 0 0000541b77264f34 dc9d5823 created 6 days ago
ubuntu1804-k8s2 default helloworld-golang cdb85f1e 1 0000eea8600e2b9a 82dfe69f created 4 days ago
ubuntu1804-k8s2 default helloworld-python 45139ac8 0 0000eea7d9821155 9147da4f created 4 days ago
ubuntu1804-k8s2 default deathstar-657477f57d-pflwg 63bd1b5d 0 00005421e8b30d05 fa336b14 created 6 days ago
ubuntu1804-k8s2 default helloworld-golang cdb85f1e 0 0000eeb0142b9961 b07b2652 created 4 days ago

traceloop 相關 commands
kubectl gadget traceloop pod default helloworld-golang 1
kubectl gadget traceloop show 0004ac1f39174f40

kubectl gadget traceloop pod sock-shop front-end-85c54f4fb-wftx6 0

# 須進入到gadget pod內才能使用
curl --silent --unix-socket /run/traceloop.socket http://localhost/dump-by-traceid?traceid=0000b7ec718351b6


Network Policy
Blog 介紹Network Policy Advisor

執行 network-policy monitor command
kubectl gadget network-policy monitor --namespaces default --output ./networktrace.log

製造traffic by curl
#For instance:
kubectl exec helloworld-python -- curl -s -XGET http://helloworld-golang.default.svc.cluster.local:8001/
kubectl exec xwing -- curl -s -XGET http://helloworld-python.default.svc.cluster.local:8080/

產生network-policy
kubectl gadget network-policy report --input ./networktrace.log > network-policy-default.yaml

Prometheus & Grafana & eBPF exporter 設定

 Prometheus


編輯 prometheus.yml
- job_name: "eBPF"
static_configs:
- targets: ['localhost:9435']

執行Prometheus, Grafana and ebpf_exporter
~/git/ebpf_exporter$ sudo ~/SourceCode/golang/bin/ebpf_exporter --config.file=examples/bio.yaml

cd ~/Prometheus/prometheus-2.16.0.linux-amd64
./prometheus --config.file=prometheus.yml

cd ~/Downloads/grafana-6.0.0
./bin/grafana-server web

# admin/bbyanlfe
#http://140.96.27.24:3000/dashboard/

eBPF exporter 使用說明
build eBPF exporter and run it
cd ~/SourceCode/go_SourceCode/ebpf_exporter
go build cmd/ebpf_exporter/main.go
sudo ./main --config.file=examples/timers.yaml --debbug

#http://140.96.27.24:9435/metrics

visual studio code to run ebpf_exporter with sudo
sudo code --user-data-dir=/home/liudanny/
### launch.json ###
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"name": "Launch",
"type": "go",
"request": "launch",
"mode": "auto",
"program": "${workspaceFolder}/cmd/ebpf_exporter/main.go",
"env": {},
"args": ["--config.file=${workspaceFolder}/examples/timers.yaml"]
}
]
}