Golang 設定與常用指令說明

 Golang設定

加入下列於 ~/.profile

#Golang path

export PATH=$PATH:/usr/local/go/bin

export GOPATH=$HOME/go

export PATH=$PATH:$GOPATH/bin

---

常用的指令與使用說明

初始化 Module

這個會在資料夾底下產生一個初始的 go.mod 檔案

go mod init <module>

• 在 build go 專案的時候，會根據 module name 的最後一路路徑 myproject 當作輸出檔名，會產生 myproject.exe 的檔案
• 在 import 自己專案 package 的路徑的時候，github.com/mileslin/myproject 是根目錄，例如我新增一個 pg1 的 package ，則我在 import 的時候要輸入 import "github.com/mileslin/myproject/pg1"。

取得套件

go get -u github.com/gorilla/mux (-u 是說明，取得套件的時候不要從 cache 取得

#go get 的時候指定版本

go get github.com/gorilla/mux@v1.6.1

除了會新增一個 go.sum 檔案之外呢， go.mod 也會記錄你下載的套件，如下範例

module github.com/mileslin/myproject

go 1.13

require github.com/gorilla/mux v1.7.3 // indirect

// indirect 的意思是說，此套件還沒被用到。 所以只要在程式碼中使用此到件，並且 build 專案後，// indirect 就會消失

最後提一下，下載的套件會被放在 GOPATH/pkg/mod 底下。 (可以使用 go env 指令查看 GOPATH 路徑)

列出套件與套件版本

go list 取得可以使用的套件列表

$ go list all 會列出所有套件，包含 GOROOT 套件

$ go list -m all 只列出 module 列出的套件(包含自己的 package)

$ go list -m -versions github.com/gorilla/mux 列出此套件的所有版本

檢查套件使用情形

列出使套件在哪裡有被用到

$ go mod why github.com/gorilla/mux

Module download

這個會將 go.mod 定義的 library 都下載下來，但基本上 go build, go test 也會自動下載 library 並新增到 go.mod 上，

並會產出 go.sum 來確保 library 是否有更新。可以說能達到一樣的效果

go mod download

如果你忘了 import ，則會出現以下的錯誤

// github.com/streadway/amqp

go build -o ./bin/rabbitmq ./cmd

# rabbitmq/cmd

cmd/main.go:16:15: undefined: amqp

cmd/main.go:41:4: undefined: amqp

Module tidy

移除在 go.mod 不需要的 library

go mod tidy

---

go test

go test -timeout 30s ./probe/endpoint/... -run "TestSpyNoProcesses|TestSpyWithProcesses"

go test -timeout 30s ./probe/endpoint/...

go test -v -run=TestNewWithAssert ./example18-write-testing-and-doc/...

inspektor gadget 常用commands

 編譯與安裝kubectl-gadget

git clone https://github.com/kinvolk/inspektor-gadget.git

cd inspektor-gadget

make kubectl-gadget-linux-amd64

sudo cp kubectl-gadget-linux-amd64 /usr/local/bin/kubectl-gadget

佈署Gadgets on K8S 使用kubectl gadget

kubectl gadget deploy | kubectl apply -f -

or

#指定image

kubectl gadget deploy --image=kinvolk/gadget:fix63 | kubectl apply -f -

# make build & get image kinvolk/gadget:alban-fix63

kubectl gadget deploy --image=kinvolk/gadget:alban-fix63 | kubectl apply -f -

查詢在每個Node上的 gadget's pod name

kubectl get pod -n kube-system -l k8s-app=gadget -o=jsonpath='{.items[*].metadata.name}'

kubectl exec -it <gadget-xxxx> -n kube-system -- bash

Run BCC commands

$ kubectl gadget tcpconnect -A

$ kubectl gadget tcptracer -A

---

Traceloop

查看traceloop 清單

kubectl gadget traceloop list

NODE NAMESPACE PODNAME PODUID INDEX TRACEID CONTAINERID STATUS

ubuntu1804-k8s1 kube-system hubble-bdqn9 851a4185 0 0000b7ec718351b6 b2d36529 created 6 days ago

ubuntu1804-k8s1 default deathstar-657477f57d-crf8w bac896ec 0 0000b7f1f2b1d065 941b6b07 created 6 days ago

ubuntu1804-k8s2 kube-system coredns-6955765f44-nqr56 024f3577 0 0000541b48333690 62d1fade created 6 days ago

ubuntu1804-k8s2 kube-system coredns-6955765f44-pbwt5 00be4172 0 0000541b483869c5 59c000af created 6 days ago

ubuntu1804-k8s2 kube-system hubble-ui-7b767c4654-j7fs7 50ab917c 0 0000541b77264f34 dc9d5823 created 6 days ago

ubuntu1804-k8s2 default helloworld-golang cdb85f1e 1 0000eea8600e2b9a 82dfe69f created 4 days ago

ubuntu1804-k8s2 default helloworld-python 45139ac8 0 0000eea7d9821155 9147da4f created 4 days ago

ubuntu1804-k8s2 default deathstar-657477f57d-pflwg 63bd1b5d 0 00005421e8b30d05 fa336b14 created 6 days ago

ubuntu1804-k8s2 default helloworld-golang cdb85f1e 0 0000eeb0142b9961 b07b2652 created 4 days ago

traceloop 相關 commands

kubectl gadget traceloop pod default helloworld-golang 1

kubectl gadget traceloop show 0004ac1f39174f40

kubectl gadget traceloop pod sock-shop front-end-85c54f4fb-wftx6 0

# 須進入到gadget pod內才能使用

curl --silent --unix-socket /run/traceloop.socket http://localhost/dump-by-traceid?traceid=0000b7ec718351b6

---

Network Policy

Blog 介紹Network Policy Advisor

https://kinvolk.io/blog/2020/03/writing-kubernetes-network-policies-with-inspektor-gadgets-network-policy-advisor/

執行 network-policy monitor command

kubectl gadget network-policy monitor --namespaces default --output ./networktrace.log

製造traffic by curl

#For instance:

kubectl exec helloworld-python -- curl -s -XGET http://helloworld-golang.default.svc.cluster.local:8001/

kubectl exec xwing -- curl -s -XGET http://helloworld-python.default.svc.cluster.local:8080/

產生network-policy

kubectl gadget network-policy report --input ./networktrace.log > network-policy-default.yaml

Prometheus & Grafana & eBPF exporter 設定

 Prometheus

https://k2r2bai.com/2018/06/10/cncf/prometheus/

編輯 prometheus.yml

- job_name: "eBPF"

static_configs:

- targets: ['localhost:9435']

執行Prometheus, Grafana and ebpf_exporter

~/git/ebpf_exporter$ sudo ~/SourceCode/golang/bin/ebpf_exporter --config.file=examples/bio.yaml

cd ~/Prometheus/prometheus-2.16.0.linux-amd64

./prometheus --config.file=prometheus.yml

cd ~/Downloads/grafana-6.0.0

./bin/grafana-server web

# admin/bbyanlfe

#http://140.96.27.24:3000/dashboard/

eBPF exporter 使用說明

build eBPF exporter and run it

cd ~/SourceCode/go_SourceCode/ebpf_exporter

go build cmd/ebpf_exporter/main.go

sudo ./main --config.file=examples/timers.yaml --debbug

#http://140.96.27.24:9435/metrics

visual studio code to run ebpf_exporter with sudo

sudo code --user-data-dir=/home/liudanny/

### launch.json ###

{

// Use IntelliSense to learn about possible attributes.

// Hover to view descriptions of existing attributes.

// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387

"version": "0.2.0",

"configurations": [

{

"name": "Launch",

"type": "go",

"request": "launch",

"mode": "auto",

"program": "${workspaceFolder}/cmd/ebpf_exporter/main.go",

"env": {},

"args": ["--config.file=${workspaceFolder}/examples/timers.yaml"]

}

]

}

[Cilium] Hubble Source Code Study

 

Source Code Study:

https://github.com/cilium/cilium/blob/master/bpf/bpf_lxc.c#L1223

int handle_policy(struct __ctx_buff *ctx)

| 

-> tail_ipv4_policy()

    |

    -> ipv4_policy()

        |

        V

        https://github.com/cilium/cilium/blob/master/bpf/lib/policy_log.h#L46

        send_policy_verdict_notify()

在此function放入 struct policy_verdict_notify 格式化的資料

        |

    -> ctx_event_output()

   (這function是被remap來自skb_event_output()

   會將event送到perf's ring buffer in Kernel)

  

#Cilium 有 perf reader 可以從 Kernel 讀 perf's ring buffer 然後解開msg

https://github.com/cilium/ebpf/blob/master/perf/reader.go

#另外一個會送send_trace_notify()的來源 (有很多bpf的點都會送)

https://github.com/cilium/cilium/blob/master/bpf/bpf_network.c

int from_network(struct __ctx_buff *ctx)

   |

   V

   https://github.com/cilium/cilium/blob/master/bpf/lib/trace.h

   send_trace_notify()

   這個function會發送packet tracing events是很重要的flow來源

#上面兩個例子的notify type 是 CILIUM_NOTIFY_POLICY_VERDICT & CILIUM_NOTIFY_TRACE

#到了Hubble就會是Flow (Layer3/4)主要來源

case monitorAPI.MessageTypeDrop,

monitorAPI.MessageTypeTrace,

monitorAPI.MessageTypePolicyVerdict:

return p.l34.Decode(payload, decoded)

case monitorAPI.MessageTypeAccessLog:

return p.l7.Decode(payload, decoded)

 

BPF datapath events 定義在這

https://github.com/cilium/cilium/blob/master/pkg/monitor/api/types.go

#跟container有關的send_trace_notify()的來源

https://github.com/cilium/cilium/blob/master/bpf/bpf_lxc.c

tail_handle_ipv4()

| 

-> handle_ipv4_from_lxc()

   |

   V

   https://github.com/cilium/cilium/blob/master/bpf/lib/trace.h

   send_trace_notify()

測試:

kubectl get pods -n kube-system

kubectl exec -n kube-system ds/cilium -- cilium status

hubble observe --verdict DROPPED --pod XXX

hubble observe --since 3m --protocol http --output=json | \

jq -sr 'map(select(.l7.type=="RESPONSE" and .l7.http.method=="GET")) |

group_by(.l7.http.url) |

map({url: .[0].l7.http.url, max_latency_ms: ((map(.l7.latency_ns | tonumber) | max) / 1000000 }) |

sort_by(.max_latency_ms)'

hubble observe --since 3m --protocol tcp --output=json | \

jq -sr 'map(select(.l4.TCP.destination_port==8080)) |

group_by(.l4.TCP.flags)'

hubble observe --since=10m -t l7 -j \

| jq 'select(.l7.dns.rcode==3) | .destination.namespace + "/" + .destination.pod_name' \

| sort | uniq -c | sort -r

hubble-relay serve --peer-service=unix:///var/run/cilium/hubble.sock --listen-address=:4245

curl \

--silent \

--unix-socket /var/run/cilium/hubble.sock \

--request GET \

http://localhost/ \

| jq .

hubble serve --listen-client-urls=0.0.0.0:50051 --listen-client-urls=unix:///var/run/hubble.sock --metrics-server :6943 --metric=dns:query --metric=drop --metric=tcp --metric=flow --

[ODL] Import ODL Controller using Maven into Eclipse on Windows 10

 

Prerequisites

Java

目前使用的是OpenJDK, 需在Windows的系統環境變數做"JAVA_HOME"設定, 變數值為安裝的JDK路徑

Eclipse

需先在Windows 10上安裝Eclipse IDE, 目前使用的版本是 2019-09 R (4.13)

Maven

相關安裝Maven可參考: https://howtodoinjava.com/maven/how-to-install-maven-on-windows/

Mave下載位置: https://maven.apache.org/download.cgi

下載並解壓縮完成後, 請在系統環境變數做"MAVEN_HOME" and "M2_HOME"設定如下:

Import ODL Controller using Maven into Eclipse

Eclipse 環境設定 for Maven路徑

Install m2e plugin

Help --> Install New Software

在 "Work with" 選擇 m2e release repository 或是 自行增加 ( 輸入 URL 後按下 "Add" )

勾選Maven Integration for Eclipse, 然後按下 "Finish"

把Maven's "Plugin execution not covered by lifecycle configuration" 改選為 "Ignore"

Import ODL Controller

ODL Controller GitHub: https://github.com/opendaylight/controller

切換到 tag: v3.0.2

$git checkout tags/v3.0.2

Project --> Import

以上述方式import project, 只剩下這種的Maven Error:

"Execution generate-depends-file of goal org.apache.servicemix.tooling:depends-maven-plugin:1.4.0:generate-depends-file failed"

[eBPF] The example of using BCC's trace.py script

If you want to use a container to run BCC's script, you can follow the instructions to build a Docker image and run it as a container.

Dockerfile

FROM ubuntu:18.04

RUN apt update && apt install -y lsb-core vim curl cscope cmake ctags file git locales bison flex iperf netperf android-tools-adb build-essential libedit-dev zlib1g-dev libelf-dev tree wget openjdk-8-jdk libgtk-3-dev iputils-ping
RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 4052245BD4284CDD
RUN echo "deb https://repo.iovisor.org/apt/$(lsb_release -cs) $(lsb_release -cs) main" > /etc/apt/sources.list.d/iovisor.list
RUN apt update && apt-get install -y bcc-tools libbcc-examples

Build a Docker image

sudo docker build -t bcc-ebpf .

Run the Docker image as a container 

sudo docker run -d --name bcc \
    --privileged \
    -v $(pwd):/bcc \
    -v /lib/modules:/lib/modules:ro \
    -v /usr/src:/usr/src:ro \
    -v /boot:/boot:ro \
    -v /sys/kernel/debug:/sys/kernel/debug \
    bcc-ebpf sleep 3600d

The examples of using BCC's trace.py
python trace.py 'r:bash:readline "%s", retval'
python trace.py 'u:/lib/x86_64-linux-gnu/libc-2.27.so:memory_sbrk_more "%u", arg1' -T

./trace.py 'python:_PyImport_LoadDynamicModule "name: %s path: %s" arg1, arg2' \
        'r:python:_PyImport_LoadDynamicModule "at 0x%x" retval'

./trace.py 'r:bash:readline "%s" retval'

./trace.py '/home/bgregg/functions:main.add "%d %d" arg1, arg2'

# List functions in the executable file
readelf -s test
objdump -t test
objdump -D test


Reference:
Linux eBPF/bcc uprobes
http://www.brendangregg.com/blog/2016-02-08/linux-ebpf-bcc-uprobes.html
Using user-space tracepoints with BPF

https://github.com/zoidbergwill/awesome-ebpf
https://lwn.net/Articles/753601/

[eBPF] An example of Userspace Tracing

#!/usr/bin/python

from __future__ import print_function
from bcc import BPF
from time import sleep

# load BPF program

b = BPF(text="""
#include <uapi/linux/ptrace.h>
struct key_t {
    char c[80];
};
BPF_HASH(counts, struct key_t);

int count(struct pt_regs *ctx) {
    if (!PT_REGS_PARM1(ctx))
        return 0;

    struct key_t key = {};
    u64 zero = 0, *val;
    bpf_probe_read(&key.c, sizeof(key.c), (void *)PT_REGS_PARM1(ctx));
    val = counts.lookup_or_init(&key, &zero);
    (*val)++;
    return 0;
};
""")

b.attach_uprobe(name="c", sym="strlen", fn_name="count")

# header
print("Tracing strlen()... Hit Ctrl-C to end.")

# sleep until Ctrl-C
try:
    sleep(99999999)
except KeyboardInterrupt:
    pass

# print output
print("%10s %s" % ("COUNT", "STRING"))
counts = b.get_table("counts")
for k, v in sorted(counts.items(), key=lambda counts: counts[1].value):
    print("%10d \"%s\"" % (v.value, k.c.encode('string-escape')))


Reference:
https://github.com/iovisor/bcc/blob/master/tools/bashreadline.py
https://github.com/iovisor/bcc/blob/master/tools/gethostlatency.py
https://github.com/iovisor/bcc/blob/master/tools/funccount.py
https://github.com/iovisor/bcc/blob/master/tools/memleak.py
https://github.com/iovisor/bcc/blob/master/tools/dbslower.py
https://github.com/iovisor/bcc/blob/master/tools/trace.py

Golang bcc/BPF Function Tracing
http://www.brendangregg.com/blog/2017-01-31/golang-bcc-bpf-function-tracing.html


Linux bcc/BPF Node.js USDT Tracing

//Search "attach_uprobe path:tools language:python" in github project:  iovisor/bcc 

https://github.com/iovisor/bcc/search?q=attach_uprobe+path%3Atools+language%3Apython&unscoped_q=attach_uprobe+path%3Atools+language%3Apython&type=Code

eBPF監控工具bcc系列七開發腳本
https://blog.csdn.net/weixin_33824363/article/details/89728890

[Tracing] Use BCC tools to do dynamic tracing in Linux user space

Here is a very simple example of using BCC to trace a program and get the data from the function's argument.

A Simple Test Code in C: "test.c"

#include <stdio.h>

// student structure
struct student {
  char id[15];
  char firstname[64];
  char lastname[64];
  float points;
};
// function declaration
void getDetail(struct student *);
void displayDetail(struct student *);

int main(void) {
  // student structure variable
  struct student std[1];
  // get student detail
  getDetail(std);
  // display student detail
  displayDetail(std);
  return 0;
}

// function definition
void getDetail(struct student *ptr) {
  int i;
  for (i = 0; i < 1; i++) {
    printf("Enter detail of student #%d\n", (i + 1));
    printf("Enter ID: ");
    scanf("%s", ptr->id);
    printf("Enter first name: ");
    scanf("%s", ptr->firstname);
    printf("Enter last name: ");
    scanf("%s", ptr->lastname);
    printf("Enter Points: ");
    scanf("%f", &ptr->points);
    // update pointer to point at next element
    // of the array std
    ptr++;
  }
}

void displayDetail(struct student *ptr) {
  int i;
  for (i = 0; i < 1; i++) {
    printf("\nDetail of student #%d\n", (i + 1));
    // display result via ptr variable
    printf("\nResult via ptr\n");
    printf("ID: %s\n", ptr->id);
    printf("First Name: %s\n", ptr->firstname);
    printf("Last Name: %s\n", ptr->lastname);
    printf("Points: %f\n", ptr->points);
    // update pointer to point at next element
    // of the array std
    ptr++;
  }
}

The simple BCC example code: "mytrace.py"
It will get and print out the string of firstname when it runs

from __future__ import print_function
from bcc import BPF
from time import strftime
import argparse


# load BPF program
bpf_text = """
#include <uapi/linux/ptrace.h>
#include <linux/sched.h>
struct str_t {
    u64 pid;
    char str[80];
};

// student structure
struct student {
  char id[15];
  char firstname[64];
  char lastname[64];
  float points;
};

BPF_PERF_OUTPUT(events);
int printret(struct pt_regs *ctx) {
    struct str_t data  = {};
    char comm[TASK_COMM_LEN] = {};
    u32 pid;
    if (!PT_REGS_RC(ctx))
        return 0;
    pid = bpf_get_current_pid_tgid();
    data.pid = pid;
    bpf_probe_read(&data.str, sizeof(data.str),
        ((struct student *)PT_REGS_RC(ctx))->firstname);
    bpf_get_current_comm(&comm, sizeof(comm));
    events.perf_submit(ctx,&data,sizeof(data));
    return 0;
};
"""

b = BPF(text=bpf_text)
b.attach_uprobe(name="<...your test binary file location...>/test",
        sym="displayDetail", fn_name="printret")

# header
print("%-9s %-6s %s" % ("TIME", "PID", "ARGUMENT"))

def print_event(cpu, data, size):
    event = b["events"].event(data)
    print("%-9s %-6d %s" % (strftime("%H:%M:%S"), event.pid,
                            event.str.decode('utf-8', 'replace')))

b["events"].open_perf_buffer(print_event)
while 1:
    try:
        b.perf_buffer_poll()
    except KeyboardInterrupt:
        exit()

Here is the steps:

$ sudo python mytrace.py

# open another terminal
$ gcc -o test test.c
$ ./test
Enter detail of student #1
Enter ID: 1234
Enter first name: Danny
Enter last name: Liu
Enter Points: 100

Detail of student #1

Result via ptr
ID: 1234
First Name: Danny
Last Name: Liu
Points: 100.000000

Then you will see the tracing message in your previous terminal as follows:
(print out the first name from function's argument)

$ sudo python mytrace.py
TIME      PID    ARGUMENT
17:01:32  7265   Danny