Prometheus & Grafana & eBPF exporter 設定

 Prometheus

https://k2r2bai.com/2018/06/10/cncf/prometheus/

編輯 prometheus.yml

- job_name: "eBPF"

static_configs:

- targets: ['localhost:9435']

執行Prometheus, Grafana and ebpf_exporter

~/git/ebpf_exporter$ sudo ~/SourceCode/golang/bin/ebpf_exporter --config.file=examples/bio.yaml

cd ~/Prometheus/prometheus-2.16.0.linux-amd64

./prometheus --config.file=prometheus.yml

cd ~/Downloads/grafana-6.0.0

./bin/grafana-server web

# admin/bbyanlfe

#http://140.96.27.24:3000/dashboard/

eBPF exporter 使用說明

build eBPF exporter and run it

cd ~/SourceCode/go_SourceCode/ebpf_exporter

go build cmd/ebpf_exporter/main.go

sudo ./main --config.file=examples/timers.yaml --debbug

#http://140.96.27.24:9435/metrics

visual studio code to run ebpf_exporter with sudo

sudo code --user-data-dir=/home/liudanny/

### launch.json ###

{

// Use IntelliSense to learn about possible attributes.

// Hover to view descriptions of existing attributes.

// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387

"version": "0.2.0",

"configurations": [

{

"name": "Launch",

"type": "go",

"request": "launch",

"mode": "auto",

"program": "${workspaceFolder}/cmd/ebpf_exporter/main.go",

"env": {},

"args": ["--config.file=${workspaceFolder}/examples/timers.yaml"]

}

]

}

[Cilium] Hubble Source Code Study

 

Source Code Study:

https://github.com/cilium/cilium/blob/master/bpf/bpf_lxc.c#L1223

int handle_policy(struct __ctx_buff *ctx)

| 

-> tail_ipv4_policy()

    |

    -> ipv4_policy()

        |

        V

        https://github.com/cilium/cilium/blob/master/bpf/lib/policy_log.h#L46

        send_policy_verdict_notify()

在此function放入 struct policy_verdict_notify 格式化的資料

        |

    -> ctx_event_output()

   (這function是被remap來自skb_event_output()

   會將event送到perf's ring buffer in Kernel)

  

#Cilium 有 perf reader 可以從 Kernel 讀 perf's ring buffer 然後解開msg

https://github.com/cilium/ebpf/blob/master/perf/reader.go

#另外一個會送send_trace_notify()的來源 (有很多bpf的點都會送)

https://github.com/cilium/cilium/blob/master/bpf/bpf_network.c

int from_network(struct __ctx_buff *ctx)

   |

   V

   https://github.com/cilium/cilium/blob/master/bpf/lib/trace.h

   send_trace_notify()

   這個function會發送packet tracing events是很重要的flow來源

#上面兩個例子的notify type 是 CILIUM_NOTIFY_POLICY_VERDICT & CILIUM_NOTIFY_TRACE

#到了Hubble就會是Flow (Layer3/4)主要來源

case monitorAPI.MessageTypeDrop,

monitorAPI.MessageTypeTrace,

monitorAPI.MessageTypePolicyVerdict:

return p.l34.Decode(payload, decoded)

case monitorAPI.MessageTypeAccessLog:

return p.l7.Decode(payload, decoded)

 

BPF datapath events 定義在這

https://github.com/cilium/cilium/blob/master/pkg/monitor/api/types.go

#跟container有關的send_trace_notify()的來源

https://github.com/cilium/cilium/blob/master/bpf/bpf_lxc.c

tail_handle_ipv4()

| 

-> handle_ipv4_from_lxc()

   |

   V

   https://github.com/cilium/cilium/blob/master/bpf/lib/trace.h

   send_trace_notify()

測試:

kubectl get pods -n kube-system

kubectl exec -n kube-system ds/cilium -- cilium status

hubble observe --verdict DROPPED --pod XXX

hubble observe --since 3m --protocol http --output=json | \

jq -sr 'map(select(.l7.type=="RESPONSE" and .l7.http.method=="GET")) |

group_by(.l7.http.url) |

map({url: .[0].l7.http.url, max_latency_ms: ((map(.l7.latency_ns | tonumber) | max) / 1000000 }) |

sort_by(.max_latency_ms)'

hubble observe --since 3m --protocol tcp --output=json | \

jq -sr 'map(select(.l4.TCP.destination_port==8080)) |

group_by(.l4.TCP.flags)'

hubble observe --since=10m -t l7 -j \

| jq 'select(.l7.dns.rcode==3) | .destination.namespace + "/" + .destination.pod_name' \

| sort | uniq -c | sort -r

hubble-relay serve --peer-service=unix:///var/run/cilium/hubble.sock --listen-address=:4245

curl \

--silent \

--unix-socket /var/run/cilium/hubble.sock \

--request GET \

http://localhost/ \

| jq .

hubble serve --listen-client-urls=0.0.0.0:50051 --listen-client-urls=unix:///var/run/hubble.sock --metrics-server :6943 --metric=dns:query --metric=drop --metric=tcp --metric=flow --

[ODL] Import ODL Controller using Maven into Eclipse on Windows 10

 

Prerequisites

Java

目前使用的是OpenJDK, 需在Windows的系統環境變數做"JAVA_HOME"設定, 變數值為安裝的JDK路徑

Eclipse

需先在Windows 10上安裝Eclipse IDE, 目前使用的版本是 2019-09 R (4.13)

Maven

相關安裝Maven可參考: https://howtodoinjava.com/maven/how-to-install-maven-on-windows/

Mave下載位置: https://maven.apache.org/download.cgi

下載並解壓縮完成後, 請在系統環境變數做"MAVEN_HOME" and "M2_HOME"設定如下:

Import ODL Controller using Maven into Eclipse

Eclipse 環境設定 for Maven路徑

Install m2e plugin

Help --> Install New Software

在 "Work with" 選擇 m2e release repository 或是 自行增加 ( 輸入 URL 後按下 "Add" )

勾選Maven Integration for Eclipse, 然後按下 "Finish"

把Maven's "Plugin execution not covered by lifecycle configuration" 改選為 "Ignore"

Import ODL Controller

ODL Controller GitHub: https://github.com/opendaylight/controller

切換到 tag: v3.0.2

$git checkout tags/v3.0.2

Project --> Import

以上述方式import project, 只剩下這種的Maven Error:

"Execution generate-depends-file of goal org.apache.servicemix.tooling:depends-maven-plugin:1.4.0:generate-depends-file failed"

[eBPF] The example of using BCC's trace.py script

If you want to use a container to run BCC's script, you can follow the instructions to build a Docker image and run it as a container.

Dockerfile

FROM ubuntu:18.04

RUN apt update && apt install -y lsb-core vim curl cscope cmake ctags file git locales bison flex iperf netperf android-tools-adb build-essential libedit-dev zlib1g-dev libelf-dev tree wget openjdk-8-jdk libgtk-3-dev iputils-ping
RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 4052245BD4284CDD
RUN echo "deb https://repo.iovisor.org/apt/$(lsb_release -cs) $(lsb_release -cs) main" > /etc/apt/sources.list.d/iovisor.list
RUN apt update && apt-get install -y bcc-tools libbcc-examples

Build a Docker image

sudo docker build -t bcc-ebpf .

Run the Docker image as a container 

sudo docker run -d --name bcc \
    --privileged \
    -v $(pwd):/bcc \
    -v /lib/modules:/lib/modules:ro \
    -v /usr/src:/usr/src:ro \
    -v /boot:/boot:ro \
    -v /sys/kernel/debug:/sys/kernel/debug \
    bcc-ebpf sleep 3600d

The examples of using BCC's trace.py
python trace.py 'r:bash:readline "%s", retval'
python trace.py 'u:/lib/x86_64-linux-gnu/libc-2.27.so:memory_sbrk_more "%u", arg1' -T

./trace.py 'python:_PyImport_LoadDynamicModule "name: %s path: %s" arg1, arg2' \
        'r:python:_PyImport_LoadDynamicModule "at 0x%x" retval'

./trace.py 'r:bash:readline "%s" retval'

./trace.py '/home/bgregg/functions:main.add "%d %d" arg1, arg2'

# List functions in the executable file
readelf -s test
objdump -t test
objdump -D test


Reference:
Linux eBPF/bcc uprobes
http://www.brendangregg.com/blog/2016-02-08/linux-ebpf-bcc-uprobes.html
Using user-space tracepoints with BPF

https://github.com/zoidbergwill/awesome-ebpf
https://lwn.net/Articles/753601/

[eBPF] An example of Userspace Tracing

#!/usr/bin/python

from __future__ import print_function
from bcc import BPF
from time import sleep

# load BPF program

b = BPF(text="""
#include <uapi/linux/ptrace.h>
struct key_t {
    char c[80];
};
BPF_HASH(counts, struct key_t);

int count(struct pt_regs *ctx) {
    if (!PT_REGS_PARM1(ctx))
        return 0;

    struct key_t key = {};
    u64 zero = 0, *val;
    bpf_probe_read(&key.c, sizeof(key.c), (void *)PT_REGS_PARM1(ctx));
    val = counts.lookup_or_init(&key, &zero);
    (*val)++;
    return 0;
};
""")

b.attach_uprobe(name="c", sym="strlen", fn_name="count")

# header
print("Tracing strlen()... Hit Ctrl-C to end.")

# sleep until Ctrl-C
try:
    sleep(99999999)
except KeyboardInterrupt:
    pass

# print output
print("%10s %s" % ("COUNT", "STRING"))
counts = b.get_table("counts")
for k, v in sorted(counts.items(), key=lambda counts: counts[1].value):
    print("%10d \"%s\"" % (v.value, k.c.encode('string-escape')))


Reference:
https://github.com/iovisor/bcc/blob/master/tools/bashreadline.py
https://github.com/iovisor/bcc/blob/master/tools/gethostlatency.py
https://github.com/iovisor/bcc/blob/master/tools/funccount.py
https://github.com/iovisor/bcc/blob/master/tools/memleak.py
https://github.com/iovisor/bcc/blob/master/tools/dbslower.py
https://github.com/iovisor/bcc/blob/master/tools/trace.py

Golang bcc/BPF Function Tracing
http://www.brendangregg.com/blog/2017-01-31/golang-bcc-bpf-function-tracing.html


Linux bcc/BPF Node.js USDT Tracing

//Search "attach_uprobe path:tools language:python" in github project:  iovisor/bcc 

https://github.com/iovisor/bcc/search?q=attach_uprobe+path%3Atools+language%3Apython&unscoped_q=attach_uprobe+path%3Atools+language%3Apython&type=Code

eBPF監控工具bcc系列七開發腳本
https://blog.csdn.net/weixin_33824363/article/details/89728890

[Tracing] Use BCC tools to do dynamic tracing in Linux user space

Here is a very simple example of using BCC to trace a program and get the data from the function's argument.

A Simple Test Code in C: "test.c"

#include <stdio.h>

// student structure
struct student {
  char id[15];
  char firstname[64];
  char lastname[64];
  float points;
};
// function declaration
void getDetail(struct student *);
void displayDetail(struct student *);

int main(void) {
  // student structure variable
  struct student std[1];
  // get student detail
  getDetail(std);
  // display student detail
  displayDetail(std);
  return 0;
}

// function definition
void getDetail(struct student *ptr) {
  int i;
  for (i = 0; i < 1; i++) {
    printf("Enter detail of student #%d\n", (i + 1));
    printf("Enter ID: ");
    scanf("%s", ptr->id);
    printf("Enter first name: ");
    scanf("%s", ptr->firstname);
    printf("Enter last name: ");
    scanf("%s", ptr->lastname);
    printf("Enter Points: ");
    scanf("%f", &ptr->points);
    // update pointer to point at next element
    // of the array std
    ptr++;
  }
}

void displayDetail(struct student *ptr) {
  int i;
  for (i = 0; i < 1; i++) {
    printf("\nDetail of student #%d\n", (i + 1));
    // display result via ptr variable
    printf("\nResult via ptr\n");
    printf("ID: %s\n", ptr->id);
    printf("First Name: %s\n", ptr->firstname);
    printf("Last Name: %s\n", ptr->lastname);
    printf("Points: %f\n", ptr->points);
    // update pointer to point at next element
    // of the array std
    ptr++;
  }
}

The simple BCC example code: "mytrace.py"
It will get and print out the string of firstname when it runs

from __future__ import print_function
from bcc import BPF
from time import strftime
import argparse


# load BPF program
bpf_text = """
#include <uapi/linux/ptrace.h>
#include <linux/sched.h>
struct str_t {
    u64 pid;
    char str[80];
};

// student structure
struct student {
  char id[15];
  char firstname[64];
  char lastname[64];
  float points;
};

BPF_PERF_OUTPUT(events);
int printret(struct pt_regs *ctx) {
    struct str_t data  = {};
    char comm[TASK_COMM_LEN] = {};
    u32 pid;
    if (!PT_REGS_RC(ctx))
        return 0;
    pid = bpf_get_current_pid_tgid();
    data.pid = pid;
    bpf_probe_read(&data.str, sizeof(data.str),
        ((struct student *)PT_REGS_RC(ctx))->firstname);
    bpf_get_current_comm(&comm, sizeof(comm));
    events.perf_submit(ctx,&data,sizeof(data));
    return 0;
};
"""

b = BPF(text=bpf_text)
b.attach_uprobe(name="<...your test binary file location...>/test",
        sym="displayDetail", fn_name="printret")

# header
print("%-9s %-6s %s" % ("TIME", "PID", "ARGUMENT"))

def print_event(cpu, data, size):
    event = b["events"].event(data)
    print("%-9s %-6d %s" % (strftime("%H:%M:%S"), event.pid,
                            event.str.decode('utf-8', 'replace')))

b["events"].open_perf_buffer(print_event)
while 1:
    try:
        b.perf_buffer_poll()
    except KeyboardInterrupt:
        exit()

Here is the steps:

$ sudo python mytrace.py

# open another terminal
$ gcc -o test test.c
$ ./test
Enter detail of student #1
Enter ID: 1234
Enter first name: Danny
Enter last name: Liu
Enter Points: 100

Detail of student #1

Result via ptr
ID: 1234
First Name: Danny
Last Name: Liu
Points: 100.000000

Then you will see the tracing message in your previous terminal as follows:
(print out the first name from function's argument)

$ sudo python mytrace.py
TIME      PID    ARGUMENT
17:01:32  7265   Danny

[Tracing] Dynamic tracing in Linux using uprobe and kprobe

This post is based on the following articles to trace again by myself. Through this process, I can get a deeper impression of the user-space tracing using uprobe.
https://opensource.com/article/17/7/dynamic-tracing-linux-user-and-kernel-space
http://www.brendangregg.com/perf.html

[Darknet][Python] Support using Numpy array as image input instead of image path in Darknet Python API

When I dealt with Darknet's detect function in Python, I found that there is only one way for accepting an image file path as the argument, which is convenient for inferencing images from files.  But, it is not very well for images from video files or cameras.
Due to this reason, I dig out some solutions already on the Internet, and modify as follows: