[xgo] The first phase of hacking xgo

 xgo is a great Go CGO cross compiler that can help users to build the application written in Golang to multi-platforms at the same time. 

For more information in details, please check out these URLs:

https://github.com/karalabe/xgo
https://www.jianshu.com/p/a6047d3f976e

But, without any further study, we probably cannot understand how it works well. So here I just give some information about it. I will use my Github repository as the example:

# Install xgo

$ go get github.com/karalabe/xgo

$ git clone https://github.com/teyenliu/win-shared-example

$ cd win-shared-example

# Use xgo to cross-compile to windows/amd64

$ xgo -targets windows/amd64 github.com/teyenliu/win-shared-example

Then, we finish the cross-compilation for my application
And actually, xgo, this tool, is based on its prebuilt docker image: "karalabe/xgo-latest" to complete the cross-compilation task. We can check the dockerhub for it as well: https://hub.docker.com/r/karalabe/xgo-latest/builds

So, we can directly run this Docker Image to cross-compile my application based on my case as follows:

$ docker run --rm -v /home/liudanny/git/xgo:/build \

-v /home/liudanny/.xgo-cache:/deps-cache:ro \

-e REPO_REMOTE= \

-e REPO_BRANCH= \

-e PACK= \

-e DEPS= \

-e ARGS= \

-e OUT= \

-e FLAG_V=false \

-e FLAG_X=false \

-e FLAG_RACE=false \

-e FLAG_TAGS= \

-e FLAG_LDFLAGS= \

-e FLAG_BUILDMODE=default \

-e TARGETS=windows/amd64 \

-e EXT_GOPATH= \

karalabe/xgo-latest \

github.com/teyenliu/win-shared-example

Furthermore, we can directly run xgo's build.sh to cross-compile my application in the host instead of in in the container:

$ sudo su

# set Golang path

export PATH=$PATH:/usr/local/go/bin

export GOPATH=$HOME/go

export PATH=$PATH:$GOPATH/bin

# rm /deps in advance

rm -rf /deps

# run build.sh

env GO111MODULE=off \

env REPO_REMOTE= \

env REPO_BRANCH= \

env PACK= \

env DEPS= \

env ARGS= \

env OUT= \

env FLAG_V=false \

env FLAG_X=false \

env FLAG_RACE=false \

env FLAG_TAGS= \

env FLAG_LDFLAGS= \

env FLAG_BUILDMODE=default \

env TARGETS=windows/amd64 \

env EXT_GOPATH= \

env BUILD_DEPS=/home/liudanny/git/xgo/docker/base/build_deps.sh \

env BOOTSTRAP_REPO=/home/liudanny/git/xgo/docker/base/bootstrap_repo.sh \

env BOOTSTRAP_PURE=/home/liudanny/git/xgo/docker/base/bootstrap_pure.sh \

env BOOTSTRAP=/home/liudanny/git/xgo/docker/base/bootstrap.sh \

env PATH=/usr/local/go/bin:$PATH \

./build.sh github.com/teyenliu/win-shared-example

Reference:
Golang 交叉编译
Golang交叉编译各个平台的二进制文件
Fyne Cross Compile

The following commands are not succesful to work for cross-compilation

$ CGO_ENABLED=1 CXX="x86_64-w64-mingw32-g++" CXX_FOR_TARGET="x86_64-w64-mingw32-g++" CC="x86_64-w64-mingw32-gcc" CC_FOR_TARGET="x86_64-w64-mingw32-gcc" GOOS=windows GOARCH=amd64 \

go build -ldflags -v -x -installsuffix cgo -o example.exe main.go

$ GOOS=windows GOARCH=amd64 CGO_ENABLED=1 CC=x86_64-w64-mingw32-gcc go build -ldflags -v -x -installsuffix cgo -o example.exe main.go

[Fun] Detecting the license plate and replacing with a given image

Propably two years ago, I studied a ECCV 2018 paper "License Plate Detection and Recognition in Unconstrained Scenarios", and  its GitHub (https://github.com/sergiomsilva/alpr-unconstrained) repository contains the author's implementation. During that time, I had an idea to leveraged its implementation for replacing the license plate with my image. After tryed a couple days, I successed and provided a Python script to detect and replace cars' license plate in the video with a specified image.

I recently commited my modification code and pushed it to my forked repository as follows:
https://github.com/teyenliu/alpr-unconstrained

In this repository, it contains 2 samples which are convenient for users to test.

The way to process a video which will replace any car license plate in the video with a given image content.

$ python license-plate-replacement.py --video cars_mountain_short.mov --replace src/hello.PNG --output output1.avi
$ python license-plate-replacement.py --video cars_mountain.mp4 --replace src/hello.PNG --output output2.avi

The result looks like this:

[Golang] Golang with Cgo 動態連結(dynamic linking) 與 靜態連結(static linking)函式庫範例與筆記 in Windows environment

This post is focusd on how to do dynamic/static linking using Golang with cgo on Windows environment. If you want to see it on Linux environment, please check out another post: 

[Golang] Golang with Cgo 動態連結(dynamic linking) 函式庫範例與筆記 in Linux environment

First of all, thanks to someone who had posted several useful articles describing how to do dynamic linking and static linking using Golang(Go) with Cgo in Windows environment as follows:

P.S: For using this example on Linux environment, you should deal with the build process for the static and dynamic libraries.

在 Windows 環境建置動態連結函式庫 (Dynamic-link library)，使用 MinGW gcc/g++ 以及 CodeBlock

Go with Cgo 靜態連結 (static linking) 函式庫建置範例與筆記
Go with Cgo 動態連結 (dynamic linking) 函式庫建置範例與筆記

[tmux] The most common used tmux commands

 This post can help you quickly learn how to use tmux in just a few minutes.

Get tmux session ready to start (new session)

# new session 

$ tmux new -s {session name}

# or new session without name

$ tmux

# list sessions
$ tmux ls

Split a tmux window into two panes

# <Ctrl+b> + %：split a tmux window into two panes vertically

# <Ctrl+b> + "：split a tmux window into two panes horizontally

# The way to switch to another pane

# <Ctrl + b> + <Arrow keys ← or → >

P.S: If you want to close one of the pane you created, 

you just switch on it and exit it by exit command.

For instance: split a tmux window into two panes vertically

Adjust a splitted pane size

You also are able to ajust  the splitted pane vertically or horizontally.

# The way to adjust a pane

# <Ctrl + b> + <Ctrl + Arrow keys ↔ or ↕ >

For instance: There are 5 panes in the window and I can adjust the currently focused pane.

Attach tmux session or detach it

# <Ctrl+b> + d：Put current session in the background running (detach)

# attach a background session

$ tmux attach-session -t {session name}

# Or

$ tmux a -t {session name}

Delete tmux session

$ tmux kill-session -t {session name}

$ tmux kill-session -t 0

[Golang] The example function of converting Struct or Map data to Map[string]interface{} Type

    I recently encountered a use case that I need to convert my struct or map value to the type of map[string]interface{}. After surveying and studying for a while, I figure out how to deal with this kind of task and the way to do it.

    The following function ConvertStructToMap()  is the example of converting Struct or Map data to Map[string]interface{} Type. It receives the argument as interface type and uses reflect function to check the receiver type is correct and what we want. But, it has a constrain, which if you give a map, and Map's key type must be string.

// ConvertStructToMap()

func ConvertStructToMap(message interface{}) (map[string]interface{}, error) {

msg := reflect.ValueOf(message)

msgtype := msg.Type()

// support Struct

if msgtype.Kind() == reflect.Struct {

// message should be tagged by "codec" or "msg"

kv := make(map[string]interface{})

fields := msgtype.NumField()

for i := 0; i < fields; i++ {

field := msgtype.Field(i)

name := field.Name

if n1 := field.Tag.Get("json"); n1 != "" {

name = n1

} else if n2 := field.Tag.Get("msg"); n2 != "" {

name = n2

}

kv[name] = msg.FieldByIndex(field.Index).Interface()

}

return kv, nil

}

// support map[string]interface{} or map[string]string

if msgtype.Kind() != reflect.Map {

return nil, errors.New("message must be a map")

} else if msgtype.Key().Kind() != reflect.String {

return nil, errors.New("map keys must be strings")

}

// Get the interface{}'s current value

kv := make(map[string]interface{})

for _, k := range msg.MapKeys() {

kv[k.String()] = msg.MapIndex(k).Interface()

}

return kv, nil

}

func main() {

fmt.Println("Hello, playground")

structData := struct { Name string `json:"JsonName"` 

                       Score int `msg:"MsgScore"`} { 

                       "john smith", 30, }

kv, err := ConvertStructToMap( structData )

if err != nil {

    fmt.Println(err)

}

fmt.Println(kv)

}

How to use it? Here we go! You can try it on the Go Playground:
https://play.golang.org/p/TLKBIbSxIQj

or just take a look at the sample as follows:

// The way to use ConvertStructToMap()

func main() {

structData := struct { Name string `json:"JsonName"` 

                       Score int `msg:"MsgScore"`} { 

                       "john smith", 30, }

kv, err := ConvertStructToMap( structData )

if err != nil {

    fmt.Println(err)

}

fmt.Println(kv)

        mapData := make(map[string]int)

        mapData["Score1"] = 100

        mapData["Score2"] = 60

        kv, err = ConvertStructToMap( mapData )

        if err != nil {

    fmt.Println(err)

}

fmt.Println(kv)

}

Run Result==>

map[JsonName:john smith MsgScore:30]
map[Score1:100 Score2:60]

Python matplotlib 中文亂碼解決 in Linux or Windows

 大家是否有遇過當使用matplotlib時，遇到顯示不出中文或是中文為亂碼的問題。無論是在Anaconda內使用matplotlib或是直接在Python安裝環境內使用matplotlib，基本解決方式是一樣的。本文將以簡單與條列方式描述在Linux 或是 Windows 解決方法。

1. 下載字型的來源

https://www.fontpalace.com/font-download/SimHei/

2. 字型安裝

執行python command, 並以下列方式找到matplotlib放置ttf字型檔位置與設定檔

>>> import matplotlib

>>> matplotlib.matplotlib_fname()

根據下列不同環境的執行結果內放置ttf字型檔

Windows: 

放到: C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\matplotlib\mpl-data\fonts\ttf 

修改設定檔  C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\matplotlib\mpl-data\matplotlibrc

Linux:

 

放到: /usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/ttf

修改設定檔  /usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/matplotlibrc

Anaconda:

放到: /opt/conda/lib/python3.7/site-packages/matplotlib/mpl-data/fonts/ttf

修改設定檔  /opt/conda/lib/python3.7/site-packages/matplotlib/mpl-data/matplotlibrc

3. 設定檔matplotlibrc修改方式

font.family  : sans-serif

font.serif      : SimHei   <== 增加此項目

範例如下:

4. 重新載入字體

>>> from matplotlib.font_manager import _rebuild

>>> _rebuild() 

個人常用的 Git Commands

 0. 相關初始設定

git init

git config --global user.name "teyenliu"

git config --global user.email "teyen.liu@gmail.com"

git config --list

#git config --global http.sslverify false.

#git config --global log.decorate short

#git config --global color.ui true

#git config --global core.editor "vim"

git remote -v

#首次建立repository

#git remote add origin https://github.com/teyenliu/XXXX.git

#git branch -M main  // 更換master branch name to main

#git push -u origin master

#CRLF

#Linux: git config --global core.autocrlf input

#Windows: git config --global core.autocrlf true

#Editor

#git config --global core.editor vim

#Setup commit message template

#git config --global commit.template ~/git-template

#如果是 fork a repository, 則需要設定upstream repo

git remote set-url origin <my fork's git repo>

git remote add upstream <upstream's git repo>

# 查看到我們名下的遠端項目

git remote -v

#獲取upstream的最新版本
git fetch upstream

#將upstream merge到我們當前分支
git merge upstream/master

1. check out 另一個branch

git checkout <to this branch>

git checkout -b <create_new_branch> <based on this branch>

2. 建議使用這個方式避免無謂的merges

git pull --rebase <remote name> <branch name>

3. 當要switch branch，但是有file未 commit，可以使用下列方式暫存起來

git stash save 'stash1'

#查看在 stash 中的缓存

git stash list

#恢复暂存

git stash pop

4.  git checkout 時發生 Please move or remove them before you can switch branches.

git clean -d -fx

5. 拉下來最新的檔案

git pull --rebase upstream master

git pull --rebase upstream ConfigBase

6. 推code 到 remote repo.

git push origin master

7. 補充發送

git commit --amend

git push -f origin ConfigBase

git push -f origin master

8. 建立/刪除分支

#建立

git checkout -b <branch> 從目前分支再去建立(會新增)本端分支

git push -u origin <branch> PUSH並建立遠端分支

#刪除

git branch -d <branch>

### Branch from a previous commit using Git ###

# Create the branch using a commit hash:

git branch branch_name <commit-hash>

#Or by using a symbolic reference:

git branch branch_name HEAD~3

#To checkout the branch while creating it, use:

git checkout -b branch_name <commit-hash or HEAD~3>

9. 移動某個commit點

#回到 上一次的 commit

git reset --hard HEAD

#回到 上一次的前一次 commit

git reset --hard HEAD^

#裡面最近做的所有 HEAD 的改動

git reflog

git reset --hard 904e1ba #最近的commit

10. 如果想要打tag( 以某個commit為基礎 )

# 產生一個新的branch以某個commit

git checkout -b <tagname> <commit id>

git push origin <tagname>

# 要先把 branch push上去的原因是, 之後建立的tag如果跟brnach name相同 (例如: v0.4 ), 則 branch會發生如下訊息:

error: src refspec v0.4 matches more than one.

error: failed to push some refs

如果不小心先push tags, 則必須把tag移除後, 才能push branch.

# 查看目前有的tag

git tag -l

# 打tag

git tag -a <tagname> -m "My App description"

# 刪除tag

git tag -d <tagname>

# push all tags 到 remote端

git push origin --tags

11. 如果想要檢查 submodule 是否有更新可使用下列指令：

git submodule foreach --recursive git pull origin master

12. 強制推送到remote

git push -f

13. 拉submodule的程式碼

git submodule update --recursive --remote

14. 查看完整的Commit 資料

git log --pretty=oneline

15. 退回到特定 Commit

# where [revision] is the commit hash (for example: 12345678901234567890123456789012345678ab)

git checkout [revision] .

#To rollback to a specific commit:

git reset --hard commit_sha

#To rollback 10 commits back:

git reset --hard HEAD~10

Trouble Shooting

如果遇到Tag Name與Branch Name相同時, 會有其中一種無法上傳至遠端的錯誤：

src refspec XXX 匹配多個, 例如:

error: src refspec v0.3 matches more than one

解決方式:

1. 先刪除分支然後上傳Tag：
git branch -D testtag

或是

2. 先刪除tag然後上傳分支：
git tag -d testtag

簡介 Fluentd 寫入 InfluxDB 透過 Fluent::Plugin::InfluxDB

 此文件用來說明 使用 fluent client-go API 並且 寫入資料到InfluxDB 透過 Fluentd 的Plugin Fluent::Plugin::InfluxDB (InfluxDB是用2.0)

fluent-logger-golang (fluent client-go API)

參考 URL: A structured event logger for Fluentd (Golang)

目前使用此fluent client-go API，將資料傳入Fluentd, 

會使用其Async模式來避免後續的latency event被block住, 當Fluentd server有問題時:

// Use "Async" to enable asynchronous I/O (connect and write)
// for sending events to Fluentd without blocking
setting.FluentdLogger.Logger, errF = fluent.New(
    fluent.Config{Async: true, FluentHost: fluenthost, FluentPort: intFluentPort})
if errF != nil {
		fmt.Println("[ERROR]:", errF)
		setting.FluentdLogger.Enabled = false
}

P.S: When Fluentd server has a problem, the events which are going to send will be buffered on the memory. The default size is 8192

BufferLimit  

Sets the number of events buffered on the memory.

The example of using fluent-logger-golang to send data:

tag := "apm.latency"
var data = map[string]string{
		"mytimestamp": strconv.FormatInt(time.Now().Unix(), 10),
		"mydata":      "hoge",
		"myjob":       "apm",
		"myvalue":     "55.55",
}
error := logger.Post(tag, data)Fluentd

Fluentd

Fluentd安裝，請參考官網
https://docs.fluentd.org/installation/install-by-deb

Fluentd操作

# 啟動/關閉/查看服務
sudo systemctl start td-agent.service
sudo systemctl stop td-agent.service
sudo systemctl status td-agent.service
sudo systemctl restart td-agent.service

#修改Fluentd設定
vi /etc/td-agent/td-agent.conf
#查看Fluentd logs
cat /var/log/td-agent/td-agent.log

InfluxDB 2.X

2.0 以上的InfluxDB 會需要token，Client端才有權限讀寫，其中一種方式查出Token是用InfluxDB自己的Web http://<your ip address>:8086，登入後點選"Data"

例如:

選擇GO

可以看到 token的值

// You can generate a Token from the "Tokens Tab" in the UI const token = "Iiq0TiIpL9lXn2GATwh3WeZBkLq-SEul6C0yrKLjq4T4WZ9b0BKVAsFeNs8q0Is93SMbhF0l63s4DwJja4MSbw=="

Fluent::Plugin::InfluxDB (This plugin is for using with InfluxDB 2.x 目前使用此Plugin)

This repository contains the reference Fluentd plugin for the InfluxDB 2.0.

Fluentd需要此Plugin來對應寫資料到InfluxDB2

我目前用的: The configuration of /etc/td-agent/td-agent.conf

#<match apm.**>
#  @type stdout
#</match>

<match apm.**>
    @type copy
    <store>
      @type influxdb2

      url             <https://localhost:8086>
      token           Iiq0TiIpL9lXn2GATwh3WeZBkLq-SEul6C0yrKLjq4TXXXX
      use_ssl         false
      bucket          apm
      org             com
      time_precision  s
      tag_keys          ["mytimestamp","mydata"]
      field_keys        ["myvalue"]
    </store>
    <store>
      @type stdout
    </store>
</match>

The result in InfluxDB

安裝此plugin:

sudo td-agent-gem install fluent-plugin-influxdb-v2
sudo td-agent-gem uninstall fluent-plugin-influxdb-v2

influxdb-plugin-fluent (只適用於InfluxDB 1.x instances)

fluent-plugin-influxdb is a buffered output plugin for fluentd and influxDB.

Configuration Example

<match apm.**>
  @type influxdb
  host  localhost
  port  8086
  dbname apm
  user  danny
  password  xxxxxxxxx
  use_ssl false
  time_precision s
  tag_keys ["timestamp", "data"]
  sequence_tag _seq
</match>

安裝 plugin:

sudo td-agent-gem install fluent-plugin-influxdb
sudo td-agent-gem uninstall fluent-plugin-influxdb

Reference

遇過的Error Message

[warn]: #0 failed to flush the buffer. retry_time=7 next_retry_seconds=2021-04-22 14:28:33 14429497340994114529/137438953472000000000 +0800 chunk="5c089be4bdfb9a28fb4b5ac58228a90 7" error_class=InfluxDB2::InfluxError error="failure writing points to database: partial write: points beyond retention policy dropped=1" 2021-04-22 14:27:22 +0800 [warn]: #0 suppressed same stacktrace

https://stackoverflow.com/questions/54359348/unable-to-insert-data-in-influxdb