Wednesday, December 1, 2021

[GCC] Using GCC to create static and shared library ( .so / .a )

[轉貼&修改] https://blog.xuite.net/csiewap/cc/23626229-Using+GCC+to+create+static+and+shared+library+.so 

Library可分成三種,static、shared與dynamically loaded。


1. Static libraries

Static 程式庫用於靜態連結,簡單講是把一堆object檔用ar(archiver)

包裝集合起來,檔名以 `.a' 結尾。優點是執行效能通常會比後兩者快

而且因為是靜態連結,所以不易發生執行時找不到library或版本錯置而

無法執行的問題。缺點則是檔案較大,維護度較低;例如library如果發

現bug需要更新,那麼就必須重新連結執行檔。

1.1 編譯

編譯方式很簡單,先例用 `-c' 編出 object 檔,再用 ar 包起來即可。

hello.c 

#include
void hello(){ printf("Hello "); }

world.c 

#include
void world(){ printf("world."); }

mylib.h 

void hello();
void world();

$ gcc -c hello.c world.c /* 編出 hello.o 與 world.o */

$ ar rcs libmylib.a hello.o world.o /* 包成 limylib.a */

這樣就可以建出一個檔名為 libmylib.a 的檔。輸出的檔名其實沒有硬性規定,

但如果想要配合 gcc 的 '-l' 參數來連結,一定要以 `lib' 開頭,中間是你要

的library名稱,然後緊接著 `.a' 結尾。

1.2 使用

 main.c 

#include "mylib.h"

int main() {
    hello();
    world();
}

使用上就像與一般的 object 檔連結沒有差別。

$ gcc main.c libmylib.a

也可以配合 gcc 的 `-l' 參數使用

$ gcc main.c -L. -lmylib

`-Ldir' 參數用來指定要搜尋程式庫的目錄,`.' 表示搜尋現在所在的目錄。

通常預設會搜 /usr/lib 或 /lib 等目錄。

`-llibrary' 參數用來指定要連結的程式庫 ,'mylib' 表示要與mylib進行連結

,他會搜尋library名稱前加`lib'後接`.a'的檔案來連結。

$ ./a.out
Hello world.

此時可以用 nm <exe filename> 來顯示symbols (functions, etc.)
(
To see which symbols come from static libraries requires running nm against those libraries to get a list of the symbols (functions, etc.) in them, then comparing them to what your list of symbols from nm <exefilename>.)


2. Shared libraries

Shared library 會在程式執行起始時才被自動載入。因為程式庫與執行檔

是分離的,所以維護彈性較好。有兩點要注意,shared library是在程式起始

時就要被載入,而不是執行中用到才載入,而且在連結階段需要有該程式庫

才能進行連結

首先有一些名詞要弄懂,soname、real name與linker name

soname 用來表示是一個特定 library 的名稱,像是 libmylib.so.1

前面以 `lib' 開頭,接著是該 library 的名稱,然後是 `.so' ,接著

是版號,用來表名他的介面;如果介面改變時,就會增加版號來維護相容度。

real name 是實際放有library程式的檔案名稱,後面會再加上 minor 版號與

release 版號,像是 libmylib.so.1.0.0

一般來說,版號的改變規則是(印象中在 APress-Difinitive Guide to GCC中有

提到,但目前手邊沒這本書),最尾碼的release版號用於程式內容的修正,

介面完全沒有改變。中間的minor用於有新增加介面,但相舊介面沒改變,所以

與舊版本相容。最前面的version版號用於原介面有移除或改變,與舊版不相容

時。

linker name是用於連結時的名稱,是不含版號的 soname ,如: libmylib.so。

通常 linker name與 real name是用 ln 指到對應的 real name ,用來提供

彈性與維護性。


2.1 編譯

shared library的製作過程較複雜。

$ gcc -c -fPIC hello.c world.c

編譯時要加上 -fPIC 用來產生 position-independent code。也可以用 -fpic

參數。 (不太清楚差異,只知道 -fPIC 較通用於不同平台,但產生的code較大

,而且編譯速度較慢)。

$ gcc -shared -Wl,-soname,libmylib.so.1 -o libmylib.so.1.0.0 \

hello.o world.o

-shared 表示要編譯成 shared library

-Wl 用於參遞參數給linker,因此-soname與libmylib.so.1會被傳給linker處理。

-soname用來指名 soname 為 limylib.so.1

library會被輸出成libmylib.so.1.0.0 (也就是real name)

若不指定 soname 的話,在編譯結連後的執行檔會以連時的library檔名為

soname,並載入他。否則是載入soname指定的library檔案。

可以利用 objdump 來看 library 的 soname。

$ objdump -p libmylib.so | grep SONAME

SONAME libmylib.so.1

若不指名-soname參數的話,則library不會有這個欄位資料。

在編譯後再用 ln 來建立 soname 與 linker name 兩個檔案。

$ ln -s libmylib.so.1.0.0 libmylib.so

$ ln -s libmylib.so.1.0.0 libmylib.so.1


2.2 使用

與使用 static library 同。

$ gcc main.c libmylib.so

以上直接指定與 libmylib.so 連結。

或用

$ gcc main.c -L. -lmylib

linker會搜尋 libmylib.so 來進行連結。

如果目錄下同時有static與shared library的話,會以shared為主。

使用 -static 參數可以避免使用shared連結。

$ gcc main.c -static -L. -lmylib

此時可以用 ldd 看編譯出的執行檔與shared程式庫的相依性

$ldd a.out

linux-gate.so.1 => (0xffffe000)

libmylib.so.1 => not found

libc.so.6 => /lib/libc.so.6 (0xb7dd6000)

/lib/ld-linux.so.2 (0xb7f07000)

輸出結果顯示出該執行檔需要 libmylib.so.1 這個shared library。

會顯示 not found 因為沒指定該library所在的目錄,所找不到該library。

因為編譯時有指定-soname參數為 libmylib.so.1 的關係,所以該執行檔會

載入libmylib.so.1。否則以libmylib.so連結,執行檔則會變成要求載入

libmylib.so

$ ./a.out

./a.out: error while loading shared libraries: libmylib.so.1:

cannot open shared object file: No such file or directory

因為找不到 libmylib.so.1 所以無法執行程式。

有幾個方式可以處理。

a. 把 libmylib.so.1 安裝到系統的library目錄,如/usr/lib下

b. 設定 /etc/ld.so.conf ,加入一個新的library搜尋目錄,並執行ldconfig

更新快取

c. 設定 LD_LIBRARY_PATH 環境變數來搜尋library

這個例子是加入目前的目錄來搜尋要載作的library

$ LD_LIBRARY_PATH=. ./a.out

Hello world.


3. Dynamically loaded libraries

Dynamicaaly loaded libraries 才是像 windows 所用的 DLL ,在使用到

時才載入,編譯連結時不需要相關的library。動態載入庫常被用於像plug-ins

的應用。

3.1 使用方式

動態載入是透過一套 dl function來處理。

#include

void *dlopen(const char *filename, int flag);

開啟載入 filename 指定的 library。

void *dlsym(void *handle, const char *symbol);

取得 symbol 指定的symbol name在library被載入的記憶體位址。

int dlclose(void *handle);

關閉dlopen開啟的handle。

char *dlerror(void);

傳回最近所發生的錯誤訊息。

dltest.c

#include
#include
#include

int main() {
    void *handle;
    void (*f)();
    char *error;

    /* 開啟之前所撰寫的 libmylib.so 程式庫 */
    handle = dlopen("./libmylib.so", RTLD_LAZY);
    if( !handle ) {
        fputs( dlerror(), stderr);
        exit(1);
    }

    /* 取得 hello function 的 address */
    f = dlsym(handle, "hello");
    if(( error=dlerror())!=NULL) {
        fputs(error, stderr);
        exit(1);
    }
  /* 呼叫該 function */
    f();
    dlclose(handle);
}

編譯時要加上 -ldl 參數來與 dl library 連結

$ gcc dltest.c -ldl

結果會印出 Hello 字串

$ ./a.out

Hello

關於dl的詳細內容請參閱 man dlopen

--

參考資料:

Creating a shared and static library with the gnu compiler [gcc]

Program Library HOWTO

Makefile

GCC=gcc  
CFLAGS=-Wall   -ggdb   -fPIC  
#CFLAGS=  
   
all:   libfunc  
   
libfunc:func.o   func1.o  
    $(GCC)   -shared   -Wl,-soname,libfunc.so.1   -o  libfunc.so.1.1   $<  
    ln   -sf   libfunc.so.1.1   libfunc.so.1  
    ln   -sf   libfunc.so.1   libfunc.so  
   
%.o:%.c  
    $(GCC)   -c   $(CFLAGS)   -o   $@   $<  
   
clean:  
    rm   -fr   *.o  
    rm   -fr   *.so*  




Thursday, November 25, 2021

[xgo] The first phase of hacking xgo

 xgo is a great Go CGO cross compiler that can help users to build the application written in Golang to multi-platforms at the same time. 

For more information in details, please check out these URLs:

https://github.com/karalabe/xgo
https://www.jianshu.com/p/a6047d3f976e

But, without any further study, we probably cannot understand how it works well. So here I just give some information about it. I will use my Github repository as the example:

# Install xgo
go get github.com/karalabe/xgo
$ git clone https://github.com/teyenliu/win-shared-example
$ cd win-shared-example

# Use xgo to cross-compile to windows/amd64
xgo -targets windows/amd64 github.com/teyenliu/win-shared-example


Then, we finish the cross-compilation for my application
And actually, xgo, this tool, is based on its prebuilt docker image: "karalabe/xgo-latest" to complete the cross-compilation task. We can check the dockerhub for it as well: https://hub.docker.com/r/karalabe/xgo-latest/builds

So, we can directly run this Docker Image to cross-compile my application based on my case as follows:

$ docker run --rm -v /home/liudanny/git/xgo:/build \
-v /home/liudanny/.xgo-cache:/deps-cache:ro \
-e REPO_REMOTE= \
-e REPO_BRANCH= \
-e PACK= \
-e DEPS= \
-e ARGS= \
-e OUT= \
-e FLAG_V=false \
-e FLAG_X=false \
-e FLAG_RACE=false \
-e FLAG_TAGS= \
-e FLAG_LDFLAGS= \
-e FLAG_BUILDMODE=default \
-e TARGETS=windows/amd64 \
-e EXT_GOPATH= \
karalabe/xgo-latest \
github.com/teyenliu/win-shared-example


Furthermore, we can directly run xgo's build.sh to cross-compile my application in the host instead of in in the container:

$ sudo su
# set Golang path
export PATH=$PATH:/usr/local/go/bin
export GOPATH=$HOME/go
export PATH=$PATH:$GOPATH/bin

# rm /deps in advance
rm -rf /deps

# run build.sh
env GO111MODULE=off \
env REPO_REMOTE= \
env REPO_BRANCH= \
env PACK= \
env DEPS= \
env ARGS= \
env OUT= \
env FLAG_V=false \
env FLAG_X=false \
env FLAG_RACE=false \
env FLAG_TAGS= \
env FLAG_LDFLAGS= \
env FLAG_BUILDMODE=default \
env TARGETS=windows/amd64 \
env EXT_GOPATH= \
env BUILD_DEPS=/home/liudanny/git/xgo/docker/base/build_deps.sh \
env BOOTSTRAP_REPO=/home/liudanny/git/xgo/docker/base/bootstrap_repo.sh \
env BOOTSTRAP_PURE=/home/liudanny/git/xgo/docker/base/bootstrap_pure.sh \
env BOOTSTRAP=/home/liudanny/git/xgo/docker/base/bootstrap.sh \
env PATH=/usr/local/go/bin:$PATH \
./build.sh github.com/teyenliu/win-shared-example

Reference:
Golang 交叉编译
Golang交叉编译各个平台的二进制文件
Fyne Cross Compile


The following commands are not succesful to work for cross-compilation
$ CGO_ENABLED=1 CXX="x86_64-w64-mingw32-g++" CXX_FOR_TARGET="x86_64-w64-mingw32-g++" CC="x86_64-w64-mingw32-gcc" CC_FOR_TARGET="x86_64-w64-mingw32-gcc" GOOS=windows GOARCH=amd64 \
go build -ldflags -v -x -installsuffix cgo -o example.exe main.go

$ GOOS=windows GOARCH=amd64 CGO_ENABLED=1 CC=x86_64-w64-mingw32-gcc go build -ldflags -v -x -installsuffix cgo -o example.exe main.go


Monday, November 22, 2021

[Fun] Detecting the license plate and replacing with a given image

Propably two years ago, I studied a ECCV 2018 paper "License Plate Detection and Recognition in Unconstrained Scenarios", and  its GitHub (https://github.com/sergiomsilva/alpr-unconstrained) repository contains the author's implementation. During that time, I had an idea to leveraged its implementation for replacing the license plate with my image. After tryed a couple days, I successed and provided a Python script to detect and replace cars' license plate in the video with a specified image.

I recently commited my modification code and pushed it to my forked repository as follows:
https://github.com/teyenliu/alpr-unconstrained

In this repository, it contains 2 samples which are convenient for users to test.

The way to process a video which will replace any car license plate in the video with a given image content.

$ python license-plate-replacement.py --video cars_mountain_short.mov --replace src/hello.PNG --output output1.avi
$ python license-plate-replacement.py --video cars_mountain.mp4 --replace src/hello.PNG --output output2.avi

The result looks like this:



Friday, November 19, 2021

[Golang] Golang with Cgo 動態連結(dynamic linking) 與 靜態連結(static linking)函式庫範例與筆記 in Windows environment

This post is focusd on how to do dynamic/static linking using Golang with cgo on Windows environment. If you want to see it on Linux environment, please check out another post: 

[Golang] Golang with Cgo 動態連結(dynamic linking) 函式庫範例與筆記 in Linux environment

First of all, thanks to someone who had posted several useful articles describing how to do dynamic linking and static linking using Golang(Go) with Cgo in Windows environment as follows:

P.S: For using this example on Linux environment, you should deal with the build process for the static and dynamic libraries.

在 Windows 環境建置動態連結函式庫 (Dynamic-link library),使用 MinGW gcc/g++ 以及 CodeBlock

Go with Cgo 靜態連結 (static linking) 函式庫建置範例與筆記
Go with Cgo 動態連結 (dynamic linking) 函式庫建置範例與筆記

Wednesday, November 3, 2021

[tmux] The most common used tmux commands

 This post can help you quickly learn how to use tmux in just a few minutes.

Get tmux session ready to start (new session)

# new session 
$ tmux new -s {session name}

# or new session without name
$ tmux

# list sessions
$ tmux ls

Split a tmux window into two panes

# <Ctrl+b> + %:split a tmux window into two panes vertically
# <Ctrl+b> + ":split a tmux window into two panes horizontally

# The way to switch to another pane
# <Ctrl + b> + <Arrow keys ← or → >

P.S: If you want to close one of the pane you created, 
you just switch on it and exit it by exit command.

For instance: split a tmux window into two panes vertically


Adjust a splitted pane size

You also are able to ajust  the splitted pane vertically or horizontally.

# The way to adjust a pane
# <Ctrl + b> + <Ctrl + Arrow keys  or  >

For instance: There are 5 panes in the window and I can adjust the currently focused pane.



Attach tmux session or detach it

# <Ctrl+b> + d:Put current session in the background running (detach)

# attach a background session
$ tmux attach-session -t {session name}

# Or
$ tmux a -t {session name}

Delete tmux session

$ tmux kill-session -t {session name}
$ tmux kill-session -t 0


Thursday, October 28, 2021

[Golang] The example function of converting Struct or Map data to Map[string]interface{} Type

    I recently encountered a use case that I need to convert my struct or map value to the type of map[string]interface{}. After surveying and studying for a while, I figure out how to deal with this kind of task and the way to do it.

    The following function ConvertStructToMap()  is the example of converting Struct or Map data to Map[string]interface{} Type. It receives the argument as interface type and uses reflect function to check the receiver type is correct and what we want. But, it has a constrain, which if you give a map, and Map's key type must be string.

// ConvertStructToMap()
func ConvertStructToMap(message interface{}) (map[string]interface{}, error) {
msg := reflect.ValueOf(message)
msgtype := msg.Type()

// support Struct
if msgtype.Kind() == reflect.Struct {
// message should be tagged by "codec" or "msg"
kv := make(map[string]interface{})
fields := msgtype.NumField()
for i := 0; i < fields; i++ {
field := msgtype.Field(i)
name := field.Name
if n1 := field.Tag.Get("json"); n1 != "" {
name = n1
} else if n2 := field.Tag.Get("msg"); n2 != "" {
name = n2
}
kv[name] = msg.FieldByIndex(field.Index).Interface()
}
return kv, nil
}
// support map[string]interface{} or map[string]string
if msgtype.Kind() != reflect.Map {
return nil, errors.New("message must be a map")
} else if msgtype.Key().Kind() != reflect.String {
return nil, errors.New("map keys must be strings")
}

// Get the interface{}'s current value
kv := make(map[string]interface{})
for _, k := range msg.MapKeys() {
kv[k.String()] = msg.MapIndex(k).Interface()
}
return kv, nil
}

func main() {
fmt.Println("Hello, playground")
structData := struct { Name string `json:"JsonName"` 
                       Score int `msg:"MsgScore"`} { 
                       "john smith", 30, }
kv, err := ConvertStructToMap( structData )
if err != nil {
    fmt.Println(err)
}
fmt.Println(kv)
}

How to use it? Here we go! You can try it on the Go Playground:
https://play.golang.org/p/TLKBIbSxIQj

or just take a look at the sample as follows:

// The way to use ConvertStructToMap()
func main() {
structData := struct { Name string `json:"JsonName"` 
                       Score int `msg:"MsgScore"`} { 
                       "john smith", 30, }
kv, err := ConvertStructToMap( structData )
if err != nil {
    fmt.Println(err)
}
fmt.Println(kv)
        mapData := make(map[string]int)
        mapData["Score1"] = 100
        mapData["Score2"] = 60
        kv, err = ConvertStructToMap( mapData )
        if err != nil {
    fmt.Println(err)
}
fmt.Println(kv)
}


Run Result==>

map[JsonName:john smith MsgScore:30]
map[Score1:100 Score2:60]



Monday, October 18, 2021

Python matplotlib 中文亂碼解決 in Linux or Windows

 大家是否有遇過當使用matplotlib時,遇到顯示不出中文或是中文為亂碼的問題。無論是在Anaconda內使用matplotlib或是直接在Python安裝環境內使用matplotlib,基本解決方式是一樣的。本文將以簡單與條列方式描述在Linux 或是 Windows 解決方法。

1. 下載字型的來源

https://www.fontpalace.com/font-download/SimHei/

2. 字型安裝

執行python command, 並以下列方式找到matplotlib放置ttf字型檔位置與設定檔

>>> import matplotlib
>>> matplotlib.matplotlib_fname()

根據下列不同環境的執行結果內放置ttf字型檔

Windows: 

放到: C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\matplotlib\mpl-data\fonts\ttf 

修改設定檔  C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\matplotlib\mpl-data\matplotlibrc

Linux:

 
放到: /usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/fonts/ttf
修改設定檔  /usr/local/lib/python2.7/dist-packages/matplotlib/mpl-data/matplotlibrc

Anaconda:


放到: /opt/conda/lib/python3.7/site-packages/matplotlib/mpl-data/fonts/ttf
修改設定檔  /opt/conda/lib/python3.7/site-packages/matplotlib/mpl-data/matplotlibrc

3. 設定檔matplotlibrc修改方式
font.family  : sans-serif
font.serif      : SimHei   <== 增加此項目

範例如下:


4. 重新載入字體

>>> from matplotlib.font_manager import _rebuild
>>> _rebuild() 



Wednesday, October 6, 2021

個人常用的 Git Commands

 0. 相關初始設定

git init
git config --global user.name "teyenliu"
git config --global user.email "teyen.liu@gmail.com"
git config --list

#git config --global http.sslverify false.
#git config --global log.decorate short
#git config --global color.ui true
#git config --global core.editor "vim"


git remote -v

#首次建立repository
#git remote add origin https://github.com/teyenliu/XXXX.git
#git branch -M main  // 更換master branch name to main
#git push -u origin master

#CRLF
#Linux: git config --global core.autocrlf input
#Windows: git config --global core.autocrlf true

#Editor
#git config --global core.editor vim
#Setup commit message template
#git config --global commit.template ~/git-template

#如果是 fork a repository, 則需要設定upstream repo
git remote set-url origin <my fork's git repo>
git remote add upstream <upstream's git repo>

# 查看到我們名下的遠端項目
git remote -v

#獲取upstream的最新版本
git fetch upstream

#將upstream merge到我們當前分支
git merge upstream/master

1. check out 另一個branch
git checkout <to this branch>
git checkout -b <create_new_branch> <based on this branch>

2. 建議使用這個方式避免無謂的merges
git pull --rebase <remote name> <branch name>

3. 當要switch branch,但是有file未 commit,可以使用下列方式暫存起來
git stash save 'stash1'

#查看在 stash 中的缓存
git stash list

#恢复暂存
git stash pop

4.  git checkout 時發生 Please move or remove them before you can switch branches.
git clean -d -fx

5. 拉下來最新的檔案
git pull --rebase upstream master
git pull --rebase upstream ConfigBase

6. 推code 到 remote repo.
git push origin master

7. 補充發送
git commit --amend
git push -f origin ConfigBase
git push -f origin master

8. 建立/刪除分支
#建立
git checkout -b <branch> 從目前分支再去建立(會新增)本端分支
git push -u origin <branch> PUSH並建立遠端分支
#刪除
git branch -d <branch>

### Branch from a previous commit using Git ###
# Create the branch using a commit hash:
git branch branch_name <commit-hash>

#Or by using a symbolic reference:
git branch branch_name HEAD~3

#To checkout the branch while creating it, use:
git checkout -b branch_name <commit-hash or HEAD~3>

9. 移動某個commit點
#回到 上一次的 commit
git reset --hard HEAD

#回到 上一次的前一次 commit
git reset --hard HEAD^

#裡面最近做的所有 HEAD 的改動
git reflog
git reset --hard 904e1ba #最近的commit

10. 如果想要打tag( 以某個commit為基礎 )
# 產生一個新的branch以某個commit
git checkout -b <tagname> <commit id>
git push origin <tagname>

# 要先把 branch push上去的原因是, 之後建立的tag如果跟brnach name相同 (例如: v0.4 ), 則 branch會發生如下訊息:
error: src refspec v0.4 matches more than one.
error: failed to push some refs
如果不小心先push tags, 則必須把tag移除後, 才能push branch.


# 查看目前有的tag
git tag -l

# 打tag
git tag -a <tagname> -m "My App description"

# 刪除tag
git tag -d <tagname>

# push all tags 到 remote端
git push origin --tags

11. 如果想要檢查 submodule 是否有更新可使用下列指令:
git submodule foreach --recursive git pull origin master

12. 強制推送到remote
git push -f

13. 拉submodule的程式碼
git submodule update --recursive --remote

14. 查看完整的Commit 資料
git log --pretty=oneline

15. 退回到特定 Commit
# where [revision] is the commit hash (for example: 12345678901234567890123456789012345678ab)
git checkout [revision] .

#To rollback to a specific commit:
git reset --hard commit_sha

#To rollback 10 commits back:
git reset --hard HEAD~10

Trouble Shooting

如果遇到Tag Name與Branch Name相同時, 會有其中一種無法上傳至遠端的錯誤:
src refspec XXX 匹配多個, 例如:
error: src refspec v0.3 matches more than one

解決方式:

1. 先刪除分支然後上傳Tag:
git branch -D testtag

或是

2. 先刪除tag然後上傳分支:
git tag -d testtag