Posts

[PCIe] lspci command and the PCIe devices in my server

Image
The following content is about my PCIe devices/drivers and the lspci command results.
$ cd /sys/bus/pci_express/drivers $ ls -al drwxr-xr-x 2 root root 0  7月  6 15:33 aer/ drwxr-xr-x 2 root root 0  7月  6 15:33 pciehp/ drwxr-xr-x 2 root root 0  7月  6 15:33 pcie_pme/

$ cd  pcie_pme $ ls -al


$ lspci | grep 00:1c


Or $ cd /sys/bus/pci_express/devices $ ls -al


$ cd 0000:00:1c.0:pcie01 $ ls -al total 0 drwxr-xr-x 3 root root    0  7月  5 08:56 ./ drwxr-xr-x 6 root root    0  7月  5 08:56 ../ lrwxrwxrwx 1 root root    0  7月  6 15:51 driver -> ../../../../bus/pci_express/drivers/pcie_pme/ drwxr-xr-x 2 root root    0  7月  6 15:51 power/ lrwxrwxrwx 1 root root    0  7月  6 15:51 subsystem -> ../../../../bus/pci_express/

[Caffe] Install Caffe and the depended packages

Image
This article is just for me to quickly record the all the steps to install the depended packages for Caffe. So, be careful that it maybe is not good for you to walk through them in your environment. ^_^

# Install CCMAKE $ sudo apt-get install cmake-curses-gui
# Build my own installation location $ mkdir ~/local_install # ProtoBuffer $ tar zxvf protobuf-2.5.0.tar.gz
$ cd protobuf-2.5.0
$ ./configure --prefix=/home/liudanny/local_install/
$ make -j2
$ make install
#Boost $ tar xvf boost_1_56_0.tar.bz2
$ cd boost_1_56_0
### ./bootstrap.sh --show-libraries ###
$ ./bootstrap.sh --with-libraries=program_options,filesystem,system,exception,thread
$ ./b2
$ cp -r boost/ /home/liudanny/local_install/include
$ cp stage/lib/* /home/liudanny/local_install/lib/
# Gflags $ unzip gflags-2.1.1.zip
$ cd gflags-2.1.1
$ mkdir build
$ cd build
$ cmake ..
$ ccmake ..

$ make -j2
$ make install
# Glog $ tar zxvf glog-0.3.3.tar.gz
$ cd glog-0.3.3
$ ./configure --prefix=/home/liudanny/local_install
$ make -j2
$…

[NCCL] Build and run the test of NCCL

NCCL requires at least CUDA 7.0 and Kepler or newer GPUs. Best performance is achieved when all GPUs are located on a common PCIe root complex, but multi-socket configurations are also supported.

Note: NCCL may also work with CUDA 6.5, but this is an untested configuration.

Build & run To build the library and tests.

$ cd nccl
$ make CUDA_HOME=<cuda install path> test
Test binaries are located in the subdirectories nccl/build/test/{single,mpi}.

$ ~/git/nccl$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:./build/lib
$ ~/git/nccl$ ./build/test/single/all_reduce_test 100000000
# Using devices
#   Rank  0 uses device  0 [0x04] GeForce GTX 1080 Ti
#   Rank  1 uses device  1 [0x05] GeForce GTX 1080 Ti
#   Rank  2 uses device  2 [0x08] GeForce GTX 1080 Ti
#   Rank  3 uses device  3 [0x09] GeForce GTX 1080 Ti
#   Rank  4 uses device  4 [0x83] GeForce GTX 1080 Ti
#   Rank  5 uses device  5 [0x84] GeForce GTX 1080 Ti
#   Rank  6 uses device  6 [0x87] GeForce GTX 1080 Ti
#   Rank  7 uses de…

[Mpld3] Render Matplotlib chart to web using Mpld3

Image
The following example is about rendering a matplotlib chart on web, which is based on Django framework to build up. I encountered some problems before, such as, not able to see chart on the web page or having a run-time error after reloading the page. But, all the problems are solved.

<< demo/views.py>> import matplotlib.pyplot as plt
import numpy as np
import mpld3

def plot_test1(request):
context = {}
fig, ax = plt.subplots(subplot_kw=dict(axisbg='#EEEEEE'))
N = 100

"""
Demo about using matplotlib and mpld3 to rendor charts
"""
scatter = ax.scatter(np.random.normal(size=N),
np.random.normal(size=N),
c=np.random.random(size=N),
s=1000 * np.random.random(size=N),
alpha=0.3,
cmap=plt.cm.jet)
ax.grid(color='white', linestyle='solid')

ax.set_title("Scatter Plot (with tooltips!)", size=20)

labels = ['point {0}'.format(i + 1) for i in range(N)]
tooltip = mpld3.plugins.PointLabelTooltip(scatter, labels=label…

[Hadoop] To build a Hadoop environment (a single node cluster)

For the purpose of studying Hadoop, I have to build a testing environment to do. I found some resource links are good enough to build a single node cluster of Hadoop MapReduce as follows. And there are additional changes from my environment that I want to add some comments for my reference.

http://www.thebigdata.cn/Hadoop/15184.html
http://www.powerxing.com/install-hadoop/
Login the user "hadoop" $ sudo su - hadoop
Go to the location of Hadoop $ /usr/local/hadoop
Add the variables in ~/.bashrc export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64 export HADOOP_HOME=/usr/local/hadoop export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop export HADOOP_INSTALL=/usr/local/hadoop export PATH=$PATH:$HADOOP_INSTALL/bin export PATH=$PATH:$HADOOP_INSTALL/sbin export HADOOP_MAPRED_HOME=$HADOOP_INSTALL export HADOOP_COMMON_HOME=$HADOOP_INSTALL export HADOOP_HDFS_HOME=$HADOOP_INSTALL export YARN_HOME=$HADOOP_INSTALL
Modify $JAVA_HOME in etc/hadoop/hadoop-env.sh export JAVA_HOME=/usr/lib/jvm/java-…

[Spark] To install Spark environment based on Hadoop

This document is to record how to install Spark environment based on Hadoop as the previous one. For running Spark in Ubuntu machine, it should install Java first. Using the following command is easily to install Java in Ubuntu machine.

$ sudo apt-get install openjdk-7-jre openjdk-7-jdk
$ dpkg -L openjdk-7-jdk | grep '/bin/javac'
$ /usr/lib/jvm/java-7-openjdk-amd64/bin/javac

So, we can setup the JAVA_HOME environment variable as follows:
$ vim /etc/profile
  append this ==> export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64

$ sudo tar -zxf ~/Downloads/spark-1.6.0-bin-without-hadoop.tgz -C /usr/local/
$ cd /usr/local
$ sudo mv ./spark-1.6.0-bin-without-hadoop/ ./spark
$ sudo chown -R hadoop:hadoop ./spark

$ sudo apt-get update
$ sudo apt-get install scala
$ wget http://apache.stu.edu.tw/spark/spark-1.6.0/spark-1.6.0-bin-hadoop2.6.tgz
$ tar xvf spark-1.6.0-bin-hadoop2.6.tgz
$ cd /spark-1.6.0-bin-hadoop2.6/bin
$ ./spark-shell

$ cd /usr/local/spark
$ cp ./conf/spark-env.sh.temp…

[picamera] Solving the problem of video display using Raspberry Pi Camera

When I tried to use Raspberry Pi Camera to display video or image, I encountered a problem that there is no image frame and the GUI showed a black frame on the screen. It took me a while to figure out this issue.
    After searching the similar error on the Internet, I found it is related with using picamera library v1.11 and Python 2.7. So I try downgrading to picamera v1.10 and this should resolve the blank/black frame issue:

The linux command is as follows:
$ sudo pip uninstall picamera
$ sudo pip install 'picamera[array]'==1.10

So, it seems there are some issues with the most recent version of picamera that are causing a bunch of problems for Python 2.7 and Python 3 users.