Object Detection With Deep Neural Networks

Recently I red a series of papers about object detection using deep neural networks. Here is summary of the reading.

R-CNN: Region-based Convolutional Networks for Accurate Object Detection and Segmentation
用传统方法提出region proposal，train一个classifier和一个location regressor。classifier为了提高精度先用softmax train，然后fc层提出来的feature用svm fit，regressor单独作为一个network

用到了selective search生成proposal，从一个over-segmentation开始repeatedly merge similar regions. 然后每个region用传统的descriptor提feature，用bag-of-words model+ svm分类

Fast R-CNN


Faster R-CNN
提出一个region proposal network (RPN) 共享feature extraction的networks，不增加计算开销的情况下把上一版最耗时的region proposal步骤变成自动从network生成。

[Linux Tips] Bash Shell (I)

1.to declare a new variable:
variable_name=initial_value;
2. in linux, before you run you shell-script, you need to change the mod of script file.
chmod 755 script_name
(This will change the mode of file to owner: rwx, group and others: r-x)
maybe ./script_name is more precisely. This command will add the permission of execution to this script file.
3. use a variable:
$variable_name
4. to do something on all certain files under current directory:
for file in ` ls *[^smp].byu `
do
# do something on every file $file
echo $file
done

if ...
do
else
fi # end of if :)

5. substring:
To take part of a string:
string="test"
substring=${string:1:2}
echo subtring
result: es
note:
substring=${string_variable_name:starting_position:length}
The string starting position is zero-based.

6. Searching and Replacing Substrings within Strings:
In this method you can replace one or more instances of a string with another string. Here is the basic syntax:
alpha="This is a test string in which the word \"test\" is replaced."
beta="${alpha/test/replace}"
The string "beta" now contains an edited version of the original string in which the first case of the word "test" has been replaced by "replace". To replace all cases, not just the first, use this syntax:
beta="${alpha//test/replace}"

Note the double "//" symbol.

Here is an example in which we replace one string with another in a multi-line block of text:

list="cricket frog cat dog"
poem="I wanna be a x\n\
A x is what I'd love to be\n\
If I became a x\n\
How happy I would be.\n"
for critter in $list; do
echo -e ${poem//x/$critter}
done
Run this example:
$ ./myscript.sh
result:
I wanna be a cricket
A cricket is what I'd love to be
If I became a cricket
How happy I would be.
I wanna be a frog
A frog is what I'd love to be
If I became a frog
How happy I would be.
I wanna be a cat
A cat is what I'd love to be
If I became a cat
How happy I would be.
I wanna be a dog
A dog is what I'd love to be
If I became a dog
How happy I would be.
Silly example, huh? It should be obvious that this search & replace capability could have many more useful purposes.

7. to run matlab (e.g plot some graph) without start matlab window
matlab -nodesktop -nosplash -r "plot_data;quit;"
#here we have a .m file named "plot_data.m" under current dir

[Machine Learning] from distance to kernel (classification via SVM)

Since for many machine learning techniques, you can use kernel trick. This is important to find a kernel, or explicitly construct a kernel from your data. One possible way to do this is calculate metric distance.
This article only gives some fundamental ideas for this transformation, without any strict provment.

Once you have a matrix of distance to measure the difference or distance between a pair of instances, denote I_i, I_j. You may define a kernel function /rho (I_i, I_j) = f(d(I_i,I_j)), where d(I_i,I_j) is the distance you've obtained. The most common choice is natural exponential.
http://math.stackexchange.com/questions/221704/transforming-a-distance-function-to-a-kernel

Another webpage to list some ways to calculate kernel from original feature space:
http://scikit-learn.org/stable/modules/metrics.html

For the requirement for positive kernel, you may refer to this paper:
http://www.kyb.mpg.de/fileadmin/user_upload/files/publications/attachments/scholkopf00kernel_3781%5b0%5d.pdf

[Linux Tips] Screen, date

[Linux Tips] includes a series of very common tips for development in Linux environment. By default, I am using Ubuntu.

Screen.
This is used to create/maintain a session. This is very useful when you ssh to a remote machine and running some jobs in a interactive mode. Imagine if your network is disconnected, the thread to run your job will be killed. Screen can help you to create a session. You can attach (enter) or detach (quit) from the session.
1. To create a session
screen -S session_name
or simply,
screen

2. To list all sessions available.
screen -ls

3. Attach a session
screen -r session_name (a prefix is enough)

4. Detach a session
ctrl+a+d

5. Delete a session
screen -X -S session_name kill


Date
When you run some bash script. It's common that you want to name your output with current date/time.
In bash shell script you can do it as below.
dt_suffix=`date +'%d%m%H%M'`
Basically, it just assigns the date/time to a variable. And the string comes from date command. The +"%d%m%H%M" is the optional format parameter.


Bash Shell script, some common usages.
For-loop with an array.

declare -a your_array=(
foo
bar
baz
)
array_len=${#your_array[@]}
for ((i = 0; i< ${array_len}; i++));
do
echo "processing" ${your_array[${i}]}
# Do something here.
done

Check whether directory or file already exists.
file_path=PathToYourFile
dir_path=PathToYourDir
if [ ! -f ${file_path} ]; then
echo "File ${file_path} does not exist!"
else
echo "File ${file_path} already exists!"
fi

if [ ! -d ${dir_path} ]; then
echo "Directory ${dir_path} does not exist!"
else
echo "Directory ${dir_path} already exists!"
fi

Andrew Ng的AI新课程

Andrew Ng离开baidu后三件大事之一是成立deeplearning.ai，开设五门深度学习课程。网易云课堂搬运并翻译了前三门课。免费听一下，感觉课程内容虽然比较浅显偏engineering多一些，但总体还是质量颇高，收获不少，按照教学大纲总结如下 （慢慢更新）。

第一周  深度学习概论：

学习驱动神经网络兴起的主要技术趋势，了解现今深度学习在哪里应用、如何应用。

1.1  欢迎来到深度学习工程师微专业

ng秀中文，委托网易发布中文字幕版deeplearning.ai的课程。理念是希望培养成千上万人工智能人才，构建人工智能驱动的社会。

1.2  什么是神经网络？

介绍神经网络，house price prediction的例子，因为price不能为负所以曲线变成ReLU，很有意思的一个引出ReLU的方式，hiden unit 也称之为neuron。neural network就是stack neuron (like Lego brick) together toform a network.很直观。

1.3  用神经网络进行监督学习

几乎所有有价值的机器学习／神经网络技术都是supervised learning。也就是model一个x到y的映射函数。相对应的unsupervised learning，不存在y，只能指望data can tell something itself。

举了很多监督学习的例子

house price prediction用standard NN解决；image understanding，object detection这类问题用convolutional NN （CNN）解决；time series，or temporal sequence数据问题用Recurrent NN （RNN）解决；

structural data：房屋size等；unstructual data: audio/image/text, etc.

1.4  为什么深度学习会兴起？

因为数据量变大了，算法强了，计算资源强了。

举例：ReLU代替sigmoid。主要的优势在于sigmoid在两侧gradient几乎为0，导致gradient decent优化速度变慢。

interesting graph： idea ->code->experiments->idea... computation的提高导致这个iterative process加速，能带来更多更好结果。

1.5  关于这门课

简介五门课程和第一门课。

1.6  课程资源

鼓励上forum讨论问题。其他问题也可以联系deeplearning.ai。可以看出这个公司/机构的目的是培训为主，所以提到如果有公司需要培训hundreds of empolyees with deep learning expertise，可以联系他们，大学老师想开deep learning课的也可联系他们。

第二周  神经网络基础：

学习如何用神经网络的思维模式提出机器学习问题、如何使用向量化加速你的模型。

2.1  二分分类

2.2  logistic 回归

2.3  logistic 回归损失函数

2.4  梯度下降法

2.5  导数

2.6  更多导数的例子

2.7  计算图

2.8  计算图的导数计算

2.9  logistic 回归中的梯度下降法

2.10  m 个样本的梯度下降

2.11  向量化

2.12  向量化的更多例子

2.13  向量化 logistic 回归

2.14  向量化 logistic 回归的梯度输出

2.15  Python 中的广播

2.16  关于 python / numpy 向量的说明

2.17  Jupyter / Ipython 笔记本的快速指南

2.18  （选修）logistic 损失函数的解释

第三周  浅层神经网络：

学习使用前向传播和反向传播搭建出有一个隐藏层的神经网络。

3.1  神经网络概览

3.2  神经网络表示

3.3  计算神经网络的输出

3.4  多样本向量化

3.5  向量化实现的解释

3.6  激活函数

3.7  为什么需要非线性激活函数？

3.8  激活函数的导数

3.9  神经网络的梯度下降法

3.10  （选修）直观理解反向传播

3.11  随机初始化

第四周  深层神经网络：

理解深度学习中的关键计算，使用它们搭建并训练深层神经网络，并应用在计算机视觉中。

4.1  深层神经网络

4.2  深层网络中的前向传播

4.3  核对矩阵的维数

4.4  为什么使用深层表示

4.5  搭建深层神经网络块

4.6  前向和反向传播

4.7  参数 VS 超参数

4.8  这和大脑有什么关系？

How does Matlab calculate the eccentricity of a region

In matlab, there is a built-in function to calculate properties of a region.
http://www.mathworks.com/help/images/ref/regionprops.html#bqkf8jf

And as said in help message:
'Eccentricity' — Scalar that specifies the eccentricity of the ellipse that has the same second-moments as the region. The eccentricity is the ratio of the distance between the foci of the ellipse and its major axis length. The value is between 0 and 1. (0 and 1 are degenerate cases; an ellipse whose eccentricity is 0 is actually a circle, while an ellipse whose eccentricity is 1 is a line segment.) This property is supported only for 2-D input label matrices.

So, the idea is to fit using a ellipse with same second-moments as the region.
What does it mean?
The answer is in this thread:
http://stackoverflow.com/questions/1532168/what-are-the-second-moments-of-a-region

To simplify, the idea is to calculate the co-variance matrix, then do eign-value decomposition. Eigen-values are those axis length, minor and major one. While eigen-vectors are the directions of minor/major axis.

length of major axis = 2a, minor axis = 2b, then the foci = c, then:
eccentricity E = c/a = sqrt(1-(b/a)^2)
a^2-b^2 = c^2.

Something associated with wearable health care device and service

Baidu delivered its solution of wearable health care
  http://dulife.baidu.com/

Android wearable
http://developer.android.com/wear/index.html?utm_source=ausdroid.net

Qualcomm life
http://www.qualcommlife.com/