ML_general_talk.md

why this article

I am not a newbie for machine learning any more. But still sometimes I suspect what did I gain from learning “machine learning”. By applying some classical algorithms, I get some real feeling about this hot topic.

everything is about generalization

Generalization means the ability to have good prediction on novel data samples. In other words, when you make prediction on testing data given the model you trained on training data.
You can easily get a 100% accuracy on training data, except for some ambiguous data point (same data points, but different label). This is meaningless since your decision boundary is too complex. The terminology is over-training. Instead of record every training data sample by taking photo, you need to loss the decision boundary. Some technologies play this role essentially, like margin in SVM, regularization in general optimization.
Another this is features goes first. You cannot do magic on bad features. This means spend more time on feature extraction/selection/design is worthy. In some sense, deep learning or sparse coding/representation is to put efforts on the steps before classification.

Do Preprocessing

Some simple preprocessing, e.g. normalization, whitening, etc. can really benefit your classification.

select correct classification algorithm

Linear or not, svm or lda.
I always try four algorithm to have baseline:
1. Support Vector Machine (SVM)
2. Linear Discriminant Analysis (LDA)
3. Random Forest (RF)
4. K Nearest Neighbor (KNN)

Written with StackEdit.

git_notes

Git 笔记

Some basic concepts

repository 仓库，存储code的单元。
branch 分支，同一个代码库的不同版本控制路径，默认的主线是master，可以分叉出来开发测试新功能，完成后merge到master上。
commit 提交修改。
push推送到远程，默认推到original的repository。
pull 从远程repository取回代码。

一般远程的repository默认是origin, 另一个常用的名字是upstream,这个用于从一个已有的repository fork了一个副本后，自己的副本作为origin，原始的版本作为upstream。参加open source project常用到upstream。

文件状态

untracked 即没有加入到git的index
unmodified 没有个修改过的文件
modified 修改但尚未提交
staged 已经提交，上了舞台了。。

常用命令

git status检查所有文件的状态
git diff 检查文件修改前后差异
git branch 创建新分支
git checkout 切换分支

删除和ignore不同：
ignore只是忽略跟踪，但文件保留，删除是彻底删除，先删除文件再用git -rm xx.xx 实现彻底git上的删除。

深入

关于配置：
/etc/gitconfig 针对系统 git config –system
~/gitconfig 针对当前用户 git config –global
.git/config 针对当前 repository
git config –list列出当前所有配置。
提交代码最常用的配置是信息是用户名和邮箱。当你需要提交并push的时候会要求输user pwd验证身份。另一种选择是可以直接在本地产生一对公钥私钥 (ssh-keygen -t rsa -C “your_email_address”)。
所谓分布式版本控制。事实上是说没有一个主线处于支配地位。我的理解，在远程服务器比如github上也是一个类似本地的一个repository。所以你的机器如果在线，也能直接被clone。
.git文件下存储所有配置。（比如忽略某些文件，index等）

References:

GitHub详细教程
collaborate using git
git user guide
简单明了的一个教程：廖雪峰的git教程

Written with StackEdit.

java integer pool

Here in this post, I record a bug in my project(Java).
The problem is when compare two integers, it works well only when integer is of one-byte length.
This is caused by "Integer constant pool". To avoid extra memory cost, Java will return a already-created Integer object if it's between -128 and 127. This means:
Integer i = Integer(10); Integer j = Integer(10);
bool flag = (i==j); //return true; directly check the address
bool flagE = i.equals(j);//return true; directly check the values, what we expect here.
For the Object class, which is the root class of all classes in java, these two method (or operator) are the same. But for a certain derived class, it is not necessary depends on the logic.

So if the logic of your program is to check whether two integers are equally valued, then you need to use equals instead of == operator.

You may imply that java override the equals function for Integer class. this gives us another topic to discuss, that is, when you override equals function you need override hashCode(). One naturally raised question is "why always override hashcode() given equals() overrided". Here is a good article for this question:
http://www.xyzws.com/javafaq/why-always-override-hashcode-if-overriding-equals/20
To simplify it, you have to guarantee that:
if equals(Object o) gives true, hashCode() output same integer. This means you need to generate hash code from attributes which are used to decide whether two objects are equal (equals() function).

[Python] fundamental of Python

introduction

Python is a interpreting script language, which is very popular these years. This post will summarize some fundamentals of this language.
I installed python(x,y) as my python environment. To run a python script, type in “python xx.py”, or run it in IDE, just same as running matlab script in matlab IDE.
fundamental syntax:
http://wiki.woodpecker.org.cn/moin/PyAbsolutelyZipManual
import (same as Java)
string <=> int
str(i)
string.atoi（s,[，base]) #convert to int, base is the base of the number, 10, 16, or 8
string.atof(s) #convert to float
change dir:
import os
chdir(strDir)
statement (loop/if/else)
if x<0:
x++ # note that all code-block structure is defined by indent
else:
x–

no end

for i in range(10,0,-3): # range(10,0,-3) means 10:-3:0, while 0 is excluded
print “the output num is %d”%(i)
read/write file:
http://www.cnblogs.com/allenblogs/archive/2010/09/13/1824842.html
fid = open(“xx.txt”,”r”)
line = fid.readline()
fid.close()
fid = open(“xx.txt”,”w+”)
fid.write(line) # automatically return to a new line
fid.close()

List if mutable, while tuple is immutable.


How to check the structure of a variable?
You can call type(var) to know the type of the variable. Then dir() and getattr() to check the structure.

List comprehensions
x = list([0 1 2 3 4 5])
y = [x_i for x_i in x]
y = [x_i+1 for x_i in x if x>1]
y = [x_i+1 if x_i%2==0 else x_i for x_i in x if x>1]

how to sum up a list?
sum(x)

pandas DataFrame

Written with StackEdit.

image stitcher: image composite editor (MSR)

Just found this very powerful tool:
image composite editor is designed and implemented by Microsoft Research.
My example:
original input images:

output image:






video demo:
http://research.microsoft.com/en-us/um/redmond/groups/ivm/multiimagefusion/
downloads:
http://research.microsoft.com/en-us/downloads/69699e5a-5c91-4b01-898c-ef012cbb07f7/default.aspx

Multi-kernel learning, run "SMO-MKL" on my laptop (windows 8 64 bit)

I am compiling an open-source code set of multi-kernel learning:
http://research.microsoft.com/en-us/um/people/manik/code/smo-mkl/download.html
This is not the latest ML software, but could be a good start.

Now it is working on my windows 8 (64 bit)
1. Add $Programfiles$\Microsoft Visual Studio 11.0\VC\bin to $path$
  run "vcvars32.bat" to set up all path and environment variables for Visual C++.
http://msdn.microsoft.com/en-us/library/f2ccy3wt.aspx

2. type:
nmake -f Makefile.win clean all

Done~ the compiled exe files are located in Windows sub-directory.
Have a try:
svm-train -s 0 -h 0 -m 400 -o 2.0 -a 26 -c 10.0 -l 1.0 -f 0 -j 1 -g 3 -k Example/Classification/PrecomputedKernels/kernelfile Example/Classification/PrecomputedKernels/y_train Example/Classification/PrecomputedKernels/model_file
This example is to train a model from a kernel matrix. And you may test on testing data:
svm-predict Example/Classification/PrecomputedKernels/y_test Example/Classification/PrecomputedKernels/model_file Example/Classification/PrecomputedKernels/prediction

I eventually found that nmake -f Makefile.win clean all actually did not compile the svm.cpp again, but instead, it is based on pre-compiled svm.obj. And if you made a change on svm.cpp, then nmake will give you some error message. I did not have a solution for it so far.

3. error/warning message when run "svm-predict".
This is annoying, because I wanna do grid search to find the optimal values for parameters. And actually the result is correct, but only because of exception not been handled, the error message comes out. This makes the pipeline stopped and have to manually click a confirm button to continue.
I have to change the svm-predict.c, to comment off the code to destroy svm_model.

[keep updating] Deep Learning

This is the buzzword in machine learning community recently. Let's start from a 101 article. http://markus.com/deep-learning-101/

Begin to read the tutorial:
http://ufldl.stanford.edu/wiki/index.php/UFLDL_Tutorial
Chinese version:
http://deeplearning.stanford.edu/wiki/index.php/UFLDL%E6%95%99%E7%A8%8B
and another one:
http://deeplearning.net/tutorial/gettingstarted.html#


open source code set:
http://deeplearning.net/software/theano/
introduce in Chinese
http://www.52ml.net/6.html
http://blog.csdn.net/mysee1989/article/details/11992535


先说一点直观感受，deep learning用了一段时间。
Motivation 是人的感知系统是hierarchical的，以视觉系统为例，底层的neuron负责检测local low-level features，比如边界，角点，纹理。高层的neuron负责在这些low-level特征基础上提取高级特征，最后形成high-level concept。
Neural Network在七八十年代红极一时。但后来逐渐被模型更简单的SVM取代。原因有：a. NN的parameter太多优化起来很难。b. 层数多了之后容易overfitting.
计算能力的爆炸性增长解决掉第一个问题（along with better optimization algorithm）第二个问题是优化过程中加入一些regularization来克服（e.g. sparsity）.

convolutional neural network (CNN) 考虑到了spatial 局部性，底层向上层 计算时只考虑一定领域内的值，training的结果也就变成多个convolutional filter.好处是计算效率提高而且training的modle复杂度降低了。伴随来的是另一个概念max-pooling.可以看作是一个非线性的down-sampling. 一定领域内取最大值传递到上一层，领域之间是不重叠的。
max-pooling的好处是使得算法更robust，代价是丢失信息。