These days, some interesting news in face recognition field are re-posted widely on social network.
DeepFace: Closing the Gap to Human-Level Performance in Face Verification (Facebook AI lab)
It is said the performance is close to human being.
Then, more incredible, someone claimed their algorithm outperforms humankind.
http://www.zhizhihu.com/html/y2014/4520.html
https://medium.com/the-physics-arxiv-blog/2c567adbf7fc
http://www.52ml.net/14704.html
face++
http://www.faceplusplus.com/uc/app/home?app_id=14807
Most of the result is achieved on LFW (labeled face in the wild)
http://vis-www.cs.umass.edu/lfw/index.html
My adviser want me to do some face recognition stuff.
Monday, April 28, 2014
FRR, FAR, TPR, FPR, ROC curve, ACC, SPC, PPV, NPV, etc.
In a framework that an algorithm is supposed to predict "positive" or "negative". Some concepts are really confusing. So a summary here. All the concepts or metrics are widely used to measure the performance of the algorithm or machine learning model (which is essentially an computational intensive algorithm).
A: true positive (TP)
B: false negative (FN)
C: false positive (FP)
D: true negative (TN)
A+B: positive (P)
C+D: negative(N)
False reject rate (FRR) = B/(A+B) = FN/(TP+FN) = FN/P = 1-TPR
False accept rate (FAR) = C/(C+D) = FP/(FP+TN) = FPR
True positive rate (TPR) = A/(A+B) = TP/(TP+FN) = TP/P
False positive rate (FPR) = fall out = C/(C+D) = FP/(FP+TN) = FAR = 1-SPC
Accuracy (ACC) = (A+D)/(A+B+C+D)
Sensitivity = TPR = A/(A+B) = TP/(TP+FN)
Specificity (SPC) = TNR = D/(C+D) = TN/(FP+TN) = 1-FPR
Positive predictive value (PPV) = precision = A/(A+C) = TP/(TP+FP) = TP/P = TPR
Negative predictive value (NPV) = TN/(TN+FN)
False positive rate (FPR) = fall out = C/(C+D) = FP/(FP+TN) = FAR = 1-SPC
False discover rate (FDR) = C/(A+C) = FP/(TP+FP) = FP/P = 1-TPR = 1-PPV
*In bio-medical field, positive means disease, while negative indicates healthy.
Further more,
F1 Score, harmonic mean of precision and sensitivity
F1 = 2TP/(2TP+FP+FN)
Matthews correlation coefficient (MCC)
MCC = (TP*TN+FP*FN) / ((TP+FP)*(TP+FN)*(TN+FP)*(TN+FN))^0.5
When a model (classification model) is finalized, one may want to find an operating point, i.e., confidence threshold. This will be a trade-off between precision (higher with higher threshold) and recall (lower with higher threshold). Then you need a curve to show the performance at all possible threshold levels.
1. Receiver operating characteristic (ROC) curve is a plot of true positive ratio V.S. false positive ratio.
When compare the performance of two models, it is hard to tell which one is better given two curves. So people use the area under curve (AUC) to measure and compare performance (the larger the better). But as you can see, two different curves can have the same AUC. Then to choose which one is depends on whether high precision or high recall is more desirable.
The AUC can be any value between 0 and 1. A random guess classifier will create a straight line segment from (0, 0) and (1, 1). While it can happen that auc<0.5, but then one can flip the output prediction, to make a new classifier with auc' = (1-auc). In this sense, AUC can be considered as something equal or above 0.5.
In statistic learning, the AUC represents the probability that a model outputs higher score for a randomly chosen positive class than a randomly chosen negative class. To prove this, please refer to a very nice blog: https://madrury.github.io/jekyll/update/statistics/2017/06/21/auc-proof.html.
2. Precision-Recall curve is, as the name says, a plot of precision ratio V.S. recall ratio.
references:
http://en.wikipedia.org/wiki/Sensitivity_and_specificity
| ground truth\prediction | positive | negative | rate |
| positive | A | B | TPR |
| negative | C | D | TNR |
| rate | PPV,Precision | NPV |
A: true positive (TP)
B: false negative (FN)
C: false positive (FP)
D: true negative (TN)
A+B: positive (P)
C+D: negative(N)
False reject rate (FRR) = B/(A+B) = FN/(TP+FN) = FN/P = 1-TPR
False accept rate (FAR) = C/(C+D) = FP/(FP+TN) = FPR
True positive rate (TPR) = A/(A+B) = TP/(TP+FN) = TP/P
False positive rate (FPR) = fall out = C/(C+D) = FP/(FP+TN) = FAR = 1-SPC
Accuracy (ACC) = (A+D)/(A+B+C+D)
Sensitivity = TPR = A/(A+B) = TP/(TP+FN)
Specificity (SPC) = TNR = D/(C+D) = TN/(FP+TN) = 1-FPR
Positive predictive value (PPV) = precision = A/(A+C) = TP/(TP+FP) = TP/P = TPR
Negative predictive value (NPV) = TN/(TN+FN)
False positive rate (FPR) = fall out = C/(C+D) = FP/(FP+TN) = FAR = 1-SPC
False discover rate (FDR) = C/(A+C) = FP/(TP+FP) = FP/P = 1-TPR = 1-PPV
*In bio-medical field, positive means disease, while negative indicates healthy.
Further more,
F1 Score, harmonic mean of precision and sensitivity
F1 = 2TP/(2TP+FP+FN)
Matthews correlation coefficient (MCC)
MCC = (TP*TN+FP*FN) / ((TP+FP)*(TP+FN)*(TN+FP)*(TN+FN))^0.5
When a model (classification model) is finalized, one may want to find an operating point, i.e., confidence threshold. This will be a trade-off between precision (higher with higher threshold) and recall (lower with higher threshold). Then you need a curve to show the performance at all possible threshold levels.
1. Receiver operating characteristic (ROC) curve is a plot of true positive ratio V.S. false positive ratio.
When compare the performance of two models, it is hard to tell which one is better given two curves. So people use the area under curve (AUC) to measure and compare performance (the larger the better). But as you can see, two different curves can have the same AUC. Then to choose which one is depends on whether high precision or high recall is more desirable.
The AUC can be any value between 0 and 1. A random guess classifier will create a straight line segment from (0, 0) and (1, 1). While it can happen that auc<0.5, but then one can flip the output prediction, to make a new classifier with auc' = (1-auc). In this sense, AUC can be considered as something equal or above 0.5.
In statistic learning, the AUC represents the probability that a model outputs higher score for a randomly chosen positive class than a randomly chosen negative class. To prove this, please refer to a very nice blog: https://madrury.github.io/jekyll/update/statistics/2017/06/21/auc-proof.html.
2. Precision-Recall curve is, as the name says, a plot of precision ratio V.S. recall ratio.
references:
http://en.wikipedia.org/wiki/Sensitivity_and_specificity
Monday, February 24, 2014
C++ thread-safe queue, two ways to implement
I just explored a little on this topic, how to implement thread safe queue.
The most standard method to introduce mutex and condition variable.
Take the problem of "to implement a thread-safe queue" as an example.
The idea is to lock the critical resource before operation, and release lock after. There are some terminologies, like "own the lock" means lock(); and "release" means unlock();
So, for the thread-safe queue, two operations need to implement are:
pop()
{
mutex.lock();
while(queue.empty())
{
condition_v.wait(mutex);
}
queue.pop();
mutex.unlock();
}
push(T &item)
{
mutex.lock();
queue.push(item);
mutex.unlock();
}
This is just pseudo-code above, for implementation, there are two different ways.
1. using pthread stuff. That is "piosix" API, which is a set of API in C style. For more details, please refer to:
The code for safe queue can be found here:
One thing worth to care about is "pthread_cond_wait(cond_v&, mutex&)" function will pass under the condition of "cond_v.signal() AND mutex.lock()", which means you must have cond_v be satisfied and own the lock. And at the begining of wait(...), it release the mutex. This can make sure other thread can access the critical resource.
2. another way to implement this idea is to use new feature of C++ 11.
A very good example is:
Where mutex.lock() can block in scope of block. I like this style, but really don't like that only lock(), but no unlock() operation.
They should show up as a pair naturally.
Friday, January 17, 2014
Cannot open any webpage when update to Windows 8.1
After update to Win 8.1 from Win 8. None of browser works, while other app can access to public network. This is a bug of this update.
The solution is very easy:
1. Win key + "f" to search "everywhere", you will see "command prompt ". Right click "run as administrator". 2. In command line, type in "netsh winsock reset", return.
3. Actually you have done everything, although it is said "you need to restart you PC". But you really don't need, or only need to restart your browser.
The solution is very easy:
1. Win key + "f" to search "everywhere", you will see "command prompt ". Right click "run as administrator". 2. In command line, type in "netsh winsock reset", return.
3. Actually you have done everything, although it is said "you need to restart you PC". But you really don't need, or only need to restart your browser.
Monday, November 25, 2013
Begin scikit-learn on python
As far as I know, this python tool-kit maybe the most widely used machine learning lib. Let start from installing it.
All instructions are listed on:
All packages, installers can be found here:
http://www.lfd.uci.edu/~gohlke/pythonlibs/#scikit-learn
I am now working on windows 8, 64 bit. So the sequence of installation is like this:
1. numpy-MKL, which is a package for numerical computation with python.
2. scipy, which is another package for science computation with python, depends on numpy-MKL. And Matplotlib.
3. six->Python-Dateutil->pytz->Pyparsing->(pillow->pycairo->Tornado->Pyside->pyqt), the libs enbraced are optionally required.
4. scikit-learn
Done!!
If you wanna test this new tool, need to install another package:
https://nose.readthedocs.org/en/latest/
download the tar.gz file, release, and type in "python setup.py install"
A Chinese webpage to summarize some open-source lib of machine learning:
http://blog.csdn.net/h349117102/article/details/15029777
I realize the better way to install those libraries is to use the "easy-install" which is a tool of python, as usual located in %PythonDir%/script/. This tool allows you to install any lib using "easy-install lib_name", so easy that really is worthy its name.
Another alternative is to install some pre-build distribution. I tested the "Pythonxy". Note that you need to restart your command prompt if you want those system variables in effect.
https://code.google.com/p/pythonxy/
All instructions are listed on:
All packages, installers can be found here:
http://www.lfd.uci.edu/~gohlke/pythonlibs/#scikit-learn
I am now working on windows 8, 64 bit. So the sequence of installation is like this:
1. numpy-MKL, which is a package for numerical computation with python.
2. scipy, which is another package for science computation with python, depends on numpy-MKL. And Matplotlib.
3. six->Python-Dateutil->pytz->Pyparsing->(pillow->pycairo->Tornado->Pyside->pyqt), the libs enbraced are optionally required.
4. scikit-learn
Done!!
If you wanna test this new tool, need to install another package:
https://nose.readthedocs.org/en/latest/
download the tar.gz file, release, and type in "python setup.py install"
A Chinese webpage to summarize some open-source lib of machine learning:
http://blog.csdn.net/h349117102/article/details/15029777
I realize the better way to install those libraries is to use the "easy-install" which is a tool of python, as usual located in %PythonDir%/script/. This tool allows you to install any lib using "easy-install lib_name", so easy that really is worthy its name.
Another alternative is to install some pre-build distribution. I tested the "Pythonxy". Note that you need to restart your command prompt if you want those system variables in effect.
https://code.google.com/p/pythonxy/
Tuesday, October 8, 2013
Compile SPAMS (SPArse Modeling Software) - Matlab mixed with C/C++
When you need to compile a 3rd-part open source code, as usual they will provide Matlab interface, but to compile source code(c/c++/Fortran) is the first step, and pretty annoying!!
This article lists some helpful tips, especially for those working on SPAMS (SPArse Modeling Software).
This post is a good summary:
http://www.mathworks.com/support/solutions/en/data/1-6IJJ3L/
This is my case:
"When using 64-bit MATLAB on 64-bit Windows, you must use a 64-bit compiler to build MEX-files, MATLAB Compiler & Builder components,..."
And very unfortunately,
"The default installation of Visual Studio 2008 Express is only capable of building 32-bit binaries, and will not work with MATLAB.
In order to build 64-bit binaries, the "x64 Compilers and Tools" and Microsoft Windows Software Development Kit (SDK) must both be installed. The x64 Compilers and Tools are not installed by default."
The solution is,
"To install Visual Studio 2008 Express Edition with all required components:
1...
2...
3...
"
http://www.mathworks.com/support/solutions/en/data/1-6IJJ3L/
http://stackoverflow.com/questions/3376198/configuring-64-bit-compilation-inside-visual-studio-2008-express-edition-vs2008
http://msdn.microsoft.com/en-us/library/9yb4317s.aspx
http://pixinsight.com/forum/index.php?topic=1902.0
http://software.intel.com/en-us/articles/configuring-microsoft-visual-studio-for-64-bit-applications/
Unfortunately, when you try to compile SPAMS, it will give you some error messages:
compilation of: -I./linalg/ -I./decomp/ -I./dictLearn/ dictLearn/mex/mexTrainDL.cpp
Warning: MEX could not find the library "acml"
specified with -l option on the path specified
with the -L option.
cl : Command line warning D9035 : option 'O' has been deprecated and will be removed in a future release
Creating library C:\USERS\JFENG\APPDATA\LOCAL\TEMP\MEX_KB~1\templib.x and object C:\USERS\JFENG\APPDATA\LOCAL\TEMP\MEX_KB~1\templib.exp
mexTrainDL.obj : error LNK2019: unresolved external symbol dcopy referenced in function "void __cdecl cblas_copy<double>(__int64,double *,__int64,double *,__int64)" (??$cblas_copy@N@@YAX_JPEAN010@Z)
mexTrainDL.obj : error LNK2019: unresolved external symbol daxpy referenced in function "void __cdecl cblas_axpy<double>(__int64,double,double *,__int64,double *,__int64)" (??$cblas_axpy@N@@YAX_JNPEAN010@Z)
mexTrainDL.obj : error LNK2019: unresolved external symbol dgemv referenced in function "void __cdecl cblas_gemv<double>(enum CBLAS_ORDER,enum CBLAS_TRANSPOSE,__int64,__int64,double,double *,__int64,double *,__int64,double,double *,__int64)" (??$cblas_gemv@N@@YAXW4CBLAS_ORDER@@W4CBLAS_TRANSPOSE@@_J2NPEAN232N32@Z)
...
This is because your MATLAB cannot find CBLAS lib. Or I guess the compile.m provided by SPAMS contains a small bug.
Solution:
If your cblas lib is "builtin" then modify the line 98 in compile.m
original line 98: blas_link='-lmwblas -lmwlapack';
modified line 98: blas_link=sprintf(' -L%s -L/usr/lib/ -lmwblas -lmwlapack',path_to_blas);
Where your "path_to_blas" should be specified in a previous line, e.g. line 102 for me.
path_to_blas='%MATLAB_ROOT%\extern\lib\win64\microsoft'; % you need to tell the compiler where is your BLAS lib.
This article lists some helpful tips, especially for those working on SPAMS (SPArse Modeling Software).
This post is a good summary:
http://www.mathworks.com/support/solutions/en/data/1-6IJJ3L/
This is my case:
"When using 64-bit MATLAB on 64-bit Windows, you must use a 64-bit compiler to build MEX-files, MATLAB Compiler & Builder components,..."
And very unfortunately,
"The default installation of Visual Studio 2008 Express is only capable of building 32-bit binaries, and will not work with MATLAB.
In order to build 64-bit binaries, the "x64 Compilers and Tools" and Microsoft Windows Software Development Kit (SDK) must both be installed. The x64 Compilers and Tools are not installed by default."
The solution is,
"To install Visual Studio 2008 Express Edition with all required components:
1...
2...
3...
"
http://www.mathworks.com/support/solutions/en/data/1-6IJJ3L/
http://stackoverflow.com/questions/3376198/configuring-64-bit-compilation-inside-visual-studio-2008-express-edition-vs2008
http://msdn.microsoft.com/en-us/library/9yb4317s.aspx
http://pixinsight.com/forum/index.php?topic=1902.0
http://software.intel.com/en-us/articles/configuring-microsoft-visual-studio-for-64-bit-applications/
Unfortunately, when you try to compile SPAMS, it will give you some error messages:
compilation of: -I./linalg/ -I./decomp/ -I./dictLearn/ dictLearn/mex/mexTrainDL.cpp
Warning: MEX could not find the library "acml"
specified with -l option on the path specified
with the -L option.
cl : Command line warning D9035 : option 'O' has been deprecated and will be removed in a future release
Creating library C:\USERS\JFENG\APPDATA\LOCAL\TEMP\MEX_KB~1\templib.x and object C:\USERS\JFENG\APPDATA\LOCAL\TEMP\MEX_KB~1\templib.exp
mexTrainDL.obj : error LNK2019: unresolved external symbol dcopy referenced in function "void __cdecl cblas_copy<double>(__int64,double *,__int64,double *,__int64)" (??$cblas_copy@N@@YAX_JPEAN010@Z)
mexTrainDL.obj : error LNK2019: unresolved external symbol daxpy referenced in function "void __cdecl cblas_axpy<double>(__int64,double,double *,__int64,double *,__int64)" (??$cblas_axpy@N@@YAX_JNPEAN010@Z)
mexTrainDL.obj : error LNK2019: unresolved external symbol dgemv referenced in function "void __cdecl cblas_gemv<double>(enum CBLAS_ORDER,enum CBLAS_TRANSPOSE,__int64,__int64,double,double *,__int64,double *,__int64,double,double *,__int64)" (??$cblas_gemv@N@@YAXW4CBLAS_ORDER@@W4CBLAS_TRANSPOSE@@_J2NPEAN232N32@Z)
...
This is because your MATLAB cannot find CBLAS lib. Or I guess the compile.m provided by SPAMS contains a small bug.
Solution:
If your cblas lib is "builtin" then modify the line 98 in compile.m
original line 98: blas_link='-lmwblas -lmwlapack';
modified line 98: blas_link=sprintf(' -L%s -L/usr/lib/ -lmwblas -lmwlapack',path_to_blas);
Where your "path_to_blas" should be specified in a previous line, e.g. line 102 for me.
path_to_blas='%MATLAB_ROOT%\extern\lib\win64\microsoft'; % you need to tell the compiler where is your BLAS lib.
Thursday, September 5, 2013
打包以及UML工具
写java最后打安装包的工具，izpack：http://izpack.org/
UML的免费工具：http://www.visual-paradigm.com/solution/freeumltool/ 版本比较老（05'）
cassandra database: http://cassandra.apache.org/
UML的免费工具：http://www.visual-paradigm.com/solution/freeumltool/ 版本比较老（05'）
cassandra database: http://cassandra.apache.org/
Subscribe to:
Posts (Atom)