[Data Mining] ROC Curve

Receiver operating characteristic

https://en.wikipedia.org/wiki/Receiver_operating_characteristic

ROC : is for evaluating the binary classification models. 

EX :  in the Nearest neighbor search algo, when we difine different distance for the neighbors : distance < 50 is my neighbor VS distance<100 is my neighbor, we need ROC to tell which choice is better. 

How to make a ROC curve? 

for example, I have defined some rules for the neighbor and we have the result of 20 observations : 

individu	score	classe
1	1	+
2	0.95	+
3	0.9	+
4	0.85	-
5	0.8	+
6	0.75	-
7	0.7	-
8	0.65	+
9	0.6	-
10	0.55	-
11	0.5	-
12	0.45	+
13	0.4	-
14	0.35	-
15	0.3	-
16	0.25	-
17	0.2	-
18	0.15	-
19	0.1	-
20	0.05	-

we will calcule 20 points : 

1 : we predict individu<=1  is +, the others are - : 

		reality
		+	-
predictions	+	1	0
	-	5	14
total		6	14

so we get TPR = TP/P = 1/6 = 0.166; FPR = FP/F = 0/14 = 0

 --> the first point(0.166,0)

2 : we predict individu<=2  is +, the others are - : 

		reality
		+	-
predictions	+	2	0
	-	4	14
total		6	14

so we get TPR = TP/P = 2/6 = 0.333; FPR = FP/F = 0/14 = 0

 --> the second point(0.333,0)

......

15 : we predict individu<=2  is +, the others are - : 

		reality
		+	-
predictions	+	6	9
	-	0	5
total		6	14

so we get TPR = TP/P = 1; FPR = FP/F = 9/14 = 0.643

 --> the fifth point(1,0.643)

......

so we can have a curve like : 

the line x=y (0.5 ) means we cannot make decision with this prediction.

so , the surface bigger the better the model is. 

the surface names : AUC

if we have several model to compare : we should choose the one with biggest AUC.