iv值计算（含qcut细节）
1 背景2 含有重复的数据2.1 数据准备2.2 等频分组2.3 等频分组-加上去掉重复的值
3 不含有重复的数据3.1 数据准备3.2 等频分组3.3 等频分组-加上去掉重复值
4 iv计算4.1 读入数据4.2 iv值计算4.3 结果分析
1 背景
在计算woe以及相关的iv值的时候，需要首先对数据进行分箱，分箱一般采用qcut，即等频分组。
下面希望验证qcut（等频分组）-相同的值会在一组，即如果一组数据一半都是0，这些会被分在一组。
同时计算iv值并进行相关分析
2 含有重复的数据
2.1 数据准备
import pandas
as pd
a
= [0]*50 + list(range(0,50))
print(len(a
))
a
[:5]
100
[0, 0, 0, 0, 0]
df
= pd
.DataFrame
({'a':a
})
print(df
.shape
)
df
.head
()
(100, 1)
a
0010203040
2.2 等频分组
pd
.qcut
(df
['a'], 3).value_counts
()
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-17-47659b591f84> in <module>
----> 1 pd.qcut(df['a'], 3).value_counts()
~\Anaconda3\lib\site-packages\pandas\core\reshape\tile.py in qcut(x, q, labels, retbins, precision, duplicates)
339 quantiles = q
340 bins = algos.quantile(x, quantiles)
--> 341 fac, bins = _bins_to_cuts(
342 x,
343 bins,
~\Anaconda3\lib\site-packages\pandas\core\reshape\tile.py in _bins_to_cuts(x, bins, right, labels, precision, include_lowest, dtype, duplicates)
378 if len(unique_bins) < len(bins) and len(bins) != 2:
379 if duplicates == "raise":
--> 380 raise ValueError(
381 f"Bin edges must be unique: {repr(bins)}.\n"
382 f"You can drop duplicate edges by setting the 'duplicates' kwarg"
ValueError: Bin edges must be unique: array([ 0., 0., 16., 49.]).
You can drop duplicate edges by setting the 'duplicates' kwarg
报错原因：有很多的重复值并且没有加上去掉重复值的语句！
2.3 等频分组-加上去掉重复的值
pd
.qcut
(df
['a'], 3, duplicates
='drop').value_counts
()
(-0.001, 16.0] 67
(16.0, 49.0] 33
Name: a, dtype: int64
3 不含有重复的数据
3.1 数据准备
a
= list(range(0,100))
df
= pd
.DataFrame
({'a':a
})
print(df
.shape
)
df
.head
()
(100, 1)
a
0011223344
3.2 等频分组
pd
.qcut
(df
['a'], 3).value_counts
()
(-0.001, 33.0] 34
(66.0, 99.0] 33
(33.0, 66.0] 33
Name: a, dtype: int64
3.3 等频分组-加上去掉重复值
pd
.qcut
(df
['a'], 3, duplicates
='drop').value_counts
()
(-0.001, 33.0] 34
(66.0, 99.0] 33
(33.0, 66.0] 33
Name: a, dtype: int64
4 iv计算
4.1 读入数据
import pandas
as pd
from IV_Cal
import *
df
= pd
.read_excel
('工作簿1.xlsx')
print(df
.shape
)
df
.head
()
(51000, 31)
idLoanActiveNumidLoan30dNumidLoan90dNumidLoan180dNumidLoan360dNumidOrg30dNumidOrg90dNumidOrg180dNumidOrg360dNumidOrgActiveNum...idOverdueLoanAmt60idOverdueLoanAmt90idOverdueLoanAmt120idOverdueLoanAmt150idOverdueLoanAmt180idActiveOverdueOrgNumidOverdueOrgNum30idOverdueOrgNum90idOverdueOrgNum180Y
0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0...-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.001-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0...-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.002-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0...-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.003-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0...-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.004-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0...-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.0-1.00
5 rows × 31 columns
4.2 iv值计算
iv
= IV_Cal
(df
,y_name
='Y')
iv
.cal
()
iv
.plot_iv
('./output')
iv
.save_info
('./output')
IV图生成
Try 10 bins.
Set bin label: mean.
Plotting idOverdueOrgNum180... (30/30) idOverdueLoanAmt180... (26/30)
Done.
最关键的还是看每个变量各自的iv值
df_var_iv
= pd
.read_csv
('output/iv_rank_table.csv')
df_var_iv
[:5]
var_nameiv_valueiv_rankn_of_binmissing_rate(%)n_of_uniquedtypesis_continuous
0idOverdueLoanAmt1500.657071110.810float64False1idOverdueLoanAmt1200.643382110.810float64False2idOverdueLoanAmt1800.632213110.810float64False3idOverdueLoanAmt900.625114110.810float64False4idOverdueLoanAmt600.613405110.810float64False
4.3 结果分析
df
['idOverdueLoanAmt'].value_counts
()
0.0 25484
-1.0 20030
1.0 2462
2.0 948
3.0 706
5.0 314
4.0 311
6.0 263
7.0 90
8.0 8
9.0 1
Name: idOverdueLoanAmt, dtype: int64
df
.info
()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 51000 entries, 0 to 50999
Data columns (total 31 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 idLoanActiveNum 50617 non-null float64
1 idLoan30dNum 50617 non-null float64
2 idLoan90dNum 50617 non-null float64
3 idLoan180dNum 50617 non-null float64
4 idLoan360dNum 50617 non-null float64
5 idOrg30dNum 50617 non-null float64
6 idOrg90dNum 50617 non-null float64
7 idOrg180dNum 50617 non-null float64
8 idOrg360dNum 50617 non-null float64
9 idOrgActiveNum 50617 non-null float64
10 idOverdueLoanAmt 50617 non-null float64
11 idActiveOverdueLoanBal 50617 non-null float64
12 idActiveOverdueLoanNum 50617 non-null float64
13 idOverdueLoanNum30 50617 non-null float64
14 idOverdueLoanNum60 50617 non-null float64
15 idOverdueLoanNum90 50617 non-null float64
16 idOverdueLoanNum120 50617 non-null float64
17 idOverdueLoanNum150 50617 non-null float64
18 idOverdueLoanNum180 50617 non-null float64
19 idTotalOverdueLoanNum 50617 non-null float64
20 idOverdueLoanAmt30 50617 non-null float64
21 idOverdueLoanAmt60 50617 non-null float64
22 idOverdueLoanAmt90 50617 non-null float64
23 idOverdueLoanAmt120 50617 non-null float64
24 idOverdueLoanAmt150 50617 non-null float64
25 idOverdueLoanAmt180 50617 non-null float64
26 idActiveOverdueOrgNum 50617 non-null float64
27 idOverdueOrgNum30 50617 non-null float64
28 idOverdueOrgNum90 50617 non-null float64
29 idOverdueOrgNum180 50617 non-null float64
30 Y 51000 non-null int64
dtypes: float64(30), int64(1)
memory usage: 12.1 MB
以变量idOverdueLoanAmt为例进行分析：
具体箱子怎么划分的看上面txt的结果，而需要讲故事，看该变量的iv值和bad rate是否呈线性相关，无论是正相关还是负相关！方便解释以及放入到线性模型中（比如逻辑回归）~
对应曲线为：
