文本挖掘的paper没找到统一的benchmark，只好自己跑程序，走过路过的前辈如果知道20newsgroups或者其它好用的公共数据集的分类（最好要所有类分类结果，全部或取部分特征无所谓）麻烦留言告知下现在的benchmark，万谢！
嗯，说正文。 20newsgroups官网上给出了3个数据集，这里我们用最原始的 20news-19997.tar.gz。
分为以下几个过程：
- 加载数据集
- 提feature
- 分类
- Naive Bayes
- KNN
- SVM
- 聚类
说明： scipy官网上有参考，但是看着有点乱，而且有bug。本文中我们分块来看。
Environment：Python 2.7 + Scipy （scikit-learn）
1.加载数据集
从 20news-19997.tar.gz下载数据集，解压到scikit_learn_data文件夹下，加载数据，详见code注释。
#first extract the 20 news_group dataset to /scikit_learn_data from sklearn.datasets import fetch_20newsgroups #all categories #newsgroup_train = fetch_20newsgroups(subset='train') #part categories categories = ['comp.graphics','comp.os.ms-windows.misc','comp.sys.ibm.pc.hardware','comp.sys.mac.hardware','comp.windows.x']; newsgroup_train = fetch_20newsgroups(subset = 'train',categories = categories);
可以检验是否load好了：
#print category names from pprint import pprint pprint(list(newsgroup_train.target_names))
结果：
['comp.graphics',
'comp.os.ms-windows.misc',
'comp.sys.ibm.pc.hardware',
'comp.sys.mac.hardware',
'comp.windows.x']
'comp.os.ms-windows.misc',
'comp.sys.ibm.pc.hardware',
'comp.sys.mac.hardware',
'comp.windows.x']
2. 提feature：
刚才load进来的newsgroup_train就是一篇篇document，我们要从中提取feature，即词频啊神马的，用fit_transform
#newsgroup_train.data is the original documents, but we need to extract the
#TF-IDF vectors inorder to model the text data
from sklearn.feature_extraction.text import TfidfVectorizer, HashingVectorizer
#vectorizer = TfidfVectorizer(sublinear_tf = True,
# max_df = 0.5,
# stop_words = 'english');
#however, Tf-Idf feather extractor makes the training set and testing set have
#divergent number of features. (Because they have different vocabulary in documents)
#So we use HashingVectorizer
vectorizer = HashingVectorizer(stop_words = 'english',non_negative = True,
n_features = 100)
fea_train = vectorizer.fit_transform(newsgroup_train.data)
#return feature vector 'fea_train' [n_samples,n_features]
print 'Size of fea_train:' + repr(fea_train.shape)
#11314 documents, 130107 vectors for all categories
print 'The average feature sparsity is {0:.3f}%'.format(
fea_train.nnz/float(fea_train.shape[0]*fea_train.shape[1])*100);结果：
Size of fea_train:(2936, 100)
The average feature sparsity is 51.183%
The average feature sparsity is 51.183%
因为我们只取了100个词，即100维feature，稀疏度还不算低。而实际上用TfidfVectorizer统计可得到上万维的feature，我统计的全部样本是13w多维，就是一个相当稀疏的矩阵了。
3. 分类
3.1 Multinomial Naive Bayes Classifier
见代码&comment，不解释
######################################################
#Multinomial Naive Bayes Classifier
print '*************************\nNaive Bayes\n*************************'
from sklearn.naive_bayes import MultinomialNB
from sklearn import metrics
newsgroups_test = fetch_20newsgroups(subset = 'test',
categories = categories);
fea_test = vectorizer.fit_transform(newsgroups_test.data);
#create the Multinomial Naive Bayesian Classifier
clf = MultinomialNB(alpha = 0.01)
clf.fit(fea_train,newsgroup_train.target);
pred = clf.predict(fea_test);
calculate_result(newsgroups_test.target,pred);
#notice here we can see that f1_score is not equal to 2*precision*recall/(precision+recall)
#because the m_precision and m_recall we get is averaged, however, metrics.f1_score() calculates
#weithed average, i.e., takes into the number of each class into consideration.注意我最后的3行注释，为什么f1≠2*（准确率*召回率）/（准确率+召回率）
其中，函数calculate_result计算f1：
def calculate_result(actual,pred):
m_precision = metrics.precision_score(actual,pred);
m_recall = metrics.recall_score(actual,pred);
print 'predict info:'
print 'precision:{0:.3f}'.format(m_precision)
print 'recall:{0:0.3f}'.format(m_recall);
print 'f1-score:{0:.3f}'.format(metrics.f1_score(actual,pred));3.2 KNN：
###################################################### #KNN Classifier from sklearn.neighbors import KNeighborsClassifier print '*************************\nKNN\n*************************' knnclf = KNeighborsClassifier()#default with k=5 knnclf.fit(fea_train,newsgroup_train.target) pred = knnclf.predict(fea_test); calculate_result(newsgroups_test.target,pred);
3.3 SVM:
###################################################### #SVM Classifier from sklearn.svm import SVC print '*************************\nSVM\n*************************' svclf = SVC(kernel = 'linear')#default with 'rbf' svclf.fit(fea_train,newsgroup_train.target) pred = svclf.predict(fea_test); calculate_result(newsgroups_test.target,pred);
*************************
Naive Bayes
*************************
predict info:
precision:0.448
recall:0.448
f1-score:0.447
*************************
KNN
*************************
predict info:
precision:0.415
recall:0.405
f1-score:0.406
*************************
SVM
*************************
predict info:
precision:0.440
recall:0.438
f1-score:0.438
4. 聚类
###################################################### #KMeans Cluster from sklearn.cluster import KMeans print '*************************\nKMeans\n*************************' pred = KMeans(n_clusters=5) pred.fit(fea_test) calculate_result(newsgroups_test.target,pred.labels_);
结果：
*************************
KMeans
*************************
predict info:
precision:0.177
recall:0.176
f1-score:0.171
本文全部代码下载： 在此
关于Python更多的学习资料将继续更新，敬请关注本博客和新浪微博 Rachel Zhang。
作者：abcjennifer 发表于2014-4-13 20:53:15 原文链接
阅读：306 评论：0 查看评论
