Clustering Chinese Documents Based on HybridDT and LDA Model

Abstract

In traditional VSM model based text clustering approach, the abstract similarities among documents are represented by the cosine values between the corresponding space vectors. Although it is simple and intuitive, it ignores the semantic and polysemy information of the documents. Aiming to resolve these problems, a clustering framework is proposed in this paper. The framework contains two parts: (1) pre-training text by using Doc2vec to obtain document vector and combining it into TFIDF strategy based weight space vector, which is called HybridDT; (2) using Latent Dirichlet allocation to get the topic distribution and linearly mixing it as feature into HybridDT. The experimental results using the proposed framework on Fudan University Chinese corpus show that the performance of text clustering are improved significantly by combining HybridDT and LDA model.