Bingyan Liu

Distributional word similarity on web scale Chinese web collection is an important and widespread
used preprocess phase in many NLP tasks. On English web collections vector-based models are widely used to measure word similarity out of context. This paper proposed an unsupervised Chinese word similarity measuring system based on context vector. We used the system measuring Chinese word similarity on web scale dataset and verified the Harris’ distribution hypothesis on Chinese corpus. The… Show more

Distributional word similarity on web scale Chinese web collection is an important and widespread
used preprocess phase in many NLP tasks. On English web collections vector-based models are widely used to measure word similarity out of context. This paper proposed an unsupervised Chinese word similarity measuring system based on context vector. We used the system measuring Chinese word similarity on web scale dataset and verified the Harris’ distribution hypothesis on Chinese corpus. The system shows high performance and well effect. In experiment this paper compared impact of each model’s parameters on final results. Show less

Chinese word segmentation is a fundamental problem in the area of natural language processing
problem. With the growth of Internet, Chinese word segmentation faces greater challenge of bigger data scale. The design of the word segmentation system requires trade-offs between word segmentation speed and precision rate. Current research mostly concentrates on improving precision rate using complex word segmentation model. This paper proposes a new Chinese word segmentation system to improve the… Show more

Chinese word segmentation is a fundamental problem in the area of natural language processing
problem. With the growth of Internet, Chinese word segmentation faces greater challenge of bigger data scale. The design of the word segmentation system requires trade-offs between word segmentation speed and precision rate. Current research mostly concentrates on improving precision rate using complex word segmentation model. This paper proposes a new Chinese word segmentation system to improve the system’s data through-put and speed. The system is capable of processing of web scale Chinese webpage dataset quickly. Experiment shows current system exploits cluster system performance very well and yields a performance speed-up to 136 times compared to current single-thread baseline which is capable of processing web scale data efficiently. Show less