Novosibirsk State University Journal of Information Technologies
Scientic Journal

ISSN 2410-0420 (Online), ISSN 1818-7900 (Print)

Switch to

All Issues >> Contents: Volume 11, Issue No 4 (2013)

Detecting near-duplicates for automatically forming thematical text collections on the basis of web documents
Yury Alekseyevich Zagorulko, Natalya Vasilyevna Salomatina, Aleksei Sergeyevich Sery, Elena Anatolyevna Sidorova, Vladimir Konstantinovich Shestakov

Approaches to detecting near-duplicates appearing in thematic text collections accumulated automatically on the basis of text documents obtained from the Internet are discussed. The paper is focused on various modifications of shingle algorithm since it allows comparing a large number of texts quickly and without any preprocessing. The latter is particularly important when forming collections of raw texts.

Key Words
text collection, text comparing, shingle algorithm, near-duplicate, web documents, web resources

How to cite:
Zagorulko Y. A., Salomatina N. V., Sery A. S., Sidorova E. A., Shestakov V. K. Detecting near-duplicates for automatically forming thematical text collections on the basis of web documents // Vestnik NSU Series: Information Technologies. - 2013. - Volume 11, Issue No 4. - P. 59–70. - ISSN 1818-7900. (in Russian).

Full Text in Russian

Available in PDF

1. Zelenkov Yu., Segalovich I. Comparative Analysis of Near-Duplicate Detection Methods of Web Documents // Proc. of IX All-Russian Research Conference RCDL’2007. Pereslavl-Zalesskij, 2007. Vol. 1. P. 66–174.
2. Manber U. Finding Similar Files in a Large File System // Proc. USENIX WINTER Technical Conference. 1994. P. 1–10.
3. Broder A., Glassman S., Manasse M. Zweig G. Syntactic Clustering of the Web // Comput. Netw. ISDN Syst. 1997. Vol. 29. P. 1157–1166.
4. Fetterly D., Manasse M., Najor M. et al. A Large-Scale Study of the Evolution of Web Pages // ACM. 2003. P. 669–678.
5. Rabin M. Fingerprinting by Random Polynomials. Center for Research in Computing Technology. Harvard, 1981. 24 p.
6. Kołcz A., Chowdhury A. Lexicon Randomization for near-Duplicate Detection with I-Match // The Journal of Supercomputing. 2008. Vol. 45. Is. 3. P. 255–276.
7. Sochenkov I. V. Text Comparison Method for a Search and Analytical Engine // Artificial Intelligence and Decision Making. 2013. Vol. 2. P. 32–43.

Publication information
Main title Vestnik NSU Series: Information Technologies, Volume 11, Issue No 4 (2013).
Parallel title: Novosibirsk State University Journal of Information Technologies Volume 11, Issue No 4 (2013).

Key title: Vestnik Novosibirskogo gosudarstvennogo universiteta. Seriâ: Informacionnye tehnologii
Abbreviated key title: Vestn. Novosib. Gos. Univ., Ser.: Inf. Tehnol.
Variant title: Vestnik NGU. Seriâ: Informacionnye tehnologii

Year of Publication: 2013
ISSN: 1818-7900 (Print), ISSN 2410-0420 (Online)
Publisher: Novosibirsk State University Press
DSpace handle

|Home Page| |All Issues| |Information for Authors| |Journal Boards| |Ethical principles| |Editorial Policy| |Contact Information| |Old Site in Russian|
© 2006-2017, Novosibirsk State University.