Text Analysis
Overview of text analysis tools and methods.
Contact us
Guide Contributors
- Created by Melissa Mar, 2024-2025 Fondren Fellow
- For assistance, contact Sean Smith, Data Services Specialist
Overview - Term Frequency
Term frequency measures how often each word appears in a text to identify key or repeated terms. When compared across documents with techniques like tf-idf (Term Frequency-Inverse Document Frequency), term frequency can reveal what words are more common, and likely more significant, in one text compared to others.
Examples and Tutorials - Term Frequency
-
TF-IDF — Term Frequency-Inverse Document FrequencyKarabiber, Fatih. “TF-IDF - Term Frequency-Inverse Document Frequency.” Learn Data Science - Tutorials, Books, Courses, and More, www.learndatasci.com/glossary/tf-idf-term-frequency-inverse-document-frequency/.
-
A Practical Guide to TF-IDF and Term Frequency in Text AnalysisKhetan, Jyoti. “A Practical Guide to TF-IDF and Term Frequency in Text Analysis.” Medium, 14 May 2023, medium.com/@jyotikhetan2/a-practical-guide-to-tf-idf-and-term-frequency-in-text-analysis-b332c0405639.
-
Analyzing Documents with TF-IDFLavin, Matthew J. “Analyzing Documents with TF-IDF.” Programming Historian, 13 May 2019, programminghistorian.org/en/lessons/analyzing-documents-with-tfidf.
-
Analyzing Word and Document Frequency: TF-IDFRobinson, Julia Silge and David. “3 Analyzing Word and Document Frequency: Tf-IDF: Text Mining with R.” A Tidy Approach, www.tidytextmining.com/tfidf.html.
-
Term Frequency and Inverse Document Frequency (TF-IDF) Using Tidy Data PrinciplesRobinson, Julia Silge and David. Term Frequency and Inverse Document Frequency (TF-IDF) Using Tidy Data Principles, 10 Apr. 2024, cran.r-project.org/web/packages/tidytext/vignettes/tf_idf.html.
Term Frequency Tools
-
Voyant ToolsA browser-based text analysis environment for exploring texts through visualizations and frequency tools.
-
Google Books Ngram ViewerOnline tool that charts the frequency of words or phrases across centuries using the Google Books corpus.
-
Bookworm HTRCOnline tool that searches and visualizes word trends in the HathiTrust Digital Library.
-
NLTK (Natural Language Toolkit)A comprehensive suite of Python libraries for text processing, including classification, tokenization, and parsing.
-
spaCyAn industrial-strength NLP library for Python that supports fast and accurate syntactic analysis.
-
TextBlobA beginner-friendly Python library for common natural language processing (NLP) tasks such as part-of-speech tagging, noun phrase extraction, sentiment analysis, classification, and more.
-
tidytextR package to manipulate and analyze text data using tidy data principles.
-
tm (Text Mining)A text mining framework for pre-processing, filtering, and analyzing large text corpora in R.
-
wordcloudA simple tool for generating word clouds based on term frequency in R.
-
zipfRR tools for statistical modeling of word frequency distributions, based on Zipf's law.
-
quantedaA fast, flexible, and comprehensive framework for quantitative text analysis in R.
Web Address: library.rice.edu | Physical Address: 6100 Main Street, Houston, Texas 77005 | Mailing Address: MS-44, P.O. Box 1892, Houston, Texas 77251-1892 | Phone: 713-348-5698 | © 2014 Rice University