Counting Word Frequency in Python

Published Mar 17, 2017
Counting Word Frequency in Python

Word Frequency

Counting Words

Word frequency is word counting technique in which a sorted list of words and their frequency is generated, where the frequency is the occurrences in a given composition. It is used commonly in computational linguistics.

Why Should I Care?

Word frequency has many applications in diverse fields. Within pedagogy, it allows teaching to cover high-frequency vocabulary before the low-frequency ones, enabling the development of better class curriculum.

But let's talk about a field that any developer might find compelling, which are analytics. In my case, I worked for a publisher that required analytics to suggest vocabulary improvements, detect trends in word usage, and determine payment for their writers.

Yes, you read that last part correctly — payment is determined by many publishers based on the number and complexity of the wording.

This tutorial will show you how to create a list of word frequency with a real life example.

Quick Notes

  1. Python 3.5 is used in this tutorial.
  2. This code requires a text file named The Beast.txt, which must be in the same directory as the code to work.
  3. Even though there are many libraries and tools available, this tutorial will only use standard Python.

Installing Python

Before we start, we will need Python 3. If you are using Linux, Python 2 will probably be there already since major distributions incorporate it. Mac OS usually comes with Python 2 included.

Example Input

The first thing we need is a document to perform the word frequency on, for this example I will use The Beast in the Cave, a short story by American horror fiction writer H.P. Lovecraft — more about the plot here. We will save the story (below) as a text file named The Beast.txt.

The Beast in the Cave:

The horrible conclusion which had been gradually intruding itself upon my confused and reluctant mind was now an awful certainty. I was lost, completely, hopelessly lost in the vast and labyrinthine recess of the Mammoth Cave. Turn as I might, in no direction could my straining vision seize on any object capable of serving as a guidepost to set me on the outward path. That nevermore should I behold the blessed light of day, or scan the pleasant bills and dales of the beautiful world outside, my reason could no longer entertain the slightest unbelief. Hope had departed. Yet, indoctrinated as I was by a life of philosophical study, I derived no small measure of satisfaction from my unimpassioned demeanour; for although I had frequently read of the wild frenzies into which were thrown the victims of similar situations, I experienced none of these, but stood quiet as soon as I clearly realised the loss of my bearings.

Nor did the thought that I had probably wandered beyond the utmost limits of an ordinary search cause me to abandon my composure even for a moment. If I must die, I reflected, then was this terrible yet majestic cavern as welcome a sepulchre as that which any churchyard might afford, a conception which carried with it more of tranquillity than of despair.

Starving would prove my ultimate fate; of this I was certain. Some, I knew, had gone mad under circumstances such as these, but I felt that this end would not be mine. My disaster was the result of no fault save my own, since unknown to the guide I had separated myself from the regular party of sightseers; and, wandering for over an hour in forbidden avenues of the cave, had found myself unable to retrace the devious windings which I had pursued since forsaking my companions.

Already my torch had begun to expire; soon I would be enveloped by the total and almost palpable blackness of the bowels of the earth. As I stood in the waning, unsteady light, I idly wondered over the exact circumstances of my coming end. I remembered the accounts which I had heard of the colony of consumptives, who, taking their residence in this gigantic grotto to find health from the apparently salubrious air of the underground world, with its steady, uniform temperature, pure air, and peaceful quiet, had found, instead, death in strange and ghastly form. I had seen the sad remains of their ill-made cottages as I passed them by with the party, and had wondered what unnatural influence a long sojourn in this immense and silent cavern would exert upon one as healthy and vigorous as I. Now, I grimly told myself, my opportunity for settling this point had arrived, provided that want of food should not bring me too speedy a departure from this life.

As the last fitful rays of my torch faded into obscurity, I resolved to leave no stone unturned, no possible means of escape neglected; so, summoning all the powers possessed by my lungs, I set up a series of loud shoutings, in the vain hope of attracting the attention of the guide by my clamour. Yet, as I called, I believed in my heart that my cries were to no purpose, and that my voice, magnified and reflected by the numberless ramparts of the black maze about me, fell upon no ears save my own.

All at once, however, my attention was fixed with a start as I fancied that I heard the sound of soft approaching steps on the rocky floor of the cavern.

Was my deliverance about to be accomplished so soon? Had, then, all my horrible apprehensions been for naught, and was the guide, having marked my unwarranted absence from the party, following my course and seeking me out in this limestone labyrinth? Whilst these joyful queries arose in my brain, I was on the point of renewing my cries, in order that my discovery might come the sooner, when in an instant my delight was turned to horror as I listened; for my ever acute ear, now sharpened in even greater degree by the complete silence of the cave, bore to my benumbed understanding the unexpected and dreadful knowledge that these footfalls were not like those of any mortal man. In the unearthly stillness of this subterranean region, the tread of the booted guide would have sounded like a series of sharp and incisive blows. These impacts were soft, and stealthy, as of the paws of some feline. Besides, when I listened carefully, I seemed to trace the falls of four instead of two feet.

I was now convinced that I had by my own cries aroused and attracted some wild beast, perhaps a mountain lion which had accidentally strayed within the cave. Perhaps, I considered, the Almighty had chosen for me a swifter and more merciful death than that of hunger; yet the instinct of self-preservation, never wholly dormant, was stirred in my breast, and though escape from the on-coming peril might but spare me for a sterner and more lingering end, I determined nevertheless to part with my life at as high a price as I could command. Strange as it may seem, my mind conceived of no intent on the part of the visitor save that of hostility. Accordingly, I became very quiet, in the hope that the unknown beast would, in the absence of a guiding sound, lose its direction as had I, and thus pass me by. But this hope was not destined for realisation, for the strange footfalls steadily advanced, the animal evidently having obtained my scent, which in an atmosphere so absolutely free from all distracting influences as is that of the cave, could doubtless be followed at great distance.

Seeing therefore that I must be armed for defense against an uncanny and unseen attack in the dark, I groped about me the largest of the fragments of rock which were strewn upon all parts of the floor of the cavern in the vicinity, and grasping one in each hand for immediate use, awaited with resignation the inevitable result. Meanwhile the hideous pattering of the paws drew near. Certainly, the conduct of the creature was exceedingly strange. Most of the time, the tread seemed to be that of a quadruped, walking with a singular lack of unison betwixt hind and fore feet, yet at brief and infrequent intervals I fancied that but two feet were engaged in the process of locomotion. I wondered what species of animal was to confront me; it must, I thought, be some unfortunate beast who had paid for its curiosity to investigate one of the entrances of the fearful grotto with a life-long confinement in its interminable recesses. It doubtless obtained as food the eyeless fish, bats and rats of the cave, as well as some of the ordinary fish that are wafted in at every freshet of Green River, which communicates in some occult manner with the waters of the cave. I occupied my terrible vigil with grotesque conjectures of what alteration cave life might have wrought in the physical structure of the beast, remembering the awful appearances ascribed by local tradition to the consumptives who had died after long residence in the cave. Then I remembered with a start that, even should I succeed in felling my antagonist, I should never behold its form, as my torch had long since been extinct, and I was entirely unprovided with matches. The tension on my brain now became frightful. My disordered fancy conjured up hideous and fearsome shapes from the sinister darkness that surrounded me, and that actually seemed to press upon my body. Nearer, nearer, the dreadful footfalls approached. It seemed that I must give vent to a piercing scream, yet had I been sufficiently irresolute to attempt such a thing, my voice could scarce have responded. I was petrified, rooted to the spot. I doubted if my right arm would allow me to hurl its missile at the oncoming thing when the crucial moment should arrive. Now the steady pat, pat, of the steps was close at hand; now very close. I could hear the laboured breathing of the animal, and terror-struck as I was, I realised that it must have come from a considerable distance, and was correspondingly fatigued. Suddenly the spell broke. My right hand, guided by my ever trustworthy sense of hearing, threw with full force the sharp-angled bit of limestone which it contained, toward that point in the darkness from which emanated the breathing and pattering, and, wonderful to relate, it nearly reached its goal, for I heard the thing jump, landing at a distance away, where it seemed to pause.

Having readjusted my aim, I discharged my second missile, this time most effectively, for with a flood of joy I listened as the creature fell in what sounded like a complete collapse and evidently remained prone and unmoving. Almost overpowered by the great relief which rushed over me, I reeled back against the wall. The breathing continued, in heavy, gasping inhalations and expirations, whence I realised that I had no more than wounded the creature. And now all desire to examine the thing ceased. At last something allied to groundless, superstitious fear had entered my brain, and I did not approach the body, nor did I continue to cast stones at it in order to complete the extinction of its life. Instead, I ran at full speed in what was, as nearly as I could estimate in my frenzied condition, the direction from which I had come. Suddenly I heard a sound or rather, a regular succession of sounds. In another Instant they had resolved themselves into a series of sharp, metallic clicks. This time there was no doubt. It was the guide. And then I shouted, yelled, screamed, even shrieked with joy as I beheld in the vaulted arches above the faint and glimmering effulgence which I knew to be the reflected light of an approaching torch. I ran to meet the flare, and before I could completely understand what had occurred, was lying upon the ground at the feet of the guide, embracing his boots and gibbering. despite my boasted reserve, in a most meaningless and idiotic manner, pouring out my terrible story, and at the same time overwhelming my auditor with protestations of gratitude. At length, I awoke to something like my normal consciousness. The guide had noted my absence upon the arrival of the party at the entrance of the cave, and had, from his own intuitive sense of direction, proceeded to make a thorough canvass of by-passages just ahead of where he had last spoken to me, locating my whereabouts after a quest of about four hours.

By the time he had related this to me, I, emboldened by his torch and his company, began to reflect upon the strange beast which I had wounded but a short distance back in the darkness, and suggested that we ascertain, by the flashlight's aid, what manner of creature was my victim. Accordingly I retraced my steps, this time with a courage born of companionship, to the scene of my terrible experience. Soon we descried a white object upon the floor, an object whiter even than the gleaming limestone itself. Cautiously advancing, we gave vent to a simultaneous ejaculation of wonderment, for of all the unnatural monsters either of us had in our lifetimes beheld, this was in surpassing degree the strangest. It appeared to be an anthropoid ape of large proportions, escaped, perhaps, from some itinerant menagerie. Its hair was snow-white, a thing due no doubt to the bleaching action of a long existence within the inky confines of the cave, but it was also surprisingly thin, being indeed largely absent save on the head, where it was of such length and abundance that it fell over the shoulders in considerable profusion. The face was turned away from us, as the creature lay almost directly upon it. The inclination of the limbs was very singular, explaining, however, the alternation in their use which I bad before noted, whereby the beast used sometimes all four, and on other occasions but two for its progress. From the tips of the fingers or toes, long rat-like claws extended. The hands or feet were not prehensile, a fact that I ascribed to that long residence in the cave which, as I before mentioned, seemed evident from the all-pervading and almost unearthly whiteness so characteristic of the whole anatomy. No tail seemed to be present.

The respiration had now grown very feeble, and the guide had drawn his pistol with the evident intent of despatching the creature, when a sudden sound emitted by the latter caused the weapon to fall unused. The sound was of a nature difficult to describe. It was not like the normal note of any known species of simian, and I wonder if this unnatural quality were not the result of a long continued and complete silence, broken by the sensations produced by the advent of the light, a thing which the beast could not have seen since its first entrance into the cave. The sound, which I might feebly attempt to classify as a kind of deep-tone chattering, was faintly continued.

All at once a fleeting spasm of energy seemed to pass through the frame of the beast. The paws went through a convulsive motion, and the limbs contracted. With a jerk, the white body rolled over so that its face was turned in our direction. For a moment I was so struck with horror at the eyes thus revealed that I noted nothing else. They were black, those eyes, deep jetty black, in hideous contrast to the snow-white hair and flesh. Like those of other cave denizens, they were deeply sunken in their orbits, and were entirely destitute of iris. As I looked more closely, I saw that they were set in a face less prognathous than that of the average ape, and infinitely less hairy. The nose was quite distinct. As we gazed upon the uncanny sight presented to our vision, the thick lips opened, and several sounds issued from them, after which the thing relaxed in death.

The guide clutched my coat sleeve and trembled so violently that the light shook fitfully, casting weird moving shadows on the walls.

I made no motion, but stood rigidly still, my horrified eyes fixed upon the floor ahead.

The fear left, and wonder, awe, compassion, and reverence succeeded in its place, for the sounds uttered by the stricken figure that lay stretched out on the limestone had told us the awesome truth. The creature I had killed, the strange beast of the unfathomed cave, was, or had at one time been a MAN!!!

The "Naive" Approach

In the first approach, we can read the complete text, tokenize each word, then count every single appearance of the desired word.

file = open('The Beast.txt', 'r')
book =

def tokenize():
    if book is not None:
        words = book.lower().split()
        return words
        return None

def count_word(tokens, token):
    count = 0

    for element in tokens:
        # Remove Punctuation
        word = element.replace(",","")
        word = word.replace(".","")

        # Found Word?
        if word == token:
            count += 1
    return count
# Tokenize the Book
words = tokenize()

# Get Word Count
word = 'day'
frequency = count_word(words, word)
print('Word: [' + word + '] Frequency: ' + str(frequency))

This code will return the frequency, but if we intended to ask for several words, it would be inefficient. To do so, we need a better approach.

The HashMap Approach

We will create a HashMap by using a Python dictionary to store the word frequencies of a book.
A dictionary is an associative array (also known as hashes).
Any key of the dictionary is associated, or mapped, to a value.
The values of a dictionary can be any Python data type, so dictionaries are unordered key-value-pairs.

By creating the dictionary, we will store the words as the keys and the value will represent the count. By doing this, we can retrieve any word without having to recount every single word.

file = open('The Beast.txt', 'r')
book =

def tokenize():
    if book is not None:
        words = book.lower().split()
        return words
        return None

def map_book(tokens):
    hash_map = {}

    if tokens is not None:
        for element in tokens:
            # Remove Punctuation
            word = element.replace(",","")
            word = word.replace(".","")

            # Word Exist?
            if word in hash_map:
                hash_map[word] = hash_map[word] + 1
                hash_map[word] = 1

        return hash_map
        return None

# Tokenize the Book
words = tokenize()
word_list = ['the','life','situations','since','day']

# Create a Hash Map (Dictionary)
map = map_book(words)

# Show Word Information
for word in word_list:
    print('Word: [' + word + '] Frequency: ' + str(map[word]))

Wrapping Things Up

The first example shows the most straightforward method to count word frequency, in contrast, the second uses HashMap to optimize the word counting. We can go beyond this and optimize the code to create the dictionary while reading the input — you might want to try that too!

Whenever we are confronted with having to count the frequency of anything, not only words, we must consider the use of HashMaps as a way to make the processing only once and access the result as needed to achieve our goal.

Things to Consider

Now that you know how to get word frequencies in Python, how about applying them to your favorite story or document?

Did you notice the extra step to tokenize in the complete source code? We can create the dictionary on a single method that reads the text and adds the tokens to the map.

What will happen if the source is too big to fit in memory? How about line by line tokenizing?

The code has a part in which we remove punctuation, but it only handles commas and dots. What additional punctuation should we consider?

Play around and check what happens! Test the code in a big book — the idea is to learn new things. If you like this tutorial, be sure to share it with your friends.

Discover and read more posts from Isai B. Cicourel
get started
Enjoy this post?

Leave a like and comment for Isai

Martin Graham
a month ago

Thank for the post…really helpful! I track what you’re doing with the “Hash Map” approach and it makes perfect sense. However, what confuses me is at what point do the words and their frequencies get added to the “hash_map” dictionary? You said “if word in hash_map”, but “hash_map” is empty. I ran this code on my local machine and it works perfectly. My question is why?? How do the words and the counts get added to the dictionary? It would seem, based on how it’s written, that “if word in hash_map” would always return “false” b/c the dictionary is empty. I don’t understand. Can you help explain this? Thank you!

Isai B. Cicourel
a month ago

Hi Martin,

When the line “if word in hash_map:” is executed, if the word is in the hash map then the value will be increased by one in this line “hash_map[word] = hash_map[word] + 1”.
On the other hands it there is no key (our word) in the hash map then we get into the else case in which a new key (our word) is added to the hash map as following “hash_map[word] = 1” and its value will be 1.
Hope this helps clarify.

Let me know if you need any extra help,

Get curated posts in your inbox

Read more posts to become a better developer