Automating the Game Wordle Using Computer Vision

Screenshot a partially-filled Wordle and identify the letters and colors using deep learning models for the suggestion of the next word.

How I played Wordle 231 — https://www.powerlanguage.co.uk/wordle/

Introduction

HOW TO PLAY — https://www.powerlanguage.co.uk/wordle/

I started playing this new trending game called Wordle and competed with family and friends with mixed luck.

Out of academic interest, I wanted to create a program that would use a snapshot of a partially-filled Wordle to automatically play the next best word for me.

Here are the steps to create a program that will analyze the screenshot, find out the letters and the colors in each square containing a letter and then return the next best word to play.

Steps

1. Using contour detection to separate all letters used in the Wordle
2. Creating a handwritten alphabet classifier using TensorFlow Keras
3. Creating a grey-yellow-green color classifier using TensorFlow Keras
4. Combining contour detection and the two models to display words used in a partially-filled Wordle
5. Using greedy algorithm to predict the next best word for the partially-filled Wordle(future work)

Using Contour Detection to Separate All Letters Used in the Wordle

Place the screenshot of an image (jpg, jpeg, png, etc) into the same directory containing the above code.
In lines 7–15, the image is preprocessed so that the contours of each square-containing alphabet are produced.
These are the steps for preprocessing the original image:

1. Grayscaling the image.

Applying grayscale on the image

2. Sharpening the image — the kernel used sharpens all the edges found in the image, hence, it will help detect contours.

Sharpening the edges found in the Image

3. Thresholding the image — Converts all colors below the pixel value 225 to 255 and then inverts the image in line 18 — cv2.THRESH_BINARY_INV.

Applying the threshold on the image

In line 18, you detect all those contours using cv2.findContours on the thresholded image. Each contour is a NumPy array of (x, y) coordinates of boundary points of the object.
From lines 29–32, I find the coordinates and height and width — x,y,w,h = cv2.boundingRect(c).
Then I go through each contour and essentially crop the image — ROI = image[y:y+h, x:x+w] — to every contour using its coordinates(x, y) and height(h) and width(w).
The contour is then saved as an image in the directory — cv2.imwrite(‘ROI_{}.png’.format(image_number), ROI).

How each square-containing alphabet is stored in the directory

Using a Handwritten Alphabet Classifier Using TensorFlow Keras

I could go into the details on how to code a model that would classify alphabets in TensorFlow, however, I would make another post to address this topic.
To save our time, I used a pre-saved model on Kaggle.
In this case(Wordle letters’ prediction), the model is not working at its full capacity as there are no handwritten alphabets, however, for future use, this model can classify any alphabet independent of its font/style.

Screenshot of Kaggle Webpage — https://www.kaggle.com/yairhadad1/cnn-for-handwritten-alphabets/data?select=my_model.h5

Click on Download All and save my_model.h5 into the directory.

Code displaying functions to be used for alphabet prediction

In line 6 you load the model.
Lines 10–14 display a function called crop_center which crops the center of each square-containing letter so that the letter is at an adequate size for the model for more accurate predictions.
Lines 17–43 display the predict_alphabet function which predicts the alphabet.
Each contour is converted to a grayscale image and then thresholded as shown in lines 22–23.
The image is then passed through the crop_center function. The returned image is then preprocessed in lines 32–27.
The preprocessed image then becomes the input to the model and the model returns a list containing 26 probabilities for each letter and the index of the highest probability is found in lines 40–41.
The alphabets_mapperdictionary is then used to find the corresponding letter for the resultant index pred.

Flowchart of how each letter is predicted.

Creating a Grey-Yellow-Green Color Classifier Using TensorFlow Keras

The next word in a Wordle is solely influenced by the colors of all the previous letters used.
Creating the model to classify the three colors was surprisingly the easiest part of this project.
We know that each square-containing letter contains a solid color (grey, yellow or green) — a 68x68 square-containing letter image consists of 4624 RGB NumPy arrays. An RGB NumPy array has three values corresponding to red, green, and blue respectively.
All I did was take the first RGB value of each square-containing letter image and created a neural network using that value as my training data.
For example, if ROI is the variable storing a square-containing letter image, then, RGB = (np.asarray(ROI)[0][0]) will store a NumPy list such as [127 124 119] and this RGB value comes from the color grey. Similarly, a particular green-colored square-containing letter produces the first RGB value as [123 179 127] .
As you can see for the green RGB array, the value in the middle, corresponding to the color green “179” was higher compared to the other values in that list. Whereas in the grey RGB array, all three values were somewhat close to each other. This is what we want our model to learn and predict — using the RGB NumPy array — the color of the square-containing letter.

Code to create the Color Classifier Model

Each training data in line 9 is an RGB NumPy array and the corresponding Y value or validation data is an integer (0, 1, 2) which acts as a key to different colors in line 12.
We create the model in line 15 with softmax as the activation function(always the preferred activation function for multi-classification) and use sparse_categorical_crossentropyas the loss function. This loss function is best used when the validation data is integer-coded.
We fit the model in line 21 and save the model.

Simple network architecture for our neural network

Now we need to predict the color of each square-containing letter.

Code to predict the color

We load the model in line 6 and the first RGB NumPy array is stored in a variable called RGB in line 11.
From lines 12–15 we predict the color using that NumPy array and return the color at the end of the function.

Combining Contour Detection and the Two Models to Display Words Used in Partially-Filled Wordle

We have finally finished all the components of this project.

Full code combining all components of the project

Lines 1–5, we import all libraries
Lines 7–35, we create crop_centerand predict_alphabet functions for predicting the alphabet in each square-containing letter.
Lines 37–45, we create predict_color function to predict the color of each square-containing letter.
Lines 70–85, we go through each contour and call functions and append the predicted letters into a list called lettersin line 80. We print the letter and the color with its RGB input on line 82.
Lines 88–93, we print out the words used in the Wordle.

The final output of our program using the Wordle:

Output of our program

Using Greedy Algorithm to Predict the Next Best Word for the Partially-Filled Wordle

In this post, I focused on converting the Wordle to a Python Shell output format by predicting letters and colors using my knowledge of computer vision.
I will leave the task of finding the next best word for the Wordle in a post which I will post soon and link here on this page.

More content at PlainEnglish.io. Sign up for our free weekly newsletter. Follow us on Twitter and LinkedIn. Join our community Discord.