Kadabra: a Shazam for movies, and no way to know if it worked.

The idea

Everybody has seen a scene on a screen at a friend's house, or a clip on social media, and wondered what movie it was. Shazam solved that for songs. Kadabra, which the bot introduced to users as FilmSleuth, was our attempt for movies: send it a photo of the screen, a text description of the plot, or both, and it answers with up to three candidates, their release years and an IMDb link. A /list command shows the movies it knows.

It was a collaborative project. Ivan Zepeda set up the repository in May 2023; the code I review here is the part I committed between June and July 2023.

How it worked

Kadabra is a Django 4.2 application with two halves: background jobs that build the search indexes, and a webhook that answers Telegram.

Collecting the data. Movie records lived in PostgreSQL and were managed from the Django admin. A Celery beat schedule then ran jobs every five minutes. One job fetched each movie's metadata from the OMDb API, its full cast and synopsis from IMDb, and created an actor record for every cast member. Another fetched each actor's photo gallery from IMDb and kept only usable portraits: four OpenCV Haar cascades had to (almost) agree that the picture showed one face, and the face_recognition library had to find exactly one face too. Every job shared a small state machine (ready, running, completed, error, expired) with retry counts, and a nightly task, written but switched off in the schedule, could put back jobs stuck "running" for more than a day.

Indexing faces. For each valid photo, face_recognition produced a 128-number encoding of the face. Then came the part I still like: DeepFace estimated how old the actor looked in the photo. Add that age to the actor's birthday, scraped from IMDb, and you get an estimate of the year the photo was taken. If a 30-year-old-looking Keanu Reeves is on the screen, the movie is probably from around 1994, not 2021. The encoding, the age and the estimated year went into an Elasticsearch index.

Indexing synopses. Each movie's plot, title and director were turned into a 512-number vector with Google's Universal Sentence Encoder (large, version 5, from TensorFlow Hub) and stored in a second index, together with the IDs of its cast.

Answering. When a photo arrives, the bot encodes every face in it and finds the closest stored face for each one, keeping matches above a cosine similarity of 0.93. When text arrives, it cleans it with NLTK, encodes it with the same sentence encoder and runs a nearest-neighbour search on the synopses. If both are present, one Elasticsearch request combines them: semantic similarity to the plot, plus a boost for movies that star a recognised actor or that came out within ten years of the photo's estimated year. The thresholds, the number of results and the year window are editable in the admin through django-constance, and the movie list is cached in Redis.

Three models, two vector indexes, a job queue and a chat interface: the hard parts were all there. The cheap part, a list of photos with the right answers, never was.

Going back: what I found

Reading the code in 2026, most problems were not in any one model. They were in the places where the models meet.

• Nothing measured whether it worked. The repository has no labelled examples (screenshots or descriptions paired with the right movie), and both tests.py files are the empty Django templates. The 0.93 face threshold and the 0.30 result threshold were chosen by trying the bot and adjusting. Without a test set, every change to a threshold, a model or a query was a guess, and so was every demo.
• The question did not look like the documents. Synopses were indexed as written. The user's text was lower-cased, stripped of numbers and stop words, lemmatised and then passed through set(), which removes duplicate words and also loses their order. A sentence encoder trained on natural sentences then had to compare "man find ring dark lord destroy" style word bags with real paragraphs. The text you search with should be prepared exactly like the text you index.
• Scores from different searches were added. Elasticsearch adds the nearest-neighbour score to the score of the regular query. The regular query had one optional clause per recognised actor and one per year window, so a movie from the right decade matched even if none of the actors were in it. A single 0.30 threshold was then applied to the sum of two scores on different scales.
• The "no match" reply could never appear. The function that formats results always appends a "Best Result" line, so it never returns an empty answer. When nothing passed the threshold, the bot replied "Best Result: None" instead of its friendly "there is not enough information" message.
• Face search compared against every photo. Each face was matched with a script_score query over all documents, which computes cosine similarity against every stored face. That is fine for a few thousand photos and slow for millions. The face vectors were also never mapped for approximate nearest-neighbour search, although the synopsis vectors were. And face_recognition is designed for Euclidean distance, with 0.6 as its usual tolerance. Cosine similarity can work, but the threshold had to be found again, without data to find it.
• A 129-line workaround for one wrong line. The page downloader did str(resp.read()). In Python 3 that does not decode the page. It returns the text form of the bytes, b'...', with every accented character written as an escape like \xc3\xa9 and every newline as a literal \n. The code then needed 129 replace calls to turn the escapes back into letters, and an image attribute that had to be read as "\nsrc" instead of "src". Decoding the bytes, resp.read().decode("utf-8"), would have made all of that unnecessary.
• Scraping that breaks on the next deploy. The IMDb parsers searched for generated CSS class names such as sc-dec7a8b-2 haviXP, which change whenever the site is rebuilt, and rotated random browser user agents. A list of 334 torrent release tags (317 unique, from "DVDRip" to "YIFY") cleaned up movie names, and the year detection only looked for years from 1900 to 2013. IMDb's conditions of use also restrict automated collection, so a licensed source, such as IMDb's own datasets or an API, is the right foundation for anything beyond a class project.
• Smaller operational issues. The large sentence encoder was loaded from settings.py, so every manage.py command and every Celery worker loaded it, even the ones that never encode text. A 5 MB age-detection model (age_detect_cnn_model.h5) is committed but never used: the code calls DeepFace instead. The delete signal for movie documents was copied from the actor one and refers to a field movies do not have. Errors were printed and swallowed, so the bot answered "There was an error" with nothing in the logs to explain why.

None of these is unusual. I have seen each of them in production systems: train/serve skew in preprocessing, scores combined without care, an empty-result path nobody tested, and a decoding bug patched downstream instead of fixed at the source.

What held up

• The pipeline design. Small, retryable jobs with explicit states, an expiry for repeated failures and a reset for stuck work is the same pattern I would use today for any slow, flaky collection process.
• One store for vectors and filters. Keeping embeddings, cast IDs and years in Elasticsearch let a single request mix semantic search with exact filters, instead of joining results from separate systems in Python.
• Settings outside the code. Thresholds, result counts and cache times could be changed from the admin without a deploy. That is exactly what you want while tuning, as long as you also have a way to measure the effect of each change.
• Using age as a signal. Estimating the year of a scene from an actor's apparent age is a cheap, original way to narrow a search across decades of films. It deserved an evaluation of its own.

What I would do differently

I have not rebuilt Kadabra. If I did, this would be the order:

• Start with the test set. A few hundred screenshots and plot descriptions, each labelled with the right movie, and one number to track: how often the right movie is in the top three. Every other decision below would be checked against it.
• Prepare queries and documents the same way. Send the user's text to the sentence encoder as written, as the synopses were, and let the model handle stop words and word forms.
• Combine rankings, not raw scores. Run the face, year and text searches separately and merge their ranked lists, for example with reciprocal rank fusion, so no score scale dominates. Make the year window a filter or a gentle boost, never a match on its own.
• Index faces for nearest-neighbour search. Map the face vectors for approximate kNN, like the synopses, and tune the threshold on the test set.
• Match the scene, not only the faces. Many frames have no clear face. An image-text embedding model could match the whole frame against stills and descriptions, and handle animation too.
• Use licensed data and test the plumbing. Replace scraping with a licensed dataset or API, decode text once at the edge, and add unit tests for the formatting and empty-result paths, which would have caught the "Best Result: None" reply.

What I learned

• The evaluation set is part of the product. Without one, a machine learning system is a demo, however good the architecture is.
• Most bugs live between components. Each model worked. The problems were in preprocessing that did not match, scores that did not mix, and a response path nobody exercised.
• Fix data problems at the source. One wrong decode line produced a 129-line workaround and a strange attribute name. When a cleanup function keeps growing, look upstream.
• Simple, original signals are worth measuring. The age-to-year idea was the most interesting part of the project and the one with the least evidence behind it.

Where AI assistants fit

As with my revisits of a bank marketing classifier and a Kijiji visit-prediction model, I used an AI coding assistant to read the 2023 code again and trace how a photo or a sentence moves from Telegram to Elasticsearch and back. It was quick to explain why 129 replace calls existed and to spot that the empty-result message was unreachable. Deciding which of these problems mattered, and what a real evaluation would look like, was still my job.

The project and the code

Project (2023)

Kadabra, a Telegram bot (introduced to users as FilmSleuth) that identifies a movie from a photo of the screen or a plot description

Team

Collaborative repository started by Ivan Zepeda; the bot, data pipeline and search code committed by Erwin Mongui

Models

face_recognition (128-dimension face encodings), DeepFace age estimation, Universal Sentence Encoder large v5 (512-dimension text vectors), OpenCV Haar cascades for photo filtering

Stack

Python, Django 4.2, Celery, Redis, PostgreSQL, Elasticsearch 8, TensorFlow, NLTK, python-telegram-bot

Evaluation

None in the repository: no labelled examples, no tests, thresholds set by hand

Code

training-ml-python-kadabra on GitHub