51,000 ads, 190 rows: revisiting a Kijiji visit-prediction model.

The problem

Kijiji is one of Canada's largest online classifieds sites, and cars are among its busiest categories. Visits are the top of the sales funnel: an ad nobody opens does not sell. A model that predicts visits from what is in the listing can tell a seller that an ad is under-performing compared with similar ads, help rank listings, and give dealers a benchmark for their inventory.

The data is a public Kaggle dataset (kijiji-car-sales-ads, CC BY-NC-SA 4.0) covering twelve Kijiji regions in British Columbia, Alberta, Ontario and Quebec. We worked on it as Group 6: Camilo Florez, Yasar Mansuri, Syed Zumair and me.

What we built in 2023

We started in a Jupyter notebook and then moved the project to a Python command-line program. It cleaned the data, drew about a dozen exploratory charts, removed outliers, and trained three regressors (linear regression, random forest and gradient boosting) separately for each city on a 60/40 split. The report compared mean real visits with mean predicted visits per year and concluded that the predictions "have similar behavior to the real ones". Random forest was "chosen" for about half of the rows, and a full run took about 30 minutes.

Looking back, two things in the write-up should have worried us. The README described the project as price prediction, although the target was visits. And it read the charts as seasonality: July and August "are the great months", and most ads were published in August, "related to the Summer season".

Going back: five problems

Reading the code in 2026, I found problems that meant the 2023 results could not be trusted:

• The model looked at the answer. For every row, the code computed each model's error against the real visit count and kept the prediction with the smallest error. That is choosing the answer closest to the truth after seeing the truth, so the reported accuracy was not something the model could deliver on a new ad.
• 51,000 ads became about 190 rows. Just before training, the data was grouped into total visits by city, month and year. The only features left were the month and the year. That leaves between 2 and 22 rows per city to train on, and the predictions were then made on every row, training rows included.
• Outliers removed by a union of six detectors. Any visit count flagged by Z score, robust Z score, IQR, percentile trimming, DBSCAN or Isolation Forest was deleted. The DBSCAN and Isolation Forest helpers had indexing bugs, and the very popular ads they removed were real, not errors.
• Values written to the wrong rows. Imputed values were assigned back with misaligned indexes, and the target itself was imputed instead of dropping ads without a visit count.
• A "no" that meant yes. The --storage_model no option was stored as an enum member, which Python treats as true, so the models were always saved.

None of these is exotic. Selecting results with the label, aggregating away the unit you want to predict, and deleting inconvenient data are mistakes I have also seen in production code. A chart of mean predicted against mean actual hides all three, because averages agree even when individual predictions do not.

Reading the data correctly

Before modelling again, I had to understand what the visit column really is. Two facts changed everything:

• It is a snapshot, not a time series. The file contains only the ads that were still online on the scrape date. Older ads had more time to sell and disappear, so the further back the publish month, the fewer ads remain. August "dominates" because of when the data was collected, not because of summer demand.
• Visits are cumulative. The column counts every visit since the ad was published, so an ad that has been online for two months has had far more chances to be seen than one posted yesterday. The age of the ad has to be a feature.

Visits are also extremely skewed: median 75, mean 294, 99th percentile about 3,350. A few viral ads would dominate any model trained on raw counts, so the new models learn log(1 + visits) and convert back.

The rebuild

The new pipeline predicts visits for each ad. It cleans what is clearly wrong (placeholder prices under $100, odometers over 1,000,000 km, model years before 1940, ads without a visit count) and keeps 50,540 ads. Imputation, scaling and one-hot encoding happen inside a scikit-learn Pipeline, so they are learnt from the training folds only. Outlier detectors are still there, now vectorised and correct, but only to report: they disagree wildly, flagging between 0.04% and 16% of the ads, which is a good reason not to delete rows based on any one of them.

The split is 80/20, stratified on visit deciles. Four candidates are compared with 5-fold cross-validation on the training set, and the winner is scored once on the test set:

On the 10,108 test ads, never seen during training or selection, gradient boosting has a mean absolute error of 158 visits against 266 for the baseline, and a median error of 32 visits against 70. Test scores match cross-validation, so the model is not overfitting. The full run takes about two minutes in my environment, instead of 30.

The two R² columns tell different stories, and both are true. On the log scale, 0.82 means the model gets the order of magnitude right: it knows a 20-visit ad from a 2,000-visit ad. On raw visits, 0.40 means it misses the size of the rare viral ads, which it under-predicts.

A chart of averages can agree perfectly with reality while every individual prediction is wrong. Evaluate at the level where the decision is made.

What drives visits

Permutation importance, how much the test error grows when one feature is shuffled, puts ad age far ahead of everything else, followed by car age, price, city and seller type. Equipment flags, colour and the day of the week the ad was posted add almost nothing. The exploratory analysis points the same way:

• Private sellers get about three times the visits of dealers for used cars (median 226 vs 67). New cars from dealers get a median of only 8.
• Older, cheaper, higher-mileage cars attract more visits. Spearman correlation with visits: car age +0.55, kilometres +0.47, ad age +0.35, price −0.35.
• Region matters a lot. Calgary (median 115) and the City of Toronto (112) get many more visits per ad than Vancouver (11) or the North Shore (8).

These are associations in one snapshot, not causes. A private seller's used sedan probably draws a different audience from a dealer's new truck, and the model cannot separate the two.

What I learned

• Never choose a prediction using the label. If the real value takes part in selecting the output, the score measures the selection, not the model.
• Model at the level of the decision. The question was about one ad. Aggregating to city-month totals threw away the data and the question with it.
• Understand how the data was collected. One scrape date explains both the "seasonality" and why ad age matters most.
• Report outliers before deleting them. When six detectors disagree by a factor of 400, none of them is telling you the truth about your data.
• Start with a baseline. "Always predict the median" gives every other number its meaning.
• Test the plumbing. The new repository has a pytest suite on synthetic data and runs it with ruff in GitHub Actions on every push. A test would have caught the "no means yes" bug.

Where AI assistants fit

As with my revisit of a bank marketing classifier, I used an AI coding assistant to read the old code, trace where the data was being reshaped, and write the new pipeline and tests. It was quick to spot that the model selection used the label, and quick to write cross-validation boilerplate. Deciding what the target really meant, which framing the business would use and which results were worth reporting was still my job.

Next steps

• Fix the survivorship bias. Ads that sold quickly, often the most attractive ones, are missing from a single snapshot. Several scrapes would allow a better target, such as visits in the first seven days.
• Add the missing signals. Photos, description text and promoted status are not in the data, and probably explain much of what the model misses on viral ads.
• Tune. A randomized search on gradient boosting and target encoding for high-cardinality columns like model and trim.

The project and the code

Dataset

Kijiji car sales ads on Kaggle, scraped on August 27, 2022, CC BY-NC-SA 4.0

Team (2023)

Group 6: Erwin Mongui, Camilo Florez, Yasar Mansuri, Syed Zumair

Model (2026)

HistGradientBoostingRegressor on log(1 + visits), chosen by 5-fold cross-validation

Test result

MAE 158 visits vs 266 for the median baseline; median error 32 visits; R² 0.82 on the log scale

Stack

Python, pandas, scikit-learn, matplotlib, pytest, ruff, GitHub Actions

Code

training-ml-python-kijiji on GitHub: the rebuilt pipeline, charts, tests and the full write-up