Method

How the rankings are made

Shampoo Arena ranks every shampoo sold on dm.cz by what is in it, separately for each hair type. Nobody rates a shampoo on its own. Language models compare two ingredient lists at a time, without knowing which products they are, and a statistical model turns those verdicts into a ranking that says how sure it is.

296shampoos ranked

9hair types

110,881pairwise verdicts

1judge model

The pipeline

Scrape. Every shampoo on dm.cz, with its INCI ingredient list and price, discovered through the shop’s public sitemap.
Normalise. Ingredient lists are parsed and spellings merged into canonical names. Only exact matches are merged automatically. Near-misses go to a human, because dipropylene glycol and propylene glycol look closer than a genuine typo does.
Judge. A model is shown two formulations and a hair type and picks the better one, with a confidence from 1 to 5 and its reasons.
Rank. All verdicts for a hair type are fitted at once with a Bradley–Terry model.

Judging blind

The judge sees the hair type, a short rubric for it, and two numbered ingredient lists: no brand, no product name, no price, no packaging, no reviews. This is enforced in the code, not just requested in the prompt: the functions that build prompts accept ingredient lists, never products, and a test checks real prompts for product names. The model is asked to reason from the chemistry alone.

What a judge sees (illustrative, abridged)

Hair profile: Dandruff and flaking scalp
Judging criteria: favour recognised anti-dandruff actives …

Formulation A:                 Formulation B:
1. Aqua                        1. Aqua
2. Sodium Laureth Sulfate      2. Sodium Coco-Sulfate
3. Cocamidopropyl Betaine      3. Lauryl Glucoside
…                              …
10. Piroctone Olamine          6. Hippophae Rhamnoides Fruit Oil

Which formulation is better suited to this hair profile?

Which product appears as A and which as B is randomised for every comparison, and the order is recorded. That lets each hair type’s diagnostics page measure position bias instead of assuming it away.

From verdicts to a ranking

Each shampoo \(i\) gets one hidden strength \(\beta_i\) per hair type. The Bradley–Terry model says the chance that \(i\) wins a comparison against \(j\) is

\[ P(i \text{ beats } j) \;=\; \sigma(\beta_i - \beta_j) \;=\; \frac{1}{1 + e^{-(\beta_i - \beta_j)}} \]

That is the same logistic curve chess Elo uses. The difference is how it is fitted. Elo updates ratings one game at a time, so the same games in a different order give different ratings. Here every verdict is in a database, so all strengths are fitted together, by maximising the penalised log-likelihood of every verdict at once:

\[ \hat{\beta} \;=\; \arg\max_{\beta} \; \sum_{\text{verdicts } (w,\,l)} \log \sigma(\beta_w - \beta_l) \;-\; \lambda \sum_i \beta_i^2 \]

where \(w\) is the winner and \(l\) the loser of each verdict. A tie counts as half a win each way. The small ridge penalty (\(\lambda = 0.5\)) keeps a shampoo that never lost from running off to infinity. The result does not depend on order, needs no tuning constant, and gives every strength a standard error.

Strengths are shown on the familiar Elo scale:

\[ \text{score}_i \;=\; 1500 + \frac{400}{\ln 10}\,\beta_i \;\approx\; 1500 + 174\,\beta_i \]

So 1500 is the average shampoo, and a 200-point gap means the stronger one is expected to win about three comparisons in four.

How sure is a rank?

The ± next to a score is a 95% confidence interval. Its width comes from the standard error of \(\hat\beta_i\), read off the curvature of the likelihood (the inverse Fisher information):

\[ \big[\,\ell_i,\; u_i\,\big] \;=\; \text{score}_i \;\pm\; 1.96 \cdot \frac{400}{\ln 10}\,\operatorname{SE}(\hat\beta_i) \]

The small range under each rank on the leaderboard (say, 4–11) turns those intervals into places. The best plausible rank counts only the shampoos whose whole interval lies above this one; the worst counts every shampoo that could be above it:

\[ \text{best}_i = 1 + \#\{\, j : \ell_j > u_i \,\} \qquad \text{worst}_i = \#\{\, j : u_j > \ell_i \,\} \]

If two shampoos’ ranges overlap, the data do not say which is better. On each hair type’s ranking page, heavier rules mark the points where one group is genuinely separated from the next.

Which pairs get judged

Comparing every pair would cost tens of thousands of calls per hair type, and most pairs are foregone conclusions. Instead, each shampoo is first rated once on its own, to give a rough starting order. A few random pairs across that order calibrate the scale. The remaining budget goes to the pairs the current fit is least able to call: a comparison tells the model the most when the predicted win chance is close to 50%.

Pooled and per-model rankings

Verdicts are kept separate by judge model and by prompt version, so a ranking never mixes different rubrics. Every judge gets its own ranking, and the pooled ranking is one fit over all included judges’ verdicts together. The switch on the leaderboard moves between them. When two models disagree about the same pair, the diagnostics pages show how often that happens. Currently included: wandb:zai-org/GLM-5.3-Flash.

The “Avg rank” column on the leaderboard is just the mean of a shampoo’s ranks across hair types. It is a convenience for browsing, not a fitted score.

What this can’t tell you

An INCI list gives the order of ingredients (most to least, down to 1%) but not their concentrations. Two shampoos with near-identical lists can behave differently.
It says nothing about pH, how the surfactants behave together, or clinical evidence. The judges reason about plausible chemistry; they do not test anything.
Shampoo is a rinse-off product with a short contact time, which limits what any active can do.
Some retailer listings carry the wrong ingredient list. The ones found are excluded by hand rather than ranked.

So treat a ranking as a well-grounded signal about formulation, not a verdict on how a shampoo will work on your hair, and not medical advice. The confidence intervals are shown because they matter.