For each pair, take the difference, drop the zeros, rank the absolute differences from smallest to largest, and give each rank the sign of its difference. If there is no systematic change, the positive and negative rank sums should be similar. Using ranks makes the test robust to outliers, and it assumes the differences are roughly symmetric around their median. Frank Wilcoxon introduced it in 1945.
Example: eight users complete a task before and after a redesign, and their time differences in seconds are −3, −5, −1, −8, +2, −6, −4 and −7. The absolute differences rank 1 to 8, and the only positive difference has rank 2, so W⁺ = 2 and W⁻ = 34. Only 3 of the 256 equally likely sign patterns give a positive sum of 2 or less, so the exact two-sided p = 6/256 ≈ 0.023.
Follow-ups: zero differences are dropped in Wilcoxon's original version, while Pratt's variant keeps them in the ranking; the sign test, which only counts positive and negative differences, needs even fewer assumptions but has less power; and for independent groups the matching test is the Mann-Whitney U test.