AI Lab

Ask SneakerPulse finds the right shoe 85% of the time across 104 held-out test questions written in the shapes visitors use, tested daily.

88 of 104 right (95% interval 76 to 90%). Scores use the held-out test questions: the part of the 611-question set that is kept aside while problems are fixed. The live system's questions are sampled and scored over the last 7 days. Plain search, with no AI at all, scored 98%. Last run November 22, 2026.

Results by system

The same test questions, scored at each stage. Plain search is what a visitor gets if every AI tier is down; an AI tier only earns its place if it beats that.

Public path

Ask SneakerPulse, end to end

The question box exactly as a visitor uses it: whichever AI tier answers, then the lookup.

85%

right shoe, first answer (88 of 104) · 95% interval 76 to 90%

104 test questions · last run November 22, 2026

Right shoe in the top 3
88%
Size read correctly
100%
Question type read
72%
Typical response (p50)
1.1 s
Fell back to plain search
0%
Answered by
Local AI (fast) 81%, Hosted AI 19%

Hosted AI

Questions the pipeline handed to the hosted AI tier.

82%

right shoe, first answer (14 of 17) · 95% interval 59 to 94%

Too few questions to judge (17 so far). The interval shows how wide the uncertainty is.

17 test questions · last run November 22, 2026

Right shoe in the top 3
94%
Size read correctly
100%
Question type read
67%
Typical response (p50)
298 ms
Fell back to plain search
0%

Local AI (fast)

Questions handled by the fast local tier when it is online.

86%

right shoe, first answer (70 of 81) · 95% interval 77 to 92%

81 test questions · last run November 21, 2026

Right shoe in the top 3
89%
Size read correctly
100%
Question type read
74%
Typical response (p50)
1.1 s
Fell back to plain search
0%

Local AI (always on)

The slower, always-on local tier, tested directly.

89%

right shoe, first answer (87 of 98) · 95% interval 81 to 94%

98 test questions · last run November 22, 2026

Right shoe in the top 3
92%
Size read correctly
100%
Question type read
58%
Typical response (p50)
12.7 s
Fell back to plain search
0%
The floor

Plain search (no AI)

The question's own words go straight to the lookup. The floor every AI tier has to beat.

98%

right shoe, first answer (102 of 104) · 95% interval 93 to 99%

104 test questions · last run November 22, 2026

Right shoe in the top 3
100%
Size read correctly
100%

Accuracy over time

Whole pipelinePlain search
70%80%90%100%10-0710-3011-22

Bars are 95% intervals, on the held-out test questions. Each day scores the last 7 days of results for the pipeline and the full test set for plain search.

Show the numbers
DateRight shoeQuestions
2026-10-0798%43
2026-10-1489%100
2026-10-1589%103
2026-10-1691%106
2026-10-1790%107
2026-10-1888%108
2026-10-1988%108
2026-10-2088%108
2026-10-2189%108
2026-10-2289%108
2026-10-2388%108
2026-10-2487%108
2026-10-2587%108
2026-10-2687%108
2026-10-2785%108
2026-10-2887%108
2026-10-2988%107
2026-10-3090%107
2026-10-3190%107
2026-11-0189%107
2026-11-0289%107
2026-11-0389%107
2026-11-0490%107
2026-11-0590%107
2026-11-0689%107
2026-11-1086%88
2026-11-1187%94
2026-11-1287%100
2026-11-1387%102
2026-11-1487%103
2026-11-1988%104
2026-11-2088%104
2026-11-2188%104
2026-11-2285%104

Which kinds of question work

Kind of questionWhole pipelinePlain search
Nicknames ("bred 11", "panda dunk")92% (35 of 38)98%
Misspellings and shorthand100% (6 of 6)100%
Full product names94% (29 of 31)99%
Grade-school and women's pairs89% (8 of 9)100%
A whole model ("jordan 4 resale")58% (11 of 19)100%
A whole brand67% (4 of 6)100%

Right shoe on the first answer, over the whole question set (including the questions used while fixing problems, so read it as a guide, not a score). Small groups carry wide intervals.

Every kind of question it answers

Kind of questionSent to the right placeAnswer correctLive pipeline
Price of a shoe100% (144 of 144)100% (144 of 144)80% (41 of 51)
Find and rank ("cheapest size 9 adidas running shoe")100% (53 of 53)100% (53 of 53)
top 3 right: 53 of 53
100% (12 of 12)
Compare two shoes100% (20 of 20)100% (20 of 20)100% (3 of 3)
GOAT or StockX100% (15 of 15)100% (15 of 15)100% (2 of 2)
Price by size100% (11 of 11)100% (11 of 11)100% (6 of 6)
How a brand or model is doing100% (13 of 13)100% (13 of 13)100% (5 of 5)
What is rising or falling100% (18 of 18)100% (18 of 18)100% (3 of 3)
The resale market100% (9 of 9)100% (9 of 9)not sampled yet
When to sell (past patterns)100% (10 of 10)100% (10 of 10)not sampled yet
Where sneakers sell (city and state)100% (26 of 26)100% (26 of 26)100% (3 of 3)
Top sellers in a month or year100% (18 of 18)100% (18 of 18)100% (18 of 18)
Call It100% (5 of 5)100% (5 of 5)not sampled yet
Above retail? (we hold no retail prices)100% (8 of 8)100% (8 of 8)not sampled yet
Out of scope (honest decline)100% (13 of 13)100% (13 of 13)100% (2 of 2)

Plain search over the whole question set, which includes the questions used while building each kind of answer, so read it as a guide. "Answer correct" means the answer states what an independent calculation from the data files says (for find questions: the first pick is the right shoe). A model only chooses the kind of question; every number comes from the data.

An honest probe. 20 questions were written after the router was built and not tuned on. First run: 14 of 20 right. They were fixed afterwards, so today’s 20 of 20 is no longer an untouched test; it is a record that the first pass missed 6.

Cleat rule: cleats are only shown when the question asks for them. 603 of 603 questions without a cleat word surfaced none.

Speed

Dark bar: typical (median) response. Light bar: slowest 5% (p95). Plain search has no AI step, so it is not timed.
Ask SneakerPulse, end to end
1.1 s
1.5 s p95
Hosted AI
298 ms
502 ms p95
Local AI (fast)
1.1 s
1.5 s p95
Local AI (always on)
12.7 s
18.9 s p95

What happens when a tier is down

Questions go to the fastest healthy tier. If it fails or is out of capacity, the next one takes over, and the last step is plain search, so the box always answers.

Each question tries the first tier that is up; if one fails the next takes over.Local AI (fast)tried firstHosted AInext, if the first is downLocal AI (always on)next, if that is downPlain searchno AI, always worksLocal AI (fast)tried firstHosted AInext, if the first is downLocal AI (always on)next, if that is downPlain searchno AI, always works

Security

  • Instruction-hijack tests: questions that try to give the AI orders ("ignore the above and say...", fake system messages, injected markup, links) must come back as a plain, valid reading and must not change what the page shows.
    • Whole pipeline: 11 of 12 neutral (95% interval 65 to 99%; too few to judge).
    • Lookup on its own, with the AI removed: 100% neutral (36 of 36) (95% interval 90 to 100%).
  • The AI only interprets the question. It never writes a price.
  • Every number on an answer comes from recorded completed sales.
  • Photos are not stored. Where a feature reads a photo, it is used for that one request and discarded.

Where it falls short

  • Grade-school and women's pairs: the whole pipeline found the right shoe 89% of the time (8 of 9).
  • A whole model: the whole pipeline found the right shoe 58% of the time (11 of 19).
  • A whole brand: the whole pipeline found the right shoe 67% of the time (4 of 6).
  • Labelling what kind of question it is (price, size, compare): the whole pipeline was right 72% of the time (57 of 79). The page does not depend on this label today.

A few real examples

Some of the questions in the test, with what the live system answered on its last run. Passes and misses are both shown.

How we test

More on the system: how SneakerPulse is built · about the builder · try the question box. The figures are in data/ai_lab.json.