See how strings become tokens and why index and query analysis must agree.
- Three analysis stages
- Index and query analysis
- Diagnosing with _analyze
See how strings become tokens and why index and query analysis must agree.
The course targets Elasticsearch 9.4.3. Exercise examples use the public products dataset; other requests illustrate separate teaching indices and may require setup or privileges unavailable in the trainer. Responses are shortened fragments; scores may vary. Check deployment and license requirements for advanced features.
Three analysis stages
Character filters transform characters, a tokenizer emits tokens, and token filters transform tokens. HTML stripping, word splitting and lowercasing solve different problems. Token positions retain structure for phrase matching. The visible _source string alone cannot explain a match: retrieval compares analyzed terms.
Check your understandingIndex and query analysis
Index analysis builds indexed terms. Full-text queries analyze input too, usually with the field analyzer or search_analyzer; term does not. Inconsistent normalization loses matches: a boost cannot fix lowercasing applied on only one side. A separate search_analyzer is useful for some synonym and autocomplete strategies.
Check your understandingDiagnosing with _analyze
Test small examples: hyphenated SKUs, Cyrillic variants, mixed scripts and stop words. _analyze reports tokens, positions and offsets; explain=true exposes stages. Aggressive stemming can merge different concepts, and stop-word removal changes short phrases. Inspect tokens before trying to fix the query.
The standard analyzer lowercases words; inspect the actual tokens.
{
"analyzer": "standard",
"text": "Wireless Headphones"
}Elasticsearch response
{
"tokens": [
{
"token": "wireless",
"start_offset": 0,
"end_offset": 8,
"type": "<ALPHANUM>",
"position": 0
},
{
"token": "headphones",
"start_offset": 9,
"end_offset": 19,
"type": "<ALPHANUM>",
"position": 1
}
]
}How to read this result
The standard analyzer lowercases words; inspect the actual tokens.
Catalog reasoning example
“Wi-Fi Headphones” may not become the terms you expect visually. Run _analyze for the field and inspect tokens and positions, then repeat for “wi fi headphones”. If terms differ, compare tokenization and filters rather than randomly increasing fuzziness. Include numeric and hyphenated model names in your test vocabulary. An analyzer suitable for prose can damage identifiers; an independent keyword representation helps.
Check your understandingCheck your understanding
Answer before expanding each explanation.
1. How is a tokenizer different from a token filter?
The tokenizer emits tokens from characters; a filter transforms those tokens.
2. Does term analyze input?
No, it looks for the supplied term.
3. What should be checked before boosting?
Compatibility of index-time and query-time tokens.
4. Why inspect position?
It controls order and distance in phrase queries.
5. Should all stop words be removed?
Not always: they may matter in names and short phrases.
Key takeaways
- The tokenizer emits tokens from characters; a filter transforms those tokens.
- No, it looks for the supplied term.
- Compatibility of index-time and query-time tokens.
- It controls order and distance in phrase queries.
- Not always: they may matter in names and short phrases.
Official reference
Ready to put this into practice?
Confirm reading to save your place. Reading and exercise progress are tracked separately in this browser using your guest cookie.