Define measurable ranking goals and distinguish offline judgments from user behavior.
course-products examples use the teaching mapping and product from chapter 5, “Mapping and field types”. Exercise walkthroughs below use the trainer’s separate public dataset.
- Queries and judgments
- Choosing metrics
- Online evidence and regressions
Define measurable ranking goals and distinguish offline judgments from user behavior.
The course targets Elasticsearch 9.4.3. Exercise examples use the public products dataset; other requests illustrate separate teaching indices and may require setup or privileges unavailable in the trainer. Responses are shortened fragments; scores may vary. Check deployment and license requirements for advanced features.
Queries and judgments
Collect exact SKU, brand, category, long-intent, typo and no-answer queries. Experts judge query–product pairs, not a product’s absolute value. Separate tuning and held-out sets to avoid fitting weights to known answers. Include queries from real failures.
Check your understandingChoosing metrics
Precision@k measures useful items in the top k; recall measures coverage of relevant items. MRR focuses on the first useful result, while NDCG uses graded relevance and order. Recall needs a sufficiently complete relevance set. Rank Evaluation API makes judged evaluations reproducible, but does not produce reliable judgments automatically.
Check your understandingOnline evidence and regressions
Clicks depend on position, UI and product appeal, so CTR is not relevance itself. A/B tests compare randomized groups using predefined metrics and observation periods. An analyzer change may improve averages while breaking SKUs: keep mandatory regression scenarios. Evaluate quality alongside latency and cost.
Ratings and IDs here are illustrative; meaningful measurement needs real judgments.
{
"requests": [
{
"id": "headphones",
"request": {
"query": {
"match": {
"name": "headphones"
}
}
},
"ratings": [
{
"_index": "course-products",
"_id": "sku-1",
"rating": 3
}
]
}
],
"metric": {
"discounted_cumulative_gain": {
"k": 10,
"normalize": true
}
}
}How to read this result
Ratings and IDs here are illustrative; meaningful measurement needs real judgments.
Catalog reasoning example
For exact SKU lookup, the first correct item matters more than top-ten variety. Broad headphones queries can benefit from graded judgments across several useful items. Do not average these flows in a way that hides identifier regressions behind popular-query improvements. Report overall quality and key segments. Small differences need stability checks, not declarations based on one changed click.
Check your understandingCheck your understanding
Answer before expanding each explanation.
1. What does an expert judge?
A particular product’s usefulness for a particular query.
2. When is MRR useful?
When finding the first useful result matters.
3. What does NDCG capture?
Ordering and graded relevance, emphasizing top positions.
4. Why is CTR insufficient?
It depends on position and UI as well as search quality.
5. Why hold out evaluation queries?
To detect overfitting to tuning examples.
Module case study: Relevance and search experience
A sales feature pushes popular accessories above suitable products. Propose signal normalization and judged-query evaluation, then design pagination without duplicates while the catalog changes.
Assessment criteria
Preserve eligibility, bound business signals, compare NDCG and regressions, and use PIT/search_after with stable sorting.
Key takeaways
- A particular product’s usefulness for a particular query.
- When finding the first useful result matters.
- Ordering and graded relevance, emphasizing top positions.
- It depends on position and UI as well as search quality.
- To detect overfitting to tuning examples.
Official reference
Ready to put this into practice?
Confirm reading to save your place. Reading and exercise progress are tracked separately in this browser using your guest cookie.