Deliver changes without corrupting the search projection through retries and lag.
course-products examples use the teaching mapping and product from chapter 5, “Mapping and field types”. Exercise walkthroughs below use the trainer’s separate public dataset.
- Database transaction + outbox
- Queue + retries
- Search projection
Deliver changes without corrupting the search projection through retries and lag.
The course targets Elasticsearch 9.4.3. Exercise examples use the public products dataset; other requests illustrate separate teaching indices and may require setup or privileges unavailable in the trainer. Responses are shortened fragments; scores may vary. Check deployment and license requirements for advanced features.
Operations and partial success
Index replaces a document by ID, create requires absence, update modifies an existing representation, and delete removes it. Bulk uses newline-terminated NDJSON rather than one JSON array. Inspect every item: retrying a partially successful batch can repeat completed actions. Retry transient failures selectively with bounded backoff.
Check your understandingOrdering and idempotency
A stable _id makes full replacement resilient to duplicate delivery but does not prevent an old event overwriting a new one. Use source versions, ordering or current-state reads. if_seq_no and if_primary_term protect Elasticsearch concurrency, not automatically your database event order. Deletions also need a reliable delivery contract.
Check your understandingDelivery and transformation
A queue decouples database writes from indexing, but a process can fail between commit and dispatch. An outbox commits business data and an event together; CDC reads a change stream. Measure lag, failures and the age of unsuccessful events. Ingest pipelines transform and enrich documents, not transactional delivery.
Full replacement with the same ID avoids duplicate products; event ordering still needs protection.
{
"name": "Running headphones",
"sku": "sku-1",
"price": 99
}How to read this result
Full replacement with the same ID avoids duplicate products; event ordering still needs protection.
Catalog reasoning example
Version 12 changes price to 100 and version 13 to 90. Delivering 13 before 12 can leave price 100 despite a stable ID. No duplicates does not mean correctness. Use monotonic source versions with an appropriate versioning contract or rebuild from current database state. After network timeouts, remember that a write may have succeeded even when its acknowledgment was lost.
Check your understandingCheck your understanding
Answer before expanding each explanation.
1. Does bulk HTTP 200 mean total success?
No, inspect errors and every item.
2. Does a stable _id solve event ordering?
No, an older event can still overwrite a newer one.
3. Why use an outbox?
To atomically commit database changes and an event record.
4. What do seq_no and primary_term protect?
Concurrent changes to a specific Elasticsearch document version.
5. What should be measured besides errors?
Delivery lag and the age of pending changes.
Key takeaways
- No, inspect errors and every item.
- No, an older event can still overwrite a newer one.
- To atomically commit database changes and an event record.
- Concurrent changes to a specific Elasticsearch document version.
- Delivery lag and the age of pending changes.
Official reference
Ready to put this into practice?
Confirm reading to save your place. Reading and exercise progress are tracked separately in this browser using your guest cookie.