Search & Catalog 15k SKU Q-Commerce: Relevan + In-Stock Real-time
System design search dan catalog untuk quick commerce dengan 15.000+ SKU. Elasticsearch full-text search dengan multi-match query dan typo tolerance (fuzziness), filter ketersediaan per-hub via Redis pipeline query-time filtering, ranking berbasis relevansi, ketersediaan, margin, dan popularitas, index update via Kafka streaming, dan autocomplete handler. Implementasi Golang lengkap dengan worker concurrency pattern untuk index sync.
- Search & Catalog 15k SKU Q-Commerce: Relevan + In-Stock Real-time
- Arsitektur Search & Catalog
- 1. Problem Statement: Kenapa Search Q-Commerce Susah?
- 2. Elasticsearch Product Index
- Index Mapping
- Go: Product Model
- 3. Search Service: Go + Elasticsearch
- Search Service dengan Multi-Match Query
- 4. Stock Filter: Redis Pipeline
- 5. Index Sync Worker: Kafka Consumer
- 6. Autocomplete Handler
- 7. Ranking: Relevance + Availability + Margin + Popularity
- 8. Edge Cases
- Edge Case 1: Elasticsearch dan Redis Stock Mismatch
- Edge Case 2: Zero Results Query
- Edge Case 3: Race Condition Stock Update
- Key Takeaways
- Kesimpulan
Search & Catalog 15k SKU Q-Commerce: Relevan + In-Stock Real-time
"In Q-commerce, a search that shows out-of-stock items is worse than no search at all. Every millisecond of latency costs orders; every irrelevant result loses customers."
TL;DR
Search di Q-commerce punya tantangan unik: 15.000+ SKU, real-time stock availability per hub, typo tolerance, dan ranking yang harus balance antara relevansi, profit margin, dan popularitas. Artikel ini membangun search service dari nol dengan Elasticsearch multi-match, Redis pipeline stock filter, Kafka-driven index sync, dan autocomplete with completion suggester. Semua kode dalam Go dengan pattern production-grade.
Arsitektur Search & Catalog
Key Design Decisions
- Elasticsearch untuk search — full-text, typo tolerance, scoring built-in
- Redis untuk stock — real-time per-hub, pipeline batch query, microsecond latency
- Kafka untuk index sync — near-real-time update tanpa DB polling
- Query-time stock filtering — bukan index-time, karena stock berubah terus
1. Problem Statement: Kenapa Search Q-Commerce Susah?
15.000+ SKU
Bukan jumlah besar untuk e-commerce biasa, tapi untuk Q-commerce yang inventory-nya per-hub, setiap SKU punya stock status unik di 50+ hub.
Real-time Stock
Stok berubah setiap ONG (order not go) atau restock. Elasticsearch index tidak bisa di-update per-transaksi — perlu approach query-time filtering via Redis.
Typo Tolerance
User Q-commerce mengetik cepat di mobile: 'indomie' jadi 'indomei', 'goreng' jadi 'gorengg'. Search harus forgive, bukan exact match.
Multi-faktor Ranking
Ranking tidak bisa relevansi saja. Margin (promosi produk high-margin), popularitas, ketersediaan — semuanya harus di-weight.
2. Elasticsearch Product Index
Index Mapping
{
"settings": {
"analysis": {
"analyzer": {
"product_analyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"lowercase",
"asciifolding",
"ngram_filter",
"synonym_filter"
]
},
"autocomplete_analyzer": {
"type": "custom",
"tokenizer": "keyword",
"filter": ["lowercase", "asciifolding"]
}
},
"filter": {
"ngram_filter": {
"type": "ngram",
"min_gram": 2,
"max_gram": 20
},
"synonym_filter": {
"type": "synonym",
"synonyms": [
"mie, mi, indomie, noodle",
"goreng, gorengan, fried",
"susu, milk",
"kopi, coffe, coffee",
"teh, tea, the"
]
}
}
}
},
"mappings": {
"properties": {
"sku": { "type": "keyword" },
"name": { "type": "text", "analyzer": "product_analyzer", "fields": { "raw": { "type": "keyword" } } },
"description": { "type": "text", "analyzer": "product_analyzer" },
"category_id": { "type": "keyword" },
"category_name": { "type": "text", "analyzer": "product_analyzer" },
"brand": { "type": "keyword" },
"price": { "type": "long" },
"margin_pct": { "type": "float" },
"popularity_score": { "type": "float" },
"tags": { "type": "keyword" },
"image_url": { "type": "keyword", "index": false },
"unit": { "type": "keyword" },
"is_active": { "type": "boolean" },
"created_at": { "type": "date" },
"updated_at": { "type": "date" },
"suggest": {
"type": "completion",
"analyzer": "autocomplete_analyzer"
}
}
}
}Go: Product Model
package model
import "time"
// Product adalah representasi produk di search index
type Product struct {
SKU string `json:"sku"`
Name string `json:"name"`
Description string `json:"description"`
CategoryID string `json:"category_id"`
CategoryName string `json:"category_name"`
Brand string `json:"brand"`
Price int64 `json:"price"`
MarginPct float64 `json:"margin_pct"`
PopularityScore float64 `json:"popularity_score"`
Tags []string `json:"tags"`
ImageURL string `json:"image_url"`
Unit string `json:"unit"`
IsActive bool `json:"is_active"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
// Completion suggester fields
SuggestInput []string `json:"suggest_input,omitempty"`
SuggestWeight int `json:"suggest_weight,omitempty"`
}
// ProductDocument is what's stored in Elasticsearch
type ProductDocument struct {
SKU string `json:"sku"`
Name string `json:"name"`
Description string `json:"description"`
CategoryID string `json:"category_id"`
CategoryName string `json:"category_name"`
Brand string `json:"brand"`
Price int64 `json:"price"`
MarginPct float64 `json:"margin_pct"`
PopularityScore float64 `json:"popularity_score"`
Tags []string `json:"tags"`
ImageURL string `json:"image_url"`
Unit string `json:"unit"`
IsActive bool `json:"is_active"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
Suggest SuggestField `json:"suggest"`
}
type SuggestField struct {
Input []string `json:"input"`
Weight int `json:"weight"`
}3. Search Service: Go + Elasticsearch
Search Service dengan Multi-Match Query
package service
import (
"context"
"encoding/json"
"fmt"
"strings"
"time"
"github.com/elastic/go-elasticsearch/v8"
"github.com/elastic/go-elasticsearch/v8/esapi"
"github.com/faisalaffan/qcommerce/internal/search/model"
)
// SearchRequest adalah request dari client
type SearchRequest struct {
Query string `json:"query"`
HubID string `json:"hub_id"` // required for stock filtering
Category string `json:"category,omitempty"` // filter by category
Brands []string `json:"brands,omitempty"` // filter by brand
MinPrice int64 `json:"min_price,omitempty"`
MaxPrice int64 `json:"max_price,omitempty"`
SortBy string `json:"sort_by,omitempty"` // relevance, price_asc, price_desc, popularity
Page int `json:"page"`
Size int `json:"size"`
}
// SearchResult adalah hasil pencarian yang sudah difilter stock
type SearchResult struct {
Total int64 `json:"total"`
Products []RankedProduct `json:"products"`
Page int `json:"page"`
Size int `json:"size"`
TotalPages int `json:"total_pages"`
TookMs int64 `json:"took_ms"`
}
type RankedProduct struct {
Product model.ProductDocument `json:"product"`
Score float64 `json:"score"`
InStock bool `json:"in_stock"`
StockQty int `json:"stock_qty,omitempty"`
}
// SearchService handles search operations
type SearchService struct {
es *elasticsearch.Client
stockSvc *StockFilterService
index string
}
func NewSearchService(es *elasticsearch.Client, stockSvc *StockFilterService, index string) *SearchService {
return &SearchService{
es: es,
stockSvc: stockSvc,
index: index,
}
}
// Search melakukan full-text search dengan stock filtering
func (s *SearchService) Search(ctx context.Context, req SearchRequest) (*SearchResult, error) {
start := time.Now()
if req.Page <= 0 {
req.Page = 1
}
if req.Size <= 0 || req.Size > 100 {
req.Size = 20
}
// 1. Build multi-match query dengan fuzziness
query := s.buildSearchQuery(req)
// 2. Execute search ke Elasticsearch
from := (req.Page - 1) * req.Size
searchBody := map[string]interface{}{
"from": from,
"size": req.Size,
"query": query,
"_source": true,
}
// Add sorting
if req.SortBy != "" && req.SortBy != "relevance" {
searchBody["sort"] = s.buildSort(req.SortBy)
}
body, err := json.Marshal(searchBody)
if err != nil {
return nil, fmt.Errorf("marshal search body: %w", err)
}
res, err := s.es.Search(
s.es.Search.WithContext(ctx),
s.es.Search.WithIndex(s.index),
s.es.Search.WithBody(strings.NewReader(string(body))),
)
if err != nil {
return nil, fmt.Errorf("elasticsearch search: %w", err)
}
defer res.Body.Close()
if res.IsError() {
return nil, fmt.Errorf("elasticsearch error: %s", res.String())
}
// 3. Parse response
var esResponse struct {
Hits struct {
Total struct {
Value int64 `json:"value"`
} `json:"total"`
Hits []struct {
Score float64 `json:"_score"`
Source model.ProductDocument `json:"_source"`
} `json:"hits"`
} `json:"hits"`
}
if err := json.NewDecoder(res.Body).Decode(&esResponse); err != nil {
return nil, fmt.Errorf("decode search response: %w", err)
}
// 4. Extract SKUs for stock check
var products []model.ProductDocument
skus := make([]string, 0, len(esResponse.Hits.Hits))
scores := make(map[string]float64)
for _, hit := range esResponse.Hits.Hits {
products = append(products, hit.Source)
skus = append(skus, hit.Source.SKU)
scores[hit.Source.SKU] = hit.Score
}
// 5. Batch stock check via Redis pipeline
stockMap, err := s.stockSvc.BatchCheckStock(ctx, req.HubID, skus)
if err != nil {
// Log error but still return products (without stock info)
fmt.Printf("stock check failed: %v\n", err)
stockMap = make(map[string]StockInfo)
}
// 6. Rank and return
ranked := make([]RankedProduct, 0, len(products))
for _, p := range products {
stock := stockMap[p.SKU]
ranked = append(ranked, RankedProduct{
Product: p,
Score: scores[p.SKU],
InStock: stock.Available,
StockQty: stock.Quantity,
})
}
// 7. Filter out-of-stock if configured
// In Q-commerce, we typically show in-stock first, then out-of-stock
ranked = s.applyStockPriority(ranked)
totalPages := int(esResponse.Hits.Total.Value) / req.Size
if int(esResponse.Hits.Total.Value)%req.Size > 0 {
totalPages++
}
return &SearchResult{
Total: esResponse.Hits.Total.Value,
Products: ranked,
Page: req.Page,
Size: req.Size,
TotalPages: totalPages,
TookMs: time.Since(start).Milliseconds(),
}, nil
}
// buildSearchQuery membangun multi-match query dengan fuzziness
func (s *SearchService) buildSearchQuery(req SearchRequest) map[string]interface{} {
must := make([]map[string]interface{}, 0)
filter := make([]map[string]interface{}, 0)
// Full-text search dengan multi-match
if req.Query != "" {
must = append(must, map[string]interface{}{
"multi_match": map[string]interface{}{
"query": req.Query,
"fields": []string{"name^3", "description", "category_name^2", "brand^2", "tags"},
"type": "best_fields",
"fuzziness": "AUTO",
"prefix_length": 2,
"max_expansions": 50,
"tie_breaker": 0.3,
},
})
} else {
// No query: return all (browse mode)
must = append(must, map[string]interface{}{
"match_all": map[string]interface{}{},
})
}
// Filters
if req.Category != "" {
filter = append(filter, map[string]interface{}{
"term": map[string]interface{}{"category_id": req.Category},
})
}
if len(req.Brands) > 0 {
filter = append(filter, map[string]interface{}{
"terms": map[string]interface{}{"brand": req.Brands},
})
}
if req.MinPrice > 0 || req.MaxPrice > 0 {
rangeFilter := make(map[string]interface{})
if req.MinPrice > 0 {
rangeFilter["gte"] = req.MinPrice
}
if req.MaxPrice > 0 {
rangeFilter["lte"] = req.MaxPrice
}
filter = append(filter, map[string]interface{}{
"range": map[string]interface{}{"price": rangeFilter},
})
}
// Only active products
filter = append(filter, map[string]interface{}{
"term": map[string]interface{}{"is_active": true},
})
query := map[string]interface{}{
"bool": map[string]interface{}{
"must": must,
"filter": filter,
},
}
return query
}
func (s *SearchService) buildSort(sortBy string) []map[string]interface{} {
switch sortBy {
case "price_asc":
return []map[string]interface{}{
{"price": map[string]interface{}{"order": "asc"}},
}
case "price_desc":
return []map[string]interface{}{
{"price": map[string]interface{}{"order": "desc"}},
}
case "popularity":
return []map[string]interface{}{
{"popularity_score": map[string]interface{}{"order": "desc"}},
}
default:
return nil // relevance = default Elasticsearch _score sorting
}
}
// applyStockPriority: in-stock first, then out-of-stock
func (s *SearchService) applyStockPriority(products []RankedProduct) []RankedProduct {
inStock := make([]RankedProduct, 0, len(products))
outOfStock := make([]RankedProduct, 0, len(products))
for _, p := range products {
if p.InStock {
inStock = append(inStock, p)
} else {
outOfStock = append(outOfStock, p)
}
}
return append(inStock, outOfStock...)
}4. Stock Filter: Redis Pipeline
Problem: setiap search result (20-100 SKU) harus di-cek stock-nya per hub. Kalau 1-by-1, latency-nya tidak acceptable. Solusi: Redis pipeline.
Redis Pipeline vs MGET
Pipeline mengirimkan semua command dalam satu round-trip — bukan N round-trip. Untuk 100 SKU, latency turun dari ~100ms (100x1ms) menjadi ~5ms (1 batch). Alternatif: MGET — tapi pipeline lebih fleksibel karena kita bisa mix tipe command.
package service
import (
"context"
"fmt"
"strconv"
"time"
"github.com/redis/go-redis/v9"
)
type StockInfo struct {
SKU string `json:"sku"`
Available bool `json:"available"`
Quantity int `json:"quantity"`
Reserved int `json:"reserved"`
}
// StockFilterService handles real-time stock checking via Redis
type StockFilterService struct {
client *redis.Client
keyPrefix string
ttl time.Duration // TTL for stock cache
}
func NewStockFilterService(client *redis.Client, keyPrefix string) *StockFilterService {
return &StockFilterService{
client: client,
keyPrefix: keyPrefix,
}
}
// stockKey generates Redis key for a specific hub+SKU stock
func (s *StockFilterService) stockKey(hubID, sku string) string {
return fmt.Sprintf("%s:stock:%s:%s", s.keyPrefix, hubID, sku)
}
// BatchCheckStock memeriksa stock untuk banyak SKU dalam satu pipeline
func (s *StockFilterService) BatchCheckStock(ctx context.Context, hubID string, skus []string) (map[string]StockInfo, error) {
if len(skus) == 0 {
return map[string]StockInfo{}, nil
}
pipe := s.client.Pipeline()
// Schedule all commands
cmds := make(map[string]*redis.StringCmd)
for _, sku := range skus {
key := s.stockKey(hubID, sku)
cmds[sku] = pipe.Get(ctx, key)
}
// Execute pipeline (1 round trip)
_, err := pipe.Exec(ctx)
if err != nil && err != redis.Nil {
// Pipeline bisa partially fail; Redis returns nil for missing keys
// We handle each key result individually
fmt.Printf("pipeline exec error (non-fatal): %v\n", err)
}
// Parse results
result := make(map[string]StockInfo, len(skus))
for _, sku := range skus {
cmd := cmds[sku]
val, err := cmd.Result()
if err == redis.Nil {
// Stock key not found — treat as out of stock
result[sku] = StockInfo{
SKU: sku,
Available: false,
Quantity: 0,
}
continue
}
if err != nil {
// Other error — treat as unknown stock
result[sku] = StockInfo{
SKU: sku,
Available: false,
Quantity: 0,
}
continue
}
// Parse Redis value: format "qty:reserved" or just "qty"
qty, res := s.parseStockValue(val)
result[sku] = StockInfo{
SKU: sku,
Available: qty > 0 && qty > res,
Quantity: qty,
Reserved: res,
}
}
return result, nil
}
// parseStockValue parses Redis stock value
// Format: "quantity:reserved" or just "quantity"
func (s *StockFilterService) parseStockValue(val string) (int, int) {
parts := split2(val, ":")
if len(parts) < 2 {
qty, err := strconv.Atoi(parts[0])
if err != nil {
return 0, 0
}
return qty, 0
}
qty, err := strconv.Atoi(parts[0])
if err != nil {
qty = 0
}
reserved, err := strconv.Atoi(parts[1])
if err != nil {
reserved = 0
}
return qty, reserved
}
// GetSingleStock checks stock for one SKU
func (s *StockFilterService) GetSingleStock(ctx context.Context, hubID, sku string) (*StockInfo, error) {
key := s.stockKey(hubID, sku)
val, err := s.client.Get(ctx, key).Result()
if err == redis.Nil {
return &StockInfo{SKU: sku, Available: false, Quantity: 0}, nil
}
if err != nil {
return nil, fmt.Errorf("get stock %s/%s: %w", hubID, sku, err)
}
qty, reserved := s.parseStockValue(val)
return &StockInfo{
SKU: sku,
Available: qty > 0 && qty > reserved,
Quantity: qty,
Reserved: reserved,
}, nil
}
// UpdateStock updates stock in Redis (called by stock mutation events)
func (s *StockFilterService) UpdateStock(ctx context.Context, hubID, sku string, quantity int, ttl time.Duration) error {
key := s.stockKey(hubID, sku)
val := fmt.Sprintf("%d:0", quantity)
return s.client.Set(ctx, key, val, ttl).Err()
}
// DecrementStock reduces stock on order placement (via Lua for atomicity)
const decrementStockScript = `
local key = KEYS[1]
local qty = tonumber(ARGV[1])
local current = redis.call("GET", key)
if not current then
return -1 -- no stock key
end
local parts = {}
for w in current:gmatch("([^:]+)") do
table.insert(parts, w)
end
local available = tonumber(parts[1]) or 0
local reserved = tonumber(parts[2]) or 0
if available - reserved < qty then
return -2 -- insufficient stock
end
reserved = reserved + qty
redis.call("SET", key, available .. ":" .. reserved)
return reserved
`
func (s *StockFilterService) DecrementStock(ctx context.Context, hubID, sku string, quantity int) (int, error) {
key := s.stockKey(hubID, sku)
result, err := redis.NewScript(decrementStockScript).Run(ctx, s.client,
[]string{key}, quantity).Int()
if err != nil {
return 0, fmt.Errorf("decrement stock %s/%s: %w", hubID, sku, err)
}
if result < 0 {
return 0, fmt.Errorf("insufficient stock for %s/%s", hubID, sku)
}
return result, nil
}
// split2 splits a string into at most 2 parts
func split2(s, sep string) []string {
parts := strings.SplitN(s, sep, 2)
// Use local empty check since this is a private utility
if len(parts) == 0 {
return []string{""}
}
return parts
}5. Index Sync Worker: Kafka Consumer
Product catalog berubah terus: harga di-update, produk baru masuk, deskripsi diubah. Kita perlu sync ke Elasticsearch secara near-real-time.
package worker
import (
"context"
"encoding/json"
"fmt"
"log"
"time"
"github.com/elastic/go-elasticsearch/v8"
"github.com/elastic/go-elasticsearch/v8/esapi"
"github.com/segmentio/kafka-go"
"github.com/faisalaffan/qcommerce/internal/search/model"
)
// CatalogEventType mendefinisikan tipe event catalog
type CatalogEventType string
const (
EventProductCreated CatalogEventType = "product.created"
EventProductUpdated CatalogEventType = "product.updated"
EventProductDeleted CatalogEventType = "product.deleted"
EventStockChanged CatalogEventType = "stock.changed"
EventPriceChanged CatalogEventType = "price.changed"
)
// CatalogEvent adalah event dari Kafka
type CatalogEvent struct {
EventType CatalogEventType `json:"event_type"`
SKU string `json:"sku"`
Timestamp time.Time `json:"timestamp"`
Data json.RawMessage `json:"data"`
}
// IndexSyncWorker handles syncing catalog changes to Elasticsearch
type IndexSyncWorker struct {
es *elasticsearch.Client
kafkaReader *kafka.Reader
catalogReader CatalogReader
index string
batchSize int
flushInterval time.Duration
}
// CatalogReader interface untuk membaca product dari database
type CatalogReader interface {
GetProduct(ctx context.Context, sku string) (*model.Product, error)
GetProductsByIDs(ctx context.Context, skus []string) ([]model.Product, error)
}
func NewIndexSyncWorker(
es *elasticsearch.Client,
kafkaReader *kafka.Reader,
catalogReader CatalogReader,
index string,
batchSize int,
flushInterval time.Duration,
) *IndexSyncWorker {
return &IndexSyncWorker{
es: es,
kafkaReader: kafkaReader,
catalogReader: catalogReader,
index: index,
batchSize: batchSize,
flushInterval: flushInterval,
}
}
// Start mulai consume Kafka events
func (w *IndexSyncWorker) Start(ctx context.Context) error {
log.Printf("index sync worker started: index=%s batch=%d flush=%v",
w.index, w.batchSize, w.flushInterval)
// Batch buffer for bulk indexing
type pendingDoc struct {
action string // "index" or "delete"
sku string
}
pending := make(map[string]pendingDoc)
lastFlush := time.Now()
flushBatch := func() error {
if len(pending) == 0 {
return nil
}
var bulkBody string
for sku, doc := range pending {
switch doc.action {
case "delete":
bulkBody += fmt.Sprintf(`{"delete":{"_index":"%s","_id":"%s"}}`+"\n", w.index, sku)
case "index":
// Read full product from DB
product, err := w.catalogReader.GetProduct(ctx, sku)
if err != nil {
log.Printf("get product %s for indexing: %v", sku, err)
continue
}
doc := toProductDocument(product)
data, _ := json.Marshal(doc)
bulkBody += fmt.Sprintf(`{"index":{"_index":"%s","_id":"%s"}}`+"\n", w.index, sku)
bulkBody += string(data) + "\n"
}
}
if bulkBody == "" {
return nil
}
res, err := w.es.Bulk(strings.NewReader(bulkBody))
if err != nil {
return fmt.Errorf("bulk index: %w", err)
}
defer res.Body.Close()
if res.IsError() {
return fmt.Errorf("bulk index error: %s", res.String())
}
// Parse response for errors
var bulkResponse struct {
Errors bool `json:"errors"`
Items []struct {
Index struct {
Error interface{} `json:"error,omitempty"`
} `json:"index"`
} `json:"items"`
}
if err := json.NewDecoder(res.Body).Decode(&bulkResponse); err != nil {
return fmt.Errorf("decode bulk response: %w", err)
}
if bulkResponse.Errors {
// Log individual errors but don't fail the batch
for _, item := range bulkResponse.Items {
if item.Index.Error != nil {
log.Printf("bulk index item error: %+v", item.Index.Error)
}
}
}
log.Printf("index sync: flushed %d docs to %s", len(pending), w.index)
pending = make(map[string]pendingDoc)
lastFlush = time.Now()
return nil
}
defer flushBatch() // flush remaining on shutdown
for {
select {
case <-ctx.Done():
log.Println("index sync worker shutting down")
return ctx.Err()
default:
}
msg, err := w.kafkaReader.ReadMessage(ctx)
if err != nil {
if ctx.Err() != nil {
return ctx.Err()
}
log.Printf("read kafka message: %v", err)
continue
}
var event CatalogEvent
if err := json.Unmarshal(msg.Value, &event); err != nil {
log.Printf("unmarshal catalog event: %v", err)
continue
}
// Map event type to Elasticsearch action
switch event.EventType {
case EventProductCreated, EventProductUpdated, EventPriceChanged:
pending[event.SKU] = pendingDoc{action: "index", sku: event.SKU}
case EventProductDeleted:
pending[event.SKU] = pendingDoc{action: "delete", sku: event.SKU}
case EventStockChanged:
// Stock is handled by Redis, not ES index
// But we might want to update a "has_stock" flag
continue
}
// Flush if batch full or interval reached
if len(pending) >= w.batchSize || time.Since(lastFlush) >= w.flushInterval {
if err := flushBatch(); err != nil {
log.Printf("flush batch: %v", err)
}
}
}
}
// toProductDocument converts Product to indexable document
func toProductDocument(p *model.Product) model.ProductDocument {
suggestInput := make([]string, 0, 2+len(p.Tags))
suggestInput = append(suggestInput, p.Name, p.Brand)
suggestInput = append(suggestInput, p.Tags...)
return model.ProductDocument{
SKU: p.SKU,
Name: p.Name,
Description: p.Description,
CategoryID: p.CategoryID,
CategoryName: p.CategoryName,
Brand: p.Brand,
Price: p.Price,
MarginPct: p.MarginPct,
PopularityScore: p.PopularityScore,
Tags: p.Tags,
ImageURL: p.ImageURL,
Unit: p.Unit,
IsActive: p.IsActive,
CreatedAt: p.CreatedAt,
UpdatedAt: p.UpdatedAt,
Suggest: model.SuggestField{
Input: suggestInput,
Weight: int(p.PopularityScore * 100),
},
}
}6. Autocomplete Handler
package handler
import (
"context"
"encoding/json"
"fmt"
"net/http"
"strings"
"github.com/elastic/go-elasticsearch/v8"
"github.com/elastic/go-elasticsearch/v8/esapi"
)
// AutocompleteHandler handles autocomplete suggestions
type AutocompleteHandler struct {
es *elasticsearch.Client
index string
}
func NewAutocompleteHandler(es *elasticsearch.Client, index string) *AutocompleteHandler {
return &AutocompleteHandler{es: es, index: index}
}
// AutocompleteRequest is the autocomplete query
type AutocompleteRequest struct {
Query string `json:"query"`
Size int `json:"size"`
HubID string `json:"hub_id,omitempty"` // optional for stock-aware suggestions
}
// AutocompleteResponse is the autocomplete result
type AutocompleteResponse struct {
Suggestions []Suggestion `json:"suggestions"`
TookMs int64 `json:"took_ms"`
}
type Suggestion struct {
Text string `json:"text"`
Score float64 `json:"score"`
SKU string `json:"sku,omitempty"`
Category string `json:"category,omitempty"`
ImageURL string `json:"image_url,omitempty"`
Price int64 `json:"price,omitempty"`
InStock bool `json:"in_stock,omitempty"`
}
// HandleAutocomplete HTTP handler
func (h *AutocompleteHandler) HandleAutocomplete(w http.ResponseWriter, r *http.Request) {
query := r.URL.Query().Get("q")
if query == "" || len(query) < 2 {
writeJSON(w, http.StatusBadRequest, map[string]string{
"error": "query must be at least 2 characters",
})
return
}
size := 8 // default 8 suggestions
hubID := r.URL.Query().Get("hub_id")
if s := r.URL.Query().Get("size"); s != "" {
if parsed, err := parseInt(s); err == nil && parsed > 0 && parsed <= 20 {
size = parsed
}
}
result, err := h.Autocomplete(r.Context(), AutocompleteRequest{
Query: query,
Size: size,
HubID: hubID,
})
if err != nil {
writeJSON(w, http.StatusInternalServerError, map[string]string{
"error": fmt.Sprintf("autocomplete: %v", err),
})
return
}
writeJSON(w, http.StatusOK, result)
}
// Autocomplete queries Elasticsearch Completion Suggester
func (h *AutocompleteHandler) Autocomplete(ctx context.Context, req AutocompleteRequest) (*AutocompleteResponse, error) {
start := time.Now()
// Build completion suggester query
suggestBody := map[string]interface{}{
"suggest": map[string]interface{}{
"product_suggest": map[string]interface{}{
"prefix": req.Query,
"completion": map[string]interface{}{
"field": "suggest",
"size": req.Size,
"fuzzy": map[string]interface{}{
"fuzziness": "AUTO",
"prefix_length": 1,
},
},
},
},
"_source": []string{"sku", "name", "category_name", "image_url", "price", "popularity_score"},
}
body, err := json.Marshal(suggestBody)
if err != nil {
return nil, fmt.Errorf("marshal suggest body: %w", err)
}
res, err := h.es.Search(
h.es.Search.WithContext(ctx),
h.es.Search.WithIndex(h.index),
h.es.Search.WithBody(strings.NewReader(string(body))),
)
if err != nil {
return nil, fmt.Errorf("elasticsearch suggest: %w", err)
}
defer res.Body.Close()
if res.IsError() {
return nil, fmt.Errorf("elasticsearch error: %s", res.String())
}
// Parse suggest response
var esResponse struct {
Suggest map[string][]struct {
Options []struct {
Text string `json:"text"`
Score float64 `json:"_score"`
Source struct {
SKU string `json:"sku"`
Name string `json:"name"`
CategoryName string `json:"category_name"`
ImageURL string `json:"image_url"`
Price int64 `json:"price"`
} `json:"_source"`
} `json:"options"`
} `json:"suggest"`
}
if err := json.NewDecoder(res.Body).Decode(&esResponse); err != nil {
return nil, fmt.Errorf("decode suggest response: %w", err)
}
// Build suggestions
suggestions := make([]Suggestion, 0)
if suggests, ok := esResponse.Suggest["product_suggest"]; ok && len(suggests) > 0 {
for _, opt := range suggests[0].Options {
suggestions = append(suggestions, Suggestion{
Text: opt.Source.Name,
Score: opt.Score,
SKU: opt.Source.SKU,
Category: opt.Source.CategoryName,
ImageURL: opt.Source.ImageURL,
Price: opt.Source.Price,
})
}
}
return &AutocompleteResponse{
Suggestions: suggestions,
TookMs: time.Since(start).Milliseconds(),
}, nil
}
// HTTP helpers
func writeJSON(w http.ResponseWriter, status int, data interface{}) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
json.NewEncoder(w).Encode(data)
}
func parseInt(s string) (int, error) {
var n int
for _, r := range s {
if r < '0' || r > '9' {
return 0, fmt.Errorf("not a number: %s", s)
}
n = n*10 + int(r-'0')
}
return n, nil
}Completion Suggester vs Search-as-you-type
Elasticsearch punya dua fitur autocomplete: Completion Suggester (berdasarkan prefix, sangat cepat, pre-computed) dan Search-as-you-type (field type yang mengoptimalkan query untuk partial matching). Completion Suggester lebih cocok karena: (1) response time <10ms, (2) bisa pakai fuzzy untuk typo, (3) weight-based ranking. Kekurangannya: perlu pre-build suggest input, tidak real-time untuk kata baru.
7. Ranking: Relevance + Availability + Margin + Popularity
Ranking di Q-commerce tidak bisa pure relevansi Elasticsearch. Kita perlu multi-faktor scoring:
package service
import (
"math"
)
// RankingConfig untuk weighting
type RankingConfig struct {
RelevanceWeight float64 `json:"relevance_weight"` // 0.4
StockWeight float64 `json:"stock_weight"` // 0.25
MarginWeight float64 `json:"margin_weight"` // 0.2
PopularityWeight float64 `json:"popularity_weight"` // 0.15
}
// DefaultRankingConfig adalah default weights
func DefaultRankingConfig() RankingConfig {
return RankingConfig{
RelevanceWeight: 0.40,
StockWeight: 0.25,
MarginWeight: 0.20,
PopularityWeight: 0.15,
}
}
// Ranker melakukan re-ranking products dengan multi-faktor
type Ranker struct {
config RankingConfig
}
func NewRanker(config RankingConfig) *Ranker {
return &Ranker{config: config}
}
// CompositeScore menghitung skor gabungan
func (r *Ranker) CompositeScore(
relevanceScore float64,
marginPct float64,
popularityScore float64,
stockQty int,
) float64 {
// Normalize each factor to 0-1 range
relScore := normalizeScore(relevanceScore, 0, 10)
marginScore := normalizeScore(marginPct, 0, 50) // margin 0-50%
popScore := normalizeScore(popularityScore, 0, 100)
stockScore := stockFactor(stockQty)
return r.config.RelevanceWeight*relScore +
r.config.MarginWeight*marginScore +
r.config.PopularityWeight*popScore +
r.config.StockWeight*stockScore
}
// ReRank melakukan re-ranking terhadap search results
func (r *Ranker) ReRank(products []RankedProduct) []RankedProduct {
scored := make([]scoredProduct, len(products))
for i, p := range products {
compositeScore := r.CompositeScore(
p.Score,
p.Product.MarginPct,
p.Product.PopularityScore,
p.StockQty,
)
scored[i] = scoredProduct{
product: p,
score: compositeScore,
}
}
// Sort by composite score descending
sort.Slice(scored, func(i, j int) bool {
return scored[i].score > scored[j].score
})
result := make([]RankedProduct, len(scored))
for i, s := range scored {
result[i] = s.product
}
return result
}
type scoredProduct struct {
product RankedProduct
score float64
}
// normalizeScore normalizes a value to 0-1 using min-max normalization
func normalizeScore(value, min, max float64) float64 {
if max-min == 0 {
return 0
}
normalized := (value - min) / (max - min)
return math.Max(0, math.Min(1, normalized))
}
// stockFactor: stock yang lebih banyak = skor lebih tinggi
// Tapi tidak linear: stok 5 vs 50 beda tipis, stok 0 vs 5 beda besar
func stockFactor(qty int) float64 {
if qty <= 0 {
return 0
}
// Log scale: 1->0.3, 5->0.6, 50->0.9, 100->1.0
return math.Min(1.0, math.Log10(float64(qty+1))/2.0)
}8. Edge Cases
Edge Case 1: Elasticsearch dan Redis Stock Mismatch
// Reconciliation job untuk memastikan stock di Redis match dengan PostgreSQL
type StockReconciler struct {
pgDB *sql.DB
redis *redis.Client
esClient *elasticsearch.Client
}
func (r *StockReconciler) Reconcile(ctx context.Context, hubID string) ([]ReconciliationDiff, error) {
// 1. Get all product SKUs from PostgreSQL inventory
pgRows, err := r.pgDB.QueryContext(ctx, `
SELECT sku, quantity, reserved
FROM inventory
WHERE hub_id = $1
`, hubID)
if err != nil {
return nil, fmt.Errorf("query pg inventory: %w", err)
}
defer pgRows.Close()
pgStock := make(map[string]string)
for pgRows.Next() {
var sku string
var qty, reserved int
if err := pgRows.Scan(&sku, &qty, &reserved); err != nil {
continue
}
pgStock[sku] = fmt.Sprintf("%d:%d", qty, reserved)
}
// 2. Scan all Redis keys for this hub
iter := r.redis.Scan(ctx, 0, fmt.Sprintf("qcom:stock:%s:*", hubID), 100).Iterator()
var diffs []ReconciliationDiff
for iter.Next(ctx) {
key := iter.Val()
sku := extractSKUFromKey(key, hubID)
redisVal, err := r.redis.Get(ctx, key).Result()
if err != nil {
continue
}
pgVal, exists := pgStock[sku]
if !exists {
diffs = append(diffs, ReconciliationDiff{
SKU: sku,
Field: "exists",
Redis: redisVal,
PG: "not_found",
Action: "delete_redis",
})
continue
}
if redisVal != pgVal {
diffs = append(diffs, ReconciliationDiff{
SKU: sku,
Field: "value",
Redis: redisVal,
PG: pgVal,
Action: "update_redis",
})
}
delete(pgStock, sku)
}
// Remaining PG entries are missing from Redis
for sku, val := range pgStock {
diffs = append(diffs, ReconciliationDiff{
SKU: sku,
Field: "missing",
Redis: "not_found",
PG: val,
Action: "add_redis",
})
}
// Auto-fix
for _, d := range diffs {
switch d.Action {
case "add_redis", "update_redis":
r.redis.Set(ctx, stockKey(hubID, d.SKU), d.PG, 0)
case "delete_redis":
r.redis.Del(ctx, stockKey(hubID, d.SKU))
}
}
return diffs, nil
}
type ReconciliationDiff struct {
SKU string
Field string
Redis string
PG string
Action string
}Edge Case 2: Zero Results Query
// Fallback search ketika query menghasilkan 0 hasil
func (s *SearchService) SearchWithFallback(ctx context.Context, req SearchRequest) (*SearchResult, error) {
result, err := s.Search(ctx, req)
if err != nil {
return nil, err
}
if result.Total > 0 {
return result, nil
}
// Fallback 1: deduplicate and try with less strict fuzziness
fallbackReq := req
fallbackReq.Query = simplifyQuery(req.Query) // remove special chars, normalize
if fallbackReq.Query != req.Query {
result, err = s.Search(ctx, fallbackReq)
if err == nil && result.Total > 0 {
return result, nil
}
}
// Fallback 2: category browse instead of search
// Show top products in default category
result, err = s.Search(ctx, SearchRequest{
HubID: req.HubID,
Page: 1,
Size: req.Size,
})
if err == nil && result.Total > 0 {
// Mark as fallback
return result, nil
}
// Fallback 3: return popular products
result, err = s.Search(ctx, SearchRequest{
HubID: req.HubID,
SortBy: "popularity",
Page: 1,
Size: req.Size,
})
if err == nil && result.Total > 0 {
return result, nil
}
// Last resort: empty result
return result, nil
}
func simplifyQuery(q string) string {
// Remove special characters, normalize whitespace
var result strings.Builder
for _, r := range q {
if r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' || r >= '0' && r <= '9' || r == ' ' {
result.WriteRune(r)
}
}
return strings.TrimSpace(result.String())
}Edge Case 3: Race Condition Stock Update
// Optimistic lock via Redis Lua untuk decrement stock
const atomicReserveScript = `
local key = KEYS[1]
local orderQty = tonumber(ARGV[1])
local orderID = ARGV[2]
local lockKey = key .. ":lock:" .. orderID
-- Check duplicate
if redis.call("EXISTS", lockKey) == 1 then
return -3 -- duplicate order
end
local stock = redis.call("GET", key)
if not stock then
return -1 -- no stock data
end
-- Parse "qty:reserved"
local colon = string.find(stock, ":")
local qty = tonumber(string.sub(stock, 1, colon - 1))
local reserved = tonumber(string.sub(stock, colon + 1))
if not qty or qty - reserved < orderQty then
return -2 -- insufficient
end
reserved = reserved + orderQty
redis.call("SET", key, qty .. ":" .. reserved)
redis.call("SETEX", lockKey, 30, "1") -- 30s lock untuk cegah duplicate
return reserved
`
func (s *StockFilterService) AtomicReserve(ctx context.Context, hubID, sku, orderID string, qty int) error {
key := s.stockKey(hubID, sku)
result, err := redis.NewScript(atomicReserveScript).Run(ctx, s.client,
[]string{key}, qty, orderID).Int()
if err != nil {
return fmt.Errorf("atomic reserve: %w", err)
}
switch result {
case -1:
return fmt.Errorf("stock data not found for %s/%s", hubID, sku)
case -2:
return fmt.Errorf("insufficient stock for %s/%s", hubID, sku)
case -3:
return fmt.Errorf("duplicate order %s for %s/%s", orderID, hubID, sku)
default:
return nil // success
}
}Key Takeaways
Elasticsearch Multi-Match
Gunakan multi_match dengan field boosting (name^3, description, brand^2). Fuzziness AUTO untuk typo tolerance. Prefix_length untuk performa.
Redis Pipeline Stock Filter
Batch stock check via Redis pipeline: 1 round-trip untuk 100 SKU. Query-time filtering karena stock berubah real-time. Format value qty:reserved.
Kafka-driven Index Sync
Catalog mutation events via Kafka, consume di worker, bulk index ke ES. Batch buffer (size + interval) untuk efisiensi. Stock event langsung ke Redis.
Multi-faktor Ranking
Relevance (40%) + Stock availability (25%) + Margin (20%) + Popularity (15%). Weight bisa di-tuning per campaign atau segment.
Completion Suggester
Autocomplete pakai Elasticsearch completion suggester. Pre-computed, sub-10ms. Fuzzy untuk typo. Weight based on popularity.
Fallback Strategy
Zero results: simplify query, fallback ke category browse, then popular products. Jangan pernah return empty page tanpa fallback.
Kesimpulan
Search di Q-commerce menuntut keseimbangan antara:
- Relevansi — Elasticsearch full-text dengan multi-field boosting
- Ketersediaan — Redis pipeline untuk stock check real-time per-hub
- Kecepatan — query-time filtering, not index-time, karena stock berubah terus
- Kekinian — Kafka streaming untuk near-real-time index sync
- Akurasi — reconciliation job untuk deteksi mismatch Redis vs database
Dengan desain ini, search service bisa handle 15.000+ SKU di 50+ hub dengan latency <50ms untuk search dan <10ms untuk autocomplete. Stock filter real-time via Redis pipeline memastikan user hanya melihat produk yang benar-benar tersedia.