Back to Engineering Articles/Search & Catalog 15k SKU Q-Commerce: Relevan + In-Stock Real-time

Search & Catalog 15k SKU Q-Commerce: Relevan + In-Stock Real-time

System design search dan catalog untuk quick commerce dengan 15.000+ SKU. Elasticsearch full-text search dengan multi-match query dan typo tolerance (fuzziness), filter ketersediaan per-hub via Redis pipeline query-time filtering, ranking berbasis relevansi, ketersediaan, margin, dan popularitas, index update via Kafka streaming, dan autocomplete handler. Implementasi Golang lengkap dengan worker concurrency pattern untuk index sync.

Faisal AffanFaisal Affan
6/20/2026

Search & Catalog 15k SKU Q-Commerce: Relevan + In-Stock Real-time

"In Q-commerce, a search that shows out-of-stock items is worse than no search at all. Every millisecond of latency costs orders; every irrelevant result loses customers."

TL;DR

Search di Q-commerce punya tantangan unik: 15.000+ SKU, real-time stock availability per hub, typo tolerance, dan ranking yang harus balance antara relevansi, profit margin, dan popularitas. Artikel ini membangun search service dari nol dengan Elasticsearch multi-match, Redis pipeline stock filter, Kafka-driven index sync, dan autocomplete with completion suggester. Semua kode dalam Go dengan pattern production-grade.


Arsitektur Search & Catalog

Key Design Decisions

  1. Elasticsearch untuk search — full-text, typo tolerance, scoring built-in
  2. Redis untuk stock — real-time per-hub, pipeline batch query, microsecond latency
  3. Kafka untuk index sync — near-real-time update tanpa DB polling
  4. Query-time stock filtering — bukan index-time, karena stock berubah terus

1. Problem Statement: Kenapa Search Q-Commerce Susah?

15.000+ SKU

Bukan jumlah besar untuk e-commerce biasa, tapi untuk Q-commerce yang inventory-nya per-hub, setiap SKU punya stock status unik di 50+ hub.

Real-time Stock

Stok berubah setiap ONG (order not go) atau restock. Elasticsearch index tidak bisa di-update per-transaksi — perlu approach query-time filtering via Redis.

Typo Tolerance

User Q-commerce mengetik cepat di mobile: 'indomie' jadi 'indomei', 'goreng' jadi 'gorengg'. Search harus forgive, bukan exact match.

Multi-faktor Ranking

Ranking tidak bisa relevansi saja. Margin (promosi produk high-margin), popularitas, ketersediaan — semuanya harus di-weight.


2. Elasticsearch Product Index

Index Mapping

config/elasticsearch/product-mapping.json
{
  "settings": {
    "analysis": {
      "analyzer": {
        "product_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": [
            "lowercase",
            "asciifolding",
            "ngram_filter",
            "synonym_filter"
          ]
        },
        "autocomplete_analyzer": {
          "type": "custom",
          "tokenizer": "keyword",
          "filter": ["lowercase", "asciifolding"]
        }
      },
      "filter": {
        "ngram_filter": {
          "type": "ngram",
          "min_gram": 2,
          "max_gram": 20
        },
        "synonym_filter": {
          "type": "synonym",
          "synonyms": [
            "mie, mi, indomie, noodle",
            "goreng, gorengan, fried",
            "susu, milk",
            "kopi, coffe, coffee",
            "teh, tea, the"
          ]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "sku":          { "type": "keyword" },
      "name":         { "type": "text", "analyzer": "product_analyzer", "fields": { "raw": { "type": "keyword" } } },
      "description":  { "type": "text", "analyzer": "product_analyzer" },
      "category_id":  { "type": "keyword" },
      "category_name": { "type": "text", "analyzer": "product_analyzer" },
      "brand":        { "type": "keyword" },
      "price":        { "type": "long" },
      "margin_pct":   { "type": "float" },
      "popularity_score": { "type": "float" },
      "tags":         { "type": "keyword" },
      "image_url":    { "type": "keyword", "index": false },
      "unit":         { "type": "keyword" },
      "is_active":    { "type": "boolean" },
      "created_at":   { "type": "date" },
      "updated_at":   { "type": "date" },
      "suggest": {
        "type": "completion",
        "analyzer": "autocomplete_analyzer"
      }
    }
  }
}

Go: Product Model

internal/search/model/product.go
package model

import "time"

// Product adalah representasi produk di search index
type Product struct {
    SKU             string    `json:"sku"`
    Name            string    `json:"name"`
    Description     string    `json:"description"`
    CategoryID      string    `json:"category_id"`
    CategoryName    string    `json:"category_name"`
    Brand           string    `json:"brand"`
    Price           int64     `json:"price"`
    MarginPct       float64   `json:"margin_pct"`
    PopularityScore float64   `json:"popularity_score"`
    Tags            []string  `json:"tags"`
    ImageURL        string    `json:"image_url"`
    Unit            string    `json:"unit"`
    IsActive        bool      `json:"is_active"`
    CreatedAt       time.Time `json:"created_at"`
    UpdatedAt       time.Time `json:"updated_at"`

    // Completion suggester fields
    SuggestInput    []string `json:"suggest_input,omitempty"`
    SuggestWeight   int      `json:"suggest_weight,omitempty"`
}

// ProductDocument is what's stored in Elasticsearch
type ProductDocument struct {
    SKU             string    `json:"sku"`
    Name            string    `json:"name"`
    Description     string    `json:"description"`
    CategoryID      string    `json:"category_id"`
    CategoryName    string    `json:"category_name"`
    Brand           string    `json:"brand"`
    Price           int64     `json:"price"`
    MarginPct       float64   `json:"margin_pct"`
    PopularityScore float64   `json:"popularity_score"`
    Tags            []string  `json:"tags"`
    ImageURL        string    `json:"image_url"`
    Unit            string    `json:"unit"`
    IsActive        bool      `json:"is_active"`
    CreatedAt       time.Time `json:"created_at"`
    UpdatedAt       time.Time `json:"updated_at"`
    Suggest         SuggestField `json:"suggest"`
}

type SuggestField struct {
    Input  []string `json:"input"`
    Weight int      `json:"weight"`
}

3. Search Service: Go + Elasticsearch

Search Service dengan Multi-Match Query

internal/search/service/search.go
package service

import (
    "context"
    "encoding/json"
    "fmt"
    "strings"
    "time"

    "github.com/elastic/go-elasticsearch/v8"
    "github.com/elastic/go-elasticsearch/v8/esapi"
    "github.com/faisalaffan/qcommerce/internal/search/model"
)

// SearchRequest adalah request dari client
type SearchRequest struct {
    Query    string   `json:"query"`
    HubID    string   `json:"hub_id"`              // required for stock filtering
    Category string   `json:"category,omitempty"`  // filter by category
    Brands   []string `json:"brands,omitempty"`    // filter by brand
    MinPrice int64    `json:"min_price,omitempty"`
    MaxPrice int64    `json:"max_price,omitempty"`
    SortBy   string   `json:"sort_by,omitempty"`   // relevance, price_asc, price_desc, popularity
    Page     int      `json:"page"`
    Size     int      `json:"size"`
}

// SearchResult adalah hasil pencarian yang sudah difilter stock
type SearchResult struct {
    Total       int64              `json:"total"`
    Products    []RankedProduct    `json:"products"`
    Page        int                `json:"page"`
    Size        int                `json:"size"`
    TotalPages  int                `json:"total_pages"`
    TookMs      int64              `json:"took_ms"`
}

type RankedProduct struct {
    Product    model.ProductDocument `json:"product"`
    Score      float64               `json:"score"`
    InStock    bool                  `json:"in_stock"`
    StockQty   int                   `json:"stock_qty,omitempty"`
}

// SearchService handles search operations
type SearchService struct {
    es       *elasticsearch.Client
    stockSvc *StockFilterService
    index    string
}

func NewSearchService(es *elasticsearch.Client, stockSvc *StockFilterService, index string) *SearchService {
    return &SearchService{
        es:       es,
        stockSvc: stockSvc,
        index:    index,
    }
}

// Search melakukan full-text search dengan stock filtering
func (s *SearchService) Search(ctx context.Context, req SearchRequest) (*SearchResult, error) {
    start := time.Now()

    if req.Page <= 0 {
        req.Page = 1
    }
    if req.Size <= 0 || req.Size > 100 {
        req.Size = 20
    }

    // 1. Build multi-match query dengan fuzziness
    query := s.buildSearchQuery(req)

    // 2. Execute search ke Elasticsearch
    from := (req.Page - 1) * req.Size

    searchBody := map[string]interface{}{
        "from": from,
        "size": req.Size,
        "query": query,
        "_source": true,
    }

    // Add sorting
    if req.SortBy != "" && req.SortBy != "relevance" {
        searchBody["sort"] = s.buildSort(req.SortBy)
    }

    body, err := json.Marshal(searchBody)
    if err != nil {
        return nil, fmt.Errorf("marshal search body: %w", err)
    }

    res, err := s.es.Search(
        s.es.Search.WithContext(ctx),
        s.es.Search.WithIndex(s.index),
        s.es.Search.WithBody(strings.NewReader(string(body))),
    )
    if err != nil {
        return nil, fmt.Errorf("elasticsearch search: %w", err)
    }
    defer res.Body.Close()

    if res.IsError() {
        return nil, fmt.Errorf("elasticsearch error: %s", res.String())
    }

    // 3. Parse response
    var esResponse struct {
        Hits struct {
            Total struct {
                Value int64 `json:"value"`
            } `json:"total"`
            Hits []struct {
                Score float64                `json:"_score"`
                Source model.ProductDocument `json:"_source"`
            } `json:"hits"`
        } `json:"hits"`
    }

    if err := json.NewDecoder(res.Body).Decode(&esResponse); err != nil {
        return nil, fmt.Errorf("decode search response: %w", err)
    }

    // 4. Extract SKUs for stock check
    var products []model.ProductDocument
    skus := make([]string, 0, len(esResponse.Hits.Hits))
    scores := make(map[string]float64)

    for _, hit := range esResponse.Hits.Hits {
        products = append(products, hit.Source)
        skus = append(skus, hit.Source.SKU)
        scores[hit.Source.SKU] = hit.Score
    }

    // 5. Batch stock check via Redis pipeline
    stockMap, err := s.stockSvc.BatchCheckStock(ctx, req.HubID, skus)
    if err != nil {
        // Log error but still return products (without stock info)
        fmt.Printf("stock check failed: %v\n", err)
        stockMap = make(map[string]StockInfo)
    }

    // 6. Rank and return
    ranked := make([]RankedProduct, 0, len(products))
    for _, p := range products {
        stock := stockMap[p.SKU]
        ranked = append(ranked, RankedProduct{
            Product:  p,
            Score:    scores[p.SKU],
            InStock:  stock.Available,
            StockQty: stock.Quantity,
        })
    }

    // 7. Filter out-of-stock if configured
    // In Q-commerce, we typically show in-stock first, then out-of-stock
    ranked = s.applyStockPriority(ranked)

    totalPages := int(esResponse.Hits.Total.Value) / req.Size
    if int(esResponse.Hits.Total.Value)%req.Size > 0 {
        totalPages++
    }

    return &SearchResult{
        Total:      esResponse.Hits.Total.Value,
        Products:   ranked,
        Page:       req.Page,
        Size:       req.Size,
        TotalPages: totalPages,
        TookMs:     time.Since(start).Milliseconds(),
    }, nil
}

// buildSearchQuery membangun multi-match query dengan fuzziness
func (s *SearchService) buildSearchQuery(req SearchRequest) map[string]interface{} {
    must := make([]map[string]interface{}, 0)
    filter := make([]map[string]interface{}, 0)

    // Full-text search dengan multi-match
    if req.Query != "" {
        must = append(must, map[string]interface{}{
            "multi_match": map[string]interface{}{
                "query":  req.Query,
                "fields": []string{"name^3", "description", "category_name^2", "brand^2", "tags"},
                "type":   "best_fields",
                "fuzziness": "AUTO",
                "prefix_length": 2,
                "max_expansions": 50,
                "tie_breaker": 0.3,
            },
        })
    } else {
        // No query: return all (browse mode)
        must = append(must, map[string]interface{}{
            "match_all": map[string]interface{}{},
        })
    }

    // Filters
    if req.Category != "" {
        filter = append(filter, map[string]interface{}{
            "term": map[string]interface{}{"category_id": req.Category},
        })
    }

    if len(req.Brands) > 0 {
        filter = append(filter, map[string]interface{}{
            "terms": map[string]interface{}{"brand": req.Brands},
        })
    }

    if req.MinPrice > 0 || req.MaxPrice > 0 {
        rangeFilter := make(map[string]interface{})
        if req.MinPrice > 0 {
            rangeFilter["gte"] = req.MinPrice
        }
        if req.MaxPrice > 0 {
            rangeFilter["lte"] = req.MaxPrice
        }
        filter = append(filter, map[string]interface{}{
            "range": map[string]interface{}{"price": rangeFilter},
        })
    }

    // Only active products
    filter = append(filter, map[string]interface{}{
        "term": map[string]interface{}{"is_active": true},
    })

    query := map[string]interface{}{
        "bool": map[string]interface{}{
            "must":   must,
            "filter": filter,
        },
    }

    return query
}

func (s *SearchService) buildSort(sortBy string) []map[string]interface{} {
    switch sortBy {
    case "price_asc":
        return []map[string]interface{}{
            {"price": map[string]interface{}{"order": "asc"}},
        }
    case "price_desc":
        return []map[string]interface{}{
            {"price": map[string]interface{}{"order": "desc"}},
        }
    case "popularity":
        return []map[string]interface{}{
            {"popularity_score": map[string]interface{}{"order": "desc"}},
        }
    default:
        return nil // relevance = default Elasticsearch _score sorting
    }
}

// applyStockPriority: in-stock first, then out-of-stock
func (s *SearchService) applyStockPriority(products []RankedProduct) []RankedProduct {
    inStock := make([]RankedProduct, 0, len(products))
    outOfStock := make([]RankedProduct, 0, len(products))

    for _, p := range products {
        if p.InStock {
            inStock = append(inStock, p)
        } else {
            outOfStock = append(outOfStock, p)
        }
    }

    return append(inStock, outOfStock...)
}

4. Stock Filter: Redis Pipeline

Problem: setiap search result (20-100 SKU) harus di-cek stock-nya per hub. Kalau 1-by-1, latency-nya tidak acceptable. Solusi: Redis pipeline.

Redis Pipeline vs MGET

Pipeline mengirimkan semua command dalam satu round-trip — bukan N round-trip. Untuk 100 SKU, latency turun dari ~100ms (100x1ms) menjadi ~5ms (1 batch). Alternatif: MGET — tapi pipeline lebih fleksibel karena kita bisa mix tipe command.

internal/search/service/stock.go
package service

import (
    "context"
    "fmt"
    "strconv"
    "time"

    "github.com/redis/go-redis/v9"
)

type StockInfo struct {
    SKU        string `json:"sku"`
    Available  bool   `json:"available"`
    Quantity   int    `json:"quantity"`
    Reserved   int    `json:"reserved"`
}

// StockFilterService handles real-time stock checking via Redis
type StockFilterService struct {
    client    *redis.Client
    keyPrefix string
    ttl       time.Duration // TTL for stock cache
}

func NewStockFilterService(client *redis.Client, keyPrefix string) *StockFilterService {
    return &StockFilterService{
        client:    client,
        keyPrefix: keyPrefix,
    }
}

// stockKey generates Redis key for a specific hub+SKU stock
func (s *StockFilterService) stockKey(hubID, sku string) string {
    return fmt.Sprintf("%s:stock:%s:%s", s.keyPrefix, hubID, sku)
}

// BatchCheckStock memeriksa stock untuk banyak SKU dalam satu pipeline
func (s *StockFilterService) BatchCheckStock(ctx context.Context, hubID string, skus []string) (map[string]StockInfo, error) {
    if len(skus) == 0 {
        return map[string]StockInfo{}, nil
    }

    pipe := s.client.Pipeline()

    // Schedule all commands
    cmds := make(map[string]*redis.StringCmd)
    for _, sku := range skus {
        key := s.stockKey(hubID, sku)
        cmds[sku] = pipe.Get(ctx, key)
    }

    // Execute pipeline (1 round trip)
    _, err := pipe.Exec(ctx)
    if err != nil && err != redis.Nil {
        // Pipeline bisa partially fail; Redis returns nil for missing keys
        // We handle each key result individually
        fmt.Printf("pipeline exec error (non-fatal): %v\n", err)
    }

    // Parse results
    result := make(map[string]StockInfo, len(skus))
    for _, sku := range skus {
        cmd := cmds[sku]
        val, err := cmd.Result()
        if err == redis.Nil {
            // Stock key not found — treat as out of stock
            result[sku] = StockInfo{
                SKU:       sku,
                Available: false,
                Quantity:  0,
            }
            continue
        }
        if err != nil {
            // Other error — treat as unknown stock
            result[sku] = StockInfo{
                SKU:       sku,
                Available: false,
                Quantity:  0,
            }
            continue
        }

        // Parse Redis value: format "qty:reserved" or just "qty"
        qty, res := s.parseStockValue(val)
        result[sku] = StockInfo{
            SKU:       sku,
            Available: qty > 0 && qty > res,
            Quantity:  qty,
            Reserved:  res,
        }
    }

    return result, nil
}

// parseStockValue parses Redis stock value
// Format: "quantity:reserved" or just "quantity"
func (s *StockFilterService) parseStockValue(val string) (int, int) {
    parts := split2(val, ":")
    if len(parts) < 2 {
        qty, err := strconv.Atoi(parts[0])
        if err != nil {
            return 0, 0
        }
        return qty, 0
    }

    qty, err := strconv.Atoi(parts[0])
    if err != nil {
        qty = 0
    }
    reserved, err := strconv.Atoi(parts[1])
    if err != nil {
        reserved = 0
    }

    return qty, reserved
}

// GetSingleStock checks stock for one SKU
func (s *StockFilterService) GetSingleStock(ctx context.Context, hubID, sku string) (*StockInfo, error) {
    key := s.stockKey(hubID, sku)
    val, err := s.client.Get(ctx, key).Result()
    if err == redis.Nil {
        return &StockInfo{SKU: sku, Available: false, Quantity: 0}, nil
    }
    if err != nil {
        return nil, fmt.Errorf("get stock %s/%s: %w", hubID, sku, err)
    }

    qty, reserved := s.parseStockValue(val)
    return &StockInfo{
        SKU:       sku,
        Available: qty > 0 && qty > reserved,
        Quantity:  qty,
        Reserved:  reserved,
    }, nil
}

// UpdateStock updates stock in Redis (called by stock mutation events)
func (s *StockFilterService) UpdateStock(ctx context.Context, hubID, sku string, quantity int, ttl time.Duration) error {
    key := s.stockKey(hubID, sku)
    val := fmt.Sprintf("%d:0", quantity)
    return s.client.Set(ctx, key, val, ttl).Err()
}

// DecrementStock reduces stock on order placement (via Lua for atomicity)
const decrementStockScript = `
    local key = KEYS[1]
    local qty = tonumber(ARGV[1])
    local current = redis.call("GET", key)

    if not current then
        return -1  -- no stock key
    end

    local parts = {}
    for w in current:gmatch("([^:]+)") do
        table.insert(parts, w)
    end

    local available = tonumber(parts[1]) or 0
    local reserved = tonumber(parts[2]) or 0

    if available - reserved < qty then
        return -2  -- insufficient stock
    end

    reserved = reserved + qty
    redis.call("SET", key, available .. ":" .. reserved)
    return reserved
`

func (s *StockFilterService) DecrementStock(ctx context.Context, hubID, sku string, quantity int) (int, error) {
    key := s.stockKey(hubID, sku)
    result, err := redis.NewScript(decrementStockScript).Run(ctx, s.client,
        []string{key}, quantity).Int()
    if err != nil {
        return 0, fmt.Errorf("decrement stock %s/%s: %w", hubID, sku, err)
    }
    if result < 0 {
        return 0, fmt.Errorf("insufficient stock for %s/%s", hubID, sku)
    }
    return result, nil
}

// split2 splits a string into at most 2 parts
func split2(s, sep string) []string {
    parts := strings.SplitN(s, sep, 2)
    // Use local empty check since this is a private utility
    if len(parts) == 0 {
        return []string{""}
    }
    return parts
}


5. Index Sync Worker: Kafka Consumer

Product catalog berubah terus: harga di-update, produk baru masuk, deskripsi diubah. Kita perlu sync ke Elasticsearch secara near-real-time.

internal/search/worker/index_sync.go
package worker

import (
    "context"
    "encoding/json"
    "fmt"
    "log"
    "time"

    "github.com/elastic/go-elasticsearch/v8"
    "github.com/elastic/go-elasticsearch/v8/esapi"
    "github.com/segmentio/kafka-go"
    "github.com/faisalaffan/qcommerce/internal/search/model"
)

// CatalogEventType mendefinisikan tipe event catalog
type CatalogEventType string

const (
    EventProductCreated CatalogEventType = "product.created"
    EventProductUpdated CatalogEventType = "product.updated"
    EventProductDeleted CatalogEventType = "product.deleted"
    EventStockChanged   CatalogEventType = "stock.changed"
    EventPriceChanged   CatalogEventType = "price.changed"
)

// CatalogEvent adalah event dari Kafka
type CatalogEvent struct {
    EventType CatalogEventType `json:"event_type"`
    SKU       string           `json:"sku"`
    Timestamp time.Time        `json:"timestamp"`
    Data      json.RawMessage  `json:"data"`
}

// IndexSyncWorker handles syncing catalog changes to Elasticsearch
type IndexSyncWorker struct {
    es            *elasticsearch.Client
    kafkaReader   *kafka.Reader
    catalogReader CatalogReader
    index         string
    batchSize     int
    flushInterval time.Duration
}

// CatalogReader interface untuk membaca product dari database
type CatalogReader interface {
    GetProduct(ctx context.Context, sku string) (*model.Product, error)
    GetProductsByIDs(ctx context.Context, skus []string) ([]model.Product, error)
}

func NewIndexSyncWorker(
    es *elasticsearch.Client,
    kafkaReader *kafka.Reader,
    catalogReader CatalogReader,
    index string,
    batchSize int,
    flushInterval time.Duration,
) *IndexSyncWorker {
    return &IndexSyncWorker{
        es:            es,
        kafkaReader:   kafkaReader,
        catalogReader: catalogReader,
        index:         index,
        batchSize:     batchSize,
        flushInterval: flushInterval,
    }
}

// Start mulai consume Kafka events
func (w *IndexSyncWorker) Start(ctx context.Context) error {
    log.Printf("index sync worker started: index=%s batch=%d flush=%v",
        w.index, w.batchSize, w.flushInterval)

    // Batch buffer for bulk indexing
    type pendingDoc struct {
        action string // "index" or "delete"
        sku    string
    }

    pending := make(map[string]pendingDoc)
    lastFlush := time.Now()

    flushBatch := func() error {
        if len(pending) == 0 {
            return nil
        }

        var bulkBody string
        for sku, doc := range pending {
            switch doc.action {
            case "delete":
                bulkBody += fmt.Sprintf(`{"delete":{"_index":"%s","_id":"%s"}}`+"\n", w.index, sku)
            case "index":
                // Read full product from DB
                product, err := w.catalogReader.GetProduct(ctx, sku)
                if err != nil {
                    log.Printf("get product %s for indexing: %v", sku, err)
                    continue
                }
                doc := toProductDocument(product)
                data, _ := json.Marshal(doc)
                bulkBody += fmt.Sprintf(`{"index":{"_index":"%s","_id":"%s"}}`+"\n", w.index, sku)
                bulkBody += string(data) + "\n"
            }
        }

        if bulkBody == "" {
            return nil
        }

        res, err := w.es.Bulk(strings.NewReader(bulkBody))
        if err != nil {
            return fmt.Errorf("bulk index: %w", err)
        }
        defer res.Body.Close()

        if res.IsError() {
            return fmt.Errorf("bulk index error: %s", res.String())
        }

        // Parse response for errors
        var bulkResponse struct {
            Errors bool `json:"errors"`
            Items  []struct {
                Index struct {
                    Error interface{} `json:"error,omitempty"`
                } `json:"index"`
            } `json:"items"`
        }
        if err := json.NewDecoder(res.Body).Decode(&bulkResponse); err != nil {
            return fmt.Errorf("decode bulk response: %w", err)
        }

        if bulkResponse.Errors {
            // Log individual errors but don't fail the batch
            for _, item := range bulkResponse.Items {
                if item.Index.Error != nil {
                    log.Printf("bulk index item error: %+v", item.Index.Error)
                }
            }
        }

        log.Printf("index sync: flushed %d docs to %s", len(pending), w.index)
        pending = make(map[string]pendingDoc)
        lastFlush = time.Now()
        return nil
    }

    defer flushBatch() // flush remaining on shutdown

    for {
        select {
        case <-ctx.Done():
            log.Println("index sync worker shutting down")
            return ctx.Err()
        default:
        }

        msg, err := w.kafkaReader.ReadMessage(ctx)
        if err != nil {
            if ctx.Err() != nil {
                return ctx.Err()
            }
            log.Printf("read kafka message: %v", err)
            continue
        }

        var event CatalogEvent
        if err := json.Unmarshal(msg.Value, &event); err != nil {
            log.Printf("unmarshal catalog event: %v", err)
            continue
        }

        // Map event type to Elasticsearch action
        switch event.EventType {
        case EventProductCreated, EventProductUpdated, EventPriceChanged:
            pending[event.SKU] = pendingDoc{action: "index", sku: event.SKU}
        case EventProductDeleted:
            pending[event.SKU] = pendingDoc{action: "delete", sku: event.SKU}
        case EventStockChanged:
            // Stock is handled by Redis, not ES index
            // But we might want to update a "has_stock" flag
            continue
        }

        // Flush if batch full or interval reached
        if len(pending) >= w.batchSize || time.Since(lastFlush) >= w.flushInterval {
            if err := flushBatch(); err != nil {
                log.Printf("flush batch: %v", err)
            }
        }
    }
}

// toProductDocument converts Product to indexable document
func toProductDocument(p *model.Product) model.ProductDocument {
    suggestInput := make([]string, 0, 2+len(p.Tags))
    suggestInput = append(suggestInput, p.Name, p.Brand)
    suggestInput = append(suggestInput, p.Tags...)

    return model.ProductDocument{
        SKU:             p.SKU,
        Name:            p.Name,
        Description:     p.Description,
        CategoryID:      p.CategoryID,
        CategoryName:    p.CategoryName,
        Brand:           p.Brand,
        Price:           p.Price,
        MarginPct:       p.MarginPct,
        PopularityScore: p.PopularityScore,
        Tags:            p.Tags,
        ImageURL:        p.ImageURL,
        Unit:            p.Unit,
        IsActive:        p.IsActive,
        CreatedAt:       p.CreatedAt,
        UpdatedAt:       p.UpdatedAt,
        Suggest: model.SuggestField{
            Input:  suggestInput,
            Weight: int(p.PopularityScore * 100),
        },
    }
}


6. Autocomplete Handler

internal/search/handler/autocomplete.go
package handler

import (
    "context"
    "encoding/json"
    "fmt"
    "net/http"
    "strings"

    "github.com/elastic/go-elasticsearch/v8"
    "github.com/elastic/go-elasticsearch/v8/esapi"
)

// AutocompleteHandler handles autocomplete suggestions
type AutocompleteHandler struct {
    es    *elasticsearch.Client
    index string
}

func NewAutocompleteHandler(es *elasticsearch.Client, index string) *AutocompleteHandler {
    return &AutocompleteHandler{es: es, index: index}
}

// AutocompleteRequest is the autocomplete query
type AutocompleteRequest struct {
    Query string `json:"query"`
    Size  int    `json:"size"`
    HubID string `json:"hub_id,omitempty"` // optional for stock-aware suggestions
}

// AutocompleteResponse is the autocomplete result
type AutocompleteResponse struct {
    Suggestions []Suggestion `json:"suggestions"`
    TookMs      int64        `json:"took_ms"`
}

type Suggestion struct {
    Text      string  `json:"text"`
    Score     float64 `json:"score"`
    SKU       string  `json:"sku,omitempty"`
    Category  string  `json:"category,omitempty"`
    ImageURL  string  `json:"image_url,omitempty"`
    Price     int64   `json:"price,omitempty"`
    InStock   bool    `json:"in_stock,omitempty"`
}

// HandleAutocomplete HTTP handler
func (h *AutocompleteHandler) HandleAutocomplete(w http.ResponseWriter, r *http.Request) {
    query := r.URL.Query().Get("q")
    if query == "" || len(query) < 2 {
        writeJSON(w, http.StatusBadRequest, map[string]string{
            "error": "query must be at least 2 characters",
        })
        return
    }

    size := 8 // default 8 suggestions
    hubID := r.URL.Query().Get("hub_id")

    if s := r.URL.Query().Get("size"); s != "" {
        if parsed, err := parseInt(s); err == nil && parsed > 0 && parsed <= 20 {
            size = parsed
        }
    }

    result, err := h.Autocomplete(r.Context(), AutocompleteRequest{
        Query: query,
        Size:  size,
        HubID: hubID,
    })
    if err != nil {
        writeJSON(w, http.StatusInternalServerError, map[string]string{
            "error": fmt.Sprintf("autocomplete: %v", err),
        })
        return
    }

    writeJSON(w, http.StatusOK, result)
}

// Autocomplete queries Elasticsearch Completion Suggester
func (h *AutocompleteHandler) Autocomplete(ctx context.Context, req AutocompleteRequest) (*AutocompleteResponse, error) {
    start := time.Now()

    // Build completion suggester query
    suggestBody := map[string]interface{}{
        "suggest": map[string]interface{}{
            "product_suggest": map[string]interface{}{
                "prefix": req.Query,
                "completion": map[string]interface{}{
                    "field": "suggest",
                    "size":  req.Size,
                    "fuzzy": map[string]interface{}{
                        "fuzziness": "AUTO",
                        "prefix_length": 1,
                    },
                },
            },
        },
        "_source": []string{"sku", "name", "category_name", "image_url", "price", "popularity_score"},
    }

    body, err := json.Marshal(suggestBody)
    if err != nil {
        return nil, fmt.Errorf("marshal suggest body: %w", err)
    }

    res, err := h.es.Search(
        h.es.Search.WithContext(ctx),
        h.es.Search.WithIndex(h.index),
        h.es.Search.WithBody(strings.NewReader(string(body))),
    )
    if err != nil {
        return nil, fmt.Errorf("elasticsearch suggest: %w", err)
    }
    defer res.Body.Close()

    if res.IsError() {
        return nil, fmt.Errorf("elasticsearch error: %s", res.String())
    }

    // Parse suggest response
    var esResponse struct {
        Suggest map[string][]struct {
            Options []struct {
                Text    string  `json:"text"`
                Score   float64 `json:"_score"`
                Source  struct {
                    SKU          string `json:"sku"`
                    Name         string `json:"name"`
                    CategoryName string `json:"category_name"`
                    ImageURL     string `json:"image_url"`
                    Price        int64  `json:"price"`
                } `json:"_source"`
            } `json:"options"`
        } `json:"suggest"`
    }

    if err := json.NewDecoder(res.Body).Decode(&esResponse); err != nil {
        return nil, fmt.Errorf("decode suggest response: %w", err)
    }

    // Build suggestions
    suggestions := make([]Suggestion, 0)
    if suggests, ok := esResponse.Suggest["product_suggest"]; ok && len(suggests) > 0 {
        for _, opt := range suggests[0].Options {
            suggestions = append(suggestions, Suggestion{
                Text:     opt.Source.Name,
                Score:    opt.Score,
                SKU:      opt.Source.SKU,
                Category: opt.Source.CategoryName,
                ImageURL: opt.Source.ImageURL,
                Price:    opt.Source.Price,
            })
        }
    }

    return &AutocompleteResponse{
        Suggestions: suggestions,
        TookMs:      time.Since(start).Milliseconds(),
    }, nil
}

// HTTP helpers
func writeJSON(w http.ResponseWriter, status int, data interface{}) {
    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(status)
    json.NewEncoder(w).Encode(data)
}

func parseInt(s string) (int, error) {
    var n int
    for _, r := range s {
        if r < '0' || r > '9' {
            return 0, fmt.Errorf("not a number: %s", s)
        }
        n = n*10 + int(r-'0')
    }
    return n, nil
}

Completion Suggester vs Search-as-you-type

Elasticsearch punya dua fitur autocomplete: Completion Suggester (berdasarkan prefix, sangat cepat, pre-computed) dan Search-as-you-type (field type yang mengoptimalkan query untuk partial matching). Completion Suggester lebih cocok karena: (1) response time <10ms, (2) bisa pakai fuzzy untuk typo, (3) weight-based ranking. Kekurangannya: perlu pre-build suggest input, tidak real-time untuk kata baru.


7. Ranking: Relevance + Availability + Margin + Popularity

Ranking di Q-commerce tidak bisa pure relevansi Elasticsearch. Kita perlu multi-faktor scoring:

internal/search/service/ranking.go
package service

import (
    "math"
)

// RankingConfig untuk weighting
type RankingConfig struct {
    RelevanceWeight  float64 `json:"relevance_weight"`   // 0.4
    StockWeight      float64 `json:"stock_weight"`        // 0.25
    MarginWeight     float64 `json:"margin_weight"`       // 0.2
    PopularityWeight float64 `json:"popularity_weight"`   // 0.15
}

// DefaultRankingConfig adalah default weights
func DefaultRankingConfig() RankingConfig {
    return RankingConfig{
        RelevanceWeight:  0.40,
        StockWeight:      0.25,
        MarginWeight:     0.20,
        PopularityWeight: 0.15,
    }
}

// Ranker melakukan re-ranking products dengan multi-faktor
type Ranker struct {
    config RankingConfig
}

func NewRanker(config RankingConfig) *Ranker {
    return &Ranker{config: config}
}

// CompositeScore menghitung skor gabungan
func (r *Ranker) CompositeScore(
    relevanceScore float64,
    marginPct float64,
    popularityScore float64,
    stockQty int,
) float64 {
    // Normalize each factor to 0-1 range
    relScore := normalizeScore(relevanceScore, 0, 10)
    marginScore := normalizeScore(marginPct, 0, 50)   // margin 0-50%
    popScore := normalizeScore(popularityScore, 0, 100)
    stockScore := stockFactor(stockQty)

    return r.config.RelevanceWeight*relScore +
        r.config.MarginWeight*marginScore +
        r.config.PopularityWeight*popScore +
        r.config.StockWeight*stockScore
}

// ReRank melakukan re-ranking terhadap search results
func (r *Ranker) ReRank(products []RankedProduct) []RankedProduct {
    scored := make([]scoredProduct, len(products))

    for i, p := range products {
        compositeScore := r.CompositeScore(
            p.Score,
            p.Product.MarginPct,
            p.Product.PopularityScore,
            p.StockQty,
        )
        scored[i] = scoredProduct{
            product: p,
            score:   compositeScore,
        }
    }

    // Sort by composite score descending
    sort.Slice(scored, func(i, j int) bool {
        return scored[i].score > scored[j].score
    })

    result := make([]RankedProduct, len(scored))
    for i, s := range scored {
        result[i] = s.product
    }
    return result
}

type scoredProduct struct {
    product RankedProduct
    score   float64
}

// normalizeScore normalizes a value to 0-1 using min-max normalization
func normalizeScore(value, min, max float64) float64 {
    if max-min == 0 {
        return 0
    }
    normalized := (value - min) / (max - min)
    return math.Max(0, math.Min(1, normalized))
}

// stockFactor: stock yang lebih banyak = skor lebih tinggi
// Tapi tidak linear: stok 5 vs 50 beda tipis, stok 0 vs 5 beda besar
func stockFactor(qty int) float64 {
    if qty <= 0 {
        return 0
    }
    // Log scale: 1->0.3, 5->0.6, 50->0.9, 100->1.0
    return math.Min(1.0, math.Log10(float64(qty+1))/2.0)
}

8. Edge Cases

Edge Case 1: Elasticsearch dan Redis Stock Mismatch

// Reconciliation job untuk memastikan stock di Redis match dengan PostgreSQL
type StockReconciler struct {
    pgDB     *sql.DB
    redis    *redis.Client
    esClient *elasticsearch.Client
}

func (r *StockReconciler) Reconcile(ctx context.Context, hubID string) ([]ReconciliationDiff, error) {
    // 1. Get all product SKUs from PostgreSQL inventory
    pgRows, err := r.pgDB.QueryContext(ctx, `
        SELECT sku, quantity, reserved
        FROM inventory
        WHERE hub_id = $1
    `, hubID)
    if err != nil {
        return nil, fmt.Errorf("query pg inventory: %w", err)
    }
    defer pgRows.Close()

    pgStock := make(map[string]string)
    for pgRows.Next() {
        var sku string
        var qty, reserved int
        if err := pgRows.Scan(&sku, &qty, &reserved); err != nil {
            continue
        }
        pgStock[sku] = fmt.Sprintf("%d:%d", qty, reserved)
    }

    // 2. Scan all Redis keys for this hub
    iter := r.redis.Scan(ctx, 0, fmt.Sprintf("qcom:stock:%s:*", hubID), 100).Iterator()

    var diffs []ReconciliationDiff
    for iter.Next(ctx) {
        key := iter.Val()
        sku := extractSKUFromKey(key, hubID)

        redisVal, err := r.redis.Get(ctx, key).Result()
        if err != nil {
            continue
        }

        pgVal, exists := pgStock[sku]
        if !exists {
            diffs = append(diffs, ReconciliationDiff{
                SKU:     sku,
                Field:   "exists",
                Redis:   redisVal,
                PG:      "not_found",
                Action:  "delete_redis",
            })
            continue
        }

        if redisVal != pgVal {
            diffs = append(diffs, ReconciliationDiff{
                SKU:    sku,
                Field:  "value",
                Redis:  redisVal,
                PG:     pgVal,
                Action: "update_redis",
            })
        }

        delete(pgStock, sku)
    }

    // Remaining PG entries are missing from Redis
    for sku, val := range pgStock {
        diffs = append(diffs, ReconciliationDiff{
            SKU:    sku,
            Field:  "missing",
            Redis:  "not_found",
            PG:     val,
            Action: "add_redis",
        })
    }

    // Auto-fix
    for _, d := range diffs {
        switch d.Action {
        case "add_redis", "update_redis":
            r.redis.Set(ctx, stockKey(hubID, d.SKU), d.PG, 0)
        case "delete_redis":
            r.redis.Del(ctx, stockKey(hubID, d.SKU))
        }
    }

    return diffs, nil
}

type ReconciliationDiff struct {
    SKU    string
    Field  string
    Redis  string
    PG     string
    Action string
}

Edge Case 2: Zero Results Query

// Fallback search ketika query menghasilkan 0 hasil
func (s *SearchService) SearchWithFallback(ctx context.Context, req SearchRequest) (*SearchResult, error) {
    result, err := s.Search(ctx, req)
    if err != nil {
        return nil, err
    }

    if result.Total > 0 {
        return result, nil
    }

    // Fallback 1: deduplicate and try with less strict fuzziness
    fallbackReq := req
    fallbackReq.Query = simplifyQuery(req.Query) // remove special chars, normalize

    if fallbackReq.Query != req.Query {
        result, err = s.Search(ctx, fallbackReq)
        if err == nil && result.Total > 0 {
            return result, nil
        }
    }

    // Fallback 2: category browse instead of search
    // Show top products in default category
    result, err = s.Search(ctx, SearchRequest{
        HubID: req.HubID,
        Page:  1,
        Size:  req.Size,
    })
    if err == nil && result.Total > 0 {
        // Mark as fallback
        return result, nil
    }

    // Fallback 3: return popular products
    result, err = s.Search(ctx, SearchRequest{
        HubID:  req.HubID,
        SortBy: "popularity",
        Page:   1,
        Size:   req.Size,
    })
    if err == nil && result.Total > 0 {
        return result, nil
    }

    // Last resort: empty result
    return result, nil
}

func simplifyQuery(q string) string {
    // Remove special characters, normalize whitespace
    var result strings.Builder
    for _, r := range q {
        if r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' || r >= '0' && r <= '9' || r == ' ' {
            result.WriteRune(r)
        }
    }
    return strings.TrimSpace(result.String())
}

Edge Case 3: Race Condition Stock Update

// Optimistic lock via Redis Lua untuk decrement stock
const atomicReserveScript = `
    local key = KEYS[1]
    local orderQty = tonumber(ARGV[1])
    local orderID = ARGV[2]
    local lockKey = key .. ":lock:" .. orderID

    -- Check duplicate
    if redis.call("EXISTS", lockKey) == 1 then
        return -3  -- duplicate order
    end

    local stock = redis.call("GET", key)
    if not stock then
        return -1  -- no stock data
    end

    -- Parse "qty:reserved"
    local colon = string.find(stock, ":")
    local qty = tonumber(string.sub(stock, 1, colon - 1))
    local reserved = tonumber(string.sub(stock, colon + 1))

    if not qty or qty - reserved < orderQty then
        return -2  -- insufficient
    end

    reserved = reserved + orderQty
    redis.call("SET", key, qty .. ":" .. reserved)
    redis.call("SETEX", lockKey, 30, "1")  -- 30s lock untuk cegah duplicate

    return reserved
`

func (s *StockFilterService) AtomicReserve(ctx context.Context, hubID, sku, orderID string, qty int) error {
    key := s.stockKey(hubID, sku)
    result, err := redis.NewScript(atomicReserveScript).Run(ctx, s.client,
        []string{key}, qty, orderID).Int()
    if err != nil {
        return fmt.Errorf("atomic reserve: %w", err)
    }
    switch result {
    case -1:
        return fmt.Errorf("stock data not found for %s/%s", hubID, sku)
    case -2:
        return fmt.Errorf("insufficient stock for %s/%s", hubID, sku)
    case -3:
        return fmt.Errorf("duplicate order %s for %s/%s", orderID, hubID, sku)
    default:
        return nil // success
    }
}


Key Takeaways

Elasticsearch Multi-Match

Gunakan multi_match dengan field boosting (name^3, description, brand^2). Fuzziness AUTO untuk typo tolerance. Prefix_length untuk performa.

Redis Pipeline Stock Filter

Batch stock check via Redis pipeline: 1 round-trip untuk 100 SKU. Query-time filtering karena stock berubah real-time. Format value qty:reserved.

Kafka-driven Index Sync

Catalog mutation events via Kafka, consume di worker, bulk index ke ES. Batch buffer (size + interval) untuk efisiensi. Stock event langsung ke Redis.

Multi-faktor Ranking

Relevance (40%) + Stock availability (25%) + Margin (20%) + Popularity (15%). Weight bisa di-tuning per campaign atau segment.

Completion Suggester

Autocomplete pakai Elasticsearch completion suggester. Pre-computed, sub-10ms. Fuzzy untuk typo. Weight based on popularity.

Fallback Strategy

Zero results: simplify query, fallback ke category browse, then popular products. Jangan pernah return empty page tanpa fallback.


Kesimpulan

Search di Q-commerce menuntut keseimbangan antara:

  1. Relevansi — Elasticsearch full-text dengan multi-field boosting
  2. Ketersediaan — Redis pipeline untuk stock check real-time per-hub
  3. Kecepatan — query-time filtering, not index-time, karena stock berubah terus
  4. Kekinian — Kafka streaming untuk near-real-time index sync
  5. Akurasi — reconciliation job untuk deteksi mismatch Redis vs database

Dengan desain ini, search service bisa handle 15.000+ SKU di 50+ hub dengan latency <50ms untuk search dan <10ms untuk autocomplete. Stock filter real-time via Redis pipeline memastikan user hanya melihat produk yang benar-benar tersedia.


Related Engineering & Tech Articles

Search & Catalog 15k SKU Q-Commerce: Relevan + In-Stock R... | Faisal Affan