Faisalaffan Blog
Faisalaffan Blog
Rekayasa Perangkat Lunak26 menit baca

Search & Catalog 15k SKU Q-Commerce: Relevan + In-Stock Real-time

System design search dan catalog untuk quick commerce dengan 15.000+ SKU. Elasticsearch full-text search dengan multi-match query dan typo tolerance (fuzziness), filter ketersediaan per-hub via Redis pipeline query-time filtering, ranking berbasis relevansi, ketersediaan, margin, dan popularitas, index update via Kafka streaming, dan autocomplete handler. Implementasi Golang lengkap dengan worker concurrency pattern untuk index sync.

Faisal Affan
Faisal Affan
Diterbitkan

Search & Catalog 15k SKU Q-Commerce: Relevan + In-Stock Real-time

Search & Catalog 15k SKU Q-Commerce: Relevan + In-Stock Real-time

"In Q-commerce, a search that shows out-of-stock items is worse than no search at all. Every millisecond of latency costs orders; every irrelevant result loses customers."

TL;DR

Search di Q-commerce punya tantangan unik: 15.000+ SKU, real-time stock availability per hub, typo tolerance, dan ranking yang harus balance antara relevansi, profit margin, dan popularitas. Artikel ini membangun search service dari nol dengan Elasticsearch multi-match, Redis pipeline stock filter, Kafka-driven index sync, dan autocomplete with completion suggester. Semua kode dalam Go dengan pattern production-grade.


Arsitektur Search & Catalog

graph TB
    subgraph "User-Facing"
        A[Mobile App]
        B[Web App]
    end

    subgraph "API Layer"
        C[Search API Gateway]
        D[Autocomplete Handler]
    end

    subgraph "Search Service"
        E[Search Engine]
        F[Ranker]
        G[Stock Filter]
        H[Index Sync Worker]
    end

    subgraph "Data Store"
        I[(Elasticsearch<br/>Product Index)]
        J[(Stock Redis<br/>Per-Hub Stock)]
        K[(Catalog DB<br/>PostgreSQL)]
        L[Kafka<br/>Catalog Events]
    end

    subgraph "Catalog Management"
        M[Catalog Admin]
        N[Vendor Integration]
    end

    A --> C
    B --> C
    A --> D
    B --> D
    C --> E
    D --> I
    E --> I
    E --> G
    G --> J
    F --> E
    H --> I
    K --> H
    L --> H
    M --> K
    N --> L
    H --> L
Key Design Decisions
  1. Elasticsearch untuk search — full-text, typo tolerance, scoring built-in
  2. Redis untuk stock — real-time per-hub, pipeline batch query, microsecond latency
  3. Kafka untuk index sync — near-real-time update tanpa DB polling
  4. Query-time stock filtering — bukan index-time, karena stock berubah terus

1. Problem Statement: Kenapa Search Q-Commerce Susah?


2. Elasticsearch Product Index

Index Mapping

config/elasticsearch/product-mapping.json
{
  "settings": {
    "analysis": {
      "analyzer": {
        "product_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": [
            "lowercase",
            "asciifolding",
            "ngram_filter",
            "synonym_filter"
          ]
        },
        "autocomplete_analyzer": {
          "type": "custom",
          "tokenizer": "keyword",
          "filter": ["lowercase", "asciifolding"]
        }
      },
      "filter": {
        "ngram_filter": {
          "type": "ngram",
          "min_gram": 2,
          "max_gram": 20
        },
        "synonym_filter": {
          "type": "synonym",
          "synonyms": [
            "mie, mi, indomie, noodle",
            "goreng, gorengan, fried",
            "susu, milk",
            "kopi, coffe, coffee",
            "teh, tea, the"
          ]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "sku":          { "type": "keyword" },
      "name":         { "type": "text", "analyzer": "product_analyzer", "fields": { "raw": { "type": "keyword" } } },
      "description":  { "type": "text", "analyzer": "product_analyzer" },
      "category_id":  { "type": "keyword" },
      "category_name": { "type": "text", "analyzer": "product_analyzer" },
      "brand":        { "type": "keyword" },
      "price":        { "type": "long" },
      "margin_pct":   { "type": "float" },
      "popularity_score": { "type": "float" },
      "tags":         { "type": "keyword" },
      "image_url":    { "type": "keyword", "index": false },
      "unit":         { "type": "keyword" },
      "is_active":    { "type": "boolean" },
      "created_at":   { "type": "date" },
      "updated_at":   { "type": "date" },
      "suggest": {
        "type": "completion",
        "analyzer": "autocomplete_analyzer"
      }
    }
  }
}

Go: Product Model

internal/search/model/product.go
package model

import "time"

// Product adalah representasi produk di search index
type Product struct {
    SKU             string    `json:"sku"`
    Name            string    `json:"name"`
    Description     string    `json:"description"`
    CategoryID      string    `json:"category_id"`
    CategoryName    string    `json:"category_name"`
    Brand           string    `json:"brand"`
    Price           int64     `json:"price"`
    MarginPct       float64   `json:"margin_pct"`
    PopularityScore float64   `json:"popularity_score"`
    Tags            []string  `json:"tags"`
    ImageURL        string    `json:"image_url"`
    Unit            string    `json:"unit"`
    IsActive        bool      `json:"is_active"`
    CreatedAt       time.Time `json:"created_at"`
    UpdatedAt       time.Time `json:"updated_at"`

    // Completion suggester fields
    SuggestInput    []string `json:"suggest_input,omitempty"`
    SuggestWeight   int      `json:"suggest_weight,omitempty"`
}

// ProductDocument is what's stored in Elasticsearch
type ProductDocument struct {
    SKU             string    `json:"sku"`
    Name            string    `json:"name"`
    Description     string    `json:"description"`
    CategoryID      string    `json:"category_id"`
    CategoryName    string    `json:"category_name"`
    Brand           string    `json:"brand"`
    Price           int64     `json:"price"`
    MarginPct       float64   `json:"margin_pct"`
    PopularityScore float64   `json:"popularity_score"`
    Tags            []string  `json:"tags"`
    ImageURL        string    `json:"image_url"`
    Unit            string    `json:"unit"`
    IsActive        bool      `json:"is_active"`
    CreatedAt       time.Time `json:"created_at"`
    UpdatedAt       time.Time `json:"updated_at"`
    Suggest         SuggestField `json:"suggest"`
}

type SuggestField struct {
    Input  []string `json:"input"`
    Weight int      `json:"weight"`
}

3. Search Service: Go + Elasticsearch

Search Service dengan Multi-Match Query

internal/search/service/search.go
package service

import (
    "context"
    "encoding/json"
    "fmt"
    "strings"
    "time"

    "github.com/elastic/go-elasticsearch/v8"
    "github.com/elastic/go-elasticsearch/v8/esapi"
    "github.com/faisalaffan/qcommerce/internal/search/model"
)

// SearchRequest adalah request dari client
type SearchRequest struct {
    Query    string   `json:"query"`
    HubID    string   `json:"hub_id"`              // required for stock filtering
    Category string   `json:"category,omitempty"`  // filter by category
    Brands   []string `json:"brands,omitempty"`    // filter by brand
    MinPrice int64    `json:"min_price,omitempty"`
    MaxPrice int64    `json:"max_price,omitempty"`
    SortBy   string   `json:"sort_by,omitempty"`   // relevance, price_asc, price_desc, popularity
    Page     int      `json:"page"`
    Size     int      `json:"size"`
}

// SearchResult adalah hasil pencarian yang sudah difilter stock
type SearchResult struct {
    Total       int64              `json:"total"`
    Products    []RankedProduct    `json:"products"`
    Page        int                `json:"page"`
    Size        int                `json:"size"`
    TotalPages  int                `json:"total_pages"`
    TookMs      int64              `json:"took_ms"`
}

type RankedProduct struct {
    Product    model.ProductDocument `json:"product"`
    Score      float64               `json:"score"`
    InStock    bool                  `json:"in_stock"`
    StockQty   int                   `json:"stock_qty,omitempty"`
}

// SearchService handles search operations
type SearchService struct {
    es       *elasticsearch.Client
    stockSvc *StockFilterService
    index    string
}

func NewSearchService(es *elasticsearch.Client, stockSvc *StockFilterService, index string) *SearchService {
    return &SearchService{
        es:       es,
        stockSvc: stockSvc,
        index:    index,
    }
}

// Search melakukan full-text search dengan stock filtering
func (s *SearchService) Search(ctx context.Context, req SearchRequest) (*SearchResult, error) {
    start := time.Now()

    if req.Page <= 0 {
        req.Page = 1
    }
    if req.Size <= 0 || req.Size > 100 {
        req.Size = 20
    }

    // 1. Build multi-match query dengan fuzziness
    query := s.buildSearchQuery(req)

    // 2. Execute search ke Elasticsearch
    from := (req.Page - 1) * req.Size

    searchBody := map[string]interface{}{
        "from": from,
        "size": req.Size,
        "query": query,
        "_source": true,
    }

    // Add sorting
    if req.SortBy != "" && req.SortBy != "relevance" {
        searchBody["sort"] = s.buildSort(req.SortBy)
    }

    body, err := json.Marshal(searchBody)
    if err != nil {
        return nil, fmt.Errorf("marshal search body: %w", err)
    }

    res, err := s.es.Search(
        s.es.Search.WithContext(ctx),
        s.es.Search.WithIndex(s.index),
        s.es.Search.WithBody(strings.NewReader(string(body))),
    )
    if err != nil {
        return nil, fmt.Errorf("elasticsearch search: %w", err)
    }
    defer res.Body.Close()

    if res.IsError() {
        return nil, fmt.Errorf("elasticsearch error: %s", res.String())
    }

    // 3. Parse response
    var esResponse struct {
        Hits struct {
            Total struct {
                Value int64 `json:"value"`
            } `json:"total"`
            Hits []struct {
                Score float64                `json:"_score"`
                Source model.ProductDocument `json:"_source"`
            } `json:"hits"`
        } `json:"hits"`
    }

    if err := json.NewDecoder(res.Body).Decode(&esResponse); err != nil {
        return nil, fmt.Errorf("decode search response: %w", err)
    }

    // 4. Extract SKUs for stock check
    var products []model.ProductDocument
    skus := make([]string, 0, len(esResponse.Hits.Hits))
    scores := make(map[string]float64)

    for _, hit := range esResponse.Hits.Hits {
        products = append(products, hit.Source)
        skus = append(skus, hit.Source.SKU)
        scores[hit.Source.SKU] = hit.Score
    }

    // 5. Batch stock check via Redis pipeline
    stockMap, err := s.stockSvc.BatchCheckStock(ctx, req.HubID, skus)
    if err != nil {
        // Log error but still return products (without stock info)
        fmt.Printf("stock check failed: %v\n", err)
        stockMap = make(map[string]StockInfo)
    }

    // 6. Rank and return
    ranked := make([]RankedProduct, 0, len(products))
    for _, p := range products {
        stock := stockMap[p.SKU]
        ranked = append(ranked, RankedProduct{
            Product:  p,
            Score:    scores[p.SKU],
            InStock:  stock.Available,
            StockQty: stock.Quantity,
        })
    }

    // 7. Filter out-of-stock if configured
    // In Q-commerce, we typically show in-stock first, then out-of-stock
    ranked = s.applyStockPriority(ranked)

    totalPages := int(esResponse.Hits.Total.Value) / req.Size
    if int(esResponse.Hits.Total.Value)%req.Size > 0 {
        totalPages++
    }

    return &SearchResult{
        Total:      esResponse.Hits.Total.Value,
        Products:   ranked,
        Page:       req.Page,
        Size:       req.Size,
        TotalPages: totalPages,
        TookMs:     time.Since(start).Milliseconds(),
    }, nil
}

// buildSearchQuery membangun multi-match query dengan fuzziness
func (s *SearchService) buildSearchQuery(req SearchRequest) map[string]interface{} {
    must := make([]map[string]interface{}, 0)
    filter := make([]map[string]interface{}, 0)

    // Full-text search dengan multi-match
    if req.Query != "" {
        must = append(must, map[string]interface{}{
            "multi_match": map[string]interface{}{
                "query":  req.Query,
                "fields": []string{"name^3", "description", "category_name^2", "brand^2", "tags"},
                "type":   "best_fields",
                "fuzziness": "AUTO",
                "prefix_length": 2,
                "max_expansions": 50,
                "tie_breaker": 0.3,
            },
        })
    } else {
        // No query: return all (browse mode)
        must = append(must, map[string]interface{}{
            "match_all": map[string]interface{}{},
        })
    }

    // Filters
    if req.Category != "" {
        filter = append(filter, map[string]interface{}{
            "term": map[string]interface{}{"category_id": req.Category},
        })
    }

    if len(req.Brands) > 0 {
        filter = append(filter, map[string]interface{}{
            "terms": map[string]interface{}{"brand": req.Brands},
        })
    }

    if req.MinPrice > 0 || req.MaxPrice > 0 {
        rangeFilter := make(map[string]interface{})
        if req.MinPrice > 0 {
            rangeFilter["gte"] = req.MinPrice
        }
        if req.MaxPrice > 0 {
            rangeFilter["lte"] = req.MaxPrice
        }
        filter = append(filter, map[string]interface{}{
            "range": map[string]interface{}{"price": rangeFilter},
        })
    }

    // Only active products
    filter = append(filter, map[string]interface{}{
        "term": map[string]interface{}{"is_active": true},
    })

    query := map[string]interface{}{
        "bool": map[string]interface{}{
            "must":   must,
            "filter": filter,
        },
    }

    return query
}

func (s *SearchService) buildSort(sortBy string) []map[string]interface{} {
    switch sortBy {
    case "price_asc":
        return []map[string]interface{}{
            {"price": map[string]interface{}{"order": "asc"}},
        }
    case "price_desc":
        return []map[string]interface{}{
            {"price": map[string]interface{}{"order": "desc"}},
        }
    case "popularity":
        return []map[string]interface{}{
            {"popularity_score": map[string]interface{}{"order": "desc"}},
        }
    default:
        return nil // relevance = default Elasticsearch _score sorting
    }
}

// applyStockPriority: in-stock first, then out-of-stock
func (s *SearchService) applyStockPriority(products []RankedProduct) []RankedProduct {
    inStock := make([]RankedProduct, 0, len(products))
    outOfStock := make([]RankedProduct, 0, len(products))

    for _, p := range products {
        if p.InStock {
            inStock = append(inStock, p)
        } else {
            outOfStock = append(outOfStock, p)
        }
    }

    return append(inStock, outOfStock...)
}

4. Stock Filter: Redis Pipeline

Problem: setiap search result (20-100 SKU) harus di-cek stock-nya per hub. Kalau 1-by-1, latency-nya tidak acceptable. Solusi: Redis pipeline.

Redis Pipeline vs MGET

Pipeline mengirimkan semua command dalam satu round-trip — bukan N round-trip. Untuk 100 SKU, latency turun dari ~100ms (100x1ms) menjadi ~5ms (1 batch). Alternatif: MGET — tapi pipeline lebih fleksibel karena kita bisa mix tipe command.

internal/search/service/stock.go
package service

import (
    "context"
    "fmt"
    "strconv"
    "time"

    "github.com/redis/go-redis/v9"
)

type StockInfo struct {
    SKU        string `json:"sku"`
    Available  bool   `json:"available"`
    Quantity   int    `json:"quantity"`
    Reserved   int    `json:"reserved"`
}

// StockFilterService handles real-time stock checking via Redis
type StockFilterService struct {
    client    *redis.Client
    keyPrefix string
    ttl       time.Duration // TTL for stock cache
}

func NewStockFilterService(client *redis.Client, keyPrefix string) *StockFilterService {
    return &StockFilterService{
        client:    client,
        keyPrefix: keyPrefix,
    }
}

// stockKey generates Redis key for a specific hub+SKU stock
func (s *StockFilterService) stockKey(hubID, sku string) string {
    return fmt.Sprintf("%s:stock:%s:%s", s.keyPrefix, hubID, sku)
}

// BatchCheckStock memeriksa stock untuk banyak SKU dalam satu pipeline
func (s *StockFilterService) BatchCheckStock(ctx context.Context, hubID string, skus []string) (map[string]StockInfo, error) {
    if len(skus) == 0 {
        return map[string]StockInfo{}, nil
    }

    pipe := s.client.Pipeline()

    // Schedule all commands
    cmds := make(map[string]*redis.StringCmd)
    for _, sku := range skus {
        key := s.stockKey(hubID, sku)
        cmds[sku] = pipe.Get(ctx, key)
    }

    // Execute pipeline (1 round trip)
    _, err := pipe.Exec(ctx)
    if err != nil && err != redis.Nil {
        // Pipeline bisa partially fail; Redis returns nil for missing keys
        // We handle each key result individually
        fmt.Printf("pipeline exec error (non-fatal): %v\n", err)
    }

    // Parse results
    result := make(map[string]StockInfo, len(skus))
    for _, sku := range skus {
        cmd := cmds[sku]
        val, err := cmd.Result()
        if err == redis.Nil {
            // Stock key not found — treat as out of stock
            result[sku] = StockInfo{
                SKU:       sku,
                Available: false,
                Quantity:  0,
            }
            continue
        }
        if err != nil {
            // Other error — treat as unknown stock
            result[sku] = StockInfo{
                SKU:       sku,
                Available: false,
                Quantity:  0,
            }
            continue
        }

        // Parse Redis value: format "qty:reserved" or just "qty"
        qty, res := s.parseStockValue(val)
        result[sku] = StockInfo{
            SKU:       sku,
            Available: qty > 0 && qty > res,
            Quantity:  qty,
            Reserved:  res,
        }
    }

    return result, nil
}

// parseStockValue parses Redis stock value
// Format: "quantity:reserved" or just "quantity"
func (s *StockFilterService) parseStockValue(val string) (int, int) {
    parts := split2(val, ":")
    if len(parts) < 2 {
        qty, err := strconv.Atoi(parts[0])
        if err != nil {
            return 0, 0
        }
        return qty, 0
    }

    qty, err := strconv.Atoi(parts[0])
    if err != nil {
        qty = 0
    }
    reserved, err := strconv.Atoi(parts[1])
    if err != nil {
        reserved = 0
    }

    return qty, reserved
}

// GetSingleStock checks stock for one SKU
func (s *StockFilterService) GetSingleStock(ctx context.Context, hubID, sku string) (*StockInfo, error) {
    key := s.stockKey(hubID, sku)
    val, err := s.client.Get(ctx, key).Result()
    if err == redis.Nil {
        return &StockInfo{SKU: sku, Available: false, Quantity: 0}, nil
    }
    if err != nil {
        return nil, fmt.Errorf("get stock %s/%s: %w", hubID, sku, err)
    }

    qty, reserved := s.parseStockValue(val)
    return &StockInfo{
        SKU:       sku,
        Available: qty > 0 && qty > reserved,
        Quantity:  qty,
        Reserved:  reserved,
    }, nil
}

// UpdateStock updates stock in Redis (called by stock mutation events)
func (s *StockFilterService) UpdateStock(ctx context.Context, hubID, sku string, quantity int, ttl time.Duration) error {
    key := s.stockKey(hubID, sku)
    val := fmt.Sprintf("%d:0", quantity)
    return s.client.Set(ctx, key, val, ttl).Err()
}

// DecrementStock reduces stock on order placement (via Lua for atomicity)
const decrementStockScript = `
    local key = KEYS[1]
    local qty = tonumber(ARGV[1])
    local current = redis.call("GET", key)

    if not current then
        return -1  -- no stock key
    end

    local parts = {}
    for w in current:gmatch("([^:]+)") do
        table.insert(parts, w)
    end

    local available = tonumber(parts[1]) or 0
    local reserved = tonumber(parts[2]) or 0

    if available - reserved < qty then
        return -2  -- insufficient stock
    end

    reserved = reserved + qty
    redis.call("SET", key, available .. ":" .. reserved)
    return reserved
`

func (s *StockFilterService) DecrementStock(ctx context.Context, hubID, sku string, quantity int) (int, error) {
    key := s.stockKey(hubID, sku)
    result, err := redis.NewScript(decrementStockScript).Run(ctx, s.client,
        []string{key}, quantity).Int()
    if err != nil {
        return 0, fmt.Errorf("decrement stock %s/%s: %w", hubID, sku, err)
    }
    if result < 0 {
        return 0, fmt.Errorf("insufficient stock for %s/%s", hubID, sku)
    }
    return result, nil
}

// split2 splits a string into at most 2 parts
func split2(s, sep string) []string {
    parts := strings.SplitN(s, sep, 2)
    // Use local empty check since this is a private utility
    if len(parts) == 0 {
        return []string{""}
    }
    return parts
}

sequenceDiagram
    participant User
    participant API as Search API
    participant ES as Elasticsearch
    participant Redis as Stock Redis
    participant Kafka

    User->>API: GET /search?q=indomie&hub=JKT01
    Note over API: Start timer

    API->>ES: multi_match query<br/>fuzziness=AUTO<br/>fields: name^3, description, brand^2, category^2

    ES-->>API: 50 hits with scores
    Note over API: Extract SKUs: [INDO01, INDO02, MIE03, ...]

    API->>Redis: PIPELINE GET stock:JKT01:{skus}
    Note over API,Redis: 1 round-trip untuk 50 keys

    Redis-->>API: [qty:reserved, ...]

    Note over API: Filter: out-of-stock ke bawah
    Note over API: Rank: relevance + margin + popularity

    API-->>User: SearchResult with stock flags

    Note over Kafka: Catalog mutation (async)
    Kafka->>API: stock update event
    API->>Redis: UPDATE stock:JKT01:INDO01 = 10

5. Index Sync Worker: Kafka Consumer

Product catalog berubah terus: harga di-update, produk baru masuk, deskripsi diubah. Kita perlu sync ke Elasticsearch secara near-real-time.

internal/search/worker/index_sync.go
package worker

import (
    "context"
    "encoding/json"
    "fmt"
    "log"
    "time"

    "github.com/elastic/go-elasticsearch/v8"
    "github.com/elastic/go-elasticsearch/v8/esapi"
    "github.com/segmentio/kafka-go"
    "github.com/faisalaffan/qcommerce/internal/search/model"
)

// CatalogEventType mendefinisikan tipe event catalog
type CatalogEventType string

const (
    EventProductCreated CatalogEventType = "product.created"
    EventProductUpdated CatalogEventType = "product.updated"
    EventProductDeleted CatalogEventType = "product.deleted"
    EventStockChanged   CatalogEventType = "stock.changed"
    EventPriceChanged   CatalogEventType = "price.changed"
)

// CatalogEvent adalah event dari Kafka
type CatalogEvent struct {
    EventType CatalogEventType `json:"event_type"`
    SKU       string           `json:"sku"`
    Timestamp time.Time        `json:"timestamp"`
    Data      json.RawMessage  `json:"data"`
}

// IndexSyncWorker handles syncing catalog changes to Elasticsearch
type IndexSyncWorker struct {
    es            *elasticsearch.Client
    kafkaReader   *kafka.Reader
    catalogReader CatalogReader
    index         string
    batchSize     int
    flushInterval time.Duration
}

// CatalogReader interface untuk membaca product dari database
type CatalogReader interface {
    GetProduct(ctx context.Context, sku string) (*model.Product, error)
    GetProductsByIDs(ctx context.Context, skus []string) ([]model.Product, error)
}

func NewIndexSyncWorker(
    es *elasticsearch.Client,
    kafkaReader *kafka.Reader,
    catalogReader CatalogReader,
    index string,
    batchSize int,
    flushInterval time.Duration,
) *IndexSyncWorker {
    return &IndexSyncWorker{
        es:            es,
        kafkaReader:   kafkaReader,
        catalogReader: catalogReader,
        index:         index,
        batchSize:     batchSize,
        flushInterval: flushInterval,
    }
}

// Start mulai consume Kafka events
func (w *IndexSyncWorker) Start(ctx context.Context) error {
    log.Printf("index sync worker started: index=%s batch=%d flush=%v",
        w.index, w.batchSize, w.flushInterval)

    // Batch buffer for bulk indexing
    type pendingDoc struct {
        action string // "index" or "delete"
        sku    string
    }

    pending := make(map[string]pendingDoc)
    lastFlush := time.Now()

    flushBatch := func() error {
        if len(pending) == 0 {
            return nil
        }

        var bulkBody string
        for sku, doc := range pending {
            switch doc.action {
            case "delete":
                bulkBody += fmt.Sprintf(`{"delete":{"_index":"%s","_id":"%s"}}`+"\n", w.index, sku)
            case "index":
                // Read full product from DB
                product, err := w.catalogReader.GetProduct(ctx, sku)
                if err != nil {
                    log.Printf("get product %s for indexing: %v", sku, err)
                    continue
                }
                doc := toProductDocument(product)
                data, _ := json.Marshal(doc)
                bulkBody += fmt.Sprintf(`{"index":{"_index":"%s","_id":"%s"}}`+"\n", w.index, sku)
                bulkBody += string(data) + "\n"
            }
        }

        if bulkBody == "" {
            return nil
        }

        res, err := w.es.Bulk(strings.NewReader(bulkBody))
        if err != nil {
            return fmt.Errorf("bulk index: %w", err)
        }
        defer res.Body.Close()

        if res.IsError() {
            return fmt.Errorf("bulk index error: %s", res.String())
        }

        // Parse response for errors
        var bulkResponse struct {
            Errors bool `json:"errors"`
            Items  []struct {
                Index struct {
                    Error interface{} `json:"error,omitempty"`
                } `json:"index"`
            } `json:"items"`
        }
        if err := json.NewDecoder(res.Body).Decode(&bulkResponse); err != nil {
            return fmt.Errorf("decode bulk response: %w", err)
        }

        if bulkResponse.Errors {
            // Log individual errors but don't fail the batch
            for _, item := range bulkResponse.Items {
                if item.Index.Error != nil {
                    log.Printf("bulk index item error: %+v", item.Index.Error)
                }
            }
        }

        log.Printf("index sync: flushed %d docs to %s", len(pending), w.index)
        pending = make(map[string]pendingDoc)
        lastFlush = time.Now()
        return nil
    }

    defer flushBatch() // flush remaining on shutdown

    for {
        select {
        case <-ctx.Done():
            log.Println("index sync worker shutting down")
            return ctx.Err()
        default:
        }

        msg, err := w.kafkaReader.ReadMessage(ctx)
        if err != nil {
            if ctx.Err() != nil {
                return ctx.Err()
            }
            log.Printf("read kafka message: %v", err)
            continue
        }

        var event CatalogEvent
        if err := json.Unmarshal(msg.Value, &event); err != nil {
            log.Printf("unmarshal catalog event: %v", err)
            continue
        }

        // Map event type to Elasticsearch action
        switch event.EventType {
        case EventProductCreated, EventProductUpdated, EventPriceChanged:
            pending[event.SKU] = pendingDoc{action: "index", sku: event.SKU}
        case EventProductDeleted:
            pending[event.SKU] = pendingDoc{action: "delete", sku: event.SKU}
        case EventStockChanged:
            // Stock is handled by Redis, not ES index
            // But we might want to update a "has_stock" flag
            continue
        }

        // Flush if batch full or interval reached
        if len(pending) >= w.batchSize || time.Since(lastFlush) >= w.flushInterval {
            if err := flushBatch(); err != nil {
                log.Printf("flush batch: %v", err)
            }
        }
    }
}

// toProductDocument converts Product to indexable document
func toProductDocument(p *model.Product) model.ProductDocument {
    suggestInput := make([]string, 0, 2+len(p.Tags))
    suggestInput = append(suggestInput, p.Name, p.Brand)
    suggestInput = append(suggestInput, p.Tags...)

    return model.ProductDocument{
        SKU:             p.SKU,
        Name:            p.Name,
        Description:     p.Description,
        CategoryID:      p.CategoryID,
        CategoryName:    p.CategoryName,
        Brand:           p.Brand,
        Price:           p.Price,
        MarginPct:       p.MarginPct,
        PopularityScore: p.PopularityScore,
        Tags:            p.Tags,
        ImageURL:        p.ImageURL,
        Unit:            p.Unit,
        IsActive:        p.IsActive,
        CreatedAt:       p.CreatedAt,
        UpdatedAt:       p.UpdatedAt,
        Suggest: model.SuggestField{
            Input:  suggestInput,
            Weight: int(p.PopularityScore * 100),
        },
    }
}

flowchart LR
    subgraph CatalogSource["Catalog Changes"]
        A[Admin Update]
        B[Vendor Feed]
        C[Price Engine]
    end

    subgraph Kafka["Kafka Events"]
        D[product.created]
        E[product.updated]
        F[product.deleted]
        G[price.changed]
        H[stock.changed]
    end

    subgraph Worker["Index Sync Worker"]
        I[Kafka Consumer]
        J{Batch<br/>Buffer}
        K[Batch Full?]
        L[Interval<br/>Reached?]
        M[Bulk Index<br/>to ES]
    end

    subgraph Storage["Storage"]
        N[(PostgreSQL<br/>Catalog)]
        O[(Elasticsearch)]
        P[(Stock Redis)]
    end

    subgraph Query["Query Path"]
        Q[Search Request]
        R[ES Query<br/>+ Redis Filter]
    end

    A --> D
    B --> D
    C --> G
    A --> H

    D --> I
    E --> I
    F --> I
    G --> I
    H --> P

    I --> J
    J --> K
    J --> L
    K -->|Yes| M
    L -->|Yes| M
    M --> O

    N --> I
    Q --> R
    O --> R
    P --> R

6. Autocomplete Handler

internal/search/handler/autocomplete.go
package handler

import (
    "context"
    "encoding/json"
    "fmt"
    "net/http"
    "strings"

    "github.com/elastic/go-elasticsearch/v8"
    "github.com/elastic/go-elasticsearch/v8/esapi"
)

// AutocompleteHandler handles autocomplete suggestions
type AutocompleteHandler struct {
    es    *elasticsearch.Client
    index string
}

func NewAutocompleteHandler(es *elasticsearch.Client, index string) *AutocompleteHandler {
    return &AutocompleteHandler{es: es, index: index}
}

// AutocompleteRequest is the autocomplete query
type AutocompleteRequest struct {
    Query string `json:"query"`
    Size  int    `json:"size"`
    HubID string `json:"hub_id,omitempty"` // optional for stock-aware suggestions
}

// AutocompleteResponse is the autocomplete result
type AutocompleteResponse struct {
    Suggestions []Suggestion `json:"suggestions"`
    TookMs      int64        `json:"took_ms"`
}

type Suggestion struct {
    Text      string  `json:"text"`
    Score     float64 `json:"score"`
    SKU       string  `json:"sku,omitempty"`
    Category  string  `json:"category,omitempty"`
    ImageURL  string  `json:"image_url,omitempty"`
    Price     int64   `json:"price,omitempty"`
    InStock   bool    `json:"in_stock,omitempty"`
}

// HandleAutocomplete HTTP handler
func (h *AutocompleteHandler) HandleAutocomplete(w http.ResponseWriter, r *http.Request) {
    query := r.URL.Query().Get("q")
    if query == "" || len(query) < 2 {
        writeJSON(w, http.StatusBadRequest, map[string]string{
            "error": "query must be at least 2 characters",
        })
        return
    }

    size := 8 // default 8 suggestions
    hubID := r.URL.Query().Get("hub_id")

    if s := r.URL.Query().Get("size"); s != "" {
        if parsed, err := parseInt(s); err == nil && parsed > 0 && parsed <= 20 {
            size = parsed
        }
    }

    result, err := h.Autocomplete(r.Context(), AutocompleteRequest{
        Query: query,
        Size:  size,
        HubID: hubID,
    })
    if err != nil {
        writeJSON(w, http.StatusInternalServerError, map[string]string{
            "error": fmt.Sprintf("autocomplete: %v", err),
        })
        return
    }

    writeJSON(w, http.StatusOK, result)
}

// Autocomplete queries Elasticsearch Completion Suggester
func (h *AutocompleteHandler) Autocomplete(ctx context.Context, req AutocompleteRequest) (*AutocompleteResponse, error) {
    start := time.Now()

    // Build completion suggester query
    suggestBody := map[string]interface{}{
        "suggest": map[string]interface{}{
            "product_suggest": map[string]interface{}{
                "prefix": req.Query,
                "completion": map[string]interface{}{
                    "field": "suggest",
                    "size":  req.Size,
                    "fuzzy": map[string]interface{}{
                        "fuzziness": "AUTO",
                        "prefix_length": 1,
                    },
                },
            },
        },
        "_source": []string{"sku", "name", "category_name", "image_url", "price", "popularity_score"},
    }

    body, err := json.Marshal(suggestBody)
    if err != nil {
        return nil, fmt.Errorf("marshal suggest body: %w", err)
    }

    res, err := h.es.Search(
        h.es.Search.WithContext(ctx),
        h.es.Search.WithIndex(h.index),
        h.es.Search.WithBody(strings.NewReader(string(body))),
    )
    if err != nil {
        return nil, fmt.Errorf("elasticsearch suggest: %w", err)
    }
    defer res.Body.Close()

    if res.IsError() {
        return nil, fmt.Errorf("elasticsearch error: %s", res.String())
    }

    // Parse suggest response
    var esResponse struct {
        Suggest map[string][]struct {
            Options []struct {
                Text    string  `json:"text"`
                Score   float64 `json:"_score"`
                Source  struct {
                    SKU          string `json:"sku"`
                    Name         string `json:"name"`
                    CategoryName string `json:"category_name"`
                    ImageURL     string `json:"image_url"`
                    Price        int64  `json:"price"`
                } `json:"_source"`
            } `json:"options"`
        } `json:"suggest"`
    }

    if err := json.NewDecoder(res.Body).Decode(&esResponse); err != nil {
        return nil, fmt.Errorf("decode suggest response: %w", err)
    }

    // Build suggestions
    suggestions := make([]Suggestion, 0)
    if suggests, ok := esResponse.Suggest["product_suggest"]; ok && len(suggests) > 0 {
        for _, opt := range suggests[0].Options {
            suggestions = append(suggestions, Suggestion{
                Text:     opt.Source.Name,
                Score:    opt.Score,
                SKU:      opt.Source.SKU,
                Category: opt.Source.CategoryName,
                ImageURL: opt.Source.ImageURL,
                Price:    opt.Source.Price,
            })
        }
    }

    return &AutocompleteResponse{
        Suggestions: suggestions,
        TookMs:      time.Since(start).Milliseconds(),
    }, nil
}

// HTTP helpers
func writeJSON(w http.ResponseWriter, status int, data interface{}) {
    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(status)
    json.NewEncoder(w).Encode(data)
}

func parseInt(s string) (int, error) {
    var n int
    for _, r := range s {
        if r < '0' || r > '9' {
            return 0, fmt.Errorf("not a number: %s", s)
        }
        n = n*10 + int(r-'0')
    }
    return n, nil
}
Completion Suggester vs Search-as-you-type

Elasticsearch punya dua fitur autocomplete: Completion Suggester (berdasarkan prefix, sangat cepat, pre-computed) dan Search-as-you-type (field type yang mengoptimalkan query untuk partial matching). Completion Suggester lebih cocok karena: (1) response time <10ms, (2) bisa pakai fuzzy untuk typo, (3) weight-based ranking. Kekurangannya: perlu pre-build suggest input, tidak real-time untuk kata baru.


7. Ranking: Relevance + Availability + Margin + Popularity

Ranking di Q-commerce tidak bisa pure relevansi Elasticsearch. Kita perlu multi-faktor scoring:

internal/search/service/ranking.go
package service

import (
    "math"
)

// RankingConfig untuk weighting
type RankingConfig struct {
    RelevanceWeight  float64 `json:"relevance_weight"`   // 0.4
    StockWeight      float64 `json:"stock_weight"`        // 0.25
    MarginWeight     float64 `json:"margin_weight"`       // 0.2
    PopularityWeight float64 `json:"popularity_weight"`   // 0.15
}

// DefaultRankingConfig adalah default weights
func DefaultRankingConfig() RankingConfig {
    return RankingConfig{
        RelevanceWeight:  0.40,
        StockWeight:      0.25,
        MarginWeight:     0.20,
        PopularityWeight: 0.15,
    }
}

// Ranker melakukan re-ranking products dengan multi-faktor
type Ranker struct {
    config RankingConfig
}

func NewRanker(config RankingConfig) *Ranker {
    return &Ranker{config: config}
}

// CompositeScore menghitung skor gabungan
func (r *Ranker) CompositeScore(
    relevanceScore float64,
    marginPct float64,
    popularityScore float64,
    stockQty int,
) float64 {
    // Normalize each factor to 0-1 range
    relScore := normalizeScore(relevanceScore, 0, 10)
    marginScore := normalizeScore(marginPct, 0, 50)   // margin 0-50%
    popScore := normalizeScore(popularityScore, 0, 100)
    stockScore := stockFactor(stockQty)

    return r.config.RelevanceWeight*relScore +
        r.config.MarginWeight*marginScore +
        r.config.PopularityWeight*popScore +
        r.config.StockWeight*stockScore
}

// ReRank melakukan re-ranking terhadap search results
func (r *Ranker) ReRank(products []RankedProduct) []RankedProduct {
    scored := make([]scoredProduct, len(products))

    for i, p := range products {
        compositeScore := r.CompositeScore(
            p.Score,
            p.Product.MarginPct,
            p.Product.PopularityScore,
            p.StockQty,
        )
        scored[i] = scoredProduct{
            product: p,
            score:   compositeScore,
        }
    }

    // Sort by composite score descending
    sort.Slice(scored, func(i, j int) bool {
        return scored[i].score > scored[j].score
    })

    result := make([]RankedProduct, len(scored))
    for i, s := range scored {
        result[i] = s.product
    }
    return result
}

type scoredProduct struct {
    product RankedProduct
    score   float64
}

// normalizeScore normalizes a value to 0-1 using min-max normalization
func normalizeScore(value, min, max float64) float64 {
    if max-min == 0 {
        return 0
    }
    normalized := (value - min) / (max - min)
    return math.Max(0, math.Min(1, normalized))
}

// stockFactor: stock yang lebih banyak = skor lebih tinggi
// Tapi tidak linear: stok 5 vs 50 beda tipis, stok 0 vs 5 beda besar
func stockFactor(qty int) float64 {
    if qty <= 0 {
        return 0
    }
    // Log scale: 1->0.3, 5->0.6, 50->0.9, 100->1.0
    return math.Min(1.0, math.Log10(float64(qty+1))/2.0)
}

8. Edge Cases

Edge Case 1: Elasticsearch dan Redis Stock Mismatch

Go
// Reconciliation job untuk memastikan stock di Redis match dengan PostgreSQL
type StockReconciler struct {
    pgDB     *sql.DB
    redis    *redis.Client
    esClient *elasticsearch.Client
}

func (r *StockReconciler) Reconcile(ctx context.Context, hubID string) ([]ReconciliationDiff, error) {
    // 1. Get all product SKUs from PostgreSQL inventory
    pgRows, err := r.pgDB.QueryContext(ctx, `
        SELECT sku, quantity, reserved
        FROM inventory
        WHERE hub_id = $1
    `, hubID)
    if err != nil {
        return nil, fmt.Errorf("query pg inventory: %w", err)
    }
    defer pgRows.Close()

    pgStock := make(map[string]string)
    for pgRows.Next() {
        var sku string
        var qty, reserved int
        if err := pgRows.Scan(&sku, &qty, &reserved); err != nil {
            continue
        }
        pgStock[sku] = fmt.Sprintf("%d:%d", qty, reserved)
    }

    // 2. Scan all Redis keys for this hub
    iter := r.redis.Scan(ctx, 0, fmt.Sprintf("qcom:stock:%s:*", hubID), 100).Iterator()

    var diffs []ReconciliationDiff
    for iter.Next(ctx) {
        key := iter.Val()
        sku := extractSKUFromKey(key, hubID)

        redisVal, err := r.redis.Get(ctx, key).Result()
        if err != nil {
            continue
        }

        pgVal, exists := pgStock[sku]
        if !exists {
            diffs = append(diffs, ReconciliationDiff{
                SKU:     sku,
                Field:   "exists",
                Redis:   redisVal,
                PG:      "not_found",
                Action:  "delete_redis",
            })
            continue
        }

        if redisVal != pgVal {
            diffs = append(diffs, ReconciliationDiff{
                SKU:    sku,
                Field:  "value",
                Redis:  redisVal,
                PG:     pgVal,
                Action: "update_redis",
            })
        }

        delete(pgStock, sku)
    }

    // Remaining PG entries are missing from Redis
    for sku, val := range pgStock {
        diffs = append(diffs, ReconciliationDiff{
            SKU:    sku,
            Field:  "missing",
            Redis:  "not_found",
            PG:     val,
            Action: "add_redis",
        })
    }

    // Auto-fix
    for _, d := range diffs {
        switch d.Action {
        case "add_redis", "update_redis":
            r.redis.Set(ctx, stockKey(hubID, d.SKU), d.PG, 0)
        case "delete_redis":
            r.redis.Del(ctx, stockKey(hubID, d.SKU))
        }
    }

    return diffs, nil
}

type ReconciliationDiff struct {
    SKU    string
    Field  string
    Redis  string
    PG     string
    Action string
}

Edge Case 2: Zero Results Query

Go
// Fallback search ketika query menghasilkan 0 hasil
func (s *SearchService) SearchWithFallback(ctx context.Context, req SearchRequest) (*SearchResult, error) {
    result, err := s.Search(ctx, req)
    if err != nil {
        return nil, err
    }

    if result.Total > 0 {
        return result, nil
    }

    // Fallback 1: deduplicate and try with less strict fuzziness
    fallbackReq := req
    fallbackReq.Query = simplifyQuery(req.Query) // remove special chars, normalize

    if fallbackReq.Query != req.Query {
        result, err = s.Search(ctx, fallbackReq)
        if err == nil && result.Total > 0 {
            return result, nil
        }
    }

    // Fallback 2: category browse instead of search
    // Show top products in default category
    result, err = s.Search(ctx, SearchRequest{
        HubID: req.HubID,
        Page:  1,
        Size:  req.Size,
    })
    if err == nil && result.Total > 0 {
        // Mark as fallback
        return result, nil
    }

    // Fallback 3: return popular products
    result, err = s.Search(ctx, SearchRequest{
        HubID:  req.HubID,
        SortBy: "popularity",
        Page:   1,
        Size:   req.Size,
    })
    if err == nil && result.Total > 0 {
        return result, nil
    }

    // Last resort: empty result
    return result, nil
}

func simplifyQuery(q string) string {
    // Remove special characters, normalize whitespace
    var result strings.Builder
    for _, r := range q {
        if r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' || r >= '0' && r <= '9' || r == ' ' {
            result.WriteRune(r)
        }
    }
    return strings.TrimSpace(result.String())
}

Edge Case 3: Race Condition Stock Update

Go
// Optimistic lock via Redis Lua untuk decrement stock
const atomicReserveScript = `
    local key = KEYS[1]
    local orderQty = tonumber(ARGV[1])
    local orderID = ARGV[2]
    local lockKey = key .. ":lock:" .. orderID

    -- Check duplicate
    if redis.call("EXISTS", lockKey) == 1 then
        return -3  -- duplicate order
    end

    local stock = redis.call("GET", key)
    if not stock then
        return -1  -- no stock data
    end

    -- Parse "qty:reserved"
    local colon = string.find(stock, ":")
    local qty = tonumber(string.sub(stock, 1, colon - 1))
    local reserved = tonumber(string.sub(stock, colon + 1))

    if not qty or qty - reserved < orderQty then
        return -2  -- insufficient
    end

    reserved = reserved + orderQty
    redis.call("SET", key, qty .. ":" .. reserved)
    redis.call("SETEX", lockKey, 30, "1")  -- 30s lock untuk cegah duplicate

    return reserved
`

func (s *StockFilterService) AtomicReserve(ctx context.Context, hubID, sku, orderID string, qty int) error {
    key := s.stockKey(hubID, sku)
    result, err := redis.NewScript(atomicReserveScript).Run(ctx, s.client,
        []string{key}, qty, orderID).Int()
    if err != nil {
        return fmt.Errorf("atomic reserve: %w", err)
    }
    switch result {
    case -1:
        return fmt.Errorf("stock data not found for %s/%s", hubID, sku)
    case -2:
        return fmt.Errorf("insufficient stock for %s/%s", hubID, sku)
    case -3:
        return fmt.Errorf("duplicate order %s for %s/%s", orderID, hubID, sku)
    default:
        return nil // success
    }
}

sequenceDiagram
    participant Catalog as Catalog Admin
    participant Kafka
    participant Worker as Index Sync Worker
    participant ES as Elasticsearch
    participant PG as PostgreSQL
    participant Redis

    Catalog->>Kafka: product.updated (SKU=INDO01, price=6500)
    Kafka->>Worker: Consume event
    Worker->>PG: Get full product data
    PG-->>Worker: Product{name, desc, category, price, ...}
    Worker->>Worker: toProductDocument()
    Worker->>ES: Bulk index INDO01
    ES-->>Worker: indexed (seq_no=42)

    Note over Worker: Batch buffer: pending docs
    Note over Worker: Flush when batch=100 or interval=5s

    Catalog->>Kafka: stock.changed (hub=JKT01, sku=INDO01, qty=100)
    Kafka->>Redis: Direct stock update
    Note over Redis: stock:qcomm:JKT01:INDO01 = 100:0

    Note over ES,Redis: Search path: ES for text, Redis for stock

Key Takeaways


Kesimpulan

Search di Q-commerce menuntut keseimbangan antara:

  1. Relevansi — Elasticsearch full-text dengan multi-field boosting
  2. Ketersediaan — Redis pipeline untuk stock check real-time per-hub
  3. Kecepatan — query-time filtering, not index-time, karena stock berubah terus
  4. Kekinian — Kafka streaming untuk near-real-time index sync
  5. Akurasi — reconciliation job untuk deteksi mismatch Redis vs database

Dengan desain ini, search service bisa handle 15.000+ SKU di 50+ hub dengan latency <50ms untuk search dan <10ms untuk autocomplete. Stock filter real-time via Redis pipeline memastikan user hanya melihat produk yang benar-benar tersedia.

Artikel terkait

Rekayasa Perangkat Lunak

Real-time Order Tracking Q-Commerce: Driver Bergerak di Peta Live

System design real-time order tracking untuk quick commerce. Driver GPS ingestion via WebSocket dengan throttling dan validasi, Kalman filter untuk position smoothing, Redis pub/sub fan-out by order ID untuk scale ribuan driver concurrent, dan Server-Sent Events (SSE) push ke user client. Implementasi Golang dengan pattern WebSocket goroutine pool, pub/sub broker, dan concurrent connection management.
Faisal Affan
Faisal Affan
Diterbitkan
Rekayasa Perangkat Lunak
Real-time Inventory Dark Store: Cegah Oversell di Q-Commerce

Real-time Inventory Dark Store: Cegah Oversell di Q-Commerce

Desain inventory per-hub untuk Astro (instant grocery). Reservation pattern dengan Redis DECRBY atomic + TTL, async sync ke Postgres, stock partitioning untuk hot SKU, reaper job untuk reservation leak, dan cycle counting untuk stok fisik vs sistem. Implementasi Golang lengkap.
Faisal Affan
Faisal Affan
Diterbitkan

Promo & Voucher Engine Q-Commerce: Rule Engine Anti-Abuse

System design promo engine untuk quick commerce. Rule engine dengan kondisi komposable (min transaksi, first-N-order, area, kategori) ke efek (discount, free ongkir), atomic counter untuk limit penggunaan Redis Lua, anti-fraud device fingerprinting, stackable vs eksklusif promo resolution, dan budget cap. Implementasi Golang lengkap dengan composable condition interface, AND/OR composite tree, concurrent-safe redeem, serta Kafka event streaming untuk audit trail.
Faisal Affan
Faisal Affan
Diterbitkan

Order Dispatch & Driver Allocation: Hungarian Algorithm untuk Q-Commerce

System design order dispatch untuk quick-commerce (q-commerce). Pelajari batching window untuk kumpulkan order, Hungarian/greedy matching driver ke order, Redis GEOADD/GEOSEARCH untuk cari driver terdekat, real-time driver state tracking (idle/picking/delivering), batch delivery multi-order, dan edge cases seperti driver reject, order timeout, serta starvation handling. Dilengkapi implementasi Golang dengan context.Context, error wrapping, struct-based services, channel-based event sourcing, dan concurrent pipeline.
Faisal Affan
Faisal Affan
Diterbitkan