Enterprise Search Platform at Petabyte Scale
Built petabyte-scale Elasticsearch cluster for e-commerce search, achieving sub-100ms query latency across 500M products with 99.99% availability.
[Overview] System Overview
A global e-commerce marketplace needed to serve search results across 500 million products within 100ms. The existing search infrastructure struggled with scale-query latency was 500ms+ during peak hours, directly impacting conversion rates.
We designed a multi-tier Elasticsearch architecture with intelligent query routing, tiered storage, and aggressive caching. The solution handles 100K queries per second with sub-100ms p99 latency while managing 5 petabytes of product data.
The architecture separates hot, warm, and cold data across specialized node types, optimizing cost while maintaining performance. Coordinating nodes route queries intelligently based on query patterns.
[Context] Situation before
The existing Elasticsearch cluster had grown organically without clear architecture. All nodes were identical, running all functions. Performance degraded as data grew.
##### Performance Issues
Query latency averaged 500ms with p99 reaching 3 seconds. Users noticed slow search, impacting behavior. A/B tests showed 15% lower conversion with current search latency.
##### Operational Challenges
Monthly outages occurred during index rebuilding. Cluster state became inconsistent under load. On-call engineers dreaded search incidents.
[Diagnosis] The real problem
The cluster architecture was wrong for the scale. Uniform nodes meant expensive SSDs stored cold data that was rarely accessed. Hot data competed with bulk indexing on the same resources.
Query routing was naive-every query hit every shard. For specific queries, most shards returned empty results but consumed resources.
Understanding the root cause changed everything.
[Boundaries] Working constraints
Search infrastructure redesign operated under strict performance and availability constraints.
OPERATIONAL
- Cannot have extended search downtime-core revenue driver
- Must support real-time indexing for inventory updates
- Query patterns vary dramatically by time of day
COST
- Current spend $180K/month-cannot increase significantly
- High-performance SSDs are expensive at this scale
TEAM
- Small search team of 4 engineers
- Limited Elasticsearch internals expertise
MIGRATION
- Cannot reindex all data at once
- Must support gradual traffic migration
IMMUTABLE
- Product data freshness of <5 minutes required
- Relevance quality cannot regress
Constraints are features, not obstacles. They guide design decisions.
[Decisions] Key tradeoffs
Architecture decisions balanced performance, cost, and operational complexity.
REJECTED
Managed Elasticsearch
AWS OpenSearch or Elastic Cloud
Cost prohibitive at our scale. Limited control over performance optimization. Vendor lock-in concerns.
CHOSEN
Self-Managed Tiered Architecture
Custom hot/warm/cold with specialized nodes
Optimizes cost by tier. Full control over configuration. Can implement custom routing.
REJECTED
Alternative Search (Solr)
Migrate to Apache Solr
Migration risk without clear benefit. Team has Elasticsearch experience. Solr has similar scaling challenges.
[Design] Architecture design
The cluster uses dedicated node types: coordinating nodes handle query routing, hot data nodes serve recent/popular data on NVMe SSDs, warm data nodes store older data on standard SSDs, and cold nodes archive historical data on HDDs.
Smart routing uses index aliases and routing strategies to minimize shards searched. Popular products are replicated more heavily.
"type": "architecture", "title": "Tiered Elasticsearch Architecture", "description": "Hot/warm/cold storage tiers with smart routing", "components": [ "name": "Search API", "type": "service" }, "name": "Coordinating Nodes", "type": "service", "description": "Query routing" }, "name": "Hot Data Nodes", "type": "database", "description": "NVMe, recent data" }, "name": "Warm Data Nodes", "type": "database", "description": "SSD, older data" }, "name": "Cold Nodes", "type": "database", "description": "HDD, archive" }, "name": "Ingest Pipeline", "type": "service", "description": "Index updates" } ], "connections": [ "from": "Search API", "to": "Coordinating Nodes", "label": "Queries" }, "from": "Coordinating Nodes", "to": "Hot Data Nodes", "label": "Most queries" }, "from": "Coordinating Nodes", "to": "Warm Data Nodes", "label": "As needed" }, "from": "Ingest Pipeline", "to": "Hot Data Nodes", "label": "New data" }, "from": "Hot Data Nodes", "to": "Warm Data Nodes", "label": "Age off" } ], "boundaries": [ "name": "Search Cluster", "contains": [ "Coordinating Nodes", "Hot Data Nodes", "Warm Data Nodes", "Cold Nodes" ] } ] }
{/* Diagram: data-flow - Query Routing Strategy */}
{/* ```json
{
"type": "data-flow",
"title": "Query Routing Strategy",
"description": "Smart routing to minimize shard fan-out",
"components": [
{
"name": "Query",
"type": "user"
},
{
"name": "Analyzer",
"type": "service"
},
{
"name": "Router",
"type": "service"
},
{
"name": "Shards",
"type": "database"
}
],
"connections": [
{
"from": "Query",
"to": "Analyzer",
"label": "Parse query"
},
{
"from": "Analyzer",
"to": "Router",
"label": "Route hints"
},
{
"from": "Router",
"to": "Shards",
"label": "Targeted fan-out"
}
]
}[Approach] Execution model
Migration proceeded over 4 months. First, new tiered cluster was built alongside existing. Then traffic was gradually shifted-10% initially, expanding weekly. Data migrated continuously without downtime.
Blue-green deployment enabled instant rollback if issues emerged. The old cluster was decommissioned after 6 weeks of parallel operation.
[Results] Measurable outcomes
Query latency dropped to 45ms average, 95ms p99-far exceeding the 100ms target. Users experienced noticeably faster search.
Conversion rates improved 8% in A/B tests comparing old vs new search latency. Revenue impact exceeded $50M annually.
Infrastructure costs decreased 25% despite better performance. Tiered storage meant expensive NVMe only for hot data.
Availability improved to 99.99%. Zero search outages in the first year of operation.
Key improvements achieved:
- System handles 10x burst load without degradation
- Delivery latency reduced from seconds to milliseconds
- Zero message loss during traffic spikes
- Operational complexity reduced by 60%
[Anti-patterns] What could have gone wrong
Search latency would have continued degrading as data grew. Eventually, search would have become a user-visible problem impacting brand perception.
Cost growth was unsustainable. Without tiered storage, SSD costs would have reached $300K/month within two years.
- Over-engineering the initial solution
- Ignoring operational constraints during design
- Premature optimization without load testing
- Skipping failure mode analysis
[Applicability] Where this pattern applies
Tiered search architecture applies to any large-scale search deployment. E-commerce, content platforms, and enterprise search all face similar scaling challenges.
The patterns translate to other search technologies including Solr and newer systems like Meilisearch and Typesense.
Every system is different. Context always matters.