PM.← All work

Data architecture · ChatDaddy

Keeping live messaging fast beyond 2TB of message data

Separated active and historical message access with MongoDB hot and cold partitions and Elasticsearch-backed search.

message data partitioned2TB+

01 · Context

What needed to change

ChatDaddy's MongoDB message data had grown beyond 2TB. Live conversations and historical reads were competing for the same data access path.

02 · Challenge

The hard part

Active messages needed predictable response times while older conversations remained available through the product's existing search experience.

03 · Approach

How I approached it

  1. Separated frequently accessed messages from historical data so current conversations could use a smaller hot working set.
  2. Kept historical messages searchable through an Elasticsearch-backed path.
  3. Combined hot and cold results at the service layer so callers could continue using one product interface.

04 · Outcome

What shipped

Partitioned more than 2TB of message data while keeping customer-facing message access in the double-digit-millisecond range.