Data Platform / Purpose-Built DB
Data PlatformDatabaseNoSQLDocument

Document Store

David TirabassiUpdated
Enterprise/Solution Architect · Almost 30 years' experience, specialising in integration · TOGAF 9.1 certifiedLinkedInGitHub

Problem

Rigid relational schemas struggle to store and query rapidly evolving, hierarchical, or semi-structured data whose shape varies per record. Forcing such data into normalised tables triggers costly migrations, brittle joins, and object-relational mapping overhead that slows iteration.

Solution

Use a Document Store (NoSQL) database that persists data as flexible, self-describing JSON/BSON documents, allowing schemas to evolve dynamically at the application level.

Cloud Paradigm

  • Schema-on-Read (application-level schema evolution)
  • Aggregate-Oriented Persistence
  • Polyglot Persistence
  • Denormalised Document Modelling
  • Change Data Capture (change streams)
  • Horizontal Partitioning (shard-key routing)

Solution Flow

  1. The application service serialises an aggregate (say a customer profile with nested addresses, preferences, and order history) into a single self-describing JSON/BSON document rather than fanning it across normalised tables.
  2. The document store driver writes the document into a collection, tagging it with a schema version field so evolving shapes coexist without a migration.
  3. The storage engine persists the document to a partitioned collection, using the shard key to route writes and co-locate related documents on the same node.
  4. Secondary indexes are built on frequently queried fields, including nested paths and array elements, so predicate and range queries avoid full collection scans.
  5. A query API executes rich filters, projections, and aggregation pipelines server-side, returning only the fields each consumer needs.
  6. Read replicas serve low-latency reads and analytics offload, while the primary handles writes; the client tunes read/write concern to trade consistency for latency.
  7. Downstream consumers (search indexers, event streams, or a change-data-capture feed) subscribe to the change stream to propagate document mutations in near real time.

When to Use

  • Content, catalogue, or profile data where each record has variable, sparse, or nested attributes.
  • Rapid product iteration where the schema changes weekly and coordinated table migrations are painful.
  • Aggregate-oriented access where you load and save an entire object graph in one round trip.
  • Hierarchical payloads (CMS pages, IoT device state, event envelopes) that map naturally to nested JSON.

When NOT to Use

  • Workloads dominated by multi-entity joins and complex ad-hoc relational analytics.
  • Systems needing strict multi-row ACID transactions across many entities as the norm.
  • Highly uniform, tabular data where a relational or columnar store is cheaper and simpler.
  • Cases where duplicated denormalised data would create unmanageable update anomalies.

Trade-offs

  • Schema flexibility vs the loss of database-enforced integrity, pushing validation into the application.
  • Single-document read performance vs data duplication and manual consistency across embedded copies.
  • Horizontal scale via sharding vs the difficulty of changing a shard key and cross-shard query cost.
  • Fast iteration vs weaker support for ad-hoc joins and normalised reporting.

Illustrative Scenario

A hypothetical scenario showing how the pattern applies. Not a client case study.

Consider a fitness-tracking app that stores each member's profile as one document: nested training plans, per-device sync settings, personalisation signals and A/B flags. When the product team adds a new preference, engineers simply write the new field, with no downtime and no ALTER TABLE across millions of rows. A change stream feeds the recommendation pipeline and a search index whenever a profile changes, and sharding by member ID spreads reads and writes across nodes as the membership grows in new regions.

Additional Details

  • Atomicity boundary: Writes are atomic only at the single-document level; multi-document transactions exist but incur cross-shard coordination cost, so design aggregates so one document equals one consistency boundary.
  • Change stream semantics: Change feeds deliver at-least-once and can emit duplicates or gap on failover; consumers must be idempotent and checkpoint a resume token to replay from the last processed mutation.
  • Schema evolution: Since documents are self-describing, old and new shapes coexist forever. Write migration-on-read logic keyed on the version field, and backfill lazily rather than assuming every document has new fields.
  • Index and document limits: Watch per-document size caps and unbounded array growth; embedded arrays that grow without limit bloat documents, slow updates, and eventually breach size ceilings. Cap or bucket them.
  • Cost and performance drivers: The bill is driven by index count, working-set RAM, and write amplification from index maintenance; every secondary index slows writes, so prune unused ones and cover hot queries deliberately.
  • Shard key commitment: The shard key is effectively immutable and dictates hotspotting—avoid monotonically increasing keys, and monitor chunk balancing, jumbo chunks, and cross-shard scatter-gather query latency.
  • Operational chores: Reclaiming space after large deletes often requires compaction, and read replicas add replication lag—tune read concern for consumers that cannot tolerate stale reads.

Security Controls

  • Encryption at rest: Enable transparent storage-level and optional client-side field-level encryption for sensitive attributes like PII inside documents.
  • TLS in transit: Require mutual TLS between drivers, primaries, and replica set members to protect document payloads on the wire.
  • Role-based access control: Scope roles to specific collections and operations so services read or write only the documents they own.
  • Schema validation rules: Apply collection-level JSON Schema validators to reject malformed or unexpected document structures at write time.
  • Audit logging: Capture authentication, DDL, and data-access events to a tamper-evident audit trail for compliance review.
  • Network isolation: Deploy the cluster in a private subnet with IP allowlists and no public endpoint exposure.

Related Patterns