Start with query workloads, not storage products
An IoT platform usually faces two opposing pressures: devices keep producing data, while operators and applications expect fast access to current status, recent trends, and long-term history. Keeping every raw event in one database is convenient at first, but indexing, compaction, backups, retention jobs, and cluster expansion eventually begin competing with ingestion. The real design problem is therefore not how to buy a larger database. It is how to manage the lifecycle of each event without breaking the user-facing query experience.
Before choosing a time-series database or an object store, classify the queries. Operational dashboards need low-latency access to recent measurements. Incident investigations may require individual events, device timestamps, and quality flags. Management reports often need only hourly, daily, or monthly aggregates. Data science workloads tend to prefer stable schemas and efficient batch scans over interactive response times. These workloads have different requirements for latency, granularity, consistency, and cost, so forcing all of them through one storage engine creates unnecessary compromises.
Define the hot and cold boundary from service objectives
The hot tier holds recent, frequently requested data and normally uses a time-series database, an analytical database, or a partitioned relational database. The cold tier holds older data that is queried less frequently but remains part of the official record. It is commonly stored in a columnar format such as Parquet on object storage and read through a query engine, batch job, or data-lake platform. Cold storage is not merely a backup. It needs a catalog, schema ownership, access controls, and a tested way to retrieve data.
- Query horizon: How far back do normal dashboards look, and how far back do engineers investigate incidents?
- Response expectation: Which queries must be interactive, and which may run asynchronously or produce an export?
- Required granularity: Does long-term use require every raw event, or are aggregates and quality summaries sufficient?
- Late arrivals: Can disconnected devices upload events after the corresponding period has already been archived?
- Governance: Do retention, deletion, anonymization, or audit rules vary by tenant, device class, or data type?
Hot retention should cover the normal interactive horizon, a practical window for replaying processing jobs, and expected late uploads. It should not be chosen from disk price alone. In the cold tier, keep immutable raw events when investigation or recomputation matters, and maintain governed aggregates for common reports. Aggregates reduce scan cost, but they cannot answer questions that were not anticipated when they were created. Raw and summarized datasets therefore serve different purposes and should have explicit retention policies.
Make ingestion, archival, and cross-tier queries independently controllable
At ingestion time, retain both the device event time and the platform receive time. Also capture a stable device identifier, source, schema version, unit, and quality status. Event time supports business analysis; receive time exposes network delays, clock problems, and backfilled uploads. After an event enters the hot tier, an archival process writes it into cold partitions, compacts files, and updates a manifest or catalog. Hot data should be deleted only after record counts, time ranges, checksums, or equivalent integrity checks confirm that the cold copy is complete and readable.
Partitioning deserves careful attention. Date combined with tenant, site, or data type is often useful, but the choice must match actual filter patterns. Creating a directory for every device can be harmful when device cardinality is high, because it produces excessive metadata and many small files. A safer design uses coarser time and business partitions, then relies on sorting, file statistics, or indexes within each file to reduce scanning. Compaction should be a normal part of the archival pipeline, not an emergency maintenance task.
Applications should ideally call one history API. A query router can send recent ranges to the hot database and older ranges to the cold query engine. A request that crosses the boundary is split and the results are merged. That merge must handle overlapping retention windows, stable ordering, pagination, late events, and deduplication using a deterministic event key. Cold queries that may take longer should return a job identifier with status and cancellation support instead of holding an HTTP connection open. The API should also disclose the available granularity so callers do not mistake an aggregate for raw telemetry.
Treat archival reliability as a production service
The most serious failure is not a slow historical query. It is deleting hot data because the system believes archival succeeded when the cold files are incomplete, inaccessible, or absent from the catalog. Archival jobs must be idempotent and safe to rerun. Monitor archival lag, failed partitions, unexpected file sizes, incompatible schemas, hot-to-cold count differences, and cold-query errors. Schema evolution also needs rules: adding a nullable field is usually manageable, while type changes, renamed fields, and unit conversions require versioned readers or an explicit migration.
- Sample and compare hot, cold, and aggregate results, paying particular attention to events around tier boundaries.
- Verify that cold files are readable, authorized, and registered before expiring the matching hot partitions.
- Practice rebuilding a hot partition, recomputing aggregates, and replaying events from cold storage.
- Track ingestion cost, hot-query latency, cold bytes scanned, and archival backlog separately from total capacity.
- Extend tenant isolation, encryption, deletion policies, and audit records to cold data and every derived dataset.
A durable architecture separates ingestion, interactive queries, immutable archival, aggregate generation, and cross-tier routing into clear responsibilities. Storage products may change, but event identity, time semantics, data contracts, integrity checks, and recovery procedures must remain stable. When the same platform also connects to ERP, CRM, LINE, or cloud analytics, an experienced integration team can help establish those boundaries without turning the solution into another collection of disconnected storage components.