Glenn Lum Software Engineering

3.4.6 Waste Identification: Idle Resources, Orphaned Assets, and Oversized Instances

Most engineers understand that cloud waste exists. They nod when someone mentions unused instances or oversized databases. The problem is not awareness — it is that waste identification is treated as a periodic audit when it is actually a continuous detection problem with fundamentally different mechanics depending on the category of waste. An idle instance, an orphaned EBS volume, and an oversized RDS database are three different failure modes with different root causes, different detection methods, and different risk profiles when you try to eliminate them. Lumping them into “cloud waste” and running a quarterly cleanup is like lumping together memory leaks, deadlocks, and race conditions under “bugs” and scheduling a monthly fix-it day. You need a model for how each type of waste forms, how it hides, and what makes it safe or dangerous to eliminate.

How Waste Forms: The Three Mechanisms

Cloud waste does not appear at a single point in time. It accumulates through three distinct mechanisms, and understanding these mechanisms determines whether you can build detection that actually works.

Drift: How Idle Resources Appear

An idle resource is one that is provisioned and billing but performing no meaningful work. The canonical example is a compute instance running at near-zero CPU utilization. But “near-zero CPU” is a dangerously simplistic heuristic, and this is where most waste identification efforts go wrong.

Consider an EC2 instance running a batch job that activates for twelve minutes every night at 2 AM, processes a critical data pipeline, then returns to idle. Its average daily CPU utilization is under 1%. A naive scan flags it as waste. An engineer terminates it. The next morning, a downstream dashboard has no data and an incident begins.

Idle resource detection requires understanding the utilization envelope — the full time-series behavior of a resource, not its average. The metrics that matter are peak utilization within a window, the recency of the last non-trivial utilization event, and the presence of any scheduled or event-driven workload pattern. An instance that has not exceeded 2% CPU in 30 days is a strong candidate for waste. An instance that hit 80% CPU for fifteen minutes last Tuesday is not, regardless of its monthly average.

Idle resources form through drift: the gap between what a resource was provisioned for and what it is currently doing. A staging environment spun up for a feature that shipped three months ago. A load balancer fronting a service that was migrated to a different endpoint. A NAT gateway in a VPC where all workloads have moved to a VPC with its own. The original provisioning was justified. The current state is not. Drift is the delta between provisioning intent and operational reality, and it grows monotonically unless you have a process that actively counteracts it.

Severed References: How Orphaned Assets Accumulate

Orphaned assets are the most mechanically interesting category of waste because they result from how cloud providers model resource lifecycles — specifically, from the fact that deletion does not cascade the way most engineers assume it does.

When you terminate an EC2 instance, the attached EBS volumes may or may not be deleted depending on the DeleteOnTermination flag set at launch time. The default for the root volume is true; the default for additional volumes is false. This means every instance launched with an extra data volume that is later terminated leaves behind an EBS volume that is attached to nothing, billing at the per-GB monthly rate, and invisible to anyone who is not specifically looking for unattached volumes.

The same pattern repeats across dozens of resource types. Elastic IP addresses that were associated with terminated instances remain allocated and billing. Snapshots taken of volumes that were subsequently deleted continue to exist and accrue storage charges. Security groups created for instances that no longer exist persist indefinitely. Load balancers whose target groups are empty still incur hourly charges. Custom AMIs registered from instances that were decommissioned still store their backing snapshots.

The root cause is that cloud infrastructure is a directed graph of references, not a tree with automatic garbage collection. When you delete a node, its dependents are not automatically collected. Some references are strong (deleting a VPC requires deleting its subnets first), but many are weak (deleting an instance does not delete its snapshots, its AMIs, its associated DNS records, or its CloudWatch alarms). Orphaned assets form at every weak reference edge in this graph when a parent resource is removed.

This is why orphan detection cannot work by scanning individual resource types in isolation. You need to evaluate resources in the context of their reference graph. An EBS volume is not inherently waste — it is waste only if nothing references it and no process intends to. A snapshot is not waste if it is the backing artifact for an AMI that is actively used in a launch template. Detecting orphans requires traversing the dependency graph and identifying terminal nodes with no inbound references from active resources.

In practice, the highest-volume orphan categories in most AWS accounts are: unattached EBS volumes, aged snapshots with no associated AMI or volume, unused Elastic IPs, stale security groups, and detached ENIs (Elastic Network Interfaces). In GCP, it is unattached persistent disks and unused static external IPs. In Azure, it is unattached managed disks and orphaned NICs. The taxonomy varies by provider, but the mechanism — severed references in a non-cascading resource graph — is universal.

Stale Assumptions: How Oversized Instances Persist

Oversized instances are resources whose specifications exceed what the workload requires. Unlike idle resources (which do nothing) or orphans (which serve no one), oversized resources are actively doing useful work — just on hardware that is two or four times larger than necessary.

Oversizing originates at provisioning time and persists due to the absence of a feedback loop. The initial sizing decision is almost always a guess. An engineer selects an r5.2xlarge for a new database because the workload is “memory-heavy” and they do not yet have production data to guide the choice. Six months later, the database is using 8 GB of its 64 GB of available memory. The instance is doing its job. No alerts fire. No one revisits the sizing decision because nothing is broken.

This is the core problem: oversizing has no operational signal. An undersized instance generates alerts — high CPU, OOM kills, increased latency. An oversized instance generates nothing. It is operationally silent. The only signal is in the utilization metrics, and no one is looking at utilization metrics unless they have a process that requires it.

Detecting oversized resources requires comparing provisioned capacity against actual utilization over a representative time window. For compute, this means CPU and memory utilization (memory metrics require the CloudWatch agent — they are not collected by default on EC2). For databases, it means CPU, memory, storage IOPS, and connection count. For managed services with provisioned capacity (DynamoDB provisioned throughput, ElastiCache node types, Elasticsearch instance sizes), it means the service-specific capacity metric.

The representative time window matters enormously. A database that averages 15% CPU but hits 90% during the first-of-month billing run cannot be safely downsized based on average utilization alone. You need to capture peak utilization across business cycles — weekly, monthly, and if applicable, quarterly or annual. For most workloads, 30 days of peak data is sufficient. For workloads with known periodic spikes (month-end processing, annual enrollment periods, seasonal traffic), you need a window that includes the spike.

A useful heuristic: if peak utilization over a full business cycle stays below 40% of provisioned capacity, the resource is a strong right-sizing candidate with room to downsize by at least one instance class while retaining comfortable headroom for spikes.

Where Identification Breaks Down

The Ownership Problem

The most common failure mode in waste identification is not a tooling problem — it is an ownership problem. You can generate a list of 200 unattached EBS volumes in an afternoon. The hard part is determining whether any of them are intentional.

An unattached volume might be a data volume that an engineer detached temporarily for a migration and intends to reattach. It might be a volume preserved for forensic analysis after a security incident. It might be a volume that nobody remembers creating. Without resource tagging that captures owner, purpose, and expected lifetime, every candidate for cleanup requires a manual investigation that scales linearly with the number of resources. This is why most cleanup efforts stall: not because identification is hard, but because the disposition decision is hard when you lack the metadata to make it safely.

The Level 1 post covered cost attribution through tagging. Here is the specific mechanism by which missing tags become an operational problem: every untagged orphan requires a human to determine whether deletion is safe. In an environment with thousands of untagged resources, this investigation cost exceeds the savings from cleanup, and rational teams choose to do nothing. The waste persists not because no one noticed it, but because the cost of safely eliminating it exceeds the cost of tolerating it.

The Measurement Trap

A subtler failure mode is measuring the wrong thing. Many waste identification tools report potential savings as the full on-demand cost of flagged resources. But if a flagged instance is covered by a Reserved Instance or Savings Plan commitment, terminating it saves nothing — you have already committed to paying for that capacity. The actual savings from eliminating a resource depend on whether it is covered by a commitment, and if so, whether that commitment can be reallocated to another resource.

Similarly, oversizing calculations that recommend moving from an m5.2xlarge to an m5.xlarge report the delta in on-demand pricing. But if the m5.2xlarge is covered by an RI and the m5.xlarge is not, downsizing could actually increase cost in the short term until the commitment expires or is modified. Waste identification that ignores the commitment layer produces recommendations that are technically correct but financially wrong.

The Blast Radius of Cleanup

Deleting resources in production infrastructure carries risk. The failure mode here is treating waste elimination as a low-stakes operation. An orphaned security group that you delete might be referenced by a launch template — the next autoscaling event fails. A snapshot you remove might be the only recovery point for a volume that is still in use. An “idle” Lambda function might be a critical error handler that fires only during outages, precisely the time you cannot afford to discover it is gone.

Safe cleanup requires not just identification but impact analysis: what depends on this resource, and what happens if it disappears? For orphaned resources, this means checking the reference graph in both directions — not just “does this volume have an attached instance?” but “does any launch template, backup policy, or automation script reference this volume?” For idle resources, it means verifying that the idle state is permanent, not periodic.

The Mental Model

Cloud waste is not a single problem with a single solution. It is three distinct failure modes — drift, severed references, and stale assumptions — each with its own formation mechanism, detection method, and risk profile during remediation.

The shift this post is trying to produce is from thinking of waste as “stuff we should clean up” to thinking of it as a continuous accumulation rate that must be counteracted by a continuous detection and disposition process. Every deploy, every teardown, every scaling event, every architectural change creates the conditions for new waste. If your identification process runs less frequently than your infrastructure changes, waste wins.

The second shift is recognizing that identification is the easy half. The hard half is the disposition decision — determining whether a flagged resource is safe to eliminate — and that decision is only as good as the metadata and dependency information available at the time you make it. Investing in tagging, dependency tracking, and resource lifecycle metadata is not overhead; it is what makes waste elimination operationally feasible.

Key Takeaways

  • Idle resources form through drift — the gap between what a resource was provisioned for and what it currently does — and detection must evaluate the full utilization time series, not averages, to avoid killing resources with periodic but critical usage patterns.

  • Orphaned assets result from non-cascading deletion in the cloud resource graph: terminating a parent resource does not automatically clean up dependent resources like volumes, snapshots, Elastic IPs, or security groups.

  • Oversized instances persist because oversizing produces no operational signal — no alerts, no errors, no latency spikes — making it invisible without an active process to compare provisioned capacity against actual peak utilization.

  • Right-sizing analysis must account for full business cycles: 30-day averages miss monthly, quarterly, or seasonal peaks that determine the true minimum capacity requirement.

  • Waste identification that ignores Reserved Instances and Savings Plans produces financially incorrect recommendations — terminating a committed resource saves nothing if the commitment cannot be reallocated.

  • The primary bottleneck in waste elimination is not identification but disposition: determining whether a flagged resource is safe to delete requires ownership metadata and dependency analysis that most environments lack.

  • Every untagged resource increases the marginal cost of cleanup, because each one requires manual investigation to determine intent and safety, eventually making the investigation cost exceed the savings.

  • Safe cleanup requires bidirectional reference checking — not just “is this resource attached to something?” but “does any automation, template, or policy reference this resource?”

← Back to Home