Skip to main content
HashnodeSafeDeploySafeDeploy

Command Palette

Search for a command to run...

ATLAS-OPS

MY FIRST LEARNING PROJECT:

Updated
•16 min read•View as Markdown
S
shreya baranwal

I Started ATLAS-OPS for a Hackathon. Somewhere Along the Way, I Learned Backend Engineering.

How a hackathon idea turned into a cloud incident remediation system and taught me much more than I expected.

I didn't start building ATLAS-OPS with a big architecture diagram in my head.

Honestly, it started with a hackathon.

I was looking at the Gemini Live Agent Challenge and the Amazon Nova AI Hackathon, and I had this idea of building something that could combine the two. I wanted to build something around AI, cloud infrastructure and automation, something that was not just another chatbot.

At that point, my thinking was pretty simple:

"Why not build one project and use it for both hackathons?"

That was the plan.

It did not exactly go according to plan.

And I'm actually glad it didn't.

Because while I was building it, I slowly stopped thinking about how to make a hackathon submission and started thinking about how the system itself should actually work.

That's how ATLAS-OPS started taking a completely different shape.

The Problem I Started Thinking About

One thing that kept bothering me was how repetitive some cloud incidents can be.

Imagine an EC2 instance suddenly starts showing unusually high CPU usage. The usual flow might look something like:

CPU gets high
   ↓
Monitoring detects it
   ↓
Someone gets alerted
   ↓
Someone investigates
   ↓
Someone decides what to do
   ↓
Someone performs the remediation
   ↓
Someone checks if it actually worked

For a complicated incident, having a human involved makes complete sense.

But what about incidents that are predictable?

What if we already know:

  • what signals indicate the problem,
  • what action should be taken,
  • what conditions make that action safe,
  • and how to verify whether the action worked?

That made me wonder:

Can we automate the repetitive part of incident response without turning remediation into a dangerous "if this happens, run this command" script?

That question became the foundation of ATLAS-OPS.

From a Simple Trigger to a Policy Engine

My first instinct was honestly pretty basic. Something like:

if CPU > 80% → remediate

But the more I thought about it, the more it felt wrong.

A CPU spike doesn't necessarily mean that something is broken.

  • Maybe there's a legitimate workload running.
  • Maybe the spike lasts for only a few seconds.
  • Maybe CPU is high, but the trend is actually going down.

So instead of looking at just one number, I started thinking about behavior.

That's where the policy engine came in. ATLAS-OPS looks at things like:

  • recent CPU measurements
  • exponential moving average
  • trend/slope
  • sudden spikes
  • a confidence score

The idea is simple:

Don't react just because something is high. Try to understand whether the behavior actually looks abnormal.

This was one of the first moments where I realized that I wasn't just connecting AWS services anymore. I was actually designing decision logic.

Then I Realized I Needed a Backend

This is probably the part of the project I didn't expect when I started.

I had initially thought of the backend mostly as:

API → receive request → do something → return response

But that mental model didn't survive for very long.

Once incidents became a real concept, I needed to answer questions like: What happened to this incident?

Is it:

Created
Approved
Executing
Executed
Verified

Or did something fail?

Executing
   ↓
Failed
   ↓
Retry

And what if it keeps failing?

Attempt 1 → Retry
Attempt 2 → Retry
Attempt 3 → Stop

Suddenly I was dealing with state, transitions, retries and failure handling.

And that's where ATLAS-OPS became my first project where I genuinely started understanding what backend engineering actually means.

It isn't just about creating endpoints. It's about asking:

"What happens when something goes wrong?"

Why I Added SQS

At one point I had another problem. Should my API directly perform the remediation? Something like:

API → EC2

It worked conceptually, but it didn't feel right. The API shouldn't have to sit there waiting while some infrastructure operation happens.

So I introduced SQS:

API
 ↓
SQS
 ↓
Worker
 ↓
EC2

Now the API could create and approve an incident, while a separate worker handled the actual execution.

This was one of those things that looked very simple when I read about queues. Building it made me understand why queues exist.

They aren't just another AWS service to put on a diagram. They change how the system behaves:

  • The API and the remediation worker can fail independently.
  • Work can be processed asynchronously.
  • Retries become possible.
  • The execution layer can evolve separately from the API.

That was a pretty big learning moment for me.

The Part I Didn't Expect: The Fix Can Fail Too

This ended up being one of my favorite parts of ATLAS-OPS.

Initially, I was mostly thinking about:

"How do I detect the incident?"

But I slowly realized that detection is only half the problem. The remediation itself can fail, and even a remediation that runs successfully doesn't guarantee that the system actually recovered.

That's why ATLAS-OPS doesn't stop at executing an action. It verifies the outcome, retries when needed, and rolls back if the remediation doesn't work.

ATLAS-OPS: What Is It?

Cloud systems can detect problems quickly. But detecting an incident is only the beginning.

When an EC2 instance starts behaving abnormally, for example, CPU usage keeps increasing, the usual response still involves a human investigating the metrics, deciding what action to take, executing the remediation, and then checking whether the system actually recovered.

ATLAS-OPS explores how much of this predictable incident-response loop can be automated.

ATLAS-OPS is a Go-based cloud incident detection and remediation system built around SRE automation patterns. It monitors infrastructure signals, evaluates them through a policy engine, and routes approved remediation actions through an asynchronous worker pipeline.

At a high level, the flow is:

Infrastructure Metrics
        ↓
Incident Detection
        ↓
Policy Evaluation
        ↓
Operator Approval
        ↓
Async Remediation
        ↓
Verification
        ↓
Retry / Rollback

The policy engine currently goes beyond a simple threshold. It uses EMA, trend/slope analysis, and confidence scoring to determine whether an infrastructure behavior actually looks abnormal.

Once a remediation decision is made, the action is passed through Amazon SQS to a worker that performs the operation against AWS resources such as EC2. The system also keeps track of incident state and audit information using DynamoDB.

The important part is that ATLAS-OPS is not designed to blindly automate every incident. Critical actions pass through an approval step, keeping an operator in the loop before execution.

What Problem Is It Trying to Solve?

The problem isn't that cloud systems can't detect incidents. They can.

The problem is what happens after the alert.

Consider a common scenario: an EC2 instance experiences sustained high CPU usage.

The monitoring system can tell us:

CPU → 92%

But that doesn't answer:

  • Is this actually an incident?
  • Is the CPU still increasing or already recovering?
  • Should anything be changed?
  • What remediation should be performed?
  • Who should execute it?
  • Did the remediation actually solve the problem?
  • What happens if the remediation fails?

In many environments, these questions still become part of a manual response process. That creates a few recurring problems:

  1. Detection can be faster than response. Monitoring can raise an alert immediately, but investigation and remediation still depend on human availability.
  2. Repetitive incidents consume engineering time. When the same type of infrastructure problem occurs repeatedly, engineers may end up repeating the same investigation and remediation process.
  3. Manual remediation introduces inconsistency. Under pressure, even a well-understood procedure can be executed differently or incorrectly.
  4. A successful action doesn't necessarily mean recovery. Changing an infrastructure resource successfully doesn't guarantee that the underlying problem has been resolved. The system still needs to verify the outcome.

How ATLAS-OPS Is Built

ATLAS-OPS is built as a set of independent components, where each part is responsible for a specific stage of the incident lifecycle. The idea was to keep detection, decision-making, execution, and persistence separated rather than putting everything inside one service.

1. Incident Detection: Turning Signals into Incidents

The first step in ATLAS-OPS is turning an infrastructure signal into a meaningful incident.

When the detection flow is triggered, the API fetches the EC2 instance ID, instance type, and CPU utilization. CPU metrics are retrieved from CloudWatch over the recent 5-minute window and stored for analysis.

EC2
 ↓
CloudWatch CPU Metric
 ↓
Metric Persistence
 ↓
Policy Evaluation
 ↓
HIGH_CPU?
 ├── No  → System Healthy
 └── Yes → Create Incident

The key design decision is:

A metric is an observation; an incident is a decision.

If the policy identifies abnormal behavior, ATLAS-OPS creates an Incident with its ID, instance details, CPU value, recommendation, timestamp, and initial DETECTED state.

2. Policy Engine: From Thresholds to Behavioral Signals

A simple monitoring system might say:

CPU > 75% → High CPU → Remediate

But a single threshold crossing does not necessarily mean the system is actually in trouble. A short-lived spike, noisy metric, or insufficient historical data could trigger a false remediation.

So instead of treating a single threshold crossing as an automatic remediation trigger, ATLAS-OPS evaluates how the metric is behaving over time.

The policy engine follows:

Metric
  ↓
EMA
  ↓
Slope / Trend
  ↓
Spike
  ↓
Confidence
  ↓
Policy Decision

EMA: Smooth the signal

The Exponential Moving Average (EMA) reduces the effect of short-lived fluctuations in CPU utilization.

ema := calculateEMA(metrics, alpha)

With:

alpha := 0.3

the latest observations have more influence while historical observations still contribute to the result.

The goal isn't to react to every noisy data point, but to get a more stable view of the system's recent CPU behavior.

Slope: Is the metric actually increasing?

Average CPU alone doesn't tell us whether the system is getting better or worse. The policy engine also calculates the trend:

slope := calculateSlope(metrics)

The implementation compares the first and latest CPU values and normalizes the difference by the number of samples:

rawSlope := last - first
return rawSlope / float64(len(metrics))

So the engine can distinguish between:

High but stable CPU
        vs.
High and continuously increasing CPU

That distinction matters when deciding whether remediation is justified.

Spike: Detect sudden deviation

The engine also looks at how far the current CPU value is from the EMA:

spike := detectSpike(current, ema)

The difference is normalized between 0 and 1:

diff := current - ema
if diff <= 0 {
    return 0
}
return math.Min(diff/100.0, 1)

This gives the policy engine another signal for identifying sudden changes.

Confidence: Combine the signals

Instead of letting one metric decide everything, ATLAS-OPS combines multiple signals into a single confidence score.

confidence := loadScore*0.5 + trendScore*0.2 + spike*0.2 + stabilityScore*0.1

The Actual Policy Decision

After calculating these signals, the engine applies the remediation policy:

if ema > 75 && conf > 0.7 {
    return Decision{
        Status:         "HIGH_CPU",
        Confidence:     conf,
        AverageCPU:     ema,
        Slope:          slope,
        Recommendation: "Scale up instance",
    }
}

So the actual decision is:

EMA > 75 AND Confidence > 0.7
        ↓
    HIGH_CPU
        ↓
Recommendation: Scale up instance

Otherwise:

→ HEALTHY

And if there isn't enough historical data:

→ INSUFFICIENT_DATA

That last case is important too: the system doesn't try to make a remediation decision when it doesn't have enough information.

3. SQS Async Queue

Once an operator approves a remediation, the API does not execute the infrastructure operation directly. Instead, it publishes the incident ID to Amazon SQS.

API → SQS → Worker → Remediation

The queue decouples the API from the execution layer. The API can finish its request without waiting for a potentially long-running EC2 operation, while the worker independently consumes and processes the task.

This also gives ATLAS-OPS a simple producer-consumer architecture:

  • API → Producer
  • SQS → Message buffer
  • Worker → Consumer

The result is better separation of concerns and failure isolation: the API coordinates the work, while the worker is responsible for executing it.

4. Execution Layer: The Worker

The worker runs as a separate process and follows a simple lifecycle:

SQS Message
   ↓
Read Incident ID
   ↓
Fetch Incident
   ↓
Execute Remediation
   ↓
Update State

cmd/worker/main.go acts as the worker process entry point, while worker/processor.go contains the actual processing logic.

The worker also handles execution attempts and uses an atomic state transition from APPROVED to EXECUTING, helping prevent multiple workers from processing the same incident simultaneously.

Keeping the worker separate from the API gives each service a focused responsibility: the API handles requests and approvals, while the worker handles long-running remediation tasks.

5. EC2 Remediation

The remediation layer separates AWS interaction from remediation behavior.

aws/ec2.go is responsible for communicating with the EC2 API through the AWS SDK:

Go
 ↓
AWS SDK
 ↓
EC2 API

It provides operations such as finding an EC2 instance and retrieving its current instance type.

The higher-level logic lives in execution/ec2_scaler.go. This layer decides how the remediation operation is performed:

Execute
   ↓
Verify
   ├── Success → Continue
   └── Failure
          ↓
       Rollback

The EC2Scaler provides separate methods for DryRun, Execute, and Rollback. This keeps the AWS SDK interaction relatively isolated while the execution package owns the remediation behavior.

6. DynamoDB Persistence

ATLAS-OPS uses DynamoDB to keep both incident state and infrastructure metrics persistent across the system.

The incident store handles the incident lifecycle:

Incident
   ↓
DynamoDB
   ↓
State persists across API / Worker

incident/dynamo_store.go provides operations to create, fetch, update, and transition incident states. It also uses conditional state transitions, which helps ensure that a worker only moves an incident from the expected previous state.

Infrastructure metrics are handled separately by infra/metrics_store.go. CPU measurements are stored with the instance ID and timestamp, allowing the policy engine to retrieve the recent five-minute history for trend analysis.

This separation keeps incident state and infrastructure telemetry as two different persistence concerns.

7. Audit Trail

For a remediation system, knowing the current state isn't enough. I also wanted to know how the incident got there.

That's why ATLAS-OPS maintains a separate audit trail for important lifecycle events.

incident/audit.go stores events with details such as the incident ID, timestamp, action, actor, result, and CPU values before and after remediation.

For example, after verification, the system can record:

Action:     VERIFICATION
Actor:      SYSTEM
Result:     VERIFIED
CPU Before: ...
CPU After:  ...

The audit timeline can then be retrieved for a specific incident, giving the system a history of what happened, who triggered it, and what the outcome was.

Current state tells me where the incident is. The audit trail tells me how it got there.

8. Incident Model & Lifecycle

At the center of ATLAS-OPS is the Incident model in incident/model.go.

It defines what an incident is and stores the information needed throughout its lifecycle: instance details, CPU, recommendation, state, execution attempts, and verification data.

The lifecycle is represented as explicit states:

DETECTED
   ↓
APPROVED
   ↓
EXECUTING
   ↓
EXECUTED
   ↓
VERIFIED

There is also a failure path where repeated execution attempts can eventually lead to:

EXECUTING
   ↓
FAILED / RETRY
   ↓
FAILED_PERMANENT

This state-based design is important because remediation is not a single operation. An incident moves through multiple stages, and each stage represents a different point in the execution lifecycle.

The model also stores fields such as ExecutionAttempts, CPUAfter, and VerifiedAt, allowing the system to track retries and whether the remediation actually improved the situation.

9. Incident Metrics

Apart from storing individual incidents, ATLAS-OPS also maintains incident-level metrics to understand overall remediation outcomes.

incident/metrics.go aggregates information such as:

  • Total incidents
  • Incidents by state
  • Total executions
  • Successful executions
  • Rollbacks
  • Overall success rate

This turns individual incident records into a higher-level view of how the remediation system is performing.

10. CloudWatch Integration

CloudWatch provides the infrastructure signal that feeds the detection pipeline:

EC2
 ↓
CloudWatch
 ↓
ATLAS-OPS
 ↓
Policy Engine

The integration uses the AWS SDK to query the EC2 CPUUtilization metric for a specific instance. The current implementation looks at a five-minute window and requests the average CPU value over a 300-second period.

input := &cloudwatch.GetMetricStatisticsInput{
    Namespace:  aws.String("AWS/EC2"),
    MetricName: aws.String("CPUUtilization"),
    Period:     aws.Int32(300),
    Statistics: []cloudwatchTypes.Statistic{
        cloudwatchTypes.StatisticAverage,
    },
}

The returned metric is then persisted and passed to the policy engine for trend analysis.

The important distinction is that the CloudWatch integration is implemented, while live metric ingestion is still being validated end-to-end. So ATLAS-OPS is not being presented as a continuously running production monitoring system yet.

11. Terraform: Defining the Infrastructure

I used Terraform to define the AWS infrastructure for ATLAS-OPS as code instead of creating each resource manually through the AWS Console.

This was especially useful for a project with multiple AWS components. The infrastructure could be recreated consistently and changes could be reviewed before being applied.

The Terraform configuration defines resources such as:

  • SQS: incident processing queue
  • DynamoDB: incidents, metrics, and audit data
  • IAM: roles and instance profile
  • Security Group: network access rules

For example, the incident queue is defined as:

resource "aws_sqs_queue" "incident_queue" {
  name                       = "atlas-incident-queue"
  visibility_timeout_seconds = 30
  message_retention_seconds  = 86400
}

Instead of manually creating these resources, I could use the Terraform lifecycle:

terraform init
     ↓
terraform validate
     ↓
terraform plan
     ↓
terraform apply
     ↓
terraform destroy

I also validated the infrastructure through this complete lifecycle, which made Terraform more than just a provisioning tool. It became a way to reproduce and verify the infrastructure setup.

The application code defines how ATLAS-OPS behaves. Terraform defines the AWS environment it runs with.

12. API Entry Point

The API service starts from cmd/api/main.go. Its responsibility is intentionally small: load the AWS configuration and hand it over to the API server.

cmd/api/main.go
      ↓
Load AWS Config
      ↓
api.StartServer()
      ↓
Routes & Services

cfg, err := config.LoadDefaultConfig(context.Background())
if err != nil {
    log.Fatalf("unable to load AWS config: %v", err)
}

api.StartServer(cfg)

This keeps application startup separate from the actual HTTP server and business logic.

AIOps / Intelligent Operations

There is also a natural path toward AIOps.

The current ATLAS-OPS policy engine is deliberately deterministic:

EMA + Trend + Spike + Confidence
            ↓
     Policy Decision

A future version could introduce ML-based anomaly detection or an LLM for incident explanation and recommendation, while keeping the actual infrastructure execution behind deterministic policies and approval controls.

That distinction is important:

AI can help understand the incident. Policies should still control what the system is allowed to do.

11 views

Comments

Join the discussion

No comments yet. Be the first to comment.

More from this blog

S

SafeDeploy

2 posts