Stop Deploying Garbage to Production

Stop Deploying Garbage to Production

Last month I watched a deployment workflow push code to production with three critical vulnerabilities, a failing integration test, and an API key hardcoded in plain text. The team’s response? “We’ll fix it next sprint.”

Two weeks later, that API key was on GitHub’s leaked secrets list. The vulnerabilities got exploited. And suddenly “next sprint” became “incident response war room.”

This wasn’t some junior developer’s side project. This was an enterprise team at a company with security certifications, compliance officers, and a CISO who probably makes more than all of us combined.

Here’s what their pipeline—and probably yours—looks like:

  • Failing tests treated as warnings, not blockers
  • Vulnerability scans that run but never fail the build
  • SAST (Static Application Security Testing) findings suppressed “temporarily”… for six months
  • Secrets committed to Git because “it’s a private repo”
  • Zero approval gates between git push and production
  • No rollback plan when everything catches fire

Every single pattern on this list is a breach waiting to happen. And if you’re ISO 27001 certified? These patterns also violate Controls A.14.2 and A.18.2—which means your certification is essentially fraudulent.

Let me show you what secure deployment actually looks like.

The “Ship It” Mentality (And Why It’s Destroying You)

Here’s what most deployment pipelines actually look like. I’ve seen this exact pattern—or worse—at companies you’ve definitely heard of:

name: YOLO Deploy

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Run Tests
        run: npm test || echo "Tests failed, deploying anyway"
      
      # No vulnerability scanning. No SAST. No secrets detection.
      
      - name: Deploy to Production
        env:
          API_KEY: "sk-prod-abc123"  # Hardcoded secret in Git history forever
        run: npm run deploy

What’s catastrophically wrong here?

Tests don’t block anything. That || echo pattern means failures get logged but deployment continues. You might as well not have tests at all.

No security scanning whatsoever. No vulnerability checks. No static analysis. No secrets detection. An attacker could commit a backdoor and your pipeline would cheerfully deploy it.

Secrets in plain text. That API key is now in your Git history. Forever. Even if you delete the file, anyone with repo access can find it. And if your repo ever gets leaked, cloned, or accessed by a compromised developer machine? Game over.

Zero approval gates. Push to main, deploy to production. No human verification. No “are we sure about this?” checkpoint. Just blind faith that the code works.

This pipeline doesn’t just violate security best practices—it violates ISO 27001 Controls A.14.2 (Secure Development) and A.18.2 (Security Reviews). If you have that certification and this is your pipeline, you’re one audit away from losing it.

The Excuses (And Why They’re All Wrong)

I’ve heard every rationalization for shipping without security gates. Let me save you the breath.

“We’re a startup—we’ll add security later.”
No, you won’t. Technical debt compounds. The “later” you’re imagining never arrives because you’re always shipping the next feature. Meanwhile, you’re building on a foundation of sand. When you finally try to add security, you’ll discover it requires rewriting half your infrastructure.

“Security scanning slows us down.”
By 2-5 minutes. That’s it. A proper security gate adds minutes to your pipeline. A breach costs weeks of incident response, customer notification, regulatory reporting, and reputation damage. You’re not saving time—you’re borrowing it at loan shark interest rates.

“We trust our developers.”
Cool. I trust my developers too. I also know that trusted developers make mistakes. They commit secrets by accident. They copy-paste vulnerable code from Stack Overflow. They forget to update dependencies. Trust is not a security control. Automated scanning is.

“The deadline is more important.”
Than what, exactly? Than not getting breached? Than keeping your certification? Than not explaining to your CEO why customer data is on the dark web? There is no deadline important enough to justify deploying unverified code to production.

Here’s the uncomfortable truth: Teams that bypass security gates don’t have a velocity problem. They have a discipline problem dressed up as agility.

What Actual Security Looks Like

Enough about what’s broken. Here’s a GitHub Actions workflow that actually protects your systems. Copy it. Use it. Stop deploying garbage.

name: Secure Deployment

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

permissions:
  contents: read
  security-events: write
  pull-requests: write

jobs:
  # Gate 1: Tests must pass
  tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-dotnet@v4
        with:
          dotnet-version: '10.0.x'
      - run: dotnet test --configuration Release

  # Gate 2: No vulnerable dependencies
  dependency-scan:
    runs-on: ubuntu-latest
    needs: tests
    steps:
      - uses: actions/checkout@v4
      - run: |
          dotnet list package --vulnerable --include-transitive 2>&1 | tee vuln.txt
          grep -q "vulnerable packages" vuln.txt && exit 1 || exit 0

  # Gate 3: SAST analysis
  code-scanning:
    runs-on: ubuntu-latest
    needs: tests
    steps:
      - uses: actions/checkout@v4
      - uses: github/codeql-action/init@v3
        with:
          languages: 'csharp'
      - uses: github/codeql-action/autobuild@v3
      - uses: github/codeql-action/analyze@v3

  # Gate 4: No hardcoded secrets
  secrets-scan:
    runs-on: ubuntu-latest
    needs: tests
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: trufflesecurity/trufflehog@v3.82.13

  # Staging: All gates must pass
  deploy-staging:
    runs-on: ubuntu-latest
    needs: [tests, dependency-scan, code-scanning, secrets-scan]
    environment: staging
    steps:
      - uses: actions/checkout@v4
      - run: dotnet publish -c Release -o ./publish
      - run: echo "Deploy to staging"

  # Production: Requires manual approval via GitHub Environment
  deploy-production:
    runs-on: ubuntu-latest
    needs: deploy-staging
    environment: production  # This enforces manual approval
    steps:
      - uses: actions/checkout@v4
      - run: dotnet publish -c Release -o ./publish
      - run: echo "Deploy to production"
      
      # Audit trail for compliance evidence
      - run: |
          echo "Deployed by: ${{ github.actor }}"
          echo "Commit: ${{ github.sha }}"
          echo "Timestamp: $(date -Iseconds)"

Why this actually works:

Every gate is mandatory. The needs chain means production deployment literally cannot happen unless tests pass, vulnerabilities are clean, SAST finds nothing critical, and no secrets are detected. There’s no continue-on-error escape hatch.

Secrets stay secret. No credentials in the workflow file. GitHub environments handle authentication. The permissions block enforces least privilege—the workflow can only do what it absolutely needs.

Humans verify production deployments. The environment: production line triggers GitHub’s environment protection rules. Someone has to explicitly approve before code hits production. That approval is logged forever.

Audit trail is automatic. Every deployment records who approved it, what commit was deployed, and when. When auditors ask for evidence (and they will), you have it.

This satisfies ISO 27001 Controls A.14.2 (Secure Development) and A.18.2 (Security Reviews). But more importantly, it prevents you from deploying exploitable code to production.

Setting Up the Approval Gate

That environment: production line is where the magic happens. Here’s how to configure it:

  1. Repository Settings → Environments → Create “production”
  2. Add protection rules:
    • Required reviewers: 1-2 people who aren’t the deployer
    • Deployment branches: main only (no deploying random feature branches)
    • Prevent self-review: You can’t approve your own deployment

Now every production deployment requires a second pair of eyes. Someone has to look at the PR, verify the security gates passed, and explicitly click “Approve.”

Is this slower than YOLO deploying? Yes, by about 30 seconds. Is it worth avoiding breaches and audit failures? Obviously.

The Evidence Trail (For When Auditors Come Knocking)

Auditors don’t care about your intentions. They want proof. With this setup, you automatically generate:

  • Test execution logs: Every workflow run shows what tests ran and whether they passed
  • Vulnerability scan results: CodeQL findings, dependency audit reports, Dependabot alerts
  • Approval records: Who approved each production deployment, when, and for what commit
  • Deployment history: Timestamped record of every production push

When an auditor asks “How do you ensure security validation before deployment?” you don’t explain your process. You show them the GitHub Actions logs. Evidence beats explanation.

Export these quarterly. Keep them for at least three years. You’ll need them.

Ways Teams Screw This Up

The “continue-on-error” Trap

# WRONG: Continuing on error defeats the entire purpose
- name: Security scan
  run: npm audit
  continue-on-error: true

ISO violation: If the gate can be bypassed, it’s not a control. Control A.14.2 requires enforced security measures, not suggestions.

Fix: Remove continue-on-error. If the scan finds issues, deployment must halt until remediated.

Accidentally Logging Secrets

# WRONG: Secrets in environment variables are still visible in logs if misused
- name: Deploy
  env:
    API_KEY: ${{ secrets.API_KEY }}
  run: echo "Using key $API_KEY"  # Logs the secret!

ISO violation: Control A.9.4.5 requires protection of authentication credentials. Accidental logging exposes secrets.

Fix: Never reference secrets in commands that might log them. Use secrets only in secure contexts:

- name: Deploy
  env:
    API_KEY: ${{ secrets.API_KEY }}
  run: ./deploy.sh  # Script uses $API_KEY internally, never echoed

No Rollback Plan

You deployed. It’s broken. Now what? If your answer is “panic,” you’ve already failed.

Every production deployment needs a rollback strategy. Ideally automated—health check fails, previous version gets redeployed. At minimum, documented and tested.

Deploying from Random Branches

# WRONG: Any branch can deploy to production
on:
  push:
    branches: ['**']

If feature branches can deploy to production, your approval gates are meaningless. Someone can push to my-feature-branch and bypass every review.

Fix: Lock production deployments to main only. Use GitHub environment branch restrictions to enforce it.

Going Further (Because Compliance Is the Floor, Not the Ceiling)

The workflow above is the minimum. Here’s how to make it actually effective:

Fail on Real Thresholds

  • Code coverage: 70% minimum. Below that, you’re guessing whether code works.
  • SAST severity: Block on critical and high. Medium can be warnings.
  • Dependencies: Zero critical CVEs (Common Vulnerabilities and Exposures). Period.
  • Secrets: Any detection = immediate failure. No exceptions.

Catch Problems Earlier

Security gates at deployment time are good. Security gates at commit time are better:

  • Pre-commit hooks for secrets detection (find them before they hit the repo)
  • PR checks for SAST and tests (fail fast, fix fast)
  • Dependabot alerts enabled and actually triaged (not just ignored)

Handle Exceptions Properly

Sometimes you genuinely need to deploy with a known medium-severity issue. That’s fine—but document it:

  • Written justification (why this can’t wait)
  • Approval from someone who isn’t the developer
  • Deadline for remediation (not “someday”)
  • Tracked in your issue system

The gate stays. You just document why you’re accepting the risk.

Stop Making Excuses

The workflow I showed you isn’t complicated. It’s maybe 70 lines of YAML. Copy it, adapt it to your stack, enforce it.

What’s hard isn’t the technical implementation. What’s hard is the discipline to keep the gates closed when someone with a title says “we need to ship NOW.”

But here’s what I’ve learned from 15 years of watching deployments go wrong: the shortcuts always cost more than the delay. The security scan you skipped catches the vulnerability that gets exploited. The approval you bypassed would have noticed the breaking change. The rollback you didn’t plan for becomes a 3 AM incident.

Security gates aren’t bureaucracy. They’re the engineering discipline that separates professionals from gamblers.

Implement them. Enforce them. Stop deploying garbage to production.

Comments

VG Wort