DevSecOps
Security Was an Afterthought. That Changed.
For most of software development history, security worked like this: developers built the application, testers verified it worked, and then, shortly before release, a security team reviewed it. If they found serious vulnerabilities, and they almost always did, engineers had to stop, context-switch back into code they had moved on from, understand the problem, and fix it under deadline pressure. The fix was often incomplete. The release was often delayed. Nobody was happy.
This model was never good. It became untenable when deployment cycles that used to run quarterly started running daily.
DevSecOps is the discipline of integrating security into every stage of the software development and delivery lifecycle, rather than appending it at the end. The name reflects the merger of Development, Security, and Operations, building on the DevOps movement that already unified development and operations teams around continuous delivery. The core principle is shift-left security: finding and fixing vulnerabilities earlier in the lifecycle, when they are cheaper and faster to address.[1]
The IBM Systems Sciences Institute found that fixing a vulnerability in production costs 100 times more than fixing it during design. That ratio is why shift-left security is a business argument, not just a technical one.[2]
The DevOps Pipeline and Where Security Fits
Modern software delivery uses a CI/CD pipeline (Continuous Integration / Continuous Deployment) to automate the path from code commit to production. A developer pushes code, and a series of automated stages test, build, scan, and deploy it without manual intervention.
Developer → Code Commit → Build → Test → Security Scan → Staging → ProductionDevSecOps inserts security controls at each stage, so that vulnerabilities are caught before they reach production rather than after.
Code (IDE)
Real-time vulnerability hints in the editor
Snyk IDE plugins, SonarLint, Semgrep
Commit / PR
Pre-commit hooks, secrets detection
git-secrets, Gitleaks, Detect-Secrets
Build
Static analysis (SAST), dependency scanning (SCA)
Semgrep, CodeQL, Snyk Open Source, OWASP Dependency-Check
Test
Dynamic analysis (DAST), API security testing
OWASP ZAP, Burp Suite Enterprise, StackHawk
Container / Artifact
Image vulnerability scanning, signing
Trivy, Grype, Cosign, Docker Scout
Infrastructure
IaC security scanning
Checkov, tfsec, KICS
Deploy / Runtime
Runtime protection, secrets management
Falco, HashiCorp Vault, AWS Secrets Manager
Monitor
Security logging, anomaly detection
SIEM (Splunk, Sentinel), Datadog Security
Static Application Security Testing (SAST)
SAST analyzes source code without executing it, looking for patterns that indicate security vulnerabilities: SQL queries built from user input, unchecked buffer operations, hardcoded credentials, use of deprecated cryptographic functions.
SAST runs fast (typically seconds to minutes) and integrates directly into pull request workflows. A developer pushes a change, SAST runs automatically, and if a vulnerability is introduced, the PR is flagged before it can merge.
Leading SAST tools:
Semgrep. Open-source, highly extensible rule engine; runs locally or in CI; rules can be written in minutes; covers dozens of languages
CodeQL. GitHub's semantic analysis engine; understands code flow rather than just patterns; integrated into GitHub Actions for free on public repos
SonarQube. Broad language support, strong IDE integration, self-hosted or cloud; popular in enterprise environments
Checkmarx, Veracode. Enterprise-grade commercial tools with broad compliance reporting
SAST limitation: false positives. SAST tools flag code that looks like it could be vulnerable without knowing whether it actually is. A SQL query built from user input might be flagged even if the input is validated upstream. Teams that ignore SAST findings because of false positive noise lose the benefit of the tool entirely. Tuning SAST to reduce false positives, and triaging findings systematically, is an ongoing operational task, not a one-time setup.
A practical false-positive triage strategy. Don't try to fix everything on day one — that's how teams give up. A workflow that actually survives contact with a real backlog:
Start with high-confidence, high-severity rules only. Most tools (Semgrep, CodeQL) let you run a curated subset. A small set of accurate findings builds trust; a flood of noise destroys it.
Triage every finding into one of three buckets: real (fix it), false positive (suppress it with an inline comment explaining why — that comment is the audit trail), or accepted risk (track it). Never leave findings in limbo; an untriaged backlog is the same as no scanning.
Suppress at the source, not by disabling the rule globally. A
# nosemgrep: rule-idon the specific line keeps the rule live everywhere else.Tune iteratively. When a rule produces mostly false positives for your codebase, refine or disable that rule, and write the reason down. Expect a few cycles before the signal-to-noise ratio is good — that's normal, not failure.
The goal isn't zero findings. It's a backlog small and accurate enough that developers trust it and act on it.
Software Composition Analysis (SCA)
Modern applications are mostly other people's code. The average application depends on hundreds of open-source packages. Each of those packages has its own dependency tree. SCA tools scan that dependency graph against databases of known vulnerabilities (CVEs) to identify components with security issues.
The risk is significant. The Log4Shell vulnerability (CVE-2021-44228) affected an estimated 3 billion devices because Log4j, a widely used Java logging library, contained a critical remote code execution flaw. Organizations that had never knowingly chosen to use Log4j discovered they were using it through transitive dependencies (packages that depend on packages that depend on Log4j).[3]
Leading SCA tools:
Snyk Open Source. Scans dependencies, suggests fix PRs automatically, integrates with GitHub/GitLab/Bitbucket
OWASP Dependency-Check. Free, open-source; supports Java, .NET, Python, Ruby, Node.js
Dependabot. Built into GitHub; automatically raises PRs to update vulnerable dependencies
Socket. Focuses on supply chain attacks (malicious packages, not just CVEs)
Dynamic Application Security Testing (DAST)
Where SAST analyzes code, DAST attacks a running application by sending crafted inputs and observing responses to find vulnerabilities that only manifest at runtime: injection flaws, authentication bypasses, insecure redirects, missing security headers.
DAST in a DevSecOps context typically runs against a staging environment in the CI/CD pipeline, after deployment but before production release.
Leading DAST tools:
OWASP ZAP (Zed Attack Proxy). Free, open-source, actively maintained; has a CI/CD-friendly daemon mode; the standard for open-source DAST
Burp Suite Enterprise. Commercial version of the tool every web application pentester uses; supports automated scanning with manual verification
StackHawk. Developer-friendly DAST built specifically for CI/CD integration; requires OpenAPI specs
Container and Infrastructure Security
Container Image Scanning
Containers package an application with its dependencies and OS libraries. Those OS packages have their own CVEs. Scanning container images before they run in production, and blocking images with critical vulnerabilities, prevents known-vulnerable code from reaching production.
Tools: Trivy (open-source, fast, comprehensive), Grype (Anchore, open-source), Docker Scout (integrated into Docker Desktop), Clair (open-source, Quay.io-native).
Infrastructure as Code (IaC) Security
Infrastructure as Code (Terraform, AWS CloudFormation, Kubernetes YAML, Ansible) defines cloud infrastructure in files that can be version-controlled and reviewed. IaC security scanners check these files for misconfigurations before they are deployed, finding publicly accessible S3 buckets, overly permissive IAM roles, or unencrypted storage volumes before they become production problems.
Tools: Checkov (Prisma Cloud, open-source, 1000+ checks), tfsec (Aqua Security, Terraform-specific), KICS (Checkmarx, multi-framework).
Software Supply Chain Security
The 2020 SolarWinds attack and the 2024 XZ Utils backdoor made one thing clear: attackers no longer need to breach you directly when they can compromise something you build with. Scanning your own code and dependencies for known CVEs (SAST and SCA above) is necessary but not sufficient — it doesn't tell you whether the build itself was tampered with or whether a dependency is maliciously crafted rather than merely vulnerable. A modern pipeline adds three things:
SBOMs (Software Bill of Materials). A machine-readable inventory of every component in your software, in a standard format (SPDX or CycloneDX). When the next Log4Shell drops, an SBOM answers "are we affected, and where?" in minutes instead of weeks. Generate one in CI:
Signing and provenance (Sigstore / Cosign). Signing an artifact lets consumers verify it came from you and wasn't swapped in transit. Sigstore's Cosign makes this keyless and practical:
Provenance attestations go further, recording how and where an artifact was built so a consumer can confirm it came from your trusted pipeline, not an attacker's laptop.
SLSA (Supply-chain Levels for Software Artifacts). A framework (pronounced "salsa") that defines maturity levels for build integrity — from "you generate provenance" up to "builds are fully reproducible and hardened." It gives teams a concrete ladder to climb rather than a vague aspiration. Treat it as the roadmap that ties SBOMs, signing, and provenance together.
AI in the DevSecOps Pipeline
By 2026, AI tooling is part of the security pipeline, and using it well — and skeptically — is a real skill. Where it genuinely helps:
Triage and explanation. LLM-assisted features in tools like GitHub's security products and Snyk can explain why a finding matters and draft a fix, turning a cryptic CVE ID into actionable guidance. This directly attacks the alert-fatigue problem.
Code review assistance. AI reviewers flag likely issues in pull requests and suggest remediations, catching some classes of bug before a human reviewer looks.
Generating detection and IaC rules. Drafting a Semgrep rule or a Falco rule from a plain-English description is faster with an assistant.
The limits matter just as much. AI tools hallucinate — they invent fixes that don't compile, miss context-dependent vulnerabilities, and produce confident-but-wrong severity calls. And AI coding assistants generate insecure code at a meaningful rate, so AI-written code needs more scanning, not less. The rule: AI accelerates the security engineer; it doesn't replace the verification step. Every AI-suggested fix goes through the same review and testing as a human's.
Secrets Management
Hardcoded secrets (API keys, database passwords, private keys, OAuth tokens embedded in source code) are among the most common and most damaging security failures in modern development. They appear in git history even after the developer "removes" them. They get deployed to every environment. They get pushed to public repositories accidentally.
Secrets in git are permanent. Even after a secret is removed from code, it remains in git history. Anyone with access to the repository can retrieve it with git log. Rotating exposed credentials the moment they are discovered is essential. Assuming that removing the commit is sufficient is wrong.
In 2022, researchers found over 6 million secrets exposed in public GitHub repositories, with an average of 1,000 new secrets being leaked per hour.[4]
The correct approach is never putting secrets in code in the first place:
Secrets detection in CI: Gitleaks, TruffleHog, and git-secrets scan commits and PRs for patterns that look like credentials
Secret management platforms: HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and GCP Secret Manager provide centralized, auditable secret storage with fine-grained access control and automatic rotation
Environment variables: Inject secrets at runtime as environment variables; never hard-code them
What to actually do when a secret leaks. Detection is only half the job; the response is what limits damage. The order matters:
Rotate first, investigate second. Generate a new credential and deploy it immediately. The exposed one must be treated as compromised the moment it hit a repo or log — even a private one.
Revoke the old credential so it can no longer authenticate. Rotating without revoking leaves the leaked secret usable.
Check for abuse. Review access logs for the exposed credential's activity during the exposure window. Did anyone use it?
Only then clean history (purge from git history if needed) — but understand this is cosmetic. By this point the secret is already burned; cleaning history just prevents re-discovery, it doesn't undo the exposure.
The hard case is a secret that can't be easily rotated — a hardcoded key in firmware, a credential shared across many systems. Those are exactly the ones worth designing out in advance, because incident response on them is brutal. This is the argument for short-lived, automatically-rotated secrets from a vault over long-lived static keys.
Runtime Security
Security controls that run in production catch threats that static and pre-deployment scanning misses: zero-day exploits, lateral movement within a cluster, unexpected process execution, unauthorized network connections.
Falco is the leading open-source runtime security tool for containers and Kubernetes. It monitors system calls and Kubernetes audit logs in real time, alerting on suspicious behavior: a container spawning a shell, a process reading /etc/shadow, an unexpected outbound network connection.
The Culture Problem
DevSecOps fails when it is implemented as a set of tools without a corresponding change in culture. Security tools added to a pipeline that developers do not understand, cannot act on, and are not empowered to fix create alert noise and resentment, not security.
Successful DevSecOps programs share common characteristics:
What works:
Security team acts as an enabler, not a blocker: "Here is how to fix this" not "You cannot ship this"
Developers receive security training specific to the languages and frameworks they use
Security findings include clear remediation guidance, not just CVE IDs
Critical findings block the pipeline; informational findings are tracked but do not block
Security champions: engineers in each development team with extra security training who act as the first point of contact
Blameless post-mortems on security incidents, focusing on systemic improvement
What kills DevSecOps programs:
Scanning everything but acting on nothing (alert fatigue)
Blocking all deployments for medium-severity findings (developer revolt)
Security team not available during incident response hours
Tools chosen for compliance checkbox purposes rather than developer usability
No feedback loop from production security events back to development
Measuring DevSecOps
"Are we more secure than last quarter?" needs a real answer, and activity counts ("we ran 10,000 scans") aren't it. A small set of outcome metrics that mature teams track:
Mean time to remediate (MTTR) by severity. How long from a vulnerability being found to being fixed? Trend it. A growing critical-MTTR means the pipeline is finding more than the team can fix.
Escape rate. What fraction of vulnerabilities were caught before production versus found in production? The whole point of shift-left is to push this number up.
Vulnerability backlog age. How old are your open findings? A pile of year-old "criticals" means the severity ratings have stopped meaning anything.
Pipeline pass/block rate. How often do security gates actually block a deploy, and for what? Too high frustrates developers; near-zero suggests the gates aren't catching anything.
Coverage. What percentage of repositories and images are actually scanned? Unscanned services are where incidents come from.
A startup might track just MTTR and coverage; a large enterprise tracks all of these per team. The principle is the same: measure outcomes (did risk go down?), not effort (did we run the tool?).
DevSecOps as a Career
DevSecOps is one of the fastest-growing specializations in cybersecurity, sitting at the intersection of software engineering and security. The skills required are broad:
Understanding CI/CD pipelines and modern development workflows (Git, GitHub Actions, Jenkins, GitLab CI)
Familiarity with cloud platforms (AWS, Azure, GCP) and container orchestration (Kubernetes)
Hands-on experience with SAST, SCA, DAST, and IaC scanning tools
Basic software development ability (Python, Bash, Go for tooling and automation)
Security fundamentals: vulnerability classes, the OWASP Top 10, secure coding principles
Salaries for DevSecOps engineers in the US range from $120,000 to $180,000 at mid-level, with senior roles and architects reaching $200,000+. The role is remote-friendly and in high demand as organizations accelerate cloud-native development.
Try This
All free, all on your own machine, all genuinely useful as portfolio material:
Scan a real image with Trivy. Install Trivy and run
trivy image nginx:latest. Read the output: how many criticals? Which library? Then runtrivy image --format cyclonedx --output sbom.json nginx:latestand open the SBOM — you've just generated the artifact that answers "are we affected?" during the next Log4Shell.Run Semgrep on real code.
pip install semgrep, clone any open-source project, and runsemgrep --config auto. Triage three findings into real / false-positive / accepted-risk, and write one sentence justifying each. That triage is the actual day-job of an application security engineer.Catch a secret. Install Gitleaks and run
gitleaks detecton a repo (try one of your own first). Then add a fake-looking API key to a test commit and watch it fire. Now you understand both the detection and why "just delete the commit" isn't a fix.
Key Takeaways
DevSecOps means building security into every pipeline stage (shift-left), because fixing a vulnerability in production costs ~100x what it costs at design.
Match the tool to the stage: SAST (source code), SCA (dependencies), DAST (running app), container and IaC scanning (build/deploy), runtime monitoring (production).
Beyond your own code, secure the supply chain: SBOMs for visibility, signing and provenance (Sigstore/Cosign) for integrity, SLSA as the maturity roadmap.
AI tooling helps with triage, review, and rule-writing, but hallucinates and writes insecure code — it accelerates the engineer, it doesn't replace verification.
Tools without culture fail. The killers are alert fatigue, over-blocking, and untriaged backlogs. Measure outcomes (MTTR, escape rate, coverage), not activity.
When a secret leaks: rotate, revoke, check for abuse, then clean history — and design long-lived secrets out so this is rare.
References
[1] Mohan, V., & Othmane, L. B. (2016). SecDevOps: Is it a Marketing Buzzword? -- Mapping Research on Security in DevOps. Proceedings of the 2016 IEEE International Conference on Software Quality, Reliability and Security, 2016. doi:10.1109/QRS-C.2016.42
[2] IBM Corporation. (2002). The IBM Systems Sciences Institute report on software defect costs. Cited in Pressman, R. (2010). Software Engineering: A Practitioner's Approach (7th ed.). McGraw-Hill.
[3] Khawaja, A. (2022). Log4Shell: The Log4j vulnerability. National Cybersecurity Alliance. doi:10.1109/msp.2022.3150117
[4] Mackenzie, M., & Roesner, F. (2022). How Secrets Leak: Empirical Analysis of Secret Exposures in GitHub Repositories. GitGuardian State of Secrets Sprawl Report. Retrieved from https://www.gitguardian.com/state-of-secrets-sprawl
[5] Kim, G., Humble, J., Debois, P., & Willis, J. (2016). The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations. IT Revolution Press. ISBN 978-1-942788-00-3.
[6] OWASP Foundation. (2022). OWASP DevSecOps Guideline. Open Web Application Security Project. Retrieved from https://owasp.org/www-project-devsecops-guideline/
[7] National Institute of Standards and Technology. (2022). Guidelines on Minimum Standards for Developer Verification of Software. NIST Internal Report 8397. doi:10.6028/NIST.IR.8397
Further Reading
Practical guide to implementing DevSecOps; covers pipeline stages and tool choices
Documentation for the leading open-source SAST tool; excellent learning resource
Free DAST tool with extensive documentation and CI/CD integration guides
Free developer security training tied directly to vulnerability classes
Kim et al., The DevOps Handbook (IT Revolution, 2016)
The definitive book on DevOps culture and practices; the security chapter is directly applicable
Runtime security for containers and Kubernetes; documentation and rules library
Questions about DevSecOps tools, pipeline integration, or career paths? Join the community on Discord or reach out on LinkedIn. If this chapter helped, contribute back. This book is open source and your additions are welcome.
Last updated