Berlin DevOps Meetup 2024022 - Modernizing CICD Pipelines
Modernizing CI/CD Pipelines
A Case Study on Building a Robust, Secure, and Efficient System for Cloud-Native Development
$whoarewe
Kemal Akkoyun
https://kakkoyun.me
Ali Akca
https://aweris.me
--
eBPF
Disclaimers
The still ongoing so everything in flux!
--
1 On-going work
Some of the reasons might overlap and get confusing. Feel free to ask questions
--
2 We have specific problems!
I hope you do not!
3 Probably these are not the only solutions, yet alone not the perfect one!
Feel free to suggest solutions if you think you know better
Why?
Reproducibility
Artifact Reproducibility (Security)
SolarWinds story
Supply chain security
-
Byte-by-byte repro
-
sigstore (chainguard)
renovate and dependabot -
Superuser previligies
-
Too many kernel versions to test on, too many distros, too many everything
- We need to increase development velocity!
Build Reproducibility (Developer Experience)
Vendor-locking
Environment (local, CI, test, production)
- it works on my machine!!
Energy efficiency (leads to cost)
Maintainability
Modern, understandable tooling
No make, bash scripts
3
What?
Reprodubilbe Builds
- Pinning build dependecies
- nix (devbox)
- renovate and dependabot
- Reproducible Go/Native binaries
2. Goreleaser (add link) - Reprodubicle containers (timestamps)
- podman
- buildx
renovate version pinning
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
- name: Set up Go
uses: actions/setup-go@93397bea11091df50f3d7e59dc26a7711a8bcfbe # v4.1.0
with:
go-version-file: .go-version
cache: false
Cross-platform builds
Go Toolchain
Zig Toolchain
Container-based Cross-platfom builds and images
Vendor-locking
-
The CI/CD locking
- Cost of Migrate / Upgrade
- Choose your poison:
- Dummy wrapper (execute your local scripts)
- it works on my machine
- Having separate workflows for local and ci/cd
- push and pray
- Dummy wrapper (execute your local scripts)
-
Environment (local, CI, test, production)
- it works on my machine!!
-
Local CI runs (Dagger)
- push and pray
Energy efficiency (leads to cost)
Maintainability
Cross-platform testing (future work)
Container-based solutions
- testcontainers
QEMU
Maintainability/Developer Experience/Fast Feedback cycle
How zenith? Language specific SDK
- Mage (why not make? or why not pure dagger?)
- Dagger
What is Dagger?
// Validate runs the build, format, and generate commands and checks if there are any changes in the source code except the out directory.
func (m *CI) Validate() *Container {
return m.Base.
WithFocus().
WithExec([]string{"devbox", "run", "build", "format", "generate"}).
WithExec([]string{"git", "diff", "--exit-code", ":!out/"})
}
::: source
https://github.com/parca-dev/testdata/blob/a92cebddbf420cd95ab3b22b14f15fa36e7c2ef6/ci/main.go#L19-L25
:::
Thank you!
Q&A
--
::: source
https://aweris.me
:::