When you host your own source code on the very platform you’re deploying, a single outage can create a perfect feedback loop of failure. GitHub faces this exact challenge: every deploy script runs on machines that depend on the services they’re trying to fix. In this listicle, we distill the core insights from GitHub’s adoption of eBPF (extended Berkeley Packet Filter) to break these circular dependencies and make deployments safer. Whether you manage stateful infrastructure or just want to understand modern observability, these ten points will show you how eBPF can guard against hidden, direct, and transient dependency loops.
1. The Self-Hosting Circular Dependency
GitHub stores all its source code on github.com itself. This creates a fundamental circular dependency: to deploy a fix when the site is down, you need the source code that is unavailable because the site is down. The company mitigates this with an offline mirror of the code and pre-built assets for rollbacks, but that is only the first layer of the problem. Even with a mirror, the deployment scripts themselves can introduce new loops by pulling tools or binaries from GitHub during an outage. Recognizing this core issue is the first step—if your deployment system depends on the very service it is supposed to repair, you have a vulnerability that eBPF can help address.
2. Direct Dependency Risks in Deploy Scripts
Consider a MySQL outage that prevents GitHub from serving release data. To fix the database, an operator runs a deploy script on the affected nodes. If that script tries to fetch the latest release of an open source tool from GitHub (e.g., a diagnostic binary), the operation fails because GitHub cannot serve the data. This is a direct dependency: the script explicitly relies on the service that is down. Such direct circular dependencies are easy to spot in theory but often slip through code reviews. eBPF can intercept these network calls before they fail, allowing the deployment system to block or reroute them safely.
3. Hidden Dependencies: The Silent Saboteurs
Even if the deploy script does not explicitly call GitHub, the tools it runs may have hidden dependencies. For example, a servicing tool already on disk might check for updates by contacting GitHub at startup. During an outage, that check can cause the tool to hang or fail silently unless the update check times out gracefully. These dependencies are difficult to detect because they are embedded in third‑party binaries. eBPF provides a way to observe all outbound calls from the deployment process, including those made by child processes, and block or mock them without modifying the tools themselves.
4. Transient Dependencies Through Internal APIs
A deploy script may call an internal service—like a migration orchestrator—via an API. That service, in turn, might fetch a binary from GitHub to use for the migration. This creates a transient dependency: the script does not directly touch GitHub, but the chain of service calls leads back to the broken service. The failure propagates upward, causing the deploy to stall. Transient dependencies are the hardest to detect because they involve multiple hops across microservices. eBPF can trace network packets across process boundaries, giving operators a complete map of dependencies at runtime.
5. Why Traditional Mitigation Falls Short
Before eBPF, GitHub relied on each team manually reviewing deployment scripts and identifying circular dependencies. This approach is error‑prone and does not scale. Scripts change frequently, and new dependencies appear without notice. Moreover, the review only catches explicit calls, not hidden or transient ones. Some teams resorted to air‑gapped scripts that used only local resources, but that added complexity and limited functionality. A systematic, runtime‑based solution was needed—one that could enforce dependency safety without requiring changes to every script. eBPF fills that gap by operating at the kernel level.
6. Introducing eBPF as a Deployment Guardian
eBPF allows you to run sandboxed programs inside the Linux kernel without modifying kernel source or loading modules. GitHub attached eBPF programs to the deploy process, specifically to monitor and control system calls related to network access. When a deploy script tries to make an outbound connection (e.g., to api.github.com), the eBPF program can log the attempt, allow it, or block it based on predefined policies. This gives operators a safety net: even if a script contains a circular dependency, eBPF can prevent it from reaching the downed service. The result is a deployment that fails only when truly unrecoverable, not because of a preventable loop.
7. How eBPF Monitors Network Calls
At the heart of GitHub’s solution is the eBPF tracepoint on connect and sendto syscalls. When the deploy process opens a socket to a destination IP, the eBPF program reads the destination address and compares it against a denylist of internal services that are considered critical but currently degraded. The program can also inspect the process ID and command name to ensure only deploy scripts are affected, not other system processes. This fine‑grained filtering avoids breaking unrelated network traffic. All decisions are recorded via a ring buffer that user‑space agents can read for auditing and alerts.
8. Selective Blocking Without Breaking Everything
Blocking all network calls during a deploy is too aggressive—many scripts need to access non‑circular endpoints, like package registries outside the downed service. GitHub’s eBPF policy uses a combination of whitelists and denylists updated by an orchestration service. For example, if the MySQL cluster is down, the denylist includes the MySQL configuration endpoint. Scripts can still reach S3 for pre‑built binaries or the internal CI system for logs. The blocking is also time‑bounded: after a configurable timeout, the eBPF program falls back to allowing the call to avoid hangs. This balance keeps deployments safe without crippling them.
9. Implementation at GitHub Scale
Rolling out eBPF to thousands of stateful hosts required careful testing. GitHub started by running eBPF in monitoring mode only, logging potential circular dependencies without blocking. This gave teams visibility into previously unknown dependency chains. After a period of validation, they switched to enforcement mode for a small subset of hosts, gradually expanding. They also built a centralized dashboard to view aggregated eBPF logs, helping engineers quickly identify which scripts need rewriting. The key lesson: start with observability, then enforce. The performance overhead was negligible (less than 1% CPU) because eBPF programs are just‑in‑time compiled and run in kernel context.
10. Lessons for Your Own Infrastructure
You don’t need to be GitHub to benefit from eBPF for deployment safety. Any organization that operates stateful services can apply these principles:
- Map your deployment dependencies using eBPF in monitoring mode.
- Create a denylist of services that, when down, should not be contacted by deploy scripts.
- Implement a fallback policy (allow after timeout) to prevent hangs.
- Combine eBPF with existing CI/CD pipelines to auto‑generate dependency graphs.
GitHub’s journey shows that even the most complex circular dependencies can be tamed with runtime enforcement. By layering eBPF on top of traditional code review, you create a safety net that catches what humans miss. Next time you write a deploy script, ask yourself: what would eBPF block? The answer could save your next incident response.