Skip to main content

Adopting Dagger

Once you have completed the quickstart, and learned the basic concepts, it's time to take the next step: adopting Dagger in your project. We call this daggerizing. But how does one daggerize a project, exactly?

Every project is different, and every pipeline is a unique snowflake (usually that's not a good thing). So we can't give you an exact step-by-step guide of how to daggerize your project. But we can share the collective experience of the Dagger community, distilled into a simple framework.

important

Daggerizing your pipeline may take some time, but it's a worthwhile investment. Start small, and you'll see the benefits as you progress. Check out these livestreams where others have taken the time to do it right, and see how they approached the process.

Phase 1: research​

Join our Discord!​

Before anything, drop what you're doing and join our Discord server. It is the beating heart of the Dagger community, and one of the killer features of the platform. It is the best place to ask questions, get inspiration, and ask for help when you are stuck.

Consider yourself warned: adopting Dagger without joining our Discord is certainly possible, but you probably won't be as productive, and it definitely won't be as fun!

Look for inspiration​

A useful step is to learn about other projects who have successfully adopted Dagger. This will give you a point of reference, and serve as inspiration.

You can find a large collection of case studies and demos videos from on our website.

Look for red flags​

Before going any further, you should look for reasons not to adopt Dagger. Your project may not be a good fit for Dagger if:

  • It is so early that it doesn't need CI, and nobody worries about pipelines yet
  • Your only output is a Windows, Mac, iOS or Android application.
  • You are happily using a monolithic toolchain, such as Gradle, Nix or Bazel, with no exception and no fragmentation within the team
  • Your pipelines are heavily dependent on Windows or Mac runners

In doubt, don't hesitate to ask the community - you've joined the Discord, right? If you have a use case that you’d like to talk about privately, you can also reach out to us at solutions@dagger.io.

Phase 2: the POC​

It's best to start your adoption journey with a proof-of-concept (POC for short). The goal of the POC is to adopt Dagger on a small part of your project, ideally a single pipeline, and quickly demonstrate the value. Then leverage the momentum from that POC to expand the scope of adoption.

Scoping your POC​

The key to a successful Dagger POC is to pick the right pipeline to daggerize. The ideal first pipeline has three properties:

  1. It suffers from a hair-on-fire problem which daggerizing can solve. For example:
  • It can't run across dev and CI
  • It can't run across different CI
  • It awkwardly glues together multiple systems, which makes it brittle
  • Not enough people know how it works
  • It's very slow and inefficient because of poor caching
  • It's so painful to change, that nobody wants to change it

  1. It can be daggerized within a week

  2. You have the authority to daggerize it

Depending on the size and constraints of your project, this first pipeline could perform a build, test, deployment, or all the above. It could also automate a custom project-specific task.

Choosing a language​

When choosing the language to use for your pipeline, follow these principles:

  • Optimize for participation. The more people on the team are likely to participate in the development of the pipeline, the better.

  • Check SDK availability. Dagger supports 3 stable SDKs (Python, Go, Typescript), but it also has several experimental SDKs, some of which are actively used in production despite not being officially supported. Take the time to investigate the status of available SDKs, and ask the community for their opinion. Our Discord server is a good place to start - each language has a dedicated channel.

  • Polyglot pipelines for a polyglot stack. If your team is divided in language siloes, consider writing a different module for each silo, each written in the ideal language. Dagger supports cross-language linking for exactly this reason - take advantage of it!

Implementing​

If you haven't already, make sure to complete the Quickstart, which will teach you how to daggerize an example pipeline.

Additionally, you can use the following resources:

Integrating with your dev toolchain​

Once your pipeline is daggerized, in the form of one or more Dagger modules, it's time to integrate it with your development toolchain. Dagger is designed for flexibility, so you have many choices in how to integrate it. The downside is that it may feel like too much flexibility. What is the correct way to integrate?

Here are a few guidelines:

  • If your team is willing and able to use the dagger CLI directly, we recommend that. Typically that means documenting a few dagger call commands in your README, explaining how to use dagger functions to discover more functions, and linking to the Dagger documentation to learn more.

  • If your team prefers (or is forced to) use another tool, whether off-the-shelf like make, or custom-built tool or script, it's perfectly fine to wrap your dagger call invocations in existing Makefile, configuration files or scripts. Some power users may scratch beneath the surface and call dagger call directly; others won't. Both are fine.

  • In some rare cases, invoking the dagger CLI may not possible or acceptable, and embedding Dagger as a library in an existing codebase might be the best option. This is possible, but should be used as a last resort, as there are limitations. Most notably you will only have access to the core Dagger API, without the ability to extend it with Dagger Modules. We plan to remove this limitation in the future.

Integrating with CI​

Once your pipeline is daggerized, and integrated in your development toolchain, it's time for the final step: integrating it in CI.

Since your daggerized pipelines are packaged as Dagger Modules, and modules only have one dependency - the dagger CLI - integrating your daggerized pipelines boils down to three steps:

  1. Decide which CI event should trigger which Dagger pipeline
  2. For each pipeline to trigger, determine how to map inputs from the CI environment into arguments to the Dagger functions composing the pipeline
  3. Write the resulting dagger call command for each pipeline.
  4. Find the integration page for your CI platform, and follow the instructions. For example: GitHub, GitLab CI, CircleCI, Jenkins, Azure Pipelines, AWS CodeBuild, Argo Workflows and Tekton.

A few important notes on CI integration:

  • If you're worried about breaking your existing CI workflow, remember that most CI platforms make it easy to add new pipelines without removing existing ones. Don't hesitate to take advantage of this, and run both versions of the pipeline in parallel for a while: daggerized, and non-daggerized. This can also help assess performance gains. Once you're confident that the daggerized pipeline is running well, you can remove the old one.

  • The typical CI architecture makes it easy to add new pipelines without removing existing ones. Don't hesitate to take advantage of this, especially if you're worried about breaking your existing CI.

  • Daggerizing leads to simplifying. It's common to merge several large CI pipelines into a single one that just wraps dagger call. This usually leads to massive simplification of the CI configuration, as complex YAML/Groovy/shell spaghetti is replaced by clean code. Taken to the extreme, this process reduces the entire CI configuration to a single dagger call, with everything else happening inside Dagger. Although this sometimes happens, in practice most projects converge to a middle ground, where the CI configuration shrinks to just enough dagger call invocations to take advantage of proprietary CI features. Usually these features are job scaling, and job visualization. The more dependent you are on these proprietary CI features, the more granularity you will need to keep in your CI configuration.

  • Remember that Dagger pipelines are not distributed. Each dagger call will execute a single Dagger pipeline on a single Dagger engine. It's the responsibility of the CI system to dispatch jobs across multiple machines. This means that if a single dagger call pipeline doesn't fit in a single machine, you should split it into smaller dagger call pipelines, and configure your CI to run them separately. This way, you can take advantage of the usual scaling features of your CI.

  • Caching makes everything faster. As a general rule, CI pipelines are very inefficient, so it's common that daggerizing them will make them faster. Importantly, it may also change your scaling requirements. In extreme cases, a pipeline that needed horizontal scaling across many machines, might fit comfortably on a single machine after daggerizing.

  • Caching is harder in CI. Most modern CI systems rely on ephemeral runners. This means that out of the box, the Dagger cache will always be empty. The typical symptom is that a Dagger pipeline that was very fast on your local machine, feels slower in CI. The solution is to implement distributed caching. How to do this depends on your CI infrastructure, and is an area of improvement for Dagger. One option is the experimental distributed cache feature in Dagger Cloud. Another is to deploy CI runners and Dagger engine together on a shared Kubernetes cluster. For more discussion of distributed cache options, join our Discord, contact us directly, or read this discussion thread. If you're not sure where to start, you can also start your CI integration without distributed caching, and get back to it later. Everything will still work, just not as fast as it could be. Unlike traditional CI, you don't need to change your pipeline logic to add caching: once you turn it on, everything will just "magically" be faster.

The Demo​

Once your POC is successful, it's time for a victory lap. Make sure to show your team what you accomplished, why, and how. If you'd like help to make your presentation shine, don't hesitate to ask on Discord.

If you'd like to, and are allowed, you can also show a demo at our bi-weekly community calls. If there's one thing Daggernauts love, it's a demo of a real world use of Dagger!

Phase 3: expand​

Once your POC is complete, it's time to expand your adoption of Dagger! Depending on your situation, there are 2 strategies you can adopt: incremental, or all-in.

Incremental expansion​

This is the safest and most common strategy. Once you've successfully daggerized a small pipeline with your POC... Just repeat the process with another piece, then another one, and so on.

Depending on the constraints of your project, the genuine need for Dagger, and your team's priorities, this process could be spread out over months or even years; or it could spread like wildfire. We've seen both extremes, and will happily support you regardless of how fast you expand.

All-in expansion​

This strategy usually coincides with a major "replatform" event, typically a CI migration. Because of the enormous lock-in of traditional CI platforms, a migration is a painful project that involves rewriting thousands of lines of proprietary configuration. This process is best done on an aggressive tight schedule, to avoid being stuck "in between". This creates a window of opportunity for adopting Dagger: if you're going to rewrite everything, why move from one proprietary silo to another, when you could instead move to a general purpose language, targeting an open platform which can run on any CI?

In this case, your team will typically want to move quickly once the POC is successful, and adopt Dagger for all pipelines. We have experience with this process, and encourage you to contact us to discuss it: solutions@dagger.io.

Phase 4: spin out reusable modules​

As you develop your pipelines, you will find yourself writing more and more reusable functions, perhaps to integrate with a tool or service for which Daggerverse has no module, or perhaps because you're not (yet) allowed to use third-party modules for compliance reasons.

  • Initially all your functions might be mixed together in the same module, for speed.

  • Then, as your adoption of Dagger progresses, you can spin out the most reusable functions into sub-modules.

  • For larger engineering organizations, as Dagger spreads to multiple teams, you can centralize shared modules into a dedicated repository, which can be imported from each team's pipelines. Remember, Dagger supports cross-language linking, so you can mix and match modules written in different languages without worrying about interoperability.

  • Finally, some of these modules may be suitable to be open-sourced and shared with the entire Dagger community, by publishing it on the Daggerverse.