Introduction

Welcome to the Gofer documentation! This documentation is a reference for all available features and options of Gofer.

• To kick the tires visit: Getting Started
• To read more about Gofer's Feature set visit: Features
• To understand the why of Gofer visit: Philosophy

Gofer: Run short-lived jobs easily.

Gofer is an opinionated, streamlined automation engine designed for the cloud-native era. It specializes in executing your custom scripts in a containerized environment, making it versatile for both developers and operations teams. Deploy Gofer effortlessly as a single static binary, and manage it using expressive, declarative configurations written in real programming languages. Once set up, Gofer takes care of scheduling and running your automation tasks—be it on Nomad, Kubernetes, or even Local Docker.

Its primary function is to execute short-term jobs like code linting, build automation, testing, port scanning, ETL operations, or any task you can containerize and trigger based on events.

Features:

• Simple Deployment: Install Gofer effortlessly with a single static binary and manage it through its intuitive command-line interface.
• Language Flexibility: Craft your pipelines in programming languages you're already comfortable with, such as Go or Rust—no more wrestling with unfamiliar YAML.
• Local Testing: Validate and run your pipelines locally, eliminating the guesswork of "commit and see" testing.
• Extensible Architecture: Easily extend Gofer's capabilities by writing your own plugins, backends, and more, in any language via gRPC.
• Built-In Storage: Comes with an integrated Object and Secret store for your convenience.
• DAG Support: Harness the power of Directed Acyclic Graphs (DAGs) for complex workflow automation.
• Robust Reliability: Automatic versioning, Blue/Green deployments, and canary releases ensure the stability and dependability of your pipelines.

Demo:

Documentation & Getting Started

If you want to fully dive into Gofer, check out the documentation site!

Install

Extended installation information is available through the documentation site.

Download a specific release:

You can view and download releases by version here.

Download the latest release:

• Linux: wget https://github.com/clintjedwards/gofer/releases/latest/download/gofer

Build from source:

You'll need to install protoc and its associated golang/grpc modules first

1. git clone https://github.com/clintjedwards/gofer && cd gofer
2. make build OUTPUT=/tmp/gofer

The Gofer binary comes with a CLI to manage the server as well as act as a client.

Dev Setup

Gofer is setup such that the base run mode is the development mode. So simply running the binary without any additional flags allows easy authless development.

You'll need to install the following first:

To run Gofer dev mode:

• Docker

To build protocol buffers:

• protoc
• protoc gen plugins go/grpc

Run from the Makefile

Gofer uses flags, env vars, and files to manage configuration (in order of most important). The Makefile already includes all the commands and flags you need to run in dev mode by simply running make run.

In case you want to run without the make file simply run:

export GOFER_LOG_LEVEL=debug
go build -o /tmp/$gofer
/tmp/gofer service start --dev-mode

Editing Protobufs

Gofer uses grpc and protobufs to communicate with both plugins and provide an external API. These protobuf files are located in /proto. To compile new protobufs once the original .proto files have changed you can use the make build-protos command.

Editing Documentation

Documentation is done with mdbook.

To install:

cargo install mdbook
cargo install mdbook-linkcheck

Once you have mdbook you can simply run make run-docs to give you an auto-reloading dev version of the documentation in a browser.

Regenerating Demo Gif

The Gif on the README page uses vhs; a very handy tool that allows you to write a configuration file which will pop out a gif on the other side.

In order to do this VHS has to run the commands so we must start the server first before we regenerate the gif.

rm -rf /tmp/gofer* # Start with a fresh database
make run # Start the server in dev mode
cd documentation/src/assets
vhs < demo.tape # this will start running commands against the server and output the gif as demo.gif.

Authors

• Clint Edwards - Github

This software is provided as-is. It's a hobby project, done in my free time, and I don't get paid for doing it.

How does Gofer work?

Gofer works in a very simple client-server model. You deploy Gofer as a single binary to your favorite VPS and you can configure it to connect to all the tooling you currently use to run containers.

Gofer acts as a scheduling middle man between a user's intent to run a container at the behest of an event and your already established container orchestration system.

Workflow

Interaction with Gofer is mostly done through its command line interface which is included in the same binary as the master service.

General Workflow

1. Gofer is connected to a container orchestrator of some sort. This can be just your local docker service or something like K8s or Nomad.
2. It launches it's configured extensions (extensions are just docker containers) and these extensions wait for events to happen.
3. Users create pipelines (by configuration file) that define exactly in which order and what containers they would like to run.
4. These pipelines don't have to, but usually involve extensions so that pipelines can run automatically.
5. Either by extension or manual intervention a pipeline run will start and schedule the containers defined in the configuration file.
6. Gofer will collect the logs, exit code, and other essentials from each container run and provide them back to the user along with summaries of how that particular run performed.

Extension Implementation

1. When Gofer launches the first thing it does is create the extension containers the same way it schedules any other container.
2. The extension containers are all small GRPC services that are implemented using a specific interface provided by the SDK.
3. Gofer passes the extension a secret value that only it knows so that the extension doesn't respond to any requests that might come from other sources.
4. After the extension is initialized Gofer will subscribe any pipelines that have requested this extension (through their pipeline configuration file) to that extension.
5. The extension then takes note of this subscription and waits for the relevant event to happen.
6. When the event happens it figures out which pipeline should be alerted and sends an event to the main Gofer process.
7. The main gofer process then starts a pipeline run on behalf of the extension.

Glossary

• Pipeline: A pipeline is a collection of tasks that can be run at once. Pipelines can be defined via a pipeline configuration file. Once you have a pipeline config file you can create a new pipeline via the CLI (recommended) or API.

• Run: A run is a single execution of a pipeline. A run can be started automatically via extensions or manually via the API or CLI

• Extension: A extension allow for the extension of pipeline functionality. Extension start-up with Gofer as long running docker containers and pipelines can subscribe to them to have additional functionality.

• Task: A task is the lowest unit in Gofer. It is a small abstraction over running a single container. Through tasks you can define what container you want to run, when to run it in relation to other containers, and what variables/secrets those containers should use.

• Task Run: A task run is an execution of a single task container. Referencing a specific task run is how you can examine the results, logs, and details of one of your tasks.

FAQ

> I have a job that works with a remote git repository, other CI/CD tools make this trivial, how do I mimic that?

The drawback of this model and architecture is does not specifically cater to GitOps. So certain workflows that come out of the box from other CI/CD tooling will need to be recreated, due to its inherently distributed nature.

Gofer has provided several tooling options to help with this.

There are two problems that need to be solved around the managing of git repositories for a pipeline:

1) How do I authenticate to my source control repository?

Good security practice suggests that you should be managing repository deploy keys, per repository, per team. You can potentially forgo the "per team" suggestion using a "read-only" key and the scope of things using the key isn't too big.

Gofer's suggestion here is to make deploy keys self service and then simply enter them into Gofer's secret store to be used by your pipeline's tasks. Once there you can then use it in each job to pull the required repository.

2) How do I download the repository?

Three strategies:

1. Just download it when you need it. Depending on the size of your repository and the frequency of the pull, this can work absolutely fine.
2. Use the object store as a cache. Gofer provides an object store to act as a permanent (pipeline-level) or short-lived (run-level) cache for your workloads. Simply store the repository inside the object store and pull down per job as needed.
3. Download it as you need it using a local caching git server. Once your repository starts becoming large or you do many pulls quickly it might make more sense to use a cache¹,². It also makes sense to only download what you need using git tools like sparse checkout

Feature Guide

Write your pipelines in a real programming language.

Other infrastructure tooling tried configuration languages(yaml, hcl).... and they kinda suck¹. The Gofer CLI allows you to create your pipelines in a fully featured programming language. Pipelines can be currently be written in Go or Rust².

DAG(Directed Acyclic Graph) Support.

Gofer provides the ability to run your containers in reference to your other containers.

With DAG support you can run containers:

• In parallel.
• After other containers.
• When particular containers fail.
• When particular containers succeed.

GRPC API

Gofer uses GRPC and Protobuf to construct its API surface. This means that Gofer's API is easy to use, well defined, and can easily be developed for in any language.

The use of Protobuf gives us two main advantages:

1. The most up-to-date API contract can always be found by reading the .proto files included in the source.
2. Developing against the API for developers working within Golang/Rust simply means importing the autogenerate proto package.
3. Developing against the API for developers not working within the Go/Rust language means simply importing the proto files and generating them for the language you need.

You can find more information on protobuf, proto files, and how to autogenerate the code you need to use them to develop against Gofer in the protobuf documentation.

Namespaces

Gofer allows you to separate out your pipelines into different namespaces, allowing you to organize your teams and set permissions based on those namespaces.

Extensions

Extensions are the way users can add extra functionality to their pipelines. For instance the ability to automate their pipelines by waiting on bespoke events (like the passage of time).

Extensions are nothing more than docker containers themselves that talk to the main process when they require activity.

Gofer out of the box provides some default extensions like cron and interval. But even more powerful than that, it accepts any type of extension you can think up and code using the included SDK.

Extensions are brought up alongside Gofer as long-running docker containers that it launches and manages.

Object Store

Gofer provides a built in object store you can access with the Gofer CLI. This object store provides a caching and data transfer mechanism so you can pass values from one container to the next, but also store objects that you might need for all containers.

Secret Store

Gofer provides a built in secret store you can access with the Gofer CLI. This secret store provides a way to pass secret values needed by your pipeline configuration into Gofer.

Events

Gofer provides a list of events for the most common actions performed. You can view this event stream via the Gofer API, allowing you to build on top of Gofer's actions and even using Gofer as a trigger for other events.

External Events

Gofer allows extensions to consume external events. This allows for extensions to respond to webhooks from favorite sites like Github and more.

Pluggable Everything

Gofer plugs into all your favorite backends your team is already using. This means that you never have to maintain things outside of your wheelhouse.

Whether you want to schedule your containers on K8s or AWS Lambda, or maybe you'd like to use an object store that you're more familiar with in minio or AWS S3, Gofer provides either an already created plugin or an interface to write your own.

Best Practices

In order to schedule workloads on Gofer your code will need to be wrapped in a docker container. This is a short workflow blurb about how to create containers to best work with Gofer.

1) Write your code to be idempotent.

Write your code in whatever language you want, but it's a good idea to make it idempotent. Gofer does not guarantee single container runs (but even if it did that doesn't prevent mistakes from users).

2) Follow 12-factor best practices.

Configuration is the important one. Gofer manages information into containers by environment variables so your code will need to take any input or configuration it needs from environment variables.

3) Keep things simple.

You could, in theory, create a super complicated graph of containers that run off each other. But the main theme of Gofer is simplicity. Make sure you're thinking through the benefits of managing something in separate containers vs just running a monolith container. There are good reasons for both; always err on the side of clarity and ease of understanding.

4) Keep your containers lean.

Because of the potentially distributed nature of Gofer, the larger the containers you run, the greater potential lag time between the start of execution for your container. This is because there is no guarantee that your container will end up on a machine that already has the image. Downloading large images takes a lot of time and a lot of disk space.

Troubleshooting Gofer

This page provides various tips on how to troubleshoot and find issues/errors within Gofer.

Debugging extensions

Extensions are simply long running docker containers that internally wait for an event to happen and then communicate with Gofer it's API.

Debugging information coming soon.

Debugging Tasks

When tasks aren't working quite right, it helps to have some simple tasks that you can use to debug. Gofer provides a few of these to aid in debugging.

Gofer's Philosophy

Things should be simple, easy, and fast. For if they are not, people will look for an alternate solution.

Gofer focuses on the usage of common docker containers to run workloads that don't belong as long-running applications. The ability to run containers easily is powerful tool for users who need to run various short-term workloads and don't want to care about the idiosyncrasies of the tooling that they run on top of.

How do I use Gofer? What's a common workflow?

1. Create a docker container with the workload/code you want to run.
2. Create a configuration file (kept with your workload code) in which you tell Gofer what containers to run and when they should be run.
3. Gofer takes care of the rest!

What problem is Gofer attempting to solve?

The current landscape for running short-term jobs is heavily splintered and could do with some centralization and sanity.

1) Tooling in this space is often CI/CD focused and treats gitops as a core tenet.

Initially this is really good, Gitops is something most companies should embrace. But eventually as your workload grows you'll notice that you'll want/need to have a little more control over your short term workloads without setting up complicated release scheduling.

2) Tooling in this space can lack testability.

Ever set up a CI/CD pipeline for your team and end up with a string of commits simply testing or fixing bugs in your assumptions of the system? This is usually due to not understanding how the system works, what values it will produce, or testing being difficult.

These are issues because most CI/CD systems make it hard to test locally. In order to support a wide array of job types(and lean toward being fully gitops focused) most of them run custom agents which in turn run the jobs you want.

This can be bad, since it's usually non-trivial to understand exactly what these agents will do once they handle your workload. Dealing with these agents can also be an operational burden. Operators are generally unfamiliar with these custom agents and it doesn't play to the strengths of an ops team that is already focused on other complex systems.

Gofer leverages schedulers which work locally and are already native to your environment, so testing locally is never far away!

3) Tooling in this space can lack simplicity.

Some user experience issues I've run into using other common CI/CD tooling:

• 100 line bash script (filled with sed and awk) to configure the agent's environment before my workload was loaded onto it.
• Debugging docker in docker issues.
• Reading the metric shit ton of documentation just to get a project started, only to realize everything is proprietary.
• Trying to understand a groovy script nested so deep it got into one of the layers of hell.
• Dealing with the security issues of a way too permissive plugin system.
• Agents giving vague and indecipherable errors to why my job failed.

Gofer aims to use tooling that users are already are familiar with and get out of the way. Running containers should be easy. Forgoing things like custom agents and being opinionated in how workloads should be run, allows users to understand the system immediately and be productive quickly.

Familiar with the logging, metrics, and container orchestration of a system you already use? Great! Gofer will fit right in.

Why should you not use Gofer?

1) You need to simply run tests for your code.

While Gofer can do this, the gitops process really shines here. I'd recommend using any one of the widely available gitops focused tooling. Attempting to do this with Gofer will require you to recreate some of the things these tools give you for free, namely git repository management and automatic deployments.

2) The code you run is not idempotent.

Gofer does not guarantee a single run of a container. Even though it does a good job in best effort, a perfect storm of operator error, extension errors, or sudden shutdowns could cause multiple runs of the same container.

3) The code you run does not follow cloud native best practices.

The easiest primer on cloud native best practices is the 12-factor guide, specifically the configuration section. Gofer provides tooling for container to operate following these guidelines with the most important being that your code will need to take configuration via environment variables.

4) The scheduling you need is precise.

Gofer makes a best effort to start jobs on their defined timeline, but it is at the mercy of many parts of the system (scheduling lag, image download time, competition with other pipelines). If you need precise down to the second or minute runs of code Gofer does not guarantee such a thing.

Gofer works better when jobs are expected to run +1 to +5 mins of their scheduled event/time.

Why not use <insert favorite tool> instead ?

Getting Started

Let's start by setting up our first Gofer pipeline!

Installing Gofer

Gofer comes as an easy to distribute pre-compiled binary that you can run on your machine locally, but you can always build Gofer from source if need be.

Pre-compiled (Recommended)

You can download the latest version for linux here:

wget https://github.com/clintjedwards/gofer/releases/latest/download/gofer

From Source

Gofer contains protobuf assets which will not get compiled if used via go install. To solve for this we can use make to build ourselves an impromptu version.

git clone https://github.com/clintjedwards/gofer && cd gofer
make build OUTPUT=/tmp/gofer SEMVER=0.0.dev
/tmp/gofer --version

Running the Server Locally

Gofer is deployed as a single static binary allowing you to run the full service locally so you can play with the internals before committing resources to it. Spinning Gofer up locally is also a great way to debug "what would happen if?" questions that might come up during the creation of pipeline config files.

Install Gofer

Install Docker

The way in which Gofer runs containers is called a Scheduler. When deploying Gofer at scale we can deploy it with a more serious container scheduler (Nomad, Kubernetes) but for now we're just going to use the default local docker scheduler included. This simply uses your local instance of docker instance to run containers.

But before we use your local docker service... you have to have one in the first place. If you don't have docker installed, the installation is quick. Rather than covering the specifics here you can instead find a guide on how to install docker for your operating system on its documentation site.

Start the server

By default the Gofer binary is able to run the server in development mode. Simply start the service by:

gofer service start --dev-mode

🪧 The Gofer CLI has many useful commands, try running gofer -h to see a full listing.

Create Your First Pipeline Configuration

Before you can start running containers you must tell Gofer what you want to run. To do this we create what is called a pipeline configuration.

The creation of this pipeline configuration is very easy and can be done in either Golang or Rust. This allows you to use a fully-featured programming language to organize your pipelines, instead of dealing with YAML mess.

Let's Go!

As an example, let's just copy a pipeline that has been given to us already. We'll use Go as our language, which means you'll need to install it if you don't have it. The Gofer repository gives us a simple pipeline that we can copy and use.

Let's first create a folder where we'll put our pipeline:

mkdir /tmp/simple_pipeline

Then let's copy the Gofer provided pipeline's main file into the correct place:

cd /tmp/simple_pipeline
wget https://raw.githubusercontent.com/clintjedwards/gofer/main/examplePipelines/go/simple/main.go

This should create a main.go file inside our /tmp/simple_pipeline directory.

Lastly, let's initialize the new Golang program:

To complete our Go program we simply have to initialize it with the go mod command.

go mod init test/simple_pipeline
go mod tidy

The pipeline we generated above gives you a very simple pipeline with a few pre-prepared testing docker containers. You should be able to view it using your favorite IDE.

The configuration itself is very simple. Essentially a pipeline contains of a few parts:

> Some basic attributes so we know what to call it and how to document it.

err := sdk.NewPipeline("simple", "Simple Pipeline").
		Description("This pipeline shows off a very simple Gofer pipeline that simply pulls in " +
...

> The containers we want to run are defined through tasks.

...
sdk.NewTask("simple_task", "ubuntu:latest").
    Description("This task simply prints our hello-world message and exits!").
    Command("echo", "Hello from Gofer!").Variable("test", "sample"),
...

> And when we want to automate when the pipeline runs automatically, we can do that through extensions.

Register your pipeline

Now we will register your newly created pipeline configuration with Gofer!

More CLI to the rescue

From your terminal, lets use the Gofer binary to run the following command, pointing Gofer at your newly created pipeline folder:

gofer up ./tmp/simple_pipeline

Examine created pipeline

It's that easy!

The Gofer command line application uses your local Golang compiler to compile, parse, and upload your pipeline configuration to Gofer.

You should have received a success message and some suggested commands:

 ✓ Created pipeline: [simple] "Simple Pipeline"

  View details of your new pipeline: gofer pipeline get simple
  Start a new run: gofer run start simple

We can view the details of our new pipeline by running:

gofer pipeline get simple

If you ever forget your pipeline ID you can list all pipelines that you own by using:

gofer pipeline list

Start a Run

Now that we've set up Gofer, defined our pipeline, and registered it we're ready to actually run our containers.

Press start

gofer pipeline run simple

What happens now?

When you start a run Gofer will attempt to schedule all your tasks according to their dependencies onto your chosen scheduler. In this case that scheduler is your local instance of Docker.

Your run should be chugging along now!

View a list of runs for your pipeline:

gofer run list simple

View details about your run:

gofer run get simple 1

List the containers that executed during the run:

gofer taskrun list simple 1

View a particular container's details during the run:

gofer taskrun get simple 1 <task_id>

Stream a particular container's logs during the run:

gofer taskrun logs simple 1 <task_id>

What's Next?

Anything!

1. Keep playing with Gofer locally and check out all the CLI commands.
2. Spruce up your pipeline definition!
3. Learn more about Gofer terminology.
4. Deploy Gofer for real. Pair it with your favorite scheduler and start using it to automate your jobs.

Pipeline Configuration

A pipeline is a directed acyclic graph of tasks that run together. A single execution of a pipeline is called a run. Gofer allows users to configure their pipeline via a configuration file written in Golang or Rust.

The general hierarchy for a pipeline is:

pipeline
    \_ run
         \_ task

Each execution of a pipeline is a run and every run consists of one or more tasks (containers). These tasks are where users specify their containers and settings.

SDK

Creating a pipeline involves using Gofer's SDK currently written in Go or Rust.

Extensive documentation can be found on the SDK's reference page. There you will find most of the features and idiosyncrasies available to you when creating a pipeline.

Small Walkthrough

To introduce some of the concepts slowly, lets build a pipeline step by step. We'll be using Go as our pipeline configuration language and this documentation assumes you've already set up a new Go project and are operating in a main.go file. If you haven't you can set up one following the guide instructions.

A Simple Pipeline

Every pipeline is initialized with a simple pipeline declaration. It's here that we will name our pipeline, giving it a machine referable ID and a human referable name.

err := sdk.NewPipeline("simple", "My Simple Pipeline")

It's important to note here that while your human readable name ("My Simple Pipeline" in this case) can contain a large array of characters the ID can only container alphanumeric letters, numbers, and underscores. Any other characters will result in an error when attempting to register the pipeline.

Add a Description

Next we'll add a simple description to remind us what this pipeline is used for.

err := sdk.NewPipeline("simple", "My Simple Pipeline").
        Description("This pipeline is purely for testing purposes.")

The SDK uses a builder pattern, which allows us to simply add another function onto our Pipeline object which we can type our description into.

Add a task

Lastly let's add a task(container) to our pipeline. We'll add a simple ubuntu container and change the command that gets run on container start to just say "Hello from Gofer!".

err := sdk.NewPipeline("simple", "My Simple Pipeline").
        Description("This pipeline is purely for testing purposes.").
        Tasks(sdk.NewTask("simple_task", "ubuntu:latest").
			Description("This task simply prints our hello-world message and exists!").
			Command("echo", "Hello from Gofer!"),
    )

We used the Tasks function to add multiple tasks and then we use the SDK's NewTask function to create a task. You can see we:

• Give the task an ID, much like our pipeline earlier.
• Specify which image we want to use.
• Tack on a description.
• And then finally specify the command.

To tie a bow on it, we add the .Finish() function to specify that our pipeline is in it's final form.

err := sdk.NewPipeline("my_pipeline", "My Simple Pipeline").
    Description("This pipeline is purely for testing purposes.").
    Tasks(sdk.NewTask("simple_task", "ubuntu:latest").
			Description("This task simply prints our hello-world message and exists!").
			Command("echo", "Hello from Gofer!"),
    ).Finish()

That's it! This is a fully functioning pipeline.

You can run and test this pipeline much like you would any other code you write. Running it will produce a protobuf binary output which Gofer uses to pass to the server.

Full Example

package main

import (
	"log"

	sdk "github.com/clintjedwards/gofer/sdk/go/config"
)

func main() {
	err := sdk.NewPipeline("simple", "Simple Pipeline").
		Description("This pipeline shows off a very simple Gofer pipeline that simply pulls in " +
			"a container and runs a command. Veterans of CI/CD tooling should be familiar with this pattern.\n\n" +

			"Shown below, tasks are the building blocks of a pipeline. They represent individual containers " +
			"and can be configured to depend on one or multiple other tasks.\n\n" +

			"In the task here, we simply call the very familiar Ubuntu container and run some commands of our own.\n\n" +

			"While this is the simplest example of Gofer, the vision is to move away from writing our logic code " +
			"in long bash scripts within these task definitions.\n\n" +

			"Ideally, these tasks are custom containers built with the purpose of being run within Gofer for a " +
			"particular workflow. Allowing you to keep the logic code closer to the actual object that uses it " +
			"and keeping the Gofer pipeline configurations from becoming a mess.\n").
		Tasks(
			sdk.NewTask("simple_task", "ubuntu:latest").
				Description("This task simply prints our hello-world message and exits!").
				Command("echo", "Hello from Gofer!").Variable("test", "sample"),
		).Finish()
	if err != nil {
		log.Fatal(err)
	}
}

Extra Examples

Auto Inject API Tokens

Gofer has the ability to auto-create and inject a token into your tasks. This is helpful if you want to use the Gofer CLI or the Gofer API to communicate with Gofer at some point in your task.

You can tell Gofer to do this by using the InjectAPIToken function for a particular task.

The token will be cleaned up the same time the logs for a particular run is cleaned up.

err := sdk.NewPipeline("my_pipeline", "My Simple Pipeline").
    Description("This pipeline is purely for testing purposes.").
    Tasks(
		sdk.NewTask("simple_task", "ubuntu:latest").
			Description("This task simply prints our hello-world message and exists!").
			Command("echo", "Hello from Gofer!").InjectAPIToken(true),
    ).Finish()

Tasks

Gofer's abstraction for running a container is called a Task. Specifically Tasks are containers you point Gofer to and configure to perform some workload.

A Task can be any Docker container you want to run. In the Getting Started example we take a regular standard ubuntu:latest container and customize it to run a passed in bash script.

Tasks(
    sdk.NewTask("simple_task", "ubuntu:latest").
        Description("This task simply prints our hello-world message and exists!").
        Command("echo", "Hello from Gofer!"),
)

Task Environment Variables and Configuration

Gofer handles container configuration the cloud native way. That is to say every configuration is passed in as an environment variable. This allows for many advantages, the greatest of which is standardization.

As a user, you pass your configuration in via the Variable(s) flavor of functions in your pipeline-config.

When a container is run by Gofer, the Gofer scheduler has the potential to pass in configuration from multiple sources¹:

1. Your pipeline configuration: Configs you pass in by using the Variable(s) functions.

2. Extension/Manual configurations: Extensions are allowed to pass in custom configuration for a run. Usually this configuration gives extra information the run might need. (For example, the git commit that activated the extension.).

   Alternatively, if this run was not activated by a extension and instead kicked of manually, the user who launched the run might opt to pass in configuration at that runtime.

3. Gofer's system configurations: Gofer will pass in system configurations that might be helpful to the user. (For example, what current pipeline is running.)²

The exact key names injected for each of these configurations can be seen on any taskrun by getting that taskrun's details: gofer taskrun get <pipeline_name> <run_id>

What happens when a task is run?

The high level flow is:

1. Gofer checks to make sure your task configuration is valid.
2. Gofer parses the task configuration's variables list. It attempts replace any substitution variables with their actual values from the object or secret store.
3. Gofer then passes the details of your task to the configured scheduler, variables are passed in as environment variables.
4. Usually this means the scheduler will take the configuration and attempt to pull the image mentioned in the configuration.
5. Once the image is successfully pulled the container is then run with the settings passed.

Server Configuration

Gofer runs as a single static binary that you deploy onto your favorite VPS.

While Gofer will happily run in development mode without any additional configuration, this mode is NOT recommended for production workloads and not intended to be secure.

Instead Gofer allows you to edit it's startup configuration allowing you to configure it to run on your favorite container orchestrator, object store, and/or storage backend.

Setup

There are a few steps to setting up the Gofer service for production:

1) Configuration

First you will need to properly configure the Gofer service.

Gofer accepts configuration through environment variables or a configuration file. If a configuration key is set both in an environment variable and in a configuration file, the value of the environment variable's value will be the final value.

You can view a list of environment variables Gofer takes by using the gofer service start -h command. It's important to note that each environment variable starts with a prefix of GOFER_. So setting the host configuration can be set as:

export GOFER_SERVER__HOST=localhost:8080

Configuration file

The Gofer service configuration file is written in HCL.

Load order

The Gofer service looks for its configuration in one of several places (ordered by first searched):

1. Path given through the GOFER_CONFIG_PATH environment variable
2. /etc/gofer/gofer.hcl

🪧 You can generate a sample Gofer configuration file by using the command: gofer service init-config

Bare minimum production file

These are the bare minimum values you should populate for a production ready Gofer configuration.

The values below should be changed depending on your environment; leaving them as they currently are will lead to loss of data on server restarts.

🪧 To keep your deployment of Gofer safe make sure to use your own TLS certificates instead of the default localhost ones included.

// Gofer Service configuration file is used as an alternative to providing the server configurations via envvars.
// You can find an explanation of these configuration variables and where to put this file so the server can read this
// file in the documenation: https://clintjedwards.com/gofer/ref/server_configuration/index.html
ignore_pipeline_run_events = false
run_parallelism_limit      = 200
pipeline_version_limit     = 5
event_log_retention        = "4380h"
event_prune_interval       = "3h"
log_level                  = "info"
task_run_log_expiry        = 50
task_run_logs_dir          = "/tmp"
task_run_stop_timeout      = "5m"

external_events_api {
  enable = true
  host   = "localhost:8081"
}

object_store {
  engine = "sqlite"
  sqlite {
    path = "/tmp/gofer-object.db"
  }
  pipeline_object_limit = 50
  run_object_expiry     = 50
}

secret_store {
  engine = "sqlite"
  sqlite {
    path           = "/tmp/gofer-secret.db"
    encryption_key = "changemechangemechangemechangeme"
  }
}

scheduler {
  engine = "docker"
  docker {
    prune          = true
    prune_interval = "24h"
  }
}

server {
  host                  = "localhost:8080"
  shutdown_timeout      = "15s"
  tls_cert_path         = "./localhost.crt"
  tls_key_path          = "./localhost.key"
  storage_path          = "/tmp/gofer.db"
  storage_results_limit = 200
}

extensions {
  install_base_extensions = true
  stop_timeout          = "5m"
  tls_cert_path         = "./localhost.crt"
  tls_key_path          = "./localhost.key"
}

2) Running the binary

You can find the most recent releases of Gofer on the github releases page..

Simply use whatever configuration management system you're most familiar with to place the binary on your chosen VPS and manage it. You can find a quick and dirty wget command to pull the latest version in the getting started documentation.

As an example a simple systemd service file setup to run Gofer is show below:

Example systemd service file

[Unit]
Description=gofer service
Requires=network-online.target
After=network-online.target

[Service]
Restart=on-failure
ExecStart=/usr/bin/gofer service start
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

3) First steps

You will notice upon service start that the Gofer CLI is unable to make any requests due to permissions.

You will first need to handle the problem of auth. Every request to Gofer must use an API key so Gofer can appropriately direct requests.

More information about auth in general terms can be found here.

To create your root management token use the command: gofer service token bootstrap

🪧 The token returned is a management token and as such as access to all routes within Gofer. It is advised that:

1. You use this token only in admin situations and to generate other lesser permissioned tokens.
2. Store this token somewhere safe

From here you can use your root token to provision extra lower permissioned tokens for everyday use.

When communicating with Gofer through the CLI you can set the token to be automatically passed per request in one of many ways.

Configuration Reference

Gofer has a variety of parameters that can be specified via environment variables or the configuration file.

To view a list of all possible environment variables simply type: gofer service start -h.

The most up to date config file values can be found by reading the code or running the command above, but a best effort key and description list is given below.

If examples of these values are needed you can find a sample file by using gofer service init-config.

Values

General

Development (block)

Special feature flags to make development easier

Example

development {
  bypass_auth = true
}
```                                                                                                   |

### External Events API (block)

The external events API controls webhook type interactions with extensions. HTTP requests go through the events endpoint and Gofer routes them to the proper extension for handling.

| name   | type    | default        | description                                                                               |
| ------ | ------- | -------------- | ----------------------------------------------------------------------------------------- |
| enable | boolean | true           | Enable the events api. If this is turned off the events http service will not be started. |
| host   | string  | localhost:8081 | The address and port to bind the events service to.                                       |

#### Example

```hcl
external_events_api {
  enable = true
  host   = "0.0.0.0:8081"
}

Object Store (block)

The settings for the Gofer object store. The object store assists Gofer with storing values between tasks since Gofer is by nature distributed. This helps jobs avoid having to download the same objects over and over or simply just allows tasks to share certain values.

You can find more information on the object store block here.

Sqlite (block)

The sqlite store is a built-in, easy to use object store. It is meant for development and small deployments.

object_store {
  engine = "sqlite"
  sqlite {
    path = "/tmp/gofer-object.db"
  }
}

Secret Store (block)

The settings for the Gofer secret store. The secret store allows users to securely populate their pipeline configuration with secrets that are used by their tasks, extension configuration, or scheduler.

You can find more information on the secret store block here.

Sqlite (block)

The sqlite store is a built-in, easy to use object store. It is meant for development and small deployments.

secret_store {
  engine = "sqlite"
  sqlite {
    path = "/tmp/gofer-secret.db"
    encryption_key = "changemechangemechangemechangeme"
  }
}

Scheduler (block)

The settings for the container orchestrator that Gofer will use to schedule workloads.

You can find more information on the scheduler block here.

Docker (block)

Docker is the default container orchestrator and leverages the machine's local docker engine to schedule containers.

scheduler {
  engine = "docker"
  docker {
    prune          = true
    prune_interval = "24h"
  }
}

Server (block)

Controls the settings for the Gofer service's server properties.

server {
  host                  = "localhost:8080"
  tls_cert_path         = "./localhost.crt"
  tls_key_path          = "./localhost.key"
  tmp_dir               = "/tmp"
  storage_path          = "/tmp/gofer.db"
  storage_results_limit = 200
}

Extensions (block)

Controls settings for Gofer's extension system. Extensions are different workflows for running pipelines usually based on some other event (like the passing of time).

You can find more information on the extension block here.

extensions {
  install_base_extensions = true
  stop_timeout          = "5m"
  tls_cert_path         = "./localhost.crt"
  tls_key_path          = "./localhost.key"
}

Authentication

Gofer's auth system is meant to be extremely lightweight and a stand-in for a more complex auth system.

How auth works

Gofer uses API Tokens for authorization. You pass a given token in whenever talking to the API and Gofer will evaluate internally what type of token you possess and for which namespaces does it possess access.

Management Tokens

The first type of token is a management token. Management tokens essentially act as root tokens and have access to all routes.

It is important to be extremely careful about where your management tokens end up and how they are used.

Other than system administration, the main use of management tokens are the creation of new tokens. You can explore token creation though the CLI.

It is advised that you use a single management token as the root token by which you create all user tokens.

Client Tokens

The most common token type is a client token. The client token simply controls which namespaces a user might have access to.

During token creation you can choose one or multiple namespaces for the token to have access to.

How to auth via the API

The Gofer API uses GRPC's metadata functionality to read tokens from requests:

md := metadata.Pairs("Authorization", "Bearer "+<token>)

How to auth via the CLI

The Gofer CLI accepts many ways of setting a token once you have one.

External Events

Gofer has an alternate endpoint specifically for external events streams¹. This endpoint takes in http requests from the outside and passes them to the relevant extension.

You can find more about external event configuration in the configuration-values reference.

 external_events_api {
   enable = true
   host   = "0.0.0.0:8081"
 }

It works like this:

1. When the Gofer service is started it starts the external events service on a separate port per the service configuration settings. It is also possible to just turn off this feature via the same configuration file.

2. External services can send Gofer http requests with payloads and headers specific to the extension they're trying to communicate with. It's possible to target specific extensions by using the /events endpoint.

   ex: https://mygofer.mydomain.com/events/github <- #extension label

3. Gofer serializes and forwards the request to the relevant extension where it is validated for authenticity of sender and then processed.

4. A extension may then handle this external event in any way it pleases. For example, the Github extension takes in external events which are expected to be Github webhooks and starts a pipeline if the event type matches one the user wanted.

Scheduler

Gofer runs the containers you reference in the pipeline configuration via a container orchestrator referred to here as a "scheduler".

The vision of Gofer is for you to use whatever scheduler your team is most familiar with.

Supported Schedulers

The only currently supported scheduler is local docker. This scheduler is used for small deployments and development work.

How to add new Schedulers?

Schedulers are pluggable! Simply implement a new scheduler by following the given interface.

type GetStateResponse struct {
	ExitCode int64
	State    ContainerState
}

type GetLogsRequest struct {
	ID string
}

type AttachContainerRequest struct {
	ID      string
	Command []string
}

type AttachContainerResponse struct {
	Conn   net.Conn
	Reader io.Reader
}

type Engine interface {
	// StartContainer launches a new container on scheduler.
	StartContainer(request StartContainerRequest) (response StartContainerResponse, err error)

	// StopContainer attempts to stop a specific container identified by a unique container name. The scheduler
	// should attempt to gracefully stop the container, unless the timeout is reached.
	StopContainer(request StopContainerRequest) error

	// GetState returns the current state of the container translated to the "models.ContainerState" enum.
	GetState(request GetStateRequest) (response GetStateResponse, err error)

	// GetLogs reads logs from the container and passes it back to the caller via an io.Reader. This io.reader can
	// be written to from a goroutine so that they user gets logs as they are streamed from the container.
	// Finally once finished the io.reader should be close with an EOF denoting that there are no more logs to be read.
	GetLogs(request GetLogsRequest) (logs io.Reader, err error)

	// Attach to a running container for debugging or other purposes. Returns a net connection, should be closed when finished.
	AttachContainer(request AttachContainerRequest) (response AttachContainerResponse, err error)
}

Docker scheduler

The docker scheduler uses the machine's local docker engine to run containers. This is great for small or development workloads and very simple to implement. Simply download docker and go!

scheduler {
  engine = "docker"
  docker {
    prune          = true
    prune_interval = "24h"
  }
}

Configuration

Docker needs to be installed and the Gofer process needs to have the required permissions to run containers upon it.

Other than that the docker scheduler just needs to know how to clean up after itself.

Object Store

Gofer provides an object store as a way to share values and objects between containers. It can also be used as a cache. It is common for one container to run, generate an artifact or values, and then store that object in the object store for the next container or next run. The object store can be accessed through the Gofer CLI or through the normal Gofer API.

Gofer divides the objects stored into two different lifetime groups:

Pipeline-level objects

Gofer can store objects permanently for each pipeline. You can store objects at the pipeline-level by using the gofer pipeline object store command:

gofer pipeline store put my-pipeline my_key1=my_value5
gofer pipeline store get my-pipeline my_key1
#output: my_value5

The limitation to pipeline level objects is that they have a limit of the number of objects that can be stored per-pipeline. Once that limit is reached the oldest object in the store will be removed for the newest object.

Run-level objects

Gofer can also store objects on a per-run basis. Unlike the pipeline-level objects run-level do not have a limit to how many can be stored, but instead have a limit of how long they last. Typically after a certain number of runs a object stored at the run level will expire and that object will be deleted.

You can access the run-level store using the run level store CLI commands. Here is an example:

gofer run store put simple_pipeline my_key=my_value
gofer run store get simple_pipeline my_key
#output: my_value

Supported Object Stores

The only currently supported object store is the sqlite object store. Reference the configuration reference for a full list of configuration settings and options.

How to add new Object Stores?

Object stores are pluggable! Simply implement a new object store by following the given interface.

type Engine interface {
	GetObject(key string) ([]byte, error)
	PutObject(key string, content []byte, force bool) error
	ListObjectKeys(prefix string) ([]string, error)
	DeleteObject(key string) error
}

Sqlite object store

The sqlite object store is great for development and small deployments.

object_store {
  engine = "sqlite"
  sqlite {
    path = "/tmp/gofer-object.db"
  }
}

Configuration

Sqlite needs to create a file on the local machine making the only parameter it accepts a path to the database file.

Secret Store

Gofer provides a secret store as a way to enable users to pass secrets into pipeline configuration files.

The secrets included in the pipeline file use a special syntax so that Gofer understands when it is given a secret value instead of a normal variable.

...
env_vars = {
  "SOME_SECRET_VAR" = "secret{{my_key_here}}"
}
...

Supported Secret Stores

The only currently supported secret store is the sqlite object store. Reference the configuration reference for a full list of configuration settings and options.

How to add new Secret Stores?

Secret stores are pluggable! Simply implement a new secret store by following the given interface.

type Engine interface {
	GetSecret(key string) (string, error)
	PutSecret(key string, content string, force bool) error
	ListSecretKeys(prefix string) ([]string, error)
	DeleteSecret(key string) error
}

Sqlite secret store

The sqlite object store is great for development and small deployments.

secret_store {
  engine = "sqlite"
  sqlite {
    path = "/tmp/gofer-secret.db"
    encryption_key = "changemechangemechangemechangeme"
  }
}

Configuration

Sqlite needs to create a file on the local machine making the only parameter it accepts a path to the database file.

Extensions

Extensions are Gofer's way of adding additional functionality to pipelines. You can subscribe your pipeline to an extension, allowing that extension to give your pipeline extra powers.

The most straight-forward example of this, is the interval extension. This extension allows your pipeline to run everytime some amount of time has passed. Let's say you have a pipeline that needs to run every 5 mins. You would subscribe your pipeline to the interval extension using the gofer cli command gofer extension sub internal every_5_seconds set to an interval of 5m.

On startup, Gofer launches the interval extension as a long-running container. When your pipeline subscribes to it. The interval extension starts a timer and when 5 minutes have passed the extension sends an API request to Gofer, causing Gofer to run your pipeline.

Gofer Provided Extensions

You can create your own extensions, but Gofer provides some provided extensions for use.

How do I install a Extension?

Extensions must first be installed by Gofer administrators before they can be used. They can be installed by the CLI. For more information on how to install a specific extension run:

gofer extension install -h

How do I configure a Extension?

Extensions allow for both system and pipeline configuration¹. Meaning they have both Global settings that apply to all pipelines and Pipeline specific settings. This is what makes them so dynamically useful!

Pipeline Configuration

Most Extensions allow for some pipeline specific configuration usually referred to as "Parameters" or "Pipeline configuration".

These variables are passed when the user subscribes their pipeline to the extension. Each extension defines what this might be in it's documentation.

System Configuration

Most extensions have system configurations which allow the administrator or system to inject some needed variables. These are defined when the Extension is installed.

How to add new Extensions/ How do I create my own?

Just like tasks, extensions are simply docker containers! Making them easily testable and portable. To create a new extension you simply use the included Gofer SDK.

The SDK provides an interface in which a well functioning GRPC service will be created from your concrete implementation.

// ExtensionServiceInterface provides a light wrapper around the GRPC extension interface. This light wrapper
// provides the caller with a clear interface to implement and allows this package to bake in common
// functionality among all extensions.
type ExtensionServiceInterface interface {
	// Init tells the extension it should complete it's initialization phase and return when it is ready to serve requests.
	// This is useful because sometimes we'll want to start the extension, but not actually have it do anything
	// but serve only certain routes like the installation routes.
	Init(context.Context, *proto.ExtensionInitRequest) (*proto.ExtensionInitResponse, error)

	// Info returns information on the specific plugin
	Info(context.Context, *proto.ExtensionInfoRequest) (*proto.ExtensionInfoResponse, error)

	// Subscribe registers a pipeline with said extension to provide the extension's functionality.
	Subscribe(context.Context, *proto.ExtensionSubscribeRequest) (*proto.ExtensionSubscribeResponse, error)

	// Unsubscribe allows pipelines to remove their extension subscriptions.
	Unsubscribe(context.Context, *proto.ExtensionUnsubscribeRequest) (*proto.ExtensionUnsubscribeResponse, error)

	// Shutdown tells the extension to cleanup and gracefully shutdown. If a extension
	// does not shutdown in a time defined by the Gofer API the extension will
	// instead be Force shutdown(SIGKILL). This is to say that all extensions should
	// lean toward quick cleanups and shutdowns.
	Shutdown(context.Context, *proto.ExtensionShutdownRequest) (*proto.ExtensionShutdownResponse, error)

	// ExternalEvent are json blobs of Gofer's /events endpoint. Normally webhooks.
	ExternalEvent(context.Context, *proto.ExtensionExternalEventRequest) (*proto.ExtensionExternalEventResponse, error)

	// Run the installer that helps admin user install the extension.
	RunExtensionInstaller(stream proto.ExtensionService_RunExtensionInstallerServer) error

	// Run the installer that helps pipeline users with their pipeline extension

For an commented example of a simple extension you can follow to build your own, view the interval extension:

// Extension interval simply runs the subscribed pipeline at the given interval.
//
// This package is commented in such a way to make it easy to deduce what is going on, making it
// a perfect example of how to build other extensions.
//
// What is going on below is relatively simple:
//   - All extensions are run as long-running containers.
//   - We create our extension as just a regular program, paying attention to what we want our variables to be
//     when we install the extension and when a pipeline subscribes to this extension.
//   - We assimilate the program to become a long running extension by using the Gofer SDK and implementing
//     the needed sdk.ExtensionServiceInterface.
//   - We simply call NewExtension and let the SDK and Gofer go to work.
package main

import (
	"context"
	"fmt"
	"strings"
	"time"

	// The proto package provides some data structures that we'll need to return to our interface.
	proto "github.com/clintjedwards/gofer/proto/go"

	// The sdk package contains a bunch of convenience functions that we use to build our extension.
	// It is possible to build a extension without using the SDK, but the SDK makes the process much
	// less cumbersome.
	sdk "github.com/clintjedwards/gofer/sdk/go/extensions"

	// Golang doesn't have a standardized logging interface and as such Gofer extensions can technically
	// use any logging package, but because Gofer and provided extensions use zerolog, it is heavily encouraged
	// to use zerolog. The log level for extensions is set by Gofer on extension start via Gofer's configuration.
	// And logs are interleaved in the stdout for the main program.
	"github.com/rs/zerolog/log"
)

// Extensions have two types of variables they can be passed.
//   - They take variables called "config" when they are installed.
//   - They take variables called "parameters" for each pipeline that subscribes to them.

// This extension has a single parameter called "every".
const (
	// "every" is the time between pipeline runs.
	// Supports golang native duration strings: https://pkg.go.dev/time#ParseDuration
	//
	// Examples: "1m", "60s", "3h", "3m30s"
	ParameterEvery = "every"
)

// And a single config called "min_duration".
const (
	// The minimum interval pipelines can set for the "every" parameter.
	ConfigMinInterval = "min_interval"
)

// Extensions are subscribed to by pipelines. Gofer will call the `subscribe` function for the extension and
// pass it details about the pipeline and the parameters it wants.
// This structure is meant to keep details about those subscriptions so that we may
// perform the extension's duties on those pipeline subscriptions.
type subscription struct {
	namespace              string
	pipeline               string
	pipelineExtensionLabel string
	quit                   context.CancelFunc
}

// SubscriptionID is simply a composite key of the many things that make a single subscription unique.
// We use this as the key in a hash table to lookup subscriptions. Some might wonder why label is part
// of this unique key. That is because extensions should expect that pipelines might
// want to subscribe more than once.
type subscriptionID struct {
	namespace              string
	pipeline               string
	pipelineExtensionLabel string
}

// Extension is a structure that every Gofer extension should have. It is essentially a God struct that coordinates things
// for the extension as a whole. It contains all information about our extension that we might want to reference.
type extension struct {
	// Extensions can be run without "initializing" them. This allows Gofer to run things like the installer without
	// having to pass the extension everything it needs to work for normal cases.
	// It might be useful to track whether the extension was initialized or not.
	isInitialized bool

	// The lower limit for how often a pipeline can request to be run.
	minInterval time.Duration

	// During shutdown the extension will want to stop all intervals immediately. Having the ability to stop all goroutines
	// is very useful.
	quitAllSubscriptions context.CancelFunc

	// The parent context is stored here so that we have a common parent for all goroutines we spin up.
	// This enables us to manipulate all goroutines at the same time.
	parentContext context.Context

	// Mapping of subscription id to actual subscription. The subscription in this case also contains the goroutine
	// cancel context for the specified extension. This is important, as when a pipeline unsubscribes from a this extension
	// we will need a way to stop that specific goroutine from running.
	subscriptions map[subscriptionID]*subscription

	// Generic extension configuration set by Gofer at startup. Useful for interacting with Gofer.
	systemConfig sdk.ExtensionSystemConfig
}

// Init serves to set up the extension for it's main functionality. It is needed mostly because we sometimes need
// extensions to launch, but not actually serve all requests(like when we're running the extensions install endpoints).
//
// The Gofer server when launching an extension will call the Init endpoint and then wait until
// a successful response is returned to mark the extension ready to take subscriptions.
func (e *extension) Init(ctx context.Context, request *proto.ExtensionInitRequest) (*proto.ExtensionInitResponse, error) {
	minDurationStr := request.Config[ConfigMinInterval]
	minDuration := time.Minute * 1
	if minDurationStr != "" {
		parsedDuration, err := time.ParseDuration(minDurationStr)
		if err != nil {
			return nil, err
		}
		minDuration = parsedDuration
	}

	e.parentContext, e.quitAllSubscriptions = context.WithCancel(context.Background())
	e.minInterval = minDuration
	e.subscriptions = map[subscriptionID]*subscription{}

	config, _ := sdk.GetExtensionSystemConfig()
	e.systemConfig = config
	e.isInitialized = true

	return &proto.ExtensionInitResponse{}, nil
}

// startInterval is the main logic of what enables the interval extension to work. Each pipeline that is subscribed runs
// this function which simply waits for the set duration and then calls the StartRun endpoint for Gofer.
func (e *extension) startInterval(ctx context.Context, namespace, pipeline string, pipelineExtensionLabel string, duration time.Duration,
) {
	for {
		select {
		case <-ctx.Done():
			return
		case <-time.After(duration):
			client, ctx, err := sdk.Connect()
			if err != nil {
				log.Error().Err(err).Str("namespaceID", namespace).Str("pipelineID", pipeline).
					Str("extension_label", pipelineExtensionLabel).Msg("could not connect to Gofer")

				continue
			}

			config, _ := sdk.GetExtensionSystemConfig()

			resp, err := client.StartRun(ctx, &proto.StartRunRequest{
				NamespaceId: namespace,
				PipelineId:  pipeline,
				Variables:   map[string]string{},
				Initiator: &proto.Initiator{
					Type:   proto.Initiator_EXTENSION,
					Name:   fmt.Sprintf("%s (%s)", config.Name, pipelineExtensionLabel),
					Reason: "Triggered due to the passage of time",
				},
			})
			if err != nil {
				log.Error().Err(err).Str("namespaceID", namespace).Str("pipelineID", pipeline).
					Str("extension_label", pipelineExtensionLabel).Msg("could not start new run")

				continue
			}

			log.Debug().Str("namespace_id", namespace).Str("pipeline_id", pipeline).
				Str("extension_label", pipelineExtensionLabel).Int64("run_id", resp.Run.Id).
				Msg("new tick for specified interval; new event spawned")
		}
	}
}

// Gofer calls subscribe when a pipeline wants to subscribe to this extension.
// The logic here is simple:
//   - Retrieve the pipeline's requested parameters.
//   - Validate the parameters.
//   - Create a new subscription object and enter it into our map.
//   - Call the `startInterval` function in a goroutine for that specific pipeline and return.
func (e *extension) Subscribe(ctx context.Context, request *proto.ExtensionSubscribeRequest) (*proto.ExtensionSubscribeResponse, error) {
	interval, exists := request.Config[strings.ToUpper(ParameterEvery)]
	if !exists {
		return nil, fmt.Errorf("could not find required configuration parameter %q; received config params: %+v", ParameterEvery, request.Config)
	}

	duration, err := time.ParseDuration(interval)
	if err != nil {
		return nil, fmt.Errorf("could not parse interval string: %w", err)
	}

	if duration < e.minInterval {
		return nil, fmt.Errorf("durations cannot be less than %s", e.minInterval)
	}

	subID := subscriptionID{
		request.NamespaceId,
		request.PipelineId,
		request.PipelineExtensionLabel,
	}

	// It is perfectly possible for Gofer to attempt to subscribe an already subscribed pipeline. In this case,
	// we can simply ignore the request.
	_, exists = e.subscriptions[subID]
	if exists {
		log.Debug().Str("namespace_id", request.NamespaceId).Str("extension_label", request.PipelineExtensionLabel).
			Str("pipeline_id", request.PipelineId).Msg("pipeline already subscribed; ignoring request")
		return &proto.ExtensionSubscribeResponse{}, nil
	}

	subctx, quit := context.WithCancel(e.parentContext)
	e.subscriptions[subID] = &subscription{
		namespace:              request.NamespaceId,
		pipeline:               request.PipelineId,
		pipelineExtensionLabel: request.PipelineExtensionLabel,
		quit:                   quit,
	}

	go e.startInterval(subctx, request.NamespaceId, request.PipelineId, request.PipelineExtensionLabel, duration)

	log.Debug().Str("namespace_id", request.NamespaceId).Str("extension_label", request.PipelineExtensionLabel).
		Str("pipeline_id", request.PipelineId).Msg("subscribed pipeline")
	return &proto.ExtensionSubscribeResponse{}, nil
}

// Pipelines change and this means that sometimes they will no longer want to be executed by a particular extension or maybe
// they want to change the previous settings on that extension. Because of this we need a way to remove pipelines that were
// previously subscribed.
func (e *extension) Unsubscribe(ctx context.Context, request *proto.ExtensionUnsubscribeRequest) (*proto.ExtensionUnsubscribeResponse, error) {
	subscription, exists := e.subscriptions[subscriptionID{
		namespace:              request.NamespaceId,
		pipeline:               request.PipelineId,
		pipelineExtensionLabel: request.PipelineExtensionLabel,
	}]

	// It is perfectly possible for Gofer to attempt to unsubscribe an already unsubscribed pipeline. In this case,
	// we can simply ignore the request.
	if !exists {
		log.Debug().Str("namespace_id", request.NamespaceId).Str("extension_label", request.PipelineExtensionLabel).
			Str("pipeline_id", request.PipelineId).Msg("no subscription found for pipeline")
		return &proto.ExtensionUnsubscribeResponse{}, nil
	}

	subscription.quit()
	delete(e.subscriptions, subscriptionID{
		namespace:              request.NamespaceId,
		pipeline:               request.PipelineId,
		pipelineExtensionLabel: request.PipelineExtensionLabel,
	})
	return &proto.ExtensionUnsubscribeResponse{}, nil
}

// Info is mostly used as a health check endpoint. It returns some basic info about a extension, the most important
// being where to get more documentation about that specific extension.
func (e *extension) Info(ctx context.Context, request *proto.ExtensionInfoRequest) (*proto.ExtensionInfoResponse, error) {
	registered := []string{}
	for _, sub := range e.subscriptions {
		registered = append(registered, fmt.Sprintf("%s/%s", sub.namespace, sub.pipeline))
	}

	return &proto.ExtensionInfoResponse{
		Name:          e.systemConfig.Name,
		Documentation: "https://clintjedwards.com/gofer/ref/extensions/provided/interval.html",
		Registered:    registered,
	}, nil
}

// The ExternalEvent endpoint tells the extension what to do if they get messages from Gofer's external event system.
// This system is set up to facilitate webhook interactions like those that occur for github
// (A user pushes a branch, Gofer gets an event from github).
// The ExternalEvent will come with a payload which the extension can then authenticate, process, and take action on.
func (e *extension) ExternalEvent(ctx context.Context, request *proto.ExtensionExternalEventRequest) (*proto.ExtensionExternalEventResponse, error) {
	return &proto.ExtensionExternalEventResponse{}, nil
}

// A graceful shutdown for a extension should clean up any resources it was working with that might be left hanging.
// Sometimes that means sending requests to third parties that it is shutting down, sometimes that just means
// reaping its personal goroutines.
func (e *extension) Shutdown(ctx context.Context, request *proto.ExtensionShutdownRequest) (*proto.ExtensionShutdownResponse, error) {
	if e.isInitialized {
		e.quitAllSubscriptions()
	}

	return &proto.ExtensionShutdownResponse{}, nil
}

// The ExtensionInstaller is a small script that gets piped to the admin who is trying to set up this particular
// extension. The installer is meant to guide the user through the different configuration options that the
// installer has globally.
func (e *extension) RunExtensionInstaller(stream proto.ExtensionService_RunExtensionInstallerServer) error {
	err := sdk.SendInstallerMessageToClient(stream, "The interval extension allows users to run their pipelines on the passage of "+
		"time by setting a particular duration.\n")
	if err != nil {
		return err
	}

	err = sdk.SendInstallerMessageToClient(stream, "First, let's prevent users from setting too low of an interval by "+
		"setting a minimum duration. Durations are set via Golang duration strings. For example, entering a duration "+
		"of '10h' would be 10 hours, meaning that users can only run their pipeline at most every 10 hours. "+
		"You can find more documentation on valid strings here: https://pkg.go.dev/time#ParseDuration.")
	if err != nil {
		return err
	}

	for {
		err = sdk.SendInstallerQueryToClient(stream, "Set a minimum duration for all pipelines: ")
		if err != nil {
			return err
		}

		clientMsg, err := stream.Recv()
		if err != nil {
			return err
		}

		_, err = time.ParseDuration(clientMsg.Msg)
		if err != nil {
			err = sdk.SendInstallerMessageToClient(stream, fmt.Sprintf("Malformed duration %q; %v", clientMsg.Msg, err))
			if err != nil {
				return err
			}
			continue
		}

		err = sdk.SendInstallerConfigSettingToClient(stream, ConfigMinInterval, clientMsg.Msg)
		if err != nil {
			return err
		}

		break
	}

	err = sdk.SendInstallerMessageToClient(stream, "Interval extension configuration finished")
	if err != nil {
		return err
	}

	return nil
}

// The PipelineConfigurator is a small script that a pipeline owner can run when subscribing to this extension.
// It's meant to guide the pipeline owner through the different options of the extension.
func (e *extension) RunPipelineConfigurator(stream proto.ExtensionService_RunPipelineConfiguratorServer) error {
	err := sdk.SendConfiguratorMessageToClient(stream, "The interval extension allows you to run your pipelines on the passage of "+
		"time by setting a particular duration.\n")
	if err != nil {
		return err
	}

	err = sdk.SendConfiguratorMessageToClient(stream, "Durations are set via Golang duration strings. "+
		"For example, entering a duration of '10h' would be 10 hours, meaning that your pipeline would run once every 10 hours. "+
		"You can find more documentation on valid strings here: https://pkg.go.dev/time#ParseDuration.")
	if err != nil {
		return err
	}

	for {
		err = sdk.SendConfiguratorQueryToClient(stream, "Set your pipeline run interval: ")
		if err != nil {
			return err
		}

		clientMsg, err := stream.Recv()
		if err != nil {
			return err
		}

		_, err = time.ParseDuration(clientMsg.Msg)
		if err != nil {
			err = sdk.SendConfiguratorMessageToClient(stream, fmt.Sprintf("Malformed duration %q; %v", clientMsg.Msg, err))
			if err != nil {
				return err
			}
			continue
		}

		err = sdk.SendConfiguratorParamSettingToClient(stream, ParameterEvery, clientMsg.Msg)
		if err != nil {
			return err
		}

		break
	}

	err = sdk.SendConfiguratorMessageToClient(stream, "Interval extension configuration finished")
	if err != nil {
		return err
	}

	return nil
}

// Lastly we call our personal NewExtension function, which now implements the ExtensionServerInterface and then we
// pass it to the NewExtension function within the SDK.
//
// From here the SDK will use the given interface and run a GRPC service whenever this program is called with the
// positional parameter "server". Ex. "./extension server"
//
// Whenever this program is called with the parameter "installer" then it will print out the installation instructions
// instead.
func main() {
	extension := extension{}
	sdk.NewExtension(&extension)
}

Provided Extensions

Gofer provides some pre-written extensions for quick use:

Cron Extension

Cron allows users to schedule pipeline runs on long term intervals and specific days.

It uses a stripped down version of the cron syntax to do so:

Field           Allowed values  Allowed special characters

Minutes         0-59            * , -
Hours           0-23            * , -
Day of month    1-31            * , -
Month           1-12            * , -
Day of week     0-6             * , -
Year            1970-2100       * , -

┌───────────── minute (0 - 59)
│ ┌───────────── hour (0 - 23)
│ │ ┌───────────── day of the month (1 - 31)
│ │ │ ┌───────────── month (1 - 12)
│ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday)
│ │ │ │ │ ┌───────────── Year (1970-2100)
│ │ │ │ │ │
│ │ │ │ │ │
│ │ │ │ │ │
* * * * * *

Pipeline Configuration

• expression : Specifies the cron expression of the interval desired.

Every year on Xmas

gofer pipeline extend simple cron yearly_on_xmas -s expression="0 1 25 12 * *"

Extension Configuration

None

Interval Extension

Interval simply runs the subscribed pipeline at the given time interval continously.

Parameters/Pipeline Configuration

• every : Specifies the time duration between events. Unless changed via the extension configuration, the minimum for this is 5 mins.

gofer pipeline extend simple interval every_five_mins -s every="5m"

Extension Configuration

Extension configurations are set upon extension startup and cannot be changed afterwards without restarting said extension.

Github Extension

The Github extension allows Gofer pipelines to be run on Github webhook events. This makes it possible to write event driven workloads that depend on an action happening on Github.

See the events section below for all supported events and the environment variables they pass to each pipeline.

:::info Due to the nature of Github's API and webhooks, you'll need to first set up a new Github app to use with Gofer's Github extension.

Steps to accomplish this can be found in the additional steps section. :::

:::danger The Github extension requires the external events feature of Gofer in order to accept webhooks from Github's servers. This requires your application to take traffic from external, potentially unknown sources.

Visit the external events page for more information on how to configure Gofer's external events endpoint.

If Github is your only external extension, to increase security consider limiting the IP addresses that can access Gofer's external events endpoint. :::

Pipeline Configuration

• repository : The Github repository you would like to listen for events from. The format is in the form <organization>/<repository>.

extension "github" "only_from_experimental" {
    repository = "clintjedwards/experimental"
    events = "push"
}

• events : A comma separated list of events you would like to listen for.

extension "github" "only_from_experimental" {
    repository = "clintjedwards/experimental"
    events = "push,create"
}

Extension Configuration

Extension configurations are set upon startup and cannot be changed afterwards.

The Github extension requires the setup and use of a new Github app. You can view setup instructions below which will walk you through how to retrieve the required env var variables.

extensions {
  registered_extensions "github" {
    image = "ghcr.io/clintjedwards/gofer/extension_github:latest"
    env_vars = {
      "GOFER_EXTENSION_GITHUB_APPS_ID": "112348",
      "GOFER_EXTENSION_GITHUB_APPS_INSTALLATION": "99560091",
      "GOFER_EXTENSION_GITHUB_APPS_KEY": <<EOT
TUtkUnhYY01LTUI1ejgzZU84MFhKQWhoNnBkaFlCQlg0NGl5awpUUTBuaENySGRVT2kvN3hVaHp6
eTgxb3d0RUdpdUFQakJIOVhpSlczQm9hazYrSTZKWjU2RC95YllPbkVSaTdFClIxVkRQeGdGa0lE
NHdUbmtHdU4vdFY1VzBuZ3Q1aW0yVG5OVGVqc0NnWUVBb0pBMlJXZ2ZaSDdobVo3VS82TFUKSi9a
WTBZYmNkOU80anpYdWRUTUo1TXVKcVEwY004bnZhb09tS1Q1ekRadnBla01sRDlaYmZ4Rlg2Mzh3
N2ZuZwp0N05lbGFZc3IxYUhFWi9Rd2pveFo2RXpEWUJSQ0M2SEFvQmJXZmdwc1FCMkhNV3lzb2ls
LUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
EOT
      "GOFER_EXTENSION_GITHUB_APPS_WEBHOOK_SECRET": "somereallylongstringofcharacters",
    }
  }
}

Additional setup

Due to the nature of Github's API and webhooks, you'll need to first set up a new Github app to use with Gofer's Github extension. Once this app has been set up, you'll have access to all the required environment variables that you'll need to pass into Gofer's server configuration.

Here is a quick and dirty walkthrough on the important parts of setting up the Github application.

1. Create a new Github application:

Github's documentation will be the most up to date and relevant so please see their walkthrough.

On the configuration page for the new Github application the following should be noted:

• APP ID: Take note of the id; it will be used later for extension configuration.

• Webhook URL: Should be the address of your Gofer's external extension instance and pointing to the events/github endpoint:

  ex: https://mygoferinstance.yourdomain.com/events/github

• Webhook Secret: Make this a secure, long, random string of characters and note it for future extension configuration.

• Private Keys: Generate a private key and store it somewhere safe. You'll need to base64 this key and insert it into the extension configuration.

  base64 ~/Desktop/myorg-gofer.2022-01-24.private-key.pem

2. Find the installation ID

Once the Github application has been created, install it. This will give you an opportunity to configure the permissions and scope of the Github application. It is recommended that you give read-only permissions to any permissions that might include webhooks and read-write for code-suite and code-runs.

The installation ID is unfortunately hidden in an event that gets sent once the Github app has been created and installed. You can find it by navigating to the settings page for the Github application and then viewing it in the "Recent Deliveries" page.

Events

Gofer's extensions have the ability to pass along event specific information in the form of environment variables that get injected into each container's run. Most of these variables are pulled from the webhook request that comes in.

Below is a breakdown of the environment variables that are passed to a run based on the event that was generated. You can find more information about the format the variables will be in by referencing the payloads for the event.

Events below are the only events that are supported.

Command Line

Gofer's main way of providing interaction is through a command line application included in the Gofer binary.

This command line tool is how you upload pipelines, view runs, upload artifacts and many other common Gofer tasks.

To view the possible commands for the Gofer pipeline simply run gofer -h.

Configuration

The Gofer CLI accepts configuration through flags, environment variables, or a configuration file.

When multiple configuration sources are used the hierarchy is (from lowest to highest) config file values -> environment variables -> flags. Meaning that if you give the same configurations different values through a configuration file and through flags, the value given in the flag will prevail.

Flags

You can view Gofer's global flags by simply typing gofer -h.

Environment variables

You can also set configuration values through environment variables. Each environment variable has a prefix of GOFER_CLI_.

For example, setting your API token:

export GOFER_CLI_TOKEN=mysupersecrettoken
gofer service token whoami

Each environment variable available is just the flag with a prefix of GOFER_CLI.

export GOFER_CLI_HOST=localhost:8080

Configuration file

For convenience reasons Gofer can also use a standard configuration file. The language of this file is HCL. Most of the options are simply in the form of key=value.

Configuration file locations

You can put your CLI configuration file in any of the following locations and Gofer will automatically detect and read from it(in order of first searched):

1. The path given to the --config flag
2. $HOME/.gofer.hcl
3. $HOME/.config/gofer.hcl

Configuration file options

The options available in the configuration file are the same as the global flags:

gofer -h

...
Flags:
   --detail
...

# The flag 'detail' maps back to the configuration file as the same name

# gofer.hcl
detail = false

Example configuration file

// /home/clintjedwards/.gofer.hcl
namespace = "myNamespace"
detail    = false
format    = "pretty"
host      = "localhost:8080"
no_color  = false
token     = "mysupersecrettoken"