Configuration Reference

Gofer has a variety of parameters that can be specified via environment variables or the configuration file.

To view a listing of the possible environment variables use: gofer service printenv.

The most up to date config file values can be found by reading the code, but a best effort key and description list is given below.

If examples of these values are needed you can find a sample file by using gofer service init-config.

Values

General

nametypedefaultdescription
dev_modebooleantrueDev mode controls many aspects of Gofer to make it easier to run locally for development and testing. Because of this you should not run dev mode in production as it is not safe. A non-complete list of things dev-mode helps with: the use of localhost certificates, auto-generation of encryption key, bypass of authentication for all routes.
event_log_retentionstring (duration)4380hControls how long Gofer will hold onto events before discarding them. This is important factor in disk space and memory footprint. Example: Rough math on a 5,000 pipeline Gofer instance with a full 6 months of retention puts the memory and storage footprint at about 9GB.
event_prune_intervalstring3hHow often to check for old events and remove them from the database.
ignore_pipeline_run_eventsbooleanfalseControls the ability for the Gofer service to execute jobs on startup. If this is set to false you can set it to true manually using the CLI command gofer service toggle-event-ingress.
log_levelstringdebugThe logging level that is output. It is common to start with info.
run_parallelism_limitintN/AThe limit automatically imposed if the pipeline does not define a limit. 0 is unlimited.
task_run_logs_dirstring/tmpThe path of the directory to store task run logs. Task run logs are stored as a text file on the server.
task_run_log_expiryint20The total amount of runs before logs of the oldest run will be deleted.
task_run_stop_timeoutstring5mThe amount of time Gofer will wait for a container to gracefully stop before sending it a SIGKILL.
external_events_apiblockN/AThe external events API controls webhook type interactions with triggers. HTTP requests go through the events endpoint and Gofer routes them to the proper trigger for handling.
object_storeblockN/AThe settings for the Gofer object store. The object store assists Gofer with storing values between tasks since Gofer is by nature distributed. This helps jobs avoid having to download the same objects over and over or simply just allows tasks to share certain values.
secret_storeblockN/AThe settings for the Gofer secret store. The secret store allows users to securely populate their pipeline configuration with secrets that are used by their tasks, trigger configuration, or scheduler.
schedulerblockN/AThe settings for the container orchestrator that Gofer will use to schedule workloads.
serverblockN/AControls the settings for the Gofer API service properties.
triggersblockN/AControls settings for Gofer's trigger system. Triggers are different workflows for running pipelines usually based on some other event (like the passing of time).

External Events API (block)

The external events API controls webhook type interactions with triggers. HTTP requests go through the events endpoint and Gofer routes them to the proper trigger for handling.

nametypedefaultdescription
enablebooleantrueEnable the events api. If this is turned off the events http service will not be started.
hoststringlocalhost:8081The address and port to bind the events service to.

Example

external_events_api {
  enable = true
  host   = "0.0.0.0:8081"
}

Object Store (block)

The settings for the Gofer object store. The object store assists Gofer with storing values between tasks since Gofer is by nature distributed. This helps jobs avoid having to download the same objects over and over or simply just allows tasks to share certain values.

You can find more information on the object store block here.

nametypedefaultdescription
enginestringsqliteThe engine Gofer will use to store state. The accepted values here are "sqlite".
pipeline_object_limitint50The limit to the amount of objects that can be stored at the pipeline level. Objects stored at the pipeline level are kept permanently, but once the object limit is reach the oldest object will be deleted.
run_object_expiryint50Objects stored at the run level are unlimited in number, but only last for a certain number of runs. The number below controls how many runs until the run objects for the oldest run will be deleted. Ex. an object stored on run number #5 with an expiry of 2 will be deleted on run #7 regardless of run health.

Sqlite (block)

The sqlite store is a built-in, easy to use object store. It is meant for development and small deployments.

nametypedefaultdescription
pathstring/tmp/gofer-object.dbThe path of the file that sqlite will use. If this file does not exist Gofer will create it.
sqliteblockN/AThe sqlite storage engine.
object_store {
  engine = "sqlite"
  sqlite {
    path = "/tmp/gofer-object.db"
  }
}

Secret Store (block)

The settings for the Gofer secret store. The secret store allows users to securely populate their pipeline configuration with secrets that are used by their tasks, trigger configuration, or scheduler.

You can find more information on the secret store block here.

nametypedefaultdescription
enginestringsqliteThe engine Gofer will use to store state. The accepted values here are "sqlite".
sqliteblockN/AThe sqlite storage engine.

Sqlite (block)

The sqlite store is a built-in, easy to use object store. It is meant for development and small deployments.

nametypedefaultdescription
pathstring/tmp/gofer-secret.dbThe path of the file that sqlite will use. If this file does not exist Gofer will create it.
encryption_keystring"changemechangemechangemechangeme"Key used to encrypt keys to keep them safe. This encryption key is responsible for facilitating that. It MUST be 32 characters long and cannot be changed for any reason once it is set or else all data will be lost.
secret_store {
  engine = "sqlite"
  sqlite {
    path = "/tmp/gofer-secret.db"
    encryption_key = "changemechangemechangemechangeme"
  }
}

Scheduler (block)

The settings for the container orchestrator that Gofer will use to schedule workloads.

You can find more information on the scheduler block here.

nametypedefaultdescription
enginestringsqliteThe engine Gofer will use as a container orchestrator. The accepted values here are "docker".
dockerblockN/ADocker is the default container orchestrator and leverages the machine's local docker engine to schedule containers.

Docker (block)

Docker is the default container orchestrator and leverages the machine's local docker engine to schedule containers.

nametypedefaultdescription
prunebooleanfalseControls if the docker scheduler should periodically clean up old containers.
prune_intervalstring24hControls how often the prune container job should run.
scheduler {
  engine = "docker"
  docker {
    prune          = true
    prune_interval = "24h"
  }
}

Server (block)

Controls the settings for the Gofer service's server properties.

nametypedefaultdescription
hoststringlocalhost:8080The address and port for the service to bind to.
shutdown_timeoutstring15sThe time Gofer will wait for all connections to drain before exiting.
tls_cert_pathstringThe TLS certificate Gofer will use for the main service endpoint. This is required.
tls_key_pathstringThe TLS certificate key Gofer will use for the main service endpoint. This is required.
storage_pathstring/tmp/gofer.dbWhere to put Gofer's sqlite database.
storage_results_limitint200The amount of results Gofer's database is allowed to return on one query.
server {
  host                  = "localhost:8080"
  dev_mode              = false
  tls_cert_path         = "./localhost.crt"
  tls_key_path          = "./localhost.key"
  tmp_dir               = "/tmp"
  storage_path          = "/tmp/gofer.db"
  storage_results_limit = 200
}

Triggers (block)

Controls settings for Gofer's trigger system. Triggers are different workflows for running pipelines usually based on some other event (like the passing of time).

You can find more information on the trigger block here.

nametypedefaultdescription
install_base_triggersbooleantrueAttempts to automatically install the cron and interval triggers on first startup.
stop_timeoutstring5mThe amount of time Gofer will wait until trigger containers have stopped before sending a SIGKILL.
tls_cert_pathstringThe TLS certificate path Gofer will use for the triggers. This should be a certificate that the main Gofer service will be able to access.
tls_key_pathstringThe TLS certificate path key Gofer will use for the triggers. This should be a certificate that the main Gofer service will be able to access.
triggers {
  install_base_triggers = true
  stop_timeout          = "5m"
  tls_cert_path         = "./localhost.crt"
  tls_key_path          = "./localhost.key"
}