Application Specification¶

An application is defined using JSON. We use a sample configuration below to explain the options.

Sample Application Definition¶

{
    "name": "TEST_APP", // (1)!
    "version": "1", // (2)!
    "type": "SERVICE", // (3)!
    "executable": { //(4)!
        "type": "DOCKER", // (5)!
        "url": "ghcr.io/appform-io/perf-test-server-httplib",// (6)!
        "dockerPullTimeout": "100 seconds"// (7)!
    },
    "resources": [//(20)!
        {
            "type": "CPU",
            "count": 1//(21)!
        },
        {
            "type": "MEMORY",
            "sizeInMB": 128//(22)!
        }
    ],
    "volumes": [//(12)!
        {
            "pathInContainer": "/data",//(13)!
            "pathOnHost": "/mnt/datavol",//(14)!
            "mode" : "READ_WRITE"//(15)!
        }
    ],
    "configs" : [//(16)!
        {
            "type" : "INLINE",//(17)!
            "localFilename": "/testfiles/drove.txt",//(18)!
            "data" : "RHJvdmUgdGVzdA=="//(19)!
        }
    ],
    "placementPolicy": {//(23)!
        "type": "ANY"//(24)!
    },
    "exposedPorts": [//(8)!
        {
            "name": "main",//(9)!
            "port": 8000,//(10)!
            "type": "HTTP"//(11)!
        }
    ],
    "healthcheck": {//(25)!
        "mode": {//(26)!
            "type": "HTTP", //(27)!
            "protocol": "HTTP",//(28)!
            "portName": "main",//(29)!
            "path": "/",//(30)!
            "verb": "GET",//(31)!
            "successCodes": [//(32)!
                200
            ],
            "payload": "", //(33)!
            "connectionTimeout": "1 second" //(34)!
        },
        "timeout": "1 second",//(35)!
        "interval": "5 seconds",//(36)!
        "attempts": 3,//(37)!
        "initialDelay": "0 seconds"//(38)!
    },
    "readiness": {//(39)!
        "mode": {
            "type": "HTTP",
            "protocol": "HTTP",
            "portName": "main",
            "path": "/",
            "verb": "GET",
            "successCodes": [
                200
            ],
            "payload": "",
            "connectionTimeout": "1 second"
        },
        "timeout": "1 second",
        "interval": "3 seconds",
        "attempts": 3,
        "initialDelay": "0 seconds"
    },
    "exposureSpec": {//(42)!
        "vhost": "testapp.local", //(43)!
        "portName": "main", //(44)!
        "mode": "ALL"//(45)!
    },
    "env": {//(41)!
        "CORES": "8"
    },
    "args" : [//(54)!
        "./entrypoint.sh",
        "arg1",
        "arg2"
    ],
    "tags": { //(40)!
        "superSpecialApp": "yes_i_am",
        "say_my_name": "heisenberg"
    },
    "preShutdown": {//(46)!
        "hooks": [ //(47)!
            {
                "type": "HTTP",
                "protocol": "HTTP",
                "portName": "main",
                "path": "/",
                "verb": "GET",
                "successCodes": [
                    200
                ],
                "payload": "",
                "connectionTimeout": "1 second"
            }
        ],
        "waitBeforeKill": "3 seconds"//(48)!
    },
    "logging": {//(49)!
        "type": "LOCAL",//(50)!
        "maxSize": "100m",//(51)!
        "maxFiles": 3,//(52)!
        "compress": true//(53)!
    }
}

A human readable name for the application. This will remain constant for different versions of the app.
A version number. Drove does not enforce any format for this, but it is recommended to increment this for changes in spec.
This should be fixed to SERVICE for an application/service.
Coordinates for the executable. Refer to Executable Specification for details.
Right now the only type supported is DOCKER.
Docker container address
Timeout for container pull.
The ports to be exposed from the container.
A logical name for the port. This will be used to reference this port in other sections.
Actual port number as mentioned in Dockerfile.
Type of port. Can be: HTTP, HTTPS, TCP, UDP.
Volumes to be mounted. Refer to Volume Specification for details.
Path that will be visible inside the container for this mount.
Actual path on the host machine for the mount.
Mount mode can be READ_WRITE and READ_ONLY
Configuration to be injected as file inside the container. Please refer to Config Specification for details.
Type of config. Can be INLINE, EXECUTOR_LOCAL_FILE, CONTROLLER_HTTP_FETCH and EXECUTOR_HTTP_FETCH. Specifies how drove will get the contents to be injected..
File name for the config inside the container.
Serialized form of the data, this and other parameters will vary according to the type specified above.
List of resources required to run this application. Check Resource Requirements Specification for more details.
Number of CPU cores to be allocated.
Amount of memory to be allocated expressed in Megabytes
Specifies how the container will be placed on the cluster. Check Placement Policy for details.
Type of placement can be ANY, ONE_PER_HOST, MATCH_TAG, NO_TAG, RULE_BASED, ANY and COMPOSITE. Rest of the parameters in this section will depend on the type.
Health check to ensure service is running fine. Refer to Check Specification for details.
Mode of health check, can be api call or command.
Type of this check spec. Type can be HTTP or CMD. Rest of the options in this example are HTTP specific.
API call protocol. Can be HTTP/HTTPS
Port name as mentioned in the exposedPorts section.
HTTP path. Include query params here.
HTTP method. Can be GET,PUT or POST.
Set of HTTP status codes which can be considered as success.
Payload to be sent for POST and PUT calls.
Connection timeout for the port.
Timeout for the check run.
Interval between check runs.
Max attempts after which the overall check is considered to be a failure.
Time to wait before starting check runs.
Readiness check to pass for the container to be considered as ready. Refer to Check Specification for details.
Key value metadata that can be used in external systems.
Custom environment variables. Additional variables are injected by Drove as well. See Environment Variables section for details.
Specifies the virtual host on which this container is exposed.
FQDN for the virtual host.
Port name as specified in exposedPorts section.
Mode for exposure. Set this to ALL for now.
Things to do before a container is shutdown. Check Pre Shutdown Behavior for more details.
Hooks (HTTP api call or shell command) to run before shutting down the container. Format is same as health/readiness checks. Refer to HTTP Check Actions and Command Check Options for details.
Time to wait before killing the container. The container will be in UNREADY state during this time and hence won't have api calls routed to it via Drove Gateway.
Specify how docker log files are configured. Refer to Logging Specification
Log to local file
Maximum File Size
Number of latest log files to retain
Log files will be compressed
List of command line arguments. See Command Line Arguments for details.

Executable Specification¶

Right now Drove supports only docker containers. However as engines, both docker and podman are supported. Drove executors will fetch the executable directly from the registry based on the configuration provided.

Name	Option	Description
Type	`type`	Set type to `DOCKER`.
URL	`url`	Docker container URL`.
Timeout	`dockerPullTimeout`	Timeout for docker image pull.

Note

Drove supports docker registry authentication. This can be configured in the executor configuration file.

Resource Requirements Specification¶

This section specifies the hardware resources required to run the container. Right now only CPU and MEMORY are supported as resource types that can be reserved for a container.

CPU Requirements¶

Specifies number of cores to be assigned to the container.

Name	Option	Description
Type	`type`	Set type to `CPU` for this.
Count	`count`	Number of cores to be assigned.

Memory Requirements¶

Specifies amount of memory to be allocated to a container.

Name	Option	Description
Type	`type`	Set type to `MEMORY` for this.
Count	`sizeInMB`	Amount of memory (in Mega Bytes) to be allocated.

Sample

[
    {
        "type": "CPU",
        "count": 1
    },
    {
        "type": "MEMORY",
        "sizeInMB": 128
    }
]

Note

Both CPU and MEMORY configurations are mandatory.

Volume Specification¶

Files and directories can be mounted from the executor host into the container. The volumes section contains a list of volumes that need to be mounted.

Name	Option	Description
Path In Container	`pathInContainer`	Path that will be visible inside the container for this mount.
Path On Host	`pathOnHost`	Actual path on the host machine for the mount.
Mount Mode	`mode`	Mount mode can be `READ_WRITE` and `READ_ONLY` to allow the containerized process to write or read to the volume.

Info

We do not support mounting remote volumes as of now.

Config Specification¶

Drove supports injection of configuration files into containers. The specifications for the same are discussed below.

Inline config¶

Inline configuration can be added in the Application Specification itself. This will manifest as a file inside the container.

The following details are needed for this:

Name	Option	Description
Type	`type`	Set the value to `INLINE`
Local Filename	`localFilename`	File name for the config inside the container.
Data	`data`	Base64 encoded string for the data. The value for this will be masked on UI.

Config file:

port: 8080
logLevel: DEBUG

Corresponding config specification:

{
    "type" : "INLINE",
    "localFilename" : "/config/service.yml",
    "data" : "cG9ydDogODA4MApsb2dMZXZlbDogREVCVUcK"
}

Warning

The full base 64 encoded config data will get stored in Drove ZK and will be pushed to executors inline. It is not recommended to stream large config files to containers using this method. This will probably need additional configuration on your ZK cluster.

Locally loaded config¶

Config file from a path on the executor directly. Such files can be distributed to the executor host using existing configuration management systems such as OpenTofu, Salt etc.

The following details are needed for this:

Name	Option	Description
Type	`type`	Set the value to `EXECUTOR_LOCAL_FILE`
Local Filename	`localFilename`	File name for the config inside the container.
File path	`filePathOnHost`	Path to the config file on executor host.

Sample config specification:

{
    "type" : "EXECUTOR_LOCAL_FILE",
    "localFilename" : "/config/service.yml",
    "data" : "/mnt/configs/myservice/config.yml"
}

Controller fetched Config¶

Config file can be fetched from a remote server by the controller. Once fetched, these will be streamed to the executor as part of the instance specification for starting a container.

The following details are needed for this:

Name	Option	Description
Type	`type`	Set the value to `CONTROLLER_HTTP_FETCH`
Local Filename	`localFilename`	File name for the config inside the container.
HTTP Call Details	`http`	HTTP Call related details. Please refer to HTTP Call Specification for details.

Sample config specification:

{
    "type" : "CONTROLLER_HTTP_FETCH",
    "localFilename" : "/config/service.yml",
    "http" : {
        "protocol" : "HTTP",
        "hostname" : "configserver.internal.yourdomain.net",
        "port" : 8080,
        "path" : "/configs/myapp",
        "username" : "appuser",
        "password" : "secretpassword"
    }
}

Note

The controller will make an API call for every single time it asks an executor to spin up a container. Please make sure to account for this in your configuration management system.

Executor fetched Config¶

Config file can be fetched from a remote server by the executor before spinning up a container. Once fetched, the payload will be injected as a config file into the container.

The following details are needed for this:

Name	Option	Description
Type	`type`	Set the value to `EXECUTOR_HTTP_FETCH`
Local Filename	`localFilename`	File name for the config inside the container.
HTTP Call Details	`http`	HTTP Call related details. Please refer to HTTP Call Specification for details.

Sample config specification:

{
    "type" : "EXECUTOR_HTTP_FETCH",
    "localFilename" : "/config/service.yml",
    "http" : {
        "protocol" : "HTTP",
        "hostname" : "configserver.internal.yourdomain.net",
        "port" : 8080,
        "path" : "/configs/myapp",
        "username" : "appuser",
        "password" : "secretpassword"
    }
}

Note

All executors will make an API call for every single time they spin up a container for this application. Please make sure to account for this in your configuration management system.

HTTP Call Specification¶

This section details the options that can set when making http calls to a configuration management system from controllers or executors.

The following options are available for HTTP call:

Name	Option	Description
Protocol	`protocol`	Protocol to use for upstream call. Can be `HTTP` or `HTTPS`.
Hostname	`hostname`	Host to call.
Port	`port`	Provide custom port. Defaults to 80 for http and 443 for https.
API Path	`path`	Path component of the URL. Include query parameters here. Defaults to `/`
HTTP Method	`verb`	Type of call, use `GET`, `POST` or `PUT`. Defaults to `GET`.
Success Code	`successCodes`	List of HTTP status codes which is considered as success. Defaults to `[200]`
Payload	`payload`	Data to be used for POST and PUT calls
Connection Timeout	`connectionTimeout`	Timeout for upstream connection.
Operation timeout	`operationTimeout`	Timeout for actual operation.
Username	`username`	Username to be used basic auth. This field is masked out on the UI.
Password	`password`	Password to be used for basic auth. This field is masked on the UI.
Authorization Header	`authHeader`	Data to be passed in HTTP `Authorization` header. This field is masked on the UI.
Additional Headers	`headers`	Any other headers to be passed to the upstream in the HTTP calls. This is a map of
Skip SSL Checks	`insecure`	Skip hostname and certification checks during SSL handshake with the upstream.

Placement Policy Specification¶

Placement policy governs how Drove deploys containers on the cluster. The following sections discuss the different placement policies available and how they can be configured to achieve optimal placement of containers.

Warning

All policies will work only at a {appName, version} combination level. They will not ensure constraints at an appName level. This means that for somethinge like a one per node placement, for the same appName, multiple containers can run on the same host if multiple deployments with different versions are active in a cluster. Same applies for all policies like N per host and so on.

Important details about executor tagging

All hosts have at-least one tag, it's own hostname.
The TAG policy will consider them as valid tags. This can be used to place containers on specific hosts if needed.
This is handled specially in all other policy types and they will consider executors having only the hostname tag as untagged.
A host with a tag (other than host) will not have any containers running if not placed on them specifically using the MATCH_TAG policy

Any Placement¶

Containers for a {appName, version} combination can run on any un-tagged executor host.

Name	Option	Description
Policy Type	`type`	Put `ANY` as policy.

Sample:

{
    "type" : "ANY"
}

Tip

For most use-cases this is the placement policy to use.

One Per Host Placement¶

Ensures that only one container for a particular {appName, version} combination is running on an executor host at a time.

Name	Option	Description
Policy Type	`type`	Put `ONE_PER_HOST` as policy.

Sample:

{
    "type" : "ONE_PER_HOST"
}

Max N Per Host Placement¶

Ensures that at most N containers for a {appName, version} combination is running on an executor host at a time.

Name	Option	Description
Policy Type	`type`	Put `MAX_N_PER_HOST` as policy.
Max count	`max`	The maximum num of containers that can run on an executor. Range: 1-64

Sample:

{
    "type" : "MAX_N_PER_HOST",
    "max": 3
}

Match Tag Placement¶

Ensures that containers for a {appName, version} combination are running on an executor host that has the tags as mentioned in the policy.

Name	Option	Description
Policy Type	`type`	Put `MATCH_TAG` as policy.
Max count	`tag`	The tag to match.

Sample:

{
    "type" : "MATCH_TAG",
    "tag": "gpu_enabled"
}

No Tag Placement¶

Ensures that containers for a {appName, version} combination are running on an executor host that has no tags.

Name	Option	Description
Policy Type	`type`	Put `NO_TAG` as policy.

Sample:

{
    "type" : "NO_TAG"
}

Info

The NO_TAG policy is mostly for internal use, and does not need to be specified when deploying containers that do not need any special placement logic.

Composite Policy Based Placement¶

Composite policy can be used to combine policies together to create complicated placement requirements.

Name	Option	Description
Policy Type	`type`	Put `COMPOSITE` as policy.
Polices	`policies`	List of policies to combine
Combiner	`combiner`	Can be `AND` and `OR` and signify all-match and any-match logic on the `policies` mentioned.

Sample:

{
    "type" : "COMPOSITE",
    "policies": [
        {
            "type": "ONE_PER_HOST"
        },
        {
            "type": "MATH_TAG",
            "tag": "gpu_enabled"
        }
    ],
    "combiner" : "AND"
}

The above policy will ensure that only one container of the relevant {appName,version} will run on GPU enabled machines.

Tip

It is easy to go into situations where no executors match complicated placement policies. Internally, we tend to keep things rather simple and use the ANY placement for most cases and maybe tags in a few places with over-provisioning or for hosts having special hardware

Environment variables¶

This config can be used to inject custom environment variables to containers. The values are defined as part of deployment specification, are same across the cluster and immutable to modifications from inside the container (ie any overrides from inside the container will not be visible across the cluster).

Sample:

{
    "MY_VARIABLE_1": "fizz",
    "MY_VARIABLE_2": "buzz"
}

The following environment variables are injected by Drove to all containers:

Variable Name	Value
HOST	Hostname where the container is running. This is for marathon compatibility.
PORT_`PORT_NUMBER`	A variable for every port specified in `exposedPorts` section. The value is the actual port on the host, the specified port is mapped to. For example if ports 8080 and 8081 are specified, two variables called `PORT_8080` and `PORT_8081` will be injected.
DROVE_EXECUTOR_HOST	Hostname where container is running.
DROVE_CONTAINER_ID	Container that is deployed
DROVE_APP_NAME	App name as specified in the Application Specification
DROVE_INSTANCE_ID	Actual instance ID generated by Drove
DROVE_APP_ID	Application ID as generated by Drove
DROVE_APP_INSTANCE_AUTH_TOKEN	A JWT string generated by Drove that can be used by this container to call `/apis/v1/internal/...` apis.

Warning

Do not pass secrets using environment variables. These variables are all visible on the UI as is. Please use Configs to inject secrets files and so on.

Command line arguments¶

A list of command line arguments that are sent to the container engine to execute inside the container. This is provides ways for you to configure your container behaviour based off such arguments. Please refer to docker documentation for details.

Danger

This might have security implications from a system point of view. As such Drove provides administrators a way to disable passing arguments at the cluster level by setting disableCmdlArgs to true in the controller configuration.

Check Specification¶

One of the cornerstones of managing applications on the cluster is to ensure we keep track of instance health and manage their life cycle depending on their health state. We need to define how to monitor health for containers accordingly. The checks will be executed on Applications and a Check result is generated. The result consists of the following:

Status - Healthy, Unhealthy or Stopped if the container is already in stopping state
Message - Any error message as generated by a specific checker

Common Options¶

Name	Option	Description
Mode	`mode`	The definition of a HTTP call or a Command to be executed in the container. See following sections for details.
Timeout	`timeout`	Duration for which we wait before declaring a check as failed
Interval	`interval`	Interval at which check will be retried
Attempts	`attempts`	Number of times a check is retried before it is declared as a failure
Initial Delay	`initialDelay`	Delay before executing the check for the first time.

Note

initialDelay is ignored when readiness checks and health checks are run in the recovery path as the container is already running at that point in time.

HTTP Check Options¶

Name	Option	Description
Type	`type`	Fixed to HTTP for HTTP checker
Protocol	`protocol`	HTTP or HTTPS call to be made
Port Name	`portName`	The name of the container port to make the http call on as specified in the Exposed Ports section in Application Spec
Path	`path`	The api path to call
HTTP method	`verb`	The HTTP Verb/Method to invoke. GET/PUT and POST are supported here
Success Codes	`successCodes`	A set of HTTP status codes that we should consider as a success from this API.
Payload	`payload`	A string payload that we can pass if the Verb is POST or PUT
Connection Timeout	`connectionTimeout`	Maximum time for which the checker will wait for the connection to be set up with the container.
Insecure	`insecure`	Skip hostname and certificate checks for HTTPS ports during checks.

Command Check Options¶

Field	Option	Description
Type	`type`	Fixed to CMD for command checker
Command	`command`	Command to execute in the container. (Equivalent to `docker exec -it <container> command>`)

Exposure Specification¶

Exposure spec is used to specify the virtual host Drove Gateway exposes to outside world for communication with the containers.

The following information needs to be specified:

Name	Option	Description
Virtual Host	`vhost`	The virtual host to be exposed on NGinx. This should be a fully qualified domain name.
Port Name	`portName`	The portname to be exposed on the vhost. Port names are defined in `exposedPorts` section.
Exposure Mode	`mode`	Use `ALL` here for now. Signifies that all healthy instances of the app are exposed to traffic.

Sample:

{
    "vhost": "teastapp.mydomain",
    "port": "main",
    "mode": "ALL"
}

Note

Application instances in any state other than HEALTHY are not considered for exposure. Please check Application Instance State Machine for an understanding of states of instances.

Configuring Pre Shutdown Behaviour¶

Before a container is shut down, it is desirable to ensure things are spun down properly. This behaviour can be configured in the preShutdown section of the configuration.

Name	Option	Description
Hooks	`hooks`	List of api calls and commands to be run on the container before it is killed. Each hook is either a HTTP Call Spec or Command Spec
Wait Time	`waitBeforeKill`	Time to wait before killing the container.

Sample

{
    "hooks": [
        {
            "type": "HTTP",
            "protocol": "HTTP",
            "portName": "main",
            "path": "/",
            "verb": "GET",
            "successCodes": [
                200
            ],
            "payload": "",
            "connectionTimeout": "1 second"
        }
    ],
    "waitBeforeKill": "3 seconds"//(48)!
}

Note

The waitBeforeKill timed wait kicks in after all the hooks have been executed.

Logging Specification¶

Can be used to configure how container logs are managed on the system.

Note

This section affects the docker log driver. Drove will continue to stream logs to it's own logger which can be configured at executor level through the executor configuration file.

Local Logger configuration¶

This is used to configure the json-file log driver.

Name	Option	Description
Type	`type`	Set the value to `LOCAL`
Max Size	`maxSize`	Maximum file size. Anything bigger than this will lead to rotation.
Max Files	`maxFiles`	Maximum number of logs files to keep. Range: 1-100
Compress	`compress`	Enable log file compression.

Tip

If logging section is omitted, the following configuration is applied by default: - File size: 10m - Number of files: 3 - Compression: on

Rsyslog configuration¶

In case suers want to stream logs to an rsyslog server, the logging configuration needs to be set to RSYSLOG mode.

Name	Option	Description
Type	`type`	Set the value to `RSYSLOG`
Server	`server`	URL for the rsyslog server.
Tag Prefix	`tagPrefix`	Prefix to add at the start of a tag
Tag Suffix	`tagSuffix`	Suffix to add at the en of a tag.

Note

The default tag is the DROVE_INSTANCE_ID. The tagPrefix and tagSuffix will to before and after this