Local Service Specification¶

A local service is defined using JSON. The structure is similar to application specifications, but deployment semantics are per executor host.

Sample Local Service Definition¶

{
    "name": "NODE_EXPORTER", // (1)!
    "version": "1", // (2)!
    "type": "SERVICE", // (3)!
    "executable": { // (4)!
        "type": "DOCKER", // (5)!
        "url": "ghcr.io/appform-io/perf-test-server-httplib", // (6)!
        "dockerPullTimeout": "120 seconds" // (7)!
    },
    "exposedPorts": [ // (8)!
        {
            "name": "metrics", // (9)!
            "port": 9000, // (10)!
            "type": "HTTP" // (11)!
        }
    ],
    "resources": [ // (12)!
        {
            "type": "CPU",
            "count": 1 // (13)!
        },
        {
            "type": "MEMORY",
            "sizeInMB": 256 // (14)!
        }
    ],
    "placementPolicy": { // (15)!
        "type": "LOCAL", // (16)!
        "hostLevel": false // (17)!
    },
    "healthcheck": { // (18)!
        "mode": {
            "type": "HTTP",
            "protocol": "HTTP",
            "portName": "metrics",
            "path": "/health",
            "verb": "GET",
            "successCodes": [
                200
            ],
            "payload": "",
            "connectionTimeout": "1 second"
        },
        "timeout": "2 seconds",
        "interval": "5 seconds",
        "attempts": 3,
        "initialDelay": "0 seconds"
    },
    "readiness": { // (19)!
        "mode": {
            "type": "HTTP",
            "protocol": "HTTP",
            "portName": "metrics",
            "path": "/ready",
            "verb": "GET",
            "successCodes": [
                200
            ],
            "payload": "",
            "connectionTimeout": "1 second"
        },
        "timeout": "2 seconds",
        "interval": "3 seconds",
        "attempts": 3,
        "initialDelay": "0 seconds"
    },
    "volumes": [ // (20)!
        {
            "pathInContainer": "/host/proc",
            "pathOnHost": "/proc",
            "mode": "READ_ONLY"
        }
    ],
    "configs": [ // (21)!
        {
            "type": "INLINE",
            "localFilename": "/config/runtime.yaml",
            "data": "bG9nTGV2ZWw6IElORk8="
        }
    ],
    "env": { // (22)!
        "LOG_LEVEL": "INFO"
    },
    "args": [ // (23)!
        "--scrape-all"
    ],
    "tags": { // (24)!
        "owner": "platform",
        "tier": "infra"
    },
    "logging": { // (25)!
        "type": "LOCAL",
        "maxSize": "100m",
        "maxFiles": 3,
        "compress": true
    },
    "preShutdown": { // (26)!
        "hooks": [
            {
                "type": "HTTP",
                "protocol": "HTTP",
                "portName": "metrics",
                "path": "/shutdown",
                "verb": "POST",
                "successCodes": [
                    200
                ],
                "payload": "",
                "connectionTimeout": "1 second"
            }
        ],
        "waitBeforeKill": "3 seconds"
    }
}

1. Human readable local service name.
2. Version for this service specification.
3. Use SERVICE for long-running local services.
4. Executable coordinates.
5. Currently supported executable type is DOCKER.
6. Docker image URI.
7. Image pull timeout.
8. Ports exposed from container.
9. Logical port name.
10. Container port.
11. Port type (HTTP, HTTPS, TCP, UDP).
12. CPU and memory resources.
13. CPU cores per instance.
14. Memory in MB per instance.
15. Placement policy.
16. Local services must use LOCAL placement policy.
17. hostLevel controls host-level behavior for local services. When hostLevel=true, service can run only one instance per executor.
18. Health check configuration.
19. Readiness check configuration.
20. Volume mounts.
21. File/config injection.
22. Custom environment variables.
23. Command line arguments.
24. Metadata tags.
25. Logging behavior.
26. Pre-shutdown hooks.

Host-level mode (hostLevel: true)¶

By default, Drove maps container ports to dynamically selected host ports. This works well for most workloads, but can be inconvenient for node-level infrastructure services that must be reachable on a fixed, well-known host port.

When placementPolicy.hostLevel is set to true, Drove binds each declared container port to the same port number on the host.

This has two important implications:

1. Only one instance can run per host, otherwise port binding conflicts will occur.
2. Restart-style operations must use stop-first ordering so old instance ports are released before new instances start.

For this reason, host-level services should be operated in stop-first mode for restart/replace flows.

Executable Specification¶

Right now Drove supports only docker containers. However as engines, both docker and podman are supported. Drove executors will fetch the executable directly from the registry based on the configuration provided.

Resource Requirements Specification¶

This section specifies the hardware resources required to run the container. Right now only CPU and MEMORY are supported as resource types that can be reserved for a container.

CPU Requirements¶

Specifies number of cores to be assigned to the container.

Memory Requirements¶

Specifies amount of memory to be allocated to a container.

Sample

[
    {
        "type": "CPU",
        "count": 1
    },
    {
        "type": "MEMORY",
        "sizeInMB": 128
    }
]

Volume Specification¶

Files and directories can be mounted from the executor host into the container. The volumes section contains a list of volumes that need to be mounted.

Config Specification¶

Drove supports injection of configuration files into containers. The specifications for the same are discussed below.

Inline config¶

Inline configuration can be added in the Application Specification itself. This will manifest as a file inside the container.

The following details are needed for this:

Config file:

port: 8080
logLevel: DEBUG

{
    "type" : "INLINE",
    "localFilename" : "/config/service.yml",
    "data" : "cG9ydDogODA4MApsb2dMZXZlbDogREVCVUcK"
}

Locally loaded config¶

Config file from a path on the executor directly. Such files can be distributed to the executor host using existing configuration management systems such as OpenTofu, Salt etc.

The following details are needed for this:

Sample config specification:

{
    "type" : "EXECUTOR_LOCAL_FILE",
    "localFilename" : "/config/service.yml",
    "data" : "/mnt/configs/myservice/config.yml"
}

Controller fetched Config¶

Config file can be fetched from a remote server by the controller. Once fetched, these will be streamed to the executor as part of the instance specification for starting a container.

The following details are needed for this:

Sample config specification:

{
    "type" : "CONTROLLER_HTTP_FETCH",
    "localFilename" : "/config/service.yml",
    "http" : {
        "protocol" : "HTTP",
        "hostname" : "configserver.internal.yourdomain.net",
        "port" : 8080,
        "path" : "/configs/myapp",
        "username" : "appuser",
        "password" : "secretpassword"
    }
}

Executor fetched Config¶

Config file can be fetched from a remote server by the executor before spinning up a container. Once fetched, the payload will be injected as a config file into the container.

The following details are needed for this:

Sample config specification:

{
    "type" : "EXECUTOR_HTTP_FETCH",
    "localFilename" : "/config/service.yml",
    "http" : {
        "protocol" : "HTTP",
        "hostname" : "configserver.internal.yourdomain.net",
        "port" : 8080,
        "path" : "/configs/myapp",
        "username" : "appuser",
        "password" : "secretpassword"
    }
}

HTTP Call Specification¶

This section details the options that can set when making http calls to a configuration management system from controllers or executors.

The following options are available for HTTP call:

Placement Policy Specification¶

Placement policy governs how Drove deploys containers on the cluster. The following sections discuss the different placement policies available and how they can be configured to achieve optimal placement of containers.

Any Placement¶

Containers for a {appName, version} combination can run on any un-tagged executor host.

Sample:

{
    "type" : "ANY"
}

One Per Host Placement¶

Ensures that only one container for a particular {appName, version} combination is running on an executor host at a time.

Sample:

{
    "type" : "ONE_PER_HOST"
}

Max N Per Host Placement¶

Ensures that at most N containers for a {appName, version} combination is running on an executor host at a time.

Sample:

{
    "type" : "MAX_N_PER_HOST",
    "max": 3
}

Match Tag Placement¶

Ensures that containers for a {appName, version} combination are running on an executor host that has the tags as mentioned in the policy.

Sample:

{
    "type" : "MATCH_TAG",
    "tag": "gpu_enabled"
}

No Tag Placement¶

Ensures that containers for a {appName, version} combination are running on an executor host that has no tags.

Sample:

{
    "type" : "NO_TAG"
}

Composite Policy Based Placement¶

Composite policy can be used to combine policies together to create complicated placement requirements.

Sample:

{
    "type" : "COMPOSITE",
    "policies": [
        {
            "type": "ONE_PER_HOST"
        },
        {
            "type": "MATH_TAG",
            "tag": "gpu_enabled"
        }
    ],
    "combiner" : "AND"
}

Environment variables¶

This config can be used to inject custom environment variables to containers. The values are defined as part of deployment specification, are same across the cluster and immutable to modifications from inside the container (ie any overrides from inside the container will not be visible across the cluster).

Sample:

{
    "MY_VARIABLE_1": "fizz",
    "MY_VARIABLE_2": "buzz"
}

The following environment variables are injected by Drove to all containers:

Command line arguments¶

A list of command line arguments that are sent to the container engine to execute inside the container. This is provides ways for you to configure your container behaviour based off such arguments. Please refer to docker documentation for details.

Logging Specification¶

Can be used to configure how container logs are managed on the system.

Local Logger configuration¶

This is used to configure the json-file log driver.

Rsyslog configuration¶

In case suers want to stream logs to an rsyslog server, the logging configuration needs to be set to RSYSLOG mode.