Setting up Controllers¶

Controllers are the brains of Drove cluster. For HA, at least 2 controllers should be set up.

Please note the following behaviour about controllers: Please note the following behaviour about controllers:

• Only one controller is leader at a time. A leader controller does not relinquish control till the process is stopped or dies
• A controller process will die when it loses connectivity to Zookeeper
• The process/container for controller should keep restarting till it gets connectivity
• All decisions for the cluster are taken by the leader controller only
• During maintenance and package upgrades etc, it is better to roll changes out on non-leaders first and then do the leader at the end
• The controller process holds all metadata about the cluster, the current states and other information in memory.
  • Some of this information is backed by Zookeeper based storage layer
  • The other information is recreated dynamically based on updates from executors
• Controllers being down does not affect running containers on executors

Controller Configuration File Reference¶

The Drove Controller is written on the Dropwizard framework. The configuration to the service is set using a YAML file which needs to be injected into the container. A typical controller configuration file will look like the following:

server: #(1)!
  applicationConnectors: #(2)!
    - type: http
      port: 10000
  adminConnectors: #(3)!
    - type: http
      port: 4001
  applicationContextPath: / #(4)!
  requestLog: #(5)!
    appenders:
      - type: console
        timeZone: ${DROVE_TIMEZONE}
      - type: file
        timeZone: ${DROVE_TIMEZONE}
        currentLogFilename: /logs/drove-controller-access.log
        archivedLogFilenamePattern: /logs/drove-controller-access.log-%d-%i
        archivedFileCount: 3
        maxFileSize: 100MiB


logging: #(6)!
  level: INFO
  loggers:
    com.phonepe.drove: ${DROVE_LOG_LEVEL}

  appenders:
    - type: console #(7)!
      threshold: ALL
      timeZone: ${DROVE_TIMEZONE}
      logFormat: "%(%-5level) [%date] [%logger{0} - %X{appId}] %message%n"
    - type: file #(8)!
      threshold: ALL
      timeZone: ${DROVE_TIMEZONE}
      currentLogFilename: /logs/drove-controller.log
      archivedLogFilenamePattern: /logs/drove-controller.log-%d-%i
      archivedFileCount: 3
      maxFileSize: 100MiB
      logFormat: "%(%-5level) [%date] [%logger{0} - %X{appId}] %message%n"
      archive: true


zookeeper: #(9)!
  connectionString: ${ZK_CONNECTION_STRING}

clusterAuth: #(10)!
  secrets:
  - nodeType: CONTROLLER
    secret: ${DROVE_CONTROLLER_SECRET}
  - nodeType: EXECUTOR
    secret: ${DROVE_EXECUTOR_SECRET}

userAuth: #(11)!
  enabled: true
  users:
    - username: admin
      password: ${DROVE_ADMIN_PASSWORD}
      role: EXTERNAL_READ_WRITE
    - username: guest
      password: ${DROVE_GUEST_PASSWORD}
      role: EXTERNAL_READ_ONLY

instanceAuth: #(12)!
  secret: ${DROVE_INSTANCE_AUTH_SECRET}

options: #(13)!
  maxStaleInstancesCount: 3
  staleCheckInterval: 1m
  staleAppAge: 1d
  staleInstanceAge: 18h
  staleTaskAge: 1d
  clusterOpParallelism: 4

1. Server listener configuration. See Dropwizard Server Configuration for the different options.
2. Main port configuration. This is where the UI and APIs will be exposed. Check connector configuration docs for details.
3. Admin port. You can take thread dumps, metrics, run healthchecks on the Drove controller on this port.
4. Base path for UI. Keep this as is.
5. Access logs configuration. See requestLog docs.
6. Main logging configuration. See logging docs.
7. Log to console. Useful in docker-compose.
8. Log to rotating files. Useful for running servers.
9. Configure how to connect to Zookeeper See Zookeeper Config for details.
10. Configuration for authentication between nodes in the cluster. Please check intra node auth config for details.
11. Configure user authentication to access the cluster. Please check User auth config for details.
12. Signing secret for JWT to be embedded in application and task instances. Check Instance auth config for details.
13. Special options to configure controller behaviour. See Controller Options for details.

adminConnectors:
  - type: http
    port: 10001
    bindHost: 127.0.0.1

Zookeeper Connection Configuration¶

The following details can be configured.

Sample

zookeeper:
  connectionString: "192.168.3.10:2181,192.168.3.11:2181,192.168.3.12:2181"
  namespace: drovetest

Intra Node Authentication Configuration¶

Communication between controller and executor is protected by a shared-secret based authentication. The following configuration is meant to configure this. This section consists of a list of 2 members:

• Config for controller to talk to executors
• Config for executors to talk to controller

Each section consists of the following:

Sample

clusterAuth:
  secrets:
  - nodeType: CONTROLLER
    secret: ControllerSecretValue
  - nodeType: EXECUTOR
    secret: ExecutorSecret

User Authentication Configuration¶

This section is used to configure user details for human and other systems that need to call Drove APIs or access the Drove UI. This is implemented using basic auth.

The configuration consists of:

Each entry in the user list consists of:

Sample

userAuth:
  enabled: true
  encoding: CRYPT
  users:
    - username: admin
      password: "$2y$10$pfGnPkYrJEGzasvVNPjRu.IJldV9TDa0Vh.u1UdimILWDuhvapc2O"
      role: EXTERNAL_READ_WRITE
    - username: guest
      password: "$2y$10$uCJ7WxIvd13C.1oOTs28p.xpJShGiTWuDLY/sGH9JE8nrkSGBFkc6"
      role: EXTERNAL_READ_ONLY
    - username: noread
      password: "$2y$10$8mr/zXL5rMW/s/jlBcgXHu0UvyzfdDDvyc.etfuoR.991sn9UOX/K"

htpasswd -nbBC 10 <username> <password>|cut -d ':' -f2

Instance Authentication Configuration¶

All application and task instances, get access to an unique JWT that is injected into it by Drove as the environment variable DROVE_APP_INSTANCE_AUTH_TOKEN. This token is signed using a secret. This secret can be configured by setting the secret parameter in the instanceAuth section.

Sample

instanceAuth:
  secret: RandomSecret

Controller Options¶

The following options can be set to influence the behavior of the Drove cluster and the controller.

Sample

options:
  staleCheckInterval: 5m
  staleAppAge: 2d
  maxStaleInstancesCount: 20
  staleInstanceAge: 1d
  staleTaskAge: 2d
  maxEventsStorageDuration: 30m
  clusterOpParallelism: 32
  allowedMountDirs:
   - /mnt/scratch

Stale data cleanup¶

In order to keep internal memory footprint low, reduce the amount of data stored on Zookeeper, and provide a faster experience on the UI,Drove keeps cleaning up data for stale applications, application instances, task instances and cluster events.

The retention for such metadata can be controlled using the following config options:

• staleAppAge
• maxStaleInstancesCount
• staleInstanceAge
• staleTaskAge
• maxEventsStorageDuration

Internal Operations¶

Drove may need to create and issue operations on applications and tasks to manage cluster stability, for maintenance and other reasons. The following parameters can be used to control the speed and parallelism of such operations:

• clusterOpTimeout
• clusterOpParallelism

Security related options¶

The auditedHttpMethods parameter contains a list of all HTTP methods that need to be audited. This means that if the auditedHttpMethods contains POST and PUT, any drove HTTP POST or PUT apis being called will lead to a audit in the controller logs with the details of the user that made the call.

The allowedMountDirs option whitelists only some directories to be mounted on containers. If this is not provided, containers will be able to mount any directory on the executors.

Security policies¶

How security profiles work is explained in Security Profiles.

The securityPolicies top-level key defines container security profiles per group of deployables. Each entry has a match rule list and a security profile. For every new instance, the controller finds the first policy that matches and applies its profile. If no policy matches, the controller default profile built from the allowedMountDirs, disableCmdlArgs and enableRawDeviceAccess options is used.

Each match rule is a string of the form (jobtype|*)/(*|name regex):

The name used for matching depends on the deployable type:

Sample

securityPolicies:
  - match:
      - "COMPUTATION/.*"
    profile:
      policyName: "tasks-locked-down"
      disableCmdlArgs: true
      maxOpenFiles: 100000
  - match:
      - "SERVICE/ml-training"
    profile:
      policyName: "ml-privileged"
      enablePrivilegedMode: true
      capabilities:
        addedCapabilities:
          - "SYS_ADMIN"
  - match:
      - "*/*"
    profile:
      policyName: "default"
      allowedMountDirs:
        - /mnt/scratch

Relevant directories¶

• /etc/drove/controller/ - Configuration files
• /var/log/drove/controller/ - Logs

We shall be volume mounting the config and log directories with the same name.

Setup the config file¶

Create a relevant configuration file in /etc/drove/controller/controller.yml.

Sample

server:
  applicationConnectors:
    - type: http
      port: 10000
  adminConnectors:
    - type: http
      port: 10001
  requestLog:
    appenders:
      - type: file
        timeZone: IST
        currentLogFilename: /var/log/drove/controller/drove-controller-access.log
        archivedLogFilenamePattern: /var/log/drove/controller/drove-controller-access.log-%d-%i
        archivedFileCount: 3
        maxFileSize: 100MiB

logging:
  level: INFO
  loggers:
    com.phonepe.drove: INFO


  appenders:
    - type: file
      threshold: ALL
      timeZone: IST
      currentLogFilename: /var/log/drove/controller/drove-controller.log
      archivedLogFilenamePattern: /var/log/drove/controller/drove-controller.log-%d-%i
      archivedFileCount: 3
      maxFileSize: 100MiB
      logFormat: "%(%-5level) [%date] [%logger{0} - %X{appId}] %message%n"

zookeeper:
  connectionString: "192.168.56.10:2181"

clusterAuth:
  secrets:
  - nodeType: CONTROLLER
    secret: "0v8XvJrDc7r86ZY1QCByPTDPninI4Xii"
  - nodeType: EXECUTOR
    secret: "pOd9sIEXhv0wrGOVc7ebwNvR7twZqyTN"

userAuth:
  enabled: true
  encoding: CRYPT
  users:
    - username: admin
      password: "$2y$10$pfGnPkYrJEGzasvVNPjRu.IJldV9TDa0Vh.u1UdimILWDuhvapc2O"
      role: EXTERNAL_READ_WRITE
    - username: guest
      password: "$2y$10$uCJ7WxIvd13C.1oOTs28p.xpJShGiTWuDLY/sGH9JE8nrkSGBFkc6"
      role: EXTERNAL_READ_ONLY


instanceAuth:
  secret: "bd2SIgz9OMPG2L8wA6zxj21oLVLbuLFC"

options:
  maxStaleInstancesCount: 3
  staleCheckInterval: 1m
  staleAppAge: 2d
  staleInstanceAge: 1d
  staleTaskAge: 1d
  clusterOpParallelism: 4
  allowedMountDirs:
   - /dev/null

Setup required environment variables¶

Environment variables need to run the drove controller are setup in /etc/drove/controller/controller.env.

CONFIG_FILE_PATH=/etc/drove/controller/controller.yml
JAVA_PROCESS_MIN_HEAP=2g
JAVA_PROCESS_MAX_HEAP=2g
ZK_CONNECTION_STRING="192.168.3.10:2181"
CONTROLLER_SECRET="0v8XvJrDc7r86ZY1QCByPTDPninI4Xii"
EXECUTOR_SECRET="pOd9sIEXhv0wrGOVc7ebwNvR7twZqyTN"
ADMIN_PASSWORD="<admin user password>"
GUEST_PASSWORD="<guest user password>"
INSTANCE_AUTH_SECRET="bd2SIgz9OMPG2L8wA6zxj21oLVLbuLFC"
JAVA_OPTS="-Xlog:gc:/var/log/drove/controller/gc.log -Xlog:gc:::filecount=3,filesize=10M -Xlog:gc::time,level,tags -XX:+UseNUMA -XX:+ExitOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -Dfile.encoding=utf-8 -Djute.maxbuffer=0x9fffff"

Container environment variables¶

The controller container entrypoint reads the following environment variables:

Image hardening and supply chain¶

The controller and executor images are built from a minimal ubuntu:noble base with a headless JRE. The build installs only the required packages, removes documentation, locale data and setuid binaries, and uses COPY instead of ADD for image contents.

Every release image pushed to the registry by the release workflow ships with two attestations:

• SBOM - the complete list of packages inside the image.
• Provenance - a SLSA attestation that records how the image was built, from which source revision, and by which workflow.

You can inspect the attestations for a released image with docker buildx:

docker buildx imagetools inspect ghcr.io/phonepe/drove-controller:latest

The output shows the image manifest along with the sbom and provenance attestation manifests.

If you use a tool like cosign, you can verify the attestations against the image digest:

cosign verify-attestation \
    --type slsaprovenance \
    ghcr.io/phonepe/drove-controller@<digest>
    ghcr.io/phonepe/drove-controller@<digest>

Create systemd file¶

[Unit]
Description=Drove Controller Service
After=docker.service
Requires=docker.service

[Service]
User=drove
Group=docker
TimeoutStartSec=0
Restart=always
ExecStartPre=-/usr/bin/docker pull ghcr.io/phonepe/drove-controller:latest
ExecStart=/usr/bin/docker run  \
    --env-file /etc/drove/controller/controller.env \
    --volume /etc/drove/controller:/etc/drove/controller:ro \
    --volume /var/log/drove/controller:/var/log/drove/controller \
    --publish 10000:10000  \
    --publish 10001:10001 \
    --hostname %H \
    --rm \
    --name drove.controller \
    ghcr.io/phonepe/drove-controller:latest

[Install]
WantedBy=multi-user.target

Verify the file with the following command:

systemd-analyze verify drove.controller.service

Set permissions

chmod 664 /etc/systemd/system/drove.controller.service

Start the service on all servers¶

Use the following to start the service:

systemctl daemon-reload
systemctl enable drove.controller
systemctl start drove.controller

You can tail the logs at /var/log/drove/controller/drove-controller.log.

The console would be available at http://<ip>:10000 and admin functionality will be available on http://<ip>:10001 according to the above config.

Health checks can be performed by running a curl as follows:

curl http://localhost:10001/healthcheck

Once controllers are up, one of them will become the leader. You can check the leader by running the following command:

curl http://<ip>:10000/apis/v1/ping

Only on the leader you should get the following response along with a HTTP status 200/OK:

{
    "status":"SUCCESS",
    "data":"pong",
    "message":"success"
}