FTN16: FutoIn - Continuous Integration & Delivery Tool
Version: 1.2
Date: 2018-04-25
Copyright: 2015-2018 FutoIn Project (http://futoin.org)
Authors: Andrey Galkin


1. Intro

There are many continuous integration & delivery tools, but they are primarily targeted at own infrastructure. The demand for a new meta-tool is to merge many operations of different technologies like npm, composer, bundle, nvm, rvm, php-build and others under a single tool for runtime setup, project development, build, deployment and running.

NOTE: current primary focus is on web projects, but other cases like Puppet modules are supported.

2. Concept

A command line tool must be available through shell search path or called via absolute path. The command name is "cid" - Continuous Integration & Delivery Tool.

"cid" must work on existing projects with no modifications, but with extra parameters required. If special "futoin.json" configuration file is present in project root then extra parameters' default values should be taken from the configuration file.

It should be possible to specify any of the actions manually through configuration file. Otherwise, the tool should auto-detect action implementation.

The tool should support the following actions:

2.1. Tag

A standard procedure for updating for release and tagging source code.

2.2. Preparation

A standard procedure for project development or release is source checkout, dependency checkout and workspace configuration.

For build tools, working copy clean up is expected.

2.3. Building

A standard procedure for detecting available build systems and executing them in predefined order. Binary artifact may be a product of such action.

2.4. Checking

A standard procedure for detecting available test systems and executing them in predefined order.

2.5. Packaging

A standard procedure for detecting packaging method to create a single binary artifact for deployment.

2.6. Promotion

A standard procedure for promoting a binary package into predefined release management systems (RMS).

Suggested name conventions: * Build -> CIBuilds * Build -> ReleaseBuilds -> ProductionBuilds * Build -> {Arbitrary} * {Arbitrary} -> {Arbitrary} [-> {Arbitrary}]+


Note: promotion from pool to pool must not modify binary artifacts. Otherwise, a separate project must exist, which uses original project binary artifact as dependency for input and own binary artifact promotion chains

2.7. Deployment

The primary focus of the action is for setup of web projects. The process should properly check requirements, setup file permissions, manage persistent data, manage configuration and support rolling deployment with no service interruption.

2.8. Running

This action should focused on execution in development and testing process and may not be implemented at all, if not applicable.

2.9. Integration into provisioning systems

In multi-tenant environments, it may not be desired to give application user full control over system. However, some tools may require privileged administrator access for installation and update. Also, it's not desired to have multiple copies of same tool, but secure sharing is required.

For such cases, an external to CID provisioning system may set external setup callback command to be used instead of CID itself for tool setup. The external command is responsible for proper filtering.

For purpose own easy integration, target deploy dir must have auto-generated ".futoin.merged.json" which should include all data from project, deployment, user, global and runtime. The file must not be manually editable and should be overwitten on each deployment. CID itself must never consult to it.

As some tools and services may be configured by provisioning system, there are .env.externalSetup and .env.externalServices configuration options available.

3. Detailed business logic definition

3.1. Configuration file

3.1.1. JSON tree definition in dot notation.

The same identifiers should be used in command line options. All configuration nodes are optional and auto-detectable in most cases.

Note: this tree represents actual state CID works with. All internal API either work with full configuration root or only with its .env part. There should be no other configuration data. Project configuration Environment configuration Deployment configuration Runtime configuration (available to plugins) Python-based CID implementation notes

  1. .plugins expects to fully qualified module named with {tool}Tool class.
  2. .pluginPacks expect fully qualified module name with submodules in {tool}tool.py format having {tool}Tool class

3.1.2. Process environment

Each tool may have a whitelist of related environment variables for .env sections. This variables may be passed through process environment as well. Example:

    rubyVer=2.3 cid tool install ruby
    rubyVer=2.3 cid tool exec ruby -- ruby-specific-args

3.1.3. Working Directory notes

By default working directory (--wcDir) is absolute path of current working directory, except:

  1. If ci_build command is used with --vcsRepo then absolute path of "./ci_build_{vcs_ref}" is used
  2. If ci_build command is used without --wcsRepo then:

In all cases working directory must be either empty or contain project without conflicts with --vcsRepo parameter, if supplied.

In case of ci_build, it is required to make a clean checkout. Therefore, existing wcDir must be removed (renamed for safety).

3.1.4. Resource distribution requirements

The complexity comes from these facts:


  1. Minimal and maximal memory requirements per service instance must be obeyed.
  2. Deployment must fail, if minimal memory requirements are not satisfied.
  3. All available for deployment memory must be allocated to instances based their weights.
  4. If one instance is unable to utilize multiple CPU cores then additional instances must be added until it's feasible to distribute available memory.
  5. If one instance is unable to seamlessly reload then at least one more instance must be added for rolling updates.
  6. If app is marked as not scalable then no more than one instance is allowed.

3.2. Commands

Prior to each command run:

Standard parameter processing:

3.2.1. cid tag <branch> [<next_version>] [--vcsRepo=<vcs:url>] [--wcDir wc_dir]


Versioning notes

It is required that version consists of three components: major, minor and patch. Generally, SEMVER is assumed.

3.2.2. cid prepare [<vcs_ref>] [--vcsRepo=<vcs:url>] [--wcDir wc_dir]


3.2.3. cid build [--debug]


3.2.4. cid package

Default: Package name convention:

3.2.5. cid check [--permissive]

3.2.6. cid promote <rms_pool> <packages> [--rmsRepo=<rms:url>]


3.2.7 cid deploy <deploy_type> ...

Generic options: cid deploy rms <rms_pool> [<package>] [--rmsRepo=<rms:url>] [--build] [--redeploy]

Default: cid deploy vcstag [<vcs_ref>] [--vcsRepo=<vcs:url>] [--redeploy]

Default: cid deploy vcsref <vcs_ref> [--vcsRepo=<vcs:url>] [--redeploy]

Default: cid deploy setup

Setup deploy directory to allow futoin.json modifications without deployment being is done. cid deploy set action <name> <action>...

Override .action in deployment config. '@default' can be used to call project-defined handlers. cid deploy set persistent <path>...

Add .persistent paths in deployment config. Duplicate entries are automatically merged. cid deploy set writable <path>...

Add .writable paths in deployment config. Duplicate entries are automatically merged. cid deploy set entrypoint <name> <tool> <entry_path> [<tune>...]

Override .entryPoints configuration.

The tune argument is either inline JSON or space separated key=value pairs. cid deploy set env <variable> [<value>]...

Set or remove .env variable configuration. cid deploy set webcfg <variable> [<value>]...

Set or remove .webcfg variable configuration. cid deploy set webmount <web_path> [<json>]...

Set or remove .webcfg.mounts web path configuration. cid deploy set tools <tool>...

Override .tools configuration.

The tool parameter is either tool name ('*' as version is assumed) or tool=ver pair. cid deploy set tooltune <tool> <tune>...

Override .toolTune configuration of specified tool.

The tune argument is either inline JSON or space separated key=value pairs. cid deploy reset [<type]

Reset either all of cid deploy set configurations or the specified type. common deploy procedure deployment assumptions

  1. Each web application must have own deployment root folder
  2. Each web application should have own user
  3. Each web application should get proper ownership and read-only permissions
  4. Application package must not have modifiable content
  5. Each read-write path should get symlink to {.env.persistentDir}/{path} and survive across deployments
  6. .action.migrate must run and successfully complete
  7. ${.deployDir}/current must always point to fully configured deployment
  8. For security reasons it is not possible to include project-specific config for web server running as root user. Also, sensitive data like TLS private keys must not be available to application user. Therefore a performance penalty of reverse proxy may apply, but large high available deployments should have load balancer/reverse proxy any way.
  9. Lock file must be acquired during deployment procedure.
  10. Not empty {.deploDir} must contain deploy lock file for safety reasons.

3.2.8. cid run [<command> [-- <command_args..>]]

Primary case is execution in development and testing environment.

3.2.9. cid ci_build <vcs_ref> [<rms_pool>] [--vcsRepo=<vcs:url>] [--rmsRepo=<rms:url>] [--permissive] [--debug] [--wcDir wc_dir]


3.2.10. cid tool <action> [<tool_name> [<version>] -- optional args]

Tools actions:

3.2.11. cid init [<project_name>] [--vcsRepo=<vcs:url>] [--rmsRepo=<rms:url>] [--permissive]

Initialize futoin.json. Automatically add already known information to it.

If project name is not provided and not auto-detected then use working copy folder basename.

3.2.12. cid migrate

Runs data migration tasks.

Provided for overriding default procedures in scope of deployment procedure.

3.2.13. cid vcs <action> [optional args]

These are helpers for CI environment and should not be used by developer in regular activities.

3.2.14. cid rms <action> [optional args]

This helpers help automate RMS operation neutral way.

3.2.15. cid service ...

Interface for Continuous Deployment execution control. It is expected to call this commands from systemd, sysv-init or other system daemon control functionality.

In case containers like Docker is used then there is a separate helper command to be used as entry point:

In case there is no deployment, but service run is desired in development process:

3.2.16. cid build-dep <name>...

Install development files for specific build dependency. The list of dependencies is implementation-defined.

If build dependency matches supported tool then full initialization of .config.env is performed.