The cxpm command-line tool

cxpm fetch | configure | build | test | pack | install | uninstall   [options] [directory]
cxpm project create <name> [--type T] [--tests] [--example]
cxpm search | login | logout | publish | serve                      (the package index)
cxpm generate <kind> [directory]
cxpm help [command]

directory is the project, . by default. Exit status is 0 on success, 1 when the work failed (a compile error, a failing test, a missing dependency) and 2 when the command line is wrong.

The lifecycle

Each stage first runs the ones before it, and each of those does nothing when its result is still valid: cxpm install on a fresh clone fetches, configures, builds, packs and installs; cxpm build right after it compiles nothing. --reconfigure (-r) resolves everything again, --force (-f) also rebuilds what is up to date.

cxpm test builds the project, then builds each project the manifest lists in tests and runs its executable from that project’s directory. It fails if any of them fails.

cxpm uninstall removes what cxpm install installed, through the same package manager.

Options common to the stages that build:

The manifest: package.cpp or package.json

A manifest describes one project. A project has at most one target.

#include <CXPM/Project.hpp>
#include <CXPM/Target.hpp>

using namespace CXPM;

auto target = Target()
                  .name_set("greeter")
                  .version_set("0.1.0")
                  .description_set("Says hello")
                  .type_set("shared-library")
                  .sources_append({"src/*.cpp"})
                  .include_directories_append({"include"})
                  .options_append({CompilerOptions::StandardCxx23,
                                   CompilerOptions::WarningsAll})
                  .link_libraries_append({"m"})
                  .dependencies_append({"zlib >= 1.2"})
                  .create();

auto project = Project()
                   .target_set(target)
                   .members_append({"example"})
                   .tests_append({"tests"})
                   .create();

The same project in package.json, which is read without compiling or running anything:

{
  "schema": 3,
  "target": {
    "name": "greeter",
    "version": "0.1.0",
    "type": "shared-library",
    "sources": ["src/*.cpp"],
    "include_directories": ["include"],
    "options": ["standard-c++23", "warnings-all"],
    "link_libraries": ["m"],
    "dependencies": ["zlib >= 1.2"]
  },
  "members": ["example"],
  "tests": ["tests"]
}

package.cpp wins when a directory has both. cxpm generate package-cpp|package-json writes a starting point; cxpm project create writes a whole project.

Target types

Every type also installs lib/pkgconfig/<name>.pc, which is how a dependent project finds it. The headers of a library are those under its include_directories.

A data target ships files as they are. cxpm’s own toolchain modules are packaged this way:

auto target = Target()
                  .name_set("cxpm-toolchain-g++")
                  .type_set("data")
                  .data_append("src/toolchain.cpp", "share/cxpm/toolchains/g++")
                  .create();

source is a file or a pattern relative to the project; destination is a directory relative to the install prefix.

Sources

sources entries are paths relative to the project or patterns: src/*.cpp matches in one directory, src/**/*.cpp in every directory below. A pattern that matches nothing is an error.

Options

An option is a neutral name that each toolchain translates, or a raw flag.

CompilerOptions:: names: StandardC11, StandardC17, StandardCxx17, StandardCxx20, StandardCxx23, WarningsAll, WarningsExtra, WarningsAsErrors, OptimizeNone, OptimizeSize, OptimizeSpeed, OptimizeMaximum, DebugInformation, PositionIndependentCode, PositionIndependentExecutable, StackProtectorStrong, StackProtectorAll, Threads, LinkTimeOptimization, AddressSanitizer, UndefinedBehaviorSanitizer, Coverage. In JSON they are spelled as in the example above ("standard-c++23").

Anything that starts with - is passed to the compiler and the linker as written, and ties the manifest to toolchains that understand it. A name that is neither stops the build with the list of names the toolchain knows.

Dependencies

.dependencies_append({"zlib", "fmt >= 10"})

A name refers to, whichever exists:

• a project of the workspace (a member of the same root manifest): built first and installed into .cxpm/sysroot;
• a fetched dependency: a git repository with a manifest of its own, cloned by cxpm fetch, then built and installed into .cxpm/sysroot like a member. It is fetched either because the manifest names its repository (below), or because the package index publishes the name;
• an installed package, found by pkg-config in the sysroot.

#include <CXPM/Dependency.hpp>

.dependencies_append(Dependency()
                         .name_set("greeter")
                         .constraint_set(">= 0.1")
                         .repository_set("https://example.org/greeter.git")
                         .revision_set("v0.1.0"))

revision is a branch, a tag or a commit. .cxpm/lock.json records the commit that was fetched, and later fetches check out that commit until the manifest asks for another revision.

A dependency that cannot be resolved, or whose version does not satisfy the constraint, fails configure with the name and what was found. link_libraries is only for libraries of the system that have no pkg-config file, such as m or dl.

Workspaces

A manifest lists other projects by directory in members (built with it) and tests (built and run by cxpm test). A root manifest may have no target at all and only tie projects together; cxpm’s own is one:

auto project = Project()
                   .members_append({"third-party", "libraries/cxpm-interface",
                                    "applications/cxpm"})
                   .tests_append({"tests/unit", "tests/cli"})
                   .create();

Projects are built in dependency order; a cycle is an error that names the projects in it. The workspace has one .cxpm/sysroot, at the root, and each project builds under its own .cxpm/build.

The package index

The index maps a package name and version to a commit of a git repository. It holds no code: cxpm fetch clones the repository.

Using packages

.dependencies_append({"greet >= 0.1"})

cxpm fetch asks the index for each name that is not a project of the workspace. A name the index does not know is left to the sysroot. For one it knows, fetch picks the newest version that every constraint in the workspace accepts — the constraints of fetched packages included — clones it into .cxpm/sources/<name>, and records the version and its commit in .cxpm/lock.json:

{
  "name": "greet",
  "repository": "https://example.org/greet.git",
  "revision": "ba5b12562248f856f2d8bda5cea758df42059e91",
  "commit": "ba5b12562248f856f2d8bda5cea758df42059e91",
  "version": "0.1.0",
  "index": "https://01.dev.soft-forge.dev/cxpm/index"
}

• Only cxpm fetch contacts the index. Every other command works from the lock file, so a build needs no network once the sources are there.
• A locked version is kept until a constraint stops accepting it. To move to a newer one, tighten the constraint (or delete the entry from the lock) and run cxpm fetch.
• The commit is the checksum. If the index later reports another commit for a locked version, fetch fails and names both.
• When two projects need versions that exclude each other, fetch fails listing who needs what.
• If the index cannot be reached, fetch warns, keeps what is locked, and leaves other names to the sysroot.

cxpm dependency edits the manifest instead of you:

cxpm dependency add greet                 # the newest version the index publishes, or a later one
cxpm dependency add "greet@>=0.1"         # a constraint of your own
cxpm dependency add https://example.org/json.git#v3 --name json
cxpm dependency list
cxpm dependency remove greet
cxpm dependency sync                      # fetch what the manifest names

add asks the index for the name, refuses a constraint no published version satisfies, writes the dependency into package.cpp or package.json, and fetches it (--no-sync leaves that to you). It checks that the manifest still reads the same afterwards and puts the file back when it does not; a package.cpp that builds its dependencies in some other way than .dependencies_append(...) has to be edited by hand.

The index is --index URL, else CXPM_INDEX, else https://01.dev.soft-forge.dev/cxpm/index. off means no index. An https:// index is reached through curl, which must be installed; an http:// one needs nothing.

Publishing

cxpm login --user ada          # once per index: asks for the password, keeps a token
cxpm publish                   # in the project, at the top of its git repository
cxpm search greet

An index with accounts (cx-pm-web-index ) takes login --user NAME: cxpm asks for the password without showing it (or reads it from the first line of standard input when that is not a terminal), trades it for a token of its own and keeps only the token. An index run by cxpm serve has no passwords: its operator gives you a token, for cxpm login --token <token>.

In a workspace, cxpm publish --package NAME publishes one of its projects; dependencies that are other projects of the same workspace are left out of what is published. cxpm whoami names the account the stored token belongs to.

publish sends the target’s name, version, description and dependencies, the address of the origin remote, the project’s README when there is one, and the commit of HEAD (or --revision REV). The commit must already be on the remote. --repository URL publishes another address than origin.

The first account to publish a name owns it, and a published version never changes: to fix something, publish a new version.

login stores the token in ~/.config/cxpm/credentials.json, readable by you alone. CXPM_TOKEN is used instead when set, which is what a CI job wants. logout forgets the stored token, and asks the index to revoke it where the index can (POST /-/logout).

Serving an index

cxpm serve --database-path /var/cache/soft-forge/cxpm --add-account ada    # prints ada's token once
cxpm serve --database-path /var/cache/soft-forge/cxpm \
           --listen unix:///run/cxpm/socket.sock \
           --host https://01.dev.soft-forge.dev/cxpm/index

cxpm serves plain HTTP. Put a reverse proxy in front of it for TLS and point it at the socket. The index keeps only a hash of each token, so a lost token is replaced with --add-account, not recovered.

An index with accounts

cxpm serve knows the accounts its operator made, one token each. cx-pm-web-index, a separate application in its own repository, is the same index with people in front of it, as npm has them: anyone signs up on its page with a name, an email and a password, signs in, makes and revokes tokens, and sees the packages and their versions in a browser. It is built on cxpm’s own PackageIndex, so fetch, search and publish talk to it exactly as they talk to cxpm serve, and its data directory keeps the packages in the same files.

What it adds to the requests above, for cxpm:

Its README describes the rest: the settings, the page, the account requests and how to deploy it behind cx-networking-http-proxy.

The session: cxpm repl

cxpm with no arguments, at a terminal, opens a session; cxpm repl (or cxpm shell) does the same. Every line is a cxpm command without the program’s name:

cxpm demo> search greet
cxpm demo> dependency add greet
cxpm demo> build
cxpm demo> publish

The session adds cd, pwd, status (the directory, its manifest, the index and the account logged in to it), clear, help [command] and exit. Tab completes commands, options and paths. A command runs in a process of its own, so Ctrl-C stops the command and not the session. The prompt names the directory, in green when it holds a manifest, and shows the status of the last command when it failed.

History is kept in $XDG_STATE_HOME/cxpm/shell_history (CXPM_SHELL_HISTORY names another file; empty for none). Lines that carry --token or --password, and lines that start with a space, are not written to it.

When standard input is not a terminal, cxpm repl reads one command per line and exits with the status of the last one, so cxpm repl < script runs a script.

Toolchains

A toolchain says how to drive a compiler: executables, flag spellings, file names, the sysroot it builds against and the package manager of that sysroot. It is a toolchain.cpp or a toolchain.json:

#include <CXPM/Toolchain.hpp>

CXPM::ToolchainDescriptor toolchain =
    CXPM::Toolchain()
        .name_set("al2023")
        .language_set("c++")
        .compiler_executable_set("/usr/bin/g++")
        .linker_executable_set("/usr/bin/g++")
        .archiver_executable_set("/usr/bin/ar")
        .sysroot_set("/opt/sysroots/amazonlinux-2023")
        .package_manager_set("rpm")
        .option_mappings_set(CXPM::CompilerOptions::Gnu);

cxpm ships g++, gcc, clang++, clang and nvcc, installed as source in <prefix>/share/cxpm/toolchains/<name>/toolchain.cpp. It also looks in ~/.local/share/cxpm/toolchains and in the directories of cxpm_BUILD_EXTRA_MODULES_PATH. A target picks one with toolchain_set("name"), --toolchain overrides it, and otherwise the first toolchain for the target’s language is used.

Sysroots

Dependencies that cxpm builds go into the project’s own .cxpm/sysroot. Everything else comes from the toolchain’s sysroot: / by default, which is your system.

With another sysroot (sysroot_set or --sysroot DIR), the compiler and the linker get --sysroot, pkg-config only sees that sysroot’s packages, and nothing from the host leaks into the build. That is how a project is built for another distribution from one machine: unpack that distribution’s root filesystem with its development packages, and point a toolchain at it.

Packages

cxpm pack writes one package per project into .cxpm/packages/:

Native packages are written by cxpm itself; makepkg, rpmbuild and dpkg-deb are not needed to pack, only the package manager to install. The dependencies of the manifest become the dependencies of the package. Installing into a system root usually needs root privileges.

Confinement

A package.cpp or toolchain.cpp is code, and so is every build step. When bubblewrap is installed, cxpm runs them in a namespace of their own:

• compiling a manifest, loading it, and every compile, link and archive command see the project, its workspace, the sysroot and the system’s own directories (/usr, /bin, /lib*, /etc, /opt), read-only except for the directories they write to;
• they have no network, and do not see your home directory or anything else on the machine.

A compiled manifest is also never loaded into cxpm itself: the separate cxpm-descriptor-sandbox process loads it, under CPU, memory and time limits, and hands the description back as JSON.

CXPM_CONFINEMENT_READABLE is a :-separated list of extra directories to show read-only, for a compiler installed outside the system directories.

Not confined: git (and curl, for an https:// index) during fetch, which need the network; the test executables cxpm test runs; and the package manager during install.

Environment

What belongs in version control

Commit the manifests and, if you want pinned dependencies, .cxpm/lock.json. Ignore the rest of .cxpm/.