+

Search Tips   |   Advanced Search

Developing collections

Collections are a distribution format for Ansible content. You can use collections to package and distribute playbooks, roles, modules, and plugins. You can publish and use collections through Ansible Galaxy.


Collection structure

Collections follow a simple data structure. None of the directories are required unless you have specific content that belongs in one of them. A collection does require a galaxy.yml file at the root level of the collection. This file contains all of the metadata that Galaxy and other tools need in order to package, build and publish the collection:


galaxy.yml

A collection must have a galaxy.yml file that contains the necessary information to build a collection artifact. See Collection Galaxy metadata structure for details.


docs directory

Put general documentation for the collection here. Keep the specific documentation for plugins and modules embedded as Python docstrings. Use the docs folder to describe how to use the roles and plugins the collection provides, role requirements, and so on. Use markdown and do not add subfolders.

Use ansible-doc to view documentation for plugins inside a collection:

The ansible-doc command requires the fully qualified collection name (FQCN) to display specific plugin documentation. In this example, my_namespace is the Galaxy namespace and my_collection is the collection name within that namespace.

The Galaxy namespace of an Ansible collection is defined in the galaxy.yml file. It can be different from the GitHub organization or repository name.


plugins directory

Add a 'per plugin type' specific subdirectory here, including module_utils which is usable not only by modules, but by most plugins by using their FQCN. This is a way to distribute modules, lookups, filters, and so on without having to import a role in every play.

Vars plugins are unsupported in collections. Cache plugins may be used in collections for fact caching, but are not supported for inventory plugins.

module_utils

When coding with module_utils in a collection, the Python import statement needs to take into account the FQCN along with the ansible_collections convention. The resulting Python import will look like from ansible_collections.{namespace}.{collection}.plugins.module_utils.{util} import {something}

The following example snippets show a Python and PowerShell module using both default Ansible module_utils and those provided by a collection. In this example the namespace is community, the collection is test_collection. In the Python example the module_util in question is called qradar such that the FQCN is community.test_collection.plugins.module_utils.qradar:

Note that importing something from an __init__.py file requires using the file name:

In the PowerShell example the module_util in question is called hyperv such that the FCQN is community.test_collection.plugins.module_utils.hyperv:


roles directory

Collection roles are mostly the same as existing roles, but with a couple of limitations:

  • Role names are now limited to contain only lowercase alphanumeric characters, plus _ and start with an alpha character.

  • Roles in a collection cannot contain plugins any more. Plugins must live in the collection plugins directory tree. Each plugin is accessible to all roles in the collection.

The directory name of the role is used as the role name. Therefore, the directory name must comply with the above role name rules. The collection import into Galaxy will fail if a role name does not comply with these rules.

You can migrate 'traditional roles' into a collection but they must follow the rules above. You may need to rename roles if they don't conform. You will have to move or link any role-based plugins to the collection specific directories.

For roles imported into Galaxy directly from a GitHub repository, setting the role_name value in the role's metadata overrides the role name used by Galaxy. For collections, that value is ignored. When importing a collection, Galaxy uses the role directory as the name of the role and ignores the role_name metadata value.


playbooks directory

TBD.


tests directory

Ansible Collections are tested much like Ansible itself, by using the ansible-test utility which is released as part of Ansible, version 2.9.0 and newer. Because Ansible Collections are tested using the same tooling as Ansible itself, via ansible-test, all Ansible developer documentation for testing is applicable for authoring Collections Tests with one key concept to keep in mind.

See Testing collections for specific information on how to test collections with ansible-test.

When reading the Testing Ansible documentation, there will be content that applies to running Ansible from source code via a git clone, which is typical of an Ansible developer. However, it's not always typical for an Ansible Collection author to be running Ansible from source but instead from a stable release, and to create Collections it is not necessary to run Ansible from source. Therefore, when references of dealing with ansible-test binary paths, command completion, or environment variables are presented throughout the Testing Ansible documentation; keep in mind that it is not needed for Ansible Collection Testing because the act of installing the stable release of Ansible containing ansible-test is expected to setup those things for you.


meta directory

A collection can store some additional metadata in a runtime.yml file in the collection's meta directory. The runtime.yml file supports the top level keys:


Creating a collection skeleton

To start a new collection:

Both the namespace and collection names use the same strict set of requirements. See Galaxy namespaces on the Galaxy docsite for those requirements.

Once the skeleton exists, you can populate the directories with the content you want inside the collection. See ansible-collections GitHub Org to get a better idea of what you can place inside a collection.


Creating collections

To create a collection:

  1. Create a collection skeleton with the collection init command. See Creating a collection skeleton above.

  2. Add your content to the collection.

  3. Build the collection into a collection artifact with ansible-galaxy collection build.

  4. Publish the collection artifact to Galaxy with ansible-galaxy collection publish.

A user can then install your collection on their systems.

Currently the ansible-galaxy collection command implements the following sub commands:

To learn more about the ansible-galaxy command-line tool, see the ansible-galaxy man page.


Using documentation fragments in collections

To include documentation fragments in your collection:

  1. Create the documentation fragment: plugins/doc_fragments/fragment_name.

  2. Refer to the documentation fragment with its FQCN.

Documentation fragments covers the basics for documentation fragments. The kubernetes collection includes a complete example.

You can also share documentation fragments across collections with the FQCN.


Building collections

To build a collection, run ansible-galaxy collection build from inside the root directory of the collection:

This creates a tarball of the built collection in the current directory which can be uploaded to Galaxy.:

This tarball is mainly intended to upload to Galaxy as a distribution method, but you can use it directly to install the collection on target systems.

Ignoring files and folders

By default the build step will include all the files in the collection directory in the final build artifact except for the following:

To exclude other files and folders when building the collection, you can set a list of file glob-like patterns in the build_ignore key in the collection's galaxy.yml file. These patterns use the following special characters for wildcard matching:

For example, if you wanted to exclude the sensitive folder within the playbooks folder as well any .tar.gz archives you can set the following in your galaxy.yml file:

This feature is only supported when running ansible-galaxy collection build with Ansible 2.10 or newer.


Trying collections locally

You can try your collection locally by installing it from the tarball. The following will enable an adjacent playbook to access the collection:

You should use one of the values configured in COLLECTIONS_PATHS for your path. This is also where Ansible itself will expect to find collections when attempting to use them. If you don't specify a path value, ansible-galaxy collection install installs the collection in the first path defined in COLLECTIONS_PATHS, which by default is ~/.ansible/collections.

Next, try using the local collection inside a playbook. For examples and more details see Using collections


Installing collections from a git repository

You can also test a version of your collection in development by installing it from a git repository.

You can install a collection in a git repository by providing the URI to the repository instead of a collection name or path to a tar.gz file. The collection must contain a galaxy.yml file, which will be used to generate the would-be collection artifact data from the directory. The URI should be prefixed with git+ (or with git@ to use a private repository with ssh authentication) and optionally supports a comma-separated git commit-ish version (for example, a commit or tag).

Warning

Embedding credentials into a git URI is not secure. Make sure to use safe auth options for security reasons. For example, use SSH, netrc or http.extraHeader/url.<base>.pushInsteadOf in Git config to prevent your creds from being exposed in logs.

In a requirements.yml file, you can also use the type and version keys in addition to using the git+repo,version syntax for the collection name.

Git repositories can be used for collection dependencies as well. This can be helpful for local development and testing but built/published artifacts should only have dependencies on other artifacts.


Default repository search locations

There are two paths searched in a repository for collections by default.

The first is the galaxy.yml file in the top level of the repository path. If the galaxy.yml file exists it's used as the collection metadata and the individual collection will be installed.

The second is a galaxy.yml file in each directory in the repository path (one level deep). In this scenario, each directory with a galaxy.yml is installed as a collection.


Specifying the location to search for collections

If you have a different repository structure or only want to install a subset of collections, you can add a fragment to the end of your URI (before the optional comma-separated version) to indicate which path ansible-galaxy should inspect for galaxy.yml file(s). The path should be a directory to a collection or multiple collections (rather than the path to a galaxy.yml file).


Publishing collections

You can publish collections to Galaxy using the ansible-galaxy collection publish command or the Galaxy UI itself. You need a namespace on Galaxy to upload your collection. See Galaxy namespaces on the Galaxy docsite for details.

Once you upload a version of a collection, you cannot delete or modify that version. Ensure that everything looks okay before you upload it.

Getting your API token

To upload your collection to Galaxy, you must first obtain an API token (--token in the ansible-galaxy CLI command or token in the ansible.cfg file under the galaxy_server section). The API token is a secret token used to protect your content.

To get your API token:

Storing or using your API token

Once you have retrieved your API token, you can store or use the token for collections in two ways:

Using the token argument

You can use the --token argument with the ansible-galaxy command (in conjunction with the --server argument or GALAXY_SERVER setting in your ansible.cfg file). You cannot use apt-key with any servers defined in your Galaxy server list.

Specify the token within a Galaxy server list

With this option, you configure one or more servers for Galaxy in your ansible.cfg file under the galaxy_server_list section. For each server, you also configure the token.

See Configuring the ansible-galaxy client for complete details.

Upload using ansible-galaxy

By default, ansible-galaxy uses https://galaxy.ansible.com as the Galaxy server (as listed in the ansible.cfg file under GALAXY_SERVER). If you are only publishing your collection to Ansible Galaxy, you do not need any further configuration. If you are using Red Hat Automation Hub or any other Galaxy server, see Configuring the ansible-galaxy client.

To upload the collection artifact with the ansible-galaxy command:

The above command assumes you have retrieved and stored your API token as part of a Galaxy server list. See Getting your API token for details.

The ansible-galaxy collection publish command triggers an import process, just as if you uploaded the collection through the Galaxy website. The command waits until the import process completes before reporting the status back. If you want to continue without waiting for the import result, use the --no-wait argument and manually look at the import progress in your My Imports page.

Upload a collection from the Galaxy website

To upload your collection artifact directly on Galaxy:

  1. Go to the My Content page, and click the Add Content button on one of your namespaces.

  2. From the Add Content dialogue, click Upload New Collection, and select the collection archive file from your local filesystem.

When uploading collections it doesn't matter which namespace you select. The collection will be uploaded to the namespace specified in the collection metadata in the galaxy.yml file. If you're not an owner of the namespace, the upload request will fail.

Once Galaxy uploads and accepts a collection, you will be redirected to the My Imports page, which displays output from the import process, including any errors or warnings about the metadata and content contained in the collection.


Collection versions

Once you upload a version of a collection, you cannot delete or modify that version. Ensure that everything looks okay before uploading. The only way to change a collection is to release a new version. The latest version of a collection (by highest version number) will be the version displayed everywhere in Galaxy; however, users will still be able to download older versions.

Collection versions use Semantic Versioning for version numbers. Please read the official documentation for details and examples. In summary:


Migrating Ansible content to a different collection

First, look at Ansible Collection Checklist.

To migrate content from one collection to another, if the collections are parts of Ansible distribution:

  1. Copy content from the source (old) collection to the target (new) collection.

  2. Deprecate the module/plugin with removal_version scheduled for the next major version in meta/runtime.yml of the old collection. The deprecation must be released after the copied content has been included in a release of the new collection.

  3. When the next major release of the old collection is prepared:

  • remove the module/plugin from the old collection

  • remove the symlink stored in plugin/modules directory if appropriate (mainly when removing from community.general and community.network)

  • remove related unit and integration tests

  • remove specific module utils

  • remove specific documentation fragments if there are any in the old collection

  • add a changelog fragment containing entries for removed_features and breaking_changes; you can see an example of a changelog fragment in this pull request

  • change meta/runtime.yml in the old collection:

    • add redirect to the corresponding module/plugin's entry

    • in particular, add redirect for the removed module utils and documentation fragments if applicable

    • remove removal_version from there

  • remove related entries from tests/sanity/ignore.txt files if exist

  • remove changelog fragments for removed content that are not yet part of the changelog (in other words, do not modify changelogs/changelog.yaml and do not delete files mentioned in it)

  • remove requirements that are no longer required in tests/unit/requirements.txt, tests/requirements.yml and galaxy.yml

According to the above, you need to create at least three PRs as follows:

  1. Create a PR against the new collection to copy the content.

  2. Deprecate the module/plugin in the old collection.

  3. Later create a PR against the old collection to remove the content according to the schedule.


Adding the content to the new collection

Create a PR in the new collection to:

  1. Copy ALL the related files from the old collection.

  2. If it is an action plugin, include the corresponding module with documentation.

  3. If it is a module, check if it has a corresponding action plugin that should move with it.

  4. Check meta/ for relevant updates to runtime.yml if it exists.

  5. Carefully check the moved tests/integration and tests/units and update for FQCN.

  6. Review tests/sanity/ignore-*.txt entries in the old collection.

  7. Update meta/runtime.yml in the old collection.


Removing the content from the old collection

Create a PR against the source collection repository to remove the modules, module_utils, plugins, and docs_fragments related to this migration:

  1. If you are removing an action plugin, remove the corresponding module that contains the documentation.

  2. If you are removing a module, remove any corresponding action plugin that should stay with it.

  3. Remove any entries about removed plugins from meta/runtime.yml. Ensure they are added into the new repo.

  4. Remove sanity ignore lines from tests/sanity/ignore\*.txt

  5. Remove associated integration tests from tests/integrations/targets/ and unit tests from tests/units/plugins/.

  6. if you are removing from content from community.general or community.network, remove entries from .github/BOTMETA.yml.

  7. Carefully review meta/runtime.yml for any entries you may need to remove or update, in particular deprecated entries.

  8. Update meta/runtime.yml to contain redirects for EVERY PLUGIN, pointing to the new collection name.

Warning

Maintainers for the old collection have to make sure that the PR is merged in a way that it does not break user experience and semantic versioning:

  1. A new version containing the merged PR must not be released before the collection the content has been moved to has been released again, with that content contained in it. Otherwise the redirects cannot work and users relying on that content will experience breakage.

  2. Once 1.0.0 of the collection from which the content has been removed has been released, such PRs can only be merged for a new major version (in other words, 2.0.0, 3.0.0, and so on).


BOTMETA.yml

The BOTMETA.yml, for example in community.general collection repository, is the source of truth for:

If the old and/or new collection has ansibullbot, its BOTMETA.yml must be updated correspondingly.

Ansibulbot will know how to redirect existing issues and PRs to the new repo. The build process for docs.ansible.com will know where to find the module docs.

Example PR


Testing collections

The main tool for testing collections is ansible-test, Ansible's testing tool described in Testing Ansible. You can run several compile and sanity checks, as well as run unit and integration tests for plugins using ansible-test. When you test collections, test against the ansible-base version(s) you are targeting.

You must always execute ansible-test from the root directory of a collection. You can run ansible-test in Docker containers without installing any special requirements. The Ansible team uses this approach in Shippable both in the ansible/ansible GitHub repository and in the large community collections such as community.general and community.network. The examples below demonstrate running tests in Docker containers.


Compile and sanity tests

To run all compile and sanity tests:

See Compile Tests and Sanity Tests for more information. See the full list of sanity tests for details on the sanity tests and how to fix identified issues.


Unit tests

You must place unit tests in the appropriate``tests/unit/plugins/`` directory. For example, you would place tests for plugins/module_utils/foo/bar.py in tests/unit/plugins/module_utils/foo/test_bar.py or tests/unit/plugins/module_utils/foo/bar/test_bar.py. For examples, see the unit tests in community.general.

To run all unit tests for all supported Python versions:

To run all unit tests only for a specific Python version:

To run only a specific unit test:

You can specify Python requirements in the tests/unit/requirements.txt file. See Unit Tests for more information, especially on fixture files.


Integration tests

You must place integration tests in the appropriate tests/integration/targets/ directory. For module integration tests, you can use the module name alone. For example, you would place integration tests for plugins/modules/foo.py in a directory called tests/integration/targets/foo/. For non-module plugin integration tests, you must add the plugin type to the directory name. For example, you would place integration tests for plugins/connections/bar.py in a directory called tests/integration/targets/connection_bar/. For lookup plugins, the directory must be called lookup_foo, for inventory plugins, inventory_foo, and so on.

You can write two different kinds of integration tests:

For examples, see the integration tests in community.general. See also Integration tests for more details.

Since integration tests can install requirements, and set-up, start and stop services, we recommended running them in docker containers or otherwise restricted environments whenever possible. By default, ansible-test supports Docker images for several operating systems. See the list of supported docker images for all options. Use the default image mainly for platform-independent integration tests, such as those for cloud modules. The following examples use the centos8 image.

To execute all integration tests for a collection:

If you want more detailed output, run the command with -vvv instead of -v. Alternatively, specify --retry-on-error to automatically re-run failed tests with higher verbosity levels.

To execute only the integration tests in a specific directory:

You can specify multiple target names. Each target name is the name of a directory in tests/integration/targets/.


Contributing to collections

If you want to add functionality to an existing collection, modify a collection you are using to fix a bug, or change the behavior of a module in a collection, clone the git repository for that collection and make changes on a branch. You can combine changes to a collection with a local checkout of Ansible (source hacking/env-setup).

This section describes the process for community.general. To contribute to other collections, replace the folder names community and general with the namespace and collection name of a different collection.

We assume that you have included ~/dev/ansible/collections/ in COLLECTIONS_PATHS, and if that path mentions multiple directories, that you made sure that no other directory earlier in the search path contains a copy of community.general. Create the directory ~/dev/ansible/collections/ansible_collections/community, and in it clone the community.general Git repository or a fork of it into the folder general:

If you clone a fork, add the original repository as a remote upstream:

Now you can use this checkout of community.general in playbooks and roles with whichever version of Ansible you have installed locally, including a local checkout of ansible/ansible's devel branch.

For collections hosted in the ansible_collections GitHub org, create a branch and commit your changes on the branch. When you are done (remember to add tests, see Testing collections), push your changes to your fork of the collection and create a Pull Request. For other collections, especially for collections not hosted on GitHub, check the README.md of the collection for information on contributing to it.


Generating changelogs for a collection

We recommend that you use the antsibull-changelog tool to generate Ansible-compatible changelogs for your collection. The Ansible changelog uses the output of this tool to collate all the collections included in an Ansible release into one combined changelog for the release.

Ansible here refers to the Ansible 2.10 or later release that includes a curated set of collections.


Understanding antsibull-changelog

The antsibull-changelog tool allows you to create and update changelogs for Ansible collections that are compatible with the combined Ansible changelogs. This is an update to the changelog generator used in prior Ansible releases. The tool adds three new changelog fragment categories: breaking_changes, security_fixes and trivial. The tool also generates the changelog.yaml file that Ansible uses to create the combined CHANGELOG.rst file and Porting Guide for the release.

See Creating a changelog fragment and the antsibull-changelog documentation for complete details.

The collection maintainers set the changelog policy for their collections. See the individual collection contributing guidelines for complete details.


Generating changelogs

To initialize changelog generation:

  1. Install antsibull-changelog: pip install antsibull-changelog.

  2. Initialize changelogs for your repository: antsibull-changelog init <path/to/your/collection>.

  3. Optionally, edit the changelogs/config.yaml file to customize the location of the generated changelog .rst file or other options. See Bootstrapping changelogs for collections for details.

To generate changelogs from the changelog fragments you created:

  1. Optionally, validate your changelog fragments: antsibull-changelog lint.

  2. Generate the changelog for your release: antsibull-changelog release [--version version_number].

Add the --reload-plugins option if you ran the antsibull-changelog release command previously and the version of the collection has not changed. antsibull-changelog caches the information on all plugins and does not update its cache until the collection version changes.


Porting Guide entries

The following changelog fragment categories are consumed by the Ansible changelog generator into the Ansible Porting Guide:


Including collection changelogs into Ansible

If your collection is part of Ansible, use one of the following three options to include your changelog into the Ansible release changelog:

For the first two options, Ansible pulls the changelog details from Galaxy so your changelogs must be included in the collection version on Galaxy that is included in the upcoming Ansible release.


See also

Using collections

Learn how to install and use collections.

Collection Galaxy metadata structure

Understand the collections metadata structure.

Ansible module development: getting started

Learn about how to write Ansible modules

Mailing List

The development mailing list

irc.freenode.net

#ansible IRC chat channel

Next Previous