Ansible + Git: Complete DevOps Workflow

Ansible has the shortest path in this pillar from a merged commit to a changed production host.

Terraform produces a plan you can read. Kubernetes converges gradually and can be paused. Ansible connects to machines and does what the playbook says, in order, now. That immediacy is why the repository engineering around it matters more than teams expect.

Before you start

• Ansible + Git — the cluster’s framing
• Ansible CI with GitHub Actions — the lint and verify workflow

The boundary

Validation and execution are different activities and must not share a workflow.

Everything in this cluster follows from that.

Linting a playbook is safe. It reads files and reports.

Running a playbook is not. It connects to hosts and changes them.

A pipeline doing the first is a quality gate. A pipeline doing the second is a deployment system holding production credentials, triggered by whatever somebody pushed.

Danger

Running a production playbook as pull request validation gives arbitrary submitted code the ability to reach production hosts with your SSH keys. It happens because “the tests should be realistic” sounds reasonable until you write down what it permits: anybody who can open a pull request can add a task that runs a command on every host in your inventory.

What belongs in the repository

In Git	Not in Git
Playbooks	Vault passwords, in any form
Roles you wrote	Private SSH keys
requirements.yml	Downloaded roles and collections
Inventory structure and groups	*.retry files
group_vars / host_vars, non-secret	Real credentials, encoded or not
Encrypted vault files	Fact caches
ansible.cfg	.vault_pass
.ansible-lint, CI workflows	Anything generated at run time

Three rows deserve explanation.

Downloaded content does not belong in the repository. requirements.yml declares dependencies; ansible-galaxy install fetches them. Committing roles/ and collections/ produces a repository where nobody can tell which code is yours, and where updating a dependency is a large diff nobody reviews.

An encrypted vault file may be committed. Its password may not. That distinction changes the risk model without eliminating key management, and it has its own lesson.

ansible.cfg is committed and matters more than it looks. It sets the inventory path, the roles path, whether host key checking is enabled, and the connection defaults. A repository without one depends on each user’s local configuration, which is how two people running the same playbook get different results.

The .gitignore

# Downloaded dependencies
roles/
collections/
!roles/local/
!requirements.yml

# Vault passwords — never
.vault_pass
.vault_pass.txt
vault-password*
*.vaultpass

# Keys
*.pem
*.key
id_rsa*
!*.pub

# Run artifacts
*.retry
*.log
.ansible/
fact_cache/

# Local overrides
inventory/local*
*.local.yml

The roles/ exclusion with an exception is the shape to copy: ignore the directory ansible-galaxy installs into, and re-include anything genuinely yours.

The vault password patterns are the ones that matter. A committed vault password makes every encrypted file in the repository plaintext, permanently, in every clone.

Repository structure

Covered fully in repository structure; the shape most repositories converge on:

ansible/
├── ansible.cfg
├── requirements.yml
├── inventories/
│   ├── development/
│   │   ├── hosts.yml
│   │   ├── group_vars/
│   │   └── host_vars/
│   └── production/
├── playbooks/
│   ├── site.yml
│   └── webservers.yml
├── roles/
│   └── local/
│       └── app_deploy/
├── tests/
└── .github/workflows/

Inventories are separated by environment, which is the primary safety boundary — running against production requires naming production’s inventory.

Playbooks are thin. A playbook lists hosts and applies roles. Logic lives in roles, which are testable in isolation with Molecule — a playbook full of inline tasks is logic that cannot be tested without a target.

Inventories are the sensitive part

A playbook says what to do. An inventory says what to do it to, and that is where operational detail leaks.

inventories/production/hosts.yml

all:
  children:
    webservers:
      hosts:
        web1.example.com:
        web2.example.com:
    databases:
      hosts:
        db1.example.com:

Use reserved example names in anything you publish or share. example.com, and the RFC 5737 documentation ranges 192.0.2.0/24, 198.51.100.0/24 and 203.0.113.0/24. Copying a tutorial’s plausible-looking internal hostname into a real inventory is a recorded way people cause incidents.

Structure so production is hard to hit by accident. Separate files, explicit -i, and never a default inventory that includes production. An ansible.cfg whose inventory points at development is a cheap safety measure.

Consider whether the inventory belongs in the same repository at all. For a small estate, yes — the coordination cost of splitting exceeds the benefit. For anything where the host list is itself sensitive, a separate repository with tighter access is defensible, and it means somebody can contribute a role improvement without being able to read your production topology. Dynamic inventory from a cloud provider removes the question entirely by generating the list at run time from tags, which is the better answer wherever the hosts are cloud instances.

Variables in group_vars are where credentials appear. A group_vars/production/vault.yml encrypted with Ansible Vault is the normal pattern, and the encryption is only as good as the password’s storage.

Branching and review

The general model from Pillar 2 applies, with Ansible-specific emphasis.

Trunk-based, short-lived branches. Ansible has no state file and no plan, so there is less to go stale than with Terraform — but a change that has been merged and not run is still a difference between the repository and reality.

Review roles more carefully than playbooks. A playbook change affects one run; a role change affects every playbook using it.

What a reviewer should look for:

shell or command where a module exists. Modules are idempotent and report changes correctly; shell commands usually do neither.

Missing changed_when on a command task. A task that reports changed on every run makes the whole playbook’s change report meaningless.

A task without a name. Unreadable output when something fails at 3am.

become added. Privilege escalation, and where it is not needed it should not be there.

Anything touching the inventory, particularly production’s.

A new when condition guarding something dangerous. Read it carefully — a condition that is subtly wrong runs the task somewhere unintended.

Deprecated modules or syntax, which ansible-lint catches better than a human.

Variable precedence

Ansible has a long variable precedence order, and it is the source of more confusion than any other feature.

The practical consequence for a repository: the same variable can be defined in a role’s defaults, a group_vars file, a host_vars file, the playbook, the inventory and the command line — and which one wins is not obvious from reading any single file.

The discipline that keeps this manageable:

Role defaults are the lowest precedence and are where a role’s variables belong. roles/x/defaults/main.yml sets a sensible value that anything can override.

group_vars is where environment differences live. One place, diffable between environments, which is the same argument as Terraform’s values files.

host_vars is for genuine per-host facts and should be small. A host_vars directory doing real configuration work means the group structure is wrong.

Avoid vars: in playbooks for anything an environment might need to change. It has high precedence and it is invisible to somebody reading group_vars.

Never rely on -e for anything routine. Extra vars beat almost everything, which makes them useful for a one-off override and dangerous as a habit — a value passed on the command line is a value nobody else’s run will have.

Prefix role variables with the role name. nginx_port rather than port. Ansible has one flat variable namespace, and two roles both using port will collide in a way that is genuinely hard to debug.

ansible-inventory --graph --vars shows what a host actually resolves to, which is the tool for answering “where is this value coming from” and is worth reaching for before reading files.

Repository conventions worth adopting

Small decisions that pay back repeatedly.

Name every task. Unnamed tasks produce output identifying them by module and arguments, which is unreadable when a playbook fails at 3am.

One role, one responsibility. A role that installs a package, configures it and manages its data is three roles, and only one of them is likely to be reusable.

Tag consistently. Tags let a run target part of a playbook, and inconsistent tagging means --tags produces surprising results. Decide a small set and apply it.

Use ansible.builtin. fully-qualified names. Explicit about which collection a module comes from, and it prevents a collection you install later shadowing a builtin.

Put a README in every role, stating what it does, its variables and its dependencies. Roles outlive the person who wrote them.

Keep playbooks under a screen. A playbook listing hosts and applying five roles is readable. One containing forty inline tasks is a role that has not been extracted.

Pin collection versions in requirements.yml. A floating version means a collection update changes behaviour without a commit in your repository — the same argument as everywhere else in this pillar.

Idempotency

Ansible’s central promise and the property most broken roles fail.

A playbook run twice should report no changes the second time. If it does not, something is not idempotent, and that matters beyond tidiness: a non-idempotent task means you cannot safely re-run a playbook after a partial failure, which is exactly when you need to.

The usual culprits: command and shell tasks with no creates, removes or changed_when; template tasks whose content varies per run — a timestamp is the classic; and tasks that append rather than declaring a desired state.

Test it in CI. Run the playbook against a disposable container twice and assert the second run reports zero changed. Molecule has this built in as an idempotence step, which is one of the strongest arguments for using it rather than assembling the equivalent by hand.

--check mode is the closest thing to a plan, and its limits are worth knowing: modules implement it individually, some cannot predict what they would do, and any task depending on a previous task’s actual effect reports inaccurately in check mode. Useful, and not equivalent to terraform plan.

Where Ansible sits

Being explicit, because the comparison with the rest of this pillar clarifies its use.

It is imperative. An ordered list of tasks. Excellent where ordering genuinely matters, poor at continuous reconciliation.

It has no state file. Nothing records what it previously did; idempotent modules re-derive the situation by inspecting the host. That removes an entire category of problem — no state to lose, corrupt or commit — and creates another: no plan, no drift report, no reliable answer to “what would this change?”.

It is push-based, with no agent. The control node connects outward. It therefore does not satisfy the GitOps principles and should not be described as GitOps regardless of how the playbooks are stored.

It configures hosts. It has good cloud and Kubernetes modules, and its centre of gravity is machines. Where Terraform or a Kubernetes controller would fit better, they usually do — and stretching Ansible to cover provisioning it does not naturally model is a common source of unmaintainable repositories.

Releasing and versioning

Whether an Ansible repository needs versions depends on what consumes it.

A repository run only by its own team does not need release tags. main is what runs, and a Git SHA identifies what was executed. Recording that SHA when you run is the whole versioning story.

Roles and collections consumed by others do need versions, for the same reason Terraform modules do — and that is its own lesson.

Tag a release if you run from tags rather than from main. Some teams prefer this for production: development runs main, production runs a tag, and promotion is moving which tag production runs. It adds a step and gives you a defined artifact.

Record the SHA in the run log, whichever model you use. “Which version of the playbook ran on Tuesday” is a question with no answer unless somebody wrote it down, and it is the first question after an unexplained change.

A changelog is worth it above a certain size. Not for every task change — for the ones that alter behaviour on managed hosts. A team returning to a playbook after three months wants to know what changed, and git log on a large repository does not answer that quickly.

Common mistakes

Running production playbooks in CI. Arbitrary code reaching production hosts.

Committing downloaded roles and collections. Nobody can tell what is yours.

A committed vault password. Every encrypted file is plaintext, permanently.

Real hostnames in examples. Copied into somebody’s inventory.

A default inventory including production. One forgotten flag from an incident.

shell where a module exists. Not idempotent, and reports changes wrongly.

No ansible.cfg. Behaviour depends on each user’s local configuration.

Not testing idempotency. Re-running after a failure is unsafe.

Calling it GitOps. No agent, no pull, no continuous reconciliation.

Running against production

The part that is deliberately not automated on merge.

Naming the inventory is the safety mechanism. ansible-playbook -i inventories/production/hosts.yml site.yml is explicit. A default inventory that includes production removes it.

--limit narrows the blast radius. Running against one host first, confirming, then the group. For anything consequential this is the difference between one broken host and all of them.

--check first, where the modules support it honestly. It is not a plan and it catches the obvious.

--diff shows file changes that will be made, which is the closest Ansible gets to showing you the effect of a template change before applying it.

Serial execution for anything behind a load balancer. serial: 1 or a percentage in the play means the playbook rolls through hosts rather than restarting a service everywhere simultaneously. Without it, a playbook restarting a service is an outage.

any_errors_fatal and max_fail_percentage stop a run that is going wrong rather than continuing through every host.

Run from somewhere accountable. A person’s laptop with their own SSH key is traceable to a person, and CI with an environment approval is traceable to a decision. A shared jump host with a shared key is neither, and it is what many estates actually have.

Record what was run. Which playbook, which inventory, which limit, by whom, when. Ansible does not do this for you, and the log of what was actually executed is the thing you want during an incident review.

Handlers and change reporting

A detail specific to Ansible that affects how reviewable a run is.

Handlers run once, at the end, if notified. A configuration task that changes a file notifies a handler that restarts the service. That is the correct pattern and it means a run’s effect is not visible in task order.

meta: flush_handlers forces them to run earlier, which is necessary when a later task depends on the restart having happened.

A handler that does not run is invisible. If the notifying task reports no change, the handler is skipped — which is right, and means a run showing zero changes genuinely did nothing.

This is why change reporting matters. A playbook whose tasks report changed on every run makes it impossible to tell whether anything actually happened, and it means handlers fire on every run too — restarting services unnecessarily.

The review implication: a task using command or shell without changed_when is not just untidy. It makes the run’s output meaningless and can cause a service restart on every execution.

Migrating shell scripts to Ansible

The common origin story: a directory of shell scripts that configure servers, and a decision to adopt Ansible.

Do not translate line by line. A shell script’s steps become command tasks, which are not idempotent, do not report changes correctly, and produce a playbook with none of Ansible’s advantages. This is the most common bad migration and it results in a team concluding Ansible is not much better than what they had.

Translate intent instead. apt-get install -y nginx is not a command task; it is the package module. sed -i on a config file is a template or lineinfile task. systemctl restart is a handler.

Do the easy modules first. Package installation, file placement, service management and user creation cover most of what configuration scripts do, and all have well-behaved modules.

Leave genuinely awkward steps as command, with creates, removes or changed_when so they are at least idempotent and honest about whether they changed anything.

Test each translated piece against a container before running it anywhere real. A container is close enough for package and file operations, which are the bulk of it.

Expect the playbook to be longer than the script. Explicit is longer. The return is idempotency, readable change reporting and the ability to run it against a hundred hosts.

Run both for a period — the script on some hosts, the playbook on others — and compare the results. A divergence is a translation error, and finding it during the migration is much cheaper than afterwards.

Mental model

A playbook is a procedure and an inventory is a target list. Version control makes the procedure reviewable; keeping the target list separate and explicit is what stops the procedure running somewhere you did not mean.

Everything else in this cluster is an application of that: CI validates procedures without targets, execution names its target deliberately, and the credentials for each are different.

What you learned

• Validation and execution are separate activities and must not share a workflow
• requirements.yml is committed; the roles and collections it fetches are not
• An encrypted vault file may be committed; its password may not
• ansible.cfg is committed and determines behaviour that would otherwise vary per user
• Inventories carry the operational detail — use reserved example names and make production explicit
• Idempotency is testable and is what makes re-running after a partial failure safe
• --check is not a plan: modules implement it individually and dependent tasks report inaccurately
• Ansible is imperative, stateless and push-based, and is not GitOps

Exercise

Use a disposable directory and a local container. No production hosts, no real credentials.

1. Create an Ansible repository with ansible.cfg, a requirements.yml, one playbook and one local role.

2. Run ansible-galaxy install -r requirements.yml. Run git status. Predict: with the .gitignore above, is the downloaded content ignored?

3. Write a task using shell to create a file. Run the playbook twice. Predict: what does the second run report?

4. Rewrite it with the file module. Run twice again. Compare.

5. Add creates: to the original shell task. Run twice. Predict: is it idempotent now?

6. Run the playbook with --check against a container. Predict: does every task report accurately?

7. Create inventories/production/hosts.yml with example.com hosts, and set ansible.cfg to default to the development inventory. Run without -i. Predict: which hosts are targeted?

8. Delete the directory and the container.

Related lessons

Ansible Repository StructureLayouts for small and large estates.

Ansible CIValidation that never touches production.

Ansible Vault and GitEncryption, and the password you must not commit.

Molecule Testing with GitTesting roles against disposable targets.