Get the free coreGet core

Remove secrets from Git history (and why rotating comes first)

Updated 6 min readSection 05 / Threats
SHORT ANSWER

To remove secrets from Git history, first revoke or rotate the key with its provider, because that may be enough without rewriting anything. If you still want a clean history, work in a fresh clone. Run git-filter-repo --sensitive-data-removal --invert-paths --path PATH-TO-YOUR-FILE-WITH-SENSITIVE-DATA, or use BFG. Then force-push, which replaces the history on GitHub, and take a few more steps to clear the data from GitHub itself. Rewriting has side effects, including changed commit hashes.

You pushed a key. Deleting the file and committing again does not help, because Git keeps every earlier commit. The order that works is: revoke the key, find where it sits, rewrite a fresh clone, push, and clean up what GitHub keeps.

01Step 1: revoke the key before anything else

Consider any leaked secret immediately compromised. Removing it from the codebase, pushing a new commit, or deleting and recreating the repository does not stop it being exploited. The most important step is revoking the secret with its provider, which usually means logging into the provider's portal and finding where the secret is managed.

Speed matters, because automated scanners can locate publicly exposed secrets in minutes. The order of steps is on the page about a leaked key. It also covers how to avoid downtime and how to check whether the key was used. The repository side is on a key exposed on GitHub.

02Do you still need to remove secrets from Git history?

Once the secret is revoked or rotated, it can no longer be used for access, and that may be sufficient. Going through the extra steps to rewrite the history may not be warranted. Rewriting is cleanup, not rescue. A dead key in old commits is just a string that no longer opens anything.

If you do it, coordinate first. Sensitive data can be removed from a repository's history if you coordinate with everyone who has cloned it and are willing to manage the side effects. On a public repository or one with many forks, you will not reach everyone. GitHub Support will not remove non-sensitive data, and helps with sensitive data only when the risk cannot be mitigated by rotating the affected credentials. So the order is fixed: rotate first, then decide whether a rewrite is worth the trouble for your repository.

03Step 2: find where the key is

Search the commit history with git log -S followed by the text of the key to see who committed it. Besides the file you know about, check old branches, tags and test files. git-filter-repo has an --analyze report that helps you decide what to prune.

Fix the current files first. By default, BFG does not modify the contents of your latest commit on the HEAD branch, so a key still in the working tree stays there. If the key came from a .env file, stop tracking the file and add it to .gitignore. The lines are on the page about where to store API keys.

04Step 3: remove secrets from Git history in a fresh clone

Work in a fresh clone. The author of git-filter-repo notes that almost everyone does so, because wiping out the clone in case of error is a vastly easier recovery mechanism. You have two documented tools.

git-filter-repoBFG Repo-Cleaner
StatusThe Git project recommends it instead of git filter-branchThe git-filter-repo README calls it a great tool for its time, but limited to a few kinds of rewrites
Needsgit 2.36.0 or newer and python3 3.6 or newer. For --sensitive-data-removal, git-filter-repo 2.47 or newerA fresh clone made with the --mirror flag
Removes a filegit-filter-repo --sensitive-data-removal --invert-paths --path PATH-TO-YOUR-FILE-WITH-SENSITIVE-DATAbfg --delete-files id_{dsa,rsa} my-repo.git
Replaces textgit-filter-repo --sensitive-data-removal --replace-text ../passwords.txtbfg --replace-text passwords.txt my-repo.git

With git-filter-repo

Install it with Homebrew, then clone the repository and step into it:

brew install git-filter-repo
git clone https://github.com/YOUR-USERNAME/YOUR-REPOSITORY
cd YOUR-REPOSITORY

To remove a file from all branches and tags, run:

git-filter-repo --sensitive-data-removal --invert-paths --path PATH-TO-YOUR-FILE-WITH-SENSITIVE-DATA

To replace passwords everywhere in history, run:

git-filter-repo --sensitive-data-removal --replace-text ../passwords.txt

Replace the placeholder with the Git path to the file, such as config/.env, not just its name. If the file was moved or renamed, add a --path for each old path. ../passwords.txt is a file you write that lists the passwords to replace. Check the result before you go on. The rewrite becomes irreversible once you move to the next step, so if the result is wrong, discard the clone and start again.

With BFG

BFG is a Java program you download from its project site, rtyley.github.io/bfg-repo-cleaner; in its examples bfg stands for java -jar bfg.jar. It starts from a fresh clone made with the --mirror flag. To replace every password listed in passwords.txt with ***REMOVED*** wherever it occurs in your repository, run:

bfg --replace-text passwords.txt my-repo.git

Here my-repo.git is the mirror clone. To drop a whole file instead, use --delete-files with its name, as in bfg --delete-files id_{dsa,rsa} my-repo.git. BFG updates the commits but does not physically delete the old data, so step into the mirror clone and follow it with this command:

git reflog expire --expire=now --all && git gc --prune=now --aggressive

05Step 4: push, and the extra GitHub steps

After rewriting with git-filter-repo, force-push the cleaned history to overwrite the repository on GitHub:

git push --force --mirror origin

Refs that start with refs/pull/ will fail to push, because GitHub marks them read-only. After a BFG run in a mirror clone, a plain git push updates all refs. What you push is final, because the rewrite becomes irreversible at that point. Then more work remains: after git-filter-repo and a push, a few more steps are needed to fully remove the data from GitHub.

This is why the rotation in step 1 is the real fix. The rewrite only shrinks the places where a dead key sits.

06Side effects to expect

Rewriting is not free. These are the costs, and the first is the one that bites most teams.

Side effectWhat happens
RecontaminationIt is easy to re-push the sensitive data and make a bigger mess
Old clonesEveryone should delete old clones and make fresh ones, because they hold dirty history
Changed hashesThe hashes of the commits that introduced the data, and all commits after them, change
Curious usersA user with an existing clone will notice the history divergence and can use it to find the removed data

Tell your teammates before you push, and ask each of them to re-clone afterward. Do the checking while the original repository is still untouched, so you can start over if the result looks wrong.

07Next time: catch it before the push

GitHub's push protection blocks pushes that contain secrets before they reach your repository. A pre-commit scanner on your own machine catches it earlier still. The options are compared on the secret scanning page, and the GitHub steps are on GitHub secret scanning. The cheapest habit is to keep keys out of the repository entirely: see where to store API keys. If you are unsure what a key can do, read what an API key is.

08The short version

Rotate the key first. Rewrite a fresh clone only if you still want a clean history, using git-filter-repo or BFG, then push and ask GitHub Support for the rest. When you remove secrets from Git history, the key should already be dead.

FAQ

Can you remove something from Git history?

Yes, if you can coordinate with everyone who has cloned the repository and accept the side effects. For a secret, revoking it comes first.

How do I completely remove a commit from Git history?

For a file with sensitive data, git-filter-repo can remove it from all branches and tags. The hashes of that commit and every later one change. On GitHub, forks and cached copies need extra steps.

How do I clean up Git history after a leak?

Revoke the key first. Then work in a fresh clone, run git-filter-repo or BFG, check the result, and push. The rewrite is irreversible once you move to the push step.

Is deleting the file in a new commit enough?

No. A file deleted in the web interface stays in Git history. Removing a secret from the codebase or pushing a new commit does not prevent it being exploited.

Do I need to rewrite history at all?

Often not. Once the secret is revoked or rotated it can no longer be used for access, and that may be sufficient.

Why do my teammates still have the secret after I cleaned the repo?

Their old clones hold the dirty history, and it is easy to re-push it. Everyone should delete old clones and make fresh ones.

Sources

  1. GitHub docs, removing sensitive data from a repository
  2. GitHub docs, remediating a leaked secret
  3. git-filter-repo README
  4. BFG Repo-Cleaner
Cleaning history is the last step. Artificial Immunity watches program starts on the Mac where your agents work.Get the free core