← Blog

How to Track Agent Skill and MCP Server Changes (and Pin Versions)

October 31, 2026 · backskills · 7 min read

To stop a third-party skill or MCP server from changing under you, pin it: a SKILL.md to a git commit SHA, an MCP server to an exact package version. Then watch upstream for new versions so you upgrade on purpose, after reviewing the diff, instead of finding out from a broken agent.

Two jobs. Pin. Watch. The rest of this post is how.

Why this matters more than it does for normal dependencies

A library change usually breaks loudly. A type error, a failing test.

A skill change usually doesn't. Someone rewrites a description and your agent stops loading the skill for half its old triggers. Someone adds a step and outputs change shape. An MCP server renames a tool or edits its description, and the model picks a different tool. Nothing throws. Behavior just drifts.

And the model under it drifts too. Chen, Zaharia and Zou (2023) compared versions of the same hosted LLM service a few months apart and found behavior on the same tasks changed substantially. If both the skill and the model can move, you want at least one of them nailed down.

Skill pinning means referencing an exact, immutable revision of a skill (a commit SHA or a published version) rather than "whatever is on main."

Step 1: Pin SKILL.md files to a commit

Branch names move. Tags can be moved. Commit SHAs can't.

Vendor the skill at a known SHA:

#!/usr/bin/env bash
# vendor-skill.sh OWNER/REPO PATH/TO/SKILL SHA
set -euo pipefail
repo="$1"; path="$2"; sha="$3"
name="$(basename "$path")"
dest="skills/$name"
tmp="$(mktemp -d)"
git clone --filter=blob:none --no-checkout "https://github.com/$repo" "$tmp"
git -C "$tmp" checkout "$sha" -- "$path"
rm -rf "$dest" && mkdir -p skills && cp -r "$tmp/$path" "$dest"
echo "$repo $path $sha" >> skills/LOCK
echo "pinned $name @ $sha"

Keep a lockfile so anyone can see what's installed and from where:

# skills/LOCK  (repo path sha)
anthropics/skills skills/pdf 3f2a9c1e...
someorg/agent-skills release-notes 9b81d07a...

Now upgrades are a pull request. You see the diff of SKILL.md and every script before your agent does.

Step 2: Pin MCP servers to an exact version

The MCP Registry versioning docs say each publication must have a unique version string, and published metadata can't be changed afterward. That's what makes pinning possible. Use it.

The common mistake is a client config that installs "latest" on every start:

{
  "mcpServers": {
    "weather": {
      "command": "npx",
      "args": ["-y", "@my-username/mcp-weather-server"]
    }
  }
}

Pin it:

{
  "mcpServers": {
    "weather": {
      "command": "npx",
      "args": ["-y", "@my-username/mcp-weather-server@1.0.1"]
    }
  }
}

Same idea for Python (uvx package==1.0.1) and containers (pin by digest, image@sha256:..., not by tag).

Remote MCP servers are harder. You connect to a URL, and the operator can deploy whenever they like. You can't pin those. You can only watch them, which is step 3.

Step 3: Watch upstream

You have pins. Now you need to know when a newer version exists, or when something you can't pin has changed.

DIY: poll GitHub for skill changes

#!/usr/bin/env python3
"""Report pinned skills whose upstream path has new commits."""
import os, httpx

TOKEN = os.environ.get("GITHUB_TOKEN")
H = {"Authorization": f"Bearer {TOKEN}"} if TOKEN else {}

def latest_sha(repo: str, path: str) -> str:
    r = httpx.get(f"https://api.github.com/repos/{repo}/commits",
                  params={"path": path, "per_page": 1}, headers=H)
    r.raise_for_status()
    return r.json()[0]["sha"]

for line in open("skills/LOCK"):
    if line.startswith("#") or not line.strip():
        continue
    repo, path, pinned = line.split()
    head = latest_sha(repo, path)
    if not head.startswith(pinned[:7]):
        print(f"CHANGED {repo}/{path}: pinned {pinned[:7]} -> upstream {head[:7]}")

DIY: poll the MCP Registry

The registry has a public search API (quickstart):

curl -s "https://registry.modelcontextprotocol.io/v0.1/servers?search=io.github.my-username/weather"

Compare the returned version with your pin and alert when it differs. Run both checks daily in CI (a scheduled GitHub Action works) and open an issue when something moved.

This works. It also becomes a chore around the tenth dependency: rate limits, repos that move, skills nested at odd paths, remote servers with no version field you can rely on.

Managed: backskills

backskills does the watching for public skills and servers. It tracks SKILL.md files on GitHub and servers in the MCP Registry, checks them daily, keeps version history so you can pin to a known revision, and sends a signed webhook within 24 hours when something you watch changes. Agents can use it directly through the MCP endpoint at https://backskills.ai/mcp; the llms.txt describes what's available.

Step 4: Verify the webhook signature

Any change-alert webhook should be signed, and you should reject anything that isn't. backskills signs every delivery with your per-key secret (get it from GET /api/webhooks/secret with your X-Agent-Key). Each request carries three headers:

Including the timestamp in the signed string lets you reject replays. Send yourself a real signed alert with POST /api/webhooks/test and verify it like this:

import hmac, hashlib, os, time
from fastapi import FastAPI, Request, HTTPException

app = FastAPI()
SECRET = os.environ["BACKSKILLS_WEBHOOK_SECRET"].encode()   # the whsec_... value

@app.post("/hooks/skill-changed")
async def skill_changed(req: Request):
    body = await req.body()                                   # raw bytes, before JSON parsing
    ts = req.headers.get("X-Backskills-Timestamp", "")
    sent = req.headers.get("X-Backskills-Signature", "")
    if not ts.isdigit() or abs(time.time() - int(ts)) > 300:  # 5-minute replay window
        raise HTTPException(401, "stale timestamp")
    expected = "sha256=" + hmac.new(SECRET, f"{ts}.".encode() + body, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(sent, expected):
        raise HTTPException(401, "bad signature")
    event = await req.json()
    # open an issue, ping Slack, or kick off your regression tests
    return {"ok": True}

Two rules: sign the raw bytes, not re-serialized JSON, and never use == for the comparison.

Step 5: Review before you bump

When an alert fires, don't just bump the pin. Five minutes of review:

  1. Read the SKILL.md diff. Did the description change? That changes when the skill triggers.
  2. Read script diffs. New network calls, new dependencies, new file writes. Treat it like any third-party code.
  3. For MCP servers, diff the tool list. Names, descriptions, input schemas. A changed description can change which tool the model picks.
  4. Run your trigger tests (the should-trigger / should-not-trigger lists from How to write a SKILL.md) on every model you use.
  5. Bump the pin in a PR with a one-line note on what changed.

A tiny tool-list differ for MCP servers, using the official Python SDK client:

import asyncio, json, sys
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

async def tools(cmd: list[str]) -> dict:
    params = StdioServerParameters(command=cmd[0], args=cmd[1:])
    async with stdio_client(params) as (r, w):
        async with ClientSession(r, w) as s:
            await s.initialize()
            res = await s.list_tools()
            return {t.name: {"desc": t.description, "schema": t.inputSchema}
                    for t in res.tools}

async def main():
    old = await tools(["npx", "-y", sys.argv[1]])
    new = await tools(["npx", "-y", sys.argv[2]])
    for name in sorted(set(old) | set(new)):
        if old.get(name) != new.get(name):
            print(f"~ {name}")
            print("  old:", json.dumps(old.get(name))[:200])
            print("  new:", json.dumps(new.get(name))[:200])

asyncio.run(main())
# python diff_tools.py @org/server@1.0.1 @org/server@1.1.0

For authors: make this easy for your users

If you publish skills or servers, you can make everyone downstream safer:

FAQ

How do I know when an agent skill I use has changed? Pin it to a commit, then either poll the GitHub commits API for its path daily or use a tracker like backskills that sends a signed webhook within 24 hours of a change.

How do I pin an MCP server version? Put the exact version in the install command (package@1.0.1, package==1.0.1, or an image digest). Avoid bare package names that resolve to latest.

Can an MCP server change its tools without me knowing? Yes, if you run unpinned or use a remote server. Pin where you can, and diff the tool list when a new version appears.

How do I roll back a skill? If you vendored it at a SHA, revert the PR that bumped the pin. That's the whole point of the lockfile.

Is version history available for skills that don't publish versions? Git history is the version history. Trackers like backskills record each observed revision, so you can pin even when the author never tagged anything.

Start watching the skills and MCP servers your agents depend on, free: backskills.ai.