One Slack channel for every deploy and alarm

A client's platform (I'll call it Northwind here) posted its production alarms into a shared channel in my old agency Slack. Deploy notices went out by email only. The client has its own Slack now, so I moved everything into one channel there, #notifications, and gave every app the same message format.

What was sending what

Before changing anything, I listed every source. There were fewer than I expected:

One surprise: dev and prod used the same webhook URL. So a staging alarm and a production alarm landed in the same place, and nothing in the setup said so.

flowchart LR
    CW["CloudWatch alarm"] --> SNS["SNS topic"] --> L["Relay Lambda"]
    GA["GitHub Actions deploy"] --> S
    AM["Amplify build"] --> W["deploy-notify workflow"] --> S
    L --> S["#notifications"]

Agree on the format before writing it

My first proposal was a list of fields in a chat window, and that was hard to judge. So I built a small HTML page that looks like the Slack channel, with one example day: deploys to dev, a failed build, a build whose status is unknown, a CI failure, a prod release and an alarm. It got a "looks good" straight away.

Every deploy message has three lines and a colour bar:

1. Result, app, stage and build number (or version).
2. The first line of the commit message, with the PR linked.
3. Author, short SHA, and links to the app, the build and the run.

Merge first, switch on later

I wanted the PRs to be safe to merge before the webhook existed. GitHub Actions doesn't let you test secrets.* in a step's if, but you can map the secret to a job-level env and test that:

jobs:
  notify:
    env:
      SLACK_WEBHOOK_URL: ${{ secrets.SLACK_DEPLOY_WEBHOOK_URL }}
    steps:
      - name: Notify Slack
        if: env.SLACK_WEBHOOK_URL != ''
        continue-on-error: true
        uses: slackapi/[email protected]
        with:
          webhook: ${{ env.SLACK_WEBHOOK_URL }}
          webhook-type: incoming-webhook

With no secret, the step is skipped. With the secret, it posts. continue-on-error means a Slack outage never turns a good deploy red. In the Amplify workflows, the commit message goes into the payload through jq --arg, so quotes or $(...) in a commit message stay plain text.

Swapping the webhook

The alarm relay reads its webhook URL from Secrets Manager, and the CDK stack only references the secret by name. CDK never owns the value, so a deploy can't overwrite a URL someone filled in by hand. Rotating it was one put-secret-value per account.

A Slack webhook belongs to one app, and an app belongs to one workspace forever. The old app couldn't be moved, so I created a new app in the client's workspace and pointed one webhook at #notifications. A test message through each SNS topic, dev and prod, showed up right away.

What it took

Most of the work was finding out what existed. The code changes were small. Next time I'd build the HTML mock first and list the sources second. People agree on a format much faster when they can see it, and the mock also shows which fields you forgot.