I've had Slack bot on the to-do list for a whilst, in addition to recreating an SE coaching bot that I created internally by using Gemini Gems, NotebookLMs and workflows. The Coaching concept centers around MEDDPICC deal qualification, objection handling, competitive positioning, demo planning. Combining the two gave me a real use case rather than a hello-world bot, and a reason to build the retrieval and security layers properly rather than bolting them on afterwards. I pivoted away from Cyber as it's too close to work, and much like the GTM Playbook website isn't a repo I'd want to make public.

The result is se-coaching-bot: a Slack-native assistant that coaches sales engineers using a synthetic corpus built around a fictional AI vendor called Corvus AI and four fictional enterprise customers. It runs on AWS (Bedrock, Knowledge Bases, Lambda, API Gateway) and was built test-first from the first commit across seven chat sessions. All content is synthetic; no employer IP.

As the build progressed, the security gateway became the most interesting part of the project: PII redaction, injection screening, RBAC, and intent classification all running before and after the model sees anything. The retrieval is managed (Bedrock Knowledge Bases); the gateway is hand-built. If you read the regulatory architecture post, this is the system that post was drawn from.

Architecture

Slack sends an app mention or DM to API Gateway. A receiver Lambda verifies the Slack signature, acknowledges within the three-second budget, adds an eyes reaction so the user knows it heard them, and invokes the worker asynchronously. The worker walks the security gateway and, only if the request clears, runs the engine. Every outcome resolves the eyes to a terminal reaction with a reply in-thread: a tick for a result, a no-entry sign for a security rejection, a question mark for unclear intent, a cross for a failure.

Slack @mention or DM API Gateway Public endpoint Private subnet Receiver verify · ack · react async Worker gateway · engine Security gateway pipeline redact → injection screen → classify → RBAC → engine → outbound redact Bedrock Claude · KB SQS + Reaper Case 3 fallback on-failure

The async split is the first decision worth explaining. Slack's Events API gives you three seconds to respond with a 200, or it retries. Any model call blows that budget. So the receiver does the minimum (verify, ack, react) and hands off; the worker has as long as Lambda allows. The two functions share no state beyond the event payload, and the receiver's IAM role can invoke the worker but cannot touch Bedrock, S3, or Secrets Manager. That asymmetry is deliberate: the public-facing function has the smallest possible blast radius.

Where to place the eyes reaction was a genuine debate during the build. Eyes-in-worker is architecturally cleaner (the receiver holds nothing beyond the signing secret), but it reintroduces the latency the eyes were meant to mask. On a cold worker, the user types a message and watches nothing happen for ten, twenty, thirty seconds before the eyes even appear. The reaction is feedback; feedback that arrives after a cold start isn't feedback. Eyes-in-receiver puts the acknowledgement on the message within the same three seconds Slack is already holding open. The receiver needs the bot token for this, scoped to reactions:write and chat:write, but a leaked token at that scope lets an attacker add emoji and post messages; it cannot reach Bedrock, the KB, or the spend.

The security gateway

The gateway pipeline runs in a fixed order: redact, screen for injection, classify intent, check RBAC. Retrieved chunks pass a separate context screen before reaching the prompt. Replies pass the redactor on the way out.

Two structural rules, both from the start rather than retrofitted. The system/user prompt split is a security boundary: policy (the MEDDPICC rubric, the skill instructions) sits in the system channel; user text and retrieved context sit in the user channel. Retrieved text is never trusted as instruction, because the indirect injection vector arrives wearing the corpus's credibility rather than the user's.

The injection screen is pattern-based and deterministic rather than model-based. It runs ahead of classification, so a prompt-injection attempt should be caught before the classifier spends a model call on it. That ordering matters: if classification ran first, an injection disguised as a coaching question would clear the classifier and reach the engine with the corpus context in front of it.

RBAC is the last inbound stage. Assignments come from SSM (seeded outside Terraform, managed with ignore_changes), and the role matrix is parity-tested against a YAML fixture so the code and the documentation cannot drift apart. A user with the se role can use all four skills; an observer can read but not invoke; an unassigned user is rejected before the engine is constructed.

The redactor had a bug that only appeared during integration testing. The outbound path was applying the full inbound redactor chain, including the person-name heuristic, to the bot's generated replies. The heuristic was tuned for short conversational Slack messages where a capitalised word pair is likely a name. In a KB-grounded reply dense with corpus proper nouns ("In-Country Dedicated", "SOC 2 Type II"), it shredded the answer into [REDACTED:person_name] noise. The fix was a protocol-level split: redact for inbound (full chain including the name heuristic), redact_structured for outbound (card numbers, NINOs, emails only). Putting the distinction in the Redactor protocol means any future implementation inherits the inbound/outbound contract as an obligation. And the inbound name-redaction already guarantees a real user name cannot appear in the reply; the model never held it to echo back.

Four skill and no slash commands

The bot has four skills: qualify (MEDDPICC assessment against a pasted transcript), objection handling, competitive positioning, and demo planning. They are classified from natural language rather than slash commands, because a coaching assistant that requires command syntax is not a true coaching assistant. The classifier is a structured model call returning one of six labels: the four skills, unclear, and out_of_scope. Unclear intent asks for clarification rather than guessing. Out-of-scope requests are declined.

Qualify is the only skill that takes no retrieval. The SE pastes a transcript; the engine sends it to the model with the MEDDPICC rubric as policy; the model returns a structured qualification with a score and a finding per element. The rubric is policy, not corpus: it lives in content/policy/, is bundled into the Lambda package, loads into the system channel, and is excluded from Knowledge Base ingestion. If it were retrievable, the model could retrieve its own instructions as evidence and cite them as findings; the architectural version of a student marking their own homework.

The three retrieval skills share a shape: retrieve from the Knowledge Base, screen the chunks for indirect injection, assemble a prompt from the skill's instructions plus the surviving context, call the model, return the reply. The Knowledge Base is Bedrock-managed with Titan Embeddings v2 and S3 Vectors; the ingestion boundary is enforced by IAM (only content/kb/ is synced, and the ingestion role is pinned to that prefix). An ingestion-boundary eval proves this against the live Knowledge Base using provenance as the invariant, with a positive control so an empty index cannot pass it vacuously.

Failure modes

Three cases, all contracted during the build rather than hoped away.

Case 1: the worker throws. A try/finally guarantees the eyes reaction resolves to a cross and a generic apology lands in-thread. The exception does not propagate to the caller, because an escaping exception would mark the invocation failed and, once the DLQ reaper fires, send a duplicate apology.

Case 2: the reply post to Slack fails. Bounded retry with exponential backoff (three attempts, 0.5s / 1s / 2s), then a structured log and give up. The reaction swap is attempted separately, so the user gets a terminal signal from at least one of them.

Case 3: the worker Lambda never starts. Nothing in the worker can apologise for this, because nothing in it ran. Lambda's on-failure destination routes the abandoned invocation to SQS, and a reaper Lambda swaps the eyes for a cross and replies in-thread. This is the only case where the user's message sits with no resolution, and it is the one I spent the most time on, because a hanging eyes reaction is the worst outcome in the system: the user believes it is still working.

A config-poisoning bug was caught during handler testing: a partial SSM read left a half-built state dict cached in the worker's module-level _state, so subsequent invocations silently operated on incomplete config. The deliberately-expected-to-fail test that exposed this became a test that guards against it.

Infrastructure

Lambda, zip-plus-layer packaging, no containers. That is an architectural KISS decision and not an oversight. The receiver is stdlib Python plus runtime boto3; the worker adds the application package as a layer. Terraform lives under infra/ with an S3 backend, applied by CI (GitHub Actions with OIDC) rather than locally. No tfvars, no local-apply dependency. Values seeded outside Terraform (the bot token in Secrets Manager, RBAC assignments in SSM) use the shell-parameter pattern: Terraform owns the resource but ignore_changes on the value, so the resource exists in state without the contents being fought over on every apply.

How the build went

The project was built across numerous chat sessions and the architecture benefited from the breaks between them. The first session designed the content pack and settled the architecture through structured reasoning. The second scaffolded the repo test-first with nothing but a smoke test, a CI workflow, and a Makefile. The third built infrastructure and the engine substrate, but ended with me dissatisfied: design decisions kept surfacing mid-build, and the session spent more time splitting and reordering issues than writing code (something I consistently found when using Opus 4.8). I went for a clean handoff document and started fresh.

The fourth session replanned everything into four numbered steps and executed them cleanly: Lambda infrastructure, security gateway, engine wiring, eval harness. This was the most productive session because the planning was done before the building started. The fifth built the DLQ/reaper (Case 3) and the answer-quality eval for the retrieval skills. A short fixes session patched the outbound redaction bug. The seventh was a final code review and hygiene pass.

What is measured

The eval harness runs two tiers: an offline pre-flight (content invariants, schema validation, ingestion boundary proof) that gates the graded layers, and the graded layers themselves (classification accuracy, retrieval precision, qualification grading, answer quality). The pre-flight hard-stops: grading against malformed ground truth produces numbers that look like results.

Qualification grading is the arithmetic layer. Twenty transcripts with eight-element ground truth, graded across three models. The headline finding from the three-model comparison was an accuracy/obedience split: Haiku scored the highest agreement at 0.76 but never once emitted instruction-conformant output (every response needed repair); GPT-oss was the mirror image at 20/20 clean output but mid-pack accuracy. That finding only exists because the parse records how much repair each response needed via a Repair enum, rather than scoring presentation habits as capability failures.

Answer quality is the harder layer, and the next post covers it properly. The short version: each scenario carries expected points, named traps, and a grounding expectation; two independent judges score the answer; invention rate is pooled over substantive factual claims. It took nine calibration runs on the objection set alone to settle the rubric.

The source code is on GitHub.