๐Ÿ”’ PAZ-297 โ€” Slack App Configuration Tokens

Pre-Implementation QA Testing Plan

๐Ÿ“… 2026-04-13 ๐ŸŽฏ Staging (pazi.dev) ๐Ÿ“‹ PRE-QA Plan ๐Ÿ”ง Cross-cutting: API + Frontend + Agent Repo
17
API Tests
16
Browser Tests
8
Categories
33
Total Tests
๐Ÿ“‹ Ticket Information
TicketPAZ-297 โ€” Change how users connect Slack
ScopeComplete overhaul of Slack connection flow for new users using Slack App Configuration Tokens
ComponentsAPI (new endpoints, models, token storage, webhooks), Frontend (new setup UI), Agent Repo (Connect Slack tool, auth hook)
EnvironmentStaging (pazi.dev)
Test Accountqa-agent@pazi.ai / PaziQA2026!
๐Ÿ“ Task Summary
What's changing: Instead of manually creating Slack apps, users provide App Configuration Tokens (access + refresh). Pazi's API then programmatically creates Slack apps per-agent via apps.manifest.create, handles token rotation (12h expiry), generates OAuth authorize URLs, registers per-user webhooks for channel events, and enforces user-level DM authorization where only the agent creator and channel co-members can talk to the agent.
Key complexity: This is a cross-cutting change touching 3 repositories. Token security (encryption at rest, rotation), per-agent app isolation, webhook scoping, and backward compatibility with existing Slack connections are all critical areas.
โš™๏ธ Prerequisites
  1. Deploy PAZ-297 feature branches (API + frontend + agent repo) to the staging environment
  2. Ensure the staging environment's MongoDB has the new schema/collection for config tokens
  3. Have a Slack workspace available for testing (Pythagora test workspace)
  4. Have app configuration tokens generated from api.slack.com/apps โ†’ "Your App Configuration Tokens" โ†’ "Generate Token"
  5. Log into the staging environment with qa-agent@pazi.ai
  6. Ensure at least 1 pre-provisioned workspace is available (poolSize โ‰ฅ 1)
  7. Have a second Slack user ID available for multi-user authorization testing
โš ๏ธ Risk Areas
๐Ÿ”ด Token SecurityConfig tokens must be encrypted at rest. Plaintext storage = critical vulnerability. Access tokens expire in 12h โ€” refresh logic must work transparently.
๐Ÿ”ด Cross-user IsolationUser A must never be able to create apps with User B's tokens or access User B's agents. Webhook events must be scoped per-user.
๐Ÿ”ด Backward CompatibilityExisting Slack connections (old bot+app token flow) must continue working. No silent migration or data loss.
๐ŸŸ  Token RotationWhen the 12h access token expires, tooling.tokens.rotate must be called with the refresh token. Both new tokens must be stored. Failure = all Slack operations break.
๐ŸŸ  Per-agent App CreationEach agent needs its own Slack app with correct name/avatar. Manifest must include all required scopes and event subscriptions.
๐ŸŸ  Authorization ModelOnly agent creator + channel co-members should be authorized. Webhook must correctly process member_joined_channel events. Unauthorized users must get rejection.
๐ŸŸก Error HandlingSlack API failures, network timeouts, invalid manifests โ€” all need clear user-facing errors with recovery steps.
๐ŸŸก State PersistenceAuthorization lists, Slack connection state, and webhook registrations must survive agent restarts (recurring bug pattern).
๐Ÿ’ก Implementation Guidance
Token storage location: Config tokens should be stored on the User model (or a dedicated collection referenced by userId) โ€” NOT on the Agent model. They're per-user, per-workspace credentials. Encrypt with AES-256-GCM using an env-var key. Store iv, authTag, and ciphertext separately.
Slack API flow: apps.manifest.create returns app_id, credentials (client_id, client_secret, signing_secret), and oauth_authorize_url. After app creation, the bot token + app-level token still need to be obtained โ€” the user installs via OAuth, then those tokens come back via the OAuth callback. The API needs an OAuth callback handler.
Avatar upload: Setting the Slack app icon requires a separate API call (apps.manifest.update or manual upload). The apps.manifest.create manifest schema doesn't have an icon field. The developer should use POST https://slack.com/api/apps.icon.upload (undocumented but functional with config token) or note this as a limitation.
Webhook architecture: Two approaches โ€” (1) single webhook endpoint with user-routing based on app_id, or (2) per-user webhook endpoints. Approach (1) is simpler. The webhook URL goes in the app manifest's request_url field, but since socket mode is enabled, event delivery is via WebSocket, not HTTP. The webhook in the ticket description likely refers to an internal API endpoint that processes Slack events received via socket mode.
๐Ÿ” Category 1: Token Storage & Security (API)
API-1.1 Store app configuration tokens โ€” valid tokens accepted HIGH
Steps:
  1. Authenticate as qa-agent@pazi.ai and obtain a session cookie
  2. Send POST to /api/slack/config-tokens (or actual endpoint) with valid access and refresh tokens
  3. Verify 200 response with success indicator
  4. Query MongoDB directly to verify tokens are stored encrypted (not plaintext)
  5. Verify tokens are associated with the authenticated user's profile
Expected: 200 response. Tokens stored encrypted in MongoDB under the user's document/collection. Raw DB query should show ciphertext, not xoxe.xoxp-... plaintext.
curl -X POST ${API_URL}/slack/config-tokens \
  -H 'Cookie: <session>' \
  -H 'Content-Type: application/json' \
  -d '{"accessToken":"xoxe.xoxp-VALID","refreshToken":"xoxe-VALID"}'
API-1.2 Store tokens โ€” missing accessToken rejected HIGH
Steps:
  1. Send POST with only refreshToken field (no accessToken)
  2. Verify 400 response
Expected: 400 with clear error message indicating accessToken is required.
curl -X POST ${API_URL}/slack/config-tokens \
  -H 'Cookie: <session>' \
  -H 'Content-Type: application/json' \
  -d '{"refreshToken":"xoxe-VALID"}'
API-1.3 Store tokens โ€” missing refreshToken rejected NORMAL
Steps:
  1. Send POST with only accessToken field (no refreshToken)
  2. Verify 400 response
Expected: 400 with clear error indicating refreshToken is required.
curl -X POST ${API_URL}/slack/config-tokens \
  -H 'Cookie: <session>' \
  -H 'Content-Type: application/json' \
  -d '{"accessToken":"xoxe.xoxp-VALID"}'
API-1.4 Store tokens โ€” invalid token format rejected NORMAL
Steps:
  1. Send POST with tokens that don't match expected prefixes (xoxe.xoxp- for access, xoxe- for refresh)
  2. Verify 400 response with format guidance
Expected: 400 error indicating invalid token format. Access tokens should start with xoxe.xoxp-, refresh tokens with xoxe-.
curl -X POST ${API_URL}/slack/config-tokens \
  -H 'Cookie: <session>' \
  -H 'Content-Type: application/json' \
  -d '{"accessToken":"invalid-format","refreshToken":"also-invalid"}'
API-1.5 Store tokens โ€” unauthenticated request rejected HIGH
Steps:
  1. Send POST without any session cookie
  2. Verify 401 response
  3. Verify no tokens are stored in DB
Expected: 401 Unauthorized. No tokens stored for anonymous requests.
curl -X POST ${API_URL}/slack/config-tokens \
  -H 'Content-Type: application/json' \
  -d '{"accessToken":"xoxe.xoxp-VALID","refreshToken":"xoxe-VALID"}'
API-1.6 Store tokens โ€” whitespace-only tokens rejected NORMAL
Steps:
  1. Send POST with whitespace-only values for both tokens
  2. Verify 400 response
Expected: 400. Whitespace is trimmed and treated as empty (recurring bug pattern โ€” see whitespace-only input pattern in knowledgebase).
curl -X POST ${API_URL}/slack/config-tokens \
  -H 'Cookie: <session>' \
  -H 'Content-Type: application/json' \
  -d '{"accessToken":"   ","refreshToken":"  "}'
API-3.1 Token encryption at rest โ€” verify DB storage HIGH
Steps:
  1. After storing valid tokens via API-1.1, connect to the staging MongoDB
  2. Query the relevant collection for the test user's tokens
  3. Inspect the stored fields โ€” should be ciphertext, not plaintext
  4. Verify encryption key is in environment variables, not in DB
Expected: Token values in MongoDB are encrypted binary/base64, NOT readable xoxe.xoxp-... strings. Encryption key stored in env vars (e.g., SLACK_TOKEN_ENCRYPTION_KEY).
mongosh --eval 'db.users.findOne({email:"qa-agent@pazi.ai"}, {slackConfigTokens:1})'
# OR check a dedicated collection:
mongosh --eval 'db.slackconfigtokens.findOne({userId: ObjectId("...")})'
API-3.2 Token isolation โ€” user A cannot read user B's tokens HIGH
Steps:
  1. Store tokens for User A
  2. Authenticate as User B
  3. Attempt to read or use User A's config tokens (via any endpoint)
  4. Verify isolation
Expected: User B only sees their own token status. No endpoint leaks another user's token values. Token values should never be returned in full โ€” at most last 4 chars for display.
curl -X GET ${API_URL}/slack/config-tokens -H 'Cookie: <session_user_B>'
API-7.1 Overwrite existing tokens โ€” update replaces old tokens NORMAL
Steps:
  1. Store initial tokens
  2. Store new tokens for the same user
  3. Verify DB has only the new tokens (upsert behavior)
  4. Verify old tokens no longer work for app creation
Expected: New tokens replace old ones (upsert, no duplicates). Subsequent operations use the new tokens.
API-7.2 Rate limiting on token storage endpoint NORMAL
Steps:
  1. Send 20 rapid POST requests to the token storage endpoint
  2. Check for 429 responses
  3. Verify no duplicate entries in DB
Expected: Rate limiting returns 429 after threshold. No duplicate DB entries.
for i in $(seq 1 20); do
  curl -s -o /dev/null -w '%{http_code}\n' -X POST ${API_URL}/slack/config-tokens \
    -H 'Cookie: <session>' -H 'Content-Type: application/json' \
    -d '{"accessToken":"xoxe.xoxp-TOKEN","refreshToken":"xoxe-REFRESH"}'
done
๐Ÿ—๏ธ Category 2: Slack App Creation (API)
API-2.1 Create Slack app for agent โ€” happy path HIGH
Steps:
  1. Ensure valid config tokens are stored for the user
  2. Send POST to create a Slack app for a specific agent
  3. Verify response contains app_id, credentials, and oauth_authorize_url
  4. Verify the app appears at api.slack.com/apps
  5. Verify the app name matches the agent's display name
  6. Verify manifest includes required scopes and event subscriptions
Expected: 200 with app_id, client_id, client_secret, signing_secret, and oauth_authorize_url. App visible at api.slack.com/apps with correct agent name and manifest configuration.
curl -X POST ${API_URL}/slack/create-app \
  -H 'Cookie: <session>' \
  -H 'Content-Type: application/json' \
  -d '{"agentId":"<agent_id>"}'
API-2.2 Create Slack app โ€” no config tokens stored HIGH
Steps:
  1. Authenticate as a user who has NOT stored config tokens
  2. Attempt to create a Slack app
  3. Verify clear error directing user to set up tokens first
Expected: 400 or 422 with message like "Please set up your Slack configuration tokens first" with instructions.
curl -X POST ${API_URL}/slack/create-app \
  -H 'Cookie: <session_without_tokens>' \
  -H 'Content-Type: application/json' \
  -d '{"agentId":"<agent_id>"}'
API-2.3 Create Slack app โ€” expired token triggers automatic refresh HIGH
Steps:
  1. Store valid config tokens
  2. Wait >12 hours OR manually expire the access token in DB
  3. Attempt to create a Slack app
  4. Verify the API transparently calls tooling.tokens.rotate
  5. Verify both new access and refresh tokens are stored in DB
  6. Verify app creation succeeds with the refreshed token
Expected: 200 success. Token rotation happens transparently. New tokens stored in DB. User sees no error or extra step. The tooling.tokens.rotate endpoint returns new token, refresh_token, iat, and exp.
# Simulate expired token in DB:
mongosh --eval 'db.users.updateOne(
  {email:"qa-agent@pazi.ai"},
  {$set:{"slackConfigTokens.expiresAt": new Date("2020-01-01")}}
)'
# Then try app creation โ€” should auto-refresh
API-2.4 Create Slack app โ€” expired refresh token returns clear error NORMAL
Steps:
  1. Store tokens with an intentionally invalid/expired refresh token
  2. Expire the access token
  3. Attempt to create a Slack app
  4. Verify clear error message and no partial state
Expected: Clear error: "Your Slack configuration tokens have expired. Please generate new ones at api.slack.com/apps". No half-created app in the system.
API-2.5 Create Slack app โ€” agent name with special characters NORMAL
Steps:
  1. Create an agent named "๐Ÿค– My Super Agent With A Very Long Name!!!"
  2. Attempt to create a Slack app for it
  3. Verify the manifest name is sanitized (max 35 chars) and the slash command is valid
Expected: App created successfully. Name truncated to 35 chars. Slash command derived safely (no invalid characters). Manifest accepted by Slack API.
API-2.6 Create Slack app โ€” agent belonging to different user rejected HIGH
Steps:
  1. Authenticate as User A
  2. Attempt to create a Slack app for an agent belonging to User B
  3. Verify 403 Forbidden
Expected: 403 Forbidden. Cannot create Slack apps for another user's agents. No Slack app created. Critical security boundary.
curl -X POST ${API_URL}/slack/create-app \
  -H 'Cookie: <session_user_A>' \
  -H 'Content-Type: application/json' \
  -d '{"agentId":"<user_B_agent_id>"}'
๐Ÿ”— Category 3: Webhooks & Authorization (API)
API-4.1 Webhook registration โ€” per-user webhook created on app setup HIGH
Steps:
  1. Complete Slack app creation and installation
  2. Verify a webhook endpoint exists that handles Slack events for this user
  3. Send a Slack verification challenge to the webhook
  4. Verify the webhook responds correctly
  5. Verify the webhook rejects events from other users' workspaces
Expected: Webhook endpoint responds to Slack's url_verification challenge. Events are scoped to the specific user. Cross-user events are rejected.
curl -X POST ${API_URL}/slack/webhook \
  -H 'Content-Type: application/json' \
  -d '{"type":"url_verification","challenge":"test_challenge_string"}'
API-4.2 Webhook โ€” member_joined_channel adds user to authorized list HIGH
Steps:
  1. Set up a channel where the agent bot is present
  2. A new user joins the channel
  3. Verify member_joined_channel event is received by webhook
  4. Verify the new user is added to the agent's authorized users list
  5. Verify the user can now DM the agent
  6. Verify the authorization list persists across agent restarts
Expected: User joining a channel with the agent โ†’ automatically authorized to DM the agent. Authorization persists across restarts.
curl -X POST ${API_URL}/slack/webhook \
  -H 'Content-Type: application/json' \
  -d '{"type":"event_callback","event":{"type":"member_joined_channel","user":"U_NEW","channel":"C_AGENT"}}'
API-4.3 Webhook โ€” member_left_channel behavior NORMAL
Steps:
  1. Have an authorized user (from channel join) leave the channel
  2. Attempt to DM the agent from that user
  3. Document the behavior: is authorization revoked or preserved?
Expected: Behavior needs to be defined. Document whether leaving a channel revokes DM access. Either way, verify it's intentional and consistent.
API-5.1 Unauthorized DM โ€” rejection message sent HIGH
Steps:
  1. From a Slack user who is NOT authorized (not the creator, not in any shared channel), send a DM to the agent
  2. Verify the rejection message is sent
  3. Verify the message is NOT processed by the agent
  4. Check agent logs for the rejection
Expected: Unauthorized user receives "You are not authorized to talk to this agent." (or similar). Message is NOT processed. Agent logs show the rejection event.
API-5.2 Creator is always authorized HIGH
Steps:
  1. From the agent creator's Slack account, DM the agent
  2. Verify the agent processes the message and responds normally
  3. Verify no rejection message is shown
Expected: Creator always authorized. Normal agent response. No rejection.
๐Ÿ”ง Category 4: Connect Slack Tool (Agent Repo)
API-6.1 Connect Slack tool โ€” agent has the tool available HIGH
Steps:
  1. Check the agent's registered tools via API or agent config
  2. Verify "Connect Slack" (or equivalent) is in the list
  3. Verify the tool can be invoked
Expected: The "Connect Slack" tool is registered and available in the agent's tool manifest. It can be invoked by the agent in response to user requests.
curl -X GET ${API_URL}/agents/<agentId>/tools -H 'Cookie: <session>'
๐Ÿ–ฅ๏ธ Category 5: Frontend UX (Browser Tests)
PW-1.1 New user โ€” Connect Slack shows config token instructions HIGH
Steps:
  1. Navigate to /login
  2. Log in with qa-agent@pazi.ai / PaziQA2026!
  3. Create a new agent or select an existing one
  4. Click "Agent setup" in the left sidebar (gear icon)
  5. Find the Slack section and click "Connect Slack"
  6. Observe the setup instructions
Expected: Instructions say to generate "App Configuration Tokens" from api.slack.com/apps. Input fields for Access Token (xoxe.xoxp-...) and Refresh Token (xoxe-...). The OLD flow (create app from manifest, install, paste bot/app tokens) should NOT appear for new users.
PW-1.2 New user โ€” Submit config tokens triggers app creation HIGH
Steps:
  1. Navigate to Slack connection setup
  2. Enter a valid App Configuration Access Token
  3. Enter a valid Refresh Token
  4. Click submit/save
  5. Observe the response: tokens saved, app created, authorize URL presented
Expected: After submitting: (1) tokens saved, (2) Slack app created for current agent, (3) app has agent's name and avatar, (4) OAuth authorization URL presented via new tab or in-chat button.
PW-1.3 New user โ€” OAuth authorization button/link appears HIGH
Steps:
  1. Complete PW-1.2 (tokens submitted, app created)
  2. Look for an authorization button/link
  3. Click it
  4. Verify Slack OAuth consent screen loads at slack.com/oauth/v2/authorize?...
Expected: Either a new tab opens to slack.com/oauth/v2/authorize?client_id=..., OR a button in the chat says "Authorize in Slack" / "Install to your workspace". Clicking leads to Slack's OAuth consent screen.
PW-2.1 Second agent โ€” auto-creates app using existing tokens HIGH
Steps:
  1. Log in with account that already has stored config tokens
  2. Create a second agent named "Support Bot"
  3. Click "Connect Slack" for the new agent
  4. Observe: should NOT ask for tokens again
Expected: System reuses stored config tokens. Automatically creates a new Slack app for "Support Bot". Presents a new authorization URL. The new app has the second agent's name/avatar.
PW-2.2 Second agent โ€” app has correct name and avatar NORMAL
Steps:
  1. After PW-2.1, go to api.slack.com/apps
  2. Find the newly created app
  3. Verify name and icon
Expected: Slack app for second agent has name "Support Bot" and the correct avatar. NOT the first agent's name or generic "Pazi Agent".
PW-4.1 Token input validation โ€” empty fields disable submit NORMAL
Steps:
  1. Navigate to Slack config token setup
  2. Leave both fields empty โ†’ observe submit button (should be disabled)
  3. Fill access token only โ†’ observe (should be disabled)
  4. Fill refresh token only โ†’ observe (should be disabled)
  5. Fill both โ†’ observe (should be enabled)
Expected: Submit disabled unless both fields are non-empty.
PW-4.2 Token input โ€” password-masked fields NORMAL
Steps:
  1. Navigate to the token setup form
  2. Type a token value in the access token field
  3. Observe: field should show dots/bullets, not plaintext
Expected: Token fields are type="password" โ€” values masked. Prevents shoulder-surfing.
PW-9.1 Double-click prevention โ€” submit button debounced NORMAL
Steps:
  1. Fill in valid config tokens
  2. Rapidly double-click the submit button
  3. Observe: only 1 request, button shows "Connecting..." and is disabled
Expected: One API request. Button disabled during request. No duplicate apps created. No duplicate DB entries.
๐Ÿ›ก๏ธ Category 6: DM Authorization (Browser Tests)
PW-3.1 Only creator can DM the agent initially HIGH
Steps:
  1. Complete Slack app setup and installation
  2. From creator's Slack: DM the agent with "Hello, are you working?"
  3. Verify normal agent response
  4. From unauthorized Slack user: DM the agent with "Hey can you help me?"
  5. Verify rejection message
Expected: Creator gets a normal response. Unauthorized user gets "You are not authorized to talk to this agent." Agent does NOT process the unauthorized message.
PW-3.2 Channel join authorizes members HIGH
Steps:
  1. Create channel #test-agent-channel
  2. Invite the agent bot to the channel
  3. Invite a second user (previously unauthorized)
  4. From the second user, DM the agent
  5. Verify normal response (no rejection)
Expected: After joining a shared channel, the second user can DM the agent normally. No rejection.
PW-3.3 Rapid unauthorized DMs โ€” rate limiting NORMAL
Steps:
  1. From unauthorized Slack user, send 5 DMs within 10 seconds
  2. Count rejection messages received
Expected: Only 1 rejection message within the 60-second rate limit window. Other 4 messages silently dropped.
โŒ Category 7: Error Handling (Browser Tests)
PW-5.1 Slack API error during app creation NORMAL
Steps:
  1. Store invalid/expired config tokens
  2. Trigger app creation
  3. Observe error message
Expected: User-friendly error (not raw API error). Suggests: "Your configuration tokens may have expired. Please generate new ones." No partial state in DB.
PW-5.2 Network timeout during Slack API call NORMAL
Steps:
  1. Start Connect Slack flow
  2. Simulate network issue during API call
  3. Observe: should not hang forever
Expected: After 10-30s timeout: "Connection to Slack timed out. Please try again." Submit button re-enabled for retry.
๐Ÿ”„ Category 8: Backward Compatibility & Stability
PW-6.1 Connect Slack tool โ€” agent can trigger connection via tool HIGH
Steps:
  1. Open dashboard chat with an agent
  2. Type "Connect to Slack"
  3. Observe: agent should invoke the Connect Slack tool
  4. If tokens exist: proceeds to app creation
  5. If no tokens: guides user through token setup
Expected: Agent recognizes intent, invokes "Connect Slack" tool, and either creates the app (tokens exist) or guides token setup (no tokens).
PW-7.1 Existing Slack connections โ€” backward compatibility HIGH
Steps:
  1. Log in with account that has existing old-style Slack connection (bot + app tokens)
  2. Navigate to Slack config / Agent setup
  3. Verify existing connection still works
  4. Verify edit flow still shows bot/app token fields
  5. Send a Slack message to the agent and confirm it responds
Expected: Existing connections continue working. No forced migration. Edit flow shows old-style fields. Agent responds to messages normally.
PW-7.2 No silent data loss during migration NORMAL
Steps:
  1. Before deploying, snapshot existing user's Slack config in DB
  2. Deploy PAZ-297
  3. Re-check the same user's DB record
  4. Verify no token fields were deleted or overwritten
Expected: No existing botToken/appToken deleted. Migration (if any) is additive only. Existing connections unaffected.
PW-8.1 Mobile responsive โ€” token setup on mobile viewport NORMAL
Steps:
  1. Set viewport to 375ร—812 (iPhone)
  2. Navigate to Slack connection setup
  3. Verify all elements visible and usable
Expected: Token fields full-width, instructions readable, submit button tappable, no horizontal overflow.
PW-10.1 Page refresh โ€” token fields cleared (no localStorage leak) NORMAL
Steps:
  1. Enter tokens in the setup form (don't submit)
  2. Refresh the page (F5)
  3. Observe: fields should be empty
Expected: Token fields empty after refresh. Sensitive tokens NOT stored in localStorage (security requirement, also avoids quota exhaustion pattern).
PW-10.2 Agent restart โ€” Slack connection survives HIGH
Steps:
  1. Complete full Slack setup (tokens โ†’ app โ†’ install โ†’ working DMs)
  2. Restart the agent via gateway restart or settings
  3. Send a DM to the agent via Slack
  4. Verify response
Expected: After restart: Slack connection active, DMs processed, authorization lists preserved. No re-setup needed. (Recurring bug pattern: state not persisted to disk.)