Every signup, password reset and invite flow ends in an email, and that email is the step most end-to-end suites skip. This guide adds it to your pipeline. You will test four things about the email your app sends, run those tests in two layers (a local SMTP catcher on every pull request, and hosted Mailsac inboxes for capture and real-delivery checks), and wire both into GitHub Actions and GitLab CI.
The complete project is the playwright-signup-ci example in Mailsac’s integration-test examples repository: a small Express app that sends a verification link and code, one Playwright test, and both CI configurations. Every code block below is copied from it. Clone it, or copy the pieces into your own repository.
What to test in an email
Four things break, roughly in this order:
- The link. The URL in the email must open the right page with a valid token. The classic bug is an email that says
http://localhost:3000/verify?... because the deployed environment inherited a development APP_URL.
- The code. The 6-digit code in the email must be the one the app accepts, and it must still be accepted after the email has been rendered, wrapped and copied.
- The content. The subject, sender and recipient are right, there is a plain-text part, and no
{{firstName}} placeholders leak through.
- Delivery. The email actually leaves the app through your email provider, with the credentials, sending domain and DNS records (SPF and DKIM) you ship with, and lands in a real mailbox.
The first three can be tested without any mail leaving CI. The fourth cannot. That is why this guide uses two layers.
Two layers, three CI jobs
| Layer |
CI job |
Runs |
Mail goes to |
Proves |
| Local SMTP catcher |
mailpit |
every pull request |
a Mailpit service container beside the app, port 1025 |
the app sends the right email with a working link and code |
| Hosted test inbox |
mailsac-capture |
every run where MAILSAC_API_KEY is set |
Mailsac Email Capture: the app’s SMTP points at capture.mailsac.com:5587; the test reads the message over the Mailsac API |
the same, with inboxes that any runner or person can read from anywhere |
| Hosted test inbox |
real-delivery |
pushes to main, a daily schedule, manual runs |
your deployed staging app sends through its real provider to a Mailsac inbox |
delivery works: provider credentials, sending domain, SPF and DKIM |
The local layer is fast and never leaves CI. The hosted layer catches what a local catcher cannot: expired provider credentials, DNS mistakes, a throttled or blocked sending domain, and templates that only break in production. The test code is identical in all three jobs; only environment variables change.
The example app and the test
The app under test is app/server.js: an Express form at / that takes an email address, then sends a message containing a confirmation link (/verify?token=...) and a 6-digit code with Nodemailer. It reads SMTP_HOST, SMTP_PORT, SMTP_SECURE, SMTP_USER and SMTP_PASS from the environment, plus APP_URL for the link and MAIL_FROM for the sender, so the same code sends to Mailpit, to Mailsac Email Capture, or to your real provider without changes.
The test reaches inboxes through one small interface in tests/inbox.ts. Excerpt from tests/inbox.ts (the interface, the address generator and the backend switch):
export interface TestInbox {
/** A new, unique address for one test. */
newAddress(prefix?: string): string;
/** Poll until an email for `address` arrives (optionally matching the subject). */
waitForEmail(address: string, opts?: { subject?: string | RegExp; timeoutMs?: number }): Promise<ReceivedEmail>;
/** Remove the message after the test, so private inboxes stay tidy. */
cleanup(address: string, id: string): Promise<void>;
}
const unique = (prefix: string) =>
`${prefix}-${Date.now().toString(36)}-${randomBytes(4).toString('hex')}`.toLowerCase();
// ...
export const testInbox = (): TestInbox => (process.env.INBOX === 'mailpit' ? new MailpitInbox() : new MailsacInbox());
MailsacInbox builds addresses on MAILSAC_DOMAIN (default mailsac.com) and uses three REST calls, each with your key in the Mailsac-Key header:
GET https://mailsac.com/api/addresses/{email}/messages lists the inbox, newest first, with a links array per message that Mailsac extracted from the text and HTML. Nothing has to be created before your app sends to the address.GET https://mailsac.com/api/text/{email}/{messageId} returns the plain text, which is where the code is matched.DELETE https://mailsac.com/api/addresses/{email}/messages/{messageId} removes the message when the test is done.
The wait is a plain polling loop. Excerpt from tests/inbox.ts (MailsacInbox.waitForEmail):
async waitForEmail(address: string, { subject, timeoutMs = 60_000 }: { subject?: string | RegExp; timeoutMs?: number } = {}) {
const inbox = encodeURIComponent(address);
const deadline = Date.now() + timeoutMs;
while (Date.now() < deadline) {
const messages: Array<{ _id: string; subject: string; links?: string[] }> =
await (await this.api(`/addresses/${inbox}/messages`)).json();
const message = messages.find((m) => matches(m.subject || '', subject));
if (message) {
const text = await (await this.api(`/text/${inbox}/${encodeURIComponent(message._id)}`)).text();
return { id: message._id, subject: message.subject, text, links: message.links?.length ? message.links : extractLinks(text) };
}
await sleep(2_000);
}
throw new Error(`No email for ${address} within ${timeoutMs / 1000}s`);
}
MailpitInbox does the same against a local Mailpit at MAILPIT_URL (default http://localhost:8025): GET /api/v1/search?query=to:"{address}", GET /api/v1/message/{ID} and DELETE /api/v1/messages, polling every 500 ms for up to 30 seconds. The whole file is about 110 lines and needs nothing beyond Node’s built-in fetch.
The test itself is tests/signup.spec.ts, complete:
// End-to-end signup verification: sign up with a fresh address, wait for the
// verification email, then confirm the account with the link and with the code.
import { test, expect } from '@playwright/test';
import { testInbox } from './inbox';
const inbox = testInbox();
test.describe('signup verification email', () => {
test('arrives and its link verifies the account', async ({ page }) => {
const address = inbox.newAddress('signup-link');
await page.goto('/');
await page.getByLabel('Email').fill(address);
await page.getByRole('button', { name: 'Sign up' }).click();
await expect(page.getByRole('heading', { name: 'Check your email' })).toBeVisible();
const email = await inbox.waitForEmail(address, { subject: 'Confirm your email address' });
// assert on content, not only on arrival
expect(email.text).toContain('Confirm your email address');
const link = email.links.find((l) => l.includes('/verify?token='));
expect(link, 'verification link in the email').toBeTruthy();
await page.goto(link!);
await expect(page.getByRole('heading', { name: 'Email verified' })).toBeVisible();
await expect(page.getByText(address)).toBeVisible();
await inbox.cleanup(address, email.id);
});
test('arrives with a 6-digit code that verifies the account', async ({ page }) => {
const address = inbox.newAddress('signup-code');
await page.goto('/');
await page.getByLabel('Email').fill(address);
await page.getByRole('button', { name: 'Sign up' }).click();
const email = await inbox.waitForEmail(address, { subject: 'Confirm your email address' });
const code = email.text.match(/\b(\d{6})\b/)?.[1];
expect(code, '6-digit code in the email').toMatch(/^\d{6}$/);
await page.getByLabel('6-digit code').fill(code!);
await page.getByRole('button', { name: 'Verify' }).click();
await expect(page.getByRole('heading', { name: 'Email verified' })).toBeVisible();
await inbox.cleanup(address, email.id);
});
});
playwright.config.ts starts the example app unless APP_URL points at a deployed one, allows one retry on CI, writes an HTML report on CI, and sets the test timeout above the 60-second email wait so a missing email fails with the helper’s clear message rather than a generic timeout:
import { defineConfig } from '@playwright/test';
const PORT = Number(process.env.PORT || 3000);
export default defineConfig({
testDir: './tests',
timeout: 90_000,
retries: process.env.CI ? 1 : 0,
reporter: process.env.CI ? [['list'], ['html', { open: 'never' }]] : 'list',
use: { baseURL: process.env.APP_URL || `http://localhost:${PORT}` },
// Starts the example app. In your project, point baseURL at your own app or staging URL
// and remove webServer if the app is already running.
webServer: process.env.APP_URL
? undefined
: { command: 'node app/server.js', url: `http://localhost:${PORT}`, reuseExistingServer: !process.env.CI },
});
Run it locally
After cd playwright-signup-ci, npm ci and npx playwright install chromium, the three ways to run it are, from the example’s README:
With Mailpit, no account needed:
docker run -d -p 1025:1025 -p 8025:8025 axllent/mailpit
INBOX=mailpit SMTP_HOST=localhost SMTP_PORT=1025 npx playwright test
With Mailsac, sending through Email Capture, which keeps every message in Mailsac instead of delivering it:
export MAILSAC_API_KEY=... # your API key
export SMTP_HOST=capture.mailsac.com SMTP_PORT=5587 SMTP_USER=your-mailsac-username SMTP_PASS=$MAILSAC_API_KEY
npx playwright test
Against your deployed app, which sends through your real provider to a Mailsac inbox:
APP_URL=https://staging.example.com MAILSAC_API_KEY=... npx playwright test
The example’s .env.example lists the same variables for a local .env; the only secret among them is the API key, which Email Capture also uses as the SMTP password.
GitHub Actions
Add the repository secret MAILSAC_API_KEY and the repository variables MAILSAC_USERNAME (for Email Capture), MAILSAC_DOMAIN (your private domain, optional) and STAGING_URL (for the real-delivery job) under Settings, Secrets and variables, Actions. Save this as .github/workflows/email-tests.yml. It is the example’s workflow file without the paths filters, the defaults.run.working-directory block, the cache-dependency-path lines, the two working-directory: . lines and the folder prefix on the report path, which exist there only because the example sits in the playwright-signup-ci/ subfolder of a larger repository:
name: Playwright signup email tests (playwright-signup-ci)
on:
pull_request:
push:
branches: [main]
schedule:
- cron: "17 6 * * *" # daily real-delivery check
workflow_dispatch:
jobs:
# 1. Every pull request: a local SMTP catcher. Fast, free, nothing leaves CI.
# Proves the app sends the right email with a working link and code.
mailpit:
runs-on: ubuntu-latest
services:
mailpit:
image: axllent/mailpit:latest
ports:
- 1025:1025
- 8025:8025
env:
INBOX: mailpit
SMTP_HOST: localhost
SMTP_PORT: "1025"
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 22
cache: npm
- run: npm ci
- run: npx playwright install --with-deps chromium
- run: npx playwright test
- uses: actions/upload-artifact@v4
if: failure()
with:
name: playwright-report-mailpit
path: playwright-report
# 2. Hosted inboxes with Mailsac Email Capture: the app sends to capture.mailsac.com
# and the test reads the message over the Mailsac API from any runner.
# Needs the repository secret MAILSAC_API_KEY and variable MAILSAC_USERNAME.
mailsac-capture:
runs-on: ubuntu-latest
env:
MAILSAC_API_KEY: ${{ secrets.MAILSAC_API_KEY }}
MAILSAC_DOMAIN: ${{ vars.MAILSAC_DOMAIN }}
SMTP_HOST: capture.mailsac.com
SMTP_PORT: "5587"
SMTP_USER: ${{ vars.MAILSAC_USERNAME }}
SMTP_PASS: ${{ secrets.MAILSAC_API_KEY }}
steps:
- name: Skip when Mailsac is not configured
id: configured
run: echo "ok=${{ env.MAILSAC_API_KEY != '' }}" >> "$GITHUB_OUTPUT"
- uses: actions/checkout@v4
if: steps.configured.outputs.ok == 'true'
- uses: actions/setup-node@v4
if: steps.configured.outputs.ok == 'true'
with:
node-version: 22
cache: npm
- run: npm ci
if: steps.configured.outputs.ok == 'true'
- run: npx playwright install --with-deps chromium
if: steps.configured.outputs.ok == 'true'
- run: npx playwright test
if: steps.configured.outputs.ok == 'true'
# 3. Real delivery, daily and on main: sign up on your deployed staging app, which sends
# through your real email provider (SES, SendGrid, Postmark...) to a Mailsac inbox.
# Catches expired provider credentials, DNS/SPF/DKIM problems and blocked sending domains.
# Needs MAILSAC_API_KEY and the variable STAGING_URL. A private Mailsac domain
# (MAILSAC_DOMAIN) keeps the mail from real signups out of public inboxes.
real-delivery:
if: github.event_name != 'pull_request'
runs-on: ubuntu-latest
env:
MAILSAC_API_KEY: ${{ secrets.MAILSAC_API_KEY }}
MAILSAC_DOMAIN: ${{ vars.MAILSAC_DOMAIN }}
APP_URL: ${{ vars.STAGING_URL }}
steps:
- name: Skip when staging is not configured
id: configured
run: echo "ok=${{ env.MAILSAC_API_KEY != '' && env.APP_URL != '' }}" >> "$GITHUB_OUTPUT"
- uses: actions/checkout@v4
if: steps.configured.outputs.ok == 'true'
- uses: actions/setup-node@v4
if: steps.configured.outputs.ok == 'true'
with:
node-version: 22
cache: npm
- run: npm ci
if: steps.configured.outputs.ok == 'true'
- run: npx playwright install --with-deps chromium
if: steps.configured.outputs.ok == 'true'
- run: npx playwright test
if: steps.configured.outputs.ok == 'true'
Three things to notice:
mailpit starts Mailpit as a service container. On a Linux runner its ports are published on localhost, so the app sends to localhost:1025 and the test reads from localhost:8025. It needs no secrets, so it runs on every pull request, including ones from forks.mailsac-capture runs the app inside the job with its SMTP pointed at Email Capture. SMTP_USER is your Mailsac username and SMTP_PASS is the API key. Mail to any recipient is captured into that recipient’s Mailsac inbox rather than delivered, and the test reads it with the same API it would use for real delivery. The first step checks whether the secret is present and every later step is skipped if not, so a fork without secrets gets a green skip instead of a failure.real-delivery does not start the app at all. APP_URL is set from the STAGING_URL variable, so Playwright signs up on your deployed staging environment, which sends through whatever provider and domain it is configured with. That is the point of the job. It is limited to pushes, the daily schedule and manual runs, because pull requests from forks never see the secret anyway.
GitLab CI
Set MAILSAC_API_KEY (masked) and MAILSAC_USERNAME as CI/CD variables under Settings, CI/CD, Variables, add STAGING_URL and schedule a pipeline for the real-delivery job. GitLab reads .gitlab-ci.yml from the repository root, so the example keeps its copy in its folder as a template to copy to the root of your project. Complete:
# GitLab CI template for this example. GitLab reads .gitlab-ci.yml from the repository root:
# copy this file there in your own project (paths below assume the example is the project root).
# The same two layers on GitLab CI. Set MAILSAC_API_KEY (masked) and MAILSAC_USERNAME
# as CI/CD variables; STAGING_URL enables the scheduled real-delivery job.
default:
image: mcr.microsoft.com/playwright:v1.63.0-noble
before_script:
- npm ci
email-tests:mailpit:
services:
- name: axllent/mailpit:latest
alias: mailpit
variables:
INBOX: mailpit
MAILPIT_URL: http://mailpit:8025
SMTP_HOST: mailpit
SMTP_PORT: "1025"
script:
- npx playwright test
artifacts:
when: on_failure
paths: [playwright-report/]
email-tests:mailsac-capture:
rules:
- if: $MAILSAC_API_KEY
variables:
SMTP_HOST: capture.mailsac.com
SMTP_PORT: "5587"
script:
- SMTP_USER="$MAILSAC_USERNAME" SMTP_PASS="$MAILSAC_API_KEY" npx playwright test
email-tests:real-delivery:
rules:
- if: $CI_PIPELINE_SOURCE == "schedule" && $MAILSAC_API_KEY && $STAGING_URL
script:
- APP_URL="$STAGING_URL" npx playwright test
In GitLab a service is reachable by its alias rather than localhost, so the app sends to mailpit:1025 and the test reads from mailpit:8025. Playwright’s Docker image already contains the browsers, which is why there is no playwright install step; its tag matches the @playwright/test version pinned in package.json (1.63.0), and the two must move together. The rules: blocks make the hosted jobs appear only when their variables exist, and the real-delivery job only on scheduled pipelines.
Handling secrets
- The Mailsac API key is used only by the test runner, which runs in Node, and by the app’s SMTP transport. It never reaches the browser or the page under test.
- The key is the SMTP password for Email Capture.
MAILSAC_USERNAME is not a secret, but keep the key masked everywhere it appears.
- Use one key per pipeline. Business plans and up have multiple named API keys, so a leaked key can be rotated without touching other pipelines.
- Pull requests from forks do not get repository secrets. The
mailsac-capture job skips itself when the key is empty, and real-delivery does not run on pull requests at all. In GitLab, mark the variables protected so only protected branches and schedules see them.
- Treat test output as sensitive. The HTML report is uploaded only on failure and this config records no traces, screenshots or videos. If you enable traces, remember they capture the codes typed and the links opened, including password-reset links. Do not log email bodies.
- Prefer a private domain (below) so the messages themselves are not readable by anyone who guesses an address.
Keeping email tests from flaking
Email tests fail for timing reasons far more often than for product reasons. These habits remove most of it:
- A new address for every test.
newAddress() builds one from a prefix, a base-36 timestamp and 4 random bytes, for example [email protected]. Parallel workers and retries then never read each other’s mail, and you never have to ask which of five messages is yours. Never share one fixed inbox across tests.
- Poll with a deadline, not a sleep.
waitForEmail checks every 2 seconds and gives up after 60 seconds (every 500 ms for 30 seconds with Mailpit). A fixed sleep(10000) is both too slow on good days and too short on bad ones.
- Match on more than “a message exists”. The test filters by subject. If your inbox can hold older mail, also record a timestamp before the trigger and ignore anything received earlier, as the Playwright tutorial does.
- Set the test timeout above the email wait. The config uses 90 seconds against a 60-second wait. Otherwise Playwright’s generic timeout fires first and hides the useful error, which names the address and the deadline.
- Assert on content, not only arrival. The test checks the subject text, the link path and the code format before using them.
- Clean up. Each test deletes its message at the end. On Mailsac this keeps private inboxes inside the plan’s stored-message limit (1,000 on Indie, 5,000 on Business). Move
cleanup into test.afterEach if you want it to run even when an assertion fails.
- Retries are safe only because addresses are fresh. The config allows one retry on CI. A retry re-runs the whole test with a new address, so it cannot read the previous attempt’s email. For a slow provider, a longer deadline is the better fix.
- Check the provider when nothing arrives. Sandbox and test modes on some providers accept the message and deliver nothing. The error tells you which address it waited for; compare it with the provider’s log.
- Public inboxes are throttled. A busy suite pointed at
@mailsac.com can see mail delayed by up to about a minute. Run busy suites on a custom domain, which also has higher inbound limits.
Private domain or public inbox
Where the test email goes decides who else can read it.
- Public
@mailsac.com addresses work on every plan, including the free one, with no setup, and are the default when MAILSAC_DOMAIN is unset. Anyone can open a public inbox on the Mailsac website, and messages are temporary. Use them for a first run and for made-up accounts only.
- A private address (
POST /api/addresses/{email}, one included free) is readable only by your account. It suits one fixed test account, not a fresh address per test.
- A custom domain, on Indie ($18 a month) and up, gives every test its own private address. A zero-setup
yourteam.msdc.co subdomain receives mail immediately; your own domain works once its DNS is verified. Set MAILSAC_DOMAIN and nothing else changes. It also gets past signup forms that reject well-known disposable-email domains.
Each received message, API call and webhook or WebSocket push counts as one Op. A test that polls every 2 seconds typically uses 3 to 6 Ops for a message that arrives within a few seconds, plus one per extra poll if delivery is slow. The free plan’s 1,500 covers a few hundred runs a month, and Indie’s 25,000 covers a delivery check on every merge for most teams. Turning on webhook or WebSocket forwarding for a private address or domain delivers each message for one Op and removes the polling.
Next steps
Create a free Mailsac account to get an API key, add it to your pipeline as a secret, and run the delivery check on your next merge.
