Email delivery testing is an essential part of ensuring your application sends emails correctly and efficiently. With Mailsac, you can take advantage of its powerful REST API to simplify your email testing process. In this blog post, we’ll show you how to use Cypress, a popular end-to-end testing framework, in combination with Mailsac for seamless email delivery testing. We’ll cover setting up the Cypress environment, running tests, and provide plenty of code samples to get you started.
Setting up the Cypress Environment
First, you’ll need to have Node.js installed on your computer. Once that’s done, follow these steps to set up Cypress:
1. Create a new directory for your project and navigate to it in the terminal. b. Run npm init to create a package.json file. c. Install Cypress by running npm install cypress. d. Add a script to your package.json file to run Cypress:
"scripts": {
"cypress:open": "cypress open"
}
2. Configuring Cypress and Mailsac API
Next, create a cypress.json file in your project’s root directory. This file will store your Mailsac API key and other configuration options:
Replace your_mailsac_api_key with your actual Mailsac API key.
3. Writing Your First Cypress Test
Now, let’s create a test file in the cypress/integration folder. Name it email_delivery_spec.js. In this file, we’ll write a test that sends an email to a random Mailsac address and then checks whether the email was received.
// cypress/integration/email_delivery_spec.js
describe("Email Delivery Test", () => {
it("sends an email and verifies its receipt", async () => {
const randomEmail = `test-${Math.random().toString(36).substring(2)}@mailsac.com`;
const testSubject = `Cypress Email Delivery Test ${Math.random().toString(36).substring(2)}`;
const testBody = "This is a test email sent using Cypress and Mailsac.";
// Send an email using your application's email sending method
// ...
// TOOD: integrate your app here!
// Function to poll Mailsac API for received messages. It will be called
// recursively.
const checkEmail = async () => {
let response = await cy.request({
method: "GET",
url: `https://mailsac.com/api/addresses/${randomEmail}/messages`,
headers: {
// Get a mailsac api key at: mailsac.com/api-keys
"Mailsac-Key": Cypress.env("mailsac_api_key")
}
});
const messages = response.body;
const message = messages.find(msg => msg.subject === testSubject);
if (!message) {
return cy.wait(1000).then(() => {
checkEmail();
});
}
expect(message.from[0].address).to.equal("[email protected]");
expect(message.inbox).to.equal(randomEmail);
// Check email content for testBody text
const textResponse = await cy.request(`https://mailsac.com/api/text/${randomEmail}/${msg._id}`);
expect(textResponse.body).to.contain(testBody);
// Clean up by deleting the received messages. This could also be done in an afterEach block.
await Promise.all(messages.map((msg) =>
cy.request({
method: "DELETE",
url: `https://mailsac.com/api/addresses/${randomEmail}/messages/${msg._id}`,
headers: {
"Mailsac-Key": Cypress.env("mailsac_api_key")
}
})
));
};
// Start polling for received messages
await checkEmail();
}
);
});
Replace [email protected] with your application’s sender email address.
This will open the Cypress Test Runner, and you’ll see your email_delivery_spec.js test listed. Click on the test to run it.
Welcome to the second part of the two-part series on running selenium tests with GitHub Actions. In the first article, we outlined how to get started and how to set up your repo for Actions. In this guide, we’ll outline how to run email integration tests and pass secrets to GitHub Actions.
To start… Why would you want to test emails with live email services in the first place? Can’t you simply write to a log file or standard out? Yes… and no. Writing to log files or standard out is ok at the beginning of the application lifecycle. Building out the email send feature takes a back seat to ensure the application actually works.
But let’s say your application is almost ready for its initial release. You want to test that you can even connect to an SMTP server. Or test to ensure the right message can go to the right inbox. Testing via log files or standard out starts to get a bit limiting in that regard. Additionally, say you want to be really sure that the contents of an email you send are what you expect them to be. At this point, you’d like to simulate as much of the email delivery as possible.
You can accomplish this with a disposable email service like Mailsac.
Allow us to plug our mail service
At Mailsac we focus on the developer experience around email automation and testing. That’s why we’ve made it so you can test out the API mentioned in this guide with a free account. You can sign up here.
So let’s lay out our testing goal:
Simulate an email send with our sample application.
Let’s craft the email by driving Selenium through the form. Start by crafting a selenium test:
const chrome = require('selenium-webdriver/chrome');
const {Builder, Browser, By } = require('selenium-webdriver');
const screen = {
width: 1920,
height: 1080
};
(async function emailSendTest() {
let driver = await new Builder()
.forBrowser(Browser.CHROME)
// .setChromeOptions(new chrome.Options().headless().windowSize(screen))
.build();
try {
await driver.get('<http://localhost:3000>');
let didSendButtonRender = await driver.findElement(By.id('sendbutton')).isDisplayed()
if (!didSendButtonRender){
throw new Error(`Send button was not rendered properly.`);
}
await driver.findElement(By.id('email')).sendKeys("[email protected]");
await driver.findElement(By.id('comment')).sendKeys("This is some text from our Selenium test.");
await driver.findElement(By.id('sendbutton')).click();
} finally {
await driver.quit();
}
})();
tests/email-send.js
Note: I left the headless option off for this first test. You’ll want to turn the headless option back on for the test run via our continuous integration environment.
Note that you’ll need to install a couple of dev packages to get this to work: dotenv and superagent. dotenv is needed in this instance since our tests don’t load the entire next framework and as such, we need a method to read your .env file. superagent is a small client-side HTTP request library for doing quick HTTP calls like the one we’re about to do.
So go ahead and add them to your developer dependencies:
npm install --save-dev dotenv superagent
And add our own text comparison to Mailsac’s sample code:
require('dotenv').config()
const superagent = require('superagent')
const mailsac_api_key = process.env.MAILSAC_API_KEY
const expected_message = 'This is some text from our Selenium test.'
superagent
.get('<https://mailsac.com/api/addresses/[email protected]/messages>')
.set('Mailsac-Key', mailsac_api_key)
.then((messages) => {
const messageId = messages.body[0]._id
superagent
.get('<https://mailsac.com/api/text/[email protected]/>' + messageId)
.set('Mailsac-Key', mailsac_api_key)
.then((messageText) => {
if (messageText.text !== expected_message) {
throw new Error(`Message '${messageText.text}' does not match expected text '${expected_message}'`)
}
else{
console.log("Message comparison passed");
}
})
})
.catch(err => {
console.log(err.message)
process.exit(-1)
})
tests/email-read.js
Running the test locally should result in a passing test:
Of course, if we’ll run these tests many times we’ll also want to ensure that we delete the email contents after our successful read. Let’s add a cleanup step to our read test:
require('dotenv').config()
const superagent = require('superagent')
const mailsac_api_key = process.env.MAILSAC_API_KEY;
const expected_message = 'This is some text from our Selenium test.';
const testInbox = '[email protected]';
superagent
.get(`https://mailsac.com/api/addresses/${testInbox}/messages`)
.set('Mailsac-Key', mailsac_api_key)
.then((messages) => {
const messageId = messages.body[0]._id
superagent
.get(`https://mailsac.com/api/text/${testInbox}/` + messageId)
.set('Mailsac-Key', mailsac_api_key)
.then((messageText) => {
if (messageText.text !== expected_message) {
throw new Error(`Message to delete '${messageText.text}' does not match expected text '${expected_message}'`)
}
else{
console.log("API Read Op: Message comparison passed");
superagent
.delete(`https://mailsac.com/api/addresses/${testInbox}/messages/${messageId}`)
.set('Mailsac-Key', mailsac_api_key)
.then((messageResponse) => {
console.log(`API Deletion Op: ${messageResponse.body.message}`)
})
}
})
})
.catch(err => {
console.log(err.message)
process.exit(-1)
})
tests/email-read.js
Let’s add it to a test script and our workflow YAML file:
Note that we added a run mail-tests script to our end-to-end testing.
Try your end-to-end script locally to ensure it works:
GitHub Action Test
Now that you have a working test on your local workstation, it’s time to push it up so GitHub Actions can start running your tests. If you haven’t already, read through the first article to catch up on GitHub Actions configuration and initialization.
As a reminder, this is our main.yml workflow file:
on: [push]
jobs:
tests:
runs-on: ubuntu-latest
name: Run Selenium Tests
steps:
- name: Start selenoid
uses: Xotabu4/selenoid-github-action@v2
- uses: actions/checkout@v1
- run: npm ci
- name: Run end to end tests
run: npm run e2e-test
- name: Run external login test
run: npm run external-tests
.github/workflows/main.yml
With all that said, let’s try and see if this email test sends an email on GitHub.
Do a git push and check your results on GitHub:
Looks like a failure. On closer inspection:
Ah, that’s right! We forgot to set our API keys at the GitHub level. Let’s go ahead and do that.
GitHub Actions Secrets
You can find it under Settings in your repo:
Then under Secrets → Actions click New repository secret
Add each secret that will be needed:
Finally, ensure you add it to the workflow file:
- name: Run end to end tests
run: npm run e2e-test
env:
MAILSAC_API_KEY: ${{secrets.MAILSAC_API_KEY}}
MAILSAC_USERNAME: ${{ secrets.MAILSAC_USERNAME }}
MAILSAC_HOST: ${{ secrets.MAILSAC_HOST }}
MAILSAC_PORT: ${{ secrets.MAILSAC_PORT }}
.github/workflows/main.yml
Do a git commit and git push and see the results on GitHub:
Success! You can check the details to ensure the API read and write options fire fired off:
Conclusion
GitHub Actions is a powerful CI/CD tool and we have only scratched the surface of its capabilities with Selenium. We’ve also shown the power of email testing and how you can ensure the contents of every email are intentional.
We hope you enjoyed the guide, and hope to hear from you on our forums or follow our other content on LinkedIn.
Browser automation is an invaluable tool. At a personal level, you can use it to automate repetitive tasks online. But browser automation can deliver so much more. At its best, it can run tasks with consistent results. Tasks that need lots of manual execution and complexity. Tasks like checking button placement, evaluating user login, and registration workflows. And in our modern era, we have our pick of browser automation frameworks.
We’ve covered browser automation before, but in this guide, we’ll do a deep dive into one of those frameworks. Additionally, we’ll cover using those testing frameworks alongside a specific continuous integration application.
In this guide we’ll walk through three testing scenarios using Selenium and GitHub Actions:
Check if a button rendered
Test a login workflow
Ensure the contents of a sent email
We’ll use our sample application to make our examples concrete, informative, and useful.
First, let’s talk about what you need to enable GitHub Actions.
Getting started with GitHub Actions
First, if you’d like a primer on GitHub Actions their page is a great resource. To summarize, it’s an easy way to run tasks when someone takes an “action” against your repo.
Actions can be:
Pull request merges
Commits
Repo pushes
GitHub takes your action and runs a task like integration tests, unit tests, etc. All you need to do is create a .github/workflows/main.yml file. Note: Running tasks is just one of the many possibilities for GitHub Actions. You can encompass entire workflows and even produce packages.
We’ll introduce more details further along the guide. For now, you can start by adding an empty file at:
We’ll use this repo to walk through the examples mentioned in the intro. These sample scenarios are not meant to be comprehensive, of course. They’re meant to kick-start your journey with GitHub Actions.
To start, you’ll need to run the standard node installation and run the command:
npm install
npm run dev
Navigate to http://localhost:3000 and see our application in action:
With the preliminary steps done, on to the tests.
First Selenium Test: Check if the button rendered
Let’s start with the simplest test, a button render assertion test.
For this first section, we will:
Add the node packages required to test
Add our selenium test
Add testing scripts to our package.json so that GitHub Actions can call our tests
Add a simple Github Actions workflow file
Let’s start by adding a new branch (or you can fork the repo if you’d like):
We’ll need to add selenium and a start-server-and-test package. start-server-and-test starts our application. Then it calls our selenium tests so they can run against our running application.
start-server-and-test is one of our most straightforward ways to add live server testing capabilities. If you need more flexible frameworks, you may want to investigate mocha or cypress.
We’ll just focus on the core GitHub actions and selenium testing in this guide.
Selenium Test: Button Render
On to the test itself.
Let’s create a test under tests/button-render.js using selenium and headless chrome:
const chrome = require('selenium-webdriver/chrome');
const {Builder, Browser, By } = require('selenium-webdriver');
const screen = {
width: 1920,
height: 1080
};
(async function buttonRender() {
let driver = await new Builder()
.forBrowser(Browser.CHROME)
.setChromeOptions(new chrome.Options().headless().windowSize(screen))
.build();
try {
await driver.get('<http://localhost:3000>');
let didSendButtonRender = await driver.findElement(By.id('sendbutton')).isDisplayed()
if (!didSendButtonRender){
throw new Error(`Send button was not rendered properly.`);
}
} finally {
await driver.quit();
}
})();
Now let’s add the workflow main.yml file to the repo. This is the main file that kicks off a GitHub Action workflow. To keep things simple, we’ll execute our actions based on when a comitter pushes code. You can find the full list over at GitHub’s documentation pages.
on: [push]
jobs:
tests:
runs-on: ubuntu-latest
name: Run Selenium Tests
steps:
- name: Start selenoid
uses: Xotabu4/selenoid-github-action@v2
- uses: actions/checkout@v1
- run: npm ci
- name: Run end to end tests
run: npm run e2e-test
.github/workflows/main.yml
For our project, we’ll use solenoid-github-action, a GitHub Action that starts a selenium grid instance in a docker container. Solenoid is a golang reimplementation of Selenium. It makes it very easy to integrate with any continuous integration/deployment environment.
In the last portion of the file, the npm run e2e-test section kicks off the end-to-end test that starts our server and runs the selenium tests.
That’s it! Before you commit and push your code, try running it locally:
npm run e2e-test
The test should pass in your local environment. If it fails due to chrome driver issues you can find a full guide on browser driver installations here.
Head over to your repo on GitHub and under the Actions tab you should now see a selenium test run
You can drill down and see where our specific tests ran
Congrats! That completes our first successful test using selenium and GitHub actions. Let’s move on to something a bit more useful.
Second Selenium Test: Open and login into a web app
Let’s expand on our tests a bit. Since we’re aiming for simplicity in this guide, I won’t add a whole authentication workflow to our application. Instead, let’s focus on an existing website and attempt (and fail) to log in.
As a software developer, it’s crucial to have effective testing tools. They run the gamut from quick libraries to full-fledged analytic frameworks. They also range from free to paid. But which ones are the best in the testing space? In this article, we will list the top 10 best testing tools for software development teams.
But why test?
We can hear the groaning now. Testing is like exercise. We know we’re supposed to, but only so many of us do. Even fewer of us genuinely enjoy it. But testing doesn’t have to be a grind. In fact, we’re willing to bet a portion of you will enjoy it. Why bother with tests?
Testing can be fun – A lot of these tools are automation based. You can focus on crafting a comprehensive test as part of your feature building.
Testing can be done by other groups – It can create a bridge between you and, for example, the QA team. Commiserating around shared testing struggles can go a long way in building trust between teams.
An investment in yourself – Much like the exercise analogy, an investment in testing is an investment in yourself. Refactoring code, deployment flight checks, and all-around confidence about your changes can only be achieved by proper testing.
Now that you know some of the whys, let’s walk through the 10 best testing tools for software developers.
The 10 Best Testing Tools For Software Developers in 2022
1. Puppeteer
Puppeteer is a favorite of the NodeJS community due to its easy integration into your existing build system. It automates form submission, UI testing, keyboard inputs, and more. The main limitation it does have surrounds browser support. As of this writing Puppeteer only supports Chrome. Firefox support is still in the experimental phase.
Puppeteer’s killer feature is that it installs the browser binary for you, making integrating it into your build system easy.
We’ve written an article on using Puppeteer to walk through a common testing scenario. It guides through testing a login screen to dev.to and ensuring entering a bad password does not allow you to log in. You can find it under Automate the Testing Pain Away with Browser Automation.
2. JMeter
JMeter is a popular open-source tool that can test web applications, network performance, and more. It has a versatile GUI for manual testing and a CLI for automated testing. It also offers recording capabilities much like some other tools on this list. It’s a powerful tool with one enormous drawback: it can only integrate with Java applications.
3. Selenium WebDriver
Selenium WebDriver is an open-source automation tool that can test web browsers across different platforms. It’s by far one of the more popular testing tools available around browser automation testing. It can incorporate into a variety of different continuous integration / continuous deployment tools.
Additionally, WebDriver is one of the few tools to be W3C recommended! In their words:
Major browser vendors (Mozilla, Google, Apple, Microsoft) support WebDriver and work constantly to improve the browsers and browser controlling code, which leads to a more uniform behavior across the different browsers, making your automation scripts more stable.
Allow us to shamelessly plug ourselves for a moment, but for your email testing and capture needs, Mailsac is top-notch. We provide just-in-time subdomains and a robust API for automation-driven email testing.
A video walkthrough of using ad-hoc Developer Environments in your testing
5. Postman
Postman is a flexible tool for managing and automating testing requests. It has an intuitive GUI and can generate scripts in various programming languages. You can also store playbooks in Postman to call later using their collection runner.
Give Postman a shot, it’s one of the most popular web API testing tools for a reason.
6. Selenium IDE
Selenium IDE is WebDriver’s GUI-driven sibling. It does a similar job of orchestrating browser functions but with the twist that it can “record” actions as you perform them in your browser.
Successful test suite
Selenium IDE can test web applications, API endpoints, and everything else that WebDriver can do. It’s available as Firefox, Chrome, and Edge extensions.
The built-in Chrome DevTools offer a powerful extension that allows you to inspect and test web pages in real-time. Most other browsers offer this capability but Chrome is hard to beat. It offers:
Built-in Lighthouse report capable of grading the accessibility of your page
Artificial network throttling to simulate slow connection speeds
Performance measurement across pages
Hundreds of extra plugins
8. JUnit5
JUnit5 is a popular open-source unit testing framework that can test Java applications. It offers the ability to have test runners for your test cases and enables you to focus on Test Driven Development.
Cucumber is an open-source testing tool that can be used to test web applications and APIs. Cucumber is a rather unique one on this list in that it focuses on getting the specifications right the first time. It’s called Behavior Driven Development and it allows project managers and technical contributors to collaborate on concrete aspects of the application.
10. Firefox Developer Tools
Firefox does a great job innovating on the developer toolset. (3D View anyone?) Since it can do anything the Chrome DevTool set can, we’ll use this section to focus on specific plugin shoutouts:
User-Agent Switcher
Test your browser detection logic with this extension.
Ghostery
Ghostery is great for a variety of reasons, not least of which is to prevent distractions. It blocks a variety of trackers, ads, and generally improves page performance.
Cookie Manager
A versatile extension that can help you test all sorts of functionality in your application. Cookie Manager can help you with everyday tasks from authentication testing to session switching and inspection. You can additionally export and import cookie sets.
Conclusion
These are some of the best software testing tools available for developers. They all have their strengths and weaknesses, but they are all powerful tools that can help you improve your code quality.
If you’d like to discuss some of the tools you use for your software testing we’ll love to hear about it! Head on over to our community at forum.mailsac.com and discuss your must-have or time-saving tools.
Today’s release of Mailsac’s frontend services includes many fixes.
We tend to release weekly with no downtime and no fanfare. Occasionally we’ll post updates here.
fix: infrequent crash on logout
Most notable is a fix to a UI crash when logging out. In certain situations, you may have seen an error message, even though the logout was successful.
Prettier and more usable charts
We heard your feedback. Usage charts have been aging. Sometimes the styling makes the charts hard to read.
Some additional PostgreSQL optimizations are coming soon. We’ll continue reducing the load time on charts. Thanks for the patience!
Dependency upgrades
For better or worse, modern software stacks have huge numbers of dependencies. Staying ahead of security issues is a daily effort.
At Mailsac we use security scans, Dependabot, and npm audit to stay on top of upgrades. Hundreds of automated tests run to give us confidence that minor and patch semver updates don’t introduce breaking changes.
We subscribe to security mailing lists for our software, such as:
When the topic of domains and email comes up most people begin and end the conversation at the top domain level. Subdomains seem to be left out of the conversation in their entirety. Are we trapped in our thinking about subdomains as mere marketing and newsletter features? Maybe it’s too difficult to use subdomains without an IT team involved. Maybe no one has brought up subdomains outside of meme-filled newsletters. Maybe you just haven’t thought about subdomains in general.
Well, let’s break up that thinking. Subdomains have a lot to offer. Do you have trouble testing 10 different email features in your application? Does the thought of accidentally sending an email to thousands of users that says “Test” make you break into a cold sweat? Subdomains can help.
We’ll show you some of the possibilities of subdomains and walk through some use cases. We’ll also provide a quick 15-second walkthrough at the end that will setup up 2 new subdomains for development and testing purposes.
But before we show you some of the juicy scenarios, let’s do a quick rundown of what a subdomain actually is.
What’s the difference between an email domain and a subdomain?
Subdomains are a way to slice up domains for specific functions like newsletters and blogs. The advantage of a subdomain is having a clear purpose tied to the name. Receiving emails from [email protected] and[email protected] show their intent from their name alone. Receiving emails from[email protected] and [email protected] is a lot vaguer. The former set clearly sends memes and educational nuggets. The latter could be a friendly name for our billing bots.
Subdomains = an easy way to differentiate email by function.
Alright, sorry about that. Had to make sure everyone was on the same page on subdomains. Let’s move on to 3 different subdomain scenarios.
Subdomain Use Case 1: Developers Get Their Own Email Domains
You work on a team of developers, and each of you needs to test the same features on a few different applications. Additionally, each feature has an email workflow attached to it. The usual response to this is to have a shared inbox, for example, [email protected]. But the pain around that approach comes fast. Issues like:
Difficulty in separating out each developer’s testing scenarios
Having to sift through 1000 other unrelated emails while looking for that 1 workflow email is painful
Complex workflows are pretty difficult to track
A domain for me, a domain for you
Creating an email subdomain per developer is an effective way to isolate these emails across systems:
Remember that you don’t need to create these inboxes ahead of time. They are made on the fly and removed when they make sense for you, the person knee-deep in the application.
Also not shown above just yet: Mailsac’s unified inbox in action.
Subdomain Use Case 2: Company-Wide Domains per Environment
Ensure emails stay in their zone
Environments for each set of applications are a pretty common scenario amongst enterprises. A sample above shows 2 applications split between 3 environments:
The upside of this approach is having predefined email subdomains for each environment. Developers, QA teams, and operations all know which environment the emails are associated with. QA testers can review the messages easily knowing which environment sent the emails. Operations and developers know which email address and domain to use as variables when configuring tests or environments. Ultimately, this saves time for all the teams involved.
Subdomain Use Case 3: Email Driven API Workflow
An email-driven API workflow is a workflow that kicks off when an email arrives. The approach resembles the first scenario, where each developer gets their own domain. The difference is the usernames are less flexible. You pin it once to an API and use it for the long term. For example:
An email to [email protected] can trigger a Submit API action that can create a case in the HR management system
An email to [email protected] can trigger an Integration API action that can automatically create a ticketing workflow in your Incident Management system.
You can even string together a received email to a webhook using Mailsac’s webhook service. If you’d like to poll for updates instead we have websocket for close to real-time processing or the rest API for polling.
Alright, enough theory let’s do a walkthrough.
Walkthrough: Company-Wide Environments
Using the company-wide scenario we can have a working subdomain in a few seconds using Mailsac. A partner video will walk through the individual developer scenario.
You also have the option of using your own domain. This requires an external domain service provider. There are lots of guides out there on which domain registrar is the best.
But let’s make this easy. Let’s use Mailsac’s Zero-Config Subdomain tool and bring up a new subdomain in 2 clicks. Note that you will need at least a Business Plan to make this scenario work. You can still enjoy the benefits of a single subdomain through the Indie Plan.
Type the name of the subdomain you’d like, in our case acme-dev and acme-test
And that’s it! You should have 2 custom subdomains ready to use. Let’s put it them rough their paces. We’ll send out these emails from any client (I’ll use Gmail):
To: [email protected] Subject: Email sent to Billing (DEV) Content: This is meant for dev
…or you could use the Unified Inbox feature that displays all of your custom domains, subdomains, and private addresses in one convenient location:
Much better
It’s just that easy!
Wrap Up
With this new superpower, you should be able to conjure up lots of different use cases for subdomains. The friction of creating and importing domains is completely taken care of for you. No need to register a domain with an external registrar, or manage an IT team to handle registration for you.
We’re always looking for feedback, so let us know what you think or if you run into any problems on our forums.
When I say “You need to test your code”, do you wince? Is it a feeling of guilt, one of “I know I should, but…”. Testing may not conjure up the sexiest of images. We as developers frequently put tests off until the end of our feature cycle. Or respond to a production bug by issuing a quick patch. Or worse, just bury our heads in the sand and pretend that we don’t have any bugs in our code at all. (Note: All code has bugs).
The reality is that testing is an incredible investment in your code’s future. Investing in tests is like an insurance policy: hedging your bets against an unknown future. An unknown future consisting of bitrot, dependency deprecations, or service API changes. Testing provides the ability to patch those unknowns through refactoring or flat-out removing stale dependencies. Testing can also buffer against those risks, providing peace of mind.
In this post, I’ll outline 3 different types of testing tools:
Selenium WebDriver
Selenium IDE
Puppeteer
To do an apples-to-apples comparison the testing scenario will be the same for all three tools. I’ll also model my testing after a user’s typical behavior. Behaviors like login attempts, searching, and form submissions. They also try to hit every layer of the application, from the user interface to the database.
Benefits of Testing a User Interface
Testing isn’t just limited to the backend. Testing your interface can provide complete end-to-end testing scenarios such as:
Repeated calls to your modal. Does the modal come back after the first call?
Does your submit button produce an error if the form has an incorrect value?
Does the UI load after a successful login to an empty state in your application?
Does a specific result come back after a form search?
I’m going to walk through a straightforward testing scenario with three tools. Not to rank them, but to touch on the nuances of each. Some of these tools allow you to create tests through simple browsing. Others are headless, allowing you to drive through programming languages.
What’s a headless browser?
Conventional browsing involves rendering forms, buttons, and images to the user. A headless browser interacts with websites through code without displaying any controls. Headless browsing opens up possibilities that are tough to achieve with conventional browsers like:
Integration with your build systems
Consistency in testing
Decreasing the duration of your tests
Layout screen captures and comparisons
Tools of the Automated Browser Trade
Onto the good stuff: The tools and testing scenarios.
WebDriver targets as its core base Developers and QA Team members who can write code.
The Easiest To Get Started with – Selenium IDE
Selenium designed the IDE version for exploratory testing and bug replication. It’s perfect for walking through a bug with someone else or creating a recording of a bug for your ticketing system.
Puppeteer is a favorite of the NodeJS community due to its easy integration into your existing build system. It automates form submission, UI testing, keyboard inputs, and more. It’s main limitation however is the browsers it supports. As of this writing Puppeteer only supports Chrome. Firefox support is, as of this writing, experimental.
Puppeteer’s killer feature is that it installs the browser binary for you, making the integrating into your build system easy.
Testing Scenario: A Failed Login to dev.to
Here’s our testing scenario:
Load https://dev.to
Click the “Log in” button
Load a page with “Welcome! – DEV Community” in its title.
Click on the “Continue” button
Ensure an “Unable to login” banner appears on the page.
For consistency throughout the walkthrough, I’ll use:
Chrome as my browser
Javascript as the programming language of choice
Test Case 1 – Selenium WebDriver
Let’s begin with an empty directory and selenium package installation:
npm init tests
cd tests
npm install selenium-webdriver
Next, download a browser driver. You can find the full supported list in selenium’s code repository. You can place the binary anywhere. For this walkthrough, I’ll place it in the current project directory under the bin/ path.
Set your specific browser driver path:
export PATH=$PATH:$PWD/bin
I’ll be using this quick test setup (selenium.js):
const {Builder, Browser, By, Key, until} = require('selenium-webdriver');
(async function example() {
let driver = await new Builder().forBrowser(Browser.CHROME).build();
try {
await driver.get('http://dev.to');
await driver.findElement(By.linkText('Log in')).click();
await driver.wait(until.titleIs('Welcome! - DEV Community'), 3000);
await driver.findElement(By.name('commit')).click();
await driver.wait(until.titleIs(''), 3000);
let errorBox = await driver.findElement(By.className('registration__error-notice'));
await driver.wait(until.elementIsVisible(errorBox));
let errorText = await errorBox.getText();
if (!errorText.includes('Error')){
throw new Error(`Error text does not contain expected value: ${errorText}`);
}
} finally {
await driver.quit();
}
})();
Set your driver and run the file
SELENIUM_BROWSER=chrome node selenium.js
A failed Selenium Test
In general I like to ensure my tests fail from the start, followed by working towards passing the tests:
const {Builder, Browser, By, Key, until} = require('selenium-webdriver');
(async function example() {
let driver = await new Builder().forBrowser(Browser.CHROME).build();
try {
await driver.get('http://dev.to');
await driver.findElement(By.linkText('Log in')).click();
await driver.wait(until.titleIs('Welcome! - DEV Community'), 3000);
await driver.findElement(By.name('commit')).click();
await driver.wait(until.titleIs(''), 3000);
let errorBox = await driver.findElement(By.className('registration__error-notice'));
await driver.wait(until.elementIsVisible(errorBox));
let errorText = await errorBox.getText();
if (!errorText.includes('Unable to login')){
throw new Error(`Error text does not contain expected value "${errorText}"`);
}
} catch(e) {
console.error(`Error running test suite: ${e.message}`)
}
finally {
await driver.quit();
}
})();
With line 15 fixed, rerun the script:
A successful Selenium Test
Success!
The above was a taste of what you can do with Selenium. You can even break out of the testing mindset and use Selenium for scraping and populating activity trackers.
On to the next tool.
Test Case 2 – Selenium IDE
While the previous test requires some programming ability, Selenium IDE is friendly to anyone who can drive a browser. The IDE version’s main use case is bug discovery, recording and profiling.
After you hit “Start Recording”, Selenium will launch a new Chrome window and redirect you to dev.to
Initial Dev.To Walkthrough with Selenium IDE
From the video, we:
Let the initial dev.to page load
Clicked on the “Log in” button
Clicked on the Selenium IDE extension
Stopped the Extension recording
Arrived at the Commands window below
Selenium testing properties loaded automatically
To continue our test scenario, let’s ensure that the page title is Welcome! - DEV Community and that our login attempt fails with an empty submission.
Again, I always like to have my tests fail first, so let’s start with that case. Use Selenium’s assert title command to ensure the title is what we expect. Add it to the command list:
Asserting the page title to fail
If you run the test, it should fail:
Example of a failed test
Let’s go ahead and fix it with the correct title and rerun the test:
Successful Title Check
And success! Now let’s add the login check:
Walking through a complete test to success.
To summarize the video, we:
Started a new recording
Hit the Log in button
Clicked Continue without supplying credentials
Used the Selenium element picker to pick out the element we were interested in asserting.
The Commands window should now look like this:
Added Command Check
Success!
The IDE version is the simplest to get started with and I recommend it for initial test write-ups. It can help you identify which elements you need to test against, think about app flow and what counts as a failure.
One question remains: Rendering the browser is nice, but I want to hook this into my continuous integration system. How can I do that when every test wants to load an application that requires a windowing system?
The answer is to go headless.
Test Case 3 – Puppeteer
Puppeteer is the perfect match to test web UI components inside a continuous integration system. It’s fast, headless, brings its own dependencies and runs the latest versions of Chrome and Firefox.
Let’s start by installing puppeteer on a new project:
mkdir tests
npm i puppeteer
Keep in mind that this automatically installs the chrome driver we had to manually download in the Selenium example. From Puppeteer’s documents:
When you install Puppeteer, it downloads a recent version of Chromium (~170MB Mac, ~282MB Linux, ~280MB Win) that is guaranteed to work with the API (customizable through Environment Variables).
https://pptr.dev/#installation
With that said, let’s create a test file that will run (and fail) our test scenario (puppeteer.js):
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch({headless: false});
const page = await browser.newPage();
const loginSelector = 'a[href="/enter"]';
const submitLoginSelector = '[name="commit"]';
const errorBoxSelector = '.bad notice';
try {
await page.goto('https://dev.to');
await page.waitForSelector(loginSelector,{ timeout: 3000 });
await page.click(loginSelector);
const pageTitle = await page.title();
if (pageTitle !== 'Welcome! - DEV Community'){
throw new Error(`Page title ${pageTitle} does not match expected value`);
}
await page.waitForSelector(submitLoginSelector,{ timeout: 3000 });
await page.click(submitLoginSelector);
await page.waitForSelector(errorBoxSelector,{ timeout: 3000 });
}catch(e){
console.error(`Error in test suite: ${e.message}`)
}finally {
await browser.close();
}
})();
Failed test due to incorrect error box selection
Some notes on the above code:
Lines 6-8 are Puppeteer’s method of selecting elements on the page.
Like Selenium WebDriver, you have to manually check a page’s attributes and decide on what to do should they fail
In the above code it’s line 15, asserting the title matches the expected value
It’ll also implicitly fail on line 20, due to the error div class not matching what dev.to sends to the browser.
I’ve disabled the headless feature to show that Puppeteer lets you do that!
Let’s fix the test. Change it to the correct value *and* turn on headless mode:
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch({headless: true});
const page = await browser.newPage();
const loginSelector = 'a[href="/enter"]';
const submitLoginSelector = '[name="commit"]';
const errorBoxSelector = '.registration__error-notice';
try {
await page.goto('https://dev.to');
await page.waitForSelector(loginSelector,{ timeout: 3000 });
await page.click(loginSelector);
const pageTitle = await page.title();
if (pageTitle !== 'Welcome! - DEV Community'){
throw new Error(`Page title ${pageTitle} does not match expected value`);
}
await page.waitForSelector(submitLoginSelector,{ timeout: 3000 });
await page.click(submitLoginSelector);
await page.waitForSelector(errorBoxSelector,{ timeout: 3000 });
}catch(e){
console.error(`Error in test suite: ${e.message}`)
}finally {
await browser.close();
}
})();
Now rerunning the test simply gets you the empty prompt:
I’ve gone through three different sets of tools for different needs. The best part about these tools is that you can string them all together or pick and choose the ones that are right for you.
I hope the main takeaway is the same: Testing can be painless and even fun!
Developing an application that sends emails is straightforward but not without its risks. Ensuring deliverability but not actually having any of those emails land inside real inboxes is a top concern for any developer. Which leads to questions like: “How do you test your application’s outbound email capabilities?” or “How do I manage email testing for free?”
Enter email capture services. While the term “email capture service” tends to focus on the marketing aspects (capturing information from your calls to action, ensuring emails don’t get caught in spam, etc) they also include SMTP deliverability. Mailsac offers an email capture service that addresses the deliverability aspect, specifically not delivering any email to its intended recipient. Effectively a “black hole” where no email should escape to the outside world.
In this post, we’ll walk through a sample application in Next.js that will generate emails and have those emails captured by Mailsac’s email sandboxing service.
Do I Really Need To Do Email Testing?
Some frameworks do come with email previewing capabilities like Rails’ ActionMailer. Said frameworks don’t actually attempt to send anything but instead preview the email on your machine. We recommend real testing during the development and quality assurance phase by using an external SMTP server to mimic the application’s behavior in production.
Testing that capability has to be done safely unless you want to land on Twitter’s trending page for accidentally sending customers an integration test email.
Test Email Sending With A Next.js Application
For the rest of this guide, we’ll focus on wiring up a simple application that will allow users to send an email when a button is pushed from a UI. We’ll then demonstrate the capture of those emails in our development environment.
The components we’ll use are:
Next.js with a React frontend and API Routes backend
nodemailer package for routing email on the backend
While the focus of this guide isn’t a line-by-line walkthrough of the sample code, we’ll focus on the key aspects of the application that mainly involve emailing capabilities.
The application source can be found in our git repository.
1. Application setup
Let’s start by creating a quick next app with tailwind support:
mailsac % npx create-next-app
…
Success! Created nodejs-send-email at /Users/mailsac/code/nodejs-send-email
cd nodejs-send-email
npm install -D tailwindcss postcss autoprefixer @tailwindcss/forms
npm install @headlessui/react@latest @heroicons/react
npx tailwindcss init -p
The second line brings in a component that takes in a message and formats it as a pop-up notification. The useEffect() method sends your email recipient and body input to the backend, which will forward that data to Mailsac’s servers.
const nodemailer = require("nodemailer");
export default async function handler(req, res) {
let emailEnvelope = JSON.parse(req.body)
if (
req.method === 'POST'
&& typeof(emailEnvelope.to) !== 'undefined'
&& emailEnvelope.to !== ''
){
const mailsaUserName = process.env.MAILSAC_USERNAME
const mailsacAPIKey = process.env.MAILSAC_API_KEY
const transporter = nodemailer.createTransport({
host: 'capture.mailsac.com',
port: 5587,
// will use TLS by upgrading later in the connection with STARTTLS
secure: false,
auth: {
user: mailsaUserName,
pass: mailsacAPIKey
}
})
try {
const results = await transporter.sendMail({
from: '"Sample App" [email protected]',
to: emailEnvelope.to,
subject: 'Sample App Send',
text: emailEnvelope.body
})
res.status(200).json(
{
message: "You should now see an email in Mailsac's capture service",
response: results.data
}
)
} catch (error){
console.log(`ERROR ${error}`)
res.status(500).json({ message: `${error.response}`, response: error })
}
} else {
return res.status(200).json({message: "No data"});
}
}
In the highlighted line, we’re ensuring the useEffect() hook gets called with input data before we allow the rest of the function to continue. useEffect() gets called a variety of times in the component lifecycle, and this check is to ensure it was initiated by an end user and not as part of the component mounting.
5. Test driving the app
Fire up the application via
npm run dev
Navigate to http://localhost:3000 and type a text message:
Then navigate over to mailsac.com to view the message
6. Capturing other email domains
While that works well as a contrived example, the real value comes when using any arbitrary email in the recipient field:
Capturing emails outside the mailsac.com domain is extremely valuable when switching between different environments. For example, in the demo application example above, the .env environment file could instead look like
const nodemailer = require("nodemailer");
export default async function handler(req, res) {
let emailEnvelope = JSON.parse(req.body)
if (
req.method === 'POST'
&& typeof(emailEnvelope.to) !== 'undefined'
&& emailEnvelope.to !== ''
){
const mailsaUserName = process.env.MAILSAC_USERNAME
const mailsacAPIKey = process.env.MAILSAC_API_KEY
const transporter = nodemailer.createTransport({
host: process.env.MAILSAC_HOST,
port: process.env.MAILSAC_PORT,
// will use TLS by upgrading later in the connection with STARTTLS
secure: false,
auth: {
user: mailsaUserName,
pass: mailsacAPIKey
}
})
try {
const results = await transporter.sendMail({
from: '"Sample App" [email protected]',
to: emailEnvelope.to,
subject: 'Sample App Send',
text: emailEnvelope.body
})
res.status(200).json(
{
message: "You should now see an email in Mailsac's capture service",
response: results.data
}
)
} catch (error){
res.status(500).json({ message: `${error.response}`, response: error })
}
} else {
return res.status(200).json({message: "No data"});
}
}
The above edits would allow you to deploy to a testing or production environment and the only changes required would be in the .env file. Specifically, the SMTP host and authentication settings.
Conclusion
The above guide just scratches the surface of what you can do with our email services. We provide a unified inbox that allows testers to view their bulk email testing in one unified view and custom domains for those who do not have their own domains with zero setup configurations.
Google Analytics (GA), the ubiquitous web analytics service provided by Google, has been removed from all Mailsac properties. GA is used to see which pages people visit, how frequently users return to a website, and where they were referred from.
Despite Google Analytics being the de facto tracking service on the internet – mostly because it is entirely free – we decided it was not a good fit for our users.
It is worth noting that Mailsac does not use Facebook analytics, nor any other 3rd party trackers. We do leverage the privacy-focused – and paid – Cloudflare Analytics, which is included with our DNS and caching service. We also track minimal usage metrics on the server side of Mailsac.com, for billing purposes.
Mailsac was created over 10 years ago with the goal of providing a reliable disposable email platform for software testers. We serve no advertisements, sell no data to 3rd parties. We rely on users to pay us for providing a good service. From the time of Mailsac’s creation – until recently – we did leverage Google Analytics as a tool to understand website load. As a “free” product, GA helped keep costs down and worked well. We have come to understand that “free” meant we effectively shared our users browsing habits with Google, for the purpose of serving ads. So we no longer do that.
Screenshot of mailsac.com on 2022-04-28 in Safari showing zero trackers
Outbound SMTP service will no longer be supported by Mailsac.
What Does this Mean for Me?
Mail will no longer be able to sent from Mailsac addresses or custom domains using the outbound Mailsac SMTP service.
If you are sending from the REST API, compose email form, or Unified Inbox we encourage you to seek out other SMTP sending services. Mailsac has always supported direct SMTP from anywhere, without a mail relay, as well.
Reasons for the Change
Email delivery is not an easy problem at scale.
Our customers, especially quality assurance teams, are using Mailsac as a receive-only service.
The likelihood that Mailsac’s outbound messages are delivered to the intended inbox has been trending lower over time, despite increased effort.
We made the decision to focus our efforts on improving our core product – disposable inbound email for testing.
