Streamline Email Testing in Healthcare with Mailsac

In healthcare, email communication intersects with patient care and data security. The margin for error is virtually nonexistent. Mailsac’s SaaS platform offers a robust email delivery suite, tailored to meet the rigorous demands of healthcare IT security, compliance, and operational efficiency. Mailsac has been integrated into systems at many enterprises in the healthcare industry, and has been relied on 24/7 for email testing for over a decade.

Enhanced Security and Compliance

Security Audits: Mailsac is built to pass IT department security audits, aligning with healthcare standards like HIPAA. Its infrastructure ensures patient data is safeguarded during email testing.

Data Privacy: With disposable email addresses, Mailsac supports testing that avoids exposing real patient information, maintaining privacy and compliance.

Collaborative Testing Environment

Unified Inbox: Devs and QAs can collaborate effectively using Mailsac’s unified inbox feature, which consolidates test emails from both persistent and temporary email accounts into a single view.

SSO Integration: Simplify access while enhancing security with Single Sign-On (SSO), allowing seamless integration into existing healthcare IT ecosystems.

Automated and Efficient Testing

CI/CD Integration: Mailsac’s API automates email tests within CI/CD pipelines, reducing manual effort and accelerating development cycles in fast-paced healthcare settings.

Scalable Solutions: Whether scaling up operations or integrating new services, Mailsac’s scalable platform adapts to the evolving needs of healthcare enterprises, ensuring email testing is never a bottleneck. There’s no need to manage server resources, deployments or upgrades.

Comprehensive Compatibility Testing

Device and Platform Coverage: Guarantee that critical communications are accessible across all devices and platforms used by healthcare professionals and patients alike.

Deliverability Assurance

Inbox Placement: Rigorous testing with Mailsac ensures healthcare emails achieve high deliverability, crucial for appointment reminders, test results, and other sensitive communications.

Optimizing Patient Communication

Email Optimization: Test and refine email content for clarity and engagement, ensuring messages to patients are both accessible and actionable.

Conclusion

For healthcare organizations, Mailsac offers a precision toolset for email testing — ensuring security, efficiency, and compliance while enhancing collaborative efforts between QA and development teams. Since 2012, Mailsac has practiced technical excellence and has helped hundreds of customers in healthcare manage their unique challenges.

10 Common Email Testing Pitfalls and How to Avoid Them

Email testing is an essential step in the QA process, ensuring that communications reach their intended recipients accurately and efficiently. However, even experienced QA teams can fall into common traps that undermine their efforts. Here are ten frequent email testing pitfalls and strategic ways to avoid them, streamlining your workflow and enhancing email reliability.

1. Ignoring Mobile Responsiveness

Pitfall:

Not testing how emails render on mobile devices, leading to formatting issues or poor user experiences.

Solution:

Use email testing tools that simulate various mobile devices and screen sizes to ensure your emails look great everywhere.

2. Overlooking Email Client Diversity

Pitfall:

Focusing on a single email client, ignoring the fact that your audience uses a wide range of email services with different rendering engines.

Solution:

Test emails across multiple clients (like Gmail, Outlook, Yahoo) to identify and fix client-specific issues.

3. Neglecting Deliverability Tests

Pitfall:

Assuming emails reach the inbox without verifying, risking them being flagged as spam.

Solution:

Conduct deliverability tests with tools that provide insights into spam scores and help optimize for better inbox placement.

4. Underestimating the Importance of Content Clarity

Pitfall:

Creating content that’s confusing or misleading, leading to poor user engagement.

Solution:

Ensure your messages are clear, concise, and actionable. Test variations to see which performs best in terms of user engagement.

5. Skipping Accessibility Checks

Pitfall:

Forgetting to make emails accessible to all users, including those with disabilities.

Solution:

Include text alternatives for images, use sufficient contrast ratios, and test with screen readers to ensure accessibility.

6. Failing to Test Links and Attachments

Pitfall:

Assuming all links and attachments work without thoroughly testing them, which could lead to a frustrating user experience.

Solution:

Manually check each link and attachment in different environments to ensure functionality and security.

7. Ignoring Email Load Times

Pitfall:

Overloading emails with high-resolution images or complex HTML, leading to slow loading times.

Solution:

Optimize images and streamline code to improve load times, ensuring a smooth user experience.

8. Forgetting to Validate Email Lists

Pitfall:

Sending tests to outdated or incorrect email addresses, skewing testing results.

Solution:

Regularly cleanse and validate your email lists to ensure accuracy and relevance.

9. Overlooking Privacy and Compliance

Pitfall:

Neglecting privacy laws and email regulations, risking legal issues and damaged reputation.

Solution:

Stay informed about regulations like GDPR and CAN-SPAM, ensuring your email practices are compliant.

10. Not Leveraging Automation

Pitfall:

Performing repetitive tests manually, which is time-consuming and prone to human error.

Solution:

Incorporate automated testing workflows to save time, reduce errors, and increase efficiency.

In Conclusion

By being mindful of these common pitfalls and implementing the suggested solutions, QA teams can significantly improve their email testing processes. Tools like Mailsac offer zero-configuration custom private domains, comprehensive Swagger REST APIs, and a generous free tier, making it easier for teams to test emails effectively and efficiently. Remember, the goal is not just to send emails but to ensure they are delivered, readable, and engaging across all devices and clients.

Have Playwright Automatically Write Tests For You with Codegen

These days, we have better options than writing our test specs by hand.

The open source community has released a variety of frameworks to relieve us from that particular tedium: Cypress, Selenium and Pupeteer. And in this video companion guide, I’ll focus on Playwright. Specifically, how playwright can help automate a lot of the boilerplate involved in writing test specs with codepen.

The video will do a quick walk you through playwright in itself. This article will provide the core login.spec.ts file I used and where to go from next.

So, What Is Playwright?

Playwright is an open-source automation library created by Microsoft. It’s designed to enable developers and testers to write reliable and efficient tests for web applications.

It’s cross platform and officially compatible with the major browsers.

Diving Into Codegen

Start by installing playwright inside your project

npm init playwright@latest

I’ll be using the defaults for this guide. After installation you’ll have these files generated in your project:

Files generated
playwright.config.ts
package.json
package-lock.json
tests/
example.spec.ts
tests-examples/
demo-todo-app.spec.ts
For more: https://playwright.dev/docs/test-configuration

Putting Codegen Through Its Paces

First Run

Fire up codegen via the built in console

npx playwright codegen

As you click around your application you’ll see codegen record each click based on its css class.

It will build each line as you go about your test. In our video, we perform a login with an incorrect set of credentials and a correct set.

Our login.spec.ts file

Before our manual edits, this is what codegen generated for us:

import { test, expect } from '@playwright/test';

test('test', async ({ page }) => {
  await page.goto('http://localhost:3000/');
  await page.getByRole('button', { name: 'Sign in' }).click();
  await page.getByLabel('Email Address').click();
  await page.locator('form div').filter({ hasText: 'Email AddressEmail Address' }).getByRole('paragraph').click();
  await page.getByLabel('Email Address').click();
  await page.getByLabel('Email Address').fill('[email protected]');
  await page.getByLabel('Email Address').press('Tab');
  await page.getByLabel('Password').fill('password');
  await page.getByLabel('Password').press('Enter');
  await page.getByLabel('Email Address').click();
  await page.getByLabel('Email Address').fill('[email protected]');
  await page.getByLabel('Email Address').press('Tab');
  await page.getByLabel('Password').fill('password123');
  await page.getByLabel('Password').press('Enter');
  await page.getByRole('button', { name: 'Settings' }).click();
  await page.getByText('Note: Your email mailsac.demo').click({
    button: 'middle'
  });
  await page.getByRole('button', { name: 'Next boilerplate' }).click();
});

All we had to do was manually add was the highlighted lines (13 and 19) to turn it into a real test:

import { test, expect } from '@playwright/test';

test('test', async ({ page }) => {
  await page.goto('http://localhost:3000/');
  await page.getByRole('button', { name: 'Sign in' }).click();
  await page.getByLabel('Email Address').click();
  await page.locator('form div').filter({ hasText: 'Email AddressEmail Address' }).getByRole('paragraph').click();
  await page.getByLabel('Email Address').click();
  await page.getByLabel('Email Address').fill('[email protected]');
  await page.getByLabel('Email Address').press('Tab');
  await page.getByLabel('Password').fill('password');
  await page.getByLabel('Password').press('Enter');
  await expect(page).toHaveURL('http://localhost:3000/login');
  await page.getByLabel('Email Address').click();
  await page.getByLabel('Email Address').fill('[email protected]');
  await page.getByLabel('Email Address').press('Tab');
  await page.getByLabel('Password').fill('password123');
  await page.getByLabel('Password').press('Enter');
  await expect(page).toHaveURL('http://localhost:3000/home');
  await page.getByRole('button', { name: 'Settings' }).click();
  await page.getByText('Note: Your email mailsac.demo').click({
    button: 'middle'
  });
  await page.getByRole('button', { name: 'Next boilerplate' }).click();
});

Running the Test

Running the test by default shows no visual progress. But if you’d like to see the browser run through your steps visually, you’ll need to issue the command:

npx playwright test --headed

Where to go next

The most natural next step is integrating playwright tests with a continuous integration platform like Travis or Github Actions. Plugging playwright into a CI system like Github Actions is fully supprted by playwright natively.

Another possible progression is using playwright to test critical paths in your application like user registration or password reset flows. We have a full guide on how to do that with another framework, Cypress.

If you want us to explore how you can integrate playwright with email testing and Github Actions or any other potential playwright integrations, let us know on our forums. We’ve only scratched the very surface of what playwright can do.

Until next time.

Integrating Email Testing into Your CI/CD Pipeline: A Step-by-Step Guide

Continuous Integration/Continuous Deployment (CI/CD) pipelines are at the heart of modern software development practices, enabling teams to automate testing and deployment processes to increase efficiency and reliability. Integrating email testing into your CI/CD pipeline is crucial for applications that rely on email functionality, ensuring that every update maintains or enhances the quality of email communications. This guide walks you through the process step by step, ensuring a seamless integration of email testing into your CI/CD workflows.

Step 1: Understand Your Email Testing Needs

Before diving into integration, clearly define what aspects of your email functionality need testing. This might include deliverability, content accuracy, responsiveness, and interaction with email clients. Understanding these needs helps in selecting the right tools and defining the scope of testing within your pipeline.

Step 2: Select the Right Email Testing Tools

Choose tools that offer API integration capabilities and can simulate various email scenarios. Tools like Mailsac allow for seamless integration into CI/CD pipelines, offering features such as disposable email addresses, REST API access for automated testing, and custom domain testing capabilities.

Step 3: Set Up Your Email Testing Environment

Configure your chosen email testing tool within your development and staging environments. Ensure it’s capable of capturing and analyzing the emails sent by your application during the automated testing phase. This setup should mimic your production environment as closely as possible to ensure accurate results.

Step 4: Integrate Email Testing into Your CI/CD Pipeline

Incorporate email testing tasks into your CI/CD pipeline configuration. This involves:

  • Triggering Email Tests: Automate the sending of emails based on specific triggers within your pipeline, such as a successful build or deployment to a staging environment.
  • Monitoring and Logging: Ensure your email testing tool captures detailed logs and analytics, providing insights into the success or failure of email deliveries and content rendering.
  • Analyzing Results: Set up mechanisms to analyze the results of email tests, identifying issues like failed deliveries, content errors, or rendering issues across email clients.

Step 5: Automate Feedback Loops

Implement automated feedback loops to alert developers and QA teams of any issues detected during email testing. This can be done through integration with project management tools, sending notifications via email, Slack, or other communication platforms used by your team.

Step 6: Continuous Monitoring and Optimization

Even after successful integration, continuously monitor the performance and effectiveness of your email testing within the CI/CD pipeline. Use insights gained from testing to refine and optimize email functionality, ensuring ongoing improvement and adherence to best practices.

Step 7: Document and Educate Your Team

Document the integration process, tools used, and best practices for email testing within your CI/CD pipeline. Educate your development and QA teams on the importance of email testing and how it fits into the broader context of quality assurance and software reliability.

Conclusion

Integrating email testing into your CI/CD pipeline is a strategic move that enhances the quality assurance process for applications relying on email communications. By following this step-by-step guide, your team can ensure that email functionality remains robust, reliable, and responsive to the needs of your users. Tools like Mailsac can play a pivotal role in this process, offering the flexibility, ease of integration, and comprehensive testing capabilities necessary to meet the demands of modern software development practices.

The Complete Checklist for Email Testing in QA Projects

Email remains a critical communication tool in business, making its reliability and functionality crucial for a wide range of applications. For QA teams, ensuring that emails are sent, received, and displayed as intended across various environments and platforms is a non-negotiable part of the software development process. This checklist provides a comprehensive guide for QA professionals to thoroughly test email functionality, ensuring a seamless user experience.

1. Preparation and Planning

  • Define Objectives: Clearly outline what you need to test, such as deliverability, content, links, and attachments.
  • Identify Email Scenarios: List all the scenarios in which an email would be sent. This includes transactional emails, marketing emails, notifications, and any others specific to your application.
  • Understand the Audience: Know the email clients (e.g., Gmail, Outlook) and devices (e.g., smartphones, tablets) your audience uses.

2. Functional Testing

  • Deliverability: Ensure emails reach the recipient’s inbox, not the spam folder.
  • Send and Receive Verification: Confirm that emails are sent and received without errors in all scenarios.
  • Link and Attachment Testing: Check all links and attachments for correct functionality and security.
  • Formatting and Layout: Verify that emails display correctly across different email clients and devices.
  • Accessibility Testing: Ensure emails are accessible, including alternative text for images and readable fonts for those with visual impairments.

3. Content and Design Testing

  • Spelling and Grammar: Verify the accuracy of content, including spelling and grammar.
  • Branding Consistency: Ensure the email design aligns with your brand’s guidelines and messaging.
  • Responsive Design: Test emails on various screen sizes to ensure the design is responsive and elements are clickable.

4. Security and Compliance Testing

  • Data Protection: Confirm that personal data is handled securely, in compliance with laws like GDPR.
  • Email Authentication: Check for SPF, DKIM, and DMARC records to prevent phishing and ensure sender authenticity.

5. Performance Testing

  • Load Testing: Assess the system’s ability to handle high volumes of emails without performance degradation.
  • Speed Testing: Evaluate the time taken to send, receive, and load emails, ensuring it meets user expectations.

6. Integration and Automation Testing

  • Integration Checks: Verify that email functionality integrates seamlessly with other systems and workflows.
  • Automation Suitability: Identify processes that can be automated, such as regression tests for email functionalities.

7. User Acceptance Testing

  • Real User Simulation: Conduct tests that mimic real-user scenarios to ensure the email meets user needs and expectations.
  • Feedback Collection: Gather and incorporate feedback from actual users to refine email functionality.

Review

Email testing is a critical component of QA that ensures your application communicates effectively and reliably with users. By following this comprehensive checklist, QA teams can systematically address and rectify potential issues, enhancing the overall user experience.

Interested in simplifying and accelerating your email testing process? Mailsac offers a robust platform designed to streamline email testing for QA teams. With features like disposable email addresses, zero-setup custom domain support, and extensive API access, Mailsac enables you to focus on what matters most—delivering a flawless product. Try Mailsac for free today, getting test email in seconds, and discover how we can elevate your QA email testing to the next level.

Transform Your Email Testing with Cypress and Mailsac

If you’re in the business of making or testing software, you’re well aware of the critical role email functionality can play in applications. Particularly in sensitive workflows like user sign-ups and password resets. Testing these features however can often be a complex and daunting task. This is where Cypress and Mailsac come in. Together these two streamline the email testing process; cypress driving it and mailsac capturing and validating any emails that may come out of your testing.

Join me as we explore how you can leverage these powerful tools to automate and enhance your testing workflow, ensuring a seamless user experience in your applications.

This article acts as a companion guide to the video linked at the top. Here, we’ll quickly walk through the topics mentioned in the video but also expand some areas and provide the code that the video uses.

The general path we’ll take in this guide is:

  1. Walk through our application’s reset flow
  2. Safeguard against sending emails to real customers
  3. Automate the password reset process with subject verification with cypress.

Let’s get started!

Setting the Stage: Manually Testing our Next.js App

We’ll focus on a simple next.js application equipped with local authentication features. Instead of starting from scratch, we’ll use Martin Persson’s Next.js Authentication Template. The concepts we’ll test are a common part of almost every application, and by using his template we’ll have a starting point (most) of us can agree on. Martin’s application comes complete with login functionality, member-exclusive sections, and importantly, a password reset flow.

Our goal is to show how we can streamline testing the password reset flow, ensuring that branch of your code behaves the same way every time. By the end of this, you’ll have a clear blueprint for applying these techniques in your projects.

Regular Password Reset Speedrun

Let’s do a quick walk through on how we would reset our password in our app.

Create an Account

The app doesn’t do any email validation out of the box but it does create and store account credentials in its database.

Reset Password To a Real Email Address

This is a standard password reset flow. The main thing to note here is that the application is sending out transactional emails to real email addresses.

Caveats

During continuous testing we could be building our application and testing its capabilities tens to hundreds of times per month. We’ll need to ensure we don’t send any emails real customers.

This is where Mailsac comes in.

Safeguarding Against Sending Emails to Real Customers

Mailsac has an email capture feature that enables customers to “reroute” any emails generated by your application to an inbox on the mailsac platform. Additional testing capabilities are available including verification of delivered content and ensuring the emails that get sent have the correct subject.

Start Capturing Emails from your Application

Getting started is extremely easy. Just point your application’s SMTP settings to the mailsac’s servers and all outbound emails will be captured.

Our sample application reads in a .env file:

MONGODB_URI="mongodb://localhost:27017"
WEB_URI="http://localhost:3000"
MAIL_PASSWORD="k_99dfuuifjd"         # Your Mailsac API Key
MAIL_USER="mailsacuser"              # Your Mailsac Username
MAIL_HOST="capture.mailsac.com"
MAIL_PORT="5587"

Going through the password reset flow in our application again, we successfully have no emails in our real gmail inbox, and an email in the Mailsac inbox.

Also, we did not create this inbox ahead of time. It was done ad-hoc as the email came into mailsac.

Success. Now we are safe against sending out real emails to customers. Let’s begin the automated testing processes with Cypress

Automatic Testing with Cypress

Our goal with cypress is to automate the clicking of a password reset link, and to verify that an email was actually initiated and sent by our application.

Installing and Configuring Cypress

Let’s start by installing cypress

npm cypress install

And then opening up the cypress testing center

npx cypress open

You’ll be greeted by what kind of testing type to initiate. In our case, we’ll use end to end testing

And the included Chrome browser

We are then greeted by the testing specs list

And now the real work begins.

Configure Cypress

Our configuration of cypress will need mailsac’s client and dotenv library. Cypress will need this to have the ability to read our application’s .env file

npm install @mailsac/api dotenv

We can now define what exactly a how exactly to execute a checkMailsacMail and deleteAllEmails function

import { defineConfig } from "cypress"
import { Mailsac } from "@mailsac/api"
import * as dotenv from 'dotenv';
dotenv.config();

export default defineConfig({
  e2e: {
    // https://docs.cypress.io/guides/references/configuration#e2e
    baseUrl: "http://localhost:3000",
    defaultCommandTimeout: 15000,
    // Whether or not test isolation is enabled to ensure a clean browser context between tests.
    testIsolation: false,
    setupNodeEvents(on,config ) {
      on('task', {
        async checkMailsacMail(address) {
          const mailsac = new Mailsac({ headers: { "Mailsac-Key": process.env.MAIL_PASSWORD } })
          const messages = await mailsac.messages.listMessages(address);
          return messages.data;
        },
        async deleteAllEmails(address) {
          const mailsac = new Mailsac({ headers: { "Mailsac-Key": process.env.MAIL_PASSWORD } })
          const messages = await mailsac.messages.deleteAllMessages(address);
          if(messages.status === 204) {
            return true;
          } else {
            return false;
          }
        }
      })
    },
  },
  component: {
    devServer: {
      framework: "next",
      bundler: "webpack",
    },
  },
  // Settings
  env: {
    baseUrl: "http://localhost:3000",
    username: "[email protected]",
    password: "password123",
  },
})

The contents of this file are fully explained in the video (Starting at 6:24) but to call out the highlighted lines (13-30):

checkMailsacMail

The checkMailsacMail function initiates a mailsac client and uses the API key provided from the .env file variable, MAIL_PASSWORD. It then calls the listMessages function and passes it up to the calling function to do as it pleases. The function returns a JSON response as outlined in the Mailsac ListMessages documentation.

deleteAllEmails

Similar to checkMailsacMail but of course, deletes all messages in an inbox. Note that the return is a 204 to confirm deletion.

Let’s move on to the actual password testing spec

Password Reset Flow Test Spec

The spec itself is placed in the cypress/e2e/password_reset_flow_success.cy.ts path and contains:

describe('Password Reset Change', () => {
  it('Should successfully change the password', () => {
    cy.visit('http://localhost:3000')

    // Find a button with class and contains text
    cy.get(".MuiButton-root").contains("Sign in").click()

    // The new url should include
    cy.url().should("include", "/login")

    const { username, password } = Cypress.env()

    cy.get("a").contains("Forgot password").click()

    cy.url().should("include", "/forgot-password")

    cy.get("input[name='email']").type(username)
    cy.get("input[name=email]").should("have.value", username);

    cy.get(".MuiButton-root").contains("Reset").click();
    cy.url().should("include", "/login")    

    cy.task("checkMailsacMail", username).then((messages) => {
      const resetUrl = messages[0].links[0];
      const subject = messages[0].subject;
      const originalInbox = messages[0].originalInbox;

      expect(subject).to.eq("next-boilerplate: Reset your password.");
      expect(originalInbox).to.eq(username);

      cy.visit(resetUrl);
      cy.url().should("include", resetUrl);

      cy.task("deleteAllEmails", username).then((result) => {
        expect(result).to.eq(true);
      });
    });

  })
})

Cypress will run through this spec file line by line and execute the steps we manually ran through at the beginning of the guide. The highlighted lines are really the structure of the returned JSON from mailsac’s ListMessages API

Final Automation

The the spec file in place, go ahead and run the spec

Wrap up

Combining Cypress and Mailsac is like giving your email testing a super boost. We’ve walked you through the nitty-gritty of automating a password reset flow, making sure your emails hit the mark without bothering real customers. It’s all about making your software solid while keeping things simple and stress-free. Give it a try, save time, and keep your users smiling.

If you have any questions about the guide or if you get stuck feel free to ask us anything in our forums.

Email Delivery Testing: Node.js, Cypress, and Mailsac

Email delivery testing is an essential part of ensuring your application sends emails correctly and efficiently. With Mailsac, you can take advantage of its powerful REST API to simplify your email testing process. In this blog post, we’ll show you how to use Cypress, a popular end-to-end testing framework, in combination with Mailsac for seamless email delivery testing. We’ll cover setting up the Cypress environment, running tests, and provide plenty of code samples to get you started.

Setting up the Cypress Environment

First, you’ll need to have Node.js installed on your computer. Once that’s done, follow these steps to set up Cypress:

1. Create a new directory for your project and navigate to it in the terminal. b. Run npm init to create a package.json file. c. Install Cypress by running npm install cypress. d. Add a script to your package.json file to run Cypress:

"scripts": {
  "cypress:open": "cypress open"
}

2. Configuring Cypress and Mailsac API

Next, create a cypress.json file in your project’s root directory. This file will store your Mailsac API key and other configuration options:

{
  "baseUrl": "https://mailsac.com",
  "env": {
    "mailsac_api_key": "your_mailsac_api_key"
  }
}

Replace your_mailsac_api_key with your actual Mailsac API key.

3. Writing Your First Cypress Test

Now, let’s create a test file in the cypress/integration folder. Name it email_delivery_spec.js. In this file, we’ll write a test that sends an email to a random Mailsac address and then checks whether the email was received.

// cypress/integration/email_delivery_spec.js

describe("Email Delivery Test", () => {
  it("sends an email and verifies its receipt", async () => {
      const randomEmail = `test-${Math.random().toString(36).substring(2)}@mailsac.com`;
      const testSubject = `Cypress Email Delivery Test ${Math.random().toString(36).substring(2)}`;
      const testBody = "This is a test email sent using Cypress and Mailsac.";

      // Send an email using your application's email sending method
      // ...
      // TOOD: integrate your app here!

      // Function to poll Mailsac API for received messages. It will be called
      // recursively.
      const checkEmail = async () => {
        let response = await cy.request({
          method: "GET",
          url: `https://mailsac.com/api/addresses/${randomEmail}/messages`,
          headers: {
            // Get a mailsac api key at: mailsac.com/api-keys
            "Mailsac-Key": Cypress.env("mailsac_api_key")
          }
        });
        const messages = response.body;
        const message = messages.find(msg => msg.subject === testSubject);

        if (!message) {
          return cy.wait(1000).then(() => {
            checkEmail();
          });
        }

        expect(message.from[0].address).to.equal("[email protected]");
        expect(message.inbox).to.equal(randomEmail);

        // Check email content for testBody text
        const textResponse = await cy.request(`https://mailsac.com/api/text/${randomEmail}/${msg._id}`);
        expect(textResponse.body).to.contain(testBody);

        // Clean up by deleting the received messages. This could also be done in an afterEach block.
        await Promise.all(messages.map((msg) =>
          cy.request({
            method: "DELETE",
            url: `https://mailsac.com/api/addresses/${randomEmail}/messages/${msg._id}`,
            headers: {
              "Mailsac-Key": Cypress.env("mailsac_api_key")
            }
          })
        ));
      };

      // Start polling for received messages
      await checkEmail();
    }
  );
});

Replace [email protected] with your application’s sender email address.

This will open the Cypress Test Runner, and you’ll see your email_delivery_spec.js test listed. Click on the test to run it.

Running Selenium Tests in GitHub actions email test cover image

Run Selenium Tests in GitHub Actions – Email Testing

Welcome to the second part of the two-part series on running selenium tests with GitHub Actions. In the first article, we outlined how to get started and how to set up your repo for Actions. In this guide, we’ll outline how to run email integration tests and pass secrets to GitHub Actions.

To start… Why would you want to test emails with live email services in the first place? Can’t you simply write to a log file or standard out? Yes… and no. Writing to log files or standard out is ok at the beginning of the application lifecycle. Building out the email send feature takes a back seat to ensure the application actually works.

But let’s say your application is almost ready for its initial release. You want to test that you can even connect to an SMTP server. Or test to ensure the right message can go to the right inbox. Testing via log files or standard out starts to get a bit limiting in that regard. Additionally, say you want to be really sure that the contents of an email you send are what you expect them to be. At this point, you’d like to simulate as much of the email delivery as possible.

You can accomplish this with a disposable email service like Mailsac.

Allow us to plug our mail service

At Mailsac we focus on the developer experience around email automation and testing. That’s why we’ve made it so you can test out the API mentioned in this guide with a free account. You can sign up here.

So let’s lay out our testing goal:

  1. Simulate an email send with our sample application.
    1. Refer to our Guide To Stress Free Email Testing with Next.js for more information on how we created this sample application
    2. Use Selenium to drive the form and hit send
  2. Use our capture service API to send the email
  3. Use Mailsac’s API to read the email from the destination inbox
  4. Verify the contents of said email
  5. Do all this on GitHub Actions

Let’s get started.

API Credentials

If you don’t already have one, go ahead and create a mailsac account and generate an API key.

Plug those API keys in a file called .env at the root of the project:

MAILSAC_USERNAME=$MAILSAC_USERNAME
MAILSAC_API_KEY=$MAILSAC_GENERATED_KEY
MAILSAC_HOST=capture.mailsac.com
MAILSAC_PORT=5587

.env

Craft the Email

Let’s craft the email by driving Selenium through the form. Start by crafting a selenium test:

const chrome = require('selenium-webdriver/chrome');
const {Builder, Browser, By } = require('selenium-webdriver');

const screen = {
  width: 1920,
  height: 1080
};

(async function emailSendTest() {
    
  let driver = await new Builder()
    .forBrowser(Browser.CHROME)
    // .setChromeOptions(new chrome.Options().headless().windowSize(screen))
    .build();

  try {
    await driver.get('<http://localhost:3000>');
    let didSendButtonRender = await driver.findElement(By.id('sendbutton')).isDisplayed()
    
    if (!didSendButtonRender){
      throw new Error(`Send button was not rendered properly.`);
    }
 
    await driver.findElement(By.id('email')).sendKeys("[email protected]");
    await driver.findElement(By.id('comment')).sendKeys("This is some text from our Selenium test.");
    await driver.findElement(By.id('sendbutton')).click();

  } finally {
    await driver.quit();
  }
})();

tests/email-send.js

Note: I left the headless option off for this first test. You’ll want to turn the headless option back on for the test run via our continuous integration environment.

You can do a local test by running

node tests/email-send.js

And checking the inbox we tested ([email protected]) manually:

Success! Though that’s part of the way there. Let’s assert that the email contents match via the API.

Read the Email via Mailsac’s API

Thankfully, Mailsac has not only a robust API but also lots of sample code inside the docs. We’ll lift the sample email read from here:

https://docs.mailsac.com/en/latest/services/reading_mail/reading_mail.html

Note that you’ll need to install a couple of dev packages to get this to work: dotenv and superagent. dotenv is needed in this instance since our tests don’t load the entire next framework and as such, we need a method to read your .env file. superagent is a small client-side HTTP request library for doing quick HTTP calls like the one we’re about to do.

So go ahead and add them to your developer dependencies:

npm install --save-dev dotenv superagent

And add our own text comparison to Mailsac’s sample code:

require('dotenv').config()

const superagent = require('superagent')
const mailsac_api_key = process.env.MAILSAC_API_KEY
const expected_message = 'This is some text from our Selenium test.'

superagent
  .get('<https://mailsac.com/api/addresses/[email protected]/messages>')
  .set('Mailsac-Key', mailsac_api_key)
  .then((messages) => {
      const messageId = messages.body[0]._id
      superagent
          .get('<https://mailsac.com/api/text/[email protected]/>' + messageId)
          .set('Mailsac-Key', mailsac_api_key)
              .then((messageText) => {
                  if (messageText.text !== expected_message)  {
                    throw new Error(`Message '${messageText.text}' does not match expected text '${expected_message}'`)
                  }
                  else{
                    console.log("Message comparison passed");
                  }
              })
  })
  .catch(err => {
      console.log(err.message)
			process.exit(-1)
  })

tests/email-read.js

Running the test locally should result in a passing test:

$ node tests/email-read.js
Message comparison passed

Of course, if we’ll run these tests many times we’ll also want to ensure that we delete the email contents after our successful read. Let’s add a cleanup step to our read test:

require('dotenv').config()

const superagent = require('superagent')
const mailsac_api_key = process.env.MAILSAC_API_KEY;
const expected_message = 'This is some text from our Selenium test.';
const testInbox = '[email protected]';

superagent
.get(`https://mailsac.com/api/addresses/${testInbox}/messages`)
  .set('Mailsac-Key', mailsac_api_key)
  .then((messages) => {
      const messageId = messages.body[0]._id
      superagent
          .get(`https://mailsac.com/api/text/${testInbox}/` + messageId)
          .set('Mailsac-Key', mailsac_api_key)
              .then((messageText) => {
                  if (messageText.text !== expected_message)  {
                    throw new Error(`Message to delete '${messageText.text}' does not match expected text '${expected_message}'`)
                  }
                  else{
                    console.log("API Read Op: Message comparison passed");
                    superagent
                    .delete(`https://mailsac.com/api/addresses/${testInbox}/messages/${messageId}`)
                    .set('Mailsac-Key', mailsac_api_key)
                    .then((messageResponse) => {
                        console.log(`API Deletion Op: ${messageResponse.body.message}`)
                    })
                  }
              })
  })
  .catch(err => {
      console.log(err.message)
      process.exit(-1)
  })

tests/email-read.js

Let’s add it to a test script and our workflow YAML file:

"scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint",
    "external-tests": "node tests/external-login.js",
    "test": "node tests/button-render.js && npm run mail-tests",
    "mail-tests": "node tests/email-send.js && node tests/email-read.js",
    "e2e-test": "start-server-and-test dev <http://localhost:3000> test"
  },

package.json

Note that we added a run mail-tests script to our end-to-end testing.

Try your end-to-end script locally to ensure it works:

End to end sample test

GitHub Action Test

Now that you have a working test on your local workstation, it’s time to push it up so GitHub Actions can start running your tests. If you haven’t already, read through the first article to catch up on GitHub Actions configuration and initialization.

As a reminder, this is our main.yml workflow file:

on: [push]

jobs:
  tests:
    runs-on: ubuntu-latest
    name: Run Selenium Tests
    steps:
    - name: Start selenoid
      uses: Xotabu4/selenoid-github-action@v2

    - uses: actions/checkout@v1
    - run: npm ci  

    - name: Run end to end tests
      run: npm run e2e-test

    - name: Run external login test
      run: npm run external-tests

.github/workflows/main.yml

With all that said, let’s try and see if this email test sends an email on GitHub.

Do a git push and check your results on GitHub:

A gitHub test failure

Looks like a failure. On closer inspection:

GitHub specific failure with details

Ah, that’s right! We forgot to set our API keys at the GitHub level. Let’s go ahead and do that.

GitHub Actions Secrets

You can find it under Settings in your repo:

Github walkthrough image : Click settings

Then under Secrets → Actions click New repository secret

Add each secret that will be needed:

GitHub Secrets listing for your Actions.

Finally, ensure you add it to the workflow file:

- name: Run end to end tests
      run: npm run e2e-test
      env:
        MAILSAC_API_KEY:  ${{secrets.MAILSAC_API_KEY}}
        MAILSAC_USERNAME: ${{ secrets.MAILSAC_USERNAME }}
        MAILSAC_HOST:     ${{ secrets.MAILSAC_HOST }}
        MAILSAC_PORT:     ${{ secrets.MAILSAC_PORT }}

.github/workflows/main.yml

Do a git commit and git push and see the results on GitHub:

Successful email test with GitHub actions.

Success! You can check the details to ensure the API read and write options fire fired off:

Github actions success detailed view.

Conclusion

GitHub Actions is a powerful CI/CD tool and we have only scratched the surface of its capabilities with Selenium. We’ve also shown the power of email testing and how you can ensure the contents of every email are intentional.

We hope you enjoyed the guide, and hope to hear from you on our forums or follow our other content on LinkedIn.

Running Selenium Tests in GitHub actions Cover image

Run Selenium Tests in GitHub Actions

Browser automation is an invaluable tool. At a personal level, you can use it to automate repetitive tasks online. But browser automation can deliver so much more. At its best, it can run tasks with consistent results. Tasks that need lots of manual execution and complexity. Tasks like checking button placement, evaluating user login, and registration workflows. And in our modern era, we have our pick of browser automation frameworks.

We’ve covered browser automation before, but in this guide, we’ll do a deep dive into one of those frameworks. Additionally, we’ll cover using those testing frameworks alongside a specific continuous integration application.

In this guide we’ll walk through three testing scenarios using Selenium and GitHub Actions:

  • Check if a button rendered
  • Test a login workflow
  • Ensure the contents of a sent email

We’ll use our sample application to make our examples concrete, informative, and useful.

Repo URL: https://github.com/mailsac/mailsac-capture-service-example-nextjs

With all that said, let’s get started!

First, let’s talk about what you need to enable GitHub Actions.

Getting started with GitHub Actions

First, if you’d like a primer on GitHub Actions their page is a great resource. To summarize, it’s an easy way to run tasks when someone takes an “action” against your repo.

Actions can be:

  • Pull request merges
  • Commits
  • Repo pushes

GitHub takes your action and runs a task like integration tests, unit tests, etc. All you need to do is create a .github/workflows/main.yml file. Note: Running tasks is just one of the many possibilities for GitHub Actions. You can encompass entire workflows and even produce packages.

We’ll introduce more details further along the guide. For now, you can start by adding an empty file at:

.github/workflows/main.yml

The Sample Application We’ll Use

We’ll fork our sample application at the URL:

https://github.com/mailsac/mailsac-capture-service-example-nextjs

We’ll use this repo to walk through the examples mentioned in the intro. These sample scenarios are not meant to be comprehensive, of course. They’re meant to kick-start your journey with GitHub Actions.

To start, you’ll need to run the standard node installation and run the command:

npm install
npm run dev

Navigate to http://localhost:3000 and see our application in action:

Our initial application

With the preliminary steps done, on to the tests.

First Selenium Test: Check if the button rendered

Let’s start with the simplest test, a button render assertion test.

For this first section, we will:

  1. Add the node packages required to test
  2. Add our selenium test
  3. Add testing scripts to our package.json so that GitHub Actions can call our tests
  4. Add a simple Github Actions workflow file

Let’s start by adding a new branch (or you can fork the repo if you’d like):

git checkout -B cicd-selenium-app-test

Now let’s move on to our sample packages.

Packages: Selenium WebDriver & Start-Server-And-Test

We’ll need to add selenium and a start-server-and-test package. start-server-and-test starts our application. Then it calls our selenium tests so they can run against our running application.

npm install --save-dev start-server-and-test selenium-webdriver

start-server-and-test is one of our most straightforward ways to add live server testing capabilities. If you need more flexible frameworks, you may want to investigate mocha or cypress.

We’ll just focus on the core GitHub actions and selenium testing in this guide.

Selenium Test: Button Render

On to the test itself.

Let’s create a test under tests/button-render.js using selenium and headless chrome:

const chrome = require('selenium-webdriver/chrome');
const {Builder, Browser, By } = require('selenium-webdriver');

const screen = {
  width: 1920,
  height: 1080
};

(async function buttonRender() {
  let driver = await new Builder()
    .forBrowser(Browser.CHROME)
    .setChromeOptions(new chrome.Options().headless().windowSize(screen))
    .build();

  try {
    await driver.get('<http://localhost:3000>');
    let didSendButtonRender = await driver.findElement(By.id('sendbutton')).isDisplayed()
    if (!didSendButtonRender){
      throw new Error(`Send button was not rendered properly.`);
    }
 
  } finally {
    await driver.quit();
  }
})();

tests/button-render.js

You can find the full usage guide on selenium’s documentation page.

To help the test out, we’ll add an ID attribute to the Send button:

...
<button
    onClick={sendEmail}
    id="sendbutton"
    className="mt-3 w-full inline-flex items-center justify-center px-4 py-2 border border-transparent shadow-sm font-medium rounded-md text-white bg-indigo-600 hover:bg-indigo-700 focus:outline-none focus:ring-2 focus:ring-offset-2 focus:ring-indigo-500 sm:mt-0 sm:ml-3 sm:w-auto sm:text-sm"
  >
    Send
</button>
...

pages/index.js

Telling GitHub Actions How to Run Our Tests

We’ll need to add a couple of testing scripts to package.json:

...
"scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint",
    "test": "node tests/button-render.js",
    "e2e-test": "start-server-and-test dev <http://localhost:3000> test"
  },
...

package.json

Now let’s add the workflow main.yml file to the repo. This is the main file that kicks off a GitHub Action workflow. To keep things simple, we’ll execute our actions based on when a comitter pushes code. You can find the full list over at GitHub’s documentation pages.

on: [push]

jobs:
  tests:
    runs-on: ubuntu-latest
    name: Run Selenium Tests
    steps:
    - name: Start selenoid
      uses: Xotabu4/selenoid-github-action@v2

    - uses: actions/checkout@v1
    - run: npm ci  

    - name: Run end to end tests
      run: npm run e2e-test

.github/workflows/main.yml

For our project, we’ll use solenoid-github-action, a GitHub Action that starts a selenium grid instance in a docker container. Solenoid is a golang reimplementation of Selenium. It makes it very easy to integrate with any continuous integration/deployment environment.

In the last portion of the file, the npm run e2e-test section kicks off the end-to-end test that starts our server and runs the selenium tests.

That’s it! Before you commit and push your code, try running it locally:

npm run e2e-test

The test should pass in your local environment. If it fails due to chrome driver issues you can find a full guide on browser driver installations here.

Send it over to GitHub via git push.

git push --set-upstream origin cicd-selenium-app-test

Head over to your repo on GitHub and under the Actions tab you should now see a selenium test run

A GitHub Actions workflow results page

You can drill down and see where our specific tests ran

Congrats! That completes our first successful test using selenium and GitHub actions. Let’s move on to something a bit more useful.

Second Selenium Test: Open and login into a web app

Let’s expand on our tests a bit. Since we’re aiming for simplicity in this guide, I won’t add a whole authentication workflow to our application. Instead, let’s focus on an existing website and attempt (and fail) to log in.

We’ll start by adding a new test to our repo.

Selenium Test: Login to Dev.to

Let’s bring in our external login Selenium test, but slightly modified:

const chrome = require('selenium-webdriver/chrome');
const {Builder, Browser, By, until } = require('selenium-webdriver');

const screen = {
    width: 1920,
    height: 1080
  };

(async function externalLogin() {
  let driver = await new Builder()
  .forBrowser(Browser.CHROME)
  .setChromeOptions(new chrome.Options().headless().windowSize(screen))
  .build();

  try {
    await driver.get('<https://dev.to>');
    await driver.findElement(By.linkText('Log in')).click();
    await driver.wait(until.titleContains('Welcome - DEV Community'), 3000);
    await driver.findElement(By.name('commit')).click();
    await driver.wait(until.titleIs(''), 3000);
    let errorBox = await driver.findElement(By.className('registration__error-notice'));
    await driver.wait(until.elementIsVisible(errorBox));
    let errorText = await errorBox.getText();

    if (!errorText.includes('Unable to login')){
      throw new Error(`Error text does not contain expected value "${errorText}"`);
    }

  }
  finally {
    await driver.quit();
  }
})();

tests/external-login.js

Only real changes are that we added a headless option for chrome.

Modifying Our Tests

Keeping things simple, let’s just add a new test script:

....
"test": "node tests/button-render.js",
"external-tests": "node tests/external-login.js",
...

package.json

And add a separate testing task to our workflow:

...
- name: Run external login test
      run: npm run external-tests
...

.github/workflows/main.yml

Let’s check our result on GitHub:

GitHub Action Failure example

Looks like a failure due to an incorrect title.

Let’s make the match less precise and rerun the test:

...
await driver.findElement(By.linkText('Log in')).click();
await driver.wait(until.titleContains('Welcome! - DEV Community'), 3000);
await driver.findElement(By.name('commit')).click();
...

tests/external-login.js

Let’s try another git push and..

A full successful external login test screen/

Success!

Conclusion

This was the first part of a two-part series about diving into GitHub Actions and Selenium tests. In the second part, we’ll run through:

  1. running our application in GitHub actions
  2. sending emails via our web form
  3. reading and comparing the email contents
  4. deleting the email contents afterward

All are driven by our selenium tests.

We hope you enjoyed the guide, and hope to hear from you on our forums or follow our other content on LinkedIn.

10 Best Testing Tools for Software Developers in 2022 Logo

10 Best Testing Tools For Software Developers in 2022

As a software developer, it’s crucial to have effective testing tools. They run the gamut from quick libraries to full-fledged analytic frameworks. They also range from free to paid. But which ones are the best in the testing space? In this article, we will list the top 10 best testing tools for software development teams.

But why test?

We can hear the groaning now. Testing is like exercise. We know we’re supposed to, but only so many of us do. Even fewer of us genuinely enjoy it. But testing doesn’t have to be a grind. In fact, we’re willing to bet a portion of you will enjoy it. Why bother with tests?

  • Testing can be fun – A lot of these tools are automation based. You can focus on crafting a comprehensive test as part of your feature building.
  • Testing can be done by other groups – It can create a bridge between you and, for example, the QA team. Commiserating around shared testing struggles can go a long way in building trust between teams.
  • An investment in yourself – Much like the exercise analogy, an investment in testing is an investment in yourself. Refactoring code, deployment flight checks, and all-around confidence about your changes can only be achieved by proper testing.

Now that you know some of the whys, let’s walk through the 10 best testing tools for software developers.

The 10 Best Testing Tools For Software Developers in 2022

1. Puppeteer 

Puppeteer is a favorite of the NodeJS community due to its easy integration into your existing build system. It automates form submission, UI testing, keyboard inputs, and more. The main limitation it does have surrounds browser support. As of this writing Puppeteer only supports Chrome. Firefox support is still in the experimental phase.

Puppeteer’s killer feature is that it installs the browser binary for you, making integrating it into your build system easy.

We’ve written an article on using Puppeteer to walk through a common testing scenario. It guides through testing a login screen to dev.to and ensuring entering a bad password does not allow you to log in. You can find it under Automate the Testing Pain Away with Browser Automation.

2. JMeter 

JMeter is a popular open-source tool that can test web applications, network performance, and more. It has a versatile GUI for manual testing and a CLI for automated testing. It also offers recording capabilities much like some other tools on this list. It’s a powerful tool with one enormous drawback: it can only integrate with Java applications.

3. Selenium WebDriver 

Selenium WebDriver is an open-source automation tool that can test web browsers across different platforms. It’s by far one of the more popular testing tools available around browser automation testing. It can incorporate into a variety of different continuous integration / continuous deployment tools.

Additionally, WebDriver is one of the few tools to be W3C recommended! In their words:

Major browser vendors (Mozilla, Google, Apple, Microsoft) support WebDriver and work constantly to improve the browsers and browser controlling code, which leads to a more uniform behavior across the different browsers, making your automation scripts more stable.

From Selenium’s Project Page
A successful Selenium WebDriver Test

To help you get started with WebDriver we’ve written a guide on creating your automated headless browser testing.

4. Mailsac 

Allow us to shamelessly plug ourselves for a moment, but for your email testing and capture needs, Mailsac is top-notch. We provide just-in-time subdomains and a robust API for automation-driven email testing.

Here’s a guide to using our on-demand subdomain feature to create Company-Wide Email Inboxes

A video walkthrough of using ad-hoc Developer Environments in your testing

5. Postman 

Postman is a flexible tool for managing and automating testing requests. It has an intuitive GUI and can generate scripts in various programming languages. You can also store playbooks in Postman to call later using their collection runner.

postman  test runner
Job Runner CLI Status From Postman’s Writing Tests Blog Post

Postman’s killer feature is the ability to integrate API calls onto your build system: 

postman   travis success
Job Runner CLI Status From Postman’s Writing Tests Blog Post

Give Postman a shot, it’s one of the most popular web API testing tools for a reason.

6. Selenium IDE

Selenium IDE is WebDriver’s GUI-driven sibling. It does a similar job of orchestrating browser functions but with the twist that it can “record” actions as you perform them in your browser.

Successful test suite
Successful test suite

Selenium IDE can test web applications, API endpoints, and everything else that WebDriver can do. It’s available as Firefox, Chrome, and Edge extensions.

Much like WebDriver we’ve written a quick start guide on automating the browser.

7. Chrome DevTools

The built-in Chrome DevTools offer a powerful extension that allows you to inspect and test web pages in real-time. Most other browsers offer this capability but Chrome is hard to beat. It offers:

  • Built-in Lighthouse report capable of grading the accessibility of your page
  • Artificial network throttling to simulate slow connection speeds
  • Performance measurement across pages
  • Hundreds of extra plugins

8. JUnit5 

JUnit5 is a popular open-source unit testing framework that can test Java applications. It offers the ability to have test runners for your test cases and enables you to focus on Test Driven Development.

It even has its own tag on Stack Overflow.

9. Cucumber Open

Cucumber is an open-source testing tool that can be used to test web applications and APIs. Cucumber is a rather unique one on this list in that it focuses on getting the specifications right the first time. It’s called Behavior Driven Development and it allows project managers and technical contributors to collaborate on concrete aspects of the application.

10. Firefox Developer Tools

Firefox does a great job innovating on the developer toolset. (3D View anyone?) Since it can do anything the Chrome DevTool set can, we’ll use this section to focus on specific plugin shoutouts:

User-Agent Switcher

Test your browser detection logic with this extension.

Ghostery

Ghostery is great for a variety of reasons, not least of which is to prevent distractions. It blocks a variety of trackers, ads, and generally improves page performance.

Cookie Manager

A versatile extension that can help you test all sorts of functionality in your application. Cookie Manager can help you with everyday tasks from authentication testing to session switching and inspection. You can additionally export and import cookie sets.

Conclusion

These are some of the best software testing tools available for developers. They all have their strengths and weaknesses, but they are all powerful tools that can help you improve your code quality.

If you’d like to discuss some of the tools you use for your software testing we’ll love to hear about it! Head on over to our community at forum.mailsac.com and discuss your must-have or time-saving tools.