15 minute partial API outage due apparent DOS

For about 15 minutes (8:13 am – 8:27 am PDT), our API was flooded with traffic due to hundreds of thousands of email attempts from 4 IP addresses. Nearly all emails were received, but HTTP requests for the API and UI frequently timed out. We do not know the percentage of requests that timed out, but it was quite high. The API is load balanced and only one API was timing out frequently.

We blocked the bad IPs immediately upon seeing traffic logs (~8:19), but because our custom IP blocking service relies on the API to fetch the blacklist, and the API was not fully responsive to HTTP on one leg, it took a while for the changes to propagate to all five inbound SMTP servers.

Faster response times on message routes

A LRU caching layer, active on each API, was added recently. Since email contents are stored separately from the database and every other microservice, many API requests will get faster. We expect faster response times for recently received messages, messages that are requested repeatedly, and any API request where the message is re-parsed in-flight.

Mailsac stores metadata about every message in MongoDB. In the interest of saving disk and RAM, the full email contents are stored across file servers. When a piece of the email is requested via the API, the full file is fetched and parsed into the desired format. Believe it or not, for Mailsac, CPU is far less resource constrained than disk and RAM, so this trade-off makes sense.

For example, when the email headers are requested, the API begins fetching the full message from the file server, parses the SMTP package, formats the headers, and sends them to the client. Fetching attachments is similar – when an attachment is requested, the API gets the original email from the file server where it is stored, extracts the attachment, and transfers it to the client.

This process involves a lot of disk IO and network latency. With the LRU cache, we seeing 80 ms – 600 ms faster requests, and the cache is used on about 45% of requests that re-parse the email.

Enhanced Outbound Sending API

Until now, all transactional messages sent through the Mailsac API only allowed text. With the latest release, the API supports HTML and attachments. Additionally supported is sending an entire raw SMTP message over the REST API.

New Transactional Email Fields

 

HTML

You can provide full HTML in the html field when doing a POST /api/outgoing-messages. This was not possible in previous versions of the API.

Attachments

Attachments can be sent and linked into the html by including an array of attachment objects in the attachments field.

Example Inline Attachment
{
  "contentDisposition": "attachment",
  "content": "3asfji32gia...93as==",
  "encoding": "base64",
  "filename": "2561a.jpg",
}
Example Attached File
{
  "cid": "[email protected].",
  "contentDisposition": "inline; filename=2561a.jpg",
  "content": "3asfji32gia...93as==",
  "encoding": "base64",
  "filename": "2561a.jpg",
}

Received Headers

Persisting received headers is now possible – send a single string or array of received headers. This allows you to receive SMTP in your application, pre-parse it, and pass along received information.

Raw SMTP over HTTP

 

If you want to test SMTP parsing or have an application that generates raw SMTP messages, put the entire SMTP contents into the raw field when doing a POST /api/outgoing-messages.

Mailsac will parse the headers, text, HTML, and attachments, then deliver it for you.

All other fields will be ignored when you send a raw message.

 

 

Read the API docs for sending transactional emails over HTTP

 

API Reference Overhaul

The API documentation received a fresh new look using a fork of the popular Slate docs framework.

new-docs-look
Significant update to our documentation

API endpoints should be easier to navigate. Many request and response examples were added. Endpoints for downloading attachments are easier to find.

In the coming weeks there will be updates to the docs for the SMTP Web Socket API and the Webhook API, including new examples and guides.

View the new API Docs