Open Source Spam Processing

Mailsac has added a new spamminess indicator in the API.

New messages will have a spam property that’s a score between 0.0 and 1.0. 1.0 indicates a high likelihood of spam. The system will get smarter over time.

The main component of spam detection is a time tested approach called Naive Bayes Classifier. The core of the code is open source at github.com/mailsac/spam-classifier, though a little special sauce is applied, too. This classifier library is ready to use and includes a simple API server and a terminal training tool.

In the works are additional spam classification projects using three different deep learning techniques: random forest, a long short term neural network, and a combination recurrent network + liquid state machine with spike dependent plasticity. The goal is to open source the useful pieces of the coming spam classifiers.

Parsed Links on Messages API

Message metadata will now include pre-parsed links. This enhancement should make automation even easier.

 

For example, say you want to test email signup confirmations on your website. Normally you would need to request the inbox for the test inbox:

GET /api/addresses/[email protected]/messages
[{
  _id: 'y39sak29gaer-0',
  subject: 'Confirm your email address',
  from: [{ address: '[email protected]' }],
  to: [{ address: '[email protected]' }]
  // more fields
}]

then download the full email or body:

GET /api/dirty/[email protected]/y39sak29gaer-0
<div>
  <a href="https://www.sampleservice.com/email-confirmation/39349234">Confirm your email!</a>
</div>

then parse the contents and search for the link.

Now the message metadata from the first request will also include a links array of strings:

GET /api/addresses/[email protected]/messages
[{
  _id: 'y39sak29gaer-0',
  subject: 'Confirm your email address',
  from: [{ address: '[email protected]' }],
  to: [{ address: '[email protected]' }]
  links: ['https://www.sampleservice.com/email-confirmation/39349234']
}]

This feature should save you a roundtrip to the API and let you not worry about properly parsing HTML links.

 

API Reference

 

Faster response times on message routes

A LRU caching layer, active on each API, was added recently. Since email contents are stored separately from the database and every other microservice, many API requests will get faster. We expect faster response times for recently received messages, messages that are requested repeatedly, and any API request where the message is re-parsed in-flight.

Mailsac stores metadata about every message in MongoDB. In the interest of saving disk and RAM, the full email contents are stored across file servers. When a piece of the email is requested via the API, the full file is fetched and parsed into the desired format. Believe it or not, for Mailsac, CPU is far less resource constrained than disk and RAM, so this trade-off makes sense.

For example, when the email headers are requested, the API begins fetching the full message from the file server, parses the SMTP package, formats the headers, and sends them to the client. Fetching attachments is similar – when an attachment is requested, the API gets the original email from the file server where it is stored, extracts the attachment, and transfers it to the client.

This process involves a lot of disk IO and network latency. With the LRU cache, we seeing 80 ms – 600 ms faster requests, and the cache is used on about 45% of requests that re-parse the email.

Enhanced Outbound Sending API

Until now, all transactional messages sent through the Mailsac API only allowed text. With the latest release, the API supports HTML and attachments. Additionally supported is sending an entire raw SMTP message over the REST API.

New Transactional Email Fields

 

HTML

You can provide full HTML in the html field when doing a POST /api/outgoing-messages. This was not possible in previous versions of the API.

Attachments

Attachments can be sent and linked into the html by including an array of attachment objects in the attachments field.

Example Inline Attachment
{
  "contentDisposition": "attachment",
  "content": "3asfji32gia...93as==",
  "encoding": "base64",
  "filename": "2561a.jpg",
}
Example Attached File
{
  "cid": "[email protected].",
  "contentDisposition": "inline; filename=2561a.jpg",
  "content": "3asfji32gia...93as==",
  "encoding": "base64",
  "filename": "2561a.jpg",
}

Received Headers

Persisting received headers is now possible – send a single string or array of received headers. This allows you to receive SMTP in your application, pre-parse it, and pass along received information.

Raw SMTP over HTTP

 

If you want to test SMTP parsing or have an application that generates raw SMTP messages, put the entire SMTP contents into the raw field when doing a POST /api/outgoing-messages.

Mailsac will parse the headers, text, HTML, and attachments, then deliver it for you.

All other fields will be ignored when you send a raw message.

 

 

Read the API docs for sending transactional emails over HTTP

 

API Reference Overhaul

The API documentation received a fresh new look using a fork of the popular Slate docs framework.

new-docs-look
Significant update to our documentation

API endpoints should be easier to navigate. Many request and response examples were added. Endpoints for downloading attachments are easier to find.

In the coming weeks there will be updates to the docs for the SMTP Web Socket API and the Webhook API, including new examples and guides.

View the new API Docs