Creating comments via email in Drupal 7

For a recent project we needed to allow users to comment on content via email. The idea was that when new comments were created, Drupal would send out notification emails that could be replied to. The system had to be able to handle long email chains and a few edge cases. Email is tricky and we learned a lot about the wild world of email. We are sharing our experiences here.

Mail Comment lets you reply via email to notifications from your website. In this article we’ll show you how to reply to content via email and outline the mechanisms that make it work. There are quite a few modules involved in the process and they all have an important role to play.

Modules

You will need the following modules installed:

• Rules
• Message
• Message Notify
• Mail Comment
• Mail Comment Message Notify
• Mailhandler
• Feeds
• Feeds Comment Processor

The Workflow

Here’s a typical workflow and how each module fits into the process:

1. User creates a comment
2. Drupal sends a notification email to recipients (Rules + Mail Comment Message Notify)
3. A recipient replies to this notification email via their email client
4. The reply email is sent to the “From” header which goes to our mailbox (Mailhandler)
5. Drupal imports the email from this mailbox (Feeds importer)
6. Drupal validates the email’s authenticity (Mail Comment)
7. Drupal maps the email’s metadata to comment fields (Feeds + Mail Comment)
8. Drupal creates a comment (Feeds Comment Processor)

The Steps

With the basic concepts in hand – let’s build out this functionality.

Create a Message type

The Message framework provides an entity type designed specifically for dealing with messages. Messages are great for creating and sending notifications. Create a message type via Structure -> Message types -> Add message type. This message type will represent a notification that occurs when someone posts a comment.

In the “Description” field label your message “New Comment”. The mandatory “Message text” field will become the notification email subject. The form requires a save before correctly displaying “Message text” fields and their associated view modes (a UI bug). Put any value in this field as we’ll be changing it after saving this form.

If you’ve previously enabled Mail Comment then you will see a very important checkbox “Allow users to reply to this message by email”. This has to be ticked.

Save the form and edit the newly created message type. Now you will see two special view modes which will be displayed as the notification email’s Subject and Body.

In “Message text -> View modes: Notify - Email subject” enter a subject:

New comment created on [nid]

In “Message text -> View modes: Notify - Email” enter a body:

[comment:user:mail] says: [comment:value] View: [comment:node:url]

Save the form and then click “Manage fields”. Create two fields:

• field_message_rendered_subject (Text field)
• field_message_rendered_body (Text area)

These fields are used by Message Notify to save the output of the Subject and Body view modes for use with email.

Create a notification rule

Now that the message type is set up we can create a notification rule that is triggered whenever someone creates a comment.

1. Create a rule that reacts on the event “After saving a new comment”.
2. Add the action “Send Message with Message Notify”.
3. In the “Data selector” select the comment that triggered this rule.
4. Set “Rendered subject field” to “Rendered subject” and “Rendered body field” to “Rendered body”.
5. Set the “The recipient of the email” to any email address you’d like.

Configure Mail Comment

Mail Comment works its magic on both sides. It modifies outgoing emails so that they can be replied to, and authenticates incoming emails and extracts metadata. Mail Comment does this by embedding a unique string in the outgoing email that contains email metadata and a hash, which we will go into more detail later.

Go to Configuration -> System -> Mail Comment (/admin/config/system/mailcomment).

Set “Reply-To address“ to the email account that the Feeds importer will be importing from.
“Insert reply to text” is a handy option to have enabled. It will help Mail Comment separate the latest content in an email from the email’s thread. Other settings are fine to leave at their defaults.

Create a Mailhandler Mailbox

Email replies will be sent to a mailbox that you control. Mailhandler allows you to create a connection to this mailbox. A Feeds importer will then be importing emails from this mailbox via Mailhandler.

Create a mailbox via Structure -> Mailhandler Mailboxes -> Add (admin/structure/mailhandler/add).
The “Username” is likely the same value as Mail Comment’s “Reply-To” email address.

Create a Feeds importer

The Feeds importer extracts the email via Mailhandler's mailbox. It runs via cron.

Create a Feeds importer via Structure -> Feeds importers -> Add importer. (/admin/structure/feeds/create)
Label it “New Comments”.

Fetcher

Configure the fetcher’s “Message filter” to “Comments only”. This filter works by checking if an email’s In-Reply-To header is set. At this time of writing the filter doesn’t handle all email situations correctly. Please see this comment for a solution.

Parser

Select “Mailhandler IMAP stream parser” with the “Authentication plugin” as “Mail Comment Authentication”. Mail Comment authenticates and validates the incoming email to ensure that it was originally sent from the website and replied to by a valid user.

Processor

Feeds’ importer processor creates the comment. Select “Comment processor” as your processor. For our particular project at Morpht we created messages instead of comments with “Message entity processor” and a custom module – but that’s another story.

The Mapping configuration here is where all the magic happens. Metadata is pulled from the email and exposed as a source value. The following source values need to be mapped from source to target:

• Message ID (message_id) -> GUID (guid)
• Parent NID (parent_nid) -> Node id (nid)
• User ID (authenticated_uid) -> User ID (uid)
• Subject (subject) -> Title (subject)
• Body (Text) (body_text) -> Comment (comment_body)

Message ID and Parent NID are source values created by Mail Comment from the email’s metadata.

Summary

The steps above detail how to reply to notifications sent when someone create a comment. Similar steps can be used to send notifications when someone creates a node. Create a new message type called “new_node” and follow the steps above, replacing the comment specific configuration with node config.

Testing

Test the system by creating a comment on a node through your website’s UI. A notification email should be sent out to recipients, including your testing email address. Reply to this email, run cron and the comment should be imported. To assist your debugging checkout Content -> Messages and the Maillog module.

Dealing with email clients

If you’d like to know what’s going on under the hood feel free to read below. There’s a reward for reading on. This section will also show you how to reply to long email chains and still have the email imported by Drupal. It also fixes a few bugs with the current version of Mail Comment at the time of this writing.

The Message-ID

<1234.4666.5056.1486965753.cb17604ac1434f792f04f724d2af7ee6@example.com>

Mail Comment stores important information in an email by creating a custom string full of metadata about the original comment. This string is sent in the email inside the Message-ID header. Message-ID is a standard email header that email clients use to insert identifying information about the email. Usually it’s a big hash but in our case it’s a dot separated string of values containing:

• Node ID
• Comment ID
• User ID
• Timestamp
• Hash

The format is four digits and one hash divided by dots, an @ symbol and finally the domain that the email was sent from.

Long email chains

Mail Comment’s metadata in the Message-ID header will be lost as soon as someone replies to an email. The email client replying to the email will overwrite Message-ID with its own value. Thankfully there is another email header specifically designed to store a history of all Message-IDs used in the email chain’s lifetime: References. The References header stores each Message-ID in chronological order like so:

<4431.5196.7356.1487571103.20e2a3d69bef550a990d7c751beb84de@example.com> <CAPbhD2K4MvcCPqSNRhp33P+CoNkY5xyb+qNqeJBuhMrxeMCBsg@mail.gmail.com> <CAPbhD2JZdgQj==1AxDWqaR2N=jrQtvrz3hQ-03b9jkgJg5eoFA@mail.gmail.com>

As you can see the original Message-ID is on the first line.

Email clients send emails in unexpected formats and can sometimes break the Message-ID metadata. This is especially true with Microsoft Outlook/Exchange, which inserts a seemingly random value at the beginning of the References header (which is actually the third value of the dot separated string).

7356 <4431.5196.7356.1487571103.20e2a3d69bef550a990d7c751beb84de@domain.com> <CAPbhD2K4MvcCPqSNRhp33P+CoNkY5xyb+qNqeJBuhMrxeMCBsg@mail.gmail.com> <CAPbhD2JZdgQj==1AxDWqaR2N=jrQtvrz3hQ-03b9jkgJg5eoFA@mail.gmail.com>

Our Patch

We have created a patch to better detail with oddly-formatted References headers and long email chains. Feel free to apply.

HTML being sent as plain text

Microsoft Outlook/Exchange occasionally sends out badly formatted HTML in the plain text version of the email. There’s no silver bullet solution. We came up with a simple fix that just strips out all the bad HTML leaving you the plain text.

In your custom module create the following function:

<?php
/**
 * Implements hook_feeds_after_parse()
 */
function mymodule_feeds_after_parse(FeedsSource $source, FeedsParserResult $result) {
  foreach ($result->items as &$email) {
    // Filter html
    $email['body_text'] = _mymodule_filter_html($email['body_text']);
    $email['body_html'] = _mymodule_filter_html($email['body_html']);
  }
}

This hook will run just after feeds has parsed in values, and call the filter html function below:

<?php
/**
 * Strip html
 */
function _mymodule_filter_html($text) {
  // remove all tags
  $text_remove = strip_tags($text);
  // decode html special characters
  $text_remove = html_entity_decode($text_remove, ENT_QUOTES);
  // replace multiple empty lines with on
  $text_remove = preg_replace('/\n(\s*\n)+/', "\n\n", $text_remove);
  return $text_remove;
}

Once these two functions are pasted in, clear Drupal’s cache (drush cc all).

Conclusion

Now you know how to create content in Drupal from email! The basic ingredients are Mail Comment, Feeds importer and a few modules gluing it all together.

You can use similar techniques to create nodes from email too. Just be mindful that there are security implications doing this as you can’t always trust the sender and contents of an email. Mail Comment authenticates incoming mail by creating a hash and this is harder to do when importing an email without a Mail Comment generated Message-ID.

As Drupal heads more and more towards being a REST based backend it’s easy to forget the more traditional means of inputting data. Email is ancient on the technology timeline, but its convenience coupled with Drupal’s well defined data structure can be a great combo. Happy mailing!