October 8, 2018

Let's Get Social!

By Martin Hollstein

Let's Get Social: Using AWS Lambda for Content Sharing Automation

Automation is amazing. The more one can automate processes - typically - the less work one must do. In the past, small one off automation tasks: file transfers, ETL, and scheduled batch jobs, would still require writing a full executable or using a feature constrained scripting language. Beyond that, one would also need to provision servers, VMs, and task managers to run the automation tasks. Thankfully, the present offers so many options to ease the pain of small automation jobs. One such product is AWS Lambda.

An example of some awesome automation one can achieve with AWS Lambda is to leverage it for sharing content across social media websites and community forums. In this post, one will learn an overview of the AWS services needed to accomplish scheduling events to read from an RSS feed and using the data from the feed to re-post the content to Twitter and Reddit.

Overview

There are a few steps one needs to accomplish to fulfill RSS re-posting functions. These steps are as follows: Develop a function triggered by scheduled events, configure pub-sub with AWS SNS, and any programming language required by the platform SDKs (or helpful libraries) for said social platforms and supported by Lambda to reduce delivery time.

Event Handlers

One of the most interesting portions of this task is writing out the Lambda Functions necessary to complete the task at hand. From a high level the following requirements are:

  • RSS Reader Triggered by Scheduled Event
  • SNS Topic
  • SNS Subscribers for each social media integration
  • Content and Social Media Platforms to target

With those topics in focus, once can start automating content sharing with AWS Lambda.

Scheduled Event Handler

The start of all of this sharing is triggered by a CloudWatch Alarm, publishing to SNS, subscribed to by our AWS Lambda Function. To demonstrate the ease at which features with a single unit of responsibility can be developed, here is the function that handles the scheduled event:

using System;
using System.Linq;
using System.Threading.Tasks;
using Amazon.Lambda.Core;
using System.Xml.Linq;
using Newtonsoft.Json;
using System.Text.RegularExpressions;
using System.Net;
using Amazon.SimpleNotificationService;
using Amazon.SimpleNotificationService.Model;

// Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class.
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.Json.JsonSerializer))]

namespace LetsGetSocial.ScheduledLambda
{
    public class Function
    {
        private readonly string FeedURL;
        private readonly IAmazonSimpleNotificationService SNSClient;

        // AWS Lambda will invoke the default constructor, we use the parameterized constructor for unit testing

        public Function()
        {
            FeedURL = Environment.GetEnvironmentVariable("rss_source");
            SNSClient = new AmazonSimpleNotificationServiceClient();
        }

        public Function(string atomFeedURL, IAmazonSimpleNotificationService snsClient)
        {
            FeedURL = atomFeedURL;
            SNSClient = snsClient;
        }

        public async Task<string> FunctionHandler(dynamic input, ILambdaContext context)
        {
            try
            {
                // Get atom feed from our source
                var request = await WebRequest.CreateHttp(FeedURL).GetResponseAsync();
                var latestAtom = XDocument.Load(request.GetResponseStream());

                // Filter our data
                var latestEntry = latestAtom.Root?
                    .Elements()
                    .FirstOrDefault(element => element.Name.LocalName.Equals("entry"));

                if (latestEntry == null)
                {
                    return null;
                }

                // Map the atom feed for the desired content
                var entryLink = latestEntry.Elements().First(i => i.Name.LocalName.Equals("link")).Attribute("href").Value;
                var entryTitle = latestEntry.Elements().First(i => i.Name.LocalName.Equals("title")).Value;
                var entryContent = latestEntry.Elements().First(i => i.Name.LocalName.Equals("content")).Value;

                // Another map of content for our domain
                // BlogContentUpdated is a user defined class used for serialization and domain validation.
                // (We are assuming this RSS feed is from your own blog - but it can be from anywhere)
                entryTitle = entryTitle.Length > BlogContentUpdated.MaxTitleLength ? entryTitle.Substring(0, BlogContentUpdated.MaxTitleLength) : entryTitle;
                entryContent = Regex.Replace(entryContent, "<.*?>", String.Empty).Substring(0, BlogContentUpdated.MaxSnippetLength)

                var updateMessage = new BlogContentUpdated(entryLink, entryTitle, entryContent);
                var publishRequest = new PublishRequest
                {
                    TopicArn = Environment.GetEnvironmentVariable("topic_arn"),
                    Message = JsonConvert.SerializeObject(updateMessage)
                };
                var response = await SNSClient.PublishAsync(publishRequest);
                return response.MessageId;
            }
            catch(Exception e)
            {
                // Any errors will be logged to CloudWatch logs (if the Lambda execution role allows it)
                context?.Logger.LogLine("Error with syndication.");
                context?.Logger.LogLine(e.Message);
                context?.Logger.LogLine(e.StackTrace);
                return null;
            }
        }
    }
}

Note: BlogContentUpdated is a domain object one can define to implement this function.

From there, the rest is deploying the function and configuration in AWS for the topic_arn and rss_source parameters in the AWS Lambda Function's environment variables.

SNS Subscribers

Once the AWS Lambda Function for reading RSS content is created, deployed, configured, and the CloudWatch alarms are all setup: one can continue with building and deploying AWS Lambda Functions to subscribe to the previously defined SNS topic. For this example, one will see how to write a C# subscriber to repost to Twitter using CoreTweet and another for Node using snoowrap to re-post to Reddit as a user.

Twitter Sharing Function ( C# )

The first function one will explore is the Twitter Content Sharing function that is subscribed to the previously mentioned SNS topic.

using Amazon.Lambda.Core;
using Amazon.Lambda.SNSEvents;
using CoreTweet;
using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.Text;
using System.Threading.Tasks;
using System.Linq;

// Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class.
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.Json.JsonSerializer))]

namespace LetsGetSocial.TweetIt
{
    // Adapter interface to make Unit testing simpler
    public interface ITwitterService
    {
        Task<string> Tweet(string status);
    }

    // Private implementation for example purposes
    private class CoreTweetProvider : ITwitterService
    {
        public override async Task<string> Tweet(string status)
        {
            var token = Tokens.Create(Environment.GetEnvironmentVariable("consumer_key"),
                    Environment.GetEnvironmentVariable("consumer_secret"),
                    Environment.GetEnvironmentVariable("auth_token"),
                    Environment.GetEnvironmentVariable("auth_secret"));
            var response = await token.Statuses.UpdateAsync(status);
            return response.CreatedAt.ToString();
        }
    }

    public class Function
    {
        private const int MaxTweetLength = 280;
        private readonly ITwitterService service;

        // Default constructor for AWS Lambda to invoke
        public Function()
        {
            service = new CoreTweetProvider();
        }

        // DI constructor for unit testing.
        public Function(ITwitterService service)
        {
            this.service = service;
        }

        public async Task<string> FunctionHandler(SNSEvent input, ILambdaContext context)
        {
            try
            {
                var messageJSONString = input.Records[0]?.Sns.Message;
                context?.Logger.LogLine($"Received({input.Records[0]?.Sns.MessageId}): {messageJSONString}");
                if (messageJSONString != null)
                {
                    var magicCharCountForFormat = 6;
                    // Again BlogContentUpdated is one's defined domain object, we will assume the properties defined in the following.
                    // Here, we are doing a lot of transformations needed to make a valid tweet.
                    var messageContent = JsonConvert.DeserializeObject<BlogContentUpdated>(messageJSONString);
                    var snippetLength = MaxTweetLength - messageContent.PostTitle.Length - magicCharCountForFormat;

                    var content = messageContent.ContentSnippet.Length > snippetLength ?
                        messageContent.ContentSnippet.Substring(0, snippetLength) :
                        messageContent.ContentSnippet;

                    var tweetStatus = $"{messageContent.PostTitle}. {content}... {messageContent.PostLink}";

                    return await service.Tweet(tweetStatus);
                }
                else
                {
                    return null;
                }
            }
            catch(Exception e)
            {
                context?.Logger.LogLine("Unable to Tweet SNS message");
                context?.Logger.LogLine(e.Message);
                context?.Logger.LogLine(e.StackTrace);
                return null;
            }
        }
    }
}

Note: BlogContentUpdated is a user defined class needed for the implementation.

The majority of the work is done by the CoreTweet Library. Also notice that there are no hard coded auth tokens - a good practice, especially since environment variables for lambda can be stored as encrypted values.

Core Tweet

Enough has been said about Core Tweet that it warrants it's own section. Setting up Core Tweet was extremely simple. It involved two multi-part steps. The steps were to configure Twitter to enable app authorization for your account and install the Core Tweet NuGet Package for .Net Core.

Here are the steps simplified:

  • Go to the Twitter Apps website.
    • Log in to the Twitter account you want to have the Tweets pushed to.
    • Select Create New App Button.
    • Follow the Instructions
    • Under Application Settings Authorize the App the access the account you are logged into.
  • Setup your Lambda
    • Upload your function or create it inline
    • Save your keys and access tokens from Twitter as encrypted Lambda Environment Variables

Reddit Sharing Function ( Node )

Now, one should pause right here and contemplate what just happened. The programming language switched. Yes, that is right - with AWS Lambda, one can execute Functions in a myriad of languages. This is to support a wider range of developers, engineers, and programmers. This can also apply to the same project. In the world of micro-services, micro-apps, cloud, containers, and serverless - language is the least of a project's concerns. Picking a language is as simple as picking the path of least resistance. Take a look at a Reddit API SDK library.

These days, to select a language - especially for AWS Lambda and serverless, one just needs to pick the intersects of the answers to the following questions:

1) What programming language skills does my team currently have or are interested in learning? 2) What programming languages, frameworks, and platforms does AWS Lambda support? 3) Which packages in a particular programming language will assist us in writing smarter and less code?

For this implementation, the answers to those very questions resulted in Node. And here is how much work went into writing the Lambda function to share content to Reddit - implemented in tens of lines of code:

var snoowrap = require('snoowrap');

exports.handler = function (event, context, callback) {

    if (event != null) {
        var record = event.Records[0];
        if (record != null) {
            const {
                accessToken,
                userAgent,
                clientId,
                clientSecret,
                refreshToken,
                subreddit,
            } = process.env;

            const {
                MessageId,
                Message,
            } = record.Sns;

            console.log(`Received(${MessageId}): ${Message}`);
            const {
                PostTitle: title,
                PostLink: url,
            } = JSON.parse(Message);

            const r = new snoowrap({
                accessToken,
                userAgent,
                clientId,
                clientSecret,
                refreshToken,
            });

            const linkToSubmit = {
                title,
                url
            };

            console.log(`Sending: ${JSON.stringify(linkToSubmit)}`);

            r.getSubreddit(subreddit)
                .submitLink(linkToSubmit)
                .then(result => callback(null, JSON.stringify(result || { msg: 'Success' })));
        }
        else {
            console.log('null record');
            callback(null, null);
        }
    }
    else {
        console.log('No event object');
        callback(null, null);
    }
};

Snoowrap

Parallel to CoreTweet - there are a few configuration steps required before executing the function in AWS Lambda. First, one will need to create an application in Reddit by clicking the button at the bottom of the page. After the script app is created in Reddit, one can create permanent credentials for the automated account using the usual OAuth 2.0 flows. Finally, one simply needs to securely add (via AWS KMS encryption) the generated tokens to the Lambda environment variables.

That Is A Wrap

Indeed, automation is amazing. Hopefully this has been a helpful and valuable look into how to automate various business needs with AWS Lambda. Beyond that, it is encouraged to look at how much this enables developers to focus on the real work that matters to achieve a certain goal. While the world is focused on AWS Lambda for serverless - keep in mind the power it offers for simple automation jobs in regards to business requirements.


Martin Hollstein - AWS Certified Solutions Architect

About Martin Hollstein

Martin Hollstein is an AWS Certified Solutions Architect Consultant at Centare. His developer and architectural experiences have ranged from multi-player games, aeronautics, e-commerce, financial services, and healthcare. His focus is designing highly distributed cloud applications and implementing those designs with teams as a full stack software developer.