Dynamic Serverless Honeynets on AWS

A serverless, dynamically reconfiguring, automated ICS honeynet

Ant Burke, March 2021
ant.b@camulos.io

Introduction

Industrial control systems (ICS) are vital components that underpin our modern world. What were physically isolated "offline" systems are now internet-enabled, integrated with remote admin and monitoring tools - and as a result, exposed to everything that operates on the internet. Operational gains come with increased security and safety risks. Without piling scorn on from afar, the recent breach of a US water processing plant highlights this balancing act (see this this Ars Technica story for a breakdown of events).

The T-Pot honeynet provides an industrial control system (ICS) focused "industrial" profile here. This configuration seeks to mimic internet-connected ICS equipment by running a mix of the following honeypots:
Conpot, Cowrie, Dicompot, Heralding, HoneySAP, Honeytrap, Medpot and Rdpy.

Finding internet-connected ICS devices is easier than ever before. Shodan.io is a very well known and widely used internet device search service that offers powerful search, scan and monitoring features for a broad range of devices and systems. Shodan.io has the fun claim to fame of making an appearance in the Netflix drama Mr Robot - who knew bulk internet port scanning could be so glamorous!

In my earlier article, I described the enjoyable process of combining the awesome T-Pot project (https://github.com/telekom-security/tpotce) with Amazon Web Services (AWS) Elastic Container Service (ECS) Fargate managed containers, building the stack with AWS Cloud Development Kit (CDK) so it can be spun up and down simply running "cdk deploy" and "cdk destroy".

In that blog, I mentioned a few ideas for further work, including a suggestion to build monitoring tools to understand how our honeynet is being profiled by external actors. I decided to spend some time creating a dynamic monitoring solution based on the T-Pot industrial control system (ICS) profile and the shodan.io scanning service.

Why do this?

Similar to my previous article, there are many reasons to pursue this work. The main reasons are:
  • Learning about threat scanning behaviours. Combining T-Pot logs and shodan.io query results gives us a starting dataset for exploring how honeynet target information data from scanning services impacts on the prevalence of attacks.
  • Sharpen skills and knowledge. Hands on work with new tech and new features helps us to stay current with one of our primary areas.
  • Enjoyment and curiosity. This really is our cup of tea!

What did we do?

In a nutshell, I added some features to our serverless honeynet project to search shodan.io for our honeynet IPs and alert me by email when our IPs appear in search results:

HTML5 Icon

Simple architecture view of serverless ICS honeynet plus event-driven shodan.io tracking and alerting

Here is a screenshot of the conpot kibana dashboard after a running online for 3 hours or so:

HTML5 Icon

Conpot kibana dashboard

I spent a few evenings with boto3, AWS CLI, the AWS console and the shodan python client to add the following elements to our CDK-built serverless T-Pot (configured to run conpot)
  • Lambda function to query our ECS public IP addresses, built using boto3.

  • Lambda function that queries shodan.io for an input IP address - if found, results get stored in S3 and an alert email sent, built using boto3 and shodan python library

  • SQS queue to connect the IP query lambda and shodan search lambda

  • Event Bridge scheduled rule that fires every set period (e.g. every 12 hours)
I stopped short of automatically running "cdk destroy" and "cdk deploy" to create a new stack with new IPs when I received a shodan result alert. One next step is implementing analytics to trigger automated teardown and redeploy cycles in response to changes in the external environment, based on observed honeynet traffic and results in shodan. Having a fully automated system that is continually shifting across different public IP addresses, all the while logging data about threats sounds fun.

I plan on running the setup for a while to capture data for an analysis task on my TODO list - before and after analysis on traffic volumes to see if shodan.io appears to influence malicious scanning traffic.

Of course, AWS publish their public IP ranges so it isn't a leap to imagine scanners running a simple "if ICS port on AWS IP, tag as honeypot" type rule... Even so, it will be interesting to capture some real data on scanner behaviours.


Reflections

Looking back at what you've done and considering the good and bad is always valuable. Here are some reflections on an interesting bit of tech dev:

  • Serverless tech is neat. Fast to develop, deploy and iterate. Using an event-driven approach implemented with Event Bridge, Lambda and SQS was an obvious but effective way to create the dynamic feedback loop I wanted for our serverless honeynet.

  • Working with shodan.io is really easy. Getting results from shodan is really simple. One thing to keep in mind is that shodan runs full internet scans on a monthly basis - left to its own devices you might be waiting to see your IP appear in their results. I nudged the process along by requesting a shodan scan (you can buy a useful developer account for $49 at https://www.shodan.io/store/member). A developer account allowed me to trigger scans to run different data collection trials.

  • Conpot sticks out like a sore thumb to shodan. Our IP is immediately identified as an AWS IP address in shodan results. Several hours later, our IP is tagged by shodan with metadata identifying it as a honeypot. Nevertheless we still saw ~1500 scans logged by T-Pot in the first 24 hours of running.

There are improvements that I ought to make to this prototype - ever more realistic honeypots, tracking other scanning services and building smarter automation analytics to name a few. Here are some ideas for the future:

  • AWS publishes their IP ranges! Yes I know, and I expected fewer hits on conpot. IT would be very nice to develop a hybrid architecture solution (on-prem containerised honeynet, AWS serverless tracking) that lives behind IP addresses that aren't public cloud platform IPs.

  • shodan.io spots Conpot fast! Yep, conpot is identified as a honeypot very quickly by shodan.io scans. There is much scope to explore more realistic, hard to fingerprint containerised honeypots that can be deployed in this architecture. Recent research points to new ways of creating and running honeypots that offer an interesting avenue for further research in the context of serverless honeynets.

  • Managed instance of Elastic stack. Why run our own Elastic stack as a set of containers when AWS can help take the load? Yet more serverless by looking into AWS Elastic.

  • Shift from EFS to S3/CloudWatch. Our T-Pot honeypot containers write their logs to EFS. Given we have modified several containers, we could change the behaviour to send logs to S3 and/or CloudWatch to support easier analytics with AWS services.

  • Multi-vendor cloud deployments. This one was said before - I really should get round to repeating this work in Azure and Google Cloud Platform...

Contact

Email me at ant.b@camulos.io if you have any questions or suggestions - let me know if you would like the project to get open sourced!