How the Coinbase Security team deployed CTFd to Power our First Capture the Flag contest at Defcon…

Posted on by

How the Coinbase Security team deployed CTFd to Power our First Capture the Flag contest at Defcon 27

By Nishil Shah and Peter Kacherginsky

We recently ran Coinbase’s first Capture the Flag (CTF) competition called Capture the Coin. As part of running the CTF, we needed to setup the underlying infrastructure. In this guide we’ll describe our process from choosing a CTF platform to getting it production ready. This CTF definitely would not have been possible without all the open source contributors, projects, tutorials, and free services we leveraged, so we’re making this guide as a small contribution back to the community. Overall, the process took us a few weeks to setup, but with this guide, you’ll only spend a few days.

Choosing a CTF Platform

We chose CTFd as our CTF platform based on the following criteria:

  • Challenge Input Interface
  • Support for Types of Challenges
  • Free Response (Static and Pattern-Based)
  • Dynamic Value
  • Multiple Choice
  • Custom Plugins
  • Public Scoreboard
  • User and Team Registration
  • Admin Panel
  • Player Communications
  • Hints
  • Active Maintainers

CTFd had support for most of our requirements except for multiple choice questions. However, requirements like having active maintainers was useful when the maintainers quickly patched a bug in a few hours. We also wanted to write challenges that forced users to interact with a real blockchain. This would require some custom development, so we also wanted the ability to build our own custom modules. Anecdotally, a few other CTFs had run successfully on CTFd. Given these requirements, CTFd matched our needs the closest.

CTFd mostly worked out of the box by following the README instructions; however, we did run into one issue. Out of the box, CTFd started running really slow. We did some basic debugging like looking at memory and CPU utilization to see if we needed to upgrade our EC2 instance. In general, resource consumption was generally less than 1%. We eventually found out that using Sync workers was inefficient. We changed the settings to use async gevent workers and correctly setting worker_connections and workers using the advice from this post. This solved our issue and CTFd worked great with very little latency.

Capture the Coin Infrastructure

Setting up and running this CTF was a great learning experience in getting hands-on with AWS and debugging real issues. As appsec engineers reviewing code or infrastructure, we often times can become unsympathetic to how hard an engineers’ job can be when defining security requirements even in a #SecurityFirst engineering culture. Just for something as simple as a CTF, the bill of materials starts to pile up as you can see below.

  1. Cloudflare Free Tier
  2. CTFd
  3. Google Recaptcha v2
  4. Infura
  5. Let’s Encrypt/CertBot
  6. Redis (AWS-managed or locally running)
  7. SES Email Forwarder Lambda (Here’s a guide we found very helpful to setup this lambda)
  8. AWS (Paid Tier)
  • EC2
  • IAM
  • Lambda
  • RDS Aurora
  • S3
  • SES

Here’s a network diagram of what our AWS infrastructure looked like at a high level.

Our vendor choices were typically determined by our own familiarity. We used AWS as our infrastructure provider, and we used Cloudflare as our DoS protection for similar reasons. For email and email verification, we used SES to trigger an email forwarder lambda and save a copy of the email to an S3 bucket. The lambda then pulled the email from the S3 bucket and forwarded the contents to the recipients defined in the lambda. Eventually, we swapped out our EC2 instance hosting an Ethereum node with Infura as this gave us reliable connectivity to the ETH blockchain for the smart contract challenges. Our infrastructure is rather simple compared to the complicated microservice environments today, but there was still some complexity involved.

Our Setup Notes

  1. Signup for AWS Account.
  2. Buy a domain with your favorite provider.
  3. Set up Route53 NS DNS records in AWS.
  4. Set up Route53 MX DNS records in AWS.
  5. Set up Route53 SOA DNS records in AWS.
  6. Set up Route53 SPF DNS records in AWS.
  7. Set up Route53 DKIM DNS records in AWS.
  8. Note: The AWS SES Domain Verification step will require an additional TXT record.
  9. Spin up an EC2 micro instance. You can always resize later.
  10. Attach the right security groups. Initially, we limited external access to private IP ranges so that we didn’t accidentally leak challenges until we started the competition.
  11. Spin up RDS instance. We went with the AWS managed solution because this was the simplest and we wouldn’t have to worry about load balancing or patching.
  12. Follow this guide to get inbound and outbound emails working under your CTF domain.
  13. Install CTFd by cloning the repo and installing dependencies. (We used v2.1.3)
  14. Setup Google Recaptcha v2 for spam and DoS protection.
  15. Setup Infura or your own node if you want Ethereum smart contract challenges.
  16. Setup Let’s Encrypt/Certbot for HTTPS during testing and eventually for connections from Cloudflare to CTFd.
  17. Setup AWS-managed Redis or use a Redis instance locally.
  18. Setup port forwarding 80 to 443 and 443 to 8443 so CTFd doesn’t have to run with sudo and http:// is auto-redirected to https://.
  19. Run CTFd! Note that Unix security best practices still apply like running the application under a limited user.
  20. Run the following commands to set all the environment variables.
# CTFd Database
export DATABASE_URL=
# Web3
export WEB3_PROVIDER_URI=
export GETH_ACCT_PASSWD=
# Redis Caching
export REDIS_URL=
# Infura
export WEB3_INFURA_API_KEY=
export WEB3_INFURA_API_SECRET=
# reCAPTCHA
export RECAPTCHA_SECRET=
export RECAPTCHA_SITE_KEY=
cd CTFd
gunicorn3 -name CTFd —bind 0.0.0.0:8443 —statsd-host localhost:8125 —keyfile /etc/letsencrypt/live/capturethecoin.org/privkey.pem —certfile /etc/letsencrypt/live/capturethecoin.org/fullchain.pem —workers=5 —worker-class=gevent —worker-connections=5000 —access-logfile /home/ubuntu/access.log —log-file /home/ubuntu/error.log “CTFd:create_app()”
  1. We found this guidance for optimizing workers and worker-connections helpful.
  2. When finished testing with your running CTFd instance, setup the competition dates in the CTFd admin interface.
  3. Install the Ethereum Oracle CTFd extension and generate contracts.
  4. Add SES SMTP credentials to CTFd admin interface for user registration if you want email verification.
  5. At this point, the CTF should be entirely setup. Only keep going if you’d like to setup Cloudflare for DoS protection.
  6. Setup Cloudflare DNS and modify the AWS security group for the CTFd box to allow ingress from Cloudflare IPs.
  7. Setup Cloudflare SSL/TLS. SSL — Full (strict) if you still have a valid cert from Let’s Encrypt
  8. Setup Edge Certificates — Cloudflare Universal SSL certificate unless you have the budget and security requirements for a dedicated cert or uploading your own cert.
  9. Setup Cloudflare by enabling Always Use HTTPS, HSTS, TLS 1.3, Automatic HTTP Rewrites, and Certificate Transparency Monitoring
  10. Setup Cloudflare by using minimum TLS version 1.2
  11. You can also setup whitelisting for known good IP addresses in Firewall Tools so that Cloudflare doesn’t challenge requests from these ranges.

Dynamic Oracle

One requirement for the competition was to support Ethereum smart contract challenges, after all, this was called Capture the Coin. If no (testnet) coins could be captured, we would not have lived up to the name.

The contest included the excellent CTFd Oracle plugin by nbanmp which allowed us to process on-chain events such as determining whether or not the deployed smart contract was successfully destroyed by the player. We have modified the original oracle to allow for contract pre-deployment since Ethereum Ropsten network may sometimes be unreliable.

You can find the source code for the oracle and contract deployer here: https://github.com/iphelix/ctc-eth-challenge

Future Steps

In the future, we plan to use the dockerized setup so that it is easy to spin up and down the entire CTFd platform. CTFd already allows for codification of the settings which is helpful in being able to get predictable application deploys. We also would like to codify our infrastructure so that we can get predictable and simple deploys.

Thank you to all the open source contributors without their contributions hosting this CTF would not have been possible.

If you’re interested in working on solving tough security challenges while also creating an open financial system, please join us!

This website contains links to third-party websites or other content for information purposes only (“Third-Party Sites”). The Third-Party Sites are not under the control of Coinbase, Inc., and its affiliates (“Coinbase”), and Coinbase is not responsible for the content of any Third-Party Site, including without limitation any link contained in a Third-Party Site, or any changes or updates to a Third-Party Site. Coinbase is not responsible for webcasting or any other form of transmission received from any Third-Party Site. Coinbase is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement, approval or recommendation by Coinbase of the site or any association with its operators.

All images provided herein are by Coinbase.

How the Coinbase Security team deployed CTFd to Power our First Capture the Flag contest at Defcon… was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.