It's AWS re:Invent season again and you can expect many new product launches accompanied by a bazillion of new features for existing products. We've already seen a lot of great announcements and I suspect still many more to come! For people like me who build things in the cloud, it's almost as exciting time as holiday season for a kid who eagerly awaits what presents Santa Claus will bring! But I'm already looking ahead, to year 2021 and here's some of my wishes for AWS.
As exciting the re:Invent season with all the product and feature launches is, there are some small under-the-radar features I'd really love to see AWS to improve on. Based on various bits of information what I've been able to gather around the internet (such as the public roadmaps and Github/StackOverflow issues etc), I don't expect to see these features to be addressed during the re:Invent 2020: Which is why I am writing this blog post, in the hopes it will increase the visibility of these issues.
These are things that might not seem so big-of-a-deal at first and they're definitely not show stoppers, but they can be quite annoying if you happen to bump into them; And once you start to work around them, you soon realize they're something that could almost be described as undifferentiated heavy lifting in 2020.
“Customers want to focus on their unique application logic and business needs – not on the undifferentiated heavy lifting of provisioning and scaling servers, keeping software stacks patched and up to date, handling fleet-wide deployments, or dealing with routine monitoring, logging, and web service front ends.”
- Cross-Region ACM certificates with CloudFormation
- CloudFront Origin Protection for VPC resources
- Unified authentication experience across all AWS SDKs & CLI tools
- Aurora Serverless with RDS Proxy
Cross-Region ACM certificates with CloudFormation ¶
This applies to both CloudFormation and to AWS CDK which is based on CloudFormation. I've described the issue in detail with examples in an issue comment in CloudFormation coverage roadmap repo, but to summarize: It's a pain point with ACM
SSL/TLS-certificates used by CloudFront if you deploy your infrastructure in any other region than
us-east-1 (North-Virginia) which is the “main region” for CloudFront.
The core of the issue is that CloudFront requires ACM certificates to be in
us-east-1 region, even if you deploy the CloudFront distribution via another region. Even though
us-east-1 is the “main region” for CloudFront, it's still a global service (a bit like IAM or Route53) to which you can deploy from any AWS region; And often you want to deploy via your specific region where your origins (such as API Gateway endpoints, AppSync APIs, Load Balancers, S3 buckets, etc) are defined.
You can hack around this problem by using:
- CloudFormation Custom Resources (with CDK you might want to utilize
acm.DnsValidatedCertificate) to define a Lambda function that creates the certificate in
us-east-1via AWS SDK
- Separate CloudFormation stack deployed to
us-east-1and storing the certificate ARN into your target region as SSM Parameter Store via CloudFormation Custom Resource
- Separate CloudFormation stack deployed to
us-east-1and then some local script that parses the certificate ARN from the CloudFormation output and passes it locally into the stack with CloudFront distribution as a stack parameter.
Each of these hacks have their downsides: The first often fails due to combination of Lambda maximum timeout of
15 minutes and ACM taking up to
30 minutes to validate the certificate. Second one means you must have a reliable path naming convention for SSM parameters and it still involves working with CloudFormation Custon resources which can be a pain to manage some times. Third one also adds some unnecessary custom complexity to your deployments and acts as a likely point of failure.
Why not just deploy the certificate manually you may ask? ¶
An example scenario is when you might want utilize a modern setup with multiple environments and you might want to have a different distinct subdomains for these environments, let's say
api.staging.example.com: Now CloudFront requires that the ACM certificate “proves” the ownership of that domain at the exact specified level, which in turn means we can not create a certificate with just wildcard value such as
*.example.com but instead we need to have certificate with a SAN of
*.staging.example.com. This is easy if you know all your environments ahead of time, but if you're using more dynamic environments such as feature/preview environments that are deployed by the CI-service depending on things like
git branch names - resulting in domain like
api.new-thing.feature.example.com - you're out of luck.
If you want to learn more about this issue and the context where this issue really matters, do checkout this issue comment.
The proposed solution ¶
Having some kind of option to define either
REGIONAL type in CloudFormation for ACM certificate would just resolve the whole issue really nicely. On that note, I'd kinda love to see the same cross-region deployment support for
CloudFront Origin Protection for VPC resources ¶
There's yet another blog post about a Lambda-based solution on updating security groups with CloudFront IP ranges. I've been saying for years that in my opinion this is unnecessary, undifferentiated heavy lifting.
Origin protection should be something that (in a “perfect world”) AWS would provide out-of-the-box. Yes, one can always set custom headers etc in CloudFront that the app servers are validating against, but still, there is a clear need for protecting your origin load balancer in network level with security groups.
In a perfect world, security groups could utilize the Origin Access Identity construct that already exists in CloudFront (and currently can be used with S3 origins): So that one could set a specific OAI as allowed inbound source in a security group. This would solve two things regarding VPC-based origins:
Origin could only be accessed via (any) CloudFront - would remove the need for those Lambda-based solutions updating CloudFront IP ranges to Security Groups
Origin could only be accessed via specific CloudFront distribution - would remove the need to do some header-value checking in the origin application
In a not-so-perfect world, if the above would be impossible (point 2, allow access only via specific distribution), then to solve the point 1 (allow access via any CloudFront) atleast, maybe there could be an AWS-managed Prefix list since - a construct like that already exists for similar purposes - for CloudFront IP ranges?
Fellow AWS Community Builder, Ben Bridts, pointed out a workaround using WAF & Secrets Manager (also see his blog post), which solves the issue of protecting the origin by allowing access only via specific CloudFront distribution, BUT it still requires quite a lot of setup and I think it still counts as that undifferentiated heavy lifting I'd like to avoid.
Aurora Serverless with RDS Proxy ¶
One of the most exciting launches of re:Invent 2020 for me has been Aurora Serverless
v2 (still in
preview). It solves a lot of the pain points in Aurora Serverless (
v1). I can't wait to get my hands dirty and start playing with the
v2! If you haven't heard about the Aurora Serverless
v2 yet, do read the excellent post by AWS Serverless Hero Jeremy Daly.
There is a strong indication that ASv2 will support all the major features of AWS Aurora once it goes to GA, but one feature I really, really, hope it will support is the RDS Proxy! I think the lack of RDS Proxy support in ASv1 has been the biggest feature miss in that product.
Yes, there is the Data API which is cool, but it's not the same thing as using real SQL-connection which allows using all the possible SQL-libraries and tools out there! Even ORMs if you so wish! Also, it makes local development somewhat more cumbersome, as you need to mock the AWS Data API responses. Instead, if you'd be able to use SQL-connections, then for local development you could just spin up a SQL-server of your choice inside a Docker container and hack away!
I guess there are valid technical reasons why AWS has not yet enabled pairing these two, but I really hope they will once ASv2 goes GA since Aurora Serverless and RDS Proxy would be a match made in heaven!
Limitations for RDS Proxy
• You can't use RDS Proxy with Aurora Serverless clusters.
Have to add: I really wish ASv2 will support RDS Proxy for both
PostgreSQL! 🙏 (I personally prefer PostgreSQL 🙃)
Unified authentication experience across all AWS SDKs & CLI tools ¶
The AWS CLI
v2 and the Python SDK (
boto3) do now have nice support for MFA, assuming named profiles from
~/.aws/config and caching temporary session credentials - so you don't have to insert the MFA token for each operation. The challenge is that not all AWS SDKs and CLI tools have the same level of support: For example AWS CDK added MFA support a while back now, but it lacks the support to cache the temporary session credentials; Same goes for several SDKs. Which makes sense, since they are all separate implementations.
The other elephant in the room is AWS SSO: On the surface, it is a great product and it also plays really well together with AWS CLI v2, but again tools such as CDK is missing support for SSO named profiles and the situation seems to be the same for the other SDKs.
There are community-driven projects such as the awesome
aws-sso-util built by Ben Kehoe that address these issues with AWS SSO; But even though Ben is a well-known figure in the community (which means I am keen to trust his implemenation), generally speaking I think that having to resort to 3rd party code for authentication in the enterprise just feels a bit wrong! And even if you accept the fact of using 3rd party code for the process in general, there's still the issue of having to teach hundreds (or in some larger enterprises even thousands) of developers how to setup this custom tool that is not maintained or documented by AWS!
I really hope 2021 is the year AWS manages to unify the authentication experience across the board and also fix the rough edges related to AWS SSO.
“Not all AWS SDKs have support for AWS SSO (which will change eventually). However, they all have support for
credential_process, which allows an external process to provide credentials.
aws-sso-util credential-processuses this to allow these SDKs to get credentials from AWS SSO. It's added automatically (by default) by the
While I really have been enjoying all the awesome product/feature launches (that have been made so far) of re:Invent 2020 such as the Lambda Container Image Support,
1ms billing granularity for Lambda, Aurora Serverless
v2, Aurora PostgreSQL Lambda triggers, S3 Strong Read-After-Write Consistency and many others - I have to say that fixing these four issues I've described in this blog post would make the day-to-day development for me and many others a lot more fun! 🤩
Generally speaking 2020 has been a
shit challenging year due to the pandemic as everybody knows, so here's to an awesome 2021 and who knows, maybe we'll even see each other at re:Invent in Las Vegas! 🥂
… and dear AWS, please put a cherry on top by implementing that wishlist! 😉