In this blog, we'll be learning how to set up an AWS Site-to-Site VPN.

When connecting on-premises network to AWS, there are two main options: IPSec VPN and AWS Direct Connect.

1. IPSec VPN: This option sets up a secure VPN connection between your AWS VPC and on-premises network, with traffic encrypted and transmitted over the internet. It's a popular choice for small to medium-sized companies due to its ease of setup. Once established, both networks can communicate seamlessly over private IP addresses.

2. AWS Direct Connect: This is a dedicated, point-to-point physical connection between your on-premises network and an AWS data center. It offers a more stable and faster connection but requires more setup and is generally used by larger enterprises.

In this blog, we will be setting up an AWS VPN connection.

Overview

On the AWS side, we have a VPC and a Virtual Private Gateway (VGW), which function as the AWS-side endpoint for the VPN connection. On the on-premises side, we need a router, known as the Customer Gateway (CGW), which must be accessible over the internet and have a public IP address.

We can set up the AWS-side network in our chosen AWS region. Since we don't have access to an actual on-premises network, we'll simulate the corporate network.

We'll create another VPC in a different region and use an EC2 instance with Openswan software as our router. In a real-world scenario, well-known routers are typically used for VPN connections, but to minimize costs, we'll use this setup.

Openswan is an open-source implementation of IPsec (Internet Protocol Security) for Linux. It enables secure communication between systems over an untrusted network, like the internet, by creating encrypted tunnels. It’s commonly used for setting up VPNs to protect data in transit.

On the left side, we have the AWS network, and on the right side, the simulated corporate network. The EC2 instance on the AWS side does not need internet access, so we will create a private subnet in the VPC with an EC2 instance that has only a private IP address.

On the right side, we'll have a VPC with a public subnet because we need to launch an EC2 instance there and configure Openswan on it. To set up Openswan, we need to connect to this instance over the internet, which is why it must be placed in a public subnet.

The AWS-side instance can only be reached from the on-premises EC2 instance once the VPN connection is established.

Task 1: Set Up AWS Infrastructure

Let's create VPC.

Search for VPC in the search box.

For the AWS side, I'm choosing Mumbai region.

Provide a name & CIDR range for your VPC.

Keeping all other configurations as is, click on Create VPC.

Now, it's time to create a private subnet.

Select your VPC.

Provide the name and eligible CIDR value for your subnet, then click on Create subnet.

Let's create route table now.

Provide name for your route table, select your VPC and click on Create route table.

In the public subnet's route table, we need a route for the internet, and we add the corresponding entry. However, in this case, we don't need that but we need to associate this route table with the subnet we created.

Click on your route table and in the Subnet associations tab, click on Edit subnet associations.

Select your subnet and click Save associations.

Alright, now it's time to create an EC2.

Go to the EC2 console.

Click on Launch instances.

Make sure to select the Amazon Linux 2 AMI, as the latest version (Amazon Linux 2023) does not support Openswan, which we are going to implement.

Create key pair for your instance.

Edit Network settings. Choose your VPC and subnet.

Keep Auto-assign public IP as default.

When configuring the security group (SG), refer to the network diagram and ask yourself, 'From where will this EC2 instance receive traffic?'

Since the traffic will only come from the on-premises network 11.0.0.0.0/16, we should allow all traffic from that network.

Allow all TCP traffic on every port from the on-premise network's CIDR range.

To enable ping from the on-premises EC2 to the AWS-side EC2, ensure that the ping protocol (ICMP) is also enabled.

Keeping all other configurations as is, click on Launch instance.

Task 2: Setup On-premise Infrastructures

Let's start by creating a VPC. This time, I am choosing a different region.

Provide a name for your VPC and provide CIDR range.

Note that your VPC's CIDR range must not overlap with each other.

Click on Create VPC.

Since I need a public subnet, it's required to attach an Internet Gateway to the VPC.

Provide name for your Internet Gateway and click on Create Internet gateway.

After creating your internet gateway, attach it to your VPC.

Now, let's create our public subnet.

Select your VPC.

Provide a suitable name, provide an eligible CIDR range and then click on Create subnet.

It's time to create route table.

Provide a name for your route table, select your VPC and click on Create route table.

In the public subnet's route table, we need a route to the internet, so let's add the corresponding entry.

Click on Save changes.

Now, let's associate this route table to our subnet.

Click on Edit subnet associations.

Click on Save associations.

Now that we have set up the VPC, subnet, and IGW, it's time to create an instance which will act as our Customer Gateway.

Let's select the same AMI to ensure consistency.

Create key pair for your instance.

Edit Network Settings. Choose your VPC and select your public subnet. Make sure to Enable Auto-assign public IP.

In the security group, since we'll be logging into this EC2 instance from our machine, we need to set up an SSH connection, allowing traffic from our IP address. Additionally, we'll allow all TCP traffic from the other side of the network and enable the ping protocol.

With all other configurations unchanged, click on Launch Instance.

Task 3: Set up VPN Connection

We've set up two VPCs, but they currently have no communication between them. The next step is to create and configure a Virtual Private Gateway (VGW) for the VPN. To do this, we need the public IP address of our EC2 instance acting as a router, as this is required to specify the location of the customer router while creating the VPN connection.

Copy the public IP address and switch to the AWS network side.

Go to the VPC dashboard.

Scroll down to the Virtual private network.

Click on Create virutal private gateway.

Provide a name and click on Create virtual private gateway.

Attach to your VPC.

Now, it's time to create Customer Gateway.

Click on Create customer gateway.

The Customer Gateway (CGW) represents the on-premises side of the network and is not part of the AWS network. It is where we will configure the IP address associated with the on-premises network.

After configuring the details, click on Create customer gateway.

Now that we have both the VPGW and CGW, we can use them to create a site-to-site VPN connection.

On Site-to-Site VPN connections, click on Create VPN connection.

When setting up a VPN connection, we can use either static routing or dynamic routing to manage the paths data takes between networks.

1. Static Routing

   • Routes are manually configured by specifying the exact paths for traffic.

   • It's simple to implement and is typically used for small or straightforward networks.

   • Requires manual updates if network topology changes.

2. Dynamic Routing

   • Routes are automatically determined and adjusted by routing protocols like BGP (Border Gateway Protocol).

   • It's more complex but ideal for large or changing networks.

   • Automatically adapts to network changes, providing better scalability and resilience.

Static IP Prefixes: It allows the Virtual Private Gateway (VGW) to recognize which IP prefixes belong to the on-premises network, ensuring that traffic destined for those addresses is correctly routed.

For the local IPv4, provide your on-premises CIDR value. For the remote IPv4, provide the AWS-side network CIDR.

For the Tunnel options, we can either keep the default settings or configure specific options, such as the pre-shared key. AWS VPN creates two tunnels for high availability, and the connection uses the secure IPsec protocol. This process involves sharing a key between the two endpoints. We can set up our own pre-shared key, or if we choose not to, AWS will generate one for us.

We'll let AWS handle the configuration, click Create VPN Connection.

While this is pending, let's set up our Openswan server on the on-premises instance.

Select your EC2 and click on Connect.

We're establishing an SSH connection to our instance.

Switch to the root user to ensure you have full permissions to install Openswan.

#switching to root user
sudo su

#installing openswan
yum install openswan -y

Now, let's configure Openswan.

In the ipsec.conf file, ensure there is an uncommented line at the end, allowing us to add additional files to this directory.

The next step is to update the /etc/sysctl.conf file. This is necessary because this machine will act as a router, forwarding the traffic it receives to another machine within the same on-premises network. We need to configure specific settings by adding a couple of lines to this file.

vim /etc/sysctl.conf

Press i to enter into insert mode.

# Enable IP forwarding for routing
net.ipv4.ip_forward = 1

# Disable acceptance of ICMP redirects
net.ipv4.conf.all.accept_redirects = 0

# Disable sending of ICMP redirects
net.ipv4.conf.all.send_redirects = 0

After making the changes, press Esc to exit insert mode, then type :wq! to save and exit the file.

Now, we need to restart the network service.

service network restart

Next, we need to configure our IPsec VPN. So far, we've only installed the software but haven't configured it yet. We'll need to specify details such as the AWS side of the network, the pre-shared key, and other settings.

To obtain these settings, we can download a text file from AWS that contains all the necessary configuration details, which we can then use to configure the IPsec VPN.

Select your VPN connection and click Download configuration.

If you open the file in a text editor, you'll find that it covers most of the configurations we've already completed. More importantly, it contains tunnel details.

Now, we need to create the file /etc/ipsec.d/aws.conf in our instance.

vim /etc/ipsec.d/aws.conf

Simply copy the entire block of tunnel details from the downloaded file and paste it into the /etc/ipsec.d/aws.conf file.

The default settings may cause issues with the VPN connection, particularly the line auth=esp, which is not supported. Be sure to delete this line from the file and then save your changes.

Similarly, we need to create the file /etc/ipsec.secrets. This file should contain the pre-shared key, as we mentioned earlier. Create the file and add the pre-shared key entry to it.

vim /etc/ipsec.d/aws.secrets

In the downloaded configuration file, look for your pre-shared key. Copy & paste it into the created file.

That's it. The /etc/ipsec.conf file will contain information for two tunnels because AWS provides two tunnels for high availability. While OpenSwan supports multiple VPN tunnels on a single host, careful configuration is essential to prevent conflicts. For this initial setup, we'll establish a single VPN tunnel to ensure a stable foundation.

The leftid field represents the IP address of the customer-side network, which in this case comes from the EC2 instance running OpenSwan. The rightid field represents the IP address at the other end of the connection, which is the AWS side.

vim /etc/ipsec.d/aws.conf

Next, we need to configure the leftsubnet and rightsubnet values in the file. The leftsubnet represents the CIDR block of the corporate network side, while the rightsubnet represents the CIDR block of the AWS side. Update these values in your configuration file to reflect the correct CIDR blocks for each side of the network.

Next, we will configure the route table for the private subnet by adding a route that directs traffic destined for the on-premises CIDR block to the VGW.

For this, go to the AWS side route table. Instead of manually adding a route to the VGW, we will use route propagation. This allows the VGW to automatically update the route table with the CIDR blocks from the AWS side.

Click on Edit route propagation.

Enable route propagation and save the changes. Once the VPN tunnel is active, the route table will update automatically.

Ok so, we have completed the configuration of the IPsec configuration files and the secrets file. Now, it's time to start the VPN tunnel.

Go back to your terminal and run the command to initiate the VPN tunnel.

systemctl start ipsec

Check the status of the IPsec service to confirm it's running.

systemctl status ipsec

To verify that the tunnel is functioning properly, it's important to remember that the VPN connection is always initiated from the customer side, not from AWS. If there is no traffic flowing from the customer side, the VPN tunnel may go down. To keep the tunnel active, ensure that you are sending keepalive packets or some form of traffic to maintain the connection.

Now, let's return to the AWS Management Console and check the VPN connection status. Navigate to the Site-to-Site VPN connections section. You should see that the tunnel status is up, indicating that the VPN connection has been successfully established.

With the tunnel status showing as 'Up', the final test is to verify connectivity between the machines. From our current machine, let's attempt to ping the private IP address of the EC2 instance in the AWS network.

Copy the private IP of the EC2 instance and use the ping command to test connectivity.

If you see traffic flowing and responses coming through, it confirms that the VPN tunnel between the AWS network and the OpenSwan router has been successfully established.

Task 4: Set up Extended infrastructure

Alright, now, let's proceed to set up our extended infrastructure.

Switch over to your on-premise region.

Let's create a private subnet.

Provide a name for your subnet, CIDR value then click on Create subnet.

After creating our private subnet, let's proceed to create route table.

Provide name for your route table, select your VPC and click on Create route table.

For now, we won't add any routes to this table. We'll specify the routes later. Just ensure that you associate this route table with the private subnet you created.

Now, let's create an EC2 instance.

Create a new Key pair or use existing one for your instance.

Edit Network settings. Select your VPC and choose your subnet.

Leave the Auto-assign public IP as default.

The only way to reach this server is through the Openswan server. We will log in to the Openswan server via SSH, and from there, we will access the server in the AWS network.

Also, configure the ping protocol to allow ICMP traffic.

Keeping all other configurations as default, click on Launch instance.

In the diagram, the OpenSwan server is configured as a router and must forward traffic that is not destined for itself. When we configure an instance to act as a router, like the OpenSwan server, we need to disable the source/destination check.

Disabling source/destination check allows the instance to forward traffic that is not necessarily intended for itself, enabling it to route traffic between other instances.

Since this EC2 is acting as a router, it needs to handle all traffic flowing between instances. Therefore, it must accept traffic from the IP addresses of other instances.

To configure this, we need to update the security group associated with the Openswan server.

Click your instance and head to the Security tab & click on your Security group.

Edit the inbound rules of the security group for the router instance.

Currently, there are rules for traffic from the AWS side of the network.

We need to add TCP and ICMP inbound rules to allow connections from our on-premise network and enable ping functionality for network diagnostics.

Click on Save rules.

Let's verify the connectivity by attempting to ping the target instance.

Ok, so we are able to ping our private instance.

Next, we need to SSH into our private EC2 instance.

On the router instance (Openswan), create a file named as your private EC2 key that you downloaded earlier. e.g., your-key.pem.

Locate your key file in your device and open it using text editor.

Copy the content and paste it into the created pem file in your terminal.

Save and exit.

Run the command to ensure your key is not publicly viewable.

We have logged in to our private instance via SSH.

Now we've completed the configuration, but we still need to set up a route for the subnet. We need to specify that traffic from the private subnet should be routed through the EC2 instance acting as a NAT.

To do this, go to the route table for the private subnet.

Edit the routes and add a new route indicating that traffic destined for 10.0.0.0/16 should be routed through the NAT instance.

Save the updated routes.

Now, finally we have made the connection. Let's ping the AWS-side EC2 instance from on-premise EC2 instance.

We can see the traffic has started flowing as expected.

Conclusion

We established a VPN connection between our on-premises network and AWS. Our on-premises environment includes a VPC with public and private subnets. An EC2 instance running OpenSwan is located in the public subnet, acting as the Customer Gateway (CGW). The AWS environment consists of a VPC with a private subnet and a private EC2 instance. We successfully configured the VPN tunnel and verified connectivity by pinging the AWS EC2 instance from the on-premises private network.

The successful ping test confirms that the VPN tunnel is operational and traffic can be routed between the on-premises and AWS networks.

Task 5: Clean Up

Let's clean up from on-premise side network.

1. Terminating instances.

   

2. Deleting VPC.

   

   Deleting a VPC will remove all associated resources.

Now, let's clean up resources from our AWS side.

3. Terminating instance.

   

Go to VPC dashboard.

4. Deleting VPN connection.

   

   Select & delete your Customer gateway.

   

   Detach your Virtual private gateway from your VPC.

   

   Select your Virtual private gateway & click Delete Virtual private gateway.

   

   Now, delete your VPN connection.

5. Deleting VPC.

   

   Select your VPC and click Delete VPC.

Now, we have cleaned up all the resources we used.

Alright, I hope that was fruitful blog.

See you in the next one!

In this blog, we’ll explore Amazon Cognito, a powerful AWS service for managing user authentication and access control in applications. We’ll also demonstrate how to use Amazon Cognito to secure APIs, ensuring that only authorized users can access our resources.

In this demo, we will secure an API deployed through API Gateway. By default, API Gateway does not provide authorization or security, which means anyone can access our endpoint.

We’ll use a Cognito Identity Pool to create users and configure an API that triggers a Lambda function. After successful sign-in, users will receive a token. API Gateway will validate this token to grant access to the request.

Additionally, we’ll use Postman to test our API. Postman allows us to send requests, configure details like headers, and view responses. It helps ensure that our API is properly secured and functioning as expected.

Task 1: Creating Lambda function

First, let's set up our backend server, lambda.

Search for lambda in the search box.

Click on Create a function.

Provide name for your function and select Python as runtime.

Keeping all other configurations as is, click on Create function.

We are just returning a simple hello message from lambda.

Click on Deploy.

Now, click on Test. To test our function, we have to provide a test event, for now, just provide the name for test event and click on Save.

Now, click on Test to see if the function is working as expected. We are getting the hello message.

Alright, that's the first job done.

Task 2: Creating an API

We will be creating a simple REST GET API.

Search for API Gateway in the search box.

Scroll down to REST API.

Click on Build.

Provide a name for your API and click on Create API.

Now we are at the root (/) level.

Let's create a method. Click on Create Method.

Configure Method type as GET, Integration type as Lambda function and select your created lambda function.

Click on Create method.

Click on Deploy API.

Choose New stage and provide a suitable Stage name.

Select the GET method and copy the invoke URL from your stage section.

Let's head over to the Postman.

Select GET method in Postman, paste the copied URL and click on Send.

We get the response from our lambda function.

"Hello from Lambda"

Task 3: Creating User Pool

Now, let's create a User Pool.

A User Pool is a user directory in Amazon Cognito that helps manage and authenticate users. When users sign up or sign in to your application, they interact with the user pool.

Search for Amazon Cognito in the search box.

Click on Create user pool.

For sign-in options, select Email, then click on Next.

For password policy, we can go with defaults, but I made some changes to make easier requirements for password in the Custom mode.

We decided to go for No MFA this time.

Leave the Self-service account recovery option as default.

This option configures how users will recover their account when they forget their password.

Click on Next.

Keep the Self-registration option as is. This allows new users to register for an account themselves.

Scroll down to bottom, leaving all other configurations as default.

We can choose the attributes that are required when a new user is created.

I added name as additional required attributes. You can select other attributes to add as you like.

Click on Next.

Select the Send email with Cognito option. The FROM email address shows who sent the email, while the REPLY-TO address indicates where replies will be sent, which can be different from the FROM address.

Click on Next.

Provide a suitable name for your User pool.

Tick the Use the Cognito Hosted UI box. This allows us to use Amazon Cognito's built-in user interface for authentication and sign-up, simplifying the process of integrating user sign-in and sign-up into your application.

Select the Cognito domain. We can use our custom domain, which involves configuring a custom domain name for the Cognito Hosted UI.

Essentially, this is the domain through which users will access the authentication interface for signing in or signing up.

Provide a suitable App client name. An initial app client in Amazon Cognito is essentially an identity for your application to interact with a user pool.

The client secret helps ensure that the client making requests to the Cognito service is indeed the one that was registered.

We are not generating for now.

We are using example.com as our callback URL for demo purpose.

Callback URLs are the addresses where users are sent after they log in or complete an authentication process. They direct users back to your app with the authentication details.

In the Advanced client app settings, we can customize the values as required.

Scroll down to the OAuth 2.0 grant types section.

OAuth 2.0 grant types are methods for obtaining access tokens to access protected resources.

Let's explore the workflows briefly.

Authorization Code Grant

• The user logs in via the chosen authorization server (e.g., Amazon Cognito, Google Identity, or another OAuth 2.0 provider).

• Authorization Server redirects back with an authorization code.

• Your backend exchanges the code for tokens.

• Use the tokens to access protected resources

Implicit Grant

1. User logs in via the chosen authorization server.

2. Authorization server redirects back with tokens in the URL.

3. Use the tokens directly to access resources.

We are using Implicit Grant to obtain an access token directly from the authorization server.

Keeping all other configurations as default, click on Next.

Review your selections and click on Create user pool.

Our User pool has been created. Click on your created user pool.

Let's create a user in the Users section.

Provide email address and set password, then click on Create user.

A user has been created.

Task 4: Creating & testing API Gateway Authorizer

It's time to add Authorizers in API Gateway that will authenticate and authorize requests to our API. This ensures that only users with valid credentials can access our API resources.

Let's go to the API Gateway.

In the left panel, in Authorizers, click on Create an authorizer.

Provide a suitable name, select Cognito as Authorizer type, select your user pool and provide the header as Authorization in the Token source.

Click on Create authorizer.

Task 5: Signing in and Testing API Gateway Authorizer

To test the authorizer, we need a token value, which is obtained after a successful sign-in. Therefore, let's proceed with the sign-in process.

To view the sign-in page, we have two approaches: a direct link and a customized URL method. The direct link is simpler for basic use cases, while the customized URL method offers flexibility and customization for more complex scenarios and specific needs.

Click to get the URL

Let's implement the customized URL approach first.

There are two different URLs given, one for authorization code grant and another for implicit code grant.

We are using the second URL.

https://your_domain/login?response_type=token&client_id=your_app_client_id&redirect_uri=your_callback_url

We need to replace your_domain, your_app_client_id & your_callback_url fields with the original values.

In the App integration tab, you'll find your domain link in the Domain section.

Copy and replace it with your_domain in the URL.

In the same section, scroll to the bottom, and we'll find our App client list. Click on your App client.

Copy your Client ID and replace with your_app_client_id in the URL.

Scroll down a bit, and you'll find your allowed callback URLs that we had provided earlier.

Copy that and paste it into the above URL in your_callback_URL.

#My final URL
https://demodomain1.auth.ap-south-1.amazoncognito.com/login?response_type=token&client_id=6pv54rin409v9ngddqb932lade&redirect_uri=https://example.com

Copy and paste the complete URL into a new browser tab to access the sign-in page.

Alternatively, we can use a direct link for a more straightforward approach.

In your App client, in Hosted UI section, in the top right corner, we can see View Hosted UI.

Click on that button.

Sign in with your configured email & password.

We can also create new users using the Sign up link, as we have enabled Self-service sign-up option.

Since this is my first time signing in, I need to set up a new password. Additionally, because I set the name as a required attribute, it also appears on this page.

Click on Send.

Our sign-in was a success, and we're directed to the callback URL that we had set up.

Notice in the URL, there's a token provided by Cognito. We'll copy this token to test our Authorizer.

Copy the token and paste it into your text editor.

Only copy the #id_token value from the URL.

Paste the copied token in the Token value section and click on Test authorizer.

We got this output, and this is what's known as the decoded JWT claims, or just JWT claims. It's the data returned by the API Gateway authorizer during testing, representing various details about the user contained within the token used for authorization.

We can pass an invalid token value to observe the response we receive.

Task 6: Adding Authorizer to GET Method

Now in our GET method, we have to add the authorization in the method request.

Click on Method request & scroll a bit to see the body section.

Click on Edit.

Select your authorizer and click on Save.

After making the change, redeploy the API by clicking on Deploy API.

After deploying the API, copy the URL and open Postman to test the GET request.

Paste the URL then click on Send.

This time, we see an “Unauthorized” error.

In API Gateway, the authorizer is configured to expect the token in a header field called Authorization.

Let's copy our token again.

In the Headers section, add Authorization as the key and paste the token as the value. Then, click on Send.

Now that we have provided the token, the authorizer has validated it and granted access to the backend.

Note that if your token is not working, it may have expired. In this case, you will need to sign in again to obtain a new token.

Task 7: Conclusion

In this blog, we secured an API with Amazon Cognito and API Gateway. We set up a Lambda function, deployed a REST API, configured a Cognito User Pool, and used API Gateway Authorizers. Finally, we tested the setup with Postman.

Task 8: Clean Up

1. Deleting API.

   

   Select your API and click on Delete to delete your API.

   

2. Deleting Lambda function.

   

   Select your lambda function and in the Actions drop-down menu, click on Delete.

3. Deleting User Pool.

   

   Select your User pool and click on Delete.

   

Alright, that sums up our today's work.

See you later in the next one!

Overview

We will use an S3 bucket to store our website files, CloudFront for content distribution & issue a certificate from AWS ACM for security. Finally, we will make updates in DNS records to access our static site with custom domain name.

Prerequisites

To complete this tutorial, we need a registered domain name. Amazon Route 53 (R53) provides domain registration and DNS management services, but we can choose separate providers like GoDaddy, Namecheap, or Google Domains for these services if desired.

I am expecting you already have your custom domain name before proceeding.

Task 1: Creating an S3 bucket

Search for S3 in the search box.

Click on Create bucket.

Your bucket name should match the domain name exactly. For example, I'll be using "sujanmagar.com.np" so my bucket name needs to be "sujanmagar.com.np".

Scroll down and deselect the setting for "Block all public access". This is usually not recommended, but since we're creating a static website hosting, it required to disable this setting.

Tick on the acknowledgment box.

Use the defaults for the rest of the bucket settings and then click on Create bucket.

Click on your bucket and on Properties tab, scroll down to Static website hosting section.

Click on Edit.

Enable the Static website hosting and input index.html in the Index document field. This specifies the default home page for our site.

Leaving all other configurations as default, click on Save changes.

When we created the bucket, we said we didn't want to block all public access. But even with that setting, the default behavior of S3 is to "deny" everything. So if we don't explicitly say that people can access the files in our bucket, they won't be able to so, we'll grant read permissions with a bucket policy.

On Permissions tab, on Bucket policy section, click on Edit.

Paste the following bucket policy in the Policy section.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::Bucket-Name/*"#make sure to add your bucket name
            ]
        }
    ]
}

Make sure to update the Bucket-Name with your own.

Click on Save changes.

Now, it's time to upload our files.

On Objects tab, click on Upload.

Click on Add files.

Select your files and click on Upload. You can create your static files that will make up your static website, or you can use mine.

My files for static website

Now it's time to test that our site loads. To do this, we'll need to get the S3 bucket website endpoint.

Head to the Properties tab of the bucket & scroll down to the Static website hosting section.

Click on the Bucket website endpoint link.

If everything worked, you should be able to load your static website.

We have hosted our site in S3 with public access. But as impressive as it is, it would be even more impressive if it were using a custom domain. Currently, it is using the S3 website URL, formatted as [bucketname].s3website[regionname].amazonaws.com.

Task 2: Creating a record to point to the S3 website

To create a DNS record, we need to be working within our DNS zone, which in my case is managed by Cloudflare. You might have your zone hosted with other providers like Amazon Route 53 (R53) or other DNS management services.

Click on Add record.

Copy the Bucket website endpoint and remove the protocol from the endpoint, & paste it in the Target field.

In Amazon Route 53, we have the alias option to point our domain to AWS resources. However, since I'm using Cloudflare, which doesn't have an alias option, I am using a CNAME record type to achieve the similar functionality. Cloudflare automatically enables CNAME flattening for the root domain, allowing it to point to another domain name. This is typically not allowed by DNS standards, but Cloudflare makes it possible.

Click on Save.

Now let's test! If everything worked, then when you type your domain name into a browser (like sujanmagar.com.np), DNS provider should direct you to the S3 website.

Alright, the last step is to set up a secure connection (HTTPS with a TLS/SSL certificate) to eliminate the "Not secure" message from our browser.

Task 3: Creating a Public TLS/SSL Certificate

In AWS, certificates are handled through AWS Certificate Manager (ACM), where you can create and manage certificates. ACM also allows you to import certificates from another authority if you already have them.

In your AWS console, search for Certificate Manager in the search box.

For this section, you need to switch your region to us-east-1 (N. Virginia).

For now, we choose the DNS validation method.

The request was successful, but it will have a "Pending validation" status until we validate DNS.

Click on View certificate.

While DNS validation is straightforward with Route 53, using Cloudflare as my DNS provider requires some manual steps to validate domain ownership when issuing an ACM certificate. With Cloudflare, I need to add a specific DNS record provided by ACM to my DNS settings to complete the validation process.

Copy the CNAME name and value, then add a new CNAME record in DNS provider's DNS records.

Click on Save.

We have a TLS/SSL certificate now and our website files are hosted in S3, but we can't use a certificate on an S3 bucket.

What we need instead is a CloudFront distribution that points to the S3 bucket. And then the certificate is applied to the CloudFront distribution.

Task 4: Create a CloudFront Distribution

Amazon's content delivery network (CDN), CloudFront, speeds up content delivery by caching it at various “Edge locations” globally. This setup is particularly effective for accelerating the load times of videos and images.

For a simple site with few files like ours, the performance boost might not be very noticeable. However, using CloudFront is essential for applying the TLS/SSL certificate that we created in the previous task.

Search for CloudFront in the search box.

Click on Create a CloudFront distribution.

The origin domain is where our website files are stored, which is in S3. If you filter by typing in S3, it will show our bucket.

However, we'll get a message about using the website endpoint instead of the bucket endpoint. Click on Use website endpoint and AWS will update the endpoint for us.

Scroll down to the Default cache behavior section, then under Viewer, select Redirect HTTP to HTTPS. This will help ensure that all traffic to our website is encrypted, enhancing security and protecting user data.

Scroll down to Web Application Firewall (WAF) and select Do not enable security protections.

In the next section, Settings, click on Add item.

Provide your domain name there and select your certificate in Custom SSL certificate.

Scroll to the bottom of the page.

For Default root object, enter index.html (our default home page) and then click on Create distribution.

It will take several minutes for the CloudFront distribution to finish deploying.

To test that everything is working with CloudFront and the TLS/SSL certificate, copy the Distribution domain name.

Open a new tab in the browser and navigate to that address.

Now, the last thing is to use our custom domain name to load the site instead of this lengthy CloudFront distribution domain name.

Task 5: Updating DNS Record

At the moment, the CNAME Record in Cloudflare is pointing to the S3 bucket.

Instead, we want Cloudflare to point to the CloudFront distribution, which then points to S3.

Navigate back to your DNS Records settings.

Instead of routing traffic to S3, update the DNS record to point to your CloudFront distribution.

Click on Save.

Task 6: The Final Show

Alright! Type your custom domain name in the new tab in your browser and hit the ENTER button.

If everything worked, we should be able to load our static website without our own custom domain name on a secure connection.

Aha! It worked.

The static files (coming from S3 via CloudFront) load on the custom domain name (from Cloudflare) over a secure connection using the TLS/SSL certificate (from Certificate Manager).

Conclusion

We utilized an S3 bucket for storing our website files & leveraged CloudFront for efficient content distribution. We secured our site with a certificate from AWS ACM, and updated DNS records to enable access through our custom domain, making a scalable, secure, and performant static website on AWS.

Here's what we've built:

Task 7: Clean Up

1. Deleting CloudFront distribution.

   

   Before deleting our Distribution, we have to Disable it.

   

   It'll take a couple of minutes even after being disabled to be able to finally delete.

   

2. Deleting records from DNS Records.

   

   Navigate to your DNS Records settings and delete the created records.

   

3. Delete the certificate from Certificate Manager.

   

   Navigate to Certificate Manager. Select your certificate and then click on Delete.

4. Deleting S3 bucket.

   

   Navigate to your bucket and then empty your bucket, deleting all the files contained.

   

   After you've deleted all the contents, select your bucket and click on Delete.

Now we have deleted all the resources that we used during the lab session.

Alright, that was it for this project.

See you in the next one!

VPC Endpoints

VPC endpoints are a way to privately connect your VPC to supported AWS services and VPC endpoint services without requiring an internet gateway, NAT device, VPN, or Direct Connect connection. They allow you to connect to these services using private IP addresses, which keeps your traffic within the AWS network.

There are two types of VPC endpoints:

1. Interface Endpoints: These are powered by AWS PrivateLink and are used for connecting to services like S3, DynamoDB. They provide a private IP address in your VPC. When you create an interface endpoint, an Elastic Network Interface (ENI) is created in your subnet with a private IP address that serves as the entry point for traffic destined to the service.

2. Gateway Endpoints: These are used for connecting to S3 and DynamoDB and are simpler to set up than interface endpoints. Gateway endpoints are associated with your route table and use route tables to direct traffic to the endpoint.

Overview

We will set up a VPC with two subnets: one public and one private. Each subnet will have one instance. We will attach a role to the private instance to allow it to list the available S3 buckets. This will be done utilizing the Gateway Endpoint which establishes a private connection within our VPC to access S3, eliminating the need for an internet gateway.

Task 1: Creating a VPC

Search for VPC in the search box.

Click on Create VPC.

Provide a name for your VPC and input a CIDR range.

Click on Create VPC.

Task 2: Creating an Internet Gateway

On Internet gateways, click on Create internet gateway.

Provide a suitable name and click on Create internet gateway.

After creating an Internet gateway, attach it to your VPC.

Task 3: Creating Subnets

On Subnets, click on Create subnet.

We are going to create two subnets: one public and one private.

Select your VPC.

Provide a suitable name for your subnet, choose an AZ and input an eligible CIDR range value.

Click on Add new subnet.

Provide a suitable name and CIDR range.

Click on Create subnet.

Task 4: Creating Route table

On Route tables, click on Create route table.

Provide a suitable name for your route table, select your VPC and click on Create route table.

On Subnet associations, click on Edit subnet associations.

Tick your public subnet and click on Save associations.

Now on Routes, click on Edit routes.

Configure a route to Internet gateway and click on Save changes.

Now it's time to create another route table & associate private subnet.

We should not include an Internet Gateway route in the private route table.

We have created two route tables.

Task 5: Creating a Role

Search for IAM in the search box.

On Roles, click on Create role.

Select AWS Service as the trusted entity type and EC2 for the use case.

By choosing AWS Service as the trusted entity type, we specify that this role is intended for use by an AWS service, in this case, EC2. This allows EC2 instances to assume this role and inherit the permissions defined in the role's policy.

Click on Next.

We are attaching S3 read only permission to our instance.

Click on '+' sign to expand the policy.

Click on Next.

Provide a suitable name for your role and review your created role.

Click on Create role.

We have created a role.

Task 6: Creating EC2 instance.

Search for EC2 in the search box.

On Instances, click on Launch instances.

Provide a name for your instance, select an Amazon Linux AMI (mandatory) and a free-tier instance if possible. Either use existing or create a new key pair.

Edit Network settings. Select your VPC and your subnet.

Leaving all other configurations as default, click on Launch instance.

Now, let's create a private instance.

Provide a name for your instance, select an Amazon Linux AMI and a free-tier instance if possible. Either use existing or create a new key pair.

Select your VPC, choose private subnet and keep other as default.

Leaving all other configurations as default, scroll to the bottom Advanced Details.

On IAM instance profile, select your created role.

Now, keeping all other settings as default, click on Launch instance.

We have created two EC2 instances.

diagram..

Task 7: Creating s3 buckets.

Search for s3 in the search box.

On Buckets, click on Create bucket.

Just provide a name for your bucket and leaving all other configurations as it is, click on Create bucket.

Let's create another bucket as well.

So, we have created two buckets.

Task 8: Accessing EC2s

We need to copy the private key required to SSH into the private server from our public server first.

Open your local terminal.

Modify the code below to suit your needs.

scp -i "publicServerKey.pem" "privateServerKey.pem" ec2-user@13.233.225.53:/home/ec2-user

This command copies the file privateServerKey.pem from our local machine to the /home/ec2-user directory on the remote server with IP address 13.233.225.53 using the SSH key specified by publicServerKey.pem.

Now, let's SSH into our public instance.

We have securely copied the key to our public instance, which will be used to SSH into our private instance.

Copy & paste the command in your public instance's terminal.

Now, copy the SSH command to establish a connection to the private instance.

Proceed with yes.

We are now connected to our private instance.

Even though we have attached the role that allows an EC2 instance to list S3 buckets, we are unable to do so due to a lack of connection. This is where we set up a VPC endpoint.

Task 9: Creating a VPC endpoint.

Go to the VPC dashboard.

On Endpoints, click on Create endpoint.

Provide a name for your endpoint.

In the services, look for S3 and select it. This VPC endpoint will be used to access S3, indicating that the services it will be consuming or using are related to S3.

Choose the service name com.amazonaws.<your-region>.s3 to create the VPC endpoint for standard Amazon S3 in the <your-region>.

Select the Gateway type & select your VPC.

Select only your private route table, as we're allowing EC2 instance in the private subnet to perform actions on S3 buckets using the endpoint.

Click on Create endpoint.

Now that we've created our VPC Endpoint, let's return to our EC2 terminal and run the same command to see if it's working.

We can now list the S3 buckets, demonstrating that a private instance within our VPC can securely communicate with other AWS services.

Conclusion

By attaching a role to the private instance and utilizing the Gateway Endpoint, we have enabled it to perform operations on S3 buckets securely. This setup establishes a private connection from our VPC to S3, allowing communication to occur without the need for an internet gateway.

Task 10: Clean Up

1. Terminating EC2 instances

   

   Select both of your instances and on Instance state, click on Terminate instance.

2. Deleting VPC Endpoint

   

   Select your Endpoint and in the Actions panel, click on Delete VPC endpoints.

3. Deleting subnets.

   

   Select both of your subnets and in the Actions panel, click on Delete subnet. Refresh the page if you are not able to delete.

4. Deleting VPC.

   

   Select your VPC and in the Actions panel, click on Delete VPC.

5. Deleting s3 buckets.

   

   

   Select your bucket and click on Delete button.

We have cleaned up our used resources.

That was it for today.

I'll see you in the next one!

Today we will be setting up our AWS CLI.

AWS CLI is a tool that lets us manage AWS services from the command line. With AWS CLI, we can control various AWS services and automate tasks through scripts.

To use AWS CLI, we need to configure it with access keys that grant the necessary permissions to perform actions on AWS resources. These access keys are part of AWS Identity and Access Management (IAM).

IAM Access Keys

IAM (Identity and Access Management) access keys are a pair of security credentials that AWS uses to authenticate and authorize API calls. These keys are associated with an IAM user and consist of two parts:

1. Access Key ID: A unique identifier for the access key.

2. Secret Access Key: A secret key used to sign requests.

Purpose

IAM access keys allow programmatic access to AWS services via the AWS CLI (Command Line Interface), AWS SDKs (Software Development Kits), and other tools that make API requests to AWS services.

When creating access keys, it is essential to use the IAMAdmin role or another appropriate IAM role. Creating access keys for the root account is not recommended due to security concerns.

Let's create our Access keys.

Click on your profile and select Security credentials.

Scroll a bit and you will find the Access keys section.

Click on Create access key.

Choose CLI for Use case, tick the confirmation box and click on Next.

Provide a description tag for your Access key.

Click on Create access key.

Our access key has been created.

We can see that this is the only time we can view or download our secret key. However, if we lose it, we can create a new one.

Click on Download .csv file and store it securely in your local machine.

Click on Done.

We have created our access keys. We can create up to two access keys in total. Access keys can be created, deleted, activated, and deactivated.

Downloading AWS CLI

Now that we have created & downloaded our access keys, it's time to download the AWS CLI on our local machine.

AWS CLI

Click the link above to download AWS CLI.

Choose according to your operating system.

Configure all the setup, and after your setup is successful, type the following command.

aws --version

The command displays the installed version of the AWS CLI on your machine.

Configuration

Type the following command to configure your AWS CLI.

aws configure

The command is used to configure your credentials for the AWS CLI. It will ask for your access key ID, secret access key, default region name (use the closest region code, e.g., us-east-1) & skip the default output format field associated with your IAM user.

After you have configured your credentials, input the following command.

aws configure list

The command displays information about the profiles you have configured for interacting with AWS services.

You must see your configured profile.

Simple Commands

Let's use simple AWS CLI commands.

aws s3 ls #to list the S3 buckets.

The command is used to list the S3 buckets.

As I do not have any buckets created, it is showing empty.

aws iam list-users #to list all IAM users

We used simple AWS CLI commands. Now that our AWS CLI is set up, we can do many AWS operations from the command line but more on that later.

Thank you.

Today we will be exploring Transit Gateway.

A Transit Gateway is a network transit hub that enables you to interconnect your VPCs, VPN connections, and Direct Connect gateways through a central point, simplifying your network architecture.

Let's get started.

Overview

We will be creating three VPCs, each with a subnet and an instance within that subnet. We'll set up a Transit Gateway to connect these three VPCs. Once all three VPCs are connected, we will SSH into our EC2 instances and perform some task to verify if they can communicate with instances in the other VPCs.

Task 1: Creating a VPC.

Search for VPC in the search box and on VPC dashboard, click on Create VPC.

Provide a suitable name for your VPC and input a CIDR range value.

Keeping all other configurations as default, click on Create VPC.

We have created our first VPC.

Now, let's repeat the process two more times. Ensure the CIDR range for your other VPC does not overlap.

We have created three VPCs. Let's move on to the next task.

Task 2: Creating an Internet gateway

On Internet gateways, click on Create Internet gateway.

Provide a suitable name for your Internet Gateway and click on Create Internet gateway.

Now, we should attach the Internet gateway to our VPC.

Select your first VPC and click on Create internet gateway.

Now, repeat the same process for the remaining two VPCs.

Alright, we have created three internet gateways and attached them to the VPCs.

Task 3: Creating Subnets

On Subnets, click on Create subnet.

Select your first VPC, pick an AZ and provide an eligible CIDR range.

Click on Create subnet.

We have created the first subnet. Now, let's repeat the process for the other two subnets.

Ensure that you provide eligible CIDR ranges for your subnets.

We have created three subnets, each residing in a different VPC.

Task 4: Creating Route Table

On Route table, click on Create route table.

Provide a suitable name for your route table, select your VPC and click on Create route table.

After creating route table, on Subnet associations, click on Edit subnet associations.

Tick your subnet and click on Save associations.

Now, on Routes, click on Edit routes.

Click on Add route, configure the above configurations, and click on Save changes.

Now, we have associated the subnets and set up the necessary routes for the current VPC. Let's repeat the process for our other VPCs.

We have created our route tables.

Task 5: Creating EC2 Instances

Search for EC2 in the search box.

On Instances, click on Launch Instances.

Provide a suitable name for your instance, choose an AMI, either create or use an existing key pair.

Edit Network settings, select your VPC, and Enable the Auto-assign public IP option.

We are provided with default SSH configuration. Now, click on Add security group rule to add a rule for HTTP requests.

Configure the above configurations in your new security group rule.

Scroll to bottom & click on Advanced details.

On User data, paste the following command.

#!/bin/bash

# Update and install apache2
apt-get update
apt-get install -y apache2

# Create a simple index.html file with "Hello" text
echo "Hello from the FIRST" > /var/www/html/index.html

# Restart apache2 service
systemctl restart apache2

Click on Launch instance.

Now, it's time to set up the other two instances. Repeat the process while making necessary changes.

Setting up our second instance.

Configure security rule like we did earlier.

And paste the following in the user data for our second instance.

#!/bin/bash

# Update and install apache2
apt-get update
apt-get install -y apache2

# Create a simple index.html file with "Hello" text
echo "Hello from the SECOND" > /var/www/html/index.html

# Restart apache2 service
systemctl restart apache2

Click on Launch instance.

Now, time to set up our third instance.

Setting up our third instance. Make sure you have added the additional security group rule.

On User data section, paste the following.

#!/bin/bash

# Update and install apache2
apt-get update
apt-get install -y apache2

# Create a simple index.html file with "Hello" text
echo "Hello from the THIRD" > /var/www/html/index.html

# Restart apache2 service
systemctl restart apache2

Click on Launch instance.

We have launched three instances.

Task 6: Setting Transit Gateway

Search for Transit Gateway in the search box.

Provide a suitable name for your transit gateway.

We left these settings as default. The ASN will be provided by AWS. Now, let's discuss these options for a while.

Transit Gateway is a central hub for connecting and managing different networks. ASN is a unique identifier used to help the Transit Gateway route traffic between these networks using Border Gateway Protocol.

DNS Support means the ability to resolve domain names to IP addresses. When enabled, DNS support allows resources connected through the Transit Gateway to use DNS resolution for communication.

Equal-Cost Multi-Path (ECMP) Support for VPNs allows the use of multiple paths for data traffic. This improves load balancing and redundancy by distributing traffic across multiple VPN tunnels that have the same cost (or path length).

Default Route Table Association refers to the automatic association of new VPC attachments to the Transit Gateway's default route table. This means that any new VPCs connected to the Transit Gateway will automatically use the default route table for routing traffic.

Default Route Table Propagation means that routes learned from new VPC attachments are automatically added to the default route table of the Transit Gateway. This ensures that all connected networks are aware of each other without manual route updates.

Multicast Support allows the transmission of data from one sender to multiple receivers simultaneously. This is particularly useful for applications like video conferencing, where the same data needs to be sent to multiple users at the same time.

It automatically accepts requests from other AWS accounts to attach their VPCs or VPNs to your Transit Gateway.

With all other configurations left as default, click on Create transit gateway.

We can see the details of our transit gateway.

Now that we have created a transit gateway, it's time to create transit gateway attachments.

A Transit Gateway Attachment is needed to connect your VPCs, VPN connections, or Direct Connect gateways to the Transit Gateway. The attachments essentially establish the pathways through which data can travel between your networks using the Transit Gateway.

On Transit gateway attachments, click on Create transit gateway attachment. Since we have three VPC connections, we need to create a transit gateway attachment for each of them.

Provide a suitable name and select your Transit Gateway. Since our Attachment type is VPC, we can keep it as the default option.

Select your first VPC for the VPC ID. Keep all configurations as default, then click on Create transit gateway attachment.

Now, let's repeat the process to create the other two transit gateway attachments.

We have created three transit gateway attachments and associated them with their respective VPCs.

Now, it's time to update the route tables of all three VPCs. We need to add routes that direct traffic destined for the other VPCs to the Transit Gateway.

Let's start with the route table of the first VPC.

Select your first route table and on Routes, click on Edit routes.

Duplicate your current tab and navigate to the CIDR value of your second VPC.

Now, click on Add route, paste the copied CIDR into the Destination field, and select your transit gateway attachment in the Target field on Transit Gateway.

Click on Save changes.

Now, click on Add route again.

Copy the CIDR range of the third VPC and follow the same steps as before.

Click on Save changes.

Now, we need to repeat the process for our second and third VPCs.

We have completed all the necessary route setup for our second VPC.

Click on Save changes.

Now, let's proceed with our third VPC.

Okay, the necessary routes for the third VPC have been configured.

Click on Save changes.

Now that all setups are complete, it's time to test the connectivity between the VPCs to ensure everything is working as expected.

Task 7: SSH into EC2 instances.

Go to EC2 dashboard by searching for EC2 in the search box.

Select your first instance and click on Connect.

Now on SSH client, we have to follow given procedures.

Open terminal in your local machine.

Paste the copied command into your terminal to change the key permissions.

Now copy the command and paste it into your terminal to SSH to your instance.

Click on Enter, proceed with yes.

We're now in our first instance. Let's check if we can communicate with the other two instances that are in different VPCs.

Select your second instance & copy the Private IPv4 address.

curl <Private IPv4 address>

Paste the command into your terminal.

Alright, we've received a response from our second instance.

Let's check for a response from our third instance.

We've received a response from our third instance as well.

This confirmation was from our first instance in our first VPC. To check the connectivity from our other two instances in different VPCs, we need to SSH into them and follow the same process we used for the first instance.

Let's now check from our second instance in our second VPC.

Now it's time from our third instance.

Alright, we have tested connectivity from all of our instances, and they all have a connection.

Conclusion

We have successfully set up a transit gateway and connected three VPCs to it. We tested the communication between instances in these VPCs using a simple curl command, confirming that they can communicate with each other.

Task 8: Clean Up

1. Terminating Instances.

   

   Select all your instances and in the Instance state panel, click on Terminate instance.

2. Deleting Transit Gateway attachments.

   

   Go to VPC dashboard, scroll down to Transit gateways, in the Actions panel, click on Delete transit gateway attachment. Repeat this process for the remaining two attachments. Note that it will take some time for the deletion to complete.

3. Deleting Transit Gateway.

   

   Select your transit gateway and in the Actions panel, click on Delete transit gateway.

4. Deleting Subnets.

   

   On Subnets, select all your 3 subnets and in the Actions panel, click on Delete subnet.

5. Deleting VPCs.

   

   Select your first VPC and in the Actions panel, click on Delete VPC.

   Deleting a VPC will also remove all associated route tables and internet gateways. Repeat the process for the other two VPCs.

We have removed all the used resources now.

Alright, see you in the next one!

Today, we'll be exploring VPC peering in AWS. In Amazon Web Services (AWS), a Virtual Private Cloud (VPC) is a logically isolated network segment you create within the cloud. You can launch resources like EC2 instances into your VPC.

VPC peering is a networking connection established between two VPCs that enables you to route traffic between them using private IP addresses.

Why do we need VPC peering?

VPC peering is essential for enabling private communication between VPCs without relying on public IPs or the internet. It facilitates secure and efficient sharing of resources across different VPCs, whether they reside within the same account, different accounts, or even different regions.

Alright, time to put our knowledge to work!

Overview

We will set up two VPCs, each containing instances. We will SSH into our instances and initiate communication with instances in the other VPC to test if our VPC peering connection is working.

Task 1: Creating VPCs

Search for VPC in the search box.

On VPC dashboard, click on Create VPC.

Provide a name for your VPC and input a CIDR range value for your VPC.

Click on Create VPC.

Let's create another VPC.

Provide a suitable name and input a CIDR range value for your second VPC.

Click on Create VPC.

We have created two VPCs.

Task 2: Creating Internet gateway

On Internet gateways, click on Create internet gateway.

Provide a name for your Internet gateway and click on Create internet gateway.

After creating internet gateway, attach to your first VPC.

Similarly, create another internet gateway and attach to the second VPC.

Task 3: Creating Subnets

On Subnets, click on Create subnet.

Select your first VPC.

Provide a suitable name for your VPC, choose AZ and input a valid CIDR range for your subnet.

Click on Create subnet.

Let's create another subnet in our second VPC.

Provide the name for your subnet, select AZ, and input an eligible CIDR range.

Click on Create subnet.

We have created two subnets in different VPCs.

Task 4: Creating Route tables

On Route tables, click on Create route table.

Provide the name for your route table and select your first VPC.

Click on Create route table.

On Subnet associations, click on Edit subnet associations.

Select your subnet and click on Save associations.

On Routes, click on Edit routes.

Click on Add route to add the above configurations.

Click on Save changes.

Now, let's create another route table, associate the corresponding subnet to it & add the necessary route.

And now we have set up our second route table.

We have created two route tables for two VPCs.

Task 5: Creating EC2 instances

Go to EC2 dashboard, on Instances, click on Launch instances.

Provide name for your EC2, select an AMI. Create or use an existing key pair.

Edit Network settings. Select your first VPC, subnet and Enable the Auto-assign public IP.

We are provided with default SSH configuration.

Click on Add security group rule.

Configure the above configurations in your Security group rule.

On Advanced Details, scroll to User data and paste the following command.

#!/bin/bash

# Update and install apache2
apt-get update
apt-get install -y apache2

# Create a simple index.html file with "Hello" text
echo "Hello FIRST" > /var/www/html/index.html

# Restart apache2 service
systemctl restart apache2

Leave all other configurations as default and click on Launch instance.

Now, let's create another instance.

Provide name for your instance, choose an AMI, either create or use an existing key pair.

Edit Network settings, select your second VPC, subnet and Enable the Auto-assign public IP.

Repeat the necessary instructions (security group rule) as earlier.

Here is the user data for the second instance.

#!/bin/bash

# Update and install apache2
apt-get update
apt-get install -y apache2

# Create a simple index.html file with "Hello" text
echo "Hello SECOND" > /var/www/html/index.html

# Restart apache2 service
systemctl restart apache2

Leaving all other configurations as default, click on Launch instance.

We have created two EC2 instances in different subnets residing in different VPCs.

Task 6: Establishing a VPC Peering Connection

A peering connection is a networking connection between two Virtual Private Clouds that allows them to communicate as if they are on the same network.

Go to VPC dashboard, scroll down to Peering connections, click on Create peering connection.

Provide a suitable name and select your first VPC as Requester.

One VPC initiates a peering connection request to another VPC.

We can establish a peering connection to a VPC in our own account as well as to a VPC in another account.

The owner of the target VPC accepts the peering request.

Click on Create peering connection.

Since both VPCs reside within the same account, this acceptance request message appears. In the Actions panel, click Accept request to proceed.

Now, both VPCs must update their route tables to direct traffic destined for the other VPC through the peering connection.

On Route tables, click on one of your Route table.

Click on Edit routes.

Click on Add route.

Duplicate your current tab and copy the IPv4 CIDR of your second VPC.

Note that for the first router, IPv4 CIDR of second VPC must be configured while for second router, IPv4 CIDR of the first VPC is considered.

Paste into the Destination field and select your Peering connection in Target field.

Click on Save changes.

Now, let's set up the necessary route for the second VPC.

Copy the IPv4 CIDR of your first VPC and paste it into the destination field in your second route table.

Click on Save changes.

Now, we have set up the necessary peering routes in the respective route tables.

Task 7: SSH into EC2s

On Instances, select one of your instance and click on Connect.

On SSH client, we are going to follow some instructions.

Copy the command and paste it into your terminal.

Copy the command and paste it into your terminal.

Proceed with yes.

We are into our first instance now.

Now, it's time to SSH into our second instance.

Follow all the instructions as before.

We are now connected to the second instance via SSH.

Copy the private IPv4 address of the second instance.

Type the command curl and paste the copied private IPv4 address of the second instance.

The curl command is being used to retrieve content from a web server at IP address 12.0.1.98.

We can see the second instance is responding to our request.

Let's see it the other way around.

We can see the connection is happening between two instances.

Now, let's remove the connection and see if the connection still happens.

Task 8: Removing Peering Connection

On VPC dashboard, on Route tables, click on your first route table.

Click on Edit routes.

Remove the peering route.

Click on Save changes.

Now, repeat the same process for your second route table.

Okay, now that we have updated our route tables, let's try to make the same request from both of our instances.

There is no response to the request at this time.

Let's try from another instance.

Following the removal of the VPC peering connection, the VPCs can no longer communicate directly as if they were on the same network.

Conclusion

In this session, we explored the concept of VPC peering by establishing connections between two VPCs. We successfully initiated communication between instances in these peered VPCs.

Task 9: Clean Up

1. Terminating Instances.

   

   Select both of your instances and in the Instance state panel, click on Terminate instance.

2. Deleting Subnets.

   

   Go to VPC dashboard, on Subnets, select both of your subnets and in the Actions panel, click on Delete subnet.

3. Deleting Route table.

   

   On Route tables, select both of your route tables and in the Actions panel, click on Delete route table.

4. Deleting peering connection.

   

   Select your created peering connection and in the Actions panel, click on Delete peering connection.

5. Deleting VPCs.

   

   Select one of your VPC and in the Actions panel, click on Delete VPC. Repeat the same to delete the other VPC.

Now, we have cleaned up our used resources.

Alright people, see you in the next one!

AWS WAF defends web apps from online threats with customizable rules to filter and block malicious traffic. Integrated with CloudFront and ALB, it offers real-time monitoring through CloudWatch.

Let's get started.

Overview

In this setup, we will create two subnets, each containing an EC2 instance. These instances will be fronted by an Application Load Balancer (ALB). Then we will configure AWS WAF with a simple IP set rule to restrict connection requests from our IP address.

Task 1 : Creating a VPC

Search for VPC in the search box.

Click on Create VPC.

Provide a suitable name for your VPC and input a IPv4 CIDR value.

Click on Create VPC with all other configurations left as default.

Task 2: Creating Internet Gateway

On Internet gateways, click on Create internet gateway.

Provide name for your Internet gateway & click on Create internet gateway.

Click on Attach to a VPC and select your VPC and click on Attach internet gateway.

Task 3: Creating Subnets

On Subnets, click on Create subnet.

Select your VPC.

Provide a suitable name for your subnet and an eligible CIDR range.

Click on Add new subnet.

Provide name for your second subnet and provide an eligible CIDR range value.

Click on Create subnet.

Task 4: Creating Route tables

On Route tables, click on Create route table.

Provide the name for your Route table and select your VPC.

Click on Create route table.

Now, in Subnet associations, click on Edit subnet associations to associate subnets with this route table.

Select both of your subnets and click on Save associations.

Now, on Routes, click on Edit routes to add route in our route table.

Configure the above configurations and select your internet gateway.

Click on Save changes.

Task 5: Creating EC2 instances

Search for EC2 in the search box.

Click on Launch instances.

Provide name for your instance. Choose an AMI. I have picked Ubuntu as my AMI.

Select free-tier instance if possible and create either a new or use an existing key pair.

Edit Network settings.

Select your VPC, select one of your subnets and Enable the Auto-assign public IP.

We have a default security group configured with SSH access.

Click on Add security group rule.

Configure the above configurations.

Click on Advanced details.

Scroll down to User data section and paste the following code.

#!/bin/bash
yes | sudo apt update
yes | sudo apt install apache2
echo "<h1>Hello from the first server</h1>" > /var/www/html/index.html
sudo systemctl restart apache2

Click on Launch instance.

Copy the Public IPv4 address of your instance and paste into your browser.

Set up another instance in a different subnet. Follow the same steps as before, but this time, make a slight change in the user data. Everything else can remain the same.

#!/bin/bash
yes | sudo apt update
yes | sudo apt install apache2
echo "<h1>Hello from the second server</h1>" > /var/www/html/index.html
sudo systemctl restart apache2

We have set up our second instance as well.

Task 6: Creating Target groups

On Target groups, click on Create target group.

Provide name for your target group, select your VPC while keeping all other settings as default.

Select both of your instances and click on Include as pending below.

Click on Create target group.

Task 7: Creating Load Balancer

On Load Balancers, click on Create load balancer.

Choose ALB as load balancer.

Provide a suitable name for your load balancer.

Select your VPC and also tick all of your subnets.

On Security groups, click on create a new security group.

Provide a suitable name for your security group and select your VPC.

In Inbound rules, configure the above configurations.

Click on Create security group.

Return to your ALB setup page and refresh the Security groups.

Select your security group.

Select your target group & click on Create load balancer.

It takes a while to provision.

When the load balancer is ready, copy the DNS name and paste into your browser.

You can refresh the page to see the load balancer distributing traffic to another server.

Task 8: Setting up AWS WAF

Search for waf on the search box and select WAF & Shield.

On the WAF & Shield Dashboard, on IP sets, click on Create IP set.

IP sets in AWS WAF are collections of IP addresses or IP address ranges that you can use to specify which IPs should be allowed or blocked. By creating an IP set, you can easily manage a list of trusted or suspicious IP addresses and reference this list in your Web ACL rules.

Let's restrict access to our instances by blocking our IP address.

To get your IP address: find your ip address here

Make sure to provide /32 at the end of your IP address to restrict access to only that specific address on the Internet.

Click on Create IP set.

Now on Web ACLs, click on Create web ACL.

When setting up a Web ACL, you need to choose the resource type it will protect. There are two main categories:

1. Amazon CloudFront distributions: These are globally distributed networks that deliver content with low latency.

2. Regional resources: These include services which operate within specific AWS regions.

Selecting the resource type determines how the Web ACL rules will be applied to manage and control access.

Provide name for your WAF.

Click on Add AWS resources.

Select your Application Load Balancer and click on Add.

Click on Next.

In Add rules panel, there are two modes:

1. Managed Rule Groups: These are pre-configured sets of rules created by AWS or AWS Marketplace sellers.

2. Add My Own Rules & Groups: This option allows you to create custom rules based on your specific needs.

Click on Add my own rules and rule groups.

There are different Rule type to choose from. As have already talked about IP set, let's explore the remaining.

• Rule builder: A tool to create custom rules that specify conditions for allowing or blocking web requests.

• Rule group: A collection of rules managed together, which can be custom-made or provided by AWS/third parties, simplifying rule management for web applications.

For now, we are sticking to IP set.

Name your Rule and select your IP set.

When requests come through a CDN or proxy, the source IP is the proxy's, and the original IP is in a header. Using "IP address in header" can be risky because headers can be altered or handled inconsistently by proxies. Choose between using the source IP or the IP in the header to identify the original address.

The purpose is to accurately identify the original source IP address of a request that passes through a CDN or proxy. This helps in applying correct security rules and logging the true origin of the traffic, rather than just the intermediary proxy's IP address. However, caution is needed as headers can be manipulated.

We can specify the action to take when a request originates from an IP address in a defined IP set. For now, let's stick to Block.

Click on Add rule.

AWS WAF evaluates the rules and rule groups in the order shown, starting from the top. We can move rules up or down to change the evaluation order.

The order of evaluation affects the behavior of the web ACL. For example, suppose a web request matches a rule that allows requests and matches another rule that counts requests. If you list the rule that allows requests before the rule that counts requests, AWS WAF will allow the request and won’t count it.

As we do not have multiple rules for now, click on Next.

Click on Next.

Review your Web ACL and click on Create web ACL.

We have created our Web ACL.

Now search for load balancer in the Search box.

Select your load balancer & copy the DNS name. Paste it into your browser.

Task 9: Exploring WAF

We are currently unable to process the request because the WAF is configured to block it. Try reloading the page a few times to see the effect later.

Search for WAF in the search box and select WAF & Shield.

On Web ACLs, select your Web ACLs.

We can explore different aspects here. Let's look at the Traffic Overview.

This section provides insights on various details.

Next, head over to the Sampled Requests section.

We can see the sampled requests here.

Alright, let's delete the rule that we had set up.

On Associated AWS resources section, select your associated resources and click on Disassociate.

On Web ACLs, select your Web ACL and click on Delete.

On IP sets, select your IP set and click on Delete.

Now, let's head up to our load balancer, copy the DNS name, and paste it into our browser to check if the restriction is still in place.

We can see that our request is no longer blocked.

Try refreshing the page to see if the load balancer distributes the traffic to different servers.

Conclusion

Today, we used a technique to protect our resources using AWS WAF by blocking our own IP address using an IP set. This was a simple demonstration. There are many more powerful techniques available with AWS WAF to block web exploits, but that's for another project.

Task 10: Clean Up

1. Deleting load balancer.

   

   Select your load balancer and in the Actions panel, click on Delete load balancer.

2. Deleting Target Group.

   

   Select your target group and in the Actions panel, click on Delete.

3. Terminating EC2 instance.

   

   Select both of your instances, In Instance state panel, click on Terminate instance.

4. Deleting Subnets.

   

   Search for VPC in the search box. On Subnets, select both of your subnets and in the Actions, panel, click on Delete subnet. If you can't delete, wait for some time, refresh the box and try again.

5. Deleting Route table.

   

   Select your Route table and in the Actions panel, click on Delete route table.

6. Deleting VPC.

   

   Select your VPC and in the Actions panel, click on Delete VPC.

We've cleaned up our used resources.

Alright, see you guys in the next one!

Auto Scaling Groups (ASGs), as the name suggests, automatically adjust the number of EC2 instances in response to demand. They help ensure your application is both responsive and cost-effective by scaling up during high traffic and scaling down during low traffic.

Let's get started.

Overview

In this setup, we will create an Auto Scaling Group (ASG) to manage the provisioning and scaling of EC2 instances based on changing demand. The ASG will automatically adjust the number of instances in response to load fluctuations, ensuring optimal performance and cost-efficiency.

Task 1: Creating a VPC

Search for VPC in the search box.

Click on Create VPC.

Provide a name for your VPC and input a CIDR range.

Keeping all other configurations default, click on Create VPC.

Task 2: Creating Internet Gateway

On Internet gateways, click on Create internet gateway.

Provide a name for your Internet Gateway and click on Create internet gateway.

Attach the Internet Gateway to your VPC.

Now, we have attached an internet gateway to our VPC.

Task 3: Creating Subnets

On Subnets, click on Create subnet.

Select your VPC.

Click on Add new subnet to create another subnet.

Click on Create subnet.

We have created two subnets in different AZs.

Task 4: Creating Route table

On Route tables, click on Create route table.

Provide name for your route table, select your VPC and click on Create route table.

On Subnet associations, click on Edit subnet associations.

Tick both of your subnets and click on Save associations.

On Routes, click on Edit routes.

Click on Add route and configure above configurations.

Click on Save changes.

Task 5: Creating Launch template

Previously, launching EC2 instances involved manually configuring every detail for each launch. Launch templates simplify this process by offering a reusable definition - essentially a blueprint - that stores settings such as AMI, instance type, security groups, and more. This streamlines instance creation, enables straightforward configuration versioning, and maintains consistency across your deployments.

On EC2 dashboard, on Launch Templates, click on Create launch template.

Provide a suitable name for your template.

On Quick Start, choose Ubuntu as an AMI.

Choose instance type and either create or select your existing key pair.

In the Network settings, subnets are not selected. Instead, a Security Group is created for the instances. Select your VPC and add the inbound rules to your Security Group.

In the Advanced network configuration, click on Add network interface and Enable the Auto-assign public IP option.

With all other configurations left as default, click on Create launch template.

Task 6: Creating Auto Scaling Group

On EC2 dashboard, scroll to bottom on Auto Scaling Groups.

Click on Create Auto Scaling group.

Name your Auto Scaling Group and select the Launch template you created.

Click on Next.

Select your VPC and choose your subnets.

Click on Next.

For this project, we are not going to associate a load balancer.

The Health Check Grace Period gives new instances time to start up and become healthy before health checks start. This prevents instances from being incorrectly marked as unhealthy and terminated during startup. Setting a grace period allows instances to boot up, configure themselves, and start applications, leading to more stable scaling.

Determines how long to wait before starting health checks on a new instance.

With all other configurations set to default, click on Next.

In normal operations, the Auto Scaling group keeps 2 instances running (Desired capacity). It can scale up to 3 instances in high traffic (Max desired capacity) but never goes below 1 instance in low traffic (Min desired capacity).

Scaling policies determine how the group responds to changes in demand. These policies are based on metrics like CPU utilization, network traffic, or custom CloudWatch alarms.

By setting up these policies, we can ensure our application always has enough resources to handle traffic spikes, while also optimizing costs during quieter periods.

The Target Tracking Scaling Policy in AWS Auto Scaling automatically adjusts the number of instances in a group to maintain a specific metric at a target value.

Instance warm-up refers to time before a new instance is included in scaling activities. It ensures the instance is fully operational before being counted.

Determines how long to wait before including a new instance in scaling decisions.

Our scaling policy adjusts the number of instances to maintain a 50% average CPU utilization. During high traffic, it adds instances to handle the load and maintains performance. During low traffic, it reduces instances to the minimum required, prioritizing cost optimization over meeting the 50% target.

Click on Next, keeping all other configurations default until you reach the option to Create Auto Scaling group.

Task 7: Exploring Auto Scaling groups

Let's perform various actions on our Auto Scaling Groups.

Select your Auto Scaling group and click on the Activity section.

Our Auto Scaling Group is currently launching instances.

The instances for our desired capacity have been launched.

We can see the instances in the Instances console.

Now, let's terminate one of the EC2 instances.

Let's head up to the Auto Scaling Group's Activity section to see what happens next.

We can see our terminating instance while the Auto Scaling Group (ASG) is launching a new instance to reach our desired capacity.

In the meantime, an Auto Scaling Group terminated one EC2 instance. As there is currently no traffic, the ASG is operating efficiently by keeping only the minimum required resources running.

Now, click on the ID of your running instance, then navigate to the Monitoring section.

Enlarge the CPU utilization metric.

Let's increase the CPU utilization.

On your instance details, click on Connect.

On EC2 Instance Connect, click on Connect.

We're now connected to our instance.

Copy the command and paste into your terminal.

sudo apt-get update # updating package lists
sudo apt-get install stress -y # installing stress

• stress utility is used to simulate high CPU usage scenarios for testing and benchmarking purposes.

And now it's time to spike the CPU utilization.

# Stress the CPU cores for 5 minutes
stress --cpu $(nproc) --timeout 300

This command stresses all available CPU cores for 5 minutes using the stress tool. $(nproc) dynamically determines the number of CPU cores.

Let's check the CPU utilization metric of our instance.

We can observe that the CPU utilization is increasing.

Now, let's check the Auto Scaling Group's Activity section to see if there are any changes.

We can see that the Auto Scaling Group (ASG) has launched an instance in response to the increasing load.

As the CPU utilization continues to rise, as we have configured it to spike for 5 minutes, another instance is being launched by the Auto Scaling Group (ASG), and now we've reached our maximum EC2 limit as set by us.

Let's go to the Instances list.

We can see that 2 additional instances have been launched by ASG as the CPU utilization metric crossed the threshold that we had set.

And when the CPU utilization is low, a scale-in event will occur.

That's the beauty of Auto Scaling.

Conclusion

We created an Auto Scaling Group with a minimum capacity of one instance, a desired capacity of two instances, and a maximum capacity of three instances. We observed both scale-in and scale-out events during the testing process. To stimulate CPU utilization, we used a stress utility tool.

Tasks 8: Clean Up

1. Deleting Auto Scaling Group

   

   Select your Auto Scaling Group, and in the Actions panel, click on Delete.

2. Deleting Launch Template

   

   Select your Launch Template and in Actions panel, click on Delete template.

3. Deleting Subnet

   

   Go to the VPC Dashboard. In the Subnets section, select both of your subnets, and in the Actions panel, click on Delete subnet.

4. Deleting Route Table

   

   Select your route table, and in the Actions panel, click on Delete route table. If you're unable to do so, try refreshing the box.

5. Deleting Internet Gateway & VPC

   

   Select your VPC and in the Actions panel, click on Delete VPC.

   Deleting the VPC will also delete the internet gateway associated with it.

We have now cleaned up our used resources.

Alright! See you next time!

An Application Load Balancer distributes incoming application traffic across multiple targets, such as EC2 instances, containers, and IP addresses. Operating at the application layer (OSI layer 7), ALB is ideal for HTTP and HTTPS traffic. It offers advanced routing, integrates with AWS services, and includes features like user authentication and WebSocket support, ensuring high availability and reliability for modern applications.

Overview

In this setup, we will create two EC2 instances in different subnets within a VPC. We will then configure an Application Load Balancer (ALB) to distribute incoming traffic across these instances. The ALB will serve as the entry point for all traffic and will route requests to the targets.

Task 1: Creating a VPC

Search for VPC and click on Create VPC.

Provide a name for your VPC and input a CIDR value for your VPC.

Task 2: Creating Internet Gateway

On Internet gateways, click on Create internet gateway.

After creating an internet gateway, it's time to attach it to our VPC.

Select your VPC and click on Attach internet gateway.

Task 3: Creating Subnets

On Subnets, click on Create subnet.

Select your VPC.

Provide a name for your subnet, choose AZ and input an eligible CIDR range.

For this project, we are picking different AZs.

Create another subnet.

Once you are done, click on Create subnet.

Task 4: Creating Route tables

On Route tables, click on Create route table.

Provide name for your route table, select your VPC & click on Create route table.

After creating route table, On Subnet associations, click on Edit subnet associations.

Tick both of your subnets and click on Save associations.

Now on Routes, click on Edit routes.

Add route, select your internet gateway and click on Save changes.

Task 5: Creating EC2 Instances

Search for EC2 in the search box.

On EC2 dashboard, click on Launch instance.

Provide a name for your EC2, select an AMI, create or use an existing key pair.

Edit Network settings. Select your VPC, select your subnet and Enable Auto-assign public IP.

We are provided with a default security group that has SSH configuration.

Now, let's add another security group rule to allow HTTP traffic to our instance.

Click on Add security group rule.

Configure the above settings in your new security group rule.

Now, at the bottom, click Advanced Details to see more options.

On User data section, provide the following code:

#!/bin/bash
yes | sudo apt update
yes | sudo apt install apache2
echo "<h1>Hello from the first server</h1>" > /var/www/html/index.html
sudo systemctl restart apache2

User data in EC2 instances refers to a script or set of commands you can provide when launching an instance. This script runs automatically when the instance starts for the first time. This process is known as bootstrapping.

User data automates the setup of EC2 instances, saving time and ensuring consistency by automatically installing software and configuring settings when the instance starts, eliminating manual tasks.

But for now, we are just displaying Hello message.

Click on Launch instance.

After your instance is up and running, go to the instance details, copy the Public IPv4 address, open a new tab, and paste the address there.

You may see something like this.

Now, let's create another instance.

On Network settings, for subnet, choose another subnet than you chose earlier.

Now add additional inbound rules like we did earlier.

For User data, add the following script:

#!/bin/bash
yes | sudo apt update
yes | sudo apt install apache2
echo "<h1>Hello from the second server</h1>" > /var/www/html/index.html
sudo systemctl restart apache2

And all the procedures are the same, and we get this.

We have created two instances.

Task 6: Create a Target Group

A target group is a set of resources that a load balancer routes traffic to. It ensures efficient traffic distribution and high availability by routing only to healthy targets. Target groups also support configuring rules to direct traffic based on specific conditions, helping manage and scale applications effectively.

On EC2 dashboard, scroll down to Target Groups.

Click on Create target group.

Provide a name for your target group and select your VPC, keeping all other configurations as default.

Click on Next.

Select both of your instances and click on Include as pending below.

This option allows you to add targets to a target group in a non-active state initially. These targets won't receive traffic from the ALB until they pass the configured health checks.

Click on Create target group.

Task 7: Creating Security Group

Security Groups (SGs) act as virtual firewalls for your instances to control inbound and outbound traffic. You can define rules to allow or deny specific traffic based on IP addresses, ports, and protocols. Security Groups are stateful, meaning if you allow an inbound request, the response is automatically allowed.

Let's create security group for our Load Balancer.

On EC2 dashboard, on Security Groups, click on Create security group.

Provide a suitable name, description is optional & select your VPC.

On Inbound rules, click on Add rule to configure above configurations.

We're allowing HTTP traffic from anywhere on the internet.

Click on Create security group.

Task 8: Creating Load Balancer

A Load Balancer (LB) distributes incoming traffic across multiple targets, such as EC2 instances, to ensure high availability and reliability. It helps balance the load, improving application performance and fault tolerance. There are different types of load balancers, including Application Load Balancer (ALB) for HTTP/HTTPS traffic and Network Load Balancer (NLB) for TCP/UDP traffic.

On EC2 dashboard, On Load Balancer, click on Create load balancer.

Choose ALB as the load balancer and click on Create.

Provide a name for your ALB.

When you create a load balancer, you must decide whether to make it an internal load balancer or an internet-facing load balancer.

The nodes of an internet-facing load balancer have public IP addresses.

The nodes of an internal load balancer have only private IP addresses.

Both internet-facing and internal load balancers route requests to your targets using private IP addresses.

Select your VPC and tick on your AZs.

In addition to the default security group, select the security group that we created earlier for this load balancer.

A listener is responsible for checking or listening for incoming connection requests on a specified protocol and port, such as HTTP on port 80 or HTTPS on port 443. Once a listener accepts a connection, it uses routing rules to determine how to direct the traffic.

These rules can route traffic based on hostnames, URL paths, HTTP headers, or query string parameters. The traffic is then forwarded to the appropriate target group.

You can read a summary about your Application Load Balancer.

Click on Create load balancer.

Let's wait for a while. It may take 3 to 4 minutes.

After a few minutes, our ALB is active. Copy the DNS name of your ALB and paste it into your browser.

Try refreshing the browser.

Our ALB is actively distributing traffic to the targets we set up.

Conclusion

We have successfully set up an Application Load Balancer (ALB) to distribute traffic, improving our application's availability and reliability. We added two EC2 instances as targets and configured the ALB to direct traffic to them.

Task 9: Clean Up

1. Deleting Load Balancer

   

   Select your load balancer and on Actions panel, click on Delete load balancer.

2. Deleting Target Group

   

   Select your target groups and click on Delete.

3. Terminating instances

   

   Select both of your instances and click on Terminate instance.

4. Deleting Subnets

   

   Search for VPC in the search box and click on VPC.

   

   After navigating to the VPC dashboard, select both of your subnets under Subnets and click on Delete subnet.

5. Deleting Route tables

   

   Select your Route table and in Actions panel, click on Delete route table.

   

   If an error occurs, try refreshing the box and then click on Delete.

6. Deleting VPC

   

   Select your VPC and click on Delete VPC.

   

   This action will also delete our Internet Gateway. Alternatively, we could perform these actions individually.

We have successfully cleaned up the resources that were in use.

Hope you enjoyed the project!

I'll see you in the next one!

Let's dive in right away.

Analogy

A bastion host is like a secure gatekeeper for your network, ensuring no one can bypass it to get inside. It checks everyone's identity and keeps a record of all entries, allowing only authorized users. With minimal features to reduce risks, it is well-protected against attacks, acting like a vigilant gatekeeper monitoring and controlling access to the castle.

Task 1: Create a VPC

Search for VPC in the search box.

Click on Create VPC.

Provide a suitable name for your VPC and provide CIDR range.

Leaving all other configurations as default, Click on Create VPC.

Our VPC is created.

Task 2: Creating an Internet Gateway

Internet Gateway to provide our VPC internet access.

On the VPC dashboard, On Internet gateways, click on Create Internet gateway.

Provide a suitable name for your IG and click on Create internet gateway.

Now, we have to attach it to our VPC.

Select your VPC and click on Attach internet gateway.

Task 3: Creating Subnets

On the VPC dashboard, On Subnets, click on Create subnet.

Select your VPC.

First, we are going to create Public subnet. Provide a suitable name for your subnet and an eligible CIDR range.

For this project, we're only using single Availability Zone.

Click on Add new subnet.

Now, we'll set up our Private subnet. Provide a suitable name, choose the AZ and an eligible CIDR range.

Click on Create subnet.

We've successfully created two subnets.

Task 4: Creating Route Tables

On VPC dashboard, On Route tables, click on Create route table.

Provide name for your route table, select your VPC and click on Create route table.

After creating route table, on Subnet associations, click on Edit subnet associations.

Tick your Public subnet and click on Save associations.

Now, Click on Edit routes.

Add route and configure the above configurations. Select your Internet Gateway and click on Save changes.

The Public subnet can access the Internet now.

Now, it's time to set up our Private subnet.

Provide name for your Private route table and select your VPC and click on Create route table.

On Subnet associations, click on Edit subnet associations.

Tick your Private subnet and click on Save associations.

For this Private route table, we do not have to configure any additional routes.

The route table entry 10.0.0.0/16 with the target local indicates that traffic within this IP range is allowed to communicate within the VPC.

Task 5: Creating EC2 instances

Search for EC2 on the search box.

Click on Launch instance.

We'll create our Bastion Host first.

Provide name for your Bastion Host, select Ubuntu as an AMI, choose free tier instance if available.

Create a new key pair.

Edit Network settings. Select your VPC, choose your Public subnet and Enable Auto-assign public IP.

By default, we're provided with an Inbound Security Group Rules.

We can SSH into this instance from anywhere on the internet.

Leaving all other configurations as default, click on Launch instance.

Now, let's create Private instance. Click on Launch instances.

Provide a name for your instance, select your AMI, select the instance type, and create a new key pair. You can also use an existing key pair that was created earlier.

I'm creating separate keys for different instances.

Edit Network Settings. Select your VPC, choose your Private subnet and leave Auto-assign public IP as default.

In the Inbound Security Group Rules, the Source type should not be set to "Anywhere." We must configure it to allow SSH access only from the resources that resides in our Public subnet.

This ensures that only instances within the Public subnet can SSH into the instance.

Leave all other configurations as default and Click on Launch instance.

We've created two instances.

Task 6: SSH into our Bastion Host

Select your Bastion Host and now click on Connect.

On SSH client, we have to follow some steps shown there.

I'm in my SSH client, i.e., terminal.

Copy the command and paste into your terminal.

You can view your permissions using the following command:

ls -l

We have changed the file permissions to make it readable by the owner only.

Copy the command & paste into your terminal.

Make sure your key file is present in your working directory; otherwise, provide the full path to the key file.

Proceed with yes.

Now, we have logged into our instance using SSH.

Task 7: Copying the Key File

Open new terminal in your local computer.

We're copying the private key file that is in our local computer to our Bastion Host.

ssh -i "myKey.pem" ubuntu@3.83.154.159

For ease, copy the command that we use to SSH into our instance. It's so to modify it a bit.

Paste the command in your Desktop's terminal.

scp -i "myKey.pem" privateKey.pem ubuntu@3.83.154.159:/path/to/destination/

Replace and add commands/configurations as required.

Here's a breakdown of the command:

• scp stands for Secure Copy. It is a command-line utility that allows you to securely copy files and directories between two locations.

• -i my-key.pem: Specifies the identity file (private key) to use for authentication.

• privateKey.pem: Specifies the source file on your local machine that you want to copy.

• ubuntu@3.83.154.159:/path/to/destination/: Specifies the destination on the remote

Alright.

To see your present working directory, In your Bastion terminal, type command:

pwd

Now, replace the /path/to/destination/ with your pwd location in your Desktop's terminal.

Now see if the key has been successfully copied into our Bastion Host by listing files/folders in our Bastion's terminal.

We can see that the private key has been copied from our local computer to the remote instance.

Task 8: SSH Login to Private Instance

It's time to SSH log in into our Private instance.

Select your Private instance and click on Connect.

Copy the command.

But before that, let's see what permissions are set initially,

We can see there are read/write for Owner, read/write for Group & read permissions for Others.

It is recommended to set the permissions of the file or directory to the minimum necessary for the required operations.

So, let's do that.

Paste the command that you had copied into the Bastion's terminal.

We can see the file permissions have been changed now.

Now, copy the SSH command.

Paste into your terminal.

Proceed with yes.

Now we are into our Private instance.

You can verify this by examining the details in the instance configurations.

Alright, that's our work done.

Conclusion

This configuration helps keep our Private instance secure by limiting direct access from the internet. Instead, we can access it securely through the Bastion host. This setup is a common practice for enhancing security in cloud environments.

Task 8: Clean Up

1. Terminating instances.

   

   Select both of your instances, on Instance state, click on Terminate instance.

2. Deleting Route tables.

   

   Search for VPC in the search box.

   

   On Route tables Subnet associations, click on Edit subnet associations.

   

   Uncheck the box and click on Save associations. Do the same for Private Route tables.

   

   

   Now, as we do not have any associations, we can delete the route tables.

   

3. Deleting Subnets

   

   Tick both of your subnets and on Actions panel, click Delete subnet.

4. Deleting Internet Gateway.

   

   Before we are able to delete the IG, we've to detach it from our VPC.

   

   After detaching from our VPC, we can delete our Internet gateway.

5. Deleting VPC.

   

   Now, we have cleaned up our resources that we had used in our lab session.

Alright folks, See you in the next one!

Today, we'll be setting up our Virtual Private Cloud, a.k.a. VPC.

Let's get started right away.

Overview

In this setup, we will create a VPC with an Internet Gateway for internet access. We'll configure route tables for traffic management. Inside the VPC, we'll have public subnets for instances directly accessible from the internet and private subnets for instances needing internet access but not directly reachable. To enable internet access for private instances, we'll deploy a NAT Gateway in the public subnet.

Task 1: Creating a VPC

On Services, search for VPC and select VPC.

A Virtual Private Cloud (VPC) in AWS is a customizable network environment that allows you to isolate and securely manage your AWS resources.

Click on Create VPC.

Provide a suitable name for your VPC.

Input the given CIDR in the IPv4 CIDR block.

For the 10.0.0.0/16 network, the first two octets are reserved for the network portion, while the remaining two octets are for host addresses. This allows for 2^16 = 65,536 total IP addresses.

Click on Create VPC.

We've created our VPC.

The other one is default, created by AWS.

Task 2: Creating an Internet Gateway

An Internet Gateway (IGW) allows your VPC to connect to the internet, enabling resources within the VPC, to send and receive internet traffic.

On the VPC dashboard, select Internet gateways.

Click on Create internet gateway.

Provide a name for your internet gateway and Create internet gateway.

Now, it's time to attach the Internet Gateway (IG) to the VPC.

We can do this using either of the options.

Select your VPC and Attach internet gateway.

Task 3: Creating subnets

A subnet is a segment of a VPC's IP address range where you can place resources such as EC2 instances. Subnets allow you to group resources based on security and operational needs, and can be either public (with internet access) or private (without direct internet access).

On VPC dashboard, on Subnets, click on Create subnet.

Select your VPC.

Provide suitable name for your subnets.

Our first subnet, publicSubnet, has a CIDR value of 10.0.1.0/24.

It will have a total of 2^8 = 256 addresses.

Also, for this project, we're only choosing a single availability zone.

Click on Add new subnet and create a private subnet.

Our second subnet, privateSubnet, has a CIDR value of 10.0.2.0/24.

It will have a total of 2^8 = 256 addresses.

Click on Create subnet.

We've successfully created 2 subnets.

Task 4: Creating a NAT Gateway

A NAT Gateway allows instances in a private subnet to initiate outbound traffic to the internet, while preventing inbound traffic from reaching those instances directly. It provides a way for these instances to access the internet while remaining private.

On VPC dashboard, search for Nat gateways and click on Create NAT gateway.

Provide a suitable name and select your private subnet.

For public connectivity, instances in private subnets can connect to the internet through a public NAT gateway, but they cannot receive unsolicited inbound connections from the internet. You create a public NAT gateway in a public subnet and associate an elastic IP address with it.

On the other hand, the Private connectivity type allows instances in private subnets to connect to other VPCs or your on-premises network. Unlike the Public NAT gateway, a Private NAT gateway does not use an Elastic IP address.

Click on Allocate Elastic IP.

An Elastic IP (EIP) is a static IPv4 address associated with your AWS account, not a specific resource. It's ideal for scenarios requiring a persistent public IP that can be easily reassigned to another instance during failures, updates, or scaling.

Click on Create NAT gateway.

It takes some time to configure.

Alright, let's move on.

Task 5: Creating route tables

Route tables direct network traffic within your VPC. They contain a set of rules (routes) that determine where network traffic from your subnet or gateway is directed.

On VPC dashboard, select Route tables and click on Create route table.

Since we have two subnets, one private and one public, for different purposes, we need to create two route tables.

Provide name for your Route table and select your VPC.

Click on Create route table.

After creating route table, on Subnet associations, click on Edit subnet associations.

Select your Public Subnet and click on Save associations.

Now we need to add routes so that our public subnet can access the Internet.

Click on Edit routes and add the following configurations on Add route.

So, we have added a rule to access the destination 0.0.0.0/0 using the Internet Gateway. In routing tables, a destination of 0.0.0.0/0 means all IP addresses, effectively representing the entire internet.

Click on Save changes.

Now, for private subnets, let's create a route table.

Provide a suitable name and assign your VPC.

Click on Create route table.

Click on Edit subnet associations.

Tick your Private Subnet and click on Save associations.

Now, time to add route. Click on Edit routes.

Select your NAT Gateway and click on Save changes.

This configuration allows instances in the private subnet to access the internet through the NAT Gateway.

Alright, we have now associated an Internet Gateway with the public subnet and a NAT Gateway with the private subnet.

Task 6: Creating EC2 Instances

Now, it's time to create EC2.

Search for EC2.

Click on Launch instance.

Let's create Public instance first.

Provide a suitable name. I chose Ubuntu as AMI.

Select a free tier instance if available and create a new key pair or use your existing one.

Edit Network settings. Select your VPC, select the public subnet, and enable Auto-assign public IP.

Assigning a public IP allows an instance to communicate directly over the internet.

Leave all other configurations as they are and click Launch instance.

Alright, let's go for Private instance now. Click on Launch instances.

Provide name, select an AMI, select the free-tire instance.

We can use our previously created key pair or create a new one.

Edit Network settings. Select your VPC, select Private subnet and leave the Auto-assign public IP as default, i.e., Disable mode.

A private instance does not require a public IP address, as it is not intended to communicate directly over the internet.

Leave all other configurations as they are and click Launch instance.

Alright, we've created two instances: one public and one private, each residing in different subnets, and that sums up our project.

Conclusion

To sum up, we created a Virtual Private Cloud (VPC) with two subnets in one Availability Zone. The public subnet is for resources needing direct internet access, like public EC2 instances, connected to an Internet Gateway (IG). The private subnet, for more secure resources like private EC2 instances, is connected to a Network Address Translation (NAT) Gateway. We also configured route tables to manage traffic flow.

Task 7: Clean up

Now, it's time to clean up the resources.

1. Terminating EC2 instances.

   Select your instances and, on Instance state, click Terminate instance.

   

   Click on Terminate.

   

2. Deleting route tables.

   Head up to VPC Dashboard by searching VPC on the Search box.

   1. On Route tables, select your one route table and click on Edit subnet associations. We've to disassociate subnets first to delete the route table.

      

      

      Uncheck the box and click on Save associations. And the same for another route table. Select route table, on Subnet associations, click on Edit subnet associations.

      

      Uncheck the box and click on Save associations.

      

      Now, select both of your route tables and on Actions menu, click on Delete route table.

      

      Type delete and click on Delete.

3. Release Elastic IP.

   

   On Elastic IPs, select your elastic IP and click on Release Elastic IP addresses.

4. Deleting NAT Gateway.

   

   Select your NAT Gateway and on Actions panel, click on Delete NAT gateway.

   

5. Detaching Internet Gateway.

   

   Select your Internet Gateway and, on Actions panel, click Detach from VPC.

   

   After detaching, we have to delete the internet gateway.

   

6. Deleting subnets.

   

   Select both of your subnets and on Actions menu, click on Delete subnet.

   

7. Deleting VPC.

Alright, now, we've cleaned up our used resources.

See you in the next one!

Let's get started.

Overview

In this setup, we will launch an EC2 instance on AWS. Then, using our local terminal, we will establish an SSH connection to that EC2 instance.

Task 1: Launching an EC2 instance

On services, search for EC2 and click on it.

Now, click on the yellow button Launch instance below.

Provide a suitable name for your EC2 and choose Ubuntu as AMI.

AMI: An AMI (Amazon Machine Image) is a template for launching virtual servers in the cloud, containing the operating system, software, and settings.

We're utilizing the free tier instance provided by AWS.

Now, time to create a key pair.

Key pairs are a fundamental security element for accessing EC2 instances in AWS. They utilize public-key cryptography to ensure secure logins.

When we launch an EC2 instance, we can select an existing key pair or create a new one. The public key is stored on the instance, while we securely store the private key on our local machine.

To connect to the instance, we use an SSH client and the private key to authenticate ourselves.

I've downloaded the private key file in my device.

Our current configuration allows SSH access to our instance from any location on the internet.

Leave all other configurations as default. On the right panel, we can see the Summary section on our EC2 specifications.

Click on Launch instance.

Click on the instance ID.

We can see we have successfully launched our EC2 instance.

Click on the instance ID to see various details of our EC2 instance.

Task 2: Accessing EC2 with SSH

SSH (Secure Shell) is the secure way to connect to and manage our EC2 instances. It provides encrypted communication between our local machine and the instance, protecting our login credentials and commands.

Prerequisites:

• Running EC2 Instance: A launched EC2 instance with SSH enabled.

• SSH Client: An SSH client installed on our local machine. Common options include terminal (Linux/macOS) or tools like PuTTY (Windows).

• Key Pair: Ensure you have a key pair created and downloaded securely. The public key should be associated with your instance.

As we have done all the prerequisites, it's time to get the work done.

Head up to your instance details and click on Connect.

Alright, now we have to follow the given instructions.

But before we continue, let's learn about the file permissions first.

The table shows uses octal numbers (base-8) to define three permission levels: owner, group, and others. Each permission level can be granted read, write, and execute access. For example, a chmod value of 7 (111 in octal) grants read, write, and execute permission to all three levels (owner, group, and others).

Say, there is a file with the value 777, it represents full access. This translates to all three user levels (owner, group, and others) having permission to read, write, and execute the file.

Alright, back to our demo.

I'm in my desktop terminal, and we can see the permissions of our key file here.

ls -l

Owner: read/write permission

Group: read/write permission

Others: read permission

chmod is a command used in Unix-based operating systems to change the permissions of files and directories.

Copy the command and paste it into your terminal.

Now, we see the permissions have been changed.

Only the owner has the read permissions now.

Let's continue.

Copy the command and paste it into your terminal.

Let's break down the command.

• ssh: This is the command itself, which stands for Secure Shell. It's used to initiate a secure connection to a remote server.

• -i "myKey.pem": This option specifies the identity file to be used for authentication. In this case, it's "myKey.pem" which is my private key file. The quotes around the filename are necessary if there are spaces in the path. Also, if you have your private key file in a different directory than your current working directory, you have to specify the full path.

• ubuntu: This specifies the username required to log in to the remote server. Here, the username is "ubuntu" (default).

• ec2-13-238-254-242.ap-southeast-2.compute.amazonaws.com: This is the hostname or IP address of the remote server we're trying to connect to.

Alright, now hit the Enter.

Proceed with yes when prompt.

And finally, we have logged in into our EC2 instance.

Task 3: Clean Up

Select your instance and Select Terminate instance on Instance state panel.

Click on Terminate.

This concludes our lab.

Conclusion

In this lab, we created an EC2 instance and then connected securely via SSH.

See you in the next one.

Overview

In this blog, we will explore AWS IAM roles by creating a user with no S3 permissions. Then, we will create a role with full S3 permissions. Finally, we will log in as the user with no permissions and switch to the role with full S3 access, demonstrating how roles can be used to manage access control in AWS.

IAM Role

An IAM (Identity and Access Management) role in AWS is an identity that you can create in your AWS account. It has specific permissions and is designed to be assumed by trusted entities such as users, applications, or services.

Unlike an IAM user, a role does not have long-term credentials (password or access keys) associated with it. Instead, it provides temporary security credentials for the purpose of accessing AWS resources.

Why IAM Roles?

1. Security: Roles use temporary credentials, reducing the risk if credentials are compromised.

2. Cross-Account Access: Allow users or services from one AWS account to access resources in another account.

3. Least Privilege Principle: Grant only the necessary permissions for a specific task or session.

Attaching Role directly to a user?

We cannot attach role directly to a user.

Here's why directly attaching roles to users wouldn't be ideal:

• If roles were directly attached to users, it would mean the user would always have those permissions. This could be a security risk, as it wouldn't allow for fine-grained control over access.

• Without role assumption, users would be stuck with the permissions assigned to their attached role. This wouldn't be flexible for scenarios where a user might need different permissions for different tasks.

Instead, IAM uses a concept called "assuming a role". This means a user (or another trusted entity like an application) can temporarily take on the identity of a role, along with the permissions that role provides. This grants them access to AWS resources for a specific session.

sts:AssumeRole

sts:AssumeRole is an AWS Security Token Service (STS) operation that allows a user or service to obtain temporary security credentials for assuming a different role.

Key Points

1. Purpose: To grant temporary access to AWS resources based on the permissions of a specified role.

2. How it Works:

   • Request: A user or service calls sts:AssumeRole and specifies the ARN (Amazon Resource Name) of the role they want to assume.

   • Verification: AWS STS checks if the caller is allowed to assume the role based on the role's trust policy.

   • Temporary Credentials: If allowed, STS issues temporary security credentials (Access Key ID, Secret Access Key, and Security Token).

3. Temporary Credentials:

   • Access Key ID: Like a username for programmatic access.

   • Secret Access Key: Like a password, used together with the Access Key ID.

   • Security Token: Adds an additional layer of security and has a short expiration time (usually up to 1 hour).

4. Session with Assumed Role:

   • Use Temporary Credentials: With these credentials, the user can access AWS resources as allowed by the role, for a limited time.

Trust Policy vs. Role Policy

Trust Policy:

• Determines who can assume the role.

• Attached to the role and uses sts:AssumeRole.

Role Policy:

• Determines what actions can be performed when the role is assumed.

• Attached to the role and specifies allowed actions and resources.

Dirty Hands

Alright, let's get our hands dirty.

To begin with, let's create a new user.

Go to IAM section, and on Users, click on Create user.

Click on Next until you create a user.

I've created a User named testuser.

Now, let's create a Role.

We are granting permission to users, applications, or services from different AWS accounts (including accounts we own and accounts owned by others) to access and perform specific operations in our AWS account

Click on Next.

Search for s3 and select the policy where we're granting full s3 access for this role.

Click on Next.

Provide a suitable role name.

This trust policy allows any user or entity belonging to the AWS account with ID 767398070747 to assume the role.

When someone from that account tries to assume the role (using the sts:AssumeRole API operation), as long as the account ID matches the one specified in the trust policy, they will be granted temporary security credentials to act with the role's permissions for a limited time.

While, it is the permissions policy we'd given earlier.

Click on Next and Create a role.

Our role has been created.

Now, let's log in as recently created testuser. For that, go to Dashboard, and copy the Sign-in URL.

Open private window, paste the link and login as recently created New User.

You either provide Account ID or alias. I had created alias for ease.

I've logged in as testuser.

Now, let's try to see the list of buckets if there's any.

We do not have the permissions to list buckets.

Now, let's assume the s3Access role for this testuser.

For this, we've to go back to the iamadmin console and proceed.

After you're in your iamadmin console, Click on Role, select the role that you'd created and copy the ARN.

Save it to somewhere for now.

Now, click on Users and select the testuser.

Scroll a bit and click on Add permissions.

Click on Create inline policy.

On Action, type sts:AssumeRole

Also, paste the copied ARN in Resource field.

This policy allows users from the account ID 767398070747 to assume the "s3Access" role. Once they assume the role, they can temporarily act with the permissions assigned to that role.

Click on Next and provide relevant policy name and Create Policy.

Ok, now our testuser can assume that role that grants the user full access to s3 services.

Head up to testuser console.

As we discussed earlier, role can't be assigned directly to the user. So, we've to switch role.

Click on Switch role.

Provide the details and click on Switch Role.

Now we can see We've switched our role, and we can see no restrictions as before. We've assumed a role that has full access to s3 services. We can perform different operations on s3 for a limited time.

And, to go back to our own profile, simply click on Switch back.

Now, as we've switched back, and we have no longer permissions to perform operations on s3.

In this way, by carefully crafting trust policies, we can grant controlled access to our AWS account's resources. We can specify exactly who (or what) can assume a role and what actions they can perform while acting under that role's permissions. This allows for secure and granular access management within your AWS environment.

Now, time to clean up.

Head back to your iamadmin console.

Click on roles and delete the created role.

Similarly, delete the created user.

Alright, that was it for this demo.

See you in the next one.

Thank you.

Let's dive in.

Content Based Message Routing System

Let's explore different architectures.

Say, we have an insurance system. If anyone purchase an insurance, System A get's data and sends message to the respective backend processing.

How would you design the architecture where System A can send message to different backend servers effectively?

First method: Just calling API's to different system.

System A will be having some conditioning logic like if insurance type is CAR, call CAR API, if it's house, call HOUSE API and so on. It's not considered a good practice.

As traffic grows, all the system need to scale up at the same time. And say, CAR system breaks down, the system A need to resend the message as it is synchronous architecture (API-based).

When there is need of message, routing, message based/event-based processing, always think of queue.

You can insert the message in queue. Even if there are lots of messages in the queue, the backend servers will eventually process them. Also, retries are built-in in queue.

Second method: Using SQS, Lambda

We've SQS that aces as a queue where System A stores messages & lambda/EC2. Based on the type of message, it calls 3 different systems. But here the lambda becomes the bottleneck. It's easier to scale SQS on a message based system than to scale lambda.

Also, as a single lambda is processing all 3 different types of servers, so even if the volume increases, say CAR insurance, it might impact the HOUSE & MOTORCYCLE processing. The CAR insurance's traffic volume might take over the concurrency limit of lambda.

Third method: Using SNS Topic

Here, System A publishes the message in SNS topic while doing that, it also attaches meta-data to the message. SNS can do filtering based on the metadata.

Metadata is not included inside the message body. Metadata (message attributes) provides additional context about the message, but is not part of the message content itself.

Say, message looks like this

#Message
{
    "INS_type": "CAR",
    "NAME": "RAM",
    "PRICE": 50
}

The metadata is not included in the message, but SNS can accept a message with metadata. Metadata could have field like insurance type, then have CAR, BIKE as a value.

#Meta-data
"ins_type": "CAR"

Note that the metadata insurance type is different from the insurance type within the JSON message.

So, when System A sends a message to SNS topic, it can attach metadata & based on the metadata, SNS topic can send to the respective queue, and it can be processed using lambda/EC2.

Each of the queue & backend can be scaled independently, which is considered a good approach.

When you want to use the actual message instead of attaching metadata, we can use Amazon Event Bus.

Fourth method: Using Amazon Event Bus

In this architecture, System A will simply publish the message into Amazon Event Bus & the message could have the field and the event bus can do filtering based on the value of the field. So, it can inspect the field insurance type and send it to the respective SQS.

It can also handle advanced conditions, pattern matching, and more.

Event Bus vs. SNS

• Event Bus can do simple to advanced conditioning, pattern matching, however it is more expensive. SNS on the other hand is cost-effective.

• SNS and SQS can scale almost infinitely, while Event Bus has limits on scaling.

Image Storage & Retrieval Architecture

When considering how to save or store images, there are generally two options:

1. Saving in a database.

2. Saving in object storage, such as Amazon S3.

When we save an image, and someone retrieves the image, it's not always the same image.

For example, on Facebook, we can upload a super high-resolution image, such as one with 20 megapixels. However, when we view this image in our Facebook feed on a device and try to zoom in, the image appears pixelated. While the original image remains the same, it gets downsampled based on the device being used.

When we talk about saving & retrieving images, it's not just about saving and retrieving, it's also the processing that's involved which is downsampling the images for different resolutions based on the device type. Downsampling images not only saves bandwidth but also enhances user experience, reduces storage costs, optimizes performance, improves network efficiency, and ensures consistency across devices.

If we store images in a database, which can be disk-based or memory-based, it is generally more expensive than using regular object-based storage. Databases provide faster IOPS (Input/Output Operations Per Second), enabling quicker image retrieval. However, in many architectures that handle images, such as those used by Amazon or Tinder, it is acceptable for images to take slightly longer to load, such as an additional second. In these systems, it is not critical for images to be retrieved in microseconds.

Therefore, almost all architectures that manage a high volume of images use object-based storage solutions like Amazon S3.

Simplified System Without Downsampling/Upscaling

If downsampling and upscaling are not required, the system can be simplified. Users can directly upload images to a predetermined folder in S3 using API Gateway. For retrieval, since there’s only one copy of the image, the API Gateway uses a fixed path to fetch the image from S3 and return it.

Let's consider a user named Smith. He uploads an image called cat.jpg. Smith invokes an API hosted on API Gateway and points to the image saved in his local folder. This API will call a backend Lambda function, which will save the image in an S3 bucket named Images. Within this S3 bucket, the image cat.jpg will be saved in a folder called Images/Cats.

API Gateway can save the image directly into S3. If you don't need any business processing.

But generally what happens is, the lambda does some processing for example, to check if the image contains explicit contents, it'll call Amazon Recognition.

If we consider scenarios with multiple S3 buckets and different folder structures, we want to avoid hardcoding these details into the retrieval code. For instance, hardcoding conditions like 'if the region is X, go to this bucket and this folder structure' becomes impractical when new folder structures are added, as it requires constant code changes.

Instead, the Lambda function will save the image location into a database. While it saves an image to an S3 bucket, it will also record the location in a database table, such as 'cat.jpg is saved in Images/Cats'. Additionally, we assume that the list of eligible S3 buckets is also stored in a database table. The Lambda function will fetch the appropriate bucket name from the database and save the image accordingly.

Now, regarding the upsampling and downsampling of images after they are uploaded, to enhance retrieval efficiency and storage optimization, we follow the following approach.

Once an image is uploaded to S3, it triggers a Lambda function. This Lambda function uses an image processing library to downsample/upsample the image into different resolutions such as 10 MP, 7 MP, and 5 MP. The processed images are saved into different S3 buckets. The Lambda function also records the locations of these images in a DynamoDB table, storing entries for each resolution.

If further processing is required beyond saving metadata, a backend_lambda function can be used. Alternatively, if no additional processing is needed, API Gateway can upload images directly to S3. The S3-triggered Lambda function then handles resizing and saving metadata into DynamoDB. This way, both the original and downsampled images are saved in S3 buckets, with their pointers stored in a DynamoDB table.

Image Retrieval

To retrieve an image, such as 'cat.jpg', the process involves invoking an API with the image name. A backend Lambda function queries DynamoDB to find the image location. The application can detect the user's device type (e.g., cellphone, computer, big-screen monitor) and set the appropriate header on the API Gateway. The Lambda function then retrieves the image location from DynamoDB based on the device type. With different entries for each resolution in the database, the Lambda function fetches the corresponding image from the appropriate S3 bucket and returns it.

High Priority Queuing Messaging System

Let's consider an insurance company where System A receives messages for different types of insurance: car, house, and bike. System B should process car insurance messages with higher priority.

<!--- Application logic for system A --->
{
    instype: CAR,
    accountno. = 12345678,
    amount = 500
}

Let's explore several design options.

Option 1: RabbitMQ

One potential solution involves using RabbitMQ or another message queue (MQ) service. RabbitMQ supports priority queues, allowing you to set a priority value in the message header.

System A sends messages to RabbitMQ. System A checks the insurance type. If it's car insurance, it sets a higher priority in the message header. RabbitMQ processes higher priority messages first.

Option 2: Amazon SQS

A more common and robust solution leverages Amazon SQS. Although it does not support message priority natively, we can create separate queues for different priorities.

We create two SQS queues: one for car insurance messages and one for other insurance types. System A checks the insurance type. If the message is for car insurance, it goes to the high-priority SQS queue other messages go to the standard SQS queue.

We use EC2 instances or AWS Lambda to process messages. For car insurance (high-priority queue), we use an auto-scaling EC2 group to handle varying loads efficiently. We deploy more powerful EC2 instances to ensure quick processing. For other insurances (standard queue), we use a smaller EC2 instance or a limited auto-scaling group to save costs ( can also use ASGs if required ). For Lambda, we adjust memory and concurrency settings to manage processing speed.

Option 3: AWS EventBridge

If we want to minimize logic within System A, we can use AWS EventBridge.

System A sends all messages to EventBridge without any filtering. EventBridge uses pattern matching to route messages based on the insurance type.

Messages for car insurance are sent to the car insurance SQS queue. Other messages are sent to a different SQS queue. Similar to the SQS setup, use EC2 or Lambda to process messages from the respective queues.

SQS & Amazon EventBridge

Both queues can be configured with DLQs to handle message processing failures. DLQs ensure that unprocessed messages are saved for later reprocessing.

If EventBridge fails to deliver a message, it can send the message to a DLQ. However, once a message is delivered to an EC2 instance or Lambda, EventBridge does not handle failures. The application must manage DLQ integration.

Data analytics designs

When it comes to data analytics, there are four main steps:

1. Collect the data.

2. Transform those data as per the analytics requirement.

3. Query those data to find out the necessary information.

4. Create some sort of report that gives you some business insight.

So if we divide this with services, this would look something like this:

Let's start with the sample architecture.

Query and Report on Click Stream

Let's say you need to design an architecture that can query and report on some clickstream data. When we talk about clickstream, there has to be some sort of website, and you need to track all the clicks that users are doing. Generally, you stream it using some sort of streaming service such as Amazon Kinesis Data Firehose, or you could use Kafka as well.

Amazon Kinesis Data Firehose can dump all the data into Amazon Simple Storage Service (S3). As we already know, S3 is a very scalable and cost-effective object-based storage service. The data in S3 is kind of unstructured. Even if you put structured data or comma-separated data in S3, there are no tables defined in S3 because it's object-based storage.

This is where AWS Glue comes in.

Say we're working with a CSV file in an S3 bucket containing sales data. The file has columns for things like order ID, product name, price, quantity, and customer location.

• Without AWS Glue: In raw S3 storage, this data is just a flat file. You wouldn't know the structure (columns) or data types (numeric, text) by looking at it directly. Analyzing or querying this data would be a manual process.

• With AWS Glue: You create a crawler in AWS Glue and point it to the location of your CSV file in S3. The crawler scans the file and automatically discovers the structure. Based on the scan, the crawler creates a schema in the Glue data catalog. This schema defines the table structure, including:

  • Column names (order_id, product_name, etc.)

  • Data types (integer, string, etc.)

Now you have a clear understanding of your data. You can see the table structure and know what kind of information each column holds. This makes it much easier to query the data & transform the data.

In essence, AWS Glue crawlers and the data catalog act as a bridge between your raw data in S3 and data processing or querying tools. They bring structure and understanding to your data, making it easier to work with and analyze.

You can also define dashboards using QuickSight, and in the QuickSight dashboard, you can reuse those Amazon Athena queries.

ETL and Data Warehouse

ETL stands for Extract, Transform, and Load. The data collection part remains similar.

You have clickstream data going through Amazon Kinesis Data Firehose, dumping into Amazon S3. AWS Glue can define metadata and read data based on that from one S3 bucket, transform those data (such as removing some columns or changing the values based on certain criteria), and then load that data into another S3 bucket. That's the ETL part.

The data warehousing part can be done using Amazon Redshift.

In this case, Amazon Redshift, using Redshift Spectrum, is directly running SQL queries from the Amazon S3 bucket. Note that in this case, the data is not being loaded into Redshift tables. It is being read directly from S3 using Redshift Spectrum.

The alternative is that AWS Glue, after doing the ETL, can also load the data into Redshift tables.

What's the difference between querying S3 data using Athena and loading data into Redshift and querying it?

Athena is primarily for ad-hoc (on the fly), serverless queries on S3 data, making it ideal for quick, flexible data analysis without infrastructure management. Redshift is designed for more complex, high-performance analytics on structured data, suitable for data warehousing and persistent, large-scale querying.

Now, let's take a quick detour into AWS Glue.

As we can see, this is a pretty powerful service. AWS Glue is a serverless data integration service that makes it easier to discover, prepare, and combine data for analytics, machine learning, and application development. Serverless means it will scale as needed, and you will only pay per use.

As we saw in the previous system design, Glue crawlers can run on data, discover the types of columns and types of data in it, and create metadata. Glue can visually create ETL flows and supports Python, Spark, and Scala. Glue DataBrew can be used to enrich, clean, and normalize data without writing code. Glue Elastic Views allows you to use familiar SQL to combine and replicate data across different data sources.

Unified Catalog across Multiple Data Stores

For this system design, we have the input data sources as S3, RDS, and a database running on Amazon EC2.

We have to run AWS Glue, which creates that Glue data catalog. You can even join this data catalog across different data sources and run SQL queries using Redshift or Athena. You can also run big data processing using Amazon EMR, reading data from all these different sources using the same Glue data catalog.

For dashboards and reporting, Amazon QuickSight is the service to do that, and QuickSight can utilize queries from Athena, Redshift, or EMR.

Big Data Analysis on Clickstream Data

To begin with let's take a look at Amazon EMR (Elastic MapReduce).

EMR is the managed big data platform from AWS. EMR runs open-source tools such as Apache Spark, Hive, HBase, Flink, Hudi, and Presto. EMR runs on EC2, EKS, or on-premises using EMR on Outposts. This takes away the management overhead of running your own big data cluster so that you can focus on the business problem. So whenever you need to do some big data processing, think of Amazon EMR.

One example of big data analysis on clickstream data could be the data getting dumped into Amazon S3 storage and then EMR reading from it.

However, there is an important consideration: Amazon EMR cannot directly read from S3 without metadata. AWS Glue must first crawl the data and create the Glue Data Catalog. Additionally, you can run machine learning algorithms using Amazon SageMaker, which can also read from the same Glue Data Catalog.

In Stream Querying & ETL

In a traditional ETL pipeline, data is processed in batches after it's been stored. However, some scenarios require Real-time querying and in-stream ETL**.**

What if your system design demands querying and transforming data as it arrives in the stream, rather than waiting for it to be dumped first?

Storing everything can be expensive and unnecessary. You might only need a specific subset of the data for further analysis (Selective data storage).

To achieve this we utilize streaming services like Kinesis Data Streams or Managed Streaming for Kafka to continuously collect data. We then employ Amazon Kinesis Data Analytics to process the incoming data stream in real-time using Apache Flink.

We leverage SQL queries, Java, Scala, or Python programs within Kinesis Data Analytics to analyze and transform the data stream. This allows for selective processing, focusing only on relevant data & send the transformed data to Amazon Kinesis Data Firehose, a service that acts as a delivery stream. Kinesis Data Firehose can be configured to selectively load the processed data into specific S3 buckets, ensuring only the essential information is stored. Now, the data stored in S3 can then be used for further processing with tools like AWS Glue, as previously discussed.

Data Lake

A Data Lake itself isn't a specific technology, but rather a design philosophy and approach to data management. It leverages a combination of existing services to achieve its core functionalities.

The most critical aspect to remember is that a Data Lake stores vast amounts of data. Typically, Amazon S3 is at the core of a Data Lake due to its scalability, cost-effectiveness, and multiple storage tiers. Implementing lifecycle policies allows for the migration of data between these tiers, further optimizing storage costs. This flexibility and efficiency make S3 an ideal foundation for a Data Lake, enabling organizations to handle large volumes of data efficiently and cost-effectively.

Authentication & Authorization

Authentication answers the question:

Do you have access to the application?

For example, when you log in to Tinder or Amazon, you provide a user ID and password, and if they match, you gain access.

Authorization determines:

What can you do within the application once authenticated?

For instance, can you create a Kubernetes cluster or delete a database?

This is dictated by the permissions assigned to your account.

System Design Overview

1. User and Identity Provider: Users provide their user ID and password. An identity provider (e.g., Active Directory, Auth0, AWS Cognito) stores these credentials. Upon successful authentication, the identity provider issues a token (JWT token).

2. API Gateway and Token Validation: The application sends the user ID and JWT token in the API call header. The API Gateway validates the JWT token, either by directly checking with the identity provider or through a lambda authorizer function. This validation ensures the token is still valid and hasn't expired.

3. Scopes and Policies: The identity provider can define scopes within the JWT token, indicating what actions the user can perform. Alternatively, a Lambda function (Lambda Authorizer) can handle this validation, returning policies dictating access permissions.

4. Token Caching: To optimize performance, JWT tokens are cached in the API Gateway for a predefined period. This caching reduces the need for repeated validation, improving response times for subsequent requests.

The AWS Well-Architected Framework helps cloud architects build secure, high-performing, resilient, and efficient infrastructure. It's based on six key areas called pillars:

1. Operational Excellence

Focus: How well you run and monitor your systems.

• Automate Everything: Use code to manage your systems and processes.

• Small Changes: Make frequent, small updates to minimize risk.

• Regular Updates: Continuously improve your procedures.

• Plan for Failure: Think ahead and prepare for things going wrong.

• Learn from mistakes: Use past failures to improve your processes.

• Use Managed Services: Let AWS handle some tasks to reduce your workload.

2. Security

Focus: Protecting your systems and data.

• Strong Access Controls: Only let the right people access your resources.

• Track Everything: Monitor and record all actions and changes.

• Multi-layer Security: Use multiple security measures at different levels.

• Automate Security: Use automated tools to enhance security.

• Protect Data: Encrypt data both when it's stored and when it's moving.

• Limit Direct Access: Reduce the need for manual data handling.

• Prepare for Incidents: Have a plan for dealing with security issues.

3. Reliability

Focus: Ensuring your system works correctly and consistently.

• Automatic Recovery: Set up systems to fix themselves when problems occur.

• Test Failures: Regularly test how your system handles failures.

• Scale Horizontally: Use multiple smaller resources instead of one big one.

• No Capacity Guessing: Scale resources based on actual demand.

• Automate Changes: Use automation to manage changes to your infrastructure.

4. Performance Efficiency

Focus: Using resources efficiently.

• Leverage Advanced Tech: Use AWS's services to handle complex tasks.

• Global Deployment: Deploy your systems worldwide for better performance.

• Serverless Options: Use serverless technologies to reduce management overhead.

• Frequent Experimentation: Regularly test different configurations to improve performance.

• Tech Matching: Choose the right technology for your specific needs.

5. Cost Optimization

Focus: Reducing unnecessary expenses.

• Pay-as-You-Go: Only pay for what you use.

• Measure Efficiency: Track the cost vs. benefit of your services.

• Avoid Heavy Lifting: Let AWS manage infrastructure to save costs.

6. Sustainability

Focus: Building environmentally friendly systems.

• Efficient Infrastructure: Use energy-efficient data centers.

• Reduce Carbon Footprint: Implement strategies to lower carbon emissions.

• Innovative Data Centers: Invest in new technologies for better efficiency.

• Data Management: Optimize storage to reduce waste.

• Transparency: Report on environmental impact.

• Education: Learn and promote sustainable practices.

Three-tier Architecture

It has 3 distinct layers/architecture.

1. Presentation tier (Frontend)

2. Logic tier (Backend)

3. Data tier (Database)

A brief explanation of the three distinct layers of architecture:

1. Presentation Tier (Frontend):

   • Description: This layer is responsible for the user interface and user experience. It’s what the end-user interacts with directly.

   • Components: Web pages, mobile apps, desktop applications, and other client interfaces.

   • Technologies: HTML, CSS, JavaScript, React, Angular, Flutter.

2. Logic Tier (Backend):

   • Description: This layer contains the business logic and processes user inputs from the presentation tier. It’s responsible for data processing, application logic, and communications between the frontend and the database.

   • Components: Application servers, APIs, microservices.

   • Technologies: Java, Python, Node.js, Ruby on Rails, Django.

3. Data Tier (Database):

   • Description: This layer is responsible for storing, retrieving, and managing data. It serves as the central repository where data is kept.

   • Components: Databases, data storage systems.

   • Technologies: SQL databases (MySQL, PostgreSQL), NoSQL databases (MongoDB, Cassandra).

These three layers work together to create a well-organized, scalable, and maintainable application architecture.

Securing different tiers

1. Presentation tier:

   • Use HTTPS: Encrypt data transmitted between the client & the server to prevent eavesdropping & tampering.

   • Input validation: Validate all user inputs or client-side (using JS) & server-side to prervent injection attacks.

2. Application tier:

   • Authentication & Authorization: Implement strong authentication mechanisms like OAuth, JWT, or SAML to verify user identity. Use role-based access control (RBAC) or attribute-based access control (ABAC) to restrict access to resources based on user roles.

   • Secure session Management: Use secure session management practices such as using unique session identifiers, setting expiration times & using HTTPS for sesssion data transmission.

3. Data tier:

   • Data Encryption: Encrypt sensitive data at rest (using database encryption) and in transit (using TLS/SSL) to protect it from unauthorized access.

   • Database security best practices: Follow database security best practices such as keeping db s/w update, applying patches and regularly auditing db access logs.

Three tier architecture on AWS

Can you guess single point of failures in the above architecture?

• Yes there are multiple points. Web server, App server and the DB.

• About the load balancer, they're actually highly available, they span across 3 AZs under the hood. It just appears like a single aspect.

Then, how to make them highly available?

• Deploying into multi AZs.

For network security,

• We can deploy ELB in public subnet, while keeping web & app server in private subnets.

• We can use WAF with ELB.

• Using NACLs and SGs to enhance the security for instances.

What about database?

• We can use either SQL or NoSQL database.

• Prefer AWS native databases. Say, for Aurora, it has Multi AZ, Global database replication features just to mention few.

• For database optimization, we can use read replica, caching layer & query tuning.

A well architected 3 tiered architecture example:

Three-Tier Architecture with Serverless

• S3 is used for serving static contents while any interactive contents are handle using API through API Gateway. Using lambda as compute and Amazon Aurora as database.

• Amazon S3 (Simple Storage Service) is designed for high durability, availability, and scalability. It can handle massive amounts of data and serves static content such as images, videos, CSS, and JavaScript files. S3 automatically scales to meet the needs of your application, and its integration with Amazon CloudFront (a Content Delivery Network) enhances performance and availability by distributing content closer to users globally.

• Amazon API Gateway serves as the front door for applications to access data, business logic, or functionality from backend services. It handles thousands of concurrent API calls and scales seamlessly with your application’s traffic.

  AWS Lambda is a serverless compute service that automatically scales up and down based on the number of incoming requests. Each invocation of a Lambda function is run in a separate execution environment, allowing for high concurrency without the need to manage servers.

• Amazon Aurora is a MySQL and PostgreSQL-compatible relational database built for the cloud, combining the performance and availability of high-end commercial databases with the simplicity and cost-effectiveness of open-source databases.

Three-Tier Architecture with K8's

Description:

• Web/App running in a container within a pod.

• Pods can run either in EC2 worker node under EKS service or Fargate.

• We make this scalable using K8's scaling constructs which is Replica Set, HPA (horizontal pod autoscaler) & Cluster Autoscaler.

• The load balancer fronting the webserver will be ALB ingress while in the back-end, the ELB could be another ingress or a K8's service.

Encryption is a process that converts data into a coded form to prevent unauthorized access. It uses algorithms and keys to transform readable data (plaintext) into an unreadable format (ciphertext). Only authorized parties with the correct decryption key can revert the ciphertext back to plaintext.

Encryption at Rest

Encryption at rest refers to encrypting data stored on a disk or other storage media. This protects the data from unauthorized access when it is not being transmitted or actively used. Common use cases include encrypting files on hard drives, databases, or cloud storage.

Benefits:

• Protects data from physical theft of devices

• Safeguards against unauthorized access in case of a data breach

• Ensures compliance with data protection regulations

Encryption in Transit

Encryption in transit protects data as it moves from one location to another, such as over the internet or a private network. It ensures that data remains confidential and integral while being transmitted between clients, servers, and other systems.

Benefits:

• Prevents eavesdropping and man-in-the-middle attacks

• Ensures data integrity during transmission

• Secures sensitive information exchanged over networks

Public & Private Key

Public Key: This key is intended to be shared publicly and is used for encryption. It allows anyone to encrypt a message or data that can only be decrypted by the corresponding private key. When you want to send a secure message to someone, you use their public key to encrypt it. However, even if someone knows your public key, they cannot decrypt messages encrypted with it.

Private Key: This key must be kept secret and is used for decryption. It allows you to decrypt messages or data that have been encrypted with your public key. Since only you should have access to your private key, only you can decrypt messages encrypted with your public key.

Types of Encryption

1. Symmetric Encryption:

• Uses a single key for both encryption and decryption

• Faster and more efficient for large data volumes

• Examples: AES (Advanced Encryption Standard), DES (Data Encryption Standard)

2. Asymmetric Encryption:

• Uses a pair of keys: a public key for encryption and a private key for decryption

• Provides strong security for data transmission

• Examples: RSA (Rivest-Shamir-Adleman), ECC (Elliptic Curve Cryptography)

3. Hybrid Encryption:

• Combines symmetric and asymmetric encryption

• Often used in protocols like SSL/TLS where a symmetric key is exchanged using asymmetric encryption

Client & Server Side Encryption

Client-Side Encryption

Client-side encryption involves encrypting data on the client’s side before it is sent to a server or cloud storage. The client retains control of the encryption keys, providing a higher level of security.

Benefits:

• Data is protected before it leaves the client’s environment

• Client retains control over encryption keys

• Reduces the risk of data exposure even if the server is compromised

Challenges:

• Managing and securing encryption keys can be complex

• Performance overhead on the client device

Server-Side Encryption

Server-side encryption occurs when data is encrypted after it is sent to the server or storage system. The server handles the encryption and decryption processes, often using managed encryption services provided by cloud providers.

Benefits:

• Simplifies key management for clients

• Integrates seamlessly with cloud storage services

• Provides built-in encryption options with minimal configuration

Challenges:

• Relies on the server’s security measures

• Potential exposure if the server is compromised

Digital Signature

A digital signature is a cryptographic technique that ensures the authenticity and integrity of a message, document, or software. It provides a way to verify that a message was created by a known sender (authentication) and that it was not altered in transit (integrity).

Applications: Email verification, software distribution, financial transactions.

How Does it Work?

1. Hashing: The message or document is processed through a hash function to create a unique fixed-size string of characters, known as the hash value or message digest.

2. Encryption: The hash value is encrypted using the sender’s private key, creating the digital signature. This private key is part of a public/private key pair used in asymmetric encryption.

3. Verification: The recipient decrypts the digital signature using the sender’s public key. This produces the original hash value. The recipient also hashes t

   he received the message or document and compares the two hash values. If they match, the message is verified as authentic and unaltered.

Applications: Email verification, software distribution, financial transactions.

Steganography

Steganography is the practice of hiding information within another file, message, image, or video. Unlike encryption, which makes data unreadable without a key, steganography hides the existence of the message itself.

How Does it Work?

1. Embedding Data: The hidden message is embedded into a carrier medium (e.g., an image, audio, or video file) by slightly altering the least significant bits of the carrier file. These alterations are usually imperceptible to the human eye or ear.

2. Extraction: The recipient uses a predefined algorithm or key to extract the hidden message from the carrier medium.

Applications: Covert communication, digital watermarking, secure information hiding.

Example: Imagine you have a digital photo with a clear blue sky. You can use steganography software to slightly modify the color values of some pixels in the sky. These changes might be so subtle that they're undetectable to the naked eye, but the software can interpret them as a hidden message. Someone who knows the secret technique and has the right software can then extract the hidden message from the image.

Steganography doesn't necessarily make a message secure. If someone suspects hidden information, they might use steganalysis techniques to try to uncover it.

Envelope Encryption

Envelope encryption is a technique used to improve the performance and security of encrypting large amounts of data. It combines symmetric and asymmetric encryption to efficiently protect data while maintaining strong security.

How Envelope Encryption Works

1. Data Encryption Key (DEK): A DEK is a symmetric key used to encrypt and decrypt actual data. It can be generated by AWS KMS or by the user and then encrypted using a KEK/CMK. Symmetric encryption is chosen for its speed and efficiency, making it suitable for encrypting large data sets.

2. Encrypt Data: The data is encrypted using the DEK. This step produces the ciphertext of the data.

3. Key Encryption Key (KEK): The DEK itself is then encrypted using a different key, called the Key Encryption Key (KEK). KEKs are often asymmetric keys, providing strong security for the DEK.

4. Store Encrypted DEK: The encrypted DEK (encrypted with the KEK) is stored alongside the encrypted data. This allows the DEK to be retrieved and decrypted only by authorized entities with access to the KEK.

5. Decryption Process: To decrypt the data, the encrypted DEK is first decrypted using the KEK to obtain the original DEK. The DEK is then used to decrypt the data, returning it to plaintext.

AWS Key Management Service (KMS)

AWS Key Management Service (KMS) is a managed service that makes it easy to create and control cryptographic keys used for encryption across AWS services and applications. KMS integrates with various AWS services to provide seamless encryption and key management.

Key Features:

• Centralized Key Management: Create, rotate, and manage keys from a central location.

• Integration: Easily integrates with AWS services like S3, EBS, RDS, Lambda, and more for automatic encryption.

• Fine-Grained Access Control: Define and enforce policies for key usage and access through AWS Identity and Access Management (IAM).

• Auditing: Provides detailed logging of all key usage through AWS CloudTrail, helping with compliance and auditing.

• High Availability: Ensures keys are available and secure, leveraging AWS's robust infrastructure.

Customer Master Key (CMK)

A CMK is a key stored in AWS KMS that is used to encrypt and decrypt data encryption keys (DEKs). It serves as a key encryption key (KEK).

Managing Customer Master Keys (CMKs) in AWS KMS

When using AWS Key Management Service (KMS) to secure your data, you have the option to manage your Customer Master Keys (CMKs) yourself or to let AWS KMS manage them for you. Both approaches come with their own benefits, challenges, and considerations.

AWS-Managed CMKs

AWS KMS provides AWS-managed CMKs, which are automatically created, rotated, and managed by AWS on your behalf. These keys are used by various AWS services to encrypt your data without requiring manual intervention.

• Pros: Simplified management, seamless integration, cost-effective.

• Cons: Limited control and customization.

Customer-Managed CMKs

Customer-managed CMKs are created, managed, and controlled by you. You have full control over the key lifecycle, including creation, rotation, and deletion, as well as the policies and permissions associated with the keys.

• Pros: Full control, customization, compliance support.

• Cons: Increased complexity, administrative responsibility, higher cost.

In software development, a monolithic architecture is an approach where all the components of an application are built as a single, indivisible unit. This means that the application's codebase is typically a single codebase, and all the functionality is developed, deployed, and scaled as a single unit.

Pros of Monolithic Architecture:

• Simplicity: Easy to develop and deploy, with all code in one place.

• Easy debugging: Easier to track down issues since all code is centralized.

• Performance: Can offer good performance when optimized due to minimal communication overhead.

Cons of Monolithic Architecture:

• Scalability: Difficult to scale as the application grows, requiring entire application replication.

• Agility: Slow to adapt to changes, as a small change might require a full redeployment.

• Single point of failure: If one part crashes, the entire application can be affected.

Can you use API with monolith?

• Yes, you can. API has nothing to do with monolith or microservices.

  

  Monolith can be fronted by either an API Gateway or a Load balancer. And, when different URLs or paths hit API/LB, they forward all the calls to the same monolithic. This monolithic will have entry functions or entry paragraph which will check the URL/paths and invoke different functions within the monolithic and execute the logic accordingly.

Issue of Scaling

Say, the traffic for store/get component increases, then we can see CPU reaching its limit.

Now, instead of scaling just the store/get component, you've to scale the entire monolith.

The microservices

Different components will have separate codebase and with different virtual machines.

Now, say the traffic for store/get component increases then, instead of increasing all VMs, only the required VM is scaled.

Now, since the backend servers operate independently and can be managed by separate teams, these microservices can be developed in various programming languages based on the needs.

Characteristics of microservice architectures

• Independent: Microservices are self-contained units with well-defined functionalities.

• Scaling: Microservices can be scaled independently based on their specific needs.

• Governance: Each microservice can have its governance model, such as API contracts and versioning. This provides better control and flexibility for individual services.

• Deployment: Microservices can be independently deployed using technologies like containers or virtual machines.

• Testing: Smaller codebases in each service make testing quicker and more focused.

• Functionality: Microservices are built around specific business capabilities, promoting modularity, reusability, and easier maintenance of the overall application.

Important: Not required to follow every characteristic.

Deploying microservices in AWS

An EC2-based microservice architecture. Depending upon the requirements, we're using different instance families.

A serverless microservice architecture using lambda. Lambda handles the autoscaling automatically. They can also be fronted by ELB/API Gateway.

Now, microservice architecture using containerized application and that run within Elastic Kubernetes Service (EKS) or Elastic Container Service (ECS), Here, as well they can be fronted by ELB/API Gateway.

For Kubernetes, each microservice is fronted by different services as ServiceA, ServiceB and ServiceC and the code for each microservice will be running in a container and this container will run in a pod. And all these are fronted by a single ingress, which will be an ALB.

Since each microservice is independent of each other and can be even coded in different programming languages, you can mix and match all these AWS services.

For example:

• store/get can run on a pod where the backend of the microservice is dockerized into a container.

• store/post can be hosted on Amazon EC2.

• store/delete can run on AWS Lambda.

So, depending on the requirements, you can use different services.

That is the beauty of microarchitecture.

Load Balancer

Before load balancers, web applications struggled with uneven traffic distribution, limited availability, and scalability issues. Single servers often became overwhelmed, leading to slow response times and downtime. Adding capacity was manual and inefficient. Security was also a concern.

Load balancers now distribute traffic, improve availability by rerouting from failed servers, and enhance scalability by adding or removing servers as needed. They also offload SSL/TLS, ensure session persistence, and provide an additional security layer.

API & API Gateway

What is an API?

APIs are interfaces that define how software components should interact. They specify the methods and data formats that applications can use to request and exchange information. APIs are typically used to define how different software systems or components should communicate with each other, providing a higher-level abstraction that simplifies the development of software applications.

Imagine you're at a restaurant. You (the user) tell the waiter (the API) what you want (make a request), the waiter relays your order to the kitchen (the other application), the kitchen prepares your food (processes the request), and the waiter brings it back to you (delivers the response).

APIs are everywhere in the modern world and are what make many of the things we take for granted possible. For instance, when you use a rideshare app, the app uses an API to connect to the rideshare company's system and request a ride for you. Similarly, when you use a weather app, the app uses an API to get weather data from a weather service.

What is an API Gateway?

An API Gateway is a server that acts as an API front-end, receiving API requests, enforcing throttling and security policies, passing requests to the back-end service, and then passing the response back to the requester. The API Gateway encapsulates the internal system architecture and provides an API that is tailored to the client's needs.

Functions of API Gateway

• Single Entry Point: An API gateway provides a unified interface for client applications to access all backend APIs. This eliminates the need for client applications to interact with individual backend URLs, streamlining communication and reducing complexity.

• Traffic Management: API gateways efficiently route incoming API requests to the appropriate backend services. They can distribute traffic load across multiple backend instances to ensure scalability and prevent overloading any single service.

• Security: API gateways enforce security measures to protect backend services from unauthorized access. They can implement authentication and authorization mechanisms to restrict access based on user roles or permissions.

• Monitoring and Analytics: API gateways track API usage patterns and provide valuable insights. You can monitor metrics like request rates, latencies, and error codes to identify potential issues and optimize API performance.

• Transformation: API gateways can transform data between formats to ensure compatibility between client applications and backend services. This eliminates the need for developers to write separate code for data formatting on each side.

• Versioning: API gateways can manage different versions of APIs simultaneously. This allows developers to introduce new features or make changes to existing APIs without disrupting existing client applications that are using older versions.

• Caching: API gateways can cache frequently accessed data to improve performance and reduce load on backend services. This can be especially beneficial for static data or data that doesn't change frequently.

• Policy Enforcement: API gateways can enforce policies that govern how APIs are used. These policies can include rate limiting, throttling, and quota restrictions to prevent abuse and ensure fair use of resources.

• Integration with Diverse Backends: API gateways can connect to various backend services, regardless of their programming language or technology stack. This flexibility allows you to integrate different backend systems and expose them as a unified set of APIs.

Application Load Balancer & Amazon API Gateway

Application Load Balancer

• Automatically distributes incoming traffic across backend targets

• Layer 7 load balancer

• Infrastructure managed by AWS, highly available, elastic

API Gateway

• Fully managed and serverless API service from AWS

• Automatically scales up and down

• Infrastructure managed by AWS, highly available, elastic

What is the difference?

Vertical Scaling & Horizontal Scaling

Vertical Scaling (Scaling Up): This method involves boosting the capabilities of a single machine by adding more resources like CPU, memory, storage, or a faster network card. Imagine upgrading your computer's RAM to handle more complex software.

Horizontal Scaling (Scaling Out): Here, instead of beefing up a single machine, you add more machines to distribute the workload. This is like adding more lanes to a highway to handle increased traffic. New machines are brought into the system and configured to work together as a team.

Different types of scaling for different AWS services

1. EC2 instance

   • Vertical Scaling: ALB ---> EC2. We can do vertical scaling, provisioning bigger and more powerful instance when the traffic goes up.

     

     

   • Horizontal Scaling: ALB --> ASGs. Depending on our requirements, ASGs will either add up or remove instances.

     

2. Lambda

   • Lambda can only scale horizontally only.

   • For each traffic, a new instance of your lambda scales up.

     

3. Container using EC2

   • Your container is running inside a pod.

     

     A single pod can handle multiple traffic connections. But with each traffic, the CPU & memory utilization will go up. And at certain point, using Horizontal Pod Autoscaler (HPA), it'll scale. During horizontal scaling of container, it doesn't create another container in the same pod. The smallest unit of scaling is pod. Thus, it'll create another pod within that EC2 node.

     

     But what will happen when the EC2 is filled up?

     Each EC2, depending upon the family type, can only have a certain number of pods. As you put more pods in the same EC2, the CPU utilization of this EC2 also goes up. So, at a certain point, it is required to scale the EC2 node, and we call it Cluster Autoscaler.

     

4. Container using Fargate

   • Underlying infrastructure node is managed by AWS.

   • So, we're starting with one pod, and as the traffic goes up, the no. of pods increases. Under the hood, all infrastructure is managed.

Synchronous & Asynchronous (Event Driven) Architectures

Synchronous Architecture

1. In a synchronous architecture, operations are executed one after the other, in a sequential manner.

2. When a request is made to a synchronous system, the system blocks further execution until the operation is completed and a response is returned.

3. Synchronous systems are easier to understand and debug, as the flow of operations is straightforward.

   

4. However, they can suffer from scalability issues, especially in scenarios where there are many concurrent requests, as each request must wait for the previous one to complete.

   

   At certain point, following condition may occur.

   

   Challenges of Synchronous Architecture:

   • All components must scale together,

   • Consumer needs to resend transaction for re-processing incase of failure,

   • Expensive approach.

   • Error handling in synchronous systems can be more complex, as failures at any step can result in the entire operation failing.

Asynchronous (Event-Driven) Architecture:

• In an asynchronous architecture, operations are not executed immediately. Instead, they are queued up and processed later.

• When a request is made to an asynchronous system, the system does not block further execution. It accepts the request, queues it up, and continues with other operations.

  

• Asynchronous systems are more scalable, as they can handle a large number of concurrent requests more efficiently than synchronous systems.

  

• In this event-driven architecture, the key component for handling busy traffic is SQS (Simple Queue Service). SQS acts as a buffer, absorbing spikes in incoming requests and ensuring that no data is lost even during high traffic periods. Its ability to decouple the components and store messages temporarily allows the system to scale efficiently, handling varying workloads without overwhelming downstream services.

  Advantages of event driven architectures

  • Each component can scale independently,

  • Automatically retries processing a failed event,

  • Cost effective than synchronous architecture.

Queue & Pub-Sub Model

Queue Model

A queue model mimics a waiting line, helping us understand how things are processed in a system. Producers, like people joining a line, add items to a queue. Consumers, the servers at the front, then take items and process them at their own pace. The queue acts as a buffer, holding items until a consumer is free.

This model considers factors like who gets served next (first-come-first-served?) and how many can wait in line. By analyzing these elements, we can predict waiting times, queue lengths, and overall system efficiency, allowing for smoother handling of incoming requests or tasks.

Example include: Amazon SQS

Pub-Sub Model

In contrast to the queue model's orderly line, the pub-sub model functions more like a public announcement system. Here, publishers, instead of sending messages directly, broadcast them to channels or topics. Subscribers interested in those topics can listen in and receive relevant messages. This asynchronous communication keeps publishers and subscribers independent.

Example: Amazon SNS, Amazon EventBridge.

Messaging & Streaming

Messaging

A communication method where information is exchanged in individual, self-contained messages. Each message represents a complete unit of data, similar to a letter in traditional mail.

• Use Cases: Perfect for situations where decoupling applications is important. The sender doesn't need to wait for the receiver to be available, and vice versa. Messaging queues are also great for ensuring messages are delivered reliably, even if there are failures. Examples include sending notifications, processing payments, or updating databases asynchronously.

• Benefits: Decoupling, reliability, asynchronous processing.

• Example: Amazon SQS.

Streaming

A continuous flow of data delivered in a sequence. Unlike messaging, a single chunk (stream) doesn't provide the full context. To understand the bigger picture, you need to analyze the entire stream or a significant portion of it.

• Use Cases: Ideal for scenarios where low latency and real-time processing are essential. Examples include analyzing sensor data, processing financial transactions, or building chat applications.

• Benefits: Real-time processing, low latency, large data volumes.

• Example: Kinesis Data Stream.

SQL and NoSQL Database

Understanding CAP theorem

1. Consistency (Every read operation gets the latest data or an error)

2. Availability (Every request receives a response, even if it's not the most recent data)

3. Partition Tolerance (The system continues to function despite network partitions): This property ensures the system can handle disruptions in communication between nodes. Network partitions can occur due to hardware failures, outages, or network congestion. Partition tolerance allows the system to continue operating even during such events.

NOTE: There is no one-size-fits-for-all. Amazon Aurora Serverless can support horizontal scaling & DDB Transaction follows ACID properties.

SQL Database: Amazon RDS, Amazon Aurora

NoSQL Database: DDB, Amazon DocumentDB ( with MongoDB compatibility), Amazon Managed Apache Cassandara Service.

Do not use a cannon to kill the mosquito. (choosing the appropirate database as requiements)

WebSockets

WebSockets are a powerful communication protocol that enables real-time, full-duplex communication between a client (like a web browser) and a server. Unlike traditional HTTP requests that are typically one-way (client requests, server responds), WebSockets allow for ongoing, bi-directional communication, making them ideal for applications requiring real-time updates, such as chat applications, live sports scores, collaborative editing tools, and financial trading platforms.

How does it begin?

The WebSocket protocol begins with a process known as the WebSocket handshake, which is based on HTTP. Here's a basic overview of how it starts:

1. Client Request: The process begins when a client (usually a web browser) sends a regular HTTP request to the server, indicating its desire to upgrade the connection to a WebSocket connection. This request includes a special header called Upgrade with the value websocket, along with other required headers.

2. Server Response: Upon receiving the client's request, the server checks if it supports the WebSocket protocol. If it does, the server responds with a 101 status code (Switching Protocols), indicating that the connection is being upgraded to a WebSocket connection. The response also includes a special header called Upgrade with the value websocket, confirming the upgrade.

3. Establishment of WebSocket Connection: Once the handshake is complete and the connection is upgraded, the WebSocket connection is established. From this point onward, both the client and server can send messages to each other in a full-duplex manner.

4. Communication: With the WebSocket connection established, the client and server can now exchange data using the WebSocket protocol. Messages can be sent and received in real-time, enabling interactive and responsive web applications.

5. Closing the Connection: When either the client or server decides to close the WebSocket connection, they send a special message indicating their intention to close. Upon receiving this message, the receiving party also sends a closing message, and once both sides have acknowledged the closure, the connection is closed.

Caching

Caching is the process of storing copies of files or data in a cache, which is a temporary storage area, so that they can be accessed more quickly when requested. Caching is used to improve the performance and efficiency of systems by reducing the need to access the original, often slower, data source.

• Reduces load on servers and network bandwidth usage.

• Enables faster response times and better user experience.

• Can be implemented at various levels including browser, server-side, and Content Delivery Networks (CDNs).

A cache hit occurs when a requested piece of content is found in the cache.

A cache miss occurs when a requested piece of content is not found in the cache.

How does cache get populated?

Caches get populated through a process called caching. When data is requested, the system checks if it's in the cache. If not, it fetches the data from the original source and stores it in the cache for future requests.

The process of adding data to the cache is known as cache population. This can happen in several ways:

• On-Demand: Data is added to the cache only when it's requested and not already present. This is known as lazy loading.

  

• Preloading: Data is added to the cache in advance, typically during system startup or during periods of low traffic, to ensure that frequently accessed data is already available in the cache.

• Write-Through: Data is simultaneously written to both the cache and the original data source, ensuring that the cache is always up-to-date.

  

• Write-Behind (Write-Back): Data is initially written only to the cache and then periodically flushed to the original data source in the background.

How does cache get deleted?

Caches can be deleted or cleared in several ways:

1. Manual Clearing: Users or administrators can manually clear the cache. This is often done through settings or options in the application or system.

2. Time To Live (TTL): Caches can be configured with TTL, where items are automatically removed from the cache after a certain period of time. This helps ensure that the cache does not contain stale data.

3. Cache Invalidation: When data in the original data source is modified or deleted, the corresponding data in the cache needs to be updated or removed to prevent stale data. This process is known as cache invalidation.

4. Eviction Policies: Caches can also use eviction policies to remove items from the cache when it reaches a certain size or when memory is needed for other purposes. Common eviction strategies include Least Recently Used (LRU), Most Recently Used (MRU), or First-In-First-Out (FIFO).

5. Programmatic Clearing: Applications can programmatically clear or remove items from the cache based on certain criteria or events, such as when a user logs out or when a session expires.

Caching is not restricted to database only. You can enable caching in different parts of your system.

Which caching service to use & when?

Use managed caching of the service. For example, using caching feature of API Gateway in case of API Gateway.

If service doesn't provide caching then use cache database.

• Cache databases like Redis and Memcached are suitable when you need more than just caching and require features like persistence, data structures, or advanced caching strategies. They're also suitable for applications that require high-performance caching and can benefit from the flexibility and scalability of cache databases.

High Availability & Fault Tolerant

High Availability

High Availability (HA) refers to the ability of a system or service to remain operational and accessible for a high percentage of time, typically 99.999% (often referred to as "five nines" availability). Achieving high availability involves implementing redundant components and failover mechanisms to ensure that if one component fails, another can take over seamlessly, minimizing downtime and ensuring continuous operation.

To make system highly available we've to identify single point of failure

• Servers running your applications

• Database

• Load balancer

• Analyze each component and validate single point of failure

Achieving High Availability on the Cloud

Deploying in more than one AZ makes our system more available.

This case for containers ensuring high availabilty.

Fault Tolerant

Fault tolerance refers to a system's ability to continue operating properly in the event of the failure of some of its components. It is a key aspect of high availability and is achieved through redundancy and error detection mechanisms.

It is more expensive approach than HA. For example, airplanes have redundant systems for critical functions such as engine power, navigation, and control surfaces.

Centralized & Distributed System

Centralized System

A centralized system is a computing architecture where most or all of the processing, storage, and management functions are performed by a single central node or entity. In such a system, all user requests, data, and resources typically flow through this central point.

Here are some key characteristics and considerations of centralized systems:

1. Centralized Control: One central entity has control over the entire system, making decisions and managing resources. This centralization can simplify management but can also become a bottleneck if the central node fails or is overloaded.

2. Single Point of Failure: Centralized systems are vulnerable to single points of failure. If the central node fails, the entire system can become inaccessible or non-functional. Redundancy and fault tolerance mechanisms are essential to mitigate this risk.

3. Scalability Challenges: Scaling a centralized system can be challenging. As the system grows, the central node may become a bottleneck, limiting the system's ability to handle increased traffic or data volume.

4. Data Consistency: Ensuring data consistency can be more straightforward in a centralized system compared to distributed systems. However, it also means that data access and processing may be slower if everything needs to go through the central node.

5. Resource Utilization: Centralized systems may struggle with efficient resource utilization, as resources are typically provisioned based on peak loads, leading to underutilization during off-peak times.

6. Security Concerns: Centralized systems can be more vulnerable to security threats, as compromising the central node can potentially expose the entire system. Strong security measures are crucial to protect against such threats.

7. Examples: Traditional client-server architectures, where a central server handles requests and manages resources, are examples of centralized systems. Many legacy systems and some modern applications still use centralized architectures.

Distributed System

A distributed system is a computing architecture that consists of multiple interconnected nodes that work together to achieve a common goal. In a distributed system, tasks are divided among the nodes, and coordination is achieved through message passing or shared storage.

Here are some key characteristics and considerations of distributed systems:

1. Decentralized Control: Unlike centralized systems, distributed systems have no single point of control. Each node in the system may make independent decisions based on local information, leading to more flexible and scalable architectures.

2. Scalability: Distributed systems are inherently more scalable than centralized systems. Adding more nodes to the system can increase its capacity to handle larger workloads, making it easier to accommodate growth.

3. Fault Tolerance: Distributed systems are designed to be fault-tolerant. If one node fails, the system can continue to operate using other nodes. Redundancy and replication are often used to ensure data availability and system reliability.

4. Data Consistency: Ensuring data consistency in a distributed system can be challenging, especially in the presence of network partitions or node failures. Various consistency models, such as eventual consistency or strong consistency, can be used based on the application's requirements.

5. Resource Utilization: Distributed systems can achieve better resource utilization by dynamically allocating resources based on demand. This can lead to more efficient use of hardware resources compared to centralized systems.

6. Concurrency and Parallelism: Distributed systems can take advantage of parallelism and concurrency to improve performance. Tasks can be divided among nodes, and multiple tasks can be executed simultaneously, leading to faster processing times.

7. Security Concerns: Distributed systems face unique security challenges, such as ensuring data privacy and integrity across multiple nodes. Encryption, authentication, and access control mechanisms are essential to protect against security threats.

8. Examples: Modern web applications, cloud computing platforms, and blockchain networks are examples of distributed systems. These systems leverage the benefits of distributed architectures to provide scalable, fault-tolerant services to users.

Hashing

Hashing is a process used in computer science to convert input data (often of arbitrary size) into a fixed-size value, typically a string of characters. The output of a hash function, known as a hash value or hash code, is unique to the input data, meaning that even a small change in the input will produce a significantly different hash value.

How is hashing applied in system design?

Hashing is applied in system design in various ways to improve performance, security, and efficiency. Here are some common applications of hashing in system design:

1. Hash Tables: Hash tables are a fundamental data structure that uses hashing to map keys to their corresponding values. They are used to implement associative arrays, databases, caches, and symbol tables. Hash tables provide fast insertion, deletion, and lookup operations, making them ideal for scenarios where quick data access is essential.

   

2. Caching: Hashing is used in caching mechanisms to quickly retrieve cached data. Each piece of data is hashed to determine its storage location in the cache. When retrieving data, the system computes the hash of the requested data and checks the cache for a matching hash. If found, the cached data is returned, avoiding the need to perform expensive computations or database queries.

3. Password Storage: Hashing is used to securely store passwords. Instead of storing passwords in plain text, systems hash the passwords using a cryptographic hash function before storing them. When a user attempts to log in, the system hashes the provided password and compares it to the stored hash. This ensures that even if the password database is compromised, the actual passwords remain protected (as only hash value of the password is stored).

4. Data Integrity Checking: Hashing is used to verify the integrity of data. By hashing data before transmission or storage and comparing the hash values before and after, systems can ensure that the data has not been tampered with.

5. Load Balancing: Hashing is used in load balancing algorithms to distribute incoming requests among multiple servers. Each request is hashed to determine which server should handle it, ensuring that the load is evenly distributed across the servers.

Use Case: When you create a DynamoDB table, you specify a primary key, which consists of a partition key and an optional sort key. The partition key is used to determine the partition in which the item will be stored.

DynamoDB uses a hashing algorithm on the partition key's value to determine the partition where the item will be stored. This ensures that items with the same partition key are stored together in the same partition, allowing for efficient retrieval of data using the partition key.

Challenges of hashing

1. Collision: Hash functions can produce the same hash value for different inputs, leading to a collision.

2. Hashing Speed: Generating a hash value can be computationally intensive, especially for large datasets or complex hash functions.

3. Hash Function Updates: Over time, vulnerabilities may be discovered in hash functions, necessitating updates or changes. Updating hash functions in a system can be complex and may require migrating data to a new hash function.

Suppose, we've 4 servers s0 to s3. Now we have hash function that will do function: given_value%no._of_servers.

And we see that the requests 1 and requests 2s are sent to the s2 and s0 servers respectively.

Now, say one server s3 went down,

Then, that ends up re-mapping the entire thing because now, requests are going to different servers. All the existing mappings will be lost..

So, generally when a user comes to a particular server, some of the state informations are saved on the server. Since everything gets remapped with this kind of traditional hashing, the vacant server won't be able to easily recover the state for the same user request.

How to solve the issue?

We have other method called Consistent Hashing.

Consistent Hashing

Consistent hashing is a technique used in distributed systems to map resource requestors (like users or programs) and server nodes onto a virtual ring structure. This ring has an infinite number of points, and server nodes are placed at random locations on the ring. Requests are also placed on the ring using the same hash function.

Consistent hashing is used to keep the hash table independent of the number of servers available to minimize key relocation when changes of scale occur.

In consistent hashing, the decision about which request will be served by which server node is based on the location of the request and the server nodes on the ring. Here's how it works:

1. Mapping Requests: Each request (or key) is hashed to a value that corresponds to a point on the ring. This determines the location of the request on the ring.

2. Locating Server Nodes: Server nodes are also hashed to points on the ring using the same hash function. Each server node covers a range of points on the ring, from its own point up to (but not including) the point of the next server node in clockwise order.

3. Routing Requests: When a request is made, it is hashed to a point on the ring. The request is then routed to the server node whose range includes this point. This is done by moving clockwise around the ring until the first server node is encountered whose point is greater than or equal to the point of the request. If no such server node is found (i.e., the request's point is higher than the highest addressed node), the request is served by the server node with the lowest address, as the traversal through the ring goes in a circle.

This method ensures that each request is consistently mapped to the same server node, even as nodes are added or removed from the system, minimizing the need for reorganization of data and ensuring efficient load balancing across the nodes.

What happens when a server node fails?

When a server node fails, only the range of keys that were assigned to that node need to be remapped. This remapping affects only a portion of the keys, relative to the total number of keys in the system, which is a significant advantage compared to traditional hashing techniques where a change in the hash table size can lead to remapping a large portion, if not all, of the keys.

This ability to minimize the impact of node failures or changes in the system topology is one of the key benefits of consistent hashing in distributed systems. It allows the system to maintain stability and efficient key-to-node mappings even as nodes are added, removed, or fail.

Consistent hashing is used by load balancers, distributed databases, caches, and distributed storage systems to evenly distribute traffic and data across nodes, ensuring efficient resource utilization and minimal disruption when nodes are added or removed.

Database Sharding

Database sharding is a technique used to partition a large database into smaller, more manageable pieces called shards. Each shard holds a subset of the data, and together, they make up the entire dataset. Sharding is often used to improve performance and scalability for databases that need to handle high volumes of transactions or queries.

Key Concepts in Database Sharding

1. Horizontal Partitioning: Sharding is a form of horizontal partitioning where each shard contains rows of the database table, as opposed to vertical partitioning where each shard contains columns.

2. Shard Key: This is a key that determines how the data is distributed across shards. It is usually chosen based on the most commonly queried fields to ensure even distribution and efficient querying.

3. Shard Infrastructure: Each shard is typically hosted on a separate database server, allowing for load balancing and parallel processing.

Benefits of Sharding

1. Scalability: By distributing data across multiple servers, sharding allows for horizontal scaling. When the load increases, new shards can be added to handle the extra data and queries.

2. Performance: Shards reduce the amount of data each server needs to process, which can lead to faster query response times and improved overall performance.

3. Availability: With data distributed across multiple servers, the failure of a single shard does not mean the entire database is down. This enhances the database's availability and fault tolerance.

Challenges of Sharding

1. Complexity: Implementing sharding adds complexity to the database architecture. It requires careful planning and management of the shard key, data distribution, and ensuring data consistency across shards.

2. Data Distribution: Uneven distribution of data can lead to hotspots, where some shards become overloaded while others are underutilized.

3. Joins and Transactions: Performing joins and transactions across multiple shards can be complex and less efficient. Applications need to be designed to handle distributed queries and transactions.

Disaster Recovery

Disaster recovery (DR) is a set of policies, tools, and procedures that enable the recovery or continuation of vital technology infrastructure and systems following a natural or human-induced disaster. It focuses on ensuring that an organization can quickly resume mission-critical functions after a disaster.

Key Concepts in Disaster Recovery

1. Recovery Point Objective (RPO): The maximum acceptable amount of data loss measured in time. Simply, it means how often do you want your data to be backed up? The lesser the RPO, the expensive the approach for example, say, we decide to backup every 10 minutes. Here, we accepting the data loss of 9 minutes. Real time RPO? We have real time replication.

2. Recovery Time Objective (RTO): The maximum acceptable amount of time to restore a function or system after a disaster. It represents how long it can take to get back to normal operations.

3. Disaster Recovery Plan (DRP): A documented process or set of procedures to recover and protect a business IT infrastructure in the event of a disaster. The plan includes detailed instructions on how to respond to unplanned incidents.

So, depending upon the RPO & RTO, we have different strategies to follow:

1. Backup & Restore: Restoring the system after the disaster.

   

2. Pilot Light: Minimal version of system is always running in the background.

   

3. Warm-standby: Ready to takeover (Duplicate of a system is running but not actively serving user requets)

   

4. Mulit-site active/active: Fully functional environments in multiple AWS regions simultaneously.

   

You've just created your AWS account—exciting!

Before you get started, let's make sure you do a few things to keep your account safe and secure.

Let's dive in.

Task 1: Assign MFA to your root account.

After creating your AWS account, you become the root user, which means you have all the privileges. It's crucial to secure your account properly.

On your account, on right top corner,

Click on and then Select Security Credentials.

You'll see like the following interface.

Click on Assign MFA.

Name your MFA device.

I'm using Authenticator app for MFA purpose.

It authenticates using a code generated by an app on your mobile device. This code is unique and valid only for a short time, adding an extra layer of security to your accounts.

There are multiple authenticator apps. I've installed Google's Google Authenticator.

Scan the QR code from your authenticator app.

Fill the two consecutive codes from your Authenticator device.

So, our first task is completed.

Task 2: Create Budget Alert

Click on Billing & Cost Management.

Scroll a bit and click on Billing Preferences.

Now Click on the check boxes as below and provide your email address to receive Free Tier usage alerts.

Now, we've setup our Billing preferences.

On Billing & Cost Management section, click on Budgets.

Create a budget.

Choose Customize option on Budget setup.

Scroll a bit and Click on Next.

Provide name for your budget and also enter your budgeted amount.

Scroll down and Click on Next.

Click on Add an alert threshold.

In my case, I've applied 50%. Provide your desired Email recipients.

When your actual cost is greater than 50.00% ($5.00) of your budgeted amount ($10.00), the alert threshold will be exceeded.

Click on Next.

Review all of your information and click on Create Budget.

That completes our second task.

Task 3: Create an IAM user.

Why we need IAM user?

Daily tasks with the root user in AWS are risky, like giving everyone a house key. IAM users are better - create specific users with limited access, like assigning keys only to certain rooms. This keeps your AWS environment more secure!

Alright, On the services, search for IAM and click on it.

Click on Users and then Create user and provide a suitable username with password.

Click on Next.

Search for Administrator Access, tick the box and Click on Next.

Review what you've done and proceed to Create user.

Return to dashboard.

Click on Create below Account Alias.

An AWS account alias is a custom name for your account ID, making it easier to remember and use when accessing AWS services.

Provide your preferred name.

Copy the given Sign-in URL and paste it into your browser and bookmark it.

Provide your IAM username and enter password to Sign in.

We've signed in as an IAM user.

Click on Security credentials, for this new identity we've to setup MFA just like we did earlier for our root account.

Assign MFA and do as we did earlier.

I hope you've done yours as well.

Alright. We've finished the account setup and securing process.

This concludes the blog.

In a nutshell, we accomplished the following:

1. Assigned MFA to root account,

2. Created a Budget Alert,

3. Created IAM user and assigned MFA.

Hope you guys had a wonderful time.

See you again in the next one.