How to auto-deploy Puppeteer in AWS Lambda using Github actions

This content originally appeared on DEV Community and was authored by Ivan Muñoz

Original Article: How to auto-deploy Puppeteer in AWS Lambda using Github actions

In this article, we will explore how to set up a Puppeteer project that can be automatically deployed to AWS Lambda using GitHub Actions.

It includes a repo that serves as a base template for deploying a Puppeteer project to AWS Lambda using GitHub Actions. The repo is designed to convert your Puppeteer project into an API that can be invoked via HTTP requests, allowing you to run Puppeteer scripts in a serverless environment.

You can find the complete code in the GitHub repository.

Development
1. Cloning the repository
2. Installing dependencies
3. Creating the .env file
4. Running the development server
5. Testing the API
Deployment
1. Creating the Lambda function
2. Creating the S3 bucket
3. Creating an IAM user and access keys
4. Configuring your lambda function
5. Configuring the GitHub Actions workflow
6. Testing the deployment
Troubleshooting
Conclusion
Additional Resources

Development

1. Cloning the repository

Clone the repository to your local machine:

git clone https://github.com/ivanalemunioz/puppeteer-lambda-auto-deploy.git
cd puppeteer-lambda-auto-deploy

2. Installing dependencies

Install the required dependencies using npm:

npm install

3. Creating the .env file

Create a .env file in the root of the project based on the .env.example file and fill in the required values.

4. Running the development server

Run the development server:

npm run dev

This command will start the server on http://localhost:5123 and will watch for changes in the code.

5. Testing the API

You can test the API by sending a POST request to http://localhost:5123/v1/run using a tool like Postman or curl. You should see the Puppeteer script running and returning a response.

curl --location 'http://localhost:5123/v1/run' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer YOUR_GENERATED_AUTH_TOKEN' \
--data '{
    "action": "scrape-pptr-docs",
    "params": {}
}'

Deployment

You can jump to step 4 if you already have a Lambda function, S3 bucket and IAM role set up.

1. Creating the Lambda function

Go to the AWS Lambda console.
Click on “Create function”.
Choose “Author from scratch”.
Enter a name for your function, e.g., puppeteer-lambda-auto-deploy.
Select the runtime as Node.js 22.x.
Select the architecture as x86_64. It will not work with arm64 architecture.
Under “Additional configurations”, enable “Function URL” to allow HTTP access to your Lambda function.
- “Auth type” select NONE.
- “Invoke mode” select “BUFFERED (default)”.
Click on “Create function”.

2. Creating the S3 bucket

Go to the S3 console.
Click on “Create bucket”.
As bucket type select “General purpose”.
Enter a unique name for your bucket, e.g., puppeteer-lambda-auto-deploy-bucket.
Let the rest of the settings as default.
Click on “Create bucket”.

3. Creating an IAM user and access keys

Go to the IAM console.
Click on “Users” in the left sidebar.
Click on “Create user”.
Enter a username, e.g., puppeteer-lambda-auto-deploy-user.
Click on “Next”.
Click on “Attach policies directly” and select the following policies:
- AWSLambda_FullAccess
- AmazonS3FullAccess
Click on “Create user”.
Click on the user you just created.
Under the “Security credentials” tab, click on “Create access key”.
As Use case select “Other”
Click on “Next”.
Click on “Create access key”.
Copy the Access key ID and Secret access key and store them securely.

4. Configuring your lambda function

Go to the AWS Lambda console.
Click on the function you created earlier.
Under the “Configuration” tab, click on “Environment variables”.
Click in “Edit” and add the BUGLESSTACK_ACCESS_TOKEN and BROWSER_AUTOMATIONS_ACCESS_TOKEN variables. You can get more info about how to generate these tokens in the .env.example file.
Under the “Configuration” tab, click on “General configuration” and “Edit”.
Increase the timeout to 2 minutes and memory to 1024MB (recommended) and click on “Save”.

5. Configuring the GitHub Actions workflow

Go to your GitHub repository.
Click on “Settings” in the top menu.
In the left sidebar, click on “Secrets and variables” and then “Actions”.
Add the following secrets:
- AWS_ACCESS_KEY_ID: Your AWS access key ID generated in step 3.
- AWS_SECRET_ACCESS_KEY: Your AWS secret access key generated in step 3.
- AWS_REGION: The AWS region where your Lambda function is deployed (e.g., us-east-1).
In the same section, click on “Variables” and add the following variables:
- S3_BUCKET: The name of the S3 bucket where the Lambda function package will be uploaded.
- S3_KEY: The key (path) in the S3 bucket where the Lambda function package will be stored (e.g., lambda/puppeteer.zip).
- S3_LAYER_BUCKET: The name of the S3 bucket where the Lambda layer package will be uploaded.
- S3_LAYER_KEY: The key (path) in the S3 bucket where the Lambda layer package will be stored (e.g., layers/puppeteer.zip).
- LAYER_NAME: The name of the Lambda layer to be created or updated (e.g., puppeteer-lambda-auto-deploy-layer).
- LAMBDA_FUNCTION_NAME: The name of the Lambda function to be updated (e.g., puppeteer-lambda-auto-deploy).
In “Settings > Actions > General”, ensure that “Allow all actions and reusable workflows” is selected under “Actions permissions”.

6. Testing the deployment

Push your changes to the main branch of your GitHub repository.
The GitHub Actions workflow will automatically build and deploy your Puppeteer project to AWS Lambda.
Once the workflow is complete, you can test the API by sending a POST request to https://YOUR_LAMBDA_URL/v1/run (you can get your Lambda URL in the Lambda details) using a tool like Postman or curl. You should see the Puppeteer script running and returning a response.
```
curl --location 'https://YOUR_LAMBDA_URL/v1/run' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer YOUR_GENERATED_AUTH_TOKEN' \
--data '{
    "action": "scrape-pptr-docs",
    "params": {}
}'
```

Troubleshooting

If you encounter issues with Puppeteer not launching or crashing, ensure that the Lambda function has sufficient memory allocated (at least 1024 MB is recommended).
Configure the Lambda function timeout to a reasonable value (e.g., 30 seconds) to allow Puppeteer enough time to execute.
Ensure your Lambda architecture is set to x86_64, it will not work with arm64 architecture.
Ensure your Lambda and S3 bucket are in the same region.
Check the permissions of the IAM user to ensure it has access to S3 and Lambda.
Check the AWS Lambda logs in CloudWatch Logs for any errors or issues during execution.
If you encounter issues with the Github Actions workflow, check the workflow logs for any errors or issues during the build and deployment process.

Conclusion

In this guide, you learned how to set up a CI/CD pipeline for your Puppeteer project using GitHub Actions and AWS Lambda. By following these steps, you can automate the deployment of your Puppeteer scripts to AWS Lambda, making it easier to run headless browser tasks in the cloud. This setup not only streamlines your development workflow but also leverages the scalability and reliability of AWS Lambda for running your Puppeteer scripts.