AWS S3

Connect an Amazon S3 bucket to Vayu to make files stored in the bucket available for data ingestion.

Vayu generates an AWS CloudFormation template for the selected bucket and region. Deploy this template in your AWS account to create the resources required by the integration.

Before You Begin

Make sure you have:

• An existing Amazon S3 bucket.
• The AWS Region containing the bucket.
• Permission to create AWS CloudFormation stacks.
• Permission to create the IAM and Lambda resources defined by the template.
• Permission to access the selected S3 bucket.

Generate the CloudFormation Template

1. In Vayu, go to Integrations.
2. Select AWS S3.
3. Enter the name of the S3 bucket.
4. Select the AWS Region containing the bucket.
5. Select Generate Cloud Formation.
6. Copy the generated YAML and save it locally as a .yaml file.

Do not change the bucket name, Vayu account identifier, endpoint, or other generated values unless instructed by Vayu Support.

Deploy the Template in AWS

1. Sign in to the AWS Management Console.
2. Open CloudFormation.
3. Select Create stack and then With new resources (standard).
4. Under Specify template, select Upload a template file.
5. Upload the YAML file generated by Vayu.
6. Select Next.
7. Enter a descriptive stack name.
8. Review the stack parameters and select Next.
9. Configure any required stack options.
10. Review the resources that CloudFormation will create.
11. Acknowledge that the template may create IAM resources.
12. Select Submit.

Wait until the stack reaches the CREATE_COMPLETE status before proceeding.

If the stack fails, open the Events tab in CloudFormation to identify the resource and error that caused the deployment to stop.

Verify the Connection

After the stack has been created:

1. Return to the AWS S3 integration in Vayu.
2. Confirm that the integration is shown as connected or active.
3. Upload a representative test file to the configured bucket.
4. Confirm that Vayu receives and processes the file.
5. Compare the resulting data in Vayu with the contents of the source file.

Use production-equivalent test data, but avoid including confidential customer information until the connection has been validated.

AWS Resources and Permissions

The generated CloudFormation stack creates the AWS resources used to connect the selected bucket with Vayu. These may include:

• An AWS Lambda function that processes or forwards S3 data.
• An IAM role used by the Lambda function.
• Permissions that allow the integration to locate and read the required S3 objects.
• Configuration containing the selected bucket and the applicable Vayu account and ingestion endpoint.
• A mechanism that invokes the function when eligible files are available.

Review the generated template before deployment and confirm that its resources and permissions comply with your organization's security policies.

Apply least-privilege access. The integration should have access only to the required bucket, objects, and operations.

Test Before Production Use

Before enabling production ingestion:

1. Deploy the integration in a test or controlled AWS environment where possible.
2. Upload a representative file using the expected format and object path.
3. Confirm that the integration is invoked.
4. Verify that Vayu receives the file successfully.
5. Confirm that the data maps to the expected customers, events, or measurements.
6. Upload the same file again to test duplicate handling.
7. Test an invalid or incomplete file.
8. Confirm that failures are visible and can be investigated.
9. Verify that retrying a failed transfer does not create unintended duplicate usage.

Troubleshooting

CloudFormation Stack Fails

Open the stack in AWS CloudFormation and review its Events. Common causes include:

• Insufficient permission to create IAM or Lambda resources.
• An incorrect bucket name or AWS Region.
• Organizational policies that prevent creation of the generated resources.
• A deployment package or referenced AWS resource that is unavailable.

Delete or retry a failed stack only in accordance with your organization's AWS change procedures.

Files Are Not Reaching Vayu

Confirm that:

• The CloudFormation stack completed successfully.
• The integration has permission to read the relevant S3 objects.
• The integration's invocation mechanism is configured and active.
• Files are being uploaded to the expected bucket and object path.
• The file format matches the agreed ingestion format.
• The Vayu account identifier and endpoint in the deployed configuration are correct.

Review AWS Lambda logs in Amazon CloudWatch for invocation or processing errors.

Access Is Denied

Review the IAM role and applicable S3 bucket policy. Confirm that the integration can list the required bucket and read the relevant objects.

If the bucket uses AWS Key Management Service encryption, the integration may also require permission to decrypt objects using the applicable KMS key.

Duplicate Data Appears

Confirm that each source record has a stable, unique identifier and that retries reuse the same identifier. Test duplicate handling before uploading production files.

Security Considerations

• Restrict the integration to the required S3 bucket and object paths.
• Do not grant access to unrelated buckets.
• Review IAM permissions before deploying the stack.
• Use S3 encryption and HTTPS transport.
• Restrict access to the CloudFormation stack and its Lambda logs.
• Avoid writing credentials or sensitive file contents to application logs.
• Follow your organization's retention and deletion policies for source files.