Aws Lambda functions can now mount Amazon S3 buckets as file systems

AWS Lambda has introduced support for Amazon S3 Files, allowing Lambda functions to mount Amazon S3 buckets as file systems. This enables the execution of standard file operations without the need to download data for processing. Built on Amazon EFS, S3 Files combines the ease of use and performance of a file system with the scalability, durability, and cost-efficiency of S3. Multiple Lambda functions can access the same S3 Files file system at the same time, facilitating data sharing through a common workspace without requiring custom synchronization logic.

The integration of S3 Files simplifies managing stateful workloads in Lambda by removing the need to download objects, upload results, and handle ephemeral storage constraints. This is especially beneficial for AI and machine learning tasks where retaining memory and sharing state across different pipeline steps is crucial. Lambda’s durable functions enable these complex AI workflows by coordinating parallel execution and providing automatic checkpointing. For instance, an orchestrator function can clone a repository into a shared workspace, allowing multiple agent functions to analyze the code concurrently. The durable function manages execution state checkpointing, while S3 Files ensures smooth data sharing across all stages.

To utilize S3 Files with Lambda, users can configure their functions to mount an S3 bucket via the Lambda console, AWS CLI, AWS SDKs, AWS CloudFormation, or AWS Serverless Application Model (SAM). For more detailed instructions on using S3 Files with Lambda functions, refer to the Lambda developer guide.

S3 Files is available for Lambda functions that are not configured with a capacity provider, in all AWS Regions where both Lambda and S3 Files are supported. There are no additional charges for using S3 Files beyond the standard Lambda and S3 pricing.