This is very much one of those YMMV (quite wildly depending on the project) but one option might be to write the files to S3. You can configure a Lambda to run for each new file written, and then get the result back into a separate bucket.