Skip to main content

Event Driven (Instant)

Instant import starts ingestion when your S3 bucket sends an object-created notification. Processing time depends on file size and queue load; delivery of a notification is not proof that the file has finished processing.

Set up the source in Chipper​

  1. Open Configure → Log Sources → New log source and choose the instrument.
  2. Choose Existing cloud storage → AWS and enter the bucket, region, log prefix, and read credentials.
  3. Complete the connection check, then choose Instant import.
  4. Copy the SQS queue ARN displayed by the wizard. Chipper authorizes this bucket on that queue before you configure its notification.

Use the displayed ARN for the current environment. Old log-ingestion queue lists are not the setup contract. If no ARN is available or authorization fails, contact support before continuing.

The bucket must be in the wizard's supported instant-import region. S3 requires the bucket and destination SQS queue to be in the same region. For a different region, use scheduled import, or coordinate a replica bucket in the supported region. See AWS notification destinations.

Add the S3 notification​

In S3 → your bucket → Properties → Event notifications, create a notification:

  • Name: a recognizable name such as chipper-new-files.
  • Event types: all object-created events (s3:ObjectCreated:*).
  • Destination: SQS, using the exact ARN copied from Chipper.
  • Prefix: the object prefix for this instrument. If a standalone collector writes the objects, this must cover its configured upload prefix.
  • Suffix: optional; do not exclude an extension or compression suffix the collector writes.

Save the notification, return to Chipper, and run the notification check before finishing setup. The read key needs s3:GetBucketNotification for this check.

If using the wizard's CLI example, merge its queue entry into any existing notification JSON: put-bucket-notification-configuration replaces the bucket's entire notification configuration.

Verify ingestion​

Add a new representative file under the configured prefix. Open Investigate → Log Files and confirm that it reaches Processed with the intended instrument. Inspect its runs and events. If notification verification fails, check the target ARN, same-region requirement, prefix, and read-key permissions before contacting support.