Streaming Logs to Chipper

Chipper can ingest instrument and automation logs from multiple cloud environments.
The general flow of data looks like this:

Hardware → Cloud Provider → Chipper

Choose the method that best fits your lab infrastructure and data governance requirements.

---

Step 1: Hardware → Cloud Provider

Different instruments and systems have different ways of exporting logs, but the goal is the same:
automatically copy your hardware logs to a cloud storage bucket (S3, Azure Blob, etc.) as soon as they are generated.

This is best achieved by installing or scheduling a log agent or sync process on the computer connected to your instrument.
Doing so not only enables Chipper ingestion but also provides redundant cloud backups of your hardware logs — helpful for troubleshooting or audit trails.

Recommended log agents

Cloud Provider	Tool	Description	Notes
Azure	AzCopy	Command-line utility that efficiently uploads or syncs local files to Azure Blob Storage.	Supports scheduled syncs, SAS tokens, and filters (--exclude-pattern). Ideal for Windows-based lab PCs.
AWS	AWS CLI + S3 Sync	Built-in aws s3 sync command can mirror local log directories to an S3 bucket.	Supports incremental uploads and IAM-based auth. Works on Windows, macOS, or Linux.
(Optional)	Fluent Bit or Filebeat	Lightweight log shippers that can stream directly to S3, Blob, or HTTP endpoints.	Useful for labs that already have observability pipelines or containerized environments.

Example (Azure – AzCopy JSON config):

{
	"dest_base": "https://<your-storage-account>.blob.core.windows.net/<container>?<sas-token>",
	"customer_prefix": "lab-logs",
	"interval_seconds": 60,
	"pairs": [
		{
			"instrument_id": "tecan-1",
			"source_dir": "C:\\InstrumentLogs\\Tecan"
		}
	],
	"exclude_pattern": "*.tmp;*.DS_Store;~$*"
}

You can run this on a schedule (e.g., via Task Scheduler or cron) to continuously sync your hardware logs to the cloud.

Step 2: Cloud Provider → Chipper

Once your logs are being stored in the cloud, Chipper can automatically detect, parse, and ingest them. Follow the setup instructions for your chosen provider:

Cloud Provider	Best For	Setup Time	Live Streaming
AWS	Integrating with AWS infrastructure like S3, SQS	5 min	✅
Azure	Integrating with Azure infrastructure like AzureBlobStorage	5 min	✅
Manual	Testing, one-offs	Instant	❌