At a FinTech payment processor building a real-time fraud detection lakehouse, the data engineering team must ingest streaming card transactions, compute 5-minute rolling velocity aggregates in real time, and land the enriched records as Apache Parquet files in Amazon S3 with the LEAST operational overhead. Select and order the 4 architectural steps from initial ingestion to S3 storage.
- A. Order 1 -> 2 -> 3 -> 4: (1) Ingest raw card events into Amazon Kinesis Data Streams -> (2) Process stateful 5-minute sliding windows using Amazon Managed Service for Apache Flink -> (3) Emit enriched aggregates to an Amazon Data Firehose delivery stream configured with inline Parquet conversion via a Glue Data Catalog table schema -> (4) Deliver the compressed Parquet partitions directly into the target Amazon S3 bucket.
- B. Order 2 -> 1 -> 4 -> 3: (1) Process windows in Flink -> (2) Ingest into Kinesis Data Streams -> (3) Write raw CSV to S3 -> (4) Run Data Firehose on S3 files.
- C. Order 1 -> 3 -> 2 -> 4: (1) Ingest into Kinesis Data Streams -> (2) Land raw JSON in S3 via Firehose -> (3) Run Apache Flink directly on S3 cold storage -> (4) Convert to Parquet via SQS.
- D. Order 3 -> 1 -> 2 -> 4: (1) Buffer in Data Firehose -> (2) Send to Kinesis Data Streams -> (3) Aggregate in Flink -> (4) Store in DynamoDB.
Reveal Verified Answer (A), Technical Explanation & Exam Hack
Correct Answer: Option A
In a real-time AWS streaming pipeline, events first enter Amazon Kinesis Data Streams (Step 1), are aggregated over stateful windows by Managed Service for Apache Flink (Step 2), pass to Amazon Data Firehose with Glue Catalog schema conversion enabled for Apache Parquet (Step 3), and land partitioned in Amazon S3 (Step 4). Architectural Deep Dive: Amazon Kinesis Data Streams provides sub-second real-time stream ingestion with configurable retention (24 hours up to 365 days), replayability across multiple independent consumers, and optional Enhanced Fan-Out (2 MB/s dedicated throughput per consumer per shard via HTTP/2). Why Distractors Fail: Option B (Order 2 -> 1 -> 4 -> 3: (1) Process windows in Flink -> (2) Ingest into Kin...) is incorrect because that service applies to a different workload pattern and uses a non-replayable delivery buffer or competing queue where multiple independent consumers cannot replay historical stream records at their own pace. Option C (Order 1 -> 3 -> 2 -> 4: (1) Ingest into Kinesis Data Streams -> (2) Land ra...) is incorrect because that service applies to a different workload pattern and uses a non-replayable delivery buffer or competing queue where multiple independent consumers cannot replay historical stream records at their own pace. Option D (Order 3 -> 1 -> 2 -> 4: (1) Buffer in Data Firehose -> (2) Send to Kinesis ...) is incorrect because that service applies to a different workload pattern and uses a non-replayable delivery buffer or competing queue where multiple independent consumers cannot replay historical stream records at their own pace.