[已上的传]汪兴郎-ebay用户行为数据流实时处理系统.pdfVIP

eBay用户行为数据流 实时处理系统 eBay CCOE 汪兴朗 用户行为数据流介绍 背景介绍 •  eBay作为全球性的商务平台和支付行业领 先者,拥有海量的用户行为数据 •  用户行为数据拥有巨大的价值 •  eBay对用户行为数据处理经历过从DW到 Hadoop的转变 •  现有的Hadoop不能支撑实时性高的业务需 求 eBay 用户行为数据特点 •  海量的持续的数据流 •  Bot和真实用户数据混合在一起 •  基于用户会话的数据分析 •  移动和桌面端的用户会话的合并 •  7x24 监控和运维 整体架构 关键技术 •  Jetstream Ø  Pipelining Ø  Stream processing CEP engine (Esper) Ø  Broker less messaging Ø  Clustering •  Oeap cache and Queue Ø  Session state management Ø  Store and forward •  Kaa Ø  Store and forward, bridge to hadoop Ø  Guaranteed messaging with replay capabilies Jetstream Jetstream •  Spring as containter •  Zookeeper for coordinator •  Ney4 for async messaging •  Batching •  Compress •  Kryo as serializaon •  Disruptor as Queue •  MongoDB for configuraon •  Affinity based messaging roung •  Esper for declarave rules Why Jetstream? •  Open/Auto-discover topology •  Endless stream •  Push model •  Measure loss rather than blocking •  Proacve throling •  Declarave event processing logic •  Cost effecve messaging •  Predictable latency Oeap 技术的应用 •  应用场景 •  Sessionizaon •  In-memory data aggregaon •  Bot detecon •  OffHeap Cache •  Low latency •  Concurrent by paron cache by key •  O(1) read /write by hash •  Instantaneous expiraon by an ordered linked list, no full scan required, O(1) for session expiraon •  Minimized memory footprint by opmizing serializer •  TimeSlot to support diverse session TTLs Ka