1、DRUIDINTERACTIVE EXPLORATORY ANALYTICS AT SCALEFANGJIN YANG DRUID COMMITTEROVERVIEWDEMO SEE SOME NEAT THINGS MOTIVATION WHY DRUID? ARCHITECTURE PICTURES WITH ARROWS COMMUNITY CONTRIBUTE TO DRUID 2013THE PROBLEMArbitrary and interactive exploration of time series data Ad-tech, system/app metrics, net
2、work/website traffic analysis Multi-tenancy: lots of concurrent users Scalability: 10+ TB/day, ad-hoc queries on trillions of events Recency matters! Real-time analysisDEMOIN CASE THE INTERNET DIDNT WORK PRETEND YOU SAW SOMETHING COOL2015REQUIREMENTSScalable & highly available Real-time data ingesti
3、on Arbitrary data exploration with ad-hoc queries Sub-second queries Many concurrent reads2015FINDING A SOLUTIONLoad all your data into Hadoop. Query it. Done! Good job guys, lets go home2015FINDING A SOLUTIONHadoopEvent StreamsInsight2015PROBLEMS WITH THE NAIVE SOLUTIONMapReduce can handle almost e
4、very distributed computing problem MapReduce over your raw data is flexible but slow Hadoop is not optimized for query latency To optimize queries, we need a query layer2015FINDING A SOLUTIONHadoop (pre-processing and storage) Query LayerHadoopEvent StreamsInsight2015MAKE QUERIES FASTERWhat types of
5、 queries to optimize for? Business intelligence/OLAP/pivot tables queries Aggregations, filters, groupBysWHAT WE TRIED2015FINDING A SOLUTIONHadoop (pre-processing and storage) RDBMS?HadoopEvent StreamsInsight2015Common solution in data warehousing: Star Schema Aggregate Tables Query CachingI. RDBMS
6、- THE SETUP2015Queries that were cached fast Queries against aggregate tables fast to acceptable Queries against base fact table generally unacceptableI. RDBMS - THE RESULTS2015I. RDBMS - PERFORMANCENaive benchmark scan rate 5.5M rows / second / core 1 day of summarized aggregates 60M+ rows 1 query
7、over 1 week, 16 cores 5 seconds Page load with 20 queries over a week of datalong time2015FINDING A SOLUTIONHadoop (pre-processing and storage)NoSQL K/V Stores?HadoopEvent StreamsInsight2015Pre-aggregate all dimensional combinations Store results in a NoSQL storeII. NOSQL - THE SETUPtsgender age rev
8、enue1 M 18 $0.151 F 25 $1.031 F 18 $0.01Key Value1 revenue=$1.191,M revenue=$0.151,F revenue=$1.041,18 revenue=$0.161,25 revenue=$1.031,M,18 revenue=$0.151,F,18 revenue=$0.011,F,25 revenue=$1.032015Queries were fast range scan on primary key Inflexible not aggregated, not available Does not work wel
9、l with streamsII. NOSQL - THE RESULTS2015Processing scales exponentially! Example: 500k records Precompute 11 dimensions 4.5 hours on a 15-node Hadoop cluster Precompute 14 dimensions 9 hours on a 25-node Hadoop clusterII. NOSQL - PERFORMANCE2015FINDING A SOLUTIONHadoop (pre-processing and storage)C
10、ommercial DatabasesHadoopEvent StreamsInsightDRUID AS A QUERY LAYER2013KEY FEATURES LOW LATENCY INGESTION FAST AGGREGATIONS ARBITRARY SLICE-N-DICE CAPABILITIES HIGHLY AVAILABLE APPROXIMATE & EXACT CALCULATIONSDRUIDDATA STORAGE2015DATA!timestamp page language city country . added deleted2011-01-01T00
11、:01:35Z Justin Bieber en SF USA 10 652011-01-01T00:03:63Z Justin Bieber en SF USA 15 622011-01-01T00:04:51Z Justin Bieber en SF USA 32 452011-01-01T01:00:00Z Ke$ha en Calgary CA 17 872011-01-01T02:00:00Z Ke$ha en Calgary CA 43 992011-01-01T02:00:00Z Ke$ha en Calgary CA 12 53.2015PARTITION DATAtimest
12、amp page language city country . added deleted2011-01-01T00:01:35Z Justin Bieber en SF USA 10 652011-01-01T00:03:63Z Justin Bieber en SF USA 15 622011-01-01T00:04:51Z Justin Bieber en SF USA 32 452011-01-01T01:00:00Z Ke$ha en Calgary CA 17 872011-01-01T02:00:00Z Ke$ha en Calgary CA 43 992011-01-01T0
13、2:00:00Z Ke$ha en Calgary CA 12 53Shard data by time Immutable chunks of data called “segments”Segment 2011-01-01T02/2011-01-01T03Segment 2011-01-01T01/2011-01-01T02Segment 2011-01-01T00/2011-01-01T012015IMMUTABLE SEGMENTSFundamental storage unit in Druid No contention between reads and writes One t
14、hread scans one segment Multiple threads can access same underlying data2015COLUMNAR STORAGEScan/load only what you need Compression! Indexes!timestamp page language city country . added deleted2011-01-01T00:01:35Z Justin Bieber en SF USA 10 652011-01-01T00:03:63Z Justin Bieber en SF USA 15 622011-0
15、1-01T00:04:51Z Justin Bieber en SF USA 32 452011-01-01T01:00:00Z Ke$ha en Calgary CA 17 872011-01-01T02:00:00Z Ke$ha en Calgary CA 43 992011-01-01T02:00:00Z Ke$ha en Calgary CA 12 53.2015COLUMN COMPRESSION DICTIONARIESCreate ids Justin Bieber - 0, Ke$ha - 1 Store page - 0 0 0 1 1 1 language - 0 0 0
16、0 0 0 timestamp page language city country . added deleted2011-01-01T00:01:35Z Justin Bieber en SF USA 10 652011-01-01T00:03:63Z Justin Bieber en SF USA 15 622011-01-01T00:04:51Z Justin Bieber en SF USA 32 452011-01-01T01:00:00Z Ke$ha en Calgary CA 17 872011-01-01T02:00:00Z Ke$ha en Calgary CA 43 99
17、2011-01-01T02:00:00Z Ke$ha en Calgary CA 12 53.2015BITMAP INDICESJustin Bieber - 0, 1, 2 - 111000 Ke$ha - 3, 4, 5 - 000111 timestamp page language city country . added deleted2011-01-01T00:01:35Z Justin Bieber en SF USA 10 652011-01-01T00:03:63Z Justin Bieber en SF USA 15 622011-01-01T00:04:51Z Just
18、in Bieber en SF USA 32 452011-01-01T01:00:00Z Ke$ha en Calgary CA 17 872011-01-01T02:00:00Z Ke$ha en Calgary CA 43 992011-01-01T02:00:00Z Ke$ha en Calgary CA 12 53.2015FAST AND FLEXIBLE QUERIESJUSTIN BIEBER 1, 1, 0, 0KE$HA 0, 0, 1, 1JUSTIN BIEBER OR KE$HA 1, 1, 1, 1rowpage0Justin(Bieber1Justin(Bieber2Ke$ha3Ke$ha
