1、kafka Kafka topic partition partiton partiton segment partition segment file partition msg chunk push-group Kafka Producer 1 Producer 2 Consumer 2 Zookeeper Broker 1 Broker 2 Broker 3 Broker 4 consumer Consumer 3 topic1/part-0 /part-4 topic1/part-1 /part-5 topic1/part-2 /part-6 topic1/part-3 /part-
2、7 topic partition report_push-0 report_push report_push-1 report_push-2 report_push-3 topic名称为:report_push kafka topic partition partition partiton partion( ) segment( ) segment file old segment file partiton segment 100GB 500MB 500MB 500MB 1 501 1001 1501 segment! le-0 segment! le-1 segment! le-2 p
3、artitionpartiton segment partion segment file segment file segment file 2 segment data file segment index file 2 .segment index data file message-chunk1 message-chunk3 message-chunk8 message-chunk6 00000000000000000000.index message-chunk1 message-chunk2 message-chunk4 message-chunk3 000000000000000
4、00000.log message-chunk5 message-chunk8 message-chunk7 message-chunk6 index index 4096 :partion segment 0 segment partion offset( ) 1. : log 8bytes 4 byte segment file offset - last seg file offset offset 4 byte segment file position partiton segment -index 00000000000000000000.indexpart segment -da
5、ta file msg chunk : 4 byte CRC32 1 byte “magic“ 1 byte “attributes“ 4 byte key length K byte key 4 byte payload length val bytes msgs payload . 00000000000000000000.log message1 message2 message4 message3 message5 message6 8 byte offset 4 byte chunk size (message chunk) offset(partiions msg chunk) c
6、hunk data index record size data msg chunk 1 - , 2 - topic ( ) report_push topic 0-0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 .index 00000000000000000000.log 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 6 8 7 6 9 . i n d e x 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 6 8 7 6 9 . l o g 0 0 0 0 0 0 0 0 0 0 0 0 0 0 7 3 7 3 3 7 . i
7、 n d e x 0 0 0 0 0 0 0 0 0 0 0 0 0 0 7 3 7 3 3 7 . log 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 5 8 1 4 . i n d e x 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 5 8 1 4 .log 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 . i n d e x 0. 00000000000000368769.index 368769. 00000000000000737337.index 737337. offset* * offset offset
8、 368969 368969 00000000000000368769.log 368969 368769 737337 partition segment file partition msg chunk 1,0 3,4597 8,12345 6,9807 00000000000000000000.index message-chunk1 message-chunk2 message-chunk4 message-chunk3 00000000000000000000.log message-chunk5 message-chunk8 message-chunk7 message-chunk6 offset = 1 offset = 8 0 4597 2039 6830 7912 9807 1108 12345 message-chunkN position index memory segment file IO consumer msg chunk segment file offset kafka kafka-0.8.1-src http:/kafka.apache.org/ http:/ you! Any Quest?!
