Sparkstreaming程序在消费kafka时报了一个告警
23/02/06 11:03:08 WARN scheduler.TaskSetManager: Lost task 1.0 in stage 344.0 (TID 8615, hadoop07, executor 5): org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions: {GZJY-3=30970547102}
at org.apache.kafka.clients.consumer.internals.Fetcher.parseCompletedFetch(Fetcher.java:990)
at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:491)
at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1269)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1200)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1135)
at org.apache.spark.streaming.kafka010.InternalKafkaConsumer.poll(KafkaDataConsumer.scala:200)
at org.apache.spark.streaming.kafka010.InternalKafkaConsumer.get(KafkaDataConsumer.scala:129)
at org.apache.spark.streaming.kafka010.KafkaDataConsumer$class.get(KafkaDataConsumer.scala:36)
at org.apache.spark.streaming.kafka010.KafkaDataConsumer$CachedKafkaDataConsumer.get(KafkaDataConsumer.scala:212)
at org.apache.spark.streaming.kafka010.KafkaRDDIterator.next(KafkaRDD.scala:261)
at org.apache.spark.streaming.kafka010.KafkaRDDIterator.next(KafkaRDD.scala:229)
at scala.collection.Iterator$$anon.next(Iterator.scala:410)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:149)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
at org.apache.spark.scheduler.Task.run(Task.scala:121)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun.apply(Executor.scala:407)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1408)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:413)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
命令查看对应消费id的消费情况
kafka-consumer-groups --bootstrap-server hadoop02:9092,hadoop03:9092,hadoop04:9092,hadoop05:9092,hadoop06:9092,hadoop07:9092,hadoop08:9092,hadoop09:9092,hadoop10:9092,hadoop11:9092,hadoop12:9092 --describe --group pangu_GDT003_online108
返回
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
GZJY 10 21102297666 21102810249 512583 consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19 consumer-1
GZJY 3 30971455431 30962503292 -8952139 consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19 consumer-1
GZJY 2 31581677989 31582209958 531969 consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19 consumer-1
GZJY 9 21063437255 21063932876 495621 consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19 consumer-1
GZJY 4 27081104241 27081622964 518723 consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19 consumer-1
GZJY 8 31544897443 31545428226 530783 consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19 consumer-1
GZJY 6 31479796535 31480294328 497793 consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19 consumer-1
GZJY 7 31523391532 31523930639 539107 consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19 consumer-1
GZJY 0 31344827282 31345323448 496166 consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19 consumer-1
GZJY 1 31097505976 31098074369 568393 consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19 consumer-1
GZJY 5 31478597056 31479107234 510178 consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19 consumer-1
可以看出来3号确实有些问题
通过命令修改3号为最新
kafka-consumer-groups --bootstrap-server hadoop02:9092,hadoop03:9092,hadoop04:9092,hadoop05:9092,hadoop06:9092,hadoop07:9092,hadoop08:9092,hadoop09:9092,hadoop10:9092,hadoop11:9092,hadoop12:9092 --group pangu_GDT003_online108 --topic GZJY --execute --reset-offsets --to-earliest
再次查看返回
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
GZJY 10 21102967356 21103534428 567072 consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19 consumer-1
GZJY 3 30972117204 30972666789 549585 consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19 consumer-1
GZJY 2 31582359667 31582930138 570471 consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19 consumer-1
GZJY 9 21064090203 21064628633 538430 consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19 consumer-1
GZJY 4 27081771694 27082333571 561877 consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19 consumer-1
GZJY 8 31545568770 31546140879 572109 consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19 consumer-1
GZJY 6 31480442205 31481007582 565377 consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19 consumer-1
GZJY 7 31524089705 31524647802 558097 consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19 consumer-1
GZJY 0 31345452158 31345995112 542954 consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19 consumer-1
GZJY 1 31098218435 31098770724 552289 consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19 consumer-1
GZJY 5 31479258366 31479815080 556714 consumer-1-5804be67-55cf-4164-b1d9-84a8898f607c /172.39.10.19 consumer-1
正常
观察Sparkstreaming没有报警告。