1、关于structed streaming 读取kafka的问题：

数据流从kafka过来，根据条件的不同，会被拆分成好几个流式dataFrame，这些流式的DF会进行各种join操作，在这个过程中会随机的触发KafkaConsumer is not safe for multi-threaded access，kafka多线程消费的问题，本程序就只有一个线程，怎么会有多线程消费的问题。在网上各方查找，有说是spark的架构task任务问题，有说是加个cache进行RDD缓存，有说是spark的bug，最终还是没有解决，问题还是会随机的出现，并且随着流的拆分数量增多，出现的概率也越高。（spark版本是2.3.0）

java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access
	at org.apache.kafka.clients.consumer.KafkaConsumer.acquire(KafkaConsumer.java:1431)
	at org.apache.kafka.clients.consumer.KafkaConsumer.seek(KafkaConsumer.java:1132)
	at org.apache.spark.sql.kafka010.CachedKafkaConsumer.seek(CachedKafkaConsumer.scala:305)
	at org.apache.spark.sql.kafka010.CachedKafkaConsumer.org$apache$spark$sql$kafka010$CachedKafkaConsumer$$fetchData(CachedKafkaConsumer.scala:216)
	at org.apache.spark.sql.kafka010.CachedKafkaConsumer$$anonfun$get$1.apply(CachedKafkaConsumer.scala:122)
	at org.apache.spark.sql.kafka010.CachedKafkaConsumer$$anonfun$get$1.apply(CachedKafkaConsumer.scala:106)
	at org.apache.spark.util.UninterruptibleThread.runUninterruptibly(UninterruptibleThread.scala:77)
Caused by: java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access
	at org.apache.kafka.clients.consumer.KafkaConsumer.acquire(KafkaConsumer.java:1431)
	at org.apache.kafka.clients.consumer.KafkaConsumer.seek(KafkaConsumer.java:1132)
	at org.apache.spark.sql.kafka010.CachedKafkaConsumer.seek(CachedKafkaConsumer.scala:305)
	at org.apache.spark.sql.kafka010.CachedKafkaConsumer.org$apache$spark$sql$kafka010$CachedKafkaConsumer$$fetchData(CachedKafkaConsumer.scala:216)
	at org.apache.spark.sql.kafka010.CachedKafkaConsumer$$anonfun$get$1.apply(CachedKafkaConsumer.scala:122)

2、解决办法：

由于问题是随机出现的，说明并不是代码的问题，可能是环境的问题或者其他外在的问题。目前找到唯一有效的办法就是提高spark的版本。将2.3.0改成了2.3.1。也不是最新版本，也不算老的版本，可能就是最好的版本。

<dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.11</artifactId>
            <version>2.3.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_2.11</artifactId>
            <version>2.3.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
            <version>2.3.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql-kafka-0-10_2.11</artifactId>
            <version>2.3.1</version>
        </dependency>