Warning: Ignoring non-spark config property: es.nodes.data.only=false 17/09/22 12:09:57 INFO SparkContext: Running Spark version 2.2.0 17/09/22 12:09:58 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17/09/22 12:09:58 INFO SparkContext: Submitted application: ESTest_5_1_3 17/09/22 12:09:58 INFO SecurityManager: Changing view acls to: mes 17/09/22 12:09:58 INFO SecurityManager: Changing modify acls to: mes 17/09/22 12:09:58 INFO SecurityManager: Changing view acls groups to: 17/09/22 12:09:58 INFO SecurityManager: Changing modify acls groups to: 17/09/22 12:09:58 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(mes); groups with view permissions: Set(); users with modify permissions: Set(mes); groups with modify permissions: Set() 17/09/22 12:09:59 INFO Utils: Successfully started service 'sparkDriver' on port 39363. 17/09/22 12:09:59 INFO SparkEnv: Registering MapOutputTracker 17/09/22 12:09:59 INFO SparkEnv: Registering BlockManagerMaster 17/09/22 12:09:59 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 17/09/22 12:09:59 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up 17/09/22 12:09:59 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-bedf254d-f56d-4ad4-a2d2-3c8ad56447e4 17/09/22 12:09:59 INFO MemoryStore: MemoryStore started with capacity 246.9 MB 17/09/22 12:09:59 INFO SparkEnv: Registering OutputCommitCoordinator 17/09/22 12:10:00 INFO Utils: Successfully started service 'SparkUI' on port 4040. 17/09/22 12:10:00 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.10.10:4040 17/09/22 12:10:01 INFO SparkContext: Added file file:/tmp/script_MJ2bwJO98J/5_concurrency_1_swissdata.py at spark://192.168.10.10:39363/files/5_concurrency_1_swissdata.py with timestamp 1506082201810 17/09/22 12:10:01 INFO Utils: Copying /tmp/script_MJ2bwJO98J/5_concurrency_1_swissdata.py to /tmp/spark-a3636a62-4957-4e15-af0a-c158d60f6433/userFiles-a62e9484-6525-482c-bbb1-c78946e3e35e/5_concurrency_1_swissdata.py 2017-09-22 12:10:03,503:28841(0x7f687187f700):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8 2017-09-22 12:10:03,503:28841(0x7f687187f700):ZOO_INFO@log_env@730: Client environment:host.name=mes_master 2017-09-22 12:10:03,503:28841(0x7f687187f700):ZOO_INFO@log_env@737: Client environment:os.name=Linux 2017-09-22 12:10:03,503:28841(0x7f687187f700):ZOO_INFO@log_env@738: Client environment:os.arch=4.9.0-3-amd64 2017-09-22 12:10:03,503:28841(0x7f687187f700):ZOO_INFO@log_env@739: Client environment:os.version=#1 SMP Debian 4.9.30-2+deb9u3 (2017-08-06) 2017-09-22 12:10:03,503:28841(0x7f687187f700):ZOO_INFO@log_env@747: Client environment:user.name=mes 2017-09-22 12:10:03,503:28841(0x7f687187f700):ZOO_INFO@log_env@755: Client environment:user.home=/home/mes 2017-09-22 12:10:03,503:28841(0x7f687187f700):ZOO_INFO@log_env@767: Client environment:user.dir=/tmp/script_MJ2bwJO98J 2017-09-22 12:10:03,503:28841(0x7f687187f700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=192.168.10.10:2181 sessionTimeout=10000 watcher=0x7f6879e6b712 sessionId=0 sessionPasswd= context=0x7f68b4041738 flags=0 I0922 12:10:03.505990 29208 sched.cpp:232] Version: 1.3.0 2017-09-22 12:10:03,521:28841(0x7f686df77700):ZOO_INFO@check_events@1728: initiated connection to server [192.168.10.10:2181] 2017-09-22 12:10:03,532:28841(0x7f686df77700):ZOO_INFO@check_events@1775: session establishment complete on server [192.168.10.10:2181], sessionId=0x15ea92e2ecd0013, negotiated timeout=10000 I0922 12:10:03.533036 29199 group.cpp:340] Group process (zookeeper-group(1)@192.168.10.10:38199) connected to ZooKeeper I0922 12:10:03.533231 29199 group.cpp:830] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0) I0922 12:10:03.533265 29199 group.cpp:418] Trying to create path '/mesos' in ZooKeeper I0922 12:10:03.583153 29199 detector.cpp:152] Detected a new leader: (id='25') I0922 12:10:03.584180 29199 group.cpp:699] Trying to get '/mesos/json.info_0000000025' in ZooKeeper I0922 12:10:03.602819 29199 zookeeper.cpp:262] A new leading master (UPID=master@192.168.10.10:5050) is detected I0922 12:10:03.603304 29199 sched.cpp:336] New master detected at master@192.168.10.10:5050 I0922 12:10:03.624153 29199 sched.cpp:352] No credentials provided. Attempting to register without authentication I0922 12:10:03.637948 29199 sched.cpp:759] Framework registered with b192e864-8a9b-4ffc-94ab-953d2b929bd2-0013 17/09/22 12:10:03 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 36563. 17/09/22 12:10:03 INFO NettyBlockTransferService: Server created on 192.168.10.10:36563 17/09/22 12:10:03 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 17/09/22 12:10:03 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.10.10, 36563, None) 17/09/22 12:10:03 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.10.10:36563 with 246.9 MB RAM, BlockManagerId(driver, 192.168.10.10, 36563, None) 17/09/22 12:10:03 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.10.10, 36563, None) 17/09/22 12:10:03 INFO BlockManager: external shuffle service port = 7337 17/09/22 12:10:03 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.10.10, 36563, None) 17/09/22 12:10:04 INFO EventLoggingListener: Logging events to file:/var/lib/spark/eventlog/b192e864-8a9b-4ffc-94ab-953d2b929bd2-0013 17/09/22 12:10:04 INFO Utils: Using initial executors = 0, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances 17/09/22 12:10:04 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 0 17/09/22 12:10:05 INFO MesosCoarseGrainedSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0 17/09/22 12:10:05 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/tmp/script_MJ2bwJO98J/spark-warehouse'). 17/09/22 12:10:05 INFO SharedState: Warehouse path is 'file:/tmp/script_MJ2bwJO98J/spark-warehouse'. 17/09/22 12:10:07 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint 17/09/22 12:10:07 INFO Version: Elasticsearch Hadoop v6.0.0-beta2 [66f16fdd93] 17/09/22 12:10:16 INFO ContextCleaner: Cleaned accumulator 2 17/09/22 12:10:16 INFO ContextCleaner: Cleaned accumulator 1 17/09/22 12:10:17 INFO CodeGenerator: Code generated in 908.275235 ms 17/09/22 12:10:17 INFO CodeGenerator: Code generated in 99.446633 ms 17/09/22 12:10:17 INFO CodeGenerator: Code generated in 91.947284 ms 17/09/22 12:10:18 INFO ScalaEsRowRDD: Reading from [swiss-airbnb] 17/09/22 12:10:19 INFO CodeGenerator: Code generated in 39.664684 ms 17/09/22 12:10:19 INFO ScalaEsRowRDD: Reading from [swiss-citypop] 17/09/22 12:10:19 INFO SparkContext: Starting job: collect at /tmp/script_MJ2bwJO98J/5_concurrency_1_swissdata.py:54 17/09/22 12:10:19 INFO DAGScheduler: Registering RDD 9 (collect at /tmp/script_MJ2bwJO98J/5_concurrency_1_swissdata.py:54) 17/09/22 12:10:19 INFO DAGScheduler: Registering RDD 5 (collect at /tmp/script_MJ2bwJO98J/5_concurrency_1_swissdata.py:54) 17/09/22 12:10:19 INFO DAGScheduler: Got job 0 (collect at /tmp/script_MJ2bwJO98J/5_concurrency_1_swissdata.py:54) with 12 output partitions 17/09/22 12:10:19 INFO DAGScheduler: Final stage: ResultStage 2 (collect at /tmp/script_MJ2bwJO98J/5_concurrency_1_swissdata.py:54) 17/09/22 12:10:19 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0, ShuffleMapStage 1) 17/09/22 12:10:19 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0, ShuffleMapStage 1) 17/09/22 12:10:19 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[9] at collect at /tmp/script_MJ2bwJO98J/5_concurrency_1_swissdata.py:54), which has no missing parents 17/09/22 12:10:20 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 10.5 KB, free 246.9 MB) 17/09/22 12:10:20 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 5.2 KB, free 246.9 MB) 17/09/22 12:10:20 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.10.10:36563 (size: 5.2 KB, free: 246.9 MB) 17/09/22 12:10:20 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006 17/09/22 12:10:20 INFO DAGScheduler: Submitting 15 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[9] at collect at /tmp/script_MJ2bwJO98J/5_concurrency_1_swissdata.py:54) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14)) 17/09/22 12:10:20 INFO TaskSchedulerImpl: Adding task set 0.0 with 15 tasks 17/09/22 12:10:20 INFO DAGScheduler: Submitting ShuffleMapStage 1 (MapPartitionsRDD[5] at collect at /tmp/script_MJ2bwJO98J/5_concurrency_1_swissdata.py:54), which has no missing parents 17/09/22 12:10:20 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 12.8 KB, free 246.9 MB) 17/09/22 12:10:21 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 5.8 KB, free 246.9 MB) 17/09/22 12:10:21 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.10.10:36563 (size: 5.8 KB, free: 246.9 MB) 17/09/22 12:10:21 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006 17/09/22 12:10:21 INFO DAGScheduler: Submitting 15 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[5] at collect at /tmp/script_MJ2bwJO98J/5_concurrency_1_swissdata.py:54) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14)) 17/09/22 12:10:21 INFO TaskSchedulerImpl: Adding task set 1.0 with 15 tasks 17/09/22 12:10:21 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 1 17/09/22 12:10:21 INFO ExecutorAllocationManager: Requesting 1 new executor because tasks are backlogged (new desired total will be 1) 17/09/22 12:10:22 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 3 17/09/22 12:10:22 INFO ExecutorAllocationManager: Requesting 2 new executors because tasks are backlogged (new desired total will be 3) 17/09/22 12:10:23 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 7 17/09/22 12:10:23 INFO ExecutorAllocationManager: Requesting 4 new executors because tasks are backlogged (new desired total will be 7) 17/09/22 12:10:24 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 15 17/09/22 12:10:24 INFO ExecutorAllocationManager: Requesting 8 new executors because tasks are backlogged (new desired total will be 15) 17/09/22 12:10:25 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 30 17/09/22 12:10:25 INFO ExecutorAllocationManager: Requesting 15 new executors because tasks are backlogged (new desired total will be 30) 17/09/22 12:10:35 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources 17/09/22 12:10:50 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources 2017-09-22 12:11:03,651:28841(0x7f686df77700):ZOO_WARN@zookeeper_interest@1570: Exceeded deadline by 11ms 17/09/22 12:11:05 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources 17/09/22 12:11:06 WARN MesosCoarseGrainedSchedulerBackend: Unable to parse into a key:value label for the task. 17/09/22 12:11:06 INFO MesosCoarseGrainedSchedulerBackend: Mesos task 0 is now TASK_RUNNING 17/09/22 12:11:06 INFO TransportClientFactory: Successfully created connection to /192.168.10.12:7337 after 42 ms (0 ms spent in bootstraps) 17/09/22 12:11:06 INFO MesosExternalShuffleClient: Successfully registered app b192e864-8a9b-4ffc-94ab-953d2b929bd2-0013 with external shuffle service. 17/09/22 12:11:11 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.10.12:42630) with ID 0 17/09/22 12:11:11 INFO ExecutorAllocationManager: New executor 0 has registered (new total is 1) 17/09/22 12:11:11 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 0, 192.168.10.12, executor 0, partition 2, NODE_LOCAL, 8748 bytes) 17/09/22 12:11:11 INFO TaskSetManager: Starting task 4.0 in stage 0.0 (TID 1, 192.168.10.12, executor 0, partition 4, NODE_LOCAL, 8748 bytes) 17/09/22 12:11:11 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.10.12:45473 with 366.3 MB RAM, BlockManagerId(0, 192.168.10.12, 45473, None) 17/09/22 12:11:12 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.10.12:45473 (size: 5.2 KB, free: 366.3 MB) 17/09/22 12:11:14 INFO TaskSetManager: Starting task 8.0 in stage 0.0 (TID 2, 192.168.10.12, executor 0, partition 8, NODE_LOCAL, 8748 bytes) 17/09/22 12:11:14 INFO TaskSetManager: Starting task 10.0 in stage 0.0 (TID 3, 192.168.10.12, executor 0, partition 10, NODE_LOCAL, 8748 bytes) 17/09/22 12:11:14 INFO TaskSetManager: Finished task 2.0 in stage 0.0 (TID 0) in 3684 ms on 192.168.10.12 (executor 0) (1/15) 17/09/22 12:11:14 INFO TaskSetManager: Finished task 4.0 in stage 0.0 (TID 1) in 3625 ms on 192.168.10.12 (executor 0) (2/15) 17/09/22 12:11:15 INFO TaskSetManager: Starting task 13.0 in stage 0.0 (TID 4, 192.168.10.12, executor 0, partition 13, NODE_LOCAL, 8748 bytes) 17/09/22 12:11:15 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 5, 192.168.10.12, executor 0, partition 0, NODE_LOCAL, 9036 bytes) 17/09/22 12:11:15 INFO TaskSetManager: Finished task 8.0 in stage 0.0 (TID 2) in 332 ms on 192.168.10.12 (executor 0) (3/15) 17/09/22 12:11:15 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.10.12:45473 (size: 5.8 KB, free: 366.3 MB) 17/09/22 12:11:15 INFO TaskSetManager: Finished task 10.0 in stage 0.0 (TID 3) in 311 ms on 192.168.10.12 (executor 0) (4/15) 17/09/22 12:11:15 INFO TaskSetManager: Starting task 4.0 in stage 1.0 (TID 6, 192.168.10.12, executor 0, partition 4, NODE_LOCAL, 9036 bytes) 17/09/22 12:11:15 INFO TaskSetManager: Finished task 13.0 in stage 0.0 (TID 4) in 123 ms on 192.168.10.12 (executor 0) (5/15) 17/09/22 12:11:15 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 25 17/09/22 12:11:16 INFO TaskSetManager: Starting task 7.0 in stage 1.0 (TID 7, 192.168.10.12, executor 0, partition 7, NODE_LOCAL, 9036 bytes) 17/09/22 12:11:16 INFO TaskSetManager: Finished task 4.0 in stage 1.0 (TID 6) in 970 ms on 192.168.10.12 (executor 0) (1/15) 17/09/22 12:11:16 INFO TaskSetManager: Starting task 8.0 in stage 1.0 (TID 8, 192.168.10.12, executor 0, partition 8, NODE_LOCAL, 9036 bytes) 17/09/22 12:11:16 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 5) in 1079 ms on 192.168.10.12 (executor 0) (2/15) 17/09/22 12:11:16 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 23 17/09/22 12:11:17 INFO TaskSetManager: Starting task 13.0 in stage 1.0 (TID 9, 192.168.10.12, executor 0, partition 13, NODE_LOCAL, 9036 bytes) 17/09/22 12:11:17 INFO TaskSetManager: Finished task 7.0 in stage 1.0 (TID 7) in 1104 ms on 192.168.10.12 (executor 0) (3/15) 17/09/22 12:11:17 INFO TaskSetManager: Finished task 8.0 in stage 1.0 (TID 8) in 1119 ms on 192.168.10.12 (executor 0) (4/15) 17/09/22 12:11:17 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 21 17/09/22 12:11:17 INFO TaskSetManager: Finished task 13.0 in stage 1.0 (TID 9) in 520 ms on 192.168.10.12 (executor 0) (5/15) 17/09/22 12:11:17 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 20 17/09/22 12:11:35 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 10, 192.168.10.12, executor 0, partition 0, ANY, 8748 bytes) 17/09/22 12:11:35 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 11, 192.168.10.12, executor 0, partition 1, ANY, 8748 bytes) 17/09/22 12:11:35 INFO TaskSetManager: Starting task 3.0 in stage 0.0 (TID 12, 192.168.10.12, executor 0, partition 3, ANY, 8748 bytes) 17/09/22 12:11:35 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 11) in 122 ms on 192.168.10.12 (executor 0) (6/15) 17/09/22 12:11:35 INFO TaskSetManager: Starting task 5.0 in stage 0.0 (TID 13, 192.168.10.12, executor 0, partition 5, ANY, 8748 bytes) 17/09/22 12:11:35 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 10) in 158 ms on 192.168.10.12 (executor 0) (7/15) 17/09/22 12:11:35 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 19 17/09/22 12:11:35 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 18 17/09/22 12:11:35 INFO TaskSetManager: Starting task 6.0 in stage 0.0 (TID 14, 192.168.10.12, executor 0, partition 6, ANY, 8748 bytes) 17/09/22 12:11:35 INFO TaskSetManager: Finished task 3.0 in stage 0.0 (TID 12) in 151 ms on 192.168.10.12 (executor 0) (8/15) 17/09/22 12:11:35 INFO TaskSetManager: Starting task 7.0 in stage 0.0 (TID 15, 192.168.10.12, executor 0, partition 7, ANY, 8748 bytes) 17/09/22 12:11:35 INFO TaskSetManager: Finished task 5.0 in stage 0.0 (TID 13) in 143 ms on 192.168.10.12 (executor 0) (9/15) 17/09/22 12:11:35 INFO TaskSetManager: Starting task 9.0 in stage 0.0 (TID 16, 192.168.10.12, executor 0, partition 9, ANY, 8748 bytes) 17/09/22 12:11:35 INFO TaskSetManager: Finished task 6.0 in stage 0.0 (TID 14) in 92 ms on 192.168.10.12 (executor 0) (10/15) 17/09/22 12:11:35 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 15 17/09/22 12:11:35 INFO TaskSetManager: Starting task 11.0 in stage 0.0 (TID 17, 192.168.10.12, executor 0, partition 11, ANY, 8748 bytes) 17/09/22 12:11:35 INFO TaskSetManager: Finished task 7.0 in stage 0.0 (TID 15) in 85 ms on 192.168.10.12 (executor 0) (11/15) 17/09/22 12:11:35 INFO TaskSetManager: Starting task 12.0 in stage 0.0 (TID 18, 192.168.10.12, executor 0, partition 12, ANY, 8748 bytes) 17/09/22 12:11:35 INFO TaskSetManager: Finished task 9.0 in stage 0.0 (TID 16) in 88 ms on 192.168.10.12 (executor 0) (12/15) 17/09/22 12:11:35 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 13 17/09/22 12:11:35 INFO TaskSetManager: Starting task 14.0 in stage 0.0 (TID 19, 192.168.10.12, executor 0, partition 14, ANY, 8748 bytes) 17/09/22 12:11:35 INFO TaskSetManager: Finished task 11.0 in stage 0.0 (TID 17) in 92 ms on 192.168.10.12 (executor 0) (13/15) 17/09/22 12:11:35 INFO TaskSetManager: Finished task 12.0 in stage 0.0 (TID 18) in 68 ms on 192.168.10.12 (executor 0) (14/15) 17/09/22 12:11:35 INFO TaskSetManager: Finished task 14.0 in stage 0.0 (TID 19) in 71 ms on 192.168.10.12 (executor 0) (15/15) 17/09/22 12:11:35 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 17/09/22 12:11:35 INFO DAGScheduler: ShuffleMapStage 0 (collect at /tmp/script_MJ2bwJO98J/5_concurrency_1_swissdata.py:54) finished in 74.900 s 17/09/22 12:11:35 INFO DAGScheduler: looking for newly runnable stages 17/09/22 12:11:35 INFO DAGScheduler: running: Set(ShuffleMapStage 1) 17/09/22 12:11:35 INFO DAGScheduler: waiting: Set(ResultStage 2) 17/09/22 12:11:35 INFO DAGScheduler: failed: Set() 17/09/22 12:11:35 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 10 17/09/22 12:11:37 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 20, 192.168.10.12, executor 0, partition 1, ANY, 9036 bytes) 17/09/22 12:11:37 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 21, 192.168.10.12, executor 0, partition 2, ANY, 9036 bytes) 17/09/22 12:11:37 INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID 22, 192.168.10.12, executor 0, partition 3, ANY, 9036 bytes) 17/09/22 12:11:37 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID 21) in 316 ms on 192.168.10.12 (executor 0) (6/15) 17/09/22 12:11:37 INFO TaskSetManager: Starting task 5.0 in stage 1.0 (TID 23, 192.168.10.12, executor 0, partition 5, ANY, 9036 bytes) 17/09/22 12:11:37 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 20) in 329 ms on 192.168.10.12 (executor 0) (7/15) 17/09/22 12:11:37 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 8 17/09/22 12:11:38 INFO TaskSetManager: Starting task 6.0 in stage 1.0 (TID 24, 192.168.10.12, executor 0, partition 6, ANY, 9036 bytes) 17/09/22 12:11:38 INFO TaskSetManager: Finished task 3.0 in stage 1.0 (TID 22) in 728 ms on 192.168.10.12 (executor 0) (8/15) 17/09/22 12:11:38 INFO TaskSetManager: Starting task 9.0 in stage 1.0 (TID 25, 192.168.10.12, executor 0, partition 9, ANY, 9036 bytes) 17/09/22 12:11:38 INFO TaskSetManager: Finished task 5.0 in stage 1.0 (TID 23) in 734 ms on 192.168.10.12 (executor 0) (9/15) 17/09/22 12:11:38 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 6 17/09/22 12:11:38 INFO TaskSetManager: Starting task 10.0 in stage 1.0 (TID 26, 192.168.10.12, executor 0, partition 10, ANY, 9036 bytes) 17/09/22 12:11:38 INFO TaskSetManager: Finished task 9.0 in stage 1.0 (TID 25) in 380 ms on 192.168.10.12 (executor 0) (10/15) 17/09/22 12:11:38 INFO TaskSetManager: Starting task 11.0 in stage 1.0 (TID 27, 192.168.10.12, executor 0, partition 11, ANY, 9036 bytes) 17/09/22 12:11:38 INFO TaskSetManager: Finished task 6.0 in stage 1.0 (TID 24) in 422 ms on 192.168.10.12 (executor 0) (11/15) 17/09/22 12:11:38 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 4 17/09/22 12:11:39 INFO TaskSetManager: Starting task 12.0 in stage 1.0 (TID 28, 192.168.10.12, executor 0, partition 12, ANY, 9036 bytes) 17/09/22 12:11:39 INFO TaskSetManager: Starting task 14.0 in stage 1.0 (TID 29, 192.168.10.12, executor 0, partition 14, ANY, 9036 bytes) 17/09/22 12:11:39 INFO TaskSetManager: Finished task 10.0 in stage 1.0 (TID 26) in 634 ms on 192.168.10.12 (executor 0) (12/15) 17/09/22 12:11:39 INFO TaskSetManager: Finished task 11.0 in stage 1.0 (TID 27) in 614 ms on 192.168.10.12 (executor 0) (13/15) 17/09/22 12:11:39 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 2 17/09/22 12:11:40 INFO TaskSetManager: Finished task 14.0 in stage 1.0 (TID 29) in 628 ms on 192.168.10.12 (executor 0) (14/15) 17/09/22 12:11:40 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 1 17/09/22 12:11:40 INFO TaskSetManager: Finished task 12.0 in stage 1.0 (TID 28) in 660 ms on 192.168.10.12 (executor 0) (15/15) 17/09/22 12:11:40 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 17/09/22 12:11:40 INFO DAGScheduler: ShuffleMapStage 1 (collect at /tmp/script_MJ2bwJO98J/5_concurrency_1_swissdata.py:54) finished in 78.999 s 17/09/22 12:11:40 INFO DAGScheduler: looking for newly runnable stages 17/09/22 12:11:40 INFO DAGScheduler: running: Set() 17/09/22 12:11:40 INFO DAGScheduler: waiting: Set(ResultStage 2) 17/09/22 12:11:40 INFO DAGScheduler: failed: Set() 17/09/22 12:11:40 INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[14] at collect at /tmp/script_MJ2bwJO98J/5_concurrency_1_swissdata.py:54), which has no missing parents 17/09/22 12:11:40 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 26.6 KB, free 246.8 MB) 17/09/22 12:11:40 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 9.9 KB, free 246.8 MB) 17/09/22 12:11:40 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.10.10:36563 (size: 9.9 KB, free: 246.9 MB) 17/09/22 12:11:40 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006 17/09/22 12:11:40 INFO DAGScheduler: Submitting 12 missing tasks from ResultStage 2 (MapPartitionsRDD[14] at collect at /tmp/script_MJ2bwJO98J/5_concurrency_1_swissdata.py:54) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11)) 17/09/22 12:11:40 INFO TaskSchedulerImpl: Adding task set 2.0 with 12 tasks 17/09/22 12:11:40 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 30, 192.168.10.12, executor 0, partition 0, NODE_LOCAL, 5076 bytes) 17/09/22 12:11:40 INFO TaskSetManager: Starting task 1.0 in stage 2.0 (TID 31, 192.168.10.12, executor 0, partition 1, NODE_LOCAL, 5076 bytes) 17/09/22 12:11:40 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.10.12:45473 (size: 9.9 KB, free: 366.3 MB) 17/09/22 12:11:40 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 192.168.10.12:42630 17/09/22 12:11:40 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 304 bytes 17/09/22 12:11:40 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 192.168.10.12:42630 17/09/22 12:11:40 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 1 is 303 bytes 17/09/22 12:11:41 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 2 17/09/22 12:11:41 INFO ExecutorAllocationManager: Requesting 1 new executor because tasks are backlogged (new desired total will be 2) 17/09/22 12:11:41 INFO TaskSetManager: Starting task 2.0 in stage 2.0 (TID 32, 192.168.10.12, executor 0, partition 2, NODE_LOCAL, 5076 bytes) 17/09/22 12:11:41 INFO TaskSetManager: Finished task 1.0 in stage 2.0 (TID 31) in 993 ms on 192.168.10.12 (executor 0) (1/12) 17/09/22 12:11:41 INFO TaskSetManager: Starting task 3.0 in stage 2.0 (TID 33, 192.168.10.12, executor 0, partition 3, NODE_LOCAL, 5076 bytes) 17/09/22 12:11:41 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 30) in 1017 ms on 192.168.10.12 (executor 0) (2/12) 17/09/22 12:11:41 INFO TaskSetManager: Starting task 4.0 in stage 2.0 (TID 34, 192.168.10.12, executor 0, partition 4, NODE_LOCAL, 5076 bytes) 17/09/22 12:11:41 INFO TaskSetManager: Finished task 3.0 in stage 2.0 (TID 33) in 147 ms on 192.168.10.12 (executor 0) (3/12) 17/09/22 12:11:41 INFO TaskSetManager: Starting task 5.0 in stage 2.0 (TID 35, 192.168.10.12, executor 0, partition 5, NODE_LOCAL, 5076 bytes) 17/09/22 12:11:41 INFO TaskSetManager: Finished task 2.0 in stage 2.0 (TID 32) in 353 ms on 192.168.10.12 (executor 0) (4/12) 17/09/22 12:11:41 INFO TaskSetManager: Starting task 6.0 in stage 2.0 (TID 36, 192.168.10.12, executor 0, partition 6, NODE_LOCAL, 5076 bytes) 17/09/22 12:11:41 INFO TaskSetManager: Finished task 4.0 in stage 2.0 (TID 34) in 315 ms on 192.168.10.12 (executor 0) (5/12) 17/09/22 12:11:41 INFO TaskSetManager: Finished task 6.0 in stage 2.0 (TID 36) in 189 ms on 192.168.10.12 (executor 0) (6/12) 17/09/22 12:11:41 INFO TaskSetManager: Starting task 7.0 in stage 2.0 (TID 37, 192.168.10.12, executor 0, partition 7, NODE_LOCAL, 5076 bytes) 17/09/22 12:11:41 INFO TaskSetManager: Starting task 8.0 in stage 2.0 (TID 38, 192.168.10.12, executor 0, partition 8, NODE_LOCAL, 5076 bytes) 17/09/22 12:11:41 INFO TaskSetManager: Finished task 5.0 in stage 2.0 (TID 35) in 359 ms on 192.168.10.12 (executor 0) (7/12) 17/09/22 12:11:41 INFO TaskSetManager: Starting task 9.0 in stage 2.0 (TID 39, 192.168.10.12, executor 0, partition 9, NODE_LOCAL, 5076 bytes) 17/09/22 12:11:41 INFO TaskSetManager: Finished task 7.0 in stage 2.0 (TID 37) in 107 ms on 192.168.10.12 (executor 0) (8/12) 17/09/22 12:11:41 INFO TaskSetManager: Starting task 10.0 in stage 2.0 (TID 40, 192.168.10.12, executor 0, partition 10, NODE_LOCAL, 5076 bytes) 17/09/22 12:11:41 INFO TaskSetManager: Finished task 8.0 in stage 2.0 (TID 38) in 147 ms on 192.168.10.12 (executor 0) (9/12) 17/09/22 12:11:42 INFO TaskSetManager: Starting task 11.0 in stage 2.0 (TID 41, 192.168.10.12, executor 0, partition 11, NODE_LOCAL, 5076 bytes) 17/09/22 12:11:42 INFO TaskSetManager: Finished task 9.0 in stage 2.0 (TID 39) in 156 ms on 192.168.10.12 (executor 0) (10/12) 17/09/22 12:11:42 INFO TaskSetManager: Finished task 10.0 in stage 2.0 (TID 40) in 166 ms on 192.168.10.12 (executor 0) (11/12) 17/09/22 12:11:42 INFO TaskSetManager: Finished task 11.0 in stage 2.0 (TID 41) in 158 ms on 192.168.10.12 (executor 0) (12/12) 17/09/22 12:11:42 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 17/09/22 12:11:42 INFO DAGScheduler: ResultStage 2 (collect at /tmp/script_MJ2bwJO98J/5_concurrency_1_swissdata.py:54) finished in 2.045 s 17/09/22 12:11:42 INFO DAGScheduler: Job 0 finished: collect at /tmp/script_MJ2bwJO98J/5_concurrency_1_swissdata.py:54, took 82.438661 s 17/09/22 12:11:42 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 0 17/09/22 12:11:43 INFO BlockManagerInfo: Removed broadcast_2_piece0 on 192.168.10.12:45473 in memory (size: 9.9 KB, free: 366.3 MB) 17/09/22 12:11:43 INFO BlockManagerInfo: Removed broadcast_2_piece0 on 192.168.10.10:36563 in memory (size: 9.9 KB, free: 246.9 MB) Printing 10 first results Row(room_id=u'7284708', country=u'Switzerland', city=u'Aeugst am Albis', room_type=u'Private room', bedrooms=u'1.0', bathrooms=None, price=u'42.0', reviews=u'0', overall_satisfaction=u'0.0', latitude=u'47.281159', longitude=u'8.478718', latitude=None, longitude=None, population=None, region=None) Row(room_id=u'18419767', country=u'Switzerland', city=u'Affoltern am Albis', room_type=u'Entire home/apt', bedrooms=u'1.0', bathrooms=None, price=u'48.0', reviews=u'3', overall_satisfaction=u'4.0', latitude=u'47.27947', longitude=u'8.450784', latitude=u'47.281224', longitude=u'8.45346', population=None, region=u'01') Row(room_id=u'15761219', country=u'Switzerland', city=u'Affoltern am Albis', room_type=u'Entire home/apt', bedrooms=u'1.0', bathrooms=None, price=u'75.0', reviews=u'1', overall_satisfaction=u'0.0', latitude=u'47.272614', longitude=u'8.449907', latitude=u'47.281224', longitude=u'8.45346', population=None, region=u'01') Row(room_id=u'17666068', country=u'Switzerland', city=u'Affoltern am Albis', room_type=u'Entire home/apt', bedrooms=u'2.0', bathrooms=None, price=u'129.0', reviews=u'2', overall_satisfaction=u'0.0', latitude=u'47.290323', longitude=u'8.442326', latitude=u'47.281224', longitude=u'8.45346', population=None, region=u'01') Row(room_id=u'5134515', country=u'Switzerland', city=u'Affoltern am Albis', room_type=u'Entire home/apt', bedrooms=u'2.0', bathrooms=None, price=u'129.0', reviews=u'4', overall_satisfaction=u'4.5', latitude=u'47.290284', longitude=u'8.44185', latitude=u'47.281224', longitude=u'8.45346', population=None, region=u'01') Row(room_id=u'19144898', country=u'Switzerland', city=u'Affoltern am Albis', room_type=u'Entire home/apt', bedrooms=u'1.0', bathrooms=None, price=u'48.0', reviews=u'0', overall_satisfaction=u'0.0', latitude=u'47.278981', longitude=u'8.447358', latitude=u'47.281224', longitude=u'8.45346', population=None, region=u'01') Row(room_id=u'1853391', country=u'Switzerland', city=u'Affoltern am Albis', room_type=u'Private room', bedrooms=u'1.0', bathrooms=None, price=u'102.0', reviews=u'11', overall_satisfaction=u'4.5', latitude=u'47.277891', longitude=u'8.449805', latitude=u'47.281224', longitude=u'8.45346', population=None, region=u'01') Row(room_id=u'6895426', country=u'Switzerland', city=u'Affoltern am Albis', room_type=u'Private room', bedrooms=u'1.0', bathrooms=None, price=u'59.0', reviews=u'16', overall_satisfaction=u'4.5', latitude=u'47.27642', longitude=u'8.441259', latitude=u'47.281224', longitude=u'8.45346', population=None, region=u'01') Row(room_id=u'19026774', country=u'Switzerland', city=u'Affoltern am Albis', room_type=u'Entire home/apt', bedrooms=u'4.0', bathrooms=None, price=u'193.0', reviews=u'0', overall_satisfaction=u'0.0', latitude=u'47.284338', longitude=u'8.453748', latitude=u'47.281224', longitude=u'8.45346', population=None, region=u'01') Row(room_id=u'12013387', country=u'Switzerland', city=u'Affoltern am Albis', room_type=u'Entire home/apt', bedrooms=u'2.0', bathrooms=None, price=u'78.0', reviews=u'6', overall_satisfaction=u'5.0', latitude=u'47.276212', longitude=u'8.449886', latitude=u'47.281224', longitude=u'8.45346', population=None, region=u'01') Computed 27744 positions (from collected list) 17/09/22 12:11:44 INFO SparkContext: Invoking stop() from shutdown hook 17/09/22 12:11:44 INFO SparkUI: Stopped Spark web UI at http://192.168.10.10:4040 17/09/22 12:11:44 INFO MesosCoarseGrainedSchedulerBackend: Shutting down all executors 17/09/22 12:11:44 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down 17/09/22 12:11:45 INFO MesosCoarseGrainedSchedulerBackend: Mesos task 0 is now TASK_FINISHED I0922 12:11:45.209002 30074 sched.cpp:2021] Asked to stop the driver I0922 12:11:45.209233 29199 sched.cpp:1203] Stopping framework b192e864-8a9b-4ffc-94ab-953d2b929bd2-0013 17/09/22 12:11:45 INFO MesosCoarseGrainedSchedulerBackend: driver.run() returned with code DRIVER_STOPPED 17/09/22 12:11:45 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 17/09/22 12:11:45 INFO MemoryStore: MemoryStore cleared 17/09/22 12:11:45 INFO BlockManager: BlockManager stopped 17/09/22 12:11:45 INFO BlockManagerMaster: BlockManagerMaster stopped 17/09/22 12:11:45 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 17/09/22 12:11:45 INFO SparkContext: Successfully stopped SparkContext 17/09/22 12:11:45 INFO ShutdownHookManager: Shutdown hook called 17/09/22 12:11:45 INFO ShutdownHookManager: Deleting directory /tmp/spark-a3636a62-4957-4e15-af0a-c158d60f6433 17/09/22 12:11:45 INFO ShutdownHookManager: Deleting directory /tmp/spark-a3636a62-4957-4e15-af0a-c158d60f6433/pyspark-d2855cc8-de12-4a3a-a0e0-13e538843806