Running test 1 4 === Created temporary folder : /tmp/script_JmzbrTSWNK === Copying script to temporary folder === DONE === Executing Script + curl -XDELETE 'localhost:9200/bank?pretty' + curl -XDELETE 'localhost:9200/shakespeare?pretty' + curl -XDELETE 'localhost:9200/apache-logs-*?pretty' + curl -XDELETE 'localhost:9200/swiss-*?pretty' + mkdir -p /home/mes/input_data + cd /home/mes/input_data + [[ ! -f logs.jsonl ]] + set -e + curl -XPUT 'localhost:9200/apache-logs-2015.05.18?pretty' -H 'Content-Type: application/json' '-d { "settings": { "number_of_shards" : 3, "number_of_replicas" : 1 }, "mappings": { "log": { "properties": { "geo": { "properties": { "coordinates": { "type": "geo_point" } } } } } } } ' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0{ "acknowledged" : true, "shards_acknowledged" : true, "index" : "apache-logs-2015.05.18" } 100 393 100 98 100 295 706 2125 --:--:-- --:--:-- --:--:-- 2137 + curl -XPUT 'localhost:9200/apache-logs-2015.05.19?pretty' -H 'Content-Type: application/json' '-d { "settings": { "number_of_shards" : 3, "number_of_replicas" : 1 }, "mappings": { "log": { "properties": { "geo": { "properties": { "coordinates": { "type": "geo_point" } } } } } } } ' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 393 100 98 100 295 669 2016 --:--:-- --:--:-- --:--:-- 2020 { "acknowledged" : true, "shards_acknowledged" : true, "index" : "apache-logs-2015.05.19" } + curl -XPUT 'localhost:9200/apache-logs-2015.05.20?pretty' -H 'Content-Type: application/json' '-d { "settings": { "number_of_shards" : 3, "number_of_replicas" : 1 }, "mappings": { "log": { "properties": { "geo": { "properties": { "coordinates": { "type": "geo_point" } } } } } } } ' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 393 100 98 100 295 503 1515 --:--:-- --:-{ "acknowledged" : true, "shards_acknowledged" : true, "index" : "apache-logs-2015.05.20" } -:-- --:--:-- 1520 + curl -XPUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{ "index.search.slowlog.threshold.query.debug" : "0s", "index.search.slowlog.threshold.fetch.debug" : "0s" }' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406} 100 209 100 97 100 112 11517 13298 --:--:-- --:--:-- --:--:-- 14000 + set +e + curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/_bulk?pretty' --data-binary @logs.jsonl + [[ 0 != 0 ]] + set +x Warning: Ignoring non-spark config property: es.nodes.data.only=false 17/09/21 08:14:35 INFO SparkContext: Running Spark version 2.2.0 17/09/21 08:14:35 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17/09/21 08:14:35 INFO SparkContext: Submitted application: ESTest_1_4 17/09/21 08:14:35 INFO SecurityManager: Changing view acls to: mes 17/09/21 08:14:35 INFO SecurityManager: Changing modify acls to: mes 17/09/21 08:14:35 INFO SecurityManager: Changing view acls groups to: 17/09/21 08:14:35 INFO SecurityManager: Changing modify acls groups to: 17/09/21 08:14:35 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(mes); groups with view permissions: Set(); users with modify permissions: Set(mes); groups with modify permissions: Set() 17/09/21 08:14:36 INFO Utils: Successfully started service 'sparkDriver' on port 35471. 17/09/21 08:14:36 INFO SparkEnv: Registering MapOutputTracker 17/09/21 08:14:36 INFO SparkEnv: Registering BlockManagerMaster 17/09/21 08:14:36 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 17/09/21 08:14:36 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up 17/09/21 08:14:36 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-107a6f6b-58f2-4f0a-a049-f5755441da6b 17/09/21 08:14:36 INFO MemoryStore: MemoryStore started with capacity 246.9 MB 17/09/21 08:14:36 INFO SparkEnv: Registering OutputCommitCoordinator 17/09/21 08:14:36 INFO Utils: Successfully started service 'SparkUI' on port 4040. 17/09/21 08:14:36 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.10.10:4040 17/09/21 08:14:36 INFO SparkContext: Added file file:/tmp/script_JmzbrTSWNK/1_nominal_4_test_apache-logs_df.py at spark://192.168.10.10:35471/files/1_nominal_4_test_apache-logs_df.py with timestamp 1505981676767 17/09/21 08:14:36 INFO Utils: Copying /tmp/script_JmzbrTSWNK/1_nominal_4_test_apache-logs_df.py to /tmp/spark-e89a56d6-2e6d-4b3b-9007-f876bc449be0/userFiles-b3edc292-96b1-4f69-9dfe-0ab159fffb0e/1_nominal_4_test_apache-logs_df.py 2017-09-21 08:14:37,170:26872(0x7f9c8904f700):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8 2017-09-21 08:14:37,170:26872(0x7f9c8904f700):ZOO_INFO@log_env@730: Client environment:host.name=mes_master 2017-09-21 08:14:37,170:26872(0x7f9c8904f700):ZOO_INFO@log_env@737: Client environment:os.name=Linux 2017-09-21 08:14:37,170:26872(0x7f9c8904f700):ZOO_INFO@log_env@738: Client environment:os.arch=4.9.0-3-amd64 2017-09-21 08:14:37,170:26872(0x7f9c8904f700):ZOO_INFO@log_env@739: Client environment:os.version=#1 SMP Debian 4.9.30-2+deb9u3 (2017-08-06) 2017-09-21 08:14:37,170:26872(0x7f9c8904f700):ZOO_INFO@log_env@747: Client environment:user.name=mes 2017-09-21 08:14:37,170:26872(0x7f9c8904f700):ZOO_INFO@log_env@755: Client environment:user.home=/home/mes 2017-09-21 08:14:37,170:26872(0x7f9c8904f700):ZOO_INFO@log_env@767: Client environment:user.dir=/tmp/script_JmzbrTSWNK 2017-09-21 08:14:37,170:26872(0x7f9c8904f700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=192.168.10.10:2181 sessionTimeout=10000 watcher=0x7f9c91e6b712 sessionId=0 sessionPasswd= context=0x7f9c94006468 flags=0 I0921 08:14:37.173612 26977 sched.cpp:232] Version: 1.3.0 2017-09-21 08:14:37,196:26872(0x7f9c85f48700):ZOO_INFO@check_events@1728: initiated connection to server [192.168.10.10:2181] 2017-09-21 08:14:37,196:26872(0x7f9c85f48700):ZOO_INFO@check_events@1775: session establishment complete on server [192.168.10.10:2181], sessionId=0x15e9eecb1870042, negotiated timeout=10000 I0921 08:14:37.198035 26968 group.cpp:340] Group process (zookeeper-group(1)@192.168.10.10:42717) connected to ZooKeeper I0921 08:14:37.198191 26968 group.cpp:830] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0) I0921 08:14:37.198221 26968 group.cpp:418] Trying to create path '/mesos' in ZooKeeper I0921 08:14:37.208175 26969 detector.cpp:152] Detected a new leader: (id='15') I0921 08:14:37.209451 26968 group.cpp:699] Trying to get '/mesos/json.info_0000000015' in ZooKeeper I0921 08:14:37.215078 26968 zookeeper.cpp:262] A new leading master (UPID=master@192.168.10.10:5050) is detected I0921 08:14:37.217200 26970 sched.cpp:336] New master detected at master@192.168.10.10:5050 I0921 08:14:37.225726 26970 sched.cpp:352] No credentials provided. Attempting to register without authentication I0921 08:14:37.245182 26970 sched.cpp:759] Framework registered with aa404b40-8fd8-470d-94ff-87f1621566a7-0010 17/09/21 08:14:37 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 38955. 17/09/21 08:14:37 INFO NettyBlockTransferService: Server created on 192.168.10.10:38955 17/09/21 08:14:37 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 17/09/21 08:14:37 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.10.10, 38955, None) 17/09/21 08:14:37 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.10.10:38955 with 246.9 MB RAM, BlockManagerId(driver, 192.168.10.10, 38955, None) 17/09/21 08:14:37 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.10.10, 38955, None) 17/09/21 08:14:37 INFO BlockManager: external shuffle service port = 7337 17/09/21 08:14:37 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.10.10, 38955, None) 17/09/21 08:14:37 INFO EventLoggingListener: Logging events to file:/var/lib/spark/eventlog/aa404b40-8fd8-470d-94ff-87f1621566a7-0010 17/09/21 08:14:37 INFO Utils: Using initial executors = 0, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances 17/09/21 08:14:37 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 0 17/09/21 08:14:37 INFO MesosCoarseGrainedSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0 17/09/21 08:14:37 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/tmp/script_JmzbrTSWNK/spark-warehouse'). 17/09/21 08:14:37 INFO SharedState: Warehouse path is 'file:/tmp/script_JmzbrTSWNK/spark-warehouse'. 17/09/21 08:14:38 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint 17/09/21 08:14:38 INFO Version: Elasticsearch Hadoop v6.0.0-beta2 [66f16fdd93] 17/09/21 08:14:41 INFO CodeGenerator: Code generated in 451.20363 ms 17/09/21 08:14:41 INFO ScalaEsRowRDD: Reading from [apache-logs-*] 17/09/21 08:14:41 INFO SparkContext: Starting job: collect at /tmp/script_JmzbrTSWNK/1_nominal_4_test_apache-logs_df.py:44 17/09/21 08:14:41 INFO DAGScheduler: Got job 0 (collect at /tmp/script_JmzbrTSWNK/1_nominal_4_test_apache-logs_df.py:44) with 9 output partitions 17/09/21 08:14:41 INFO DAGScheduler: Final stage: ResultStage 0 (collect at /tmp/script_JmzbrTSWNK/1_nominal_4_test_apache-logs_df.py:44) 17/09/21 08:14:41 INFO DAGScheduler: Parents of final stage: List() 17/09/21 08:14:41 INFO DAGScheduler: Missing parents: List() 17/09/21 08:14:41 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[3] at collect at /tmp/script_JmzbrTSWNK/1_nominal_4_test_apache-logs_df.py:44), which has no missing parents 17/09/21 08:14:41 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 21.5 KB, free 246.9 MB) 17/09/21 08:14:41 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 7.8 KB, free 246.9 MB) 17/09/21 08:14:41 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.10.10:38955 (size: 7.8 KB, free: 246.9 MB) 17/09/21 08:14:41 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006 17/09/21 08:14:41 INFO DAGScheduler: Submitting 9 missing tasks from ResultStage 0 (MapPartitionsRDD[3] at collect at /tmp/script_JmzbrTSWNK/1_nominal_4_test_apache-logs_df.py:44) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8)) 17/09/21 08:14:41 INFO TaskSchedulerImpl: Adding task set 0.0 with 9 tasks 17/09/21 08:14:42 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 1 17/09/21 08:14:42 INFO ExecutorAllocationManager: Requesting 1 new executor because tasks are backlogged (new desired total will be 1) 17/09/21 08:14:43 WARN MesosCoarseGrainedSchedulerBackend: Unable to parse into a key:value label for the task. 17/09/21 08:14:43 INFO MesosCoarseGrainedSchedulerBackend: Mesos task 0 is now TASK_RUNNING 17/09/21 08:14:43 INFO TransportClientFactory: Successfully created connection to /192.168.10.11:7337 after 25 ms (0 ms spent in bootstraps) 17/09/21 08:14:43 INFO MesosExternalShuffleClient: Successfully registered app aa404b40-8fd8-470d-94ff-87f1621566a7-0010 with external shuffle service. 17/09/21 08:14:43 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 3 17/09/21 08:14:43 INFO ExecutorAllocationManager: Requesting 2 new executors because tasks are backlogged (new desired total will be 3) 17/09/21 08:14:44 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 7 17/09/21 08:14:44 INFO ExecutorAllocationManager: Requesting 4 new executors because tasks are backlogged (new desired total will be 7) 17/09/21 08:14:45 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 9 17/09/21 08:14:45 INFO ExecutorAllocationManager: Requesting 2 new executors because tasks are backlogged (new desired total will be 9) 17/09/21 08:14:45 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.10.11:36406) with ID 0 17/09/21 08:14:45 INFO ExecutorAllocationManager: New executor 0 has registered (new total is 1) 17/09/21 08:14:45 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 192.168.10.11, executor 0, partition 0, NODE_LOCAL, 10332 bytes) 17/09/21 08:14:45 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, 192.168.10.11, executor 0, partition 1, NODE_LOCAL, 10332 bytes) 17/09/21 08:14:45 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.10.11:37263 with 366.3 MB RAM, BlockManagerId(0, 192.168.10.11, 37263, None) 17/09/21 08:14:46 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.10.11:37263 (size: 7.8 KB, free: 366.3 MB) 17/09/21 08:14:48 INFO TaskSetManager: Starting task 3.0 in stage 0.0 (TID 2, 192.168.10.11, executor 0, partition 3, NODE_LOCAL, 10332 bytes) 17/09/21 08:14:48 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 2733 ms on 192.168.10.11 (executor 0) (1/9) 17/09/21 08:14:48 INFO TaskSetManager: Starting task 4.0 in stage 0.0 (TID 3, 192.168.10.11, executor 0, partition 4, NODE_LOCAL, 10332 bytes) 17/09/21 08:14:48 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 2938 ms on 192.168.10.11 (executor 0) (2/9) 17/09/21 08:14:48 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 7 17/09/21 08:14:49 WARN MesosCoarseGrainedSchedulerBackend: Unable to parse into a key:value label for the task. 17/09/21 08:14:49 WARN MesosCoarseGrainedSchedulerBackend: Unable to parse into a key:value label for the task. 17/09/21 08:14:49 INFO TaskSetManager: Starting task 6.0 in stage 0.0 (TID 4, 192.168.10.11, executor 0, partition 6, NODE_LOCAL, 10332 bytes) 17/09/21 08:14:49 INFO TaskSetManager: Finished task 3.0 in stage 0.0 (TID 2) in 729 ms on 192.168.10.11 (executor 0) (3/9) 17/09/21 08:14:49 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 6 17/09/21 08:14:49 INFO TaskSetManager: Starting task 7.0 in stage 0.0 (TID 5, 192.168.10.11, executor 0, partition 7, NODE_LOCAL, 10332 bytes) 17/09/21 08:14:49 INFO TaskSetManager: Finished task 4.0 in stage 0.0 (TID 3) in 670 ms on 192.168.10.11 (executor 0) (4/9) 17/09/21 08:14:49 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 5 17/09/21 08:14:49 INFO MesosCoarseGrainedSchedulerBackend: Mesos task 1 is now TASK_RUNNING 17/09/21 08:14:49 INFO TransportClientFactory: Successfully created connection to /192.168.10.12:7337 after 39 ms (0 ms spent in bootstraps) 17/09/21 08:14:49 INFO MesosExternalShuffleClient: Successfully registered app aa404b40-8fd8-470d-94ff-87f1621566a7-0010 with external shuffle service. 17/09/21 08:14:50 INFO MesosCoarseGrainedSchedulerBackend: Mesos task 2 is now TASK_RUNNING 17/09/21 08:14:50 INFO TransportClientFactory: Successfully created connection to /192.168.10.10:7337 after 8 ms (0 ms spent in bootstraps) 17/09/21 08:14:50 INFO TaskSetManager: Finished task 7.0 in stage 0.0 (TID 5) in 804 ms on 192.168.10.11 (executor 0) (5/9) 17/09/21 08:14:50 INFO MesosExternalShuffleClient: Successfully registered app aa404b40-8fd8-470d-94ff-87f1621566a7-0010 with external shuffle service. 17/09/21 08:14:50 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 4 17/09/21 08:14:50 INFO TaskSetManager: Finished task 6.0 in stage 0.0 (TID 4) in 1276 ms on 192.168.10.11 (executor 0) (6/9) 17/09/21 08:14:50 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 3 17/09/21 08:14:53 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.10.12:35148) with ID 1 17/09/21 08:14:53 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 6, 192.168.10.12, executor 1, partition 2, NODE_LOCAL, 10332 bytes) 17/09/21 08:14:53 INFO TaskSetManager: Starting task 5.0 in stage 0.0 (TID 7, 192.168.10.12, executor 1, partition 5, NODE_LOCAL, 10332 bytes) 17/09/21 08:14:53 INFO ExecutorAllocationManager: New executor 1 has registered (new total is 2) 17/09/21 08:14:53 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.10.12:40771 with 366.3 MB RAM, BlockManagerId(1, 192.168.10.12, 40771, None) 17/09/21 08:14:53 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.10.10:54954) with ID 2 17/09/21 08:14:53 INFO TaskSetManager: Starting task 8.0 in stage 0.0 (TID 8, 192.168.10.10, executor 2, partition 8, NODE_LOCAL, 10332 bytes) 17/09/21 08:14:53 INFO ExecutorAllocationManager: New executor 2 has registered (new total is 3) 17/09/21 08:14:53 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.10.10:39297 with 366.3 MB RAM, BlockManagerId(2, 192.168.10.10, 39297, None) 17/09/21 08:14:53 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.10.12:40771 (size: 7.8 KB, free: 366.3 MB) 17/09/21 08:14:54 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.10.10:39297 (size: 7.8 KB, free: 366.3 MB) 17/09/21 08:14:58 INFO TaskSetManager: Finished task 2.0 in stage 0.0 (TID 6) in 4991 ms on 192.168.10.12 (executor 1) (7/9) 17/09/21 08:14:58 INFO TaskSetManager: Finished task 5.0 in stage 0.0 (TID 7) in 5039 ms on 192.168.10.12 (executor 1) (8/9) 17/09/21 08:14:58 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 1 17/09/21 08:15:00 INFO TaskSetManager: Finished task 8.0 in stage 0.0 (TID 8) in 6607 ms on 192.168.10.10 (executor 2) (9/9) 17/09/21 08:15:00 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 0 17/09/21 08:15:00 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 17/09/21 08:15:00 INFO DAGScheduler: ResultStage 0 (collect at /tmp/script_JmzbrTSWNK/1_nominal_4_test_apache-logs_df.py:44) finished in 18.325 s 17/09/21 08:15:00 INFO DAGScheduler: Job 0 finished: collect at /tmp/script_JmzbrTSWNK/1_nominal_4_test_apache-logs_df.py:44, took 18.857939 s Printing 10 first results Row(@message=u'149.34.15.231 - - [2015-05-20T13:34:26.203Z] "GET /styles/app.css HTTP/1.1" 200 6474 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Gecko/20110421 Firefox/6.0a1"', @tags=[u'success', u'info'], @timestamp=datetime.datetime(2015, 5, 20, 13, 34, 26, 203000), @version=u'1', agent=u'Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Gecko/20110421 Firefox/6.0a1', bytes=6474, clientip=u'149.34.15.231', extension=u'css', geo=Row(coordinates=Row(lat=40.56186833, lon=-81.07748611), dest=u'US', src=u'IN', srcdest=u'IN:US'), headings=[u'

mike-adams

', u'http://www.slate.com/success/kevin-p-chilton'], host=u'cdn.theacademyofperformingartsandscience.org', ip=u'149.34.15.231', links=[u'leonid-kizim@www.slate.com', u'http://nytimes.com/security/philippe-perrin', u'www.twitter.com'], machine=Row(os=u'win 7', ram=32212254720), memory=None, phpmemory=None, referer=u'http://www.slate.com/success/jing-haipeng', request=u'/styles/app.css', response=u'200', spaces=u'this is a thing with lots of spaces wwwwoooooo', url=u'https://cdn.theacademyofperformingartsandscience.org/styles/app.css', utc_time=datetime.datetime(2015, 5, 20, 13, 34, 26, 203000), xss=u'') Row(@message=u'198.208.197.202 - - [2015-05-20T08:46:13.123Z] "GET /uploads/marc-garneau.jpg HTTP/1.1" 404 8311 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"', @tags=[u'success', u'security'], @timestamp=datetime.datetime(2015, 5, 20, 8, 46, 13, 123000), @version=u'1', agent=u'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)', bytes=8311, clientip=u'198.208.197.202', extension=u'jpg', geo=Row(coordinates=Row(lat=32.07345972, lon=-93.76551889), dest=u'FI', src=u'IR', srcdest=u'IR:FI'), headings=[u'

gemini-9a

', u'http://www.slate.com/success/donald-peterson'], host=u'media-for-the-masses.theacademyofperformingartsandscience.org', ip=u'198.208.197.202', links=[u'thomas-akers@www.slate.com', u'http://nytimes.com/info/mark-c-lee', u'www.twitter.com'], machine=Row(os=u'osx', ram=20401094656), memory=None, phpmemory=None, referer=u'http://nytimes.com/success/konstantin-feoktistov', request=u'/uploads/marc-garneau.jpg', response=u'404', spaces=u'this is a thing with lots of spaces wwwwoooooo', url=u'https://media-for-the-masses.theacademyofperformingartsandscience.org/uploads/marc-garneau.jpg', utc_time=datetime.datetime(2015, 5, 20, 8, 46, 13, 123000), xss=u'') Row(@message=u'60.138.81.225 - - [2015-05-20T11:02:49.427Z] "GET /uploads/andrei-borisenko.jpg HTTP/1.1" 200 5863 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"', @tags=[u'error', u'info'], @timestamp=datetime.datetime(2015, 5, 20, 11, 2, 49, 427000), @version=u'1', agent=u'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)', bytes=5863, clientip=u'60.138.81.225', extension=u'jpg', geo=Row(coordinates=Row(lat=35.38507528, lon=-87.96752833), dest=u'CN', src=u'US', srcdest=u'US:CN'), headings=[u'

joe-h-engle

', u'http://www.slate.com/success/maurizio-cheli'], host=u'media-for-the-masses.theacademyofperformingartsandscience.org', ip=u'60.138.81.225', links=[u'mark-polansky@www.slate.com', u'http://twitter.com/info/rex-walheim', u'www.twitter.com'], machine=Row(os=u'win 8', ram=18253611008), memory=None, phpmemory=None, referer=u'http://twitter.com/success/apollo-15', request=u'/uploads/andrei-borisenko.jpg', response=u'200', spaces=u'this is a thing with lots of spaces wwwwoooooo', url=u'https://media-for-the-masses.theacademyofperformingartsandscience.org/uploads/andrei-borisenko.jpg', utc_time=datetime.datetime(2015, 5, 20, 11, 2, 49, 427000), xss=u'') Row(@message=u'228.105.101.199 - - [2015-05-20T11:21:11.281Z] "GET /uploads/kenneth-cockrell.jpg HTTP/1.1" 200 8292 "-" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.50 Safari/534.24"', @tags=[u'success', u'info'], @timestamp=datetime.datetime(2015, 5, 20, 11, 21, 11, 281000), @version=u'1', agent=u'Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.50 Safari/534.24', bytes=8292, clientip=u'228.105.101.199', extension=u'jpg', geo=Row(coordinates=Row(lat=33.46540139, lon=-89.72924806), dest=u'NG', src=u'CN', srcdest=u'CN:NG'), headings=[u'

mark-polansky

', u'http://www.slate.com/success/pavel-popovich'], host=u'media-for-the-masses.theacademyofperformingartsandscience.org', ip=u'228.105.101.199', links=[u'ronald-mcnair@www.slate.com', u'http://www.slate.com/info/sergei-krikalev', u'www.www.slate.com'], machine=Row(os=None, ram=6442450944), memory=None, phpmemory=None, referer=u'http://twitter.com/warning/mary-l-cleave', request=u'/uploads/kenneth-cockrell.jpg', response=u'200', spaces=u'this is a thing with lots of spaces wwwwoooooo', url=u'https://media-for-the-masses.theacademyofperformingartsandscience.org/uploads/kenneth-cockrell.jpg', utc_time=datetime.datetime(2015, 5, 20, 11, 21, 11, 281000), xss=u'') Row(@message=u'219.209.149.229 - - [2015-05-20T20:09:33.297Z] "GET /uploads/yury-usachov.jpg HTTP/1.1" 200 4127 "-" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.50 Safari/534.24"', @tags=[u'success', u'login'], @timestamp=datetime.datetime(2015, 5, 20, 20, 9, 33, 297000), @version=u'1', agent=u'Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.50 Safari/534.24', bytes=4127, clientip=u'219.209.149.229', extension=u'jpg', geo=Row(coordinates=Row(lat=29.72754583, lon=-85.02744778), dest=u'TH', src=u'NG', srcdest=u'NG:TH'), headings=[u'

marcos-pontes

', u'http://facebook.com/success/anatoly-artsebarsky'], host=u'media-for-the-masses.theacademyofperformingartsandscience.org', ip=u'219.209.149.229', links=[u'thomas-pesquet@www.slate.com', u'http://www.slate.com/info/eileen-collins', u'www.www.slate.com'], machine=Row(os=u'win 8', ram=3221225472), memory=None, phpmemory=None, referer=u'http://www.slate.com/success/karol-bobko', request=u'/uploads/yury-usachov.jpg', response=u'200', spaces=u'this is a thing with lots of spaces wwwwoooooo', url=u'https://media-for-the-masses.theacademyofperformingartsandscience.org/uploads/yury-usachov.jpg', utc_time=datetime.datetime(2015, 5, 20, 20, 9, 33, 297000), xss=u'') Row(@message=u'186.215.180.237 - - [2015-05-20T13:58:06.813Z] "GET /uploads/chiaki-mukai.jpg HTTP/1.1" 200 9034 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Gecko/20110421 Firefox/6.0a1"', @tags=[u'success', u'info'], @timestamp=datetime.datetime(2015, 5, 20, 13, 58, 6, 813000), @version=u'1', agent=u'Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Gecko/20110421 Firefox/6.0a1', bytes=9034, clientip=u'186.215.180.237', extension=u'jpg', geo=Row(coordinates=Row(lat=47.961167, lon=-97.401167), dest=u'NZ', src=u'GM', srcdest=u'GM:NZ'), headings=[u'

oleg-kotov

', u'http://facebook.com/success/virgil-i-gus-grissom'], host=u'media-for-the-masses.theacademyofperformingartsandscience.org', ip=u'186.215.180.237', links=[u'miroslaw-hermaszewski@www.slate.com', u'http://facebook.com/info/aleksandr-pavlovich-aleksandrov', u'www.nytimes.com'], machine=Row(os=u'win xp', ram=20401094656), memory=None, phpmemory=None, referer=u'http://facebook.com/warning/scott-altman', request=u'/uploads/chiaki-mukai.jpg', response=u'200', spaces=u'this is a thing with lots of spaces wwwwoooooo', url=u'https://media-for-the-masses.theacademyofperformingartsandscience.org/uploads/chiaki-mukai.jpg', utc_time=datetime.datetime(2015, 5, 20, 13, 58, 6, 813000), xss=u'') Row(@message=u'91.119.49.67 - - [2015-05-20T07:28:57.612Z] "GET /uploads/alexander-gerst.jpg HTTP/1.1" 200 3621 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Gecko/20110421 Firefox/6.0a1"', @tags=[u'success', u'info'], @timestamp=datetime.datetime(2015, 5, 20, 7, 28, 57, 612000), @version=u'1', agent=u'Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Gecko/20110421 Firefox/6.0a1', bytes=3621, clientip=u'91.119.49.67', extension=u'jpg', geo=Row(coordinates=Row(lat=40.08970806, lon=-105.9172367), dest=u'IN', src=u'CN', srcdest=u'CN:IN'), headings=[u'

yuri-baturin

', u'http://nytimes.com/success/james-f-reilly'], host=u'media-for-the-masses.theacademyofperformingartsandscience.org', ip=u'91.119.49.67', links=[u'wubbo-ockels@www.slate.com', u'http://www.slate.com/info/mike-adams', u'www.www.slate.com'], machine=Row(os=u'win 8', ram=12884901888), memory=None, phpmemory=None, referer=u'http://facebook.com/success/robert-curbeam', request=u'/uploads/alexander-gerst.jpg', response=u'200', spaces=u'this is a thing with lots of spaces wwwwoooooo', url=u'https://media-for-the-masses.theacademyofperformingartsandscience.org/uploads/alexander-gerst.jpg', utc_time=datetime.datetime(2015, 5, 20, 7, 28, 57, 612000), xss=u'') Row(@message=u'65.88.146.1 - - [2015-05-20T07:55:01.192Z] "GET /uploads/alexander-gerst.jpg HTTP/1.1" 200 2978 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Gecko/20110421 Firefox/6.0a1"', @tags=[u'warning', u'info'], @timestamp=datetime.datetime(2015, 5, 20, 7, 55, 1, 192000), @version=u'1', agent=u'Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Gecko/20110421 Firefox/6.0a1', bytes=2978, clientip=u'65.88.146.1', extension=u'jpg', geo=Row(coordinates=Row(lat=44.58036111, lon=-124.0579167), dest=u'IN', src=u'CA', srcdest=u'CA:IN'), headings=[u'

john-blaha

', u'http://twitter.com/success/richard-mullane'], host=u'media-for-the-masses.theacademyofperformingartsandscience.org', ip=u'65.88.146.1', links=[u'sergei-revin@twitter.com', u'http://www.slate.com/info/sigmund-j-hn', u'www.nytimes.com'], machine=Row(os=u'ios', ram=13958643712), memory=None, phpmemory=None, referer=u'http://twitter.com/error/soichi-noguchi', request=u'/uploads/alexander-gerst.jpg', response=u'200', spaces=u'this is a thing with lots of spaces wwwwoooooo', url=u'https://media-for-the-masses.theacademyofperformingartsandscience.org/uploads/alexander-gerst.jpg', utc_time=datetime.datetime(2015, 5, 20, 7, 55, 1, 192000), xss=u'') Row(@message=u'183.213.87.60 - - [2015-05-20T08:09:49.668Z] "GET /uploads/apollo-17.png HTTP/1.1" 200 5066 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"', @tags=[u'success', u'security'], @timestamp=datetime.datetime(2015, 5, 20, 8, 9, 49, 668000), @version=u'1', agent=u'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)', bytes=5066, clientip=u'183.213.87.60', extension=u'png', geo=Row(coordinates=Row(lat=40.12916667, lon=-91.67833333), dest=u'BR', src=u'CN', srcdest=u'CN:BR'), headings=[u'

michael-coats

', u'http://facebook.com/success/andrew-m-allen'], host=u'media-for-the-masses.theacademyofperformingartsandscience.org', ip=u'183.213.87.60', links=[u'david-saint-jacques@twitter.com', u'http://www.slate.com/info/vitali-sevastyanov', u'www.www.slate.com'], machine=Row(os=None, ram=11811160064), memory=None, phpmemory=None, referer=u'http://www.slate.com/success/dominic-a-antonelli', request=u'/uploads/apollo-17.png', response=u'200', spaces=u'this is a thing with lots of spaces wwwwoooooo', url=u'https://media-for-the-masses.theacademyofperformingartsandscience.org/uploads/apollo-17.png', utc_time=datetime.datetime(2015, 5, 20, 8, 9, 49, 668000), xss=u'') Row(@message=u'115.42.67.191 - - [2015-05-20T05:12:00.903Z] "GET /uploads/james-h-newman.jpg HTTP/1.1" 200 3722 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Gecko/20110421 Firefox/6.0a1"', @tags=[u'success', u'info'], @timestamp=datetime.datetime(2015, 5, 20, 5, 12, 0, 903000), @version=u'1', agent=u'Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Gecko/20110421 Firefox/6.0a1', bytes=3722, clientip=u'115.42.67.191', extension=u'jpg', geo=Row(coordinates=Row(lat=55.5546575, lon=-133.1016928), dest=u'JP', src=u'CN', srcdest=u'CN:JP'), headings=[u'

winston-scott

', u'http://www.slate.com/success/stephen-frick'], host=u'media-for-the-masses.theacademyofperformingartsandscience.org', ip=u'115.42.67.191', links=[u'david-m-walker@twitter.com', u'http://www.slate.com/security/voskhod-1', u'www.www.slate.com'], machine=Row(os=u'win xp', ram=18253611008), memory=None, phpmemory=None, referer=u'http://facebook.com/success/jing-haipeng', request=u'/uploads/james-h-newman.jpg', response=u'200', spaces=u'this is a thing with lots of spaces wwwwoooooo', url=u'https://media-for-the-masses.theacademyofperformingartsandscience.org/uploads/james-h-newman.jpg', utc_time=datetime.datetime(2015, 5, 20, 5, 12, 0, 903000), xss=u'') 17/09/21 08:15:03 INFO CodeGenerator: Code generated in 76.367979 ms 17/09/21 08:15:03 INFO CodeGenerator: Code generated in 19.706997 ms 17/09/21 08:15:03 INFO ScalaEsRowRDD: Reading from [apache-logs-*] 17/09/21 08:15:04 INFO SparkContext: Starting job: count at NativeMethodAccessorImpl.java:0 17/09/21 08:15:04 INFO DAGScheduler: Registering RDD 7 (count at NativeMethodAccessorImpl.java:0) 17/09/21 08:15:04 INFO DAGScheduler: Got job 1 (count at NativeMethodAccessorImpl.java:0) with 1 output partitions 17/09/21 08:15:04 INFO DAGScheduler: Final stage: ResultStage 2 (count at NativeMethodAccessorImpl.java:0) 17/09/21 08:15:04 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 1) 17/09/21 08:15:04 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 1) 17/09/21 08:15:04 INFO DAGScheduler: Submitting ShuffleMapStage 1 (MapPartitionsRDD[7] at count at NativeMethodAccessorImpl.java:0), which has no missing parents 17/09/21 08:15:04 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 11.4 KB, free 246.9 MB) 17/09/21 08:15:04 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 5.7 KB, free 246.9 MB) 17/09/21 08:15:04 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.10.10:38955 (size: 5.7 KB, free: 246.9 MB) 17/09/21 08:15:04 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006 17/09/21 08:15:04 INFO DAGScheduler: Submitting 9 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[7] at count at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8)) 17/09/21 08:15:04 INFO TaskSchedulerImpl: Adding task set 1.0 with 9 tasks 17/09/21 08:15:04 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 9, 192.168.10.10, executor 2, partition 0, NODE_LOCAL, 9989 bytes) 17/09/21 08:15:04 INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID 10, 192.168.10.12, executor 1, partition 3, NODE_LOCAL, 9989 bytes) 17/09/21 08:15:04 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 11, 192.168.10.11, executor 0, partition 1, NODE_LOCAL, 9989 bytes) 17/09/21 08:15:04 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 12, 192.168.10.10, executor 2, partition 2, NODE_LOCAL, 9989 bytes) 17/09/21 08:15:04 INFO TaskSetManager: Starting task 4.0 in stage 1.0 (TID 13, 192.168.10.12, executor 1, partition 4, NODE_LOCAL, 9989 bytes) 17/09/21 08:15:04 INFO TaskSetManager: Starting task 5.0 in stage 1.0 (TID 14, 192.168.10.11, executor 0, partition 5, NODE_LOCAL, 9989 bytes) 17/09/21 08:15:04 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.10.11:37263 (size: 5.7 KB, free: 366.3 MB) 17/09/21 08:15:04 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.10.12:40771 (size: 5.7 KB, free: 366.3 MB) 17/09/21 08:15:04 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.10.10:39297 (size: 5.7 KB, free: 366.3 MB) 17/09/21 08:15:05 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 4 17/09/21 08:15:05 INFO ExecutorAllocationManager: Requesting 4 new executors because tasks are backlogged (new desired total will be 4) 17/09/21 08:15:05 INFO TaskSetManager: Starting task 6.0 in stage 1.0 (TID 15, 192.168.10.11, executor 0, partition 6, NODE_LOCAL, 9989 bytes) 17/09/21 08:15:05 INFO TaskSetManager: Finished task 5.0 in stage 1.0 (TID 14) in 1466 ms on 192.168.10.11 (executor 0) (1/9) 17/09/21 08:15:05 INFO TaskSetManager: Starting task 7.0 in stage 1.0 (TID 16, 192.168.10.12, executor 1, partition 7, NODE_LOCAL, 9989 bytes) 17/09/21 08:15:05 INFO TaskSetManager: Finished task 3.0 in stage 1.0 (TID 10) in 1538 ms on 192.168.10.12 (executor 1) (2/9) 17/09/21 08:15:05 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 11) in 1681 ms on 192.168.10.11 (executor 0) (3/9) 17/09/21 08:15:06 INFO TaskSetManager: Starting task 8.0 in stage 1.0 (TID 17, 192.168.10.12, executor 1, partition 8, NODE_LOCAL, 9989 bytes) 17/09/21 08:15:06 INFO TaskSetManager: Finished task 4.0 in stage 1.0 (TID 13) in 1960 ms on 192.168.10.12 (executor 1) (4/9) 17/09/21 08:15:06 INFO TaskSetManager: Finished task 6.0 in stage 1.0 (TID 15) in 1062 ms on 192.168.10.11 (executor 0) (5/9) 17/09/21 08:15:06 INFO TaskSetManager: Finished task 7.0 in stage 1.0 (TID 16) in 990 ms on 192.168.10.12 (executor 1) (6/9) 17/09/21 08:15:06 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 3 17/09/21 08:15:07 INFO TaskSetManager: Finished task 8.0 in stage 1.0 (TID 17) in 1133 ms on 192.168.10.12 (executor 1) (7/9) 17/09/21 08:15:07 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 2 17/09/21 08:15:07 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID 12) in 3192 ms on 192.168.10.10 (executor 2) (8/9) 17/09/21 08:15:07 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 9) in 3220 ms on 192.168.10.10 (executor 2) (9/9) 17/09/21 08:15:07 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 17/09/21 08:15:07 INFO DAGScheduler: ShuffleMapStage 1 (count at NativeMethodAccessorImpl.java:0) finished in 3.221 s 17/09/21 08:15:07 INFO DAGScheduler: looking for newly runnable stages 17/09/21 08:15:07 INFO DAGScheduler: running: Set() 17/09/21 08:15:07 INFO DAGScheduler: waiting: Set(ResultStage 2) 17/09/21 08:15:07 INFO DAGScheduler: failed: Set() 17/09/21 08:15:07 INFO MesosCoarseGrainedSchedulerBackend: Capping the total amount of executors to 0 17/09/21 08:15:07 INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[10] at count at NativeMethodAccessorImpl.java:0), which has no missing parents 17/09/21 08:15:07 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 7.0 KB, free 246.8 MB) 17/09/21 08:15:07 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 3.7 KB, free 246.8 MB) 17/09/21 08:15:07 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.10.10:38955 (size: 3.7 KB, free: 246.9 MB) 17/09/21 08:15:07 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006 17/09/21 08:15:07 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (MapPartitionsRDD[10] at count at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0)) 17/09/21 08:15:07 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks 17/09/21 08:15:07 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 18, 192.168.10.12, executor 1, partition 0, NODE_LOCAL, 4748 bytes) 17/09/21 08:15:07 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.10.12:40771 (size: 3.7 KB, free: 366.3 MB) 17/09/21 08:15:07 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 192.168.10.12:35148 17/09/21 08:15:07 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 174 bytes 17/09/21 08:15:07 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 18) in 324 ms on 192.168.10.12 (executor 1) (1/1) 17/09/21 08:15:07 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 17/09/21 08:15:07 INFO DAGScheduler: ResultStage 2 (count at NativeMethodAccessorImpl.java:0) finished in 0.325 s 17/09/21 08:15:07 INFO DAGScheduler: Job 1 finished: count at NativeMethodAccessorImpl.java:0, took 3.719041 s Fetched 14005 logs (re-computed) Fetched 14005 logs (from collected list) 17/09/21 08:15:07 INFO SparkContext: Invoking stop() from shutdown hook 17/09/21 08:15:08 INFO SparkUI: Stopped Spark web UI at http://192.168.10.10:4040 17/09/21 08:15:08 INFO MesosCoarseGrainedSchedulerBackend: Shutting down all executors 17/09/21 08:15:08 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down 17/09/21 08:15:08 INFO MesosCoarseGrainedSchedulerBackend: Mesos task 1 is now TASK_FINISHED 17/09/21 08:15:08 INFO MesosCoarseGrainedSchedulerBackend: Mesos task 0 is now TASK_FINISHED 17/09/21 08:15:09 INFO MesosCoarseGrainedSchedulerBackend: Mesos task 2 is now TASK_FINISHED I0921 08:15:09.862349 27228 sched.cpp:2021] Asked to stop the driver I0921 08:15:09.863443 26969 sched.cpp:1203] Stopping framework aa404b40-8fd8-470d-94ff-87f1621566a7-0010 17/09/21 08:15:09 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 17/09/21 08:15:09 INFO MemoryStore: MemoryStore cleared 17/09/21 08:15:09 INFO BlockManager: BlockManager stopped 17/09/21 08:15:09 INFO BlockManagerMaster: BlockManagerMaster stopped 17/09/21 08:15:09 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 17/09/21 08:15:09 INFO SparkContext: Successfully stopped SparkContext 17/09/21 08:15:09 INFO ShutdownHookManager: Shutdown hook called 17/09/21 08:15:09 INFO ShutdownHookManager: Deleting directory /tmp/spark-e89a56d6-2e6d-4b3b-9007-f876bc449be0/pyspark-951e3781-1a4e-4045-bb47-0a3045926dde 17/09/21 08:15:09 INFO ShutdownHookManager: Deleting directory /tmp/spark-e89a56d6-2e6d-4b3b-9007-f876bc449be0 === DONE === Deleting temporary folder === DONE