Cannot run multiple SparkContexts at once; existing SparkContext(app=PySparkShell, maste
最新推荐文章于 2024-09-09 07:30:00 发布

Rachel_nana 于 2019-08-28 16:13:00 发布
版权声明:本文为博主原创文章,遵循 CC 4.0 BY-SA 版权协议,转载请附上原文出处链接和本声明。
运行SparkContext报错:
ValueError: Cannot run multiple SparkContexts at once; existing SparkContext(app=PySparkShell, master=local[*]) created by <module> at /usr/local/spark/python/pyspark/shell.py:59
hadoop@rachel-virtual-machine:/usr/local/spark/bin$ ./pyspark
./pyspark: 行 45: python: 未找到命令
Python 3.6.8 (default, Jan 14 2019, 11:02:34)
[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux
Type "help", "copyright", "credits" or "license" for more information.
2019-08-28 15:27:12 WARN Utils:66 - Your hostname, rachel-virtual-machine resolves to a loopback address: 127.0.1.1; using 192.168.80.128 instead (on interface ens33)
2019-08-28 15:27:12 WARN Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address
2019-08-28 15:27:23 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Welcome to
____ __
/ __/__ ___ _____/ /__
_ / _ / _ `/ __/ '_/
/__ / .__/_,_/_/ /_/_ version 2.3.3
/_/
Using Python version 3.6.8 (default, Jan 14 2019 11:02:34)
SparkSession available as 'spark'.
>>> from pyspark import SparkContext
>>> sc = SparkContext( 'local', 'test')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/spark/python/pyspark/context.py", line 129, in __init__
SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
File "/usr/local/spark/python/pyspark/context.py", line 328, in _ensure_initialized
callsite.function, callsite.file, callsite.linenum))
ValueError: Cannot run multiple SparkContexts at once; existing SparkContext(app=PySparkShell, master=local[*]) created by <module> at /usr/local/spark/python/pyspark/shell.py:59
出现这个错误是因为之前已经启动了SparkContext,所以需要先关闭spark,然后再启动。
>>> sc.stop()
>>> sc=SparkContext("local","test")
>>> logFile = "file:///usr/local/spark/README.md"
>>> logData = sc.textFile(logFile, 2).cache()
>>> numAs = logData.filter(lambda line: 'a' in line).count()
>>> numBs = logData.filter(lambda line: 'b' in line).count()
>>> print('Lines with a: %s, Lines with b: %s' % (numAs, numBs))
Lines with a: 61, Lines with b: 30
相关知识
成功解决报错Cannot find class [com.alibaba.druid.pool.DruidDataSource] for bean with name ‘dataSource‘
Operator '==' cannot be applied to错误
cannot open resource self.font = core.getfont(font, size, index, encoding, layout
no such file or directory
python中x y的意思,在Python中x ='y''z'的含义是什么?
【深度学习TPU+Keras+Tensorflow+EfficientNetB7】kaggle竞赛 使用TPU对104种花朵进行分类 第十八次尝试 99.9%准确率
考研英语作文:温室花朵经不起考验
Honeywell Air Purifier
2003年考研英语一作文温室里地花朵经不起风雨.pdf
基于神经结构搜索的多种植物叶片病害识别
原文链接: Cannot run multiple SparkContexts at once; existing SparkContext(app=PySparkShell, maste https://m.huajiangbk.com/newsview1055193.html
| 上一篇: 幼儿园活动计划方案(15篇) | 下一篇: 动物传播的植物有哪些 |



