Issue might happen import PyArrow
Depending on your Linux OS, especially with Ubuntu, you might run into issue like, when you installed pyarrow by conda install
1
>>> import pyarrow
2
Traceback (most recent call last):
3
File "<stdin>", line 1, in <module>
4
File "/home/dv6/anaconda3/envs/spark/lib/python3.6/site-packages/pyarrow/__init__.py", line 54, in <module>
5
from pyarrow.lib import cpu_count, set_cpu_count
6
ImportError: /home/dv6/anaconda3/envs/spark/lib/python3.6/site-packages/pyarrow/lib.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZNK5arrow5F.... B5cxx11Ev
7
>>> quit()
8
(spark) [email protected]:~$ conda uninstall pyarrow
9
Collecting package metadata (repodata.json): done
10
Solving environment: done
11
​
Copied!
I have avoided the issue with pip install pyarrow --user, after conda uninstall
1
(spark) [email protected]:~$ conda uninstall pyarrow
2
Collecting package metadata (repodata.json): done
3
Solving environment: done
4
​
5
​
6
==> WARNING: A newer version of conda exists. <==
7
current version: 4.8.2
8
latest version: 4.8.3
9
​
10
Please update conda by running
11
​
12
$ conda update -n base -c defaults conda
13
​
14
​
15
​
16
## Package Plan ##
17
​
18
environment location: /home/dv6/anaconda3/envs/spark
19
​
20
removed specs:
21
- pyarrow
22
​
23
​
24
The following packages will be REMOVED:
25
​
26
pyarrow-0.11.1-py36hbbcf98d_1002
27
python_abi-3.6-1_cp36m
28
​
29
The following packages will be SUPERSEDED by a higher-priority channel:
30
​
31
ca-certificates conda-forge::ca-certificates-2020.4.5~ --> pkgs/main::ca-ce rtificates-2020.1.1-0
32
certifi conda-forge::certifi-2020.4.5.1-py36h~ --> pkgs/main::certi fi-2020.4.5.1-py36_0
33
openssl conda-forge::openssl-1.1.1f-h516909a_0 --> pkgs/main::opens sl-1.1.1f-h7b6447c_0
34
​
35
​
36
Proceed ([y]/n)? y
37
​
38
Preparing transaction: done
39
Verifying transaction: done
40
Executing transaction: done
41
(spark) [email protected]:~$ pip install pyarrow --user
42
Collecting pyarrow
43
Downloading pyarrow-0.16.0-cp36-cp36m-manylinux2014_x86_64.whl (63.1 MB)
44
|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 63.1 MB 25 kB/s
45
Requirement already satisfied: numpy>=1.14 in ./anaconda3/envs/spark/lib/python3 .6/site-packages (from pyarrow) (1.18.1)
46
Requirement already satisfied: six>=1.0.0 in ./anaconda3/envs/spark/lib/python3. 6/site-packages (from pyarrow) (1.14.0)
47
Installing collected packages: pyarrow
48
Successfully installed pyarrow-0.16.0
49
(spark) [email protected]:~$ python -c "import pyarrow"
50
(spark) [email protected]:~$ python
51
Python 3.6.10 |Anaconda, Inc.| (default, Jan 7 2020, 21:14:29)
52
[GCC 7.3.0] on linux
53
Type "help", "copyright", "credits" or "license" for more information.
54
>>>
55
​
Copied!
Last modified 1yr ago
Copy link