首页 > 分享 > CDH集群规划

CDH集群规划

花匠小妙招
2024-11-02 21:00

Cluster1集群规划版本操作系统Redhat7.2JDK1.8mysql5.7.20CM6.3.1CDH6.2.1 iphostname10.162.5.181(SERVER)dsj-4t-32310.162.5.182dsj-4t-32410.162.5.183dsj-4t-32510.162.5.184dsj-4t-32610.162.5.185dsj-4t-32710.162.5.186dsj-4t-32810.162.5.187dsj-4t-32910.162.5.188dsj-4t-33010.162.5.189dsj-4t-331 Cluster2集群规划版本JDK1.8mysql5.7.29CM6.3.1CDH6.2.1 iphostname10.162.5.190(SERVER)dsj-4t-33210.162.5.191dsj-4t-33310.162.5.192dsj-4t-33410.162.5.193dsj-4t-33510.162.5.194dsj-4t-336 安装CM过程中遇到的问题安装agent时检测不到信号:

确保9001与9000端口没有被占用

netstat -tunlp | grep 9001 sudo kill -9 <PID> 12

注意：不要先杀死所有主机的9001的进程再统一安装agent，因为一些进程啥杀死后过一段时间还会自己启动，要逐个主机的安装agent。

出现ProtocolError:

ps -ef | grep supervisord kill -9 <PID> ps aux | grep super kill -9 <PID> sudo service cloudera-scm-agent restart 12345 安装CDH过程中遇到的问题 1. 所有CDH的安装位置

cd /opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/lib 1 2. 查看zookeeper数据

cd /opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/lib/zookeeper/bin/ sudo bash zkCli.sh -server 127.0.0.1:2181 12 3. 修改kafka的ID后还要修改meta.propertie中的id

cd /var/local/kafka/data sudo vim meta.propertie 12 4. 安装HDFS后NFS Gateway启动失败

先来到安装了NFS Gateway的主机上，启动rpcbind服务

sudo service rpcbind status sudo service rpcbind start 12 5. 安装HDFS时出现错误

Failed to add storage directory [DISK]file:/mnt/sd01/dfs/dn java.io.IOException: Incompatible clusterIDs in /mnt/sd01/dfs/dn: namenode clusterID = cluster22; datanode clusterID = cluster13 12

这种情况是所有datanode无法启动，需要去所有安装datanode的主机上删除current文件

rm -rf /mnt/sd01/dfs/dn/current/ rm -rf /mnt/sd02/dfs/dn/current/ rm -rf /mnt/sd03/dfs/dn/current/ rm -rf /mnt/sd04/dfs/dn/current/ rm -rf /mnt/sd05/dfs/dn/current/ rm -rf /mnt/sd06/dfs/dn/current/ rm -rf /mnt/sd07/dfs/dn/current/ rm -rf /mnt/sd08/dfs/dn/current/ rm -rf /mnt/sd09/dfs/dn/current/ rm -rf /mnt/sd10/dfs/dn/current/ rm -rf /mnt/sd11/dfs/dn/current/ 1234567891011 6. 安装HIve是测试数据库连接时报错：Logon denied for user/password. Able to find the database server and database, but logon request was rejected

root用户无法连接新建的hive数据库，要赋权一下

mysql -uroot -p grant all privileges on *.* to 'root'@'10.162.5.190' identified by 'HXa#@2018QdoP' with grant option; flush privileges; 123 7.修改配置文件在web上能够上传下载文件到HDFS： 8.从本地上传文件到HDFS权限不够，在linux执行：

sudo groupadd supergroup sudo usermod -a -G supergroup root hdfs dfsadmin -refreshUserToGroupsMappings grep 'supergroup:' /etc/group hadoop fs -moveFromLocal ./1.txt / 12345