Hadoop 3.3.6 + Spark 3.4.2 环境配置：CentOS 7 集群部署 5 节点避坑指南

Hadoop 3.3.6 Spark 3.4.2 环境配置CentOS 7 集群部署 5 节点避坑指南1. 集群规划与环境准备在开始部署之前我们需要对集群进行合理的规划。一个典型的5节点集群可以按照以下方式分配角色主节点Master1台运行NameNode、ResourceManager、Spark Master等主控服务从节点Slave4台运行DataNode、NodeManager、Spark Worker等计算存储服务硬件配置建议主节点16核CPU32GB内存500GB SSD从节点8核CPU64GB内存2TB HDD x 4建议使用RAID 0操作系统准备# 所有节点执行 yum install -y epel-release yum install -y java-11-openjdk-devel vim wget net-tools ntp sshpass pdsh关键配置# 设置主机名解析所有节点 vim /etc/hosts 192.168.1.101 master 192.168.1.102 slave1 192.168.1.103 slave2 192.168.1.104 slave3 192.168.1.105 slave4 # 关闭防火墙和SELinux生产环境需谨慎 systemctl stop firewalld systemctl disable firewalld setenforce 0 sed -i s/SELINUXenforcing/SELINUXdisabled/g /etc/selinux/config # 时间同步 ntpdate ntp.aliyun.com2. Hadoop 3.3.6 集群部署2.1 安装与基础配置首先下载并解压Hadoop安装包wget https://archive.apache.org/dist/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz tar -xzf hadoop-3.3.6.tar.gz -C /opt/ ln -s /opt/hadoop-3.3.6 /opt/hadoop配置环境变量所有节点echo export HADOOP_HOME/opt/hadoop /etc/profile echo export PATH$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin /etc/profile source /etc/profile2.2 关键配置文件修改core-site.xmlconfiguration property namefs.defaultFS/name valuehdfs://master:9000/value /property property namehadoop.tmp.dir/name value/data/hadoop/tmp/value /property property nameio.file.buffer.size/name value131072/value /property /configurationhdfs-site.xmlconfiguration property namedfs.namenode.name.dir/name value/data/hadoop/namenode/value /property property namedfs.datanode.data.dir/name value/data/hadoop/datanode/value /property property namedfs.replication/name value3/value /property property namedfs.blocksize/name value256m/value /property property namedfs.namenode.datanode.registration.ip-hostname-check/name valuefalse/value /property /configurationyarn-site.xmlconfiguration property nameyarn.nodemanager.aux-services/name valuemapreduce_shuffle/value /property property nameyarn.resourcemanager.hostname/name valuemaster/value /property property nameyarn.nodemanager.resource.memory-mb/name value49152/value /property property nameyarn.scheduler.maximum-allocation-mb/name value49152/value /property /configuration2.3 集群初始化与启动在主节点执行格式化hdfs namenode -format启动HDFS和YARNstart-dfs.sh start-yarn.sh验证集群状态# 检查HDFS hdfs dfsadmin -report # 检查YARN yarn node -list3. Spark 3.4.2 集群部署3.1 安装与配置下载并安装Sparkwget https://archive.apache.org/dist/spark/spark-3.4.2/spark-3.4.2-bin-hadoop3.tgz tar -xzf spark-3.4.2-bin-hadoop3.tgz -C /opt/ ln -s /opt/spark-3.4.2-bin-hadoop3 /opt/spark配置环境变量echo export SPARK_HOME/opt/spark /etc/profile echo export PATH$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin /etc/profile source /etc/profilespark-env.sh配置cp $SPARK_HOME/conf/spark-env.sh.template $SPARK_HOME/conf/spark-env.sh echo export JAVA_HOME/usr/lib/jvm/java-11-openjdk $SPARK_HOME/conf/spark-env.sh echo export HADOOP_CONF_DIR$HADOOP_HOME/etc/hadoop $SPARK_HOME/conf/spark-env.sh echo export SPARK_MASTER_HOSTmaster $SPARK_HOME/conf/spark-env.sh echo export SPARK_WORKER_MEMORY48g $SPARK_HOME/conf/spark-env.shworkers文件配置slave1 slave2 slave3 slave43.2 集群启动与验证启动Spark集群$SPARK_HOME/sbin/start-all.sh提交测试任务spark-submit --class org.apache.spark.examples.SparkPi \ --master spark://master:7077 \ $SPARK_HOME/examples/jars/spark-examples_2.12-3.4.2.jar 1004. 关键调优参数4.1 Hadoop性能优化hdfs-site.xml追加property namedfs.client.read.shortcircuit/name valuetrue/value /property property namedfs.domain.socket.path/name value/var/lib/hadoop-hdfs/dn_socket/value /propertymapred-site.xmlproperty namemapreduce.map.memory.mb/name value4096/value /property property namemapreduce.reduce.memory.mb/name value8192/value /property4.2 Spark性能优化spark-defaults.confspark.executor.memory 48g spark.executor.cores 8 spark.driver.memory 8g spark.default.parallelism 200 spark.sql.shuffle.partitions 200 spark.memory.fraction 0.8 spark.serializer org.apache.spark.serializer.KryoSerializer5. 常见问题解决方案5.1 部署阶段问题问题1DataNode无法启动报错Invalid directory原因数据目录权限不正确解决chown -R hadoop:hadoop /data/hadoop问题2YARN资源分配不足现象应用无法获取足够资源解决调整yarn-site.xml中的内存配置5.2 运行阶段问题问题3Spark作业OOM内存溢出解决方案增加executor内存--executor-memory 16g调整内存分配比例spark.memory.fraction0.6问题4HDFS写入速度慢优化方案增加DataNode磁盘数量调整dfs.datanode.max.transfer.threads参数6. 集群监控与维护6.1 监控方案Hadoop监控NameNode UI: http://master:9870YARN UI: http://master:8088Spark监控Spark UI: http://master:4040History Server: http://master:180806.2 日常维护命令检查HDFS健康状态hdfs fsck / -files -blocks -locations平衡HDFS数据hdfs balancer -threshold 10查看YARN应用日志yarn logs -applicationId application_id