大数据一——Hadoop单机部署
1 背景
由于我本人服务器有限,因此尝试进行单机安装,主要为了学习使用。
2 前提
hadoop安装依赖JDK。
我这里尝试用下JDK18看看。
下载地址:https://jdk.java.net/18/
这里咱们选择Linux/x64的版本。
2.1 配置JDK环境变量
# vi /etc/profile
在文件最后补充:
export JAVA_HOME=/usr/local/java/jdk-18.0.2.1
export PATH=$PATH:$JAVA_HOME/bin
配置生效
# source /etc/profile
4 下载
http://archive.apache.org/dist/hadoop/core/hadoop-3.3.4/
在这个链接,你可以看到各个版本,我这里选择的是3.3.4
在服务器上下载命令:
# wget http://archive.apache.org/dist/hadoop/core/hadoop-3.3.4/hadoop-3.3.4.tar.gz
5 解压
我这边上传到centos上,进行解压
# tar -zxvf hadoop-3.3.4.tar.gz
6 配置环境变量
同样,在/etc/profile文件中配置hadoop配置文件:
# vi /etc/profile
在文件末尾补充:
export HADOOP_HOME=/usr/local/hadoop/hadoop-3.3.4
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
配置生效
# source /etc/profile
检查配置是否生效
[root@instance-1mfp2mc1 jdk-18.0.2.1]# hadoop version
Hadoop 3.3.4
Source code repository https://github.com/apache/hadoop.git -r a585a73c3e02ac62350c136643a5e7f6095a3dbb
Compiled by stevel on 2022-07-29T12:32Z
Compiled with protoc 3.7.1
From source with checksum fb9dd8918a7b8a5b430d61af858f6ec
This command was run using /usr/local/hadoop/hadoop-3.3.4/share/hadoop/common/hadoop-common-3.3.4.jar
7 配置Hadoop文件
所有配置文件都在etc目录下:hadoop-3.3.4/etc/hadoop,所以,我们只要关心这个目录下相关文件的配置。
各个配置文件说明如下:
配置文件的名称 | 作用 |
---|---|
hadoop-env.sh | 主要配置我们的java路径 |
core-site.xml | 核心配置文件,主要定义了我们文件访问的格式 hdfs:// |
hdfs-site.xml | 主要定义配置我们的hdfs的相关配置 |
mapred-site.xml | 主要定义我们的mapreduce相关的一些配置 |
slaves | 控制我们的从节点在哪里 datanode nodemanager在哪些机器上 |
yarm-site.xml | 配置我们的resourcemanager资源调度 |
7.1 修改hadoop-env.sh
在文件末尾加上如下配置:
export JAVA_HOME=/usr/local/java/jdk-18.0.2.1
export HDFS_NAMENODE_USER=root
export HDFS_DATANODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root
7.2 修改 core-site.xml
将配置文件修改为:
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9820</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/hadoop/hadoopdata</value>
</property>
</configuration>
7.3 修改hdfs-site.xml
将配置文件修改为如下,表示只有一个副本,因为我这里是单机:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
7.4 修改mapred-site.xml
将配置文件修改为:
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
7.5 修改yarn-site.xml
将配置文件修改为:
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.env-whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
</property>
</configuration>
8 启动
8.1 对安装目录进行授权
# chmod -R 777 hadoop-3.3.4
8.2 格式化hdfs文件系统
# cd /usr/local/hadoop/hadoop-3.3.4/
# bin/hdfs namenode -format
8.3 启动hadoop
[root@instance-1mfp2mc1 hadoop-3.3.4]# sbin/start-all.sh
Starting namenodes on [localhost]
Last login: Wed Sep 14 21:45:50 CST 2022 from 218.76.52.120 on pts/3
Last failed login: Wed Sep 14 22:25:26 CST 2022 from 158.69.32.51 on ssh:notty
There were 10 failed login attempts since the last successful login.
localhost: Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
localhost: root@localhost: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
Starting datanodes
Last login: Wed Sep 14 22:28:54 CST 2022 on pts/3
localhost: root@localhost: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
Starting secondary namenodes [instance-1mfp2mc1]
Last login: Wed Sep 14 22:28:54 CST 2022 on pts/3
instance-1mfp2mc1: Warning: Permanently added 'instance-1mfp2mc1,192.168.16.4' (ECDSA) to the list of known hosts.
instance-1mfp2mc1: root@instance-1mfp2mc1: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
Starting resourcemanager
Last login: Wed Sep 14 22:28:55 CST 2022 on pts/3
Starting nodemanagers
Last login: Wed Sep 14 22:28:59 CST 2022 on pts/3
localhost: root@localhost: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
不过,我这里失败了。
9 配置ssh并再次启动
操作步骤我就直接贴出来了:
[root@instance-1mfp2mc1 .ssh]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:5UT8ePM907UkTWhPTy5b8lExBD1Jbfs3woLFZaNwOY0 root@instance-1mfp2mc1
The key's randomart image is:
---[RSA 3072]----
| .. .*= |
| o.E B **|
| *o* * =|
| . .o*=|
| S ..ooB*|
| . . oo=*|
| . . =|
| |
| |
----[SHA256]-----
[root@instance-1mfp2mc1 .ssh]# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[root@instance-1mfp2mc1 .ssh]# ssh localhost
Activate the web console with: systemctl enable --now cockpit.socket
Last failed login: Wed Sep 14 22:32:14 CST 2022 from 158.69.32.51 on ssh:notty
There were 4 failed login attempts since the last successful login.
Last login: Wed Sep 14 22:29:01 2022
再次启动下hadoop:
[root@instance-1mfp2mc1 ~]# cd /usr/local/hadoop/hadoop-3.3.4/
[root@instance-1mfp2mc1 hadoop-3.3.4]# sbin/start-all.sh
Starting namenodes on [localhost]
Last login: Wed Sep 14 22:32:53 CST 2022 from ::1 on pts/4
Last failed login: Wed Sep 14 22:34:01 CST 2022 from 91.240.118.222 on ssh:notty
There was 1 failed login attempt since the last successful login.
Starting datanodes
Last login: Wed Sep 14 22:35:12 CST 2022 on pts/4
Starting secondary namenodes [instance-1mfp2mc1]
Last login: Wed Sep 14 22:35:15 CST 2022 on pts/4
Starting resourcemanager
Last login: Wed Sep 14 22:35:20 CST 2022 on pts/4
Starting nodemanagers
Last login: Wed Sep 14 22:35:26 CST 2022 on pts/4
[root@instance-1mfp2mc1 hadoop-3.3.4]#
打开namenode,http://120.48.16.193:9870/,namenode的作用是存储元数据。
打开datanode,http://120.48.16.193:9864/,datanode作用是存储真正的数据。
这篇好文章是转载于:学新通技术网
- 版权申明: 本站部分内容来自互联网,仅供学习及演示用,请勿用于商业和其他非法用途。如果侵犯了您的权益请与我们联系,请提供相关证据及您的身份证明,我们将在收到邮件后48小时内删除。
- 本站站名: 学新通技术网
- 本文地址: /boutique/detail/tanhfheeic
系列文章
更多
-
photoshop保存的图片太大微信发不了怎么办
PHP中文网 06-15 -
Android 11 保存文件到外部存储,并分享文件
Luke 10-12 -
《学习通》视频自动暂停处理方法
HelloWorld317 07-05 -
word里面弄一个表格后上面的标题会跑到下面怎么办
PHP中文网 06-20 -
photoshop扩展功能面板显示灰色怎么办
PHP中文网 06-14 -
微信公众号没有声音提示怎么办
PHP中文网 03-31 -
excel下划线不显示怎么办
PHP中文网 06-23 -
excel打印预览压线压字怎么办
PHP中文网 06-22 -
怎样阻止微信小程序自动打开
PHP中文网 06-13 -
TikTok加速器哪个好免费的TK加速器推荐
TK小达人 10-01