关键字:debian9、ambari、安装、hadoop
时间:2019年1月

步骤

一、准备主机
二、安装ambari-server
三、配置hdp节点
四、部署HDP

一、准备主机

1、主机

名称 IP hostname
ambari服务器 172.16.1.100 ambari.fish2bird.com
hdp节点1 172.16.1.101 node-1.fish2bird.com
hdp节点2 172.16.1.102 node-2.fish2bird.com
hdp节点3 172.16.1.103 node-3.fish2bird.com

注意:请配置好/etc/hostnamt和/etc/hosts。

2、所有主机安装所需包和添加hortonworks源的公钥

root@debian:~# apt-get install ntp
root@debian:~# apt-get install openjdk-8-jdk
root@debian:~# apt-get install dirmngr
root@debian:/etc/apt/sources.list.d# apt-key adv --keyserver keyserver.ubuntu.com --recv-keys B9733A7A07513CAD
Executing: /tmp/apt-key-gpghome.MMdqGHL6AD/gpg.1.sh --keyserver keyserver.ubuntu.com --recv-keys B9733A7A07513CAD
gpg: key B9733A7A07513CAD: public key "Jenkins (HDP Builds) <jenkin@hortonworks.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1
root@debian:~# cd /etc/apt/sources.list.d/
root@debian:/etc/apt/sources.list.d# wget http://public-repo-1.hortonworks.com/ambari/debian9/2.x/updates/2.7.1.0/ambari.list
root@debian:/etc/apt/sources.list.d# apt-get update

二、安装ambari-server

root@ambari:~# apt-get install ambari-server
root@ambari:~# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
WARNING: Could not run /usr/sbin/sestatus: OK
Customize user account for ambari-server daemon [y/n] (n)? 
Enter user account for ambari-server daemon (root):
Adjusting ambari-server permissions and ownership...
Checking firewall status...
/bin/bash: ufw: command not found
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
==============================================================================
Enter choice (1): 2
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64
Validating JDK on Ambari Server...done.
Check JDK version for Ambari Server...
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? 
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? 
Configuring database...
Default properties detected. Using built-in database.
Configuring ambari database...
Checking PostgreSQL...
Configuring local database...
Configuring PostgreSQL...
Restarting PostgreSQL
Creating schema and user...
done.
Creating tables...
done.
Extracting system views...
ambari-admin-2.7.1.0.169.jar
....
Ambari repo file contains latest json url http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json, updating stacks repoinfos with it...
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

root@ambari:~# systemctl start ambari-server

三、配置hdp节点

配置ssh秘钥登录

root@ambari:~# ssh-keygen -t rsa
root@ambari:~# cd .ssh/
root@ambari:~/.ssh# ssh-copy-id -i id_rsa root@node-1.fish2bird.com
root@ambari:~/.ssh# ssh-copy-id -i id_rsa root@node-2.fish2bird.com
root@ambari:~/.ssh# ssh-copy-id -i id_rsa root@node-3.fish2bird.com
root@ambari:~/.ssh# cat id_rsa
-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----

注:复制id_rsa内容到粘贴板。

四、部署HDP

1、打开ambari的web界面

http://ambari.fish2bird.com:8080/
用户名:admin
密码:admin
创建一个集群。

2、第2步,Install Options

Target Hosts
Enter a list of hosts using the Fully Qualified Domain Name (FQDN), one per line. Or use Pattern Expressions
+————————
| node-1.fish2bird.com
| node-2.fish2bird.com
| node-3.fish2bird.com
|
+————————
选择Provide your SSH Private Key to automatically register hosts,填写上一步粘贴板内容
+————–
| —–BEGIN RSA PRIVATE KEY—–
|
| —–END RSA PRIVATE KEY—–
+————–
NEXT > NEXT

第9步,Install,Start and Test

此步,各个节点开始自动安装各种包,可能会因为网络较慢以及某些包较大造成超时(例如:phoenix包就有600M+,但下载速度只有几十KB/s),造成失败的假象,此时需要回到上一步再来一次,直到安装成功。建议先安装一个节点,然后把节点上缓存的deb包拷贝到其他节点的缓存目录后再进行安装。

四、删除无用组件

在部署HDP时,hortonworks把Smart Sense这个组件作为了强制安装,并且在ambari的web界面上无法删除该组件。下面提供脚本方式删除。

root@ambari:~# curl -u admin:${PASSWD} -i -H 'X-Requested-By: ambari' -X DELETE http://${AMBARI_HOST}:8080/api/v1/clusters/${CLUSTER}/hosts/${HOST}/host_components/HST_SERVER
HTTP/1.1 200 OK
...
root@ambari:~# curl -u admin:${PASSWD} -i -H 'X-Requested-By: ambari' -X DELETE http://${AMBARI_HOST}:8080/api/v1/clusters/${CLUSTER}/hosts/${HOST}/host_components/HST_AGENT
HTTP/1.1 200 OK
...

注意:${PASSWD}替换成密码,${CLUSTER}替换成你部署HDP时取的名字,${HOST}替换成HDP节点主机名。

参考

https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.1.0/bk_ambari-installation/content/ambari_repositories.html

关键字:hbase、修复、repair、hbck
时间:2019年1月

为什么需要检查和修复

1、当hbase节点出现宕机、或磁盘损坏的情况下,可能会造成数据错误,此时需要进行检查;
2、修复的作用主要是让hbase处于可用状态,出现错误不修复通常的现象是数据无法写入或写入缓慢等

检查命令

hbase hbck

修复参数

1. 单种错误修复
-fixTableOrphans
-fixHdfsOverlaps
-fixHdfsOrphans
-fixHdfsHoles
-fixMeta
-fixAssignments
-fixVersionFile
2. 综合修复
-repair,等价与多个单种错误修复参数同时执行
-repairHoles,等价与多个单种错误修复参数同时执行

修复举例

a. 只执行检查

root@debian:~# hbase hbck

b. 不管检查错误项,修复所有错误

root@debian:~# hbase hbck -repair

c. 修复具体某种错误

root@debian:~# hbase hbck -fixMeta

d. 几种错误同时修复

root@debian:~# hbase hbck -fixMeta -fixHdfsOrphans -fixHdfsOrphans

注意

1、通常执行一次修复无法修复所有问题,一次修复执行后,应再执行检查,确定是否所有问题都修复完成;
2、多个参数一次执行,参数的顺序不同,修复结果也许会不一样;(未详细验证过)
3、多个参数一次执行和多个参数分多次执行,修复结果不一样;
4、同一个参数反复执行应该没有太大意义;
5、修复一个错误后,再次检查可能会有新的错误出现,注意仔细阅读检查结果,确定再次执行修复使用哪个参数;
6、检查出现错误,表示数据本身存在问题,修复主要能让hbase处于可用状态,通常不能修复数据本身。