[Ubuntu] Apache Metron 설치 (0.1 Beta, Full 버전, VM, Vagrant)

*Ubuntu14.04

0. Requirements

• Ansible 2.0.0.2
• Vagrant 1.8.1
• Virtualbox 5.0.16
• Python 2.7.11
• Maven 3.3.9

1. Virtual Box^[각주:1] 설치 (5.0.16)

wget  http://download.virtualbox.org/virtualbox/5.0.16/virtualbox-5.0_5.0.16-105871~Ubuntu~trusty_amd64.deb

dpkg -i virtualbox-5.0_5.0.16-105871~Ubuntu~trusty_amd64.deb

2. Vagrant^[각주:2] 설치 (1.8.1)

wget https://releases.hashicorp.com/vagrant/1.8.1/vagrant_1.8.1_x86_64.deb

dpkg -i vagrant_1.8.1_x86_64.deb

3. Maven^[각주:3] 설치 (3.3.9)

wget http://mirror.olnevhost.net/pub/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz

sudo tar xvf apache-maven-3.3.9-bin.tar.gz -C /usr/local/

sudo ln -s /usr/local/apache-maven-3.3.9/bin/mvn /usr/bin/mvn

rm -rf apache-maven-3.3.9-bin.tar.gz

 

4. Ansible^[각주:4] 설치 (2.0.0.2) & 설정

sudo su -

easy_install pip

export CFLAGS=-Qunused-arguments

export CPPFLAGS=-Qunused-arguments

pip install ansible==2.0.0.2

 

5. Apache Metron^[각주:5] 설치 (0.1 Beta)

wget https://dist.apache.org/repos/dist/dev/incubator/metron/0.1BETA-RC7-incubating/apache-metron-0.1BETA-RC7-incubating.tar.gz

tar -zxvf apache-metron-0.1BETA-RC7-incubating.tar.gz

cd incubator-metron-Metron_0.1BETA_rc7

mvn apache-rat:check

cd metron-streaming

mvn clean integration-test

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test (integration-tests) on project Metron-Topologies: Execution integration-tests of goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test failed: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?

Solution-->

cd incubator-metron-Metron_0.1BETA_rc7

mvn clean package -DskipTests

cd metron-streaming

mvn clean package -DskipTests

cd ../deployment/vagrant/singlenode-vagrant

vagrant plugin install vagrant-hostmanager

vagrant up //Ansible scripts을 통해 Ambari, HDP, vm 위에 metron까지 실행됨. 소요시간은 45-60분.

TASK [ambari_config : Start All Hadoop Services node1] *************************
failed: [node1] (item=HDFS) => {"connection": "close", "content": "", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "item": "HDFS", "msg": "Status code was not [200, 202]: HTTP Error 500: Server Error", "redirected": false, "server": "Jetty(8.1.17.v20150415)", "set_cookie": "AMBARISESSIONID=1sgn1chrad41b1qw1kj03dd9kg;Path=/;HttpOnly", "status": 500, "url": "http://node1:8080/api/v1/clusters/metron_cluster/services/HDFS", "user": "admin"}
failed: [node1] (item=YARN) => {"connection": "close", "content": "", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "item": "YARN", "msg": "Status code was not [200, 202]: HTTP Error 500: Server Error", "redirected": false, "server": "Jetty(8.1.17.v20150415)", "set_cookie": "AMBARISESSIONID=1is6xyjxvpccqyba2f46le9sl;Path=/;HttpOnly", "status": 500, "url": "http://node1:8080/api/v1/clusters/metron_cluster/services/YARN", "user": "admin"}
failed: [node1] (item=MAPREDUCE2) => {"connection": "close", "content": "", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "item": "MAPREDUCE2", "msg": "Status code was not [200, 202]: HTTP Error 500: Server Error", "redirected": false, "server": "Jetty(8.1.17.v20150415)", "set_cookie": "AMBARISESSIONID=1nvb5mwkf1a0r19a051px2z1da;Path=/;HttpOnly", "status": 500, "url": "http://node1:8080/api/v1/clusters/metron_cluster/services/MAPREDUCE2", "user": "admin"}
failed: [node1] (item=ZOOKEEPER) => {"connection": "close", "content": "", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "item": "ZOOKEEPER", "msg": "Status code was not [200, 202]: HTTP Error 500: Server Error", "redirected": false, "server": "Jetty(8.1.17.v20150415)", "set_cookie": "AMBARISESSIONID=1t0e24juqq5be1oneug42xd9fy;Path=/;HttpOnly", "status": 500, "url": "http://node1:8080/api/v1/clusters/metron_cluster/services/ZOOKEEPER", "user": "admin"}
failed: [node1] (item=HBASE) => {"connection": "close", "content": "", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "item": "HBASE", "msg": "Status code was not [200, 202]: HTTP Error 500: Server Error", "redirected": false, "server": "Jetty(8.1.17.v20150415)", "set_cookie": "AMBARISESSIONID=1v10mm1sa85tj1jd0cxg53kmfr;Path=/;HttpOnly", "status": 500, "url": "http://node1:8080/api/v1/clusters/metron_cluster/services/HBASE", "user": "admin"}
failed: [node1] (item=STORM) => {"connection": "close", "content": "", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "item": "STORM", "msg": "Status code was not [200, 202]: HTTP Error 500: Server Error", "redirected": false, "server": "Jetty(8.1.17.v20150415)", "set_cookie": "AMBARISESSIONID=1xso3y6yntj86150i2ne8ixqdl;Path=/;HttpOnly", "status": 500, "url": "http://node1:8080/api/v1/clusters/metron_cluster/services/STORM", "user": "admin"}
failed: [node1] (item=KAFKA) => {"connection": "close", "content": "", "content_type": "text/plain;charset=ISO-8859-1", "failed": true, "item": "KAFKA", "msg": "Status code was not [200, 202]: HTTP Error 500: Server Error", "redirected": false, "server": "Jetty(8.1.17.v20150415)", "set_cookie": "AMBARISESSIONID=1e2nei5aurn69eq1lcnvfcnna;Path=/;HttpOnly", "status": 500, "url": "http://node1:8080/api/v1/clusters/metron_cluster/services/KAFKA", "user": "admin"}
to retry, use: --limit @../../playbooks/metron_full_install.retry


PLAY RECAP *********************************************************************
node1                      : ok=38   changed=28   unreachable=0    failed=1

----------------->Solution // ansible 버전 문제

pip uninstall ansible

pip install ansible==2.0.0.2

vagrant destroy

vagrant up 

 

ERROR:

==> node1: Waiting for machine to boot. This may take a few minutes...
    node1: SSH address: 127.0.0.1:2222
    node1: SSH username: vagrant
    node1: SSH auth method: private key
    node1: Warning: Remote connection disconnect. Retrying...
    node1:
    node1: Vagrant insecure key detected. Vagrant will automatically replace
    node1: this with a newly generated keypair for better security.
    node1:
    node1: Inserting generated public key within guest...
    node1: Removing insecure key from the guest if it's present...
    node1: Key inserted! Disconnecting and reconnecting using new SSH key...
    node1: Warning: Authentication failure. Retrying...

Timed out while waiting for the machine to boot. This means that
Vagrant was unable to communicate with the guest machine within
the configured ("config.vm.boot_timeout" value) time period.

If you look above, you should be able to see the error(s) that
Vagrant had when attempting to connect to the machine. These errors
are usually good hints as to what may be wrong.

If you're using a custom box, make sure that networking is properly
working and you're able to connect to the machine. It is a common
problem that networking isn't setup properly in these boxes.
Verify that authentication configurations are also setup properly,
as well.

If the box appears to be booting properly, you may want to increase
the timeout ("config.vm.boot_timeout") value.

Solution---->

vi deployment/vagrant/singlenode-vagrant/Vagrantfile

// modify  config.ssh.insert_key = false

vagrant destroy

vagrant up

6. Apache Metron 실행 확인

웹브라우저를 통해 http://node1:8080 접속

ID : admin

PW : admin

 

 

 

TASK [ambari_common : Download Ambari repo] ************************************
fatal: [node1]: FAILED! => {"changed": false, "dest": "/etc/yum.repos.d/ambari.repo", "failed": true, "msg": "Request failed", "response": "Request failed: <urlopen error [Errno -3] Temporary failure in name resolution>", "state": "absent", "status_code": -1, "url": "http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.1.2.1/ambari.repo"}

 

------------> CentOS 용이라서 안됨.불가능. 

1. Hypervisor [본문으로]
2. vm 관리 도구 - vm 이미지(box) 다운로드, vm에 ssh 연결 [본문으로]
3. 프로젝트 관리 툴 : 빌드 툴을 바탕으로 여러 기능들을 제공 - 보고서 작성, 웹사이트 생성, 팀원간 소통 기능 [본문으로]
4. 환경설정 관리 & 배포 자동화 툴 : ssh 기반, python 기반 [본문으로]
5. 실시간 빅 데이터 사이버 보안 분석을 위한 개방형 연결 데이터 플랫폼 [본문으로]