An insight into IBM BigInsight..

I was installing IBM BigInsight and during the install I kept getting this error addnode.sh failed

[INFO] @bivm2 - OS - Red Hat Enterprise Linux Server release 5.5 (Tikanga) Kernel \r on an \m : supported
[INFO] Progress - 3%
[INFO] Prerequisite check - Server configuration check - succeed
[ERROR] Prerequisite check - Disk space check - failed on bivm2
[ERROR] Prerequisite check - 255 -- java.io.IOException: exit code: 2 -- "/data1/opt/ibm/biginsights/hdm/bin/_ssh-remote.exp" "{xor}Lz4sLCgwLTs=" "ssh" "-o" "NumberOfPasswordPrompts=1" "root@bivm2" "/tmp/cal-diskOfDir-bivm2.sh <%= dfsDataDir %>"
root@bivm2's password:
bash: -c: line 0: syntax error near unexpected token `newline'
bash: -c: line 0: `/tmp/cal-diskOfDir-bivm2.sh <%= dfsDataDir %>'

        at com.ibm.xap.mgmt.util.ExecUtil.exec(ExecUtil.java:101)
        at com.ibm.xap.mgmt.hdm.AccountSshSetup.runAsRootRemote(AccountSshSetup.java:746)
        at com.ibm.xap.mgmt.hdm.AccountSshSetup.runAsRootRemote(AccountSshSetup.java:735)
        at com.ibm.xap.mgmt.hdm.AccountSshSetup.runAsRootRemote(AccountSshSetup.java:707)
        at com.ibm.xap.mgmt.hdm.BaseTask.runCmdRemote(BaseTask.java:134)
        at com.ibm.xap.mgmt.hdm.CheckDiskTask.getDiskWrittenTo(CheckDiskTask.java:110)
        at com.ibm.xap.mgmt.hdm.CheckDiskTask.checkOrGetFreeSpaceForDatanode(CheckDiskTask.java:195)
        at com.ibm.xap.mgmt.hdm.CheckDiskTask.doTask(CheckDiskTask.java:176)
        at com.ibm.xap.mgmt.util.Task.run(Task.java:77)
        at com.ibm.xap.mgmt.util.TaskRunner$1.run(TaskRunner.java:52)

[INFO] Progress - 4%
[ERROR] Prerequisite check - Datanode disk space check - failed on bivm2
[ERROR] Prerequisite check - 255 -- java.io.IOException: exit code: 2 -- "/data1/opt/ibm/biginsights/hdm/bin/_ssh-remote.exp" "{xor}Lz4sLCgwLTs=" "ssh" "-o" "NumberOfPasswordPrompts=1" "root@bivm2" "/tmp/cal-diskOfDir-bivm2.sh <%= dfsDataDir %>"
The authenticity of host 'bivm2 (172.20.168.233)' can't be established.
RSA key fingerprint is 7d:3e:bc:25:6e:f6:92:f8:35:87:fb:35:4d:98:bc:45.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'bivm2,172.20.168.233' (RSA) to the list of known hosts.
root@bivm2's password:
bash: -c: line 0: syntax error near unexpected token `newline'
bash: -c: line 0: `/tmp/cal-diskOfDir-bivm2.sh <%= dfsDataDir %>'

        at com.ibm.xap.mgmt.util.ExecUtil.exec(ExecUtil.java:101)
        at com.ibm.xap.mgmt.hdm.AccountSshSetup.runAsRootRemote(AccountSshSetup.java:746)
        at com.ibm.xap.mgmt.hdm.AccountSshSetup.runAsRootRemote(AccountSshSetup.java:735)
        at com.ibm.xap.mgmt.hdm.AccountSshSetup.runAsRootRemote(AccountSshSetup.java:707)
        at com.ibm.xap.mgmt.hdm.BaseTask.runCmdRemote(BaseTask.java:134)
        at com.ibm.xap.mgmt.hdm.CheckDiskTask.getDiskWrittenTo(CheckDiskTask.java:110)
        at com.ibm.xap.mgmt.hdm.CheckDiskTask.checkOrGetFreeSpaceForDatanode(CheckDiskTask.java:195)
        at com.ibm.xap.mgmt.hdm.CheckDiskTask.doTask(CheckDiskTask.java:176)
        at com.ibm.xap.mgmt.util.Task.run(Task.java:77)
        at com.ibm.xap.mgmt.util.TaskRunner$1.run(TaskRunner.java:52)

[INFO] @bivm2 - Program - scp,zip,bash,tar,ssh,unzip : installed
[INFO] Progress - 5%
[INFO] Prerequisite check - Required software/libraries Check - succeed

After some research, I found that folks at IBM have written the installation scripts keeping in mind that their might be bugs but you can work around it by just running some very simple preferences settings. So here's what I found and hope other find it useful as well.

Look for script enableOrDisablePrecheker.sh under <rooot>/biginsights-enterprise linux64_b20130821_1818/installer/bin

./enableOrDisablePrechecker.sh disable disk.space

That should take care of it. 

There are other options as well in case you run into isuses.

NAME

  enableOrDisablePrechecker.sh - enable or disable install pre-checks

SYNOPSIS

  enableOrDisablePrechecker.sh [OPTION] ITEM[,ITEM]...

  enableOrDisablePrechecker.sh [OPTION] all

OPTIONS

  enable - enable one or more prechecks

  disable - disable one or more precheck

  status - view current status

DESCRIPTION

  Each installation precheck item can be enabled or disabled, supported item names are

  * service.port - check service ports be used by biginsights services

  * selinux - check if SELINUX option is disabled on all biginsights nodes

  * disk.space - check if available disk space applies for minimal installation requirement

  * os.dist - check operation system

  * host.nameorip - check host names or ip addresses are correct and reachable

  * port.connectivity - check basic TCP connectivity on certain ports from client to server

  * software.dependencies - check software dependencies are installed, e.g ssh, scp

If you want to make any changes to Hadoop config files, you need to follow the following procedure

1) Make the required changes in the directory <biginsightroot>/opt/ibm/biginsights/hdm/hadoop-conf-staging

2) Once the changes are done on the head node, you can run /ibm/opt/ibm/biginsights/bin//syncconf.sh hadoop (Provided you want to make only changes for the haddop) if you don't provide any argument, it will sync the config changes for all services.

Change IP or Hostname for an already installed Cloudera Manager 4.x (CM 4.0)

I installed the Cloudera Manager 4 on a 1GbE default setup on eth0/bond0 and now I need to use 10GbE or Infiniband IB or 40GbE without the reinstall. How I did it!!

 The assumptions here are that you already have a CM 4.x already installed with the default embedded postgres DB on a linux server
. If you want to learn how-to just ask...

Let's start here where I have a 3 cluster setup with yosemite001 - yosemite003

 hostname = yosemite00[1-3].somedomain.com = CM Installed on 1GbE

1. Shutdown all services
2. service cloudera-scm-agent stop on all nodes 
3. service cloudera-scm-server stop on CM server

From here if you have a paid or supported version, You are about to lose it or maybe they say "Cool good job!!"  proceed at your own risk...

from the CM Server

as root su – postgres (If you have installed on a different database then login using that credentials)

psql -h localhost -p 7432 -U scm

à When asked for the password open another terminal and run

[root@yosemite001 ~]# grep password /etc/cloudera-scm-server/db.properties

com.cloudera.cmf.db.password=TVkDZxuNCw

paste the password & you should see the scm prompt

scm=>  select host_id,host_identifier,name,ip_address from hosts;  --> Check the current config and save the values in a file/notepad you will need this if you need to flip back.

host_id |    host_identifier    |         name          |   ip_address

---------+-----------------------+-----------------------+----------------

       2 | yosemite001.somedomain.com | yosemite001.somedomain.com | 192.168.0.11

       3 | yosemite002.somedomain.com | yosemite002.somedomain.com | 192.168.0.12

       4 | yosemite003.somedomain.com | yosemite003.somedomain.com | 192.168.0.13

(3 rows)

scm=> update hosts set (host_identifier,name,ip_address) = ('yosemite001-10g.somedomain.com','yosemite001-10g.somedomain.com','192.168.10.11') where host_id=2;

UPDATE 1

Update all the other rows.

Check if the updates went through

scm=>  select host_id,host_identifier,name,ip_address from hosts;

host_id |     host_identifier                       |           name                                   |   ip_address

---------+-------------------------------------+---------------------------------------+----------------

       3 | yosemite002.somedomain.com    | yosemite002.somedomain.com    | 192.168.0.12

       4 | yosemite003.somedomain.com    | yosemite003.somedomain.com    | 192.168.0.13

       2 | yosemite001-somedomain.com | yosemite001-ib.somedomain.com | 192.168.10.11

(3 rows)

Exit the tool  “\q;”

edit the /etc/cloudera-scm-agent/config.ini and update the server and the listen ip & hostname section on all nodes to the new interface ip & address

edit /etc/sysconfig/network with the new hostname for the interface

run hostname yosemite001-10g (Update all hostnames on the servers)

run “exec bash” and verify the hostname change is done

Run on the CM Server

chkconfig cloudera-scm-server on

service cloudera-scm-server start

Run the following on all the nodes

chkconfig cloudera-scm-agent on

service cloudera-scm-agent start on all the nodes

Login to the GUI verify the host changes and “Good health”

Force CM to rerun the configuration to register the changes (mimic changes and revert to force CM a rerun of config )

Go to the hdfs->system-wide config edit/changes and make any minimal changes and save and revert back to original and save

Do the same for mapreduce as well

Restart the services

Verify the client functionality by running Terasort.