Add support for Amazon EC2 to Ambari (part of hadoop)

Add support for Amazon EC2 to Ambari (part of hadoop)


Job Description

Currently at first login Ambari performs initial Hadoop deployment. Ambari has two options now:
1. Using SSH key;
2. Using self registration of Agent.

Need to add 3rd option: create on amazon EC2.
If this option chosen Ambari should create instances in amazon and self register all of them.
When EC2 chosen Ambari should ask for required information.
Ambari initial cluster setup requires to enter manula
Common set of parameters:
1. Amazon Key;
2. Amazon secure key;
3. SSH key
4. Security group. Available security groups should be pulled from Amazon account. The system should have check box to show all security groups or only compatible with hadoop requirements. Also "Create new security group" option should be available. "Create new" should create default hadoop settings and provide possibilities to modify it.
5. Generate Key
6. Test Connection
7. Any other required?

Example of look for this section in "Common Set" screenshot. Region from screenshot should be in Instances section.

User should be able to add any amount of blocks for instances. Instances set can have differences by type of instances, region, type of order. As an example user can setup all master tasks on "On-demand" instances and computational nodes on "Spot Instances". Example of collected info below and in screenshots.
1. Region (options pulled from Amazon)
2. Availability zone (pulled from Amazon)
3. Remote user
4. Root command prefix
5. Description
6. AMI. We should provide several options as build in for RedHat and CentOs. User should also be able to enter it manually.
7. Command for package installer (yum vs apt-get)
8. Instance type (options pulled from Amazon)
9. Tags. User should be able to add any amount of tags.
10. Amount of instances of this type
11. ssh port?
12. any required for hadoop options.
13. Type of request: "On demand" or "Spot Instances". If spot are chosen: additional options should be available: "maximal bid price" and "Percistant vs single order".
14. VPN?
15. DNS?
16. UserData

Install should be done through through CloudInit and self registration. Example of scripts for cloud init are below:

$COMMAND_For_Package_Installer -y install wget openjdk (or use oracle java installer)
wget ADDRESS/to/AmbariServer/slave.jar
start slave agent.

When all options set and "Continue Button pressed", new window should appear which shows progress of filling order, instance creation and self registration. orders should be in a table/tree format which is grouped by instance groups.
Progress bars should be provided for creation, registration....

If in the specific zone absent key, other parameters they should be created automatically.

The user should be able to proceed on the next step without waiting all processes to complete.

Options to view registered slaves should be added in the pages where user choosing where to install each service.

In the system should be available metadata for each EC2 agent.
Metadata should include Amazon specific info. Example:
Availability zone, instance type, Order type, for Spot Instances - maximal bid price, ....

Additional smaller tasks:
1. EC2 nodes supports should be added to running cluster.
2. Add set of services which should automatically installed on any new joined nodes
3. Spot Instances can be terminated at any-time. If we are using Persistent request this instances can be started later.
The system should remember set of services which were associated with Spot instance request and re install all services when instance is back.
4. add option to remove nodes from cluster and cancel Persistent request for spotInstances (if they present).