samedi 28 février 2015

AWS EMR validation error

I have a problem running a map-reduce java application I simplified my problem using the tutorial code given from AWS which runs a pre-defined step:



public class Main {

public static void main(String[] args) {

AWSCredentials credentials = getCredentials();
AmazonElasticMapReduceClient emr = new AmazonElasticMapReduceClient(
credentials);

StepFactory stepFactory = new StepFactory();

StepConfig enabledebugging = new StepConfig()
.withName("Enable debugging")
.withActionOnFailure("TERMINATE_JOB_FLOW")
.withHadoopJarStep(stepFactory.newEnableDebuggingStep());

StepConfig installHive = new StepConfig().withName("Install Hive")
.withActionOnFailure("TERMINATE_JOB_FLOW")
.withHadoopJarStep(stepFactory.newInstallHiveStep());

RunJobFlowRequest request = new RunJobFlowRequest()
.withName("Hive Interactive")
.withAmiVersion("3.3.1")
.withSteps(enabledebugging, installHive)
.withLogUri("s3://tweets-hadoop/")
.withServiceRole("service_role")
.withJobFlowRole("jobflow_role")
.withInstances(
new JobFlowInstancesConfig().withEc2KeyName("hadoop")
.withInstanceCount(5)
.withKeepJobFlowAliveWhenNoSteps(true)
.withMasterInstanceType("m3.xlarge")
.withSlaveInstanceType("m1.large"));

RunJobFlowResult result = emr.runJobFlow(request);
System.out.println(result);
}

private static AWSCredentials getCredentials() {
AWSCredentials credentials = null;
credentials = new BasicAWSCredentials("<KEY>","<VALUE>");
return credentials;
}


}


where , are the secret active key and 'hadoop' is a keypair I created in the EC2 console.


After running I see the Job trying to start in the EMR console, after 1 minute it changes from 'starting' to 'Terminated with errors Validation error'


no other information is given


Any ideas what goes wrong?


Thanks!





Aucun commentaire:

Enregistrer un commentaire