amazon web services: août 2015

lundi 31 août 2015

Hive table to json and upload to S3

I have a hive table in the following format:

col1 col2
1 {"test": "hi"}
2 {"test2": "hi2"}
2 {"test3": "hi3"}

I can perform all queries, such as select * from table. What would be the best way to go about transforming that table to a text file with each line being a json string such as this:

{"id": 1, "test": "hi"}
{"id": 2, "test2": "hi2"}
{"id": 3, "test2": "hi3"}

Would it be transform call with a mapper? Also, after I have the text files with the json lines, I would like to upload the text file to a S3 bucket. In python, I could use boto to upload it, but in a Hive environment, is there such functionality?

Thanks

How to get cognito id in a lambda function ?

How can I get the the cognito id of the user initiating the action in a Lambda function ?

the lambda function is triggered on s3 upload by a cognito authenticated user.

if I look in : event.Records[0].userIdentity.principalId, all i get is a string saying "CognitoIdentityCredentials"

Difference between azure search and aws search?

I am going to work on cloud search so Please explain what is the differences between Azure search and AWS search?

which is better for me in terms of Cost,Performance,Easy to develop?

python version confliction while accesing yum command

My amazon aws system has two python, 2.6 and 2.7

Default python which I say which python is : Python 2.7.9

I also set symbolic link using

sudo ln -s /usr/bin/python2.7 /usr/bin/python

When I do

yum
bash: /usr/bin/yum: /usr/bin/python2.62.7: bad interpreter: No such file or directory

also

echo $PATH
/sbin:/bin:/usr/sbin:/usr/bin:/opt/aws/bin

I guess python 2.6 and 2.7 linked togather?

is there any way to get rid of this issue?

how to resolve Exception in thread "main" com.amazonaws.AmazonClientException: Unable to build cipher: Illegal key size using aws s3

I am trying for encryption and decryption using amazon aws. I got exception like

Exception in thread "main" com.amazonaws.AmazonClientException: Unable to build cipher: Illegal key size
Make sure you have the JCE unlimited strength policy files installed and configured for your JVM
    at com.amazonaws.services.s3.internal.crypto.ContentCryptoScheme.createCipherLite(ContentCryptoScheme.java:190)
    at com.amazonaws.services.s3.internal.crypto.ContentCryptoMaterial.wrap(ContentCryptoMaterial.java:823)
    at com.amazonaws.services.s3.internal.crypto.S3CryptoModuleBase.buildContentCryptoMaterial(S3CryptoModuleBase.java:535)
    at com.amazonaws.services.s3.internal.crypto.S3CryptoModuleBase.newContentCryptoMaterial(S3CryptoModuleBase.java:483)
    at com.amazonaws.services.s3.internal.crypto.S3CryptoModuleBase.createContentCryptoMaterial(S3CryptoModuleBase.java:449)
    at com.amazonaws.services.s3.internal.crypto.S3CryptoModuleBase.putObjectUsingMetadata(S3CryptoModuleBase.java:165)
    at com.amazonaws.services.s3.internal.crypto.S3CryptoModuleBase.putObjectSecurely(S3CryptoModuleBase.java:159)
    at com.amazonaws.services.s3.internal.crypto.CryptoModuleDispatcher.putObjectSecurely(CryptoModuleDispatcher.java:107)
    at com.amazonaws.services.s3.AmazonS3EncryptionClient.putObject(AmazonS3EncryptionClient.java:485)
    at testKMSkeyUploadObject.main(testKMSkeyUploadObject.java:91)
Caused by: java.security.InvalidKeyException: Illegal key size
    at javax.crypto.Cipher.checkCryptoPerm(Cipher.java:1039)
    at javax.crypto.Cipher.implInit(Cipher.java:805)
    at javax.crypto.Cipher.chooseProvider(Cipher.java:864)
    at javax.crypto.Cipher.init(Cipher.java:1396)
    at javax.crypto.Cipher.init(Cipher.java:1327)
    at com.amazonaws.services.s3.internal.crypto.ContentCryptoScheme.createCipherLite(ContentCryptoScheme.java:187)
    ... 9 more

please help me.

when I was trying to put object for doing encryption using AmazonS3EncryptionClient I am getting exception. How to resolve this error.

AmazonS3EncryptionClient s3 = new AmazonS3EncryptionClient(credentials,materialProvider);

        PutObjectRequest putRequest = new PutObjectRequest(
                bucket, kms_cmk_id, new ByteArrayInputStream(plaintext), metadata);

        ObjectMetadata objectMetadata = new ObjectMetadata();
        objectMetadata.setSSEAlgorithm(ObjectMetadata.AES_256_SERVER_SIDE_ENCRYPTION);   

        putRequest.setMetadata(objectMetadata);



         System.out.println(putRequest.getKey());
        s3.putObject(putRequest); //getting exception here

Is there a way to get actual user ID via Amazon login?

Following this manual http://ift.tt/1Q5a3wq I was able to get user ID which as I understand is an unique for my application/user. This ID has following format: "amzn1.account.XXXXXXXXXXXXXXXXXXXXXXXXXXXX".

But I want to somehow get a link to user profile which looks like "http://ift.tt/1LPFZX8".

Is there any way to achieve this?

What is the difference between the AWS boto3 and boto3

I'm new to AWS using Python and I'm trying to learn the boto API however I notice there are two major versions/packages for Python. That would be boto, and boto3.

I haven't been able to find an article with the major advantages/disadvantages or differences between these packages.

Thank you.

How to copy/move the downloaded files from S3 bucket to a different folder under the same bucket and not download load the latest file

Hi I have a unique problem to solve, any idea, suggestions or even a code helps!! Thank you in advance!

I am using python 2.7.x, and Boto API 2.X to connect to AWS S3 bucket. I have a unique situation where I want to download files from S3 bucket that to from a specific directory/folder say ‘myBucket/foo/'. But the catch is I want to leave a latest file behind in S3 folder and not download it. Once, I download these files on my local box, I want to move these files out to a different folder under the same bucket say ‘myBucket/foo/bar/'. Has anyone worked on similar situation before? Your help is greatly appreciated!!

About SMS service from AWS SNS

I want SNS to send SMS to cell phones. I checked the AWS document, it says

SMS notifications are currently supported for phone numbers in the     
United States. SMS messages can be sent only from topics created in the 
US East (N. Virginia) region. However, you can publish messages to 
topics that you create in the US East (N. Virginia) region from any 
other region.

But I don't know how to publish a message to the topics that I create in the US East region from US West which my aws nodes are located.

Googled a lot, but without luck. Does anybody here know how?

Force visitors to use SSL by default - Apache2

I am using an Amazon EC2 instance with Ubuntu 14.04 and running an apache2 webserver. I bought my domain with godaddy and migrated the nameservers to cloudflare. Since cloudflare offers free SSL, I have that enabled. I have never built a website to use SSL only and I know this is possible because many websites are SSL by default. So my question is how to have visitors use SSL only site-wide?

Thanks

Best Config / Instance for Magento 1.3 on AWS EC2 RDS

Hello We have a dedicated server currently and wanting to move Magento to Amazon Web services.

We have 1) Magento 1.3CE ( will be upgraded in the next few months to multi store ). It uses APC cache currently and CDN. 2) 1500-2500 sessions per day : Typically 10-20 concurrent users and fortnightly 20-70 concurrent users for a few hours - very occassionally 200 concurrent users. 3) 10 wordpress sites - latest versions ( one integrated with Magento ) 4) All running MYSQL , PHP 5.4

We want 1) Room to Grow 2) Use RDS 3) Enough grunt to be very responsive 4) SSD

I have narrowed instances down to m3 , c3 or r3 AWS instances.

Question : Which instance is recommended - and ( assuming UBUNTU ) what default should I load ; some sort of LAMP Stack ( should it be without MYSQL because I am using RDS? ) . any thoughts appreciated.

Thanks so much for your time :-)

How to efficiently implement a newsletter system to prevent duplicate e-mails?

We run an e-commerce site with approximately 500,000 active customers. We regularly send newsletters via an external mail service (http://ift.tt/1lFeFI1) however as the company grows and we get more customers, our newsletter costs increase. I recently came across Amazon SES which is perfect for us as they are cheap and since we are already using AWS for our servers, it makes things a little easier to maintain. I know there's cheaper alternatives, but I would prefer to stick to Amazon SES.

Let's say I create a newsletter in my system that sends an e-mail to everyone that's having a birthday today. It just so happens that 400,000 people have a birthday today, so 400,000 e-mails get sent. The next day, the remaining 100,000 people have birthdays as well as 100,000 more people that already received the birthday e-mail yesterday but changed their birth date to this day because they entered it incorrectly when they signed up. The system now sees 200,000 e-mails to send, but should know to only send to 100,000 of them because 100,000 already received the e-mail for the time period (365 days).

The problem I'm having is regarding database and script efficiency.

On the first day, I store the e-mail addresses of 400,000 people in a table called "sent". On the second day, I retrieve the list of all e-mail addresses to send to. I then have the following options:

Query the "sent" table 200,000 times to see which e-mail addresses to send to - Extremely inefficient, plus unnecessary load on the server that is running the site itself.
Retrieve the entire "sent" table, store into an array and filter accordingly - Fast (just 1 DB query), but requires a lot of RAM. This is currently how I do things, but obviously I receive PHP memory exhausted errors, so it's no longer viable, especially as the site grows.
Sort the 200,000 e-mail addresses alphabetically and query the "sent" table in batches, such as all of the A e-mail addresses, all of the B e-mail addresses, and so on. - Fast and should use less RAM depending on the batch conditions, but what if all 200,000 e-mail addresses begin with the letter S?
Something else?!

I realize that these are very extreme examples that will never happen in a real environment, but I would prefer to (re)implement this correctly now and not have to revisit it again in the future.

If Amazon SES has this functionality built in, I would love to know where I can find it! I've searched their docs but I can't see it.

Does anyone have any experience in this field, or have any suggestions? I've been racking my brain for days trying to come up with a good solution, but I just can't come up with anything.

HELP! :)

EDIT: I'm using MySQL. I've looked into No-SQL but don't feel it's necessary.

EDIT: The birthday thing is just an example of a newsletter. It should be a completely generic system that does not rely on any predefined notions of what the data contains.

EDIT: The server already implements good e-mail practices such as SPF, DKIM, and so forth.

Amazon SQS Tagging

We are trying to setup Amazon SQS between two AWS applications. Management wants to track cost associated with all Amazon resources. Is it possible to tag Amazon Simple Queue Service resources?

Trying to use logstash to index FROM cloudwatch logs

So we have logs (apache, tomcat, etc) stored in aws cloudwatch logs. I'm trying to use logstash to index from aws logs and send them over to elasticsearch/kibana.

I can't seem to find a plugin to accomplish this. Has anyone tried this and was successful? I don't want the metrics, just the logs stored in aws logs.

How to distribute long-lived TCP connections to individual nodes, e.g. to RabbitMQ nodes in a cluster?

I'm about to deploy a cluster of RabbitMQ nodes in AWS. I expect there would be 100k+ (possibly 1m+) clients - IoT devices - which will connect to the RabbitMQ cluster over MQTT and will stay connected 24/7/365.

Now the question is: how to distribute the TCP connections from clients to individual nodes comprising the RabbitMQ cluster? I see a couple of options:

Use AWS Elastic Load Balancer
Use HAProxy on AWS EC2
Use DNS round-robin
Use DNS SRV (e.g. like in SIP service deployments)
Use a custom hand-written client-side load balancing algorithm, based on a list of addresses obtained from a server or hard-coded into the device

Which solution from the ones listed above would you recommend given the amount of load and the expected duration each connection? Or maybe a combination of these? Or is there any other solution worth knowing?

Running a .NET Web Server on AWS

I am not sure the steps I have taken are ok so I will write what I've done:
1) Created a new Windows Server 2012.
2) Moved my .NET web project files into inetpub\wwwroot.
3) With server manager -> added Web Server (IIS) role
4) On IIS Manager added my website with physical path to the project Dir.
5) Set up index.html as my Default Document.

After that I entered my instances public DNS, and got to index.html, but all the API calls from JS, get 500 (Internal server error).

I'm not sure what the problem is, can anyone point me to the right direction?

Connect IntelliJ to Amazon Redshift

I'm using the latest version of IntelliJ and I've just created a cluster in Amazon Redshift. How do I connect IntelliJ to Redshift so that I can query it from my favorite IDE?

Python Django AWS Internal Server Error

I am currently trying to deploy just a dummy django project to AWS and am having nothing but problems, i asked amazon support and they told me it is a code related issue. Every time i run the project locally i get the normal Welcome to Django page but once i use the eb deploy and publish it i received

"Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator at root@localhost to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log."

I have no way of knowing what the problem even is, the log and deploy don't specify that anything went wrong. Is there anything that could cause this issue?

i used this site as a guide http://ift.tt/1QLXiYn

thank you!

Amazon S3 Presigned URL: Forbidden

I am trying to generate pre-signed url with .Net sdk from Amazon.

I am setting config like this:

GetPreSignedUrlRequest request = new GetPreSignedUrlRequest()
        {
            BucketName = "content.enloop",
            Key = checkinId + "/" + picId + ".jpg",
            ContentType = "image/jpeg",
            Expires = DateTime.Now.AddHours(1),
            Verb = HttpVerb.PUT
        };

and this is result:

http://ift.tt/1X6PPHv

I have the same code in javascript (NodeJs) and I can upload pictures. With this one I get 403 error. The only difference I can see in urls is that nodejs version contains "&Content-Type=image%2Fjpeg&" . Any ideas what might be wrong? Method in nodejs I use is getSignedUrl()

Composite key in DynamoDB with more than 2 columns?

I'm exploring the use of DynamoDB in the application I work on, which currently only has one database component -- a MySQL database running on RDS.

We pretty heavily use AWS and use a sharding scheme for our databases, but it can only get us so far without manual intervention. Playing around with Aurora I actually saw a significant drop in performance vs our MySQL database, so I'm evaluating DynamoDB to see it will work for us, as it can efficiently store JSON data, and also scale easily (just increase the reads or writes per second in the AWS console and let Amazon do the heavy lifting).

In several of our MySQL tables we have a primary key that is an autoincrement column, but we also have several indices on top of that to support query performance in other ways. The other indices are crucial as some of our tables have over 1 billion rows in them. In essence, we partition things among a client, an object_name, etc. So I might do something like this in MySQL:

Create Table: CREATE TABLE `record` (
  `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  `client_id` int(10) unsigned NOT NULL,
  `data_id_start` bigint(20) unsigned NOT NULL,
  `data_id_end` bigint(20) unsigned NOT NULL DEFAULT '8888888888888888',
  `object_name` varchar(255) NOT NULL,
  `uuid` varchar(255) NOT NULL,
  `deleted` tinyint(1) unsigned NOT NULL DEFAULT '0',
  ...
  PRIMARY KEY (`id`),
  ...
  KEY `client_id_object_name_data_id_data_id_end_deleted` (`client_id`,`object_name`,`data_id_start`,`data_id_end`,`deleted`),
  KEY `client_id_object_name_data_id_end_uuid_id` (`client_id`,`object_name`,`data_id_end`,`uuid_id`),
  ...
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8;

I'm evaluating duplicating some of this data into DynamoDB to use as a cache, so we don't have to go out to S3 to retrieve stored data there under certain situations. Instead, I'll just store the JSON data directly in the cache. In DynamoDB, it looks like I could use a HASH or a HASH and RANGE attribute in a key. So for example, I could use the autoincrement column from our MySQL table as the HASH, but then all of the examples I see of RANGE keys, global/local secondary indices, etc. only specify ONE other attribute as the RANGE. I want to create an index for efficient lookup when 3 or more values are specified in the "where" clause.

For example, I would like to query this table using an expression like this:

var params = {
    TableName: "Cache",
    KeyConditionExpression: "clientId = :clientId and objectName = :objectName and uuid = :uuid",
    ExpressionAttributeValues: {
        ":clientId": 17,
        ":objectName": "Some name",
        ":uuid": "ABC123-KDJFK3244-CCB"
    }
};

Notice that my "where clause" in the KeyConditionExpression uses 3 values. It's possible that we might have 4 or 5 values there. So is there any way in DynamoDB to create composite keys that have more than 2 attributes (columns) in them?

If not, I suppose that I could concatenate the 3 columns into a string and use that as my primary key on each insert. Or at least concatenate clientId and objectName, then use uuid as a RANGE or something like that. Effectively I need to page through all values for a specific clientId/objectName combination, and then based on some of the attributes in each row either take its value directly from the cache, or consider it a miss and retrieve the value from S3 (which is considerably slower).

Error File not found, cakephp 3.0

I am using the cakephp framework on Amazon Web Services. I am trying to open and read a file using this code -

$pic_file = new File(WWW_ROOT . 'fb_profile_pics' . DS . $path . h($user->user_id) . '.txt'); $fb_pic = $pic_file->read(true,'rb');

the WWW_ROOT var above is - /var/app/current/webroot/

The fb_profile_pics folder(location of file) is in the webroot folder in my cakephp application. I am not able to read this file and cannot figure what is wrong. Any help to get me going again would be awesome.

Deploying Docker to AWS Elastic Beanstalk -- how to forward port to host? (-P flag)

I have a project set up with CircleCI that I am using to auto-deploy to Elastic Beanstalk. My EBS environment is a single container, auto-scaling, web environment. I am trying to run a service that listens on raw socket port 8080.

My Dockerfile:

FROM golang:1.4.2

...

EXPOSE 8080

My Dockerrun.aws.json.template:

{
  "AWSEBDockerrunVersion": "1",
  "Authentication": {
    "Bucket": "<bucket>",
    "Key": "<key>"
  },
  "Image": {
    "Name": "project/hello:<TAG>",
    "Update": "true"
  },
  "Ports": [
    {
      "ContainerPort": "8080"
    }
  ]
}

I have made sure to expose port 8080 on the "role" assigned to my project environment.

I used the exact deployment script from the CircleCI tutorial linked above (except with changed names).

Within the EC2 instance that is running my EBS application, I can see that the Docker container has run successfully, except that Docker did not forward the exposed port to the host container. I have encountered this in the past when I ran docker run .... without the -P flag. How can I force this behavior in my application?

Thanks in advance.

AWS Opsworks : AWS Flow (ruby) layer cant be combined with Custom Layer?

My objective:
To have some instances in my stack be members of an AWS Flow (Ruby) layer and a custom layer.

What have I tried:
I have a AWS Flow (Ruby) layer that is working great. (thanks AWS!)
I have also created a custom layer.
My instance is a member of both layers.
I have a custom application type 'Other' called sqspoller
I try to deploy my custom application to my custom layer.

What I expected to happen:
Opsworks chef deploys my code to directory specified in my app setup.

What has happened:
Info in deployment log (which exits as a success).

2015-08-31T16:58:48+00:00] INFO: Skipping deploy::aws-flow-ruby application sqspoller as it is not an AWS Flow Ruby app

It appears since I have an AWS-Flow Ruby layer, only aws-flow ruby app can be deployed to instances in that layer?

A general help regarding the tools required for cloud app without ssl

I know this may be basic, but i really need your help to getting started because i need to choose the right horses to ride now .

I need to have some cloud application that do this :

User can access to it via http requests (not ssl! ) and perform one of these(according to the rest api request) :

register to some basic table with an ID and a few other fields.
check some conditions from a user table, and then send a push notification (APNS) to iOS device (Amazon sns? )
check some condition and perform some https(ssl) request to some other third party service.

So basically save basic data, make basic decision based on the data, and preform one of the 2 : push notification / other https request . Should serve many users.( 1000- 10,000)

I know only C/C++/objective-C(very good).

Parse.com services are not good, because its an ssl ( i cant use ssl for the API)

Amazon AWS seems like the best choice , but i wonder how to start and if its something that would take a long learning process or not ,and what exact service do i need.

Any advice on how to make this app alive ,what service is fit the best, where to start, would be great .

How do I switch elastic IPs on a Linux AWS without getting disconnected from XShell?

I'm using XShell to access a Linux AWS. I know how to switch elastic IP addresses with Ruby, but when I do, XShell's connection to the AWS gets busted.

Here's script.rb I run on the AWS via XShell with the ruby script.rb command:

require 'json'
require 'pp'

region = 'REGION NAME'
aws_access_key_id = 'SECRET KEY ID'
aws_secret_access_key = 'SECRET ACCESS KEY'
credentials = Aws::Credentials.new(aws_access_key_id, aws_secret_access_key)
client = Aws::EC2::Client.new(
    region: region,
    credentials: credentials
)
pp client
p '==='

describeAddresses = `aws ec2 describe-addresses`
awsHash = JSON.parse(describeAddresses)
pp awsHash
p '==='

getInstanceID = `wget -O - 'http://ift.tt/1IpSH7i' 2>/dev/null`
instanceID = getInstanceID.split("\n")[-1]
pp instanceID
p '==='

resp = client.allocate_address({
  domain: "vpc", # accepts vpc, standard
})
pp resp
p '==='

resp2 = client.associate_address({
    instance_id: instanceID,
    allocation_id: resp['allocation_id'],
    allow_reassociation: true#,
})
pp resp2
p '==='

How do I implement Live streaming feature to an iOS app?

Im working on a small project that will provide its users similar features of meerkat or periscope, which is to broadcast live videos to other users. However, because this area(mobile live-streaming) is totally new to me. I don't know where to start. So, there's 3 questions i'd like to ask. 1. How do i actually implement HLS feature to my app? 2. Can you use AWS as a server for a service that broadcasts live-stream videos? 3. Are there any special libraries or functions that apple provides to support live streaming in iOS apps?

Thank you.

My SQL queries in JSP won't work on server environments

I have a JSP which makes some JDBC SQL query to a database and prints them out.

The JSP works great on my Tomcat localhost server.

But it doesn't work when i deploy the JSP to AWS Elastic Beanstalk. The logfile throws the exception below. But why does the code work on Tomcat local host then? (JSP and elastic beanstalk log files below)

Exception:

org.postgresql.util.PSQLException: Connection to postgres.ctmidyquyj6t.eu-central-1.rds.amazonaws.com:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

JSP:

<%@ page language="java" import="java.sql.*,java.util.*" %> 
<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <title>Twitter</title>
  </head>

  <body>
    <%
    Class.forName ("org.postgresql.Driver");  
    Connection postgresConnection = DriverManager.getConnection ("jdbc:http://postgresql*****/Twitter",
            "postgres", "****");

    String query = new StringBuilder().append("SELECT * from tswholeworld").toString();
    PreparedStatement ps = postgresConnection.prepareStatement(query);
    ResultSet rs = ps.executeQuery();
    StringBuilder stringBuilder = new StringBuilder();

        while(rs.next()) {
            String lat = rs.getString(3);
            String lon = rs.getString(4);
            stringBuilder.append("new google.maps.LatLng("+lat+", "+lon+"), ");
        }
    String finalString = stringBuilder.toString();
    %>
    <%=finalString%>
  </body>
</html>

Log file last 100 lines:

-------------------------------------
/var/log/tomcat8/catalina.out
-------------------------------------
    at java.net.Socket.connect(Socket.java:589)
    at org.postgresql.core.PGStream.<init>(PGStream.java:61)
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:121)
    ... 36 more
org.postgresql.util.PSQLException: Connection to postgres.ctmidyquyj6t.eu-central-1.rds.amazonaws.com:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:239)
    at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
    at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:127)
    at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29)
    at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21)
    at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:41)
    at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
    at org.postgresql.Driver.makeConnection(Driver.java:414)
    at org.postgresql.Driver.connect(Driver.java:282)
    at java.sql.DriverManager.getConnection(DriverManager.java:664)
    at java.sql.DriverManager.getConnection(DriverManager.java:247)
    at com.journaldev.program.twitterstream.getStream(twitterstream.java:14)
    at com.journaldev.servlet.HomeServlet.doGet(HomeServlet.java:24)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
    at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:676)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:617)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1521)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1478)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection timed out
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at org.postgresql.core.PGStream.<init>(PGStream.java:61)
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:121)
    ... 36 more
org.postgresql.util.PSQLException: Connection to postgres.ctmidyquyj6t.eu-central-1.rds.amazonaws.com:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:239)
    at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
    at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:127)
    at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29)
    at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21)
    at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:41)
    at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
    at org.postgresql.Driver.makeConnection(Driver.java:414)
    at org.postgresql.Driver.connect(Driver.java:282)
    at java.sql.DriverManager.getConnection(DriverManager.java:664)
    at java.sql.DriverManager.getConnection(DriverManager.java:247)
    at com.journaldev.program.twitterstream.getStream(twitterstream.java:14)
    at com.journaldev.servlet.HomeServlet.doGet(HomeServlet.java:24)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
    at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:676)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:617)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1521)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1478)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection timed out
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at org.postgresql.core.PGStream.<init>(PGStream.java:61)
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:121)
    ... 36 more

List of valid states in boto3

I want to list all stacks that not have DELETE_COMPLETE as status value

on boto2 it was very easy:

import boto3
cf = boto.cloudformation.connect_to_region(region)
status_filter = [st for st in cf.valid_states if st != 'DELETE_COMPLETE']
for stack in cf.list_stacks(stack_status_filters=status_filter):
    ...

boto3 use botocore with many service-2.json files for every AWS service. But I can't find any list all possible status flags.

I can find this in the service-2.json (botocore/data/cloudformation/2010-05-15/service-2.json) but how is the right way to access this list?

Skip stacks in the for-loop and code the list in my script is no option.

Test Kitchen adding env variable to windows vagrant box

I am trying to implement a local testing environment for our windows chef cookbooks using vagrant/test kitchen. Inside of our cookbooks we have some recipes that need authentication information for AWS. Each dev has their personal aws authentication information on their local machine.

How do I import the local AWS credentials onto the vagrant box with the kitchen.yml file?

EC2 to S3 file uploads. How can I tolerate ec2 failure during upload?

I have a service that lives in an autoscaling group. A client uploads a file to the autoscaling group - ec2 machines - some work is done and then the files are transmitted to S3.

My question is, what is a good way to recover from a failure of an EC2 machine that is in the middle of a transfer to S3.

Python version error while installation

when I do yum, it says

bash: /usr/bin/yum: /usr/bin/python2.62.7: bad interpreter: No such file or directory

my aws instance details are as given below:

Linux ip-72-311-15-127 3.14.27-25.47.amzn1.x86_64 #1 SMP Wed Dec 17 18:36:15 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

/usr/bin/apt-get also not available

/usr/bin/yum is there so I tried

/usr/bin/yum install -u mongodb-org

which gives

bash: /usr/bin/yum: /usr/bin/python2.62.7: bad interpreter: No such file or directory

Any suggestion for this?

Monitor API Gateway APIs based on API key

How can I monitor my Amazon API Gateway APIs API key wise?

Currently it is showing data for all API keys, but I want to display API calls, 5xx errors, 4xx errors etc for particular API key.

aws s3 not able to delete the object by code php

I am putting a object in my AWS S3 basket by using following code.

        $temp = explode(".", $_FILES["file"]["name"]);
        $_FILES["file"]["name"] = $newfilename = round(microtime(true)) . '.' . end($temp);

        $filepath = $_FILES['file']['tmp_name'];
        try {
            $result = $this->Amazon->S3->putObject(array(
                'Bucket' => 'mytest.sample',
                'ACL' => 'authenticated-read',
                'Key' => 'files/image/' . $id . '/' . $newfilename,
                'ServerSideEncryption' => 'aws:kms',
                'SourceFile' => $filepath,
                'ContentType' => mime_content_type($filepath),

            ));
        } catch (S3Exception $e) {
            echo $e->getMessage() . "\n";
        }

and it is working fine. files are uploading but when I try to delete the file empty response I am getting. here is the code used for deleting object.

try {
                $result = $this->Amazon->S3->deleteMatchingObjects(array(
                    'Bucket' => 'mytest.sample',
                    //'Key' => 'files/image/' . $id . '/' . $fileName['Attachment']['attachment'],
                    'Key' => 'files/image/45/1441026402.docx', //static path for checking.
                    'VersionId' => 'latest',
                ));
            } catch (S3Exception $e) {
                echo $e->getMessage() . "\n";
            }

I tried below method too

   try {
        $result = $this->Amazon->S3->deleteObject(array(
            'Bucket' => 'mytest.sample',
            //'Key' => 'files/image/' . $id . '/' . $fileName['Attachment']['attachment'],
            'Key' => 'files/image/45/1441026402.docx',
            'VersionId' => 'latest',
        ));
    } catch (S3Exception $e) {
        echo $e->getMessage() . "\n";
    }

Is there any permission issue?

My SQL queries in JSP won't work on server environments

I have a JSP which makes some SQL query to a database and prints them out (code below).

The JSP works great on my Tomcat or Google App Engine localhost server.

But it doesn't work when i upload my WAR to AWS Elastic Beanstalk or Google App Engine server environments. Can you tell me why?

<%@ page language="java" import="java.sql.*,java.util.*" %> 
<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <title>Twitter</title>
  </head>

  <body>
    <%
    Class.forName ("org.postgresql.Driver");  
    Connection postgresConnection = DriverManager.getConnection ("jdbc:http://postgresql*****/Twitter",
            "postgres", "****");

    String query = new StringBuilder().append("SELECT * from tswholeworld").toString();
    PreparedStatement ps = postgresConnection.prepareStatement(query);
    ResultSet rs = ps.executeQuery();
    StringBuilder stringBuilder = new StringBuilder();

        while(rs.next()) {
            String lat = rs.getString(3);
            String lon = rs.getString(4);
            stringBuilder.append("new google.maps.LatLng("+lat+", "+lon+"), ");
        }
    String finalString = stringBuilder.toString();
    %>
    <%=finalString%>
  </body>
</html>

Which EC2 AWS CLI tools should I be using?

Amazon seems to have multiple tools that to do the same thing to manage EC2; see below:

http://ift.tt/1ImGTZ6 (Python version)
http://ift.tt/1tveDrR (Java version)

Which ones should I be using? They both seem to be maintained by Amazon.

Run servlet before anything else

I have a dynamic web project with a servlet (structure below).

When right-clicking HomeServlet.java --> Run as --> Run on Server --> Tomcat...
It runs fine with the URL: http://localhost:8080/JSTLTest/HomeServlet

When right-clicking JSTLTest --> Run as --> Run on Server --> Tomcat...
I get an error and this URL is used: http://localhost:8080/JSTLTest/

The problem is, i want to deploy it on a server like AWS elastic beanstalk and i guess it doesn't work because he won't use the servlet to start, like:
http://ift.tt/1LNAFDvHomeServlet

Is my web.xml wrong?

web.xml

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://ift.tt/ra1lAU"
    xmlns="http://ift.tt/19L2NlC"
    xsi:schemaLocation="http://ift.tt/19L2NlC http://ift.tt/1drxgYl"
    version="3.1">
    <display-name>JSTLTest</display-name>
    <welcome-file-list>
        <welcome-file>index.html</welcome-file>
        <welcome-file>index.htm</welcome-file>
        <welcome-file>index.jsp</welcome-file>
        <welcome-file>default.html</welcome-file>
        <welcome-file>default.htm</welcome-file>
        <welcome-file>default.jsp</welcome-file>
    </welcome-file-list>
    <servlet>
        <servlet-name>Home</servlet-name>
        <servlet-class>com.journaldev.servlet.HomeServlet</servlet-class>
    </servlet>
    <servlet-mapping>
        <servlet-name>Home</servlet-name>
        <url-pattern>/HomeServlet</url-pattern>
    </servlet-mapping>
</web-app>

Structure:

Tomcat not accessible from EC2 Instance

I am not able to access the tomcat using public IP of EC2 Instance. I can access it using http://localhost:8080 internally but not from outside. In fact, It was running good since 2 months but suddenly it stopped and now I am hanging in between. locally my application is running fine and I can ping to my Instance using Public IP. Security group defines access to all traffic (TCP,HTTP,SSH). Then why Tomcat/Application is not being accessed externally. Any help will be highly appreciated. Thanks

EMR: How to specify EBS attached to instances in boto3 run_job_flow?

This is a continuation of my questions on running out of space in root device on EMR. See other questions like this here, here and here.

The hive intelligence seems to suggest that an EBS volume attached to the instances is the right way to get around this. But I can't figure out how to specify the "InstanceGroups" dict of the boto3 run_job_flow method.

There is some advice here for EC2 and boto2 here, but I am not sure how that translates into boto3 advice.

Aws data transfer rate: $0.010 per GB - regional data transfer - in/out/between EC2 AZs or using elastic IPs or ELB

AWS have charged me $0.09 for data transfer. below is the line from bill details which shows the data usage of 8.988 GB and charge of $0.09

$0.010 per GB - regional data transfer - in/out/between EC2 AZs or using elastic IPs or ELB 8.988 GB    $0.09

But in my region all servers are in the same region and same availability zone.i am also not using ELB. Is this charge apply for elastic IPs which i am using for accessing web site though browser and for doing ssh?? if yes then why AWS have not charged me for data transfer rate in the previous month bills.

Uploading multiple files at same time from local to s3 bucket through node js

I am trying to upload a files to the s3 bucket,The following code I am using to accomplish this operation.

var params = {
  localFile: "../Processor/1.wav",

  s3Params: {
    Bucket: "bucketname",
    Key: "1.wav",
  },
};
var uploader = client.uploadFile(params);
uploader.on('error', function(err) {
  console.error("unable to upload:", err.stack);
});
uploader.on('progress', function() {
  console.log("progress", uploader.progressMd5Amount,
            uploader.progressAmount, uploader.progressTotal);
});
uploader.on('end', function() {
  console.log("done uploading");
});

Every thing works fine till this point. Now lets say here, I want to upload 5 files from local path to s3 bucket, How can I achieve that is there any direct method providing amazon for multiple file uploads or I need to use async module.?

Unable to run my application after customizing AMI on aws ec2 worker tier instance

I am new to elastic bean stalk. I am running sample example given by aws for ebs which is working fine with default instances. When I am trying to rebuild this worker tier environment with customized image its showing red health and example application also failing to execute its job on this instance. Can someone please help me out to resolve this issue. In case more details require please comment.

Thanks, Nil.

Is it possbile to get the user name and password of one AWS account using the secret key and access key

I have secret key and access key of one root account, can i find the username and password for the same

Connect to Amazon ec2 through GUI from ubuntu 14.04

How can I connect to Amazon ec2 through a GUI? I believe this requires setup of gui in the instance. I tried many links, right out of the first page, after googling, but I got stuck at the stage, where I have to use the VNC client to connect. Below is a sample client I am trying to use

Taking a sample amazon instance public dns

    ec2-<ip address>.compute-1.amazonaws.com

How can I connect using VNC,or tightVNc after all the setup has been done?

Is this the right approach to index creation in DynamoDB?

I want to create an app that has a list of clients with ids (emails in this case), their phone number and other important information. Most of the time the clients table will be searched by using the client id (their email) but occasionally I want to be able to do this search using their phone number. Basically I want the app to have a text field where you can either type the email or phone number and be able to retrieve the client data.

Client ( ID, PhoneNumber, Name, LastName, etc...)

After researching on DynamoDB, I came up with the solution of having a table for the clients and having an index hash key for the ids with a lot of throughput for read and write since querying based on this attribute will be the most common task. Then, I created a global secondary index, with a key for the attribute phoneNumber, and giving this a low throughput number for reading and writing since the search of a client by its phone number won't be too occasional. However, The app will never make an update using the phone number as a key, it will only make updates using the id as a key.

Is this the right approach, or would there be a better thing to do? Are the throughput values right based on my needs or do you think that there is no need to have any write throughput values for the secondary index? Is there maybe something wrong with thought process?

Thank you very much!

dimanche 30 août 2015

Setup incoming email on ec2

I am trying to set up email accounts on AWS E2 instance. Surprisingly, I couldn't find the documentation directly from AWS. How can I set up incoming email accounts?

Amazon S3 sync Deleting excluded files

I have 2 buckets, Bucket A and Bucket B. Bucket A contains Javascript files and bucket B contains a mix of javascript and other file types. I am trying to do a sync of only JS files from bucket A to B.

I am using the following:

aws s3 sync s3://bucket-a s3://bucket-b --delete --exclude "*" --include "*.js"

I was assuming that this will leave bucket B with an exact copy of all of the js files. However the above command will start deleting all of the non js files in Bucket B.

When I run the following command:

aws s3 sync . s3://bucket-b --delete --exclude "*" --include "*.js"

With the current directory containing a copy of bucket A, bucket B will have the same js files as bucket A and non js files will not be affected.

Why is this command functioning differently when syncing local to bucket compared to bucket to bucket?

aws ec2 instances in different vpc subnets access each other

I have 2 AWS EC2 instances living inside 2 different subnets of my vpc.

I would like to allow the ruby app running on the first instance (say App#1) to call the endpoints of the app (say App#2) running on the 2nd instance.

I would also like my users to directly call the endpoints of App#2 from their browser.

Here is what I have tried (and mostly failed):

[Sucess!] I added the known IP addresses of my users to the inbound rules of Load Balancer Security Group of App#2 and have confirmed that they can access App#2 endpoints from their browsers.
[Fail!] I added the Load Balancer Security Group ID of App#1 to the inbound rules to the Load Balancer Security Group of App#2. But my logs tell me App#1 cannot access the endpoints of App#2.
[Fail!] I added the VPC Security Group ID of App#1 to the inbound rules of the Load Balancer Security Group of App#2 - nope, still doesn't work. (Somehow, when I launched the instance for App#1, aws automatically created 2 security groups for this instance - one for VPC and one for load balancer... I have no idea why/how this happened...)
[Success...Sort Of] I assigned an elastic IP for the instance running App#1 and added that to the inbound rules of the Load Balancer Security Group of App#2. This works but I would rather not use this method since I would like to elastically scale my App#1 in the future and I do not know how to automatically assign more elastic IPs for the new instances when they spin up, add them to the inbound rules, and then somehow remove them when they shut down.

I feel like there has got to be a really clean solution to this problem and I am probably missing something painfully obvious. Can someone please give me a hint?

Any help would be appreciated!

EMR: Running out of local space

I am trying to bring up an EMR cluster (using boto3), but I constantly run out local memory for tasks such as logging and reading data from S3, etc. This is a cluster of d2.xlarges, which have nearly 4 TB of ephemeral storage.

When I do a df -hi, I see:

Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/xvda1       640K  146K  495K   23% /
devtmpfs         3.8M   483  3.8M    1% /dev
tmpfs            3.8M     1  3.8M    1% /dev/shm
/dev/xvdb        373M   359  373M    1% /mnt
/dev/xvdc        373M    13  373M    1% /mnt1
/dev/xvdd        373M    13  373M    1% /mnt2

When I do a lsblk, I see:

NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
xvda    202:0    0   10G  0 disk
└─xvda1 202:1    0   10G  0 part /
xvdb    202:16   0  1.8T  0 disk /media/ephemeral0
xvdc    202:32   0  1.8T  0 disk /mnt1
xvdd    202:48   0  1.8T  0 disk /mnt2

When I do a mount, I see:

proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=15692656k,nr_inodes=3923164,mode=755)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /dev/shm type tmpfs (rw,relatime)
/dev/xvda1 on / type ext4 (rw,noatime,data=ordered)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
/dev/xvdb on /mnt type xfs (rw,relatime,attr2,inode64,noquota)
/dev/xvdc on /mnt1 type xfs (rw,relatime,attr2,inode64,noquota)
/dev/xvdd on /mnt2 type xfs (rw,relatime,attr2,inode64,noquota)
/dev/xvdb on /media/ephemeral0 type xfs (rw,relatime,attr2,inode64,noquota)

This seems to indicate that only 10 GB is allocated to the local filesystem for logging, reading in data from S3 etc. How can I set this to be a higher percentage of the (ephemeral) memory? Should I mount one of the other available drives to the local filesystem and then use that as the "/"? Which one of the above should that be?

Potentially related, but their solution was to use a paramter specific to the MapR distribution, and it is not even clear to me how that would be used in a boto script.

AWS security group settings

Currently, I connect from my pc to a RedHat instance, and from this Redhat, i connect to Ubuntu. The thing is, I cannot manage to connect to Ubuntu, I have a timeout error.

In the security group of Ubuntu, I put the following settings in inbound:

Type=SSH Protocol=TCP Port=22 Source=PublicIPofRedHat/32

And from Redhad, I connect using ssh -i ~/.ssh/ubuntu.pem ubuntu@PublicIPofRedHat

This setting does not work, but I would like to know why.

I changed this setting and put SecurityGroupOfRedHat instead of IP and it works. But with this setting, it means that any instance within this security group can also access to my Ubuntu (which I dont want), right ?

Could you please help with these settings ? Explain me what is wrong and how could I fix it ?

Remote Upload Amazon S3

I have made an script to upload remote files by link to amazon s3 but does not work. I have no idea what's wrong. Below is all the code done and I'm using the Amazon SDK in its latest version:

config.php

<?php

return [
    's3' => [
        'key' => 'mykey',
        'secret' => 'mykey',
        'bucket' => 'mybucket',
        'region' => 'us-west-2',
        'version' => 'latest'
    ]
]

?>

start.php

<?php
use Aws\S3\S3Client;
require 'aws/aws-autoloader.php';

$config = require('config.php');

//S3

$s3 = S3Client::factory([
    'key' => $config['s3']['key'],
    'secret' => $config['s3']['secret'],
    'region' => $config['s3']['region'],
    'version' => $config['s3']['version']

]);
?>

upload.php

<?php

use Aws\S3\Exception\S3Exception;
require 'start.php';

error_reporting(0);
$get_url = $_POST["url"];
$url = trim($get_url);
if($url)
{
    $file = fopen($url,"rb");
    $directory = "animes";
    $valid_exts = array("php","jpeg","gif","png","doc","docx","jpg","html","asp","xml","JPEG","bmp"); 
    $ext = end(explode(".",strtolower(basename($url))));
    if(in_array($ext,$valid_exts))
    {
        $rand = rand(1000,9999);
        $filename = "$rand.$ext";
        $newfile = fopen($directory . $filename, "wb");
        try{
            $s3->putObject([
                'Bucket' => $config['s3']['bucket'],
                'Key' => "{$directory}/{$filename}",
                'ACL' => 'public-read'
        ]);


    } catch(S3Exception $e){
        die("Error.");
    }
        if($newfile)
        {
            while(!feof($file))
            {
                fwrite($newfile,fread($file,1024 * 8),1024 * 8);
            }
            echo 'File uploaded successfully';
            echo '**$$**'.$filename;
        }
        else
        {
            echo 'File does not exists';
        }
    }
    else
    {
        echo 'Invalid URL';
    }
}
else
{
    echo 'Please enter the URL';
}
?>

and index.php

<html>
    <head>
        <title>PHP File Upload From URL</title>
        <script type="text/javascript" src="jquery.js"></script>
        <script type="text/javascript">
        $(document).ready(function(){
            $("#1").hide();
        });
        function uploadfile(){
            $("#1").show();
            $("#disp").html("");
            var url = encodeURIComponent($("#url").val());
            $.ajax({
                url: "upload.php",
                data: "url=" +url,
                type: 'post',
                success: function(data)
                {
                    var findsucc = data.indexOf("successfully");
                    out=data.split('**$$**');
                    if(findsucc!=-1)
                    {
                        $("#disp").css({"color": "green"});
                        $("#disp").html(out[0]);
                        $("#link").html("<a href='./upload/"+out[1]+"'>Click here</a> to view");
                        $("#1").hide();
                    }
                    else
                    {
                        $("#1").hide();
                        $("#disp").css({"color": "red"});
                        $("#disp").html(data);
                        $("#url").val("");
                    }
                }
            });

        }
        </script>
    </head>
<body>
<div align='center' style='padding-top: 40px;color: #4e4e4e;'><h1>PHP File Upload From URL</h1></div>
<div align='center' style='padding-top: 30px;'>
Enter Remote URL: <input type="text" name="url" id='url' size="35"> <input type="submit" value="Upload" name="submit" style='cursor: pointer;' onclick='uploadfile()'>&nbsp;&nbsp;<img src='ajax-loader.gif' id='1'>&nbsp;&nbsp;<br /><br /><div align='center'><span id='disp'></span></div><br>
<div id='link'></div><br />
<div style=" padding-left: 20px;font-size: 10px;color: #dadada;" id="dumdiv">
</div>
</body>
</html>

Solr - Importing XML

System Tech Specs:

Amazon AWS EC2, 1GHZ, CPU1GB Memory, 30GB HardDrive, 4GB Swap, PHP5.3 (Codeigniter), MySQL.

When I use this "bin/post -c gettingstarted example/exampledocs/*.xml" to import the Solr XML examples and do a search via Query it gives results.

BUT when I do a cleanup using "bin/solr stop -all ; rm -Rf example/cloud/" and then restarting solr using "bin/solr start -e cloud -noprompt" exporting my MYSQL table into XML using "mysqldump --xml -u root -p dbname tablename > tablename.xml" and then again using "bin/post -c gettingstarted example/exampledocs/*.xml" to index my own MySQL table in XML format I get

POSTing file tblcity.xml (application/xml) to [base] 1 files indexed. COMMITting Solr index changes to http://localhost:8983/solr/gettingstarted/update... Time spent: 0:00:00.398

Then when I go to search using Query for items in my file I get

"response": {
"numFound": 0,
"start": 0,
"maxScore": 0,
"docs": []

Even though there are items in my list which I have queried.

SQS message acknowledgement

My Sring Boot application listens Amazon SQS queue. Right now I need to implement correct message acknowledgement - I need to receive a message, do some business logic a only after that in case of success I need to ack the message(delete the message from the queue). For example, in case of error in my business logic the message must be re-enqueued.

This is my SQS config:

    /**
     * AWS Credentials Bean
     */
    @Bean
    public AWSCredentials awsCredentials() {
        return new BasicAWSCredentials(accessKey, secretAccessKey);
    }

    /**
     * AWS Client Bean
     */
    @Bean
    public AmazonSQS amazonSQSAsyncClient() {
        AmazonSQS sqsClient = new AmazonSQSClient(awsCredentials());
        sqsClient.setRegion(Region.getRegion(Regions.US_EAST_1));
        return sqsClient;
    }

    /**
     * AWS Connection Factory
     */
    @Bean
    public SQSConnectionFactory connectionFactory() {
        SQSConnectionFactory.Builder factoryBuilder = new SQSConnectionFactory.Builder(
                Region.getRegion(Regions.US_EAST_1));
        factoryBuilder.setAwsCredentialsProvider(new AWSCredentialsProvider() {

            @Override
            public AWSCredentials getCredentials() {
                return awsCredentials();
            }

            @Override
            public void refresh() {
            }

        });
        return factoryBuilder.build();
    }

    /**
     * Registering QueueListener for queueName
     */
    @Bean
    public DefaultMessageListenerContainer defaultMessageListenerContainer() {
        DefaultMessageListenerContainer messageListenerContainer = new DefaultMessageListenerContainer();
        messageListenerContainer.setConnectionFactory(connectionFactory());
        messageListenerContainer.setMessageListener(new MessageListenerAdapter(new MyQueueListener()));
        messageListenerContainer.setDestinationName(queueName);

        return messageListenerContainer;
    }

My queue listener:

public class MyQueueListener {

    public void handleMessage(String messageContent) throws JMSException {
        //do some job
        //TODO: ack the message
    }
}

Right now I don't know how to ack the message from my listener.

writing a distributed AWS SWF program

AWS offers a simple workflow example on their site, when implementing it in my environment it does not compile, i need assistance implementing a simple distributed SWF application, the example can be found here: http://ift.tt/1ISqQAZ

Live555 RTSP Stream Slow

I am wondering why my Live555 RTSP Stream is so slow. Here is what I did:

I set up an application based on the testOnDemandRTSPServer.cpp (http://ift.tt/1KvJ50J).

The application is supposed to stream a .mkv file which is 90MB big and 2:30 minutes long.

I am hosting the stream on a large Amazon AWS EC2 instance. Upload speed is somewhere in the range of 500 - 600Mbit/s at least.

At home I have a 20Mbit connection. Speedtests have shown a download speed of 1.8MB/s.

A calculation would say that streaming the file should not be a problem: 90 / 1.8 = 50 (Filesize divided by download speed). Therefore, I should be able to download the file within 50 seconds. The video is 2:30 minutes long. I don't see why the stream should have bad quality or lag in any way.

However, when viewing the RTSP stream with VLC Player the image quality is horrible. It's like half a frame per second.

How is that possible? What am I missing?

Windows Firewall is off on both ends.
No other connections are open, therefore the entire network speed is available for the stream.
Security Group of the Amazon EC2 Instance has been modified to allow all incoming and outgoing traffic on all protocols and all ports.

Is there a way to debug such problems?

"Missing secret key" - AWS Rails tutorial Chapter 11

I've searched long and hard and can't resolve this issue. I've even deleted the old heroku site and made another.

I created an AWS account, made a user, gave the user full permissions for my bucket.

I've set the secret keys on Heroku (via terminal and the site) and I've tried setting them in both production.rb and development.rb as so S3_ACCESS_KEY = 'xxxx'

My carrier_wave.rb looks like this

` if Rails.env.production?
  CarrierWave.configure do |config|
    config.fog_credentials = {
        # Configuration for Amazon S3
        :provider              => 'AWS',
        :aws_access_key_id     => ENV['S3_ACCESS_KEY'],
        :aws_secret_access_key => ENV['S3_SECRET_KEY']
    }
    config.fog_directory     =  ENV['S3_BUCKET']
  end
end`

I've tried adding different things like :region and :host as suggested by various SO questions but it doesn't make a difference.

My Heroku logs show

`2015-08-30T15:36:29.574898+00:00 app[web.1]: [3] * Listening on tcp://0.0.0.0:3000
2015-08-30T15:36:29.575137+00:00 app[web.1]: [3] Use Ctrl-C to stop
2015-08-30T15:36:29.842552+00:00 app[web.1]: [3] - Worker 0 (pid: 6) booted, phase: 0
2015-08-30T15:36:29.951221+00:00 app[web.1]: [3] - Worker 1 (pid: 15) booted, phase: 0
2015-08-30T15:36:46.027166+00:00 heroku[router]: at=error code=H20 desc="App boot timeout" method=GET path="/" host=intense-sierra-1318.herokuapp.com request_id=cf0eb0fb-a9a8-4e36-a17b-457af96b15f2 fwd="68.81.200.36" dyno= connect= service= status=503 bytes=
2015-08-30T15:37:10.347183+00:00 heroku[router]: at=error code=H20 desc="App boot timeout" method=GET path="/favicon.ico" host=intense-sierra-    1318.herokuapp.com request_id=6ae64831-17a7-497e-8416-9b2870b62c97 fwd="166.216.165.52" dyno= connect= service= status=503 bytes=
2015-08-30T15:37:12.635345+00:00 heroku[web.1]: Error R10 (Boot timeout) -> Web process failed to bind to $PORT within 60 seconds of launch
2015-08-30T15:37:12.635345+00:00 heroku[web.1]: Stopping process with SIGKILL
2015-08-30T15:37:14.311191+00:00 heroku[web.1]: Process exited with status 137
2015-08-30T15:37:14.325709+00:00 heroku[web.1]: State changed from starting to crashed
2015-08-30T15:37:21.671523+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=intense-sierra-1318.herokuapp.com request_id=623c2c5b-906d-4b6c-b9ca-de4503242b3b fwd="68.81.200.36" dyno= connect= service= status=503 bytes=
2015-08-30T15:37:41.716401+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=intense-sierra-1318.herokuapp.com request_id=e9f0958c-5008-45f5-bd2c-f6bbf93e8f4a fwd="166.216.165.52" dyno= connect= service= status=503 bytes=
2015-08-30T15:37:41.862125+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/favicon.ico" host=intense-sierra-1318.herokuapp.com request_id=861b02c1-2c0b-4fce-9dcc-6619fc632442 fwd="166.216.165.52" dyno= connect= service= status=503 bytes=
2015-08-30T15:37:43.445862+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=intense-sierra-1318.herokuapp.com request_id=59dd08ab-e784-416a-82e2-6d06f3ed3e1d fwd="166.216.165.52" dyno= connect= service= status=503 bytes=
2015-08-30T15:37:43.612208+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/favicon.ico" host=intense-sierra-1318.herokuapp.com request_id=0b0ac1fb-8288-49a3-b4e8-85a7ec151e15 fwd="166.216.165.52" dyno= connect= service= status=503 bytes=
2015-08-30T15:38:00.376281+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/favicon.ico" host=intense-sierra-1318.herokuapp.com request_id=307d4327-bf26-40dd-b471-5eff80579f38 fwd="68.81.200.36" dyno= connect= service= status=503 bytes= `

When I try to run rails server -e production I get

`Exiting
/Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/fog-core-1.32.0/lib/fog/core/service.rb:244:in `validate_options': Missing required arguments: aws_access_key_id, aws_secret_access_key (ArgumentError)
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/fog-core-1.32.0/lib/fog/core/service.rb:268:in `handle_settings'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/fog-core-1.32.0/lib/fog/core/service.rb:98:in `new'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/fog-core-1.32.0/lib/fog/core/services_mixin.rb:16:in `new'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/fog-core-1.32.0/lib/fog/storage.rb:22:in `new'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/carrierwave-0.10.0/lib/carrierwave/uploader/configuration.rb:83:in `eager_load_fog'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/carrierwave-0.10.0/lib/carrierwave/uploader/configuration.rb:96:in `fog_credentials='
    from /Users//trumptalk/TrumpTalk/config/initializers/carrierwave.rb:3:in `block in <top (required)>'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/carrierwave-0.10.0/lib/carrierwave/uploader/configuration.rb:118:in `configure'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/carrierwave-0.10.0/lib/carrierwave.rb:14:in `configure'
    from /Users//trumptalk/TrumpTalk/config/initializers/carrierwave.rb:2:in `<top (required)>'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/activesupport-4.2.2/lib/active_support/dependencies.rb:268:in `load'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/activesupport-4.2.2/lib/active_support/dependencies.rb:268:in `block in load'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/activesupport-4.2.2/lib/active_support/dependencies.rb:240:in `load_dependency'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/activesupport-4.2.2/lib/active_support/dependencies.rb:268:in `load'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/railties-4.2.2/lib/rails/engine.rb:652:in `block in load_config_initializer'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/activesupport-4.2.2/lib/active_support/notifications.rb:166:in `instrument'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/railties-4.2.2/lib/rails/engine.rb:651:in `load_config_initializer'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/railties-4.2.2/lib/rails/engine.rb:616:in `block (2 levels) in <class:Engine>'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/railties-4.2.2/lib/rails/engine.rb:615:in `each'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/railties-4.2.2/lib/rails/engine.rb:615:in `block in <class:Engine>'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/railties-4.2.2/lib/rails/initializable.rb:30:in `instance_exec'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/railties-4.2.2/lib/rails/initializable.rb:30:in `run'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/railties-4.2.2/lib/rails/initializable.rb:55:in `block in run_initializers'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/2.2.0/tsort.rb:226:in `block in tsort_each'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/2.2.0/tsort.rb:348:in `block (2 levels) in each_strongly_connected_component'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/2.2.0/tsort.rb:420:in `block (2 levels) in each_strongly_connected_component_from'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/2.2.0/tsort.rb:429:in `each_strongly_connected_component_from'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/2.2.0/tsort.rb:419:in `block in each_strongly_connected_component_from'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/railties-4.2.2/lib/rails/initializable.rb:44:in `each'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/railties-4.2.2/lib/rails/initializable.rb:44:in `tsort_each_child'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/2.2.0/tsort.rb:413:in `call'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/2.2.0/tsort.rb:413:in `each_strongly_connected_component_from'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/2.2.0/tsort.rb:347:in `block in each_strongly_connected_component'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/2.2.0/tsort.rb:345:in `each'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/2.2.0/tsort.rb:345:in `call'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/2.2.0/tsort.rb:345:in `each_strongly_connected_component'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/2.2.0/tsort.rb:224:in `tsort_each'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/2.2.0/tsort.rb:203:in `tsort_each'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/railties-4.2.2/lib/rails/initializable.rb:54:in `run_initializers'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/railties-4.2.2/lib/rails/application.rb:352:in `initialize!'
    from /Users//trumptalk/TrumpTalk/config/environment.rb:5:in `<top (required)>'
    from /Users//trumptalk/TrumpTalk/config.ru:3:in `require'
    from /Users//trumptalk/TrumpTalk/config.ru:3:in `block in <main>'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/rack-1.6.4/lib/rack/builder.rb:55:in `instance_eval'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/rack-1.6.4/lib/rack/builder.rb:55:in `initialize'
    from /Users//trumptalk/TrumpTalk/config.ru:in `new'
    from /Users//trumptalk/TrumpTalk/config.ru:in `<main>'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/rack-1.6.4/lib/rack/builder.rb:49:in `eval'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/rack-1.6.4/lib/rack/builder.rb:49:in `new_from_string'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/rack-1.6.4/lib/rack/builder.rb:40:in `parse_file'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/rack-1.6.4/lib/rack/server.rb:299:in `build_app_and_options_from_config'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/rack-1.6.4/lib/rack/server.rb:208:in `app'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/railties-4.2.2/lib/rails/commands/server.rb:61:in `app'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/rack-1.6.4/lib/rack/server.rb:336:in `wrapped_app'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/rack-1.6.4/lib/rack/server.rb:272:in `start'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/railties-4.2.2/lib/rails/commands/server.rb:80:in `start'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/railties-4.2.2/lib/rails/commands/commands_tasks.rb:80:in `block in server'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/railties-4.2.2/lib/rails/commands/commands_tasks.rb:75:in `tap'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/railties-4.2.2/lib/rails/commands/commands_tasks.rb:75:in `server'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/railties-4.2.2/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/railties-4.2.2/lib/rails/commands.rb:17:in `<top (required)>'
    from /Users//trumptalk/TrumpTalk/bin/rails:8:in `require'
    from /Users//trumptalk/TrumpTalk/bin/rails:8:in `<top (required)>'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/spring-1.1.3/lib/spring/client/rails.rb:27:in `load'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/spring-1.1.3/lib/spring/client/rails.rb:27:in `call'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/spring-1.1.3/lib/spring/client/command.rb:7:in `call'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/spring-1.1.3/lib/spring/client.rb:26:in `run'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/spring-1.1.3/bin/spring:48:in `<top (required)>'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/spring-1.1.3/lib/spring/binstub.rb:11:in `load'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/spring-1.1.3/lib/spring/binstub.rb:11:in `<top (required)>'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/site_ruby/2.2.0/rubygems/core_ext/kernel_require.rb:54:in `require'
    from /Users//.rbenv/versions/2.2.2/lib/ruby/site_ruby/2.2.0/rubygems/core_ext/kernel_require.rb:54:in `require'
    from /Users//trumptalk/TrumpTalk/bin/spring:16:in `<top (required)>'
    from bin/rails:3:in `load'
    from bin/rails:3:in `<main>'

BTW just in case anyone is wondering I named the site TrumpTalk (and made it Trump themed) because I find him pretty amusing, everyone is always talking about Trump.

Dynamodb - Having mulitple rangekey conditions for the same hash key

Could i know if there is a provision of having multiple range key conditions for the same hash key. For example,if the hash key is 'locality' and rangekey is 'shopNo'. Then could we have the below keyconditionset:

>HashKeyCondition: locality EQ  'NewYork'  
>RangeKeyCondition1: shopNo BETWEEN  1 to 10  
>RangeKeyCondition2: shopNo BETWEEN  20 to 30

When the above was tried, it was considering only one rangeKeyCondition . Is there any way that we can achieve giving two rangekeyconditions for the same hashkey. I am supposing that it could be possible since for a given hashkey, all the rangekeys under this partition are indexed via a 'sortedindex'.

I can see this can be achieved using a 'QueryFilter' with 'OR' ConditionalOperator but the filter is applied after the key-based retrieval and before the results are returned to you. This may not be efficient, as getting all shopNo's for a locallity and then performing a filter could be slower than getting the only items which satisfy the provided multiple rangekey conditions

Could anyone advice on how can this be achieved?

Thanks.

Can't run gunicorn on port 80 while deploying django app on AWS EC2

I am running Ubuntu 14.04 instance in my EC2 AWS.

I can run gunicorn -b 0.0.0.0:8000 myapp.wsgi

but it fails to run at port 80

gunicorn -b 0.0.0.0:80 myapp.wsgi --> Fails

I tried to follow this answer (I couldn't understand it though, whats deployment script ?) Getting Gunicorn to run on port 80

But it didn't work.

I also tried running it with elevated permission sudo in vain.

SQL Server vs. No-SQL Database

I have inherited a legacy content delivery system and I need to re-design & re-build it. The content is delivered by content suppliers (e.g. Sony Music) and is ingested by a legacy .NET app into a SQL Server database.

Each content has some common properties (e.g. Title & Artist Name) as well as some content-type specific properties (e.g. Bit Rate for MP3 files and Frame Rate for video files).

This information is stored in a relational database in multiple tables. These tables might have null values in some of their fields because those fields might not belong to a property of the content. The database is constantly under write operations because the content ingestion system is constantly receiving content files from the suppliers and then adds their metadata to the DB.

Also, there is a public facing web application which lets end users buy the ingested contents (e.g. musics, videos etc). This web application totally relies on an Elasticsearch index. In fact this application does not see the database at all and uses the Elasticsearch index as the source of data. The reason is that SQL Server does not perform as fast and as efficient as Elasticsearch when it comes to text-search.

To keep the database and Elasticsearch in sync there is a Windows service which reads the updates from SQL Sever and writes them to the Elasticsearch index!

As you can see there are a few problems here:

1- The data is saved in a relational database which the data hard to manage. e.g. there is a table of 3 billion records to store metadata of each contents as a key value pairs! To me using a no-sql database or index would make a lot more sense as they allow to store documents with different formats in them.

2- The Elasticsearch index needs to be kept in Sync with the database. If the Windows services does not work for any reason then the index will not get updated. Also when there are too many inserts/updates in the DB it takes a while for the index to get updated.

3- We need to maintain two sources of data which has cost overhead.

Now my question is that : Is there a no-sql database which has these characteristics?

A- Allows me to store documents with different structures in it? B- Provides good text-search functions and performance? e.g. Fuzzy search etc. C- Allows multiple updates to be made to its data concurrently? Based on my experience Elasticsearch has problems with concurrent updates. D- It can be installed and used at Amazon AWS infrastructure because our new products will be hosted on AWS.Auto scaling and clustering is important. e.g. DynamoDB.

E- It would have a kind of GUI so that support staff or developers could modify the data to some extent.

S3. How to protect my storage from many files upload by authorised users

As a part of Mobile-server application, we wanted users to upload images directly to S3 service instead of uploading them to the backend server, to alleviate the backend and make it more focused on its main function.

We did restrictive policy on AMI user (we are using one AMI user for all mobile users, is that ok?) who are allowed to upload to certain bucket, the question is: In case of stolen access and secret keys, how can I prevent the attacker from arbitrary uploading of images to fill my storage as a kind of harm to my service.

What techniques/strategies should I follow to more protect my account

No EXPOSE in aws docker fails deployment

I have a scrapy project run continously by cron hosted inside a docker image.

When I run and deploy this locally everything works fine. If I try to deploy the same to AWS I get the following error inside the logs:

No EXPOSE directive found in Dockerfile, abort deployment (ElasticBeanstalk::ExternalInvocationError)

The console shows that my container was build correctly but I can not use it without an EXPOSED port.

INFO: Successfully pulled python:2.7
WARN: Failed to build Docker image aws_beanstalk/staging-app, retrying...
INFO: Successfully built aws_beanstalk/staging-app
ERROR: No EXPOSE directive found in Dockerfile, abort deployment
ERROR: [Instance: i-6eebaeaf] Command failed on instance. Return code: 1 Output: No EXPOSE directive found in Dockerfile, abort deployment.
Hook /opt/elasticbeanstalk/hooks/appdeploy/enact/00run.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
INFO: Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].

But why is it not possible?

My Dockerfile looks like the following:

FROM python:2.7
MAINTAINER XDF

ENV DIRECTORY /opt/the-flat

# System
##########

RUN apt-get update -y && apt-get upgrade -y && apt-get install -y ntp vim apt-utils
WORKDIR $DIRECTORY

# GIT
##########
# http://ift.tt/12YvFHs

RUN apt-get install -y git
RUN mkdir /root/.ssh/
ADD deploy/git-deply-key /root/.ssh/id_rsa
RUN chmod 0600 /root/.ssh/id_rsa
RUN touch /root/.ssh/known_hosts
RUN ssh-keyscan -t rsa bitbucket.org >> /root/.ssh/known_hosts
RUN ssh -T -o 'ConnectionAttempts=1' git@bitbucket.org
RUN git clone --verbose git@bitbucket.org:XDF/the-flat.git .

# Install
##########

RUN pip install scrapy
RUN pip install MySQL-python

# not working
# apt-get install -y wkhtmltopdf && pip install pdfkit
# else
# http://ift.tt/1xpfdNR

RUN DEBIAN_FRONTEND=noninteractive apt-get install -y openssl build-essential xorg libssl-dev
RUN wget http://ift.tt/1X14Udt
RUN tar xvjf wkhtmltopdf-0.10.0_rc2-static-amd64.tar.bz2
RUN chown root:root wkhtmltopdf-amd64
RUN mv wkhtmltopdf-amd64 /usr/bin/wkhtmltopdf
RUN pip install pdfkit

# Cron
##########
# http://ift.tt/1M7MK4p
# http://ift.tt/ZDwrU1

RUN apt-get install -y cron
RUN crontab "${DIRECTORY}/deploy/crontab"

CMD ["cron", "-f"]

How do I know my whole app is Rails Asset Pipeline Compliant?

I am trying to figure out where my issue is for when I try to use aws cloudfront to render the rails assets. Not sure if there is a tool that will determine if my whole app is rails asset pipeline compliant (whether it meets its standards, etc). Any help would be appreciated, even helping me figure out how do I know for sure that my assets are coming from CloudFront and not from my app.

samedi 29 août 2015

HDP 2.3 - DataNodes start failed on EC2

I try to install HDP2.3 on 10 CentOS 7 EC2 instances. I follow the document on Hortonworks and successfully passed all process except start DataNodes in the last step.

My instance type is t2.medium. I tried t2.micro and m4.xlarge, same problem.

I have 5 data nodes and they are also node managers. I installed servers on other 5 nodes. I can start SNameNode and NameNode successfully.

Here is stderr: /var/lib/ambari-agent/data/errors-137.txt

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py", line 153, in <module>
    DataNode().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py", line 47, in start
    datanode(action="start")
  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_datanode.py", line 58, in datanode
    create_log_dir=True
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py", line 276, in service
    environment=hadoop_env_exports
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 258, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
    tries=tries, try_sleep=try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ;  /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start datanode'' returned 1. starting datanode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-datanode-ip-10-0-0-100.ec2.internal.out

At first I find JVM warning in .out file:

OpenJDK 64-Bit Server VM warning: You have loaded library /usr/local/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.

It’s highly recommended that you fix the library with ‘execstack -c ’, or link it with ‘-z noexecstack’.

I edit hadoop-env.sh and .bashrc with

export HADOOP_HOME=/usr/hdp/2.3.0.0-2557/hadoop
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

Also tried execstack -c So there is no JVM warning now.

Here is current /var/log/hadoop/hdfs/hadoop-hdfs-datanode-ip-10-0-0-100.ec2.internal.out

ulimit -a for user hdfs
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 15027
max locked memory   (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 128000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65536
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Seems no errors but I can not start DataNode in anyway. Really need help.

P.S. My goal is to build a Hadoop Cluster with Spark to do some tasks, and may learn Hive, Pig in the future. Is there any better deployment method for me? Thanks.

NotAuthorizedException while uploading image to Amazon S3

While trying to upload image to amazon s3 using,

AWSS3TransferManager.defaultS3TransferManager().upload(uploadRequest).continueWithBlock{...}

Getting log in verbose mode,

AWSiOSSDKv2 [Debug] AWSURLResponseSerialization.m line:82 | [AWSJSONResponseSerializer responseObjectForResponse:originalRequest:currentRequest:data:error:] | Response header: [{ "Content-Length" = 129; "Content-Type" = "application/x-amz-json-1.1"; Date = "Sun, 30 Aug 2015 04:54:14 GMT"; nnCoection = close; "x-amzn-ErrorMessage" = "Access to Identity 'us-east-1:9b28176f-XXXX-XXXX-82d6-f64569ae14ce' is forbidden."; "x-amzn-ErrorType" = "NotAuthorizedException:"; "x-amzn-RequestId" = "2a1fc807-4ed3-11e5-a8b0-9368dcfb1138"; }]

I have confirmed bucket access policy to have proper access. What would be possible reason for such error?

Is AWS Aurora truly scalable?

It is common knowledge that relational databases don't scale well for very large systems but AWS now has Aurora as a part of RDS. Aurora supposedly allows for scalable relational databases.

My questions are:

What makes relational databases unable to scale properly?

Is Aurora truly scalable?

If it does, how?

Amazon AWS Account Sharing with iOS

I am considering using Amazon AWS for the 1st time, so I am trying to get familiar with its capabilities. I've been reading through the AWS documentation for S3 and Cognito, but I'm not sure if AWS will support what I need, so I'd like to know if anyone who has experience with AWS can shed some light.

I have a client who has the following request for an iOS app that allows users to essentially upload and download files:

An end user can create an account (I assume using AWS Cognito) where he can upload/download files (AWS S3) that other people can't access by default
This end user can provide access to his account to friends, giving friends the ability to also upload and download files within the created account. The original end user has the ability to delete files (he has total control), but not his friends (friends can only upload and download). I'm not sure if AWS Cognito supports this shared account ability.

Is the above scenario possible using AWS, and if so, how straight-forward would it be to implement? If the requirement was only for the original end user to access the account, then I know from the documentation that this should be straight forward. But, with the added requirement of delegating access to friends, I can't tell from the documentation if AWS supports this, and if it does support it, how easily does it support this feature.

How can I speed up my AWS RDS Postgres performance?

I just set up a db.t2.micro instance on Amazon's AWS. I am using sinatra to load a localhost webpage. I am using Active Record to do maybe about 30~ queries and it's taking 92 seconds to load. It's extremely slow. I tried doing custom parameters as listed here: http://ift.tt/1LKbSjN

This didn't help speed anything up. I'm not sure how I can speed up this instance. This is my first time hosting a database. Any help would be appreciated.

It is only slow running through localhost. When I do queries in Postgres it is very fast.

What is the purpose of the DENY section (SID 3) in AWS Elastic Transcoder default policy?

I am looking at the default policy (below) and I do not understand why the section identified as SID 3 is necessary. By default permissions are denied unless they are allowed. I don't see that the previous sections (SID 1 and SID 2) allow the actions that are denied in SID 3. Am I missing something? It doesn't hurt to explicitly set these denials but I think they are unnecessary.

{
    "Version" : "2008-10-17",
    "Statement" : [{
            "Sid" : "1",
            "Effect" : "Allow",
            "Action" : ["s3:Put*", "s3:ListBucket", "s3:*MultipartUpload*", "s3:Get*"],
            "Resource" : "*"
        }, {
            "Sid" : "2",
            "Effect" : "Allow",
            "Action" : "sns:Publish",
            "Resource" : "*"
        }, {
            "Sid" : "3",
            "Effect" : "Deny",
            "Action" : ["s3:*Delete*", "s3:*Policy*", "sns:*Remove*", "sns:*Delete*", "sns:*Permission*"],
            "Resource" : "*"
        }
    ]

Trouble with amazon s3 uploads

I've managed to get file uploads working, but I can't seem to get the file names to be what I want. I think amazon is assigning them randomized file names upon upload because I don't think my code is specifying what to name them.

I am able to get it so that every user has their own folder in my bucket, which is how I want it, but I don't know how to get 1) tell it to name the file what it was originally named, and, 2) how to get back that URL / path after the upload is complete.

Here's my code, and for the life of me, everything I've tried isn't working. I've tried pushing the path to a user object in Mongo, but it never pushes anything, and it won't console.log the object when I try to see what it actually is (file.path or any of the methods associated like file.extension)

Here's the code I'm using:

app.use(multiMiddle); 
app.post('/api/merch/upload/:merchid', function(req, res){
    var file = req.files.file; //last property must match key from upload 
    var stream = fs.createReadStream(file.path); 

// gives every user their own folder on my bucket 
fileName = req.user._id+file.path; 
console.log(file.extension); //this doesn't log anything

return s3.writeFile(fileName, stream).then(function(){
    fs.unlink(fileName, function(err, path){
            if(err) console.error(err);
            console.log("FILE PATH", path); 
        });
    res.redirect('/');
});
});

How do I get these to be named correctly?

GPU memory on AWS instance

I have a g2.2xlarge instance (http://ift.tt/1hHKqUH) on AWS running Ubuntu 10.4 with GTX 750 Ti GPU running cuda. The GPU has 4GB of memory. I am using the instance to run Theano, (http://ift.tt/OaLIE2), a popular machine learning library. My program aborts with following error

Error allocating 1073741824 bytes of device memory (out of memory). 
Driver report 888422400 bytes free and 4294770688 bytes total

The program requests about 1GB of GPU memory while only 800MB is free. How can I know what processes are using GPU memory? I am not running any other process on the instance. I tried looking into nvidia-smi (http://ift.tt/1gNsGVN), but could not get any information.

Why does AWS Lambda uses older node version?

It seems like aws lambda uses older node version 0.10.36

Is there any reason to not to use newer versions of node or maybe even latest iojs compiler?

Thanks

How does Amazon RDS dns resolution work?

Amazon provide you a url of form like x.y.us-west-4.rds.amazonaws.com . Whenever you try to connect to database server first this dns name should get resolved.After dns resolution you get an IP address(static is guess) and you try to connect to this IP address.

There are lots of RDS servers globally. It's not possible to assign all of them an static IP. So how does amazon manage all this with limited no. of IPs?

Boto server error: Endpoint already exists with the same Token, but different attributes

My app is returning the following error:

BotoServerError: BotoServerError: 400 Bad Request {"Error":{"Code":"InvalidParameter","Message":"Invalid parameter: Token Reason: Endpoint [myendpoint] already exists with the same Token, but different attributes.","Type":"Sender"},"RequestId":"myrequestid"}

I modelled my code on http://ift.tt/10n04hv so as to avoid this kind of problem, but I can't work out what is causing this or how to fix it. It doesn't seem to matter whether I delete the endpoint from the SNS console, or set the endpointarn in my database to NULL, or both. Please help!

def createEndpoint(sns_conn, applicationArn, testUser, token):
    cpe_result = sns_conn.create_platform_endpoint(applicationArn, token, str(testUser.userid))
    try:
            endpointArn = cpe_result["CreatePlatformEndpointResponse"]["CreatePlatformEndpointResult"]["EndpointArn"]
    except BotoServerError, e:
            if "already exists with the same Token, but different attributes" in e.error_message:
                    s = e.error_message
                    endpointArn = s[s.index("Endpoint ") + len("Endpoint "):s.index("already exists") - 1]
            else:
                    raise
    testUser.endpointarn = endpointArn
    db.session.commit()
    return endpointArn

def registerWithSNS(testUser):
    # Adaptation of code from http://ift.tt/10n04hv
    endpointArn = testUser.endpointarn
    token = request.form["token"]
    platform = request.form["platform"]
    updateNeeded = False
    createNeeded = endpointArn == None

    # Init sns_conn and applicationArn
    sns_conn = sns.connect_to_region("eu-west-1")
    lpa_response = sns_conn.list_platform_applications()
    platformApps = lpa_response["ListPlatformApplicationsResponse"]["ListPlatformApplicationsResult"]["PlatformApplications"]
    if platform == "Android":
            requiredSuffix = "GCM"
    elif platform == "iOS":
            requiredSuffix = "APNS"        
    else:
            raise Exception("Unknown platform: '{}'".format(platform))
    applicationArn = None
    for pa in platformApps:
            if pa["PlatformApplicationArn"].endswith(requiredSuffix):
                    applicationArn = pa["PlatformApplicationArn"]
                    break
    if applicationArn == None:
            raise Exception("Missing SNS platform application for '{}'".format(platform))

    if createNeeded:
            # No endpoint ARN is stored; need to call createEndpoint
            endpointArn = createEndpoint(sns_conn, applicationArn, testUser, token)
            createNeeded = False

    # Look up the endpoint and make sure the data in it is current, even if
    # it was just created
    try:
            gea_result = sns_conn.get_endpoint_attributes(endpointArn)
            returnedToken = gea_result["GetEndpointAttributesResponse"]["GetEndpointAttributesResult"]["Attributes"]["Token"]
            returnedEnabled = gea_result["GetEndpointAttributesResponse"]["GetEndpointAttributesResult"]["Attributes"]["Enabled"]
            updateNeeded = (returnedToken != token) or (returnedEnabled != "true")
    except BotoServerError, e:
            if e.error_code == "NotFound":
                    # we had a stored ARN, but the endpoint associated with it
                    # disappeared. Recreate it.
                    createNeeded = True
            else:
                    raise

    if createNeeded:
            createEndpoint(sns_conn, applicationArn, testUser, token)

    if updateNeeded:
            # endpoint is out of sync with the current data;
            # update the token and enable it.
            attribs = {}
            attribs["Token"] = token
            attribs["Enabled"] = "true"
            sns_conn.set_endpoint_attributes(endpointArn, attribs)