samedi 28 février 2015

How to download data from Amazon's requester pay buckets?

I am not a computer genius and have no computer science background. I have been struggling for about a week to download arXiv articles as mentioned here: http://ift.tt/1E2ztWZ.


I have tried lots of things: s3Browser, s3cmd. I am able to login to my buckets but I am unable to download data from arXiv bucket.


I tried:


1.


$ s3cmd get s3://arxiv/pdf/arXiv_pdf_1001_001.tar




s3://arxiv/pdf/arXiv_pdf_1001_001.tar -> ./arXiv_pdf_1001_001.tar [1 of 1]
s3://arxiv/pdf/arXiv_pdf_1001_001.tar -> ./arXiv_pdf_1001_001.tar [1 of 1]
ERROR: S3 error: Unknown error


2.


$ s3cmd get --add-header="x-amz-request-payer:requester" s3://arxiv/pdf/arXiv_pdf_manifest.xml



It gave me same error again:



s3://arxiv/pdf/arXiv_pdf_manifest.xml -> ./arXiv_pdf_manifest.xml [1 of 1]
s3://arxiv/pdf/arXiv_pdf_manifest.xml -> ./arXiv_pdf_manifest.xml [1 of 1]
ERROR: S3 error: Unknown error


3.

I have tried copying files from that folder too.



$ aws s3 cp s3://arxiv/pdf/arXiv_pdf_1001_001.tar .




A client error (403) occurred when calling the HeadObject operation: Forbidden
Completed 1 part(s) with ... file(s) remaining


This probably means that I made a mistake. Problem is I dont know how and what to add that will convey my permission to pay for download.


I am unable to figure out what should I do for downloading data from s3. I have been reading a lot on AWS sites but nowhere I can get pinpoint solution to my problem. Any help on this will be really appreciated.


Thanks!





1 commentaire: