lundi 27 avril 2015

Cannot export HBase snapshots to S3

I am trying to export a HBase Snapshot into S3 storage as backup using the following code:

hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot "SNAP_X" -copy-to http://s3n<ACCESS_KEY_ID>:<SECRET_ACCESS_KEY>@<BUCKET_NAME>/2HBASE-SNAP_X -mappers 3

however I am seeing the following error:

015-04-27 05:39:49,547 FATAL [IPC Server handler 0 on 40333] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1429544880663_0004_m_000000_0 - exited : java.io.IOException: Could not get the output FileSystem with root=http://s3n<ACCESS_KEY_ID>:<SECRET_ACCESS_KEY>@<BUCKET_NAME>/2HBASE-SNAP_X
at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.setup(ExportSnapshot.java:149)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.io.IOException: No FileSystem for scheme: s3n
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.setup(ExportSnapshot.java:147)
... 8 more

2015-04-27 05:39:49,547 INFO [IPC Server handler 0 on 40333] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1429544880663_0004_m_000000_0: Error: java.io.IOException: Could not get the output FileSystem with root=http://s3n<ACCESS_KEY_ID>:<SECRET_ACCESS_KEY>@<BUCKET_NAME>/2HBASE-SNAP_X
at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.setup(ExportSnapshot.java:149)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.io.IOException: No FileSystem for scheme: s3n
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.setup(ExportSnapshot.java:147)
... 8 more

2015-04-27 05:39:49,548 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1429544880663_0004_m_000000_0: Error: java.io.IOException: Could not get the output FileSystem with root=http://s3n<ACCESS_KEY_ID>:<SECRET_ACCESS_KEY>@<BUCKET_NAME>/2HBASE-SNAP_X
at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.setup(ExportSnapshot.java:149)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.io.IOException: No FileSystem for scheme: s3n
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.setup(ExportSnapshot.java:147)
... 8 more

Is there something I'm missing here in regards to S3 credentials here? Perhaps a username needs to be specified somewhere in the code?




Aucun commentaire:

Enregistrer un commentaire