MATLAB Answers

Script to access Amazon S3 files

78 views (last 30 days)
Show older comments
André Bibaud
André Bibaud on 25 Sep 2017
  • 0
Commented: Dmitry Kaplanon 8 Feb 2022
I setup a cluster on AWS EC2 to run my scripts faster. Right now my script load input variables locally (on my C: drive) and write simulation results also locally. I setup an S3 bucket with a sub-directory for my input variables and a sub-directory for the results. How do I change the load and write function to point to the S3 bucket instead of my c: drive?
Thanks

Sign in to answer this question.

Answers (3)

Jhon Wine
Jhon Wine on 24 Jan 2018
我发现数据存储是非常容易使用。运行code below in each worker.
%Set S3 Credentials
setenv('AWS_ACCESS_KEY_ID','ID');
setenv('AWS_SECRET_ACCESS_KEY','Key');
setenv('AWS_REGION','us-west-2');
%Load Data
fp='s3://mybucket/data/file.data';
ds=fileDatastore(spectralFilePath,'ReadFcn',@AWSRead);
data=ds.read;
Also define this function
functiondata= AWSRead(fileName)
fid = fopen(fileName);
data= fread(fid,inf,'short');
fclose(fid);
end
Make sure the EC2 cloud is on the same site as S3 to avoid data transfer bottlenecks
David Fink
David Fink on 28 Sep 2017
Since S3 uses a RESTful architecture, use 'webread' and 'webwrite' to access the S3 bucket.
webread: //www.tianjin-qmedu.com/help/matlab/ref/webread.html
webwrite: //www.tianjin-qmedu.com/help/matlab/ref/webwrite.html
1 Comment
Jhon Wine
Jhon Wine 2018年11月17日
Is there an example, how to use webread and webwrite with S3? There is no good tutorial..

Sign in to comment.

Harish Sundaresh
Harish Sundaresh on 26 Jun 2021
can MATLAB read .json files from aws s3? I get these errors
%% webread
webread(fname)
Error using webread (line 119)
The 's3' protocol specified in URL, 'url', is not supported. Specify the URL with the protocol 'http://' or 'https://'.
%% jsondecode
Error using jsondecode
JSON syntax error: expected value but found end of text.
1 Comment
Dmitry Kaplan
Dmitry Kaplan on 8 Feb 2022
I could not get webread to work. I am sure there is a way, but I could not figure it out.
Instead, I use something like this
fds = fileDatastore('s3://blah/blah/filename.json','FileExtensions',{'.json'},'readfcn',@fileread);
file_data = read(fds);
This defines the read function for the datastore to be fileread(). Invoking the read() actually runs fileread() and passes to it the filename -- because the fds.ReadMode is 'file'.

Sign in to comment.

Categories

Tags

s manbetx 845

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!