Hi Francine,

As your student is looking at processing a significant of data I would be looking at utilizing some cloud processing tools on a platform such as Amazon Web Services (AWS). This would be much more efficient than batch downloading terabytes of imagery data

AWS hosts some significant imagery holdings including Landsat, Sentinel and NAIP:

https://registry.opendata.aws/sentinel-2/
https://registry.opendata.aws/landsat-8/
https://registry.opendata.aws/sentinel-1/
https://registry.opendata.aws/naip/

They basic idea is that you set up a virtual service (such as an EC2 instance) to process the data within the cloud. The results can be stored, viewed and downloaded as necessary. The links above point to some examples.

In most cases the data access is free (sentinel-2 is user pays), and you only pay for the time the EC2 instance is running/processing plus and storage costs.

If the student would prefer to batch download the raw data there are many tools that allow you to connect the 'buckets' referenced above and download (for example S3 Browser). Several scripts and command line tools can also be used to pull the data from S3.

Hope the above helps

Greg Lauer
Business Development Manager - Asia/Pacific
East View Geospatial, Inc.
10601 Wayzata Blvd, Minneapolis, MN 55305-1526 USA
Tel : AU +61 405 258 631 | US +1 (909) 979 3822
[log in to unmask] www.geospatial.com<http://www.geospatial.com/>


From: Maps-L: Map Librarians, etc. <[log in to unmask]> On Behalf Of Francine Berish
Sent: Monday, January 21, 2019 3:56 PM
To: [log in to unmask]
Subject: batch global imagery downloading

[please forgive the cross posting]

Good afternoon folks,

Helping a student who is using artificial intelligence (AI) to interpret global imagery for geological purposes. The student has been testing this workflow using LandSat 8 (bands 6-7), ASTER shortwave infrared, and Digital Globe RGB. Since the student is looking to download and process imagery in large batches - the Earth Explorer and Digital Globe portals propose some unique challenges as imagery practically has be downloaded tile by tile. Wondering if there's a way to access any of these sources through a directory or batch download?

The researcher is open sources other than LandSat 8, ASTER & Digital Globe. The project is not geographically specific, and is using global imagery sources. It would acceptable if there were a license or cost associated with this data, as long as the batch processing and possible automation capability is there.

Your insight and expertise are greatly appreciated!

Best&Cheers,
Francine
________________________________________________________________
Francine Berish | Geospatial Data Librarian | Queen's University Library
613-533-6000 x 36952 | [log in to unmask]<mailto:[log in to unmask]>
[https://library.queensu.ca/files/qul-email-signature.png]