Tuesday, October 15, 2013

Cloud-based storage in MCS 2.0.6: setting up MScan and MView to access the cloud


In MScan and MView, each user (chosen or created from the User window shown when the programs start) has his or her own cloud settings.
Click on the Settings/Cloud Storage Settings menu item on uppermost window of MScan or MView.  

For Amazon S3:
Choose Amazon S3 for the service and  enter the Access key ID and the Secret key ID.


For Microsoft Azure
Choose Microsoft Azure for the service and enter the Storage Account Name and the Primary Access Key.

Cloud-based storage in MCS 2.0.6: introduction to S3 and Azure


Amazon S3 (Simple Storage Service) and Microsoft Azure services are very similar in the way they operate. Their services can be tried for free, either for a limited time or for a certain amount of data transfer. In addition, data files located on the cloud are organized in a similar way (see table below). The the top level is called an account, which would be similar to a disk drive. The account holds either buckets or containers, which are very similar to disk folders or directories. Buckets or containers contain objects or blobs. For all practical intent, objects or blobs are MCS data files. A major difference between a file system on a disk and the S3 or Azure cloud storages is the fact that buckets or containers cannot contain other buckets or containers, only objects or blobs.

Your Amazon S3 Microsoft Azure
computer

Drive Account Storage Account
Folder Bucket     Container
File         Object Blob 

An Azure storage can store up to 200 TB of data. Blobs can be as large as 200 GB.

MScan allows you
  • to create new buckets and containers
  • and directly upload data files to one of these "folders" at the end of an imaging session.

MView is capable of
  • dowloading data files on the cloud to a local folder on your computer
  • create new buckets or containers.
  • upload files to the cloud
  • delete files in the cloud


Cloud-based storage in MCS: setting up your S3 or Azure account


For Amazon S3:

1. You need first to set up an account. Go to:
Amazon S3: http://aws.amazon.com/s3/

2. Once you are activated, you will receive a confirmation e-mail. Follow the instructions in the e-mail to set up Access Identifiers. Use the Web-based AWS Management Console to try creating a bucket, manually upload a file etc… Click on the Security Credentials menu item (under the menu item bearing your user name).

The two most important information to retrieve from your account are:
  • Your Access Key ID: it is a 20 character-long string
  • Your Secret Key: it is a 40 character-long string

Copy these two values somewhere (on paper, in a text file or in an e-mail) to register later in MScan and MView.


For Microsoft Azure:

1. You need first to set up an account. Go to:
Microsoft Azure: http://www.windowsazure.com/en-us/
This requires an account name and a password.

Log into the Windows Azure Management Portal.
Click on the NEW button at the bottom of the navigation pane 
Click on DATA SERVICES menu item on the left of the window, then click on STORAGE, QUICK CREATE.
Enter a name for the storage account in the URL input box (e.g., jimtwophotonfiles), select a location for the file repository (e.g., West US) and enable/disable geo-replication.

2. Once the storage account is created

Click on the Manage Access Key button at the bottom of the screen 
The two most important information to retrieve from your account are:
  • Your Storage account name
  • Your Primary Accesst Key

Copy these two values somewhere (on paper, in a text file or in an e-mail) to register later in MScan and MView.

Backing up and archiving your multiphoton data: the MCS 2.0.6 cloud-based solution


Optical imaging instruments such as multiphoton microscopes generate large amount of data. Because MOMs are often used to collect two-photon images many times per second over periods ranging from minutes to hours, typical files can routinely exceed hundreds of megabytes. The trend towards producing tens or hundred of gigabytes of data per day is accelerating with the use of video-rate resonant scanning imaging in behavioral experiments. As a resut, the problem of backing up and archiving terabytes of two-photon datasets is becoming particularly acute. One could use a local file server but this requires funds to buy the hardware, expertise to set up the server and some kind of periodic administration. And of course, when the file server is full, someone will have to take care of the problem.

The computer industry has already a solution, called "cloud storage" to safely store large amount of data without having to invest in a costly infrastructure. In exchange for a small fee, Web behemoths such as Amazon, Google and Microsoft offer disk space on their servers to end-users, provide a Web-based interface to administrate accounts and programming tools to upload or download files. The advantages of cloud storage include:

  • virtually unlimited or extremely large storage capacity.
  • reliability: the data is usually stored in triplicate and, optionally, at sites that are geographically distinct to prevent losing data in case of a catastrophic hardware failure.
  • universal availability through an Internet connection.
  • flexible pricing that scales up with the amount of data you store.



MCS 2.0.6 now leverages cloud storage capabilities by allowing:
  • in MScan, to systematically upload your newly created data files to a user-specific location on the cloud upon finishing an imaging session. You won't have to back up (or forget to archive...) your data files manually anymore.
  • in MView, to download data files from the cloud to your local disk storage to perform analysis. Your data is available anywhere and at all times as long as you have an Internet connection to access your cloud account.


MCS 2.0.6 at present supports two services, Amazon S3 (Simple Storage Service) and Microsoft Azure. If you opt for their fee-based plan, Amazon has no restriction on the amount of data you can store while Microsoft now caps the amount of data per storage to 200 TB. However, you can create as many storage accounts as you want. Maximal file size is 200 GB and 5 TB for Azure and S3, respectively.

There are many other cloud storage companies. Google has a service similar to S3 and Azure called Drive, but files cannot be larger than 10 GB, which could be a problem with data collected in resonant scanning mode. For those who are accustomed to Dropbox, it appears that this company uses Amazon S3 as its underlying infrastructure.