Upload files in AWS S3 Glacier in Rails

-

How to upload a file using multipart to glacier in Ruby

What is Amazon Glacier?

Amazon Glacier is a storage optimized for infrequently used data, or cold data. It is an extremely low-cost storage service that provides durable storage with security features for data archiving and backup.

# Gemfile

gem 'aws-sdk', '~> 2.3'

Run bundle install and restart the Rails server after modifying the Gemfile.

View Page :-

<%= form_tag buy_glacier_upload_path, method: :post, multipart: true do %>
    <%= file_field_tag :file, required: true %>
    <%= submit_tag "Upload File", class: "btn submit-btn" %>
<% end %>

Now in the controller :-

class HomesController < ApplicationController

    ACCOUNT_ID         =  "6363463563636536"
    AWS_ACCESS_ID    =  "HDGHDHDHHFGHRGF"
    AWS_SECRET_KEY =  "Dghghghgggggggggghghtyeryey+/"
    AWS_REGION         =  "ap-southeast-2"
    AWS_VAULT           =  "videos"


    def glacier_upload
        uploaded_file       =   params[:file]
        glacier_client       =  Aws::Glacier::Client.new(region: AWS_REGION,  access_key_id: AWS_ACCESS_ID, secret_access_key: AWS_SECRET_KEY)
        file_path              =   uploaded_file.path
        file                      =   File.open(file_path, "r")
        source_file           =   file.read
        file_size               =   file.size
        original_filename =   uploaded_file.original_filename

        response             =      glacier_client.initiate_multipart_upload({account_id: ACCOUNT_ID, part_size: 2097152, vault_name: AWS_VAULT })
        upload_id            =   response.to_h[:upload_id]

        puts "Uploaded id #{upload_id}"

        total_checksum    =   Treehash::calculate_tree_hash source_file
      
        resp              =   glacier_client.upload_archive({
          account_id: ACCOUNT_ID,
          archive_description: "Testing from RY",
          body: source_file,
          checksum: total_checksum,
          vault_name: AWS_VAULT
        })

        # split the file into chunks with the size same as range
        File.open(file_path, "rb") do |f|
          start_byte = 0
          end_byte = 2097151
          f.each_chunk(2097152) do |chunk|
            
            # last chunk's end_byte can be less than the part_size
            if end_byte > file_size
              end_byte = file_size - 1
            end
            # generate checksum for each chunk
            chunk_checksum = Treehash::calculate_tree_hash chunk
            resp = glacier_client.upload_multipart_part({
              account_id: ACCOUNT_ID,
              vault_name: AWS_VAULT,
              upload_id: upload_id,
              checksum: chunk_checksum,
              range: "bytes #{start_byte}-#{end_byte}/*",
              body: chunk
            })
            start_byte = start_byte + 2097152
            end_byte = end_byte + 2097152
          end
        end  

        resp = glacier_client.upload_multipart_part({
          account_id: ACCOUNT_ID,
          body: source_file,
          checksum: total_checksum,
          range: "bytes 0-#{file_size-1}/*",
          upload_id: upload_id,
          vault_name: AWS_VAULT
        })
        resp = glacier_client.complete_multipart_upload({
          account_id: ACCOUNT_ID,
          archive_size: file_size,
          checksum: total_checksum,
          upload_id: upload_id,
          vault_name: AWS_VAULT
        })

         archive_id = resp.to_h[:archive_id]
         location   = resp.to_h[:location]
         checksum   = resp.to_h[:checksum]
         if archive_id.present?
           puts "Successfully uploaded, Archive id is #{archive_id}"
           puts "Successfully uploaded, Location is #{location}"
           puts "Successfully uploaded, Checksum is #{checksum}"
         else
           puts "Please Try again."
         end
         redirect_to root_path
    end
end

 Test the uploaded files in rails console :-

resp = glacier_client.list_jobs({ account_id: ACCOUNT_ID, vault_name: AWS_VAULT })

resp = glacier_client.describe_vault({ account_id: ACCOUNT_ID, vault_name: AWS_VAULT })

resp = glacier_client.list_jobs({ account_id: ACCOUNT_ID, vault_name: AWS_VAULT })

resp.job_list

resp.job_list[0].to_h[:job_id]

resp.job_list.count = 12

resp = glacier_client.describe_job({ account_id: ACCOUNT_ID, vault_name: AWS_VAULT, job_id: "fdfdasfafafasfas9496456445gdfgdfgdsqjhgjgj-etY^TRwetrwetee" })


Also, see :

https://docs.aws.amazon.com/sdkforruby/api/Aws/Glacier/Client.html#upload_multipart_part-instance_method

https://www.inkoop.io/blog/upload-a-file-using-multipart-to-glacier-in-ruby/

Comments

Post a Comment

Popular posts from this blog

psql: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?

-
If we are getting error after creating database in postgres like this :- psql: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"? Then use following link :-  https://stackoverflow.com/questions/42653690/psql-could-not-connect-to-server-no-such-file-or-directory-5432-error sudo chmod 777 -R /var/run/postgresql/ Then restart the postgres:-  sudo service postgresql restart  or   sudo systemctl start postgresql@9.6-main  
Read more

E: Package 'aptitude' has no installation candidate

-
Image
E: Package 'aptitude' has no installation candidate Sometimes, we are getting broken package error. Please can easily solve this problem. Example:-        sudo apt-get install aptitude If you got error like-              E: Package 'aptitude' has no installation candidate Solution:-           sudo apt-get update       sudo apt-get install build-essential       sudo dpkg --configure -a       sudo apt-get -f install None of those worked in my case. I then went to Ubuntu Software Center>Edit> Software Sources and made these changes. After that the following commands did the job       sudo apt-get update         sudo apt-get install build-essential         sudo apt-get install aptitude Now, problem is solved. Please follow the link below:- http://musingsonsoftware.blogspot.com/2014/02/ubuntu-apt-get-package-has-no.html  
Read more

CARRIERWAVE - UPLOAD ANY MEDIA (AUDIO , VIDEO AND IMAGE ) FILE IN RAILS

-
Image
UPLOADING ANY MEDIA FILE WITH CARRIER-WAVE RAILS Multiple file uploads Add it to your Gemfile:       gem 'mime-types'       gem 'mini_magick'       gem 'rmagick'       gem 'carrierwave', '~> 1.0'       gem 'carrierwave-video'       gem 'carrierwave-video-thumbnailer'       gem 'carrierwave-ffmpeg' Run       bundle Start off by generating an uploader: app/uploaders/vimeo_uploader.rb       rails generate uploader Vimeo Check out this file for some hints on how you can customize your uploader. It should look something like this: class VimeoUploader < CarrierWave::Uploader::Base       storage :file end If we want to upload video or image then we will have to use :- class VimeoUploader < CarrierWave::Uploader::Base       include CarrierWave::MiniMagick       include CarrierWave::Video       include CarrierWave::Video::Thumbnailer       include CarrierWave::FFmpeg       storage :file          de
Read more

SEARCHING USING AJAX IN RAILS

-
Search any entry from Database using ajax Routes        get 'search' Controller        def search           @companies = Company.search(params[:search].split("=").last)           respond_to :js       end Model        def self.search(search)           if search                 Company.where("name LIKE '%#{search}%'")           else                 Company.all           end       end index.html.erb :-       <%= text_field_tag :search, "", class: "search_companies" %>       <tbody class="company_tbody_content">            <%= render 'index_content' %>       </tbody>       <script type="text/javascript">            $(".search_companies").keyup(function() {                $.ajax({                   url: '/super_admin/companies/search',                   data: {search: $(this).serialize()}                });            });       </script>
Read more

SELECT COUNTRY WITH STATE AND IT'S CITIES RAILS

-
Country State Select is a library that provides an easy API to generate Country , State / Province and City dropdowns for use in forms. We will refer :- https://github.com/arvindvyas/Country-State-Select Gemfile gem 'chosen-rails' gem 'country_state_select', git: 'git://github.com/arvindvyas/Country-State-Select', branch: :master gem "simple_form"#, "~> 3.2" gem "reform"#, "~> 2.0" gem "reform-rails" orders_controller.rb class OrdersController < ApplicationController     def shipping_address         @location = Address.new     end end shipping_address.html.erb   <%= simple_form_for @location,  :html => {class: 'subscribe-from'} ,url: orders_shipping_address_path do |f| %>   <div class="row">     <div class="col-md-6">       <div class="form-group">         <%= f.input :address_line1, label: "Country", collecti
Read more

FATAL: Peer authentication failed for user "postgres"

-
When we are getting error like When we get error at the time of import the postgres database :- : FATAL:  Peer authentication failed for user "postgres" then:- The problem is still your pg_hba.conf file ( /etc/postgresql/9.1/main/pg_hba.conf ). This line: local   all             postgres                                peer Should be local   all             postgres                                md5 After altering this file, don't forget to restart your PostgreSQL server. If you're on Linux, that would be  sudo service postgresql restart SEE ALSO, http://stackoverflow.com/questions/18664074/getting-error-peer-authentication-failed-for-user-postgres-when-trying-to-ge
Read more

RAILS: UPLOADING PHOTOS VIA AMAZON S3 AND PAPERCLIP (UPLOADING FILES TO S3 IN RUBY WITH PAPERCLIP)

-
Set up Ruby on Rails with Paperclip and S3 using AWS SDK Uploading Files to S3 in Ruby with Paperclip Paperclip requires the following gems added to your Gemfile. If your paperclip version is 5.1.0 then we are using 'aws-sdk' version 2.3. # Gemfile gem 'paperclip' gem 'aws-sdk', '~> 2.3' or our paperclip version is 4.1.0 then we need to use 'aws-sdk' version < 2  (note: add version less than 2.0 otherwise you will get paperclip error) gem 'paperclip' gem 'aws-sdk', '< 2.0' Run bundle install and restart the Rails server after modifying the Gemfile. Then run the command :- rails generate paperclip user image Define the file attribute in the Model class User < ActiveRecord::Base     has_attached_file :image, styles: { medium: "300x300>", thumb: "100x100>" }, default_url: "/images/:style/missing.png"       validates_attachment_content_type :image, cont
Read more

CHECK PASSENGER RUBY PATH

-
When we want to use the passenger with Nginx or Apache. Then we need to find the passenger ruby path. So, we can easily find the ruby path using :- passenger - config -- ruby - command When we run then we will get like :-   Command: /home/rohit/.rvm/gems/ruby -2.1.10/wrappers/ruby   Version: ruby 2.1.10p492 (2016-04-01 revision 54464) [x86_64-linux]   To use in Apache : PassengerRuby /home/rohit/.rvm/gems/ruby -2.1.10/wrappers/ruby   To use in Nginx : passenger_ruby /home/rohit/.rvm/gems/ruby -2.1.10/wrappers/ruby   To use with Standalone: /home/rohit/.rvm/gems/ruby -2.1.10/wrappers/ruby /usr/bin/passenger start Add that line in passenger.conf file
Read more

CURL COMMAND ON RUBY ON RAILS

-
HOW TO USE CURL ON RUBY ON RAILS? LIKE THIS ONE curl - d 'params1[name]=name&params2[email]' 'http://mydomain.com/file.json' Just in case you don't know, it requires 'net/http' require 'net/http'   uri = URI . parse ( "http://example.org" )   # Shortcut   #response = Net::HTTP.post_form(uri, {"user[name]" => "testusername", "user[email]" => "testemail@yahoo.com"})     # Full control   http = Net :: HTTP . new ( uri . host , uri . port )   request = Net :: HTTP :: Post . new ( uri . request_uri )   request . set_form_data ({ "user[name]" => "testusername" , "user[email]" => "testemail@yahoo.com" })   response = http . request ( request )   render : json => response .body Hope it'll helps others.. :) We can also follow the link below :- https://stackoverflow.com/questions/15804425/curl-on-rub
Read more