Upload files in AWS S3 Glacier in Rails

-

How to upload a file using multipart to glacier in Ruby

What is Amazon Glacier?

Amazon Glacier is a storage optimized for infrequently used data, or cold data. It is an extremely low-cost storage service that provides durable storage with security features for data archiving and backup.

# Gemfile

gem 'aws-sdk', '~> 2.3'

Run bundle install and restart the Rails server after modifying the Gemfile.

View Page :-

<%= form_tag buy_glacier_upload_path, method: :post, multipart: true do %>
    <%= file_field_tag :file, required: true %>
    <%= submit_tag "Upload File", class: "btn submit-btn" %>
<% end %>

Now in the controller :-

class HomesController < ApplicationController

    ACCOUNT_ID         =  "6363463563636536"
    AWS_ACCESS_ID    =  "HDGHDHDHHFGHRGF"
    AWS_SECRET_KEY =  "Dghghghgggggggggghghtyeryey+/"
    AWS_REGION         =  "ap-southeast-2"
    AWS_VAULT           =  "videos"


    def glacier_upload
        uploaded_file       =   params[:file]
        glacier_client       =  Aws::Glacier::Client.new(region: AWS_REGION,  access_key_id: AWS_ACCESS_ID, secret_access_key: AWS_SECRET_KEY)
        file_path              =   uploaded_file.path
        file                      =   File.open(file_path, "r")
        source_file           =   file.read
        file_size               =   file.size
        original_filename =   uploaded_file.original_filename

        response             =      glacier_client.initiate_multipart_upload({account_id: ACCOUNT_ID, part_size: 2097152, vault_name: AWS_VAULT })
        upload_id            =   response.to_h[:upload_id]

        puts "Uploaded id #{upload_id}"

        total_checksum    =   Treehash::calculate_tree_hash source_file
      
        resp              =   glacier_client.upload_archive({
          account_id: ACCOUNT_ID,
          archive_description: "Testing from RY",
          body: source_file,
          checksum: total_checksum,
          vault_name: AWS_VAULT
        })

        # split the file into chunks with the size same as range
        File.open(file_path, "rb") do |f|
          start_byte = 0
          end_byte = 2097151
          f.each_chunk(2097152) do |chunk|
            
            # last chunk's end_byte can be less than the part_size
            if end_byte > file_size
              end_byte = file_size - 1
            end
            # generate checksum for each chunk
            chunk_checksum = Treehash::calculate_tree_hash chunk
            resp = glacier_client.upload_multipart_part({
              account_id: ACCOUNT_ID,
              vault_name: AWS_VAULT,
              upload_id: upload_id,
              checksum: chunk_checksum,
              range: "bytes #{start_byte}-#{end_byte}/*",
              body: chunk
            })
            start_byte = start_byte + 2097152
            end_byte = end_byte + 2097152
          end
        end  

        resp = glacier_client.upload_multipart_part({
          account_id: ACCOUNT_ID,
          body: source_file,
          checksum: total_checksum,
          range: "bytes 0-#{file_size-1}/*",
          upload_id: upload_id,
          vault_name: AWS_VAULT
        })
        resp = glacier_client.complete_multipart_upload({
          account_id: ACCOUNT_ID,
          archive_size: file_size,
          checksum: total_checksum,
          upload_id: upload_id,
          vault_name: AWS_VAULT
        })

         archive_id = resp.to_h[:archive_id]
         location   = resp.to_h[:location]
         checksum   = resp.to_h[:checksum]
         if archive_id.present?
           puts "Successfully uploaded, Archive id is #{archive_id}"
           puts "Successfully uploaded, Location is #{location}"
           puts "Successfully uploaded, Checksum is #{checksum}"
         else
           puts "Please Try again."
         end
         redirect_to root_path
    end
end

 Test the uploaded files in rails console :-

resp = glacier_client.list_jobs({ account_id: ACCOUNT_ID, vault_name: AWS_VAULT })

resp = glacier_client.describe_vault({ account_id: ACCOUNT_ID, vault_name: AWS_VAULT })

resp = glacier_client.list_jobs({ account_id: ACCOUNT_ID, vault_name: AWS_VAULT })

resp.job_list

resp.job_list[0].to_h[:job_id]

resp.job_list.count = 12

resp = glacier_client.describe_job({ account_id: ACCOUNT_ID, vault_name: AWS_VAULT, job_id: "fdfdasfafafasfas9496456445gdfgdfgdsqjhgjgj-etY^TRwetrwetee" })


Also, see :

https://docs.aws.amazon.com/sdkforruby/api/Aws/Glacier/Client.html#upload_multipart_part-instance_method

https://www.inkoop.io/blog/upload-a-file-using-multipart-to-glacier-in-ruby/

Comments

Post a Comment

Popular posts from this blog

psql: could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?

-
If we are getting error after creating database in postgres like this :-psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"? Then use following link :- https://stackoverflow.com/questions/42653690/psql-could-not-connect-to-server-no-such-file-or-directory-5432-errorsudo chmod 777 -R /var/run/postgresql/Then restart the postgres:-  sudo service postgresql restart orsudo systemctl start postgresql@9.6-main
Read more

CARRIERWAVE - UPLOAD ANY MEDIA (AUDIO , VIDEO AND IMAGE ) FILE IN RAILS

-
Image
UPLOADING ANY MEDIA FILE WITH CARRIER-WAVE RAILS Multiple file uploads Add it to your Gemfile:      gem 'mime-types'
      gem 'mini_magick'
      gem 'rmagick'
      gem 'carrierwave', '~> 1.0'
      gem 'carrierwave-video'
      gem 'carrierwave-video-thumbnailer'
      gem 'carrierwave-ffmpeg'
Run
      bundle
Start off by generating an uploader:app/uploaders/vimeo_uploader.rb      rails generate uploader Vimeo Check out this file for some hints on how you can customize your uploader. It should look something like this:class VimeoUploader < CarrierWave::Uploader::Base
storage :file
end If we want to upload video or image then we will have to use :-class VimeoUploader < CarrierWave::Uploader::Base
include CarrierWave::MiniMagick
      include CarrierWave::Video
      include CarrierWave::Video::Thumbnailer
      include CarrierWave::FFmpeg

      storage :file
      def store_dir
         "uploads/#{model.class.to_s.undersc…
Read more

SELECT COUNTRY WITH STATE AND IT'S CITIES RAILS

-
Country State Select is a library that provides an easy API to generate Country , State / Province and City dropdowns for use in forms.We will refer :-

https://github.com/arvindvyas/Country-State-Select Gemfilegem 'chosen-rails'
gem 'country_state_select', git: 'git://github.com/arvindvyas/Country-State-Select', branch: :master
gem "simple_form"#, "~> 3.2"
gem "reform"#, "~> 2.0"
gem "reform-rails" orders_controller.rbclass OrdersController < ApplicationController
    def shipping_address
        @location = Address.new
    end
end shipping_address.html.erb<%= simple_form_for @location,  :html => {class: 'subscribe-from'} ,url: orders_shipping_address_path do |f| %>
  <div class="row">
    <div class="col-md-6">
      <div class="form-group">
        <%= f.input :address_line1, label: "Country", collection: CountryStateSelect.countries_co…
Read more

RAILS: UPLOADING PHOTOS VIA AMAZON S3 AND PAPERCLIP (UPLOADING FILES TO S3 IN RUBY WITH PAPERCLIP)

-
Set up Ruby on Rails with Paperclip and S3 using AWS SDK
Uploading Files to S3 in Ruby with Paperclip Paperclip requires the following gems added to your Gemfile. If your paperclip version is 5.1.0 then we are using 'aws-sdk' version 2.3.# Gemfile
gem 'paperclip'
gem 'aws-sdk', '~> 2.3' or our paperclip version is 4.1.0 then we need to use 'aws-sdk' version < 2  (note: add version less than 2.0 otherwise you will get paperclip error)gem 'paperclip'
gem 'aws-sdk', '< 2.0'

Run bundle install and restart the Rails server after modifying the Gemfile. Then run the command :-rails generate paperclip user image Define the file attribute in the Modelclass User < ActiveRecord::Base
    has_attached_file :image, styles: { medium: "300x300>", thumb: "100x100>" }, default_url: "/images/:style/missing.png"
      validates_attachment_content_type :image, content_type: /\Aimage\/.*\z/
end Migration…
Read more

SEARCHING USING AJAX IN RAILS

-
Search any entry from Database using ajaxRoutes
get 'search'

Controller

def search
          @companies = Company.search(params[:search].split("=").last)
          respond_to :js
      end

Model

def self.search(search)
          if search
                Company.where("name LIKE '%#{search}%'")
          else
                Company.all
          end
      end

index.html.erb :-

      <%= text_field_tag :search, "", class: "search_companies" %>

      <tbody class="company_tbody_content">
           <%= render 'index_content' %>
      </tbody>


      <script type="text/javascript">
           $(".search_companies").keyup(function() {
               $.ajax({
                  url: '/super_admin/companies/search',
                  data: {search: $(this).serialize()}
               });
           });
      </script>

search.js.erb

$(".company_tbody_content").html("<…
Read more

UPLOAD ANY MEDIA FILE IN RAILS USING PAPERCLIP

-
Image
Steps to Upload any type of media in ruby on rails by paperclip Upload video in RailsGemfile

gem 'paperclip'
gem 'paperclip-ffmpeg'
gem 'paperclip-av-transcoder' Apply validation in modelhas_attached_file :file, :styles => {:medium => { :geometry => "300x300", :format => 'flv'},:thumb => {:geometry => "100x100#", :format => 'jpg', :time => 15}
    }, :processors => [:ffmpeg]
When you will get some validation errors then we can use this below code :-validates_attachment_content_type :file, content_type: %w(video/mp4 video/3gp video/webm image/jpeg image/jpg image/png) Install ffmpeg on your systemsudo apt-add-repository ppa:jon-severinsson/ffmpeg
sudo apt-get update
sudo apt-get install ffmpeg

When we will get some ppa error then run:-

http://stackoverflow.com/questions/29125229/how-to-reinstall-ffmpeg-clean-on-ubuntu-14-04

Play video in rails using video_tag

video_tag("trailer.ogg", controls: true,…
Read more

CURL COMMAND ON RUBY ON RAILS

-
HOW TO USE CURL ON RUBY ON RAILS? LIKE THIS ONE curl -d 'params1[name]=name&params2[email]''http://mydomain.com/file.json' Just in case you don't know, it requires 'net/http' require 'net/http' uri = URI.parse("http://example.org")
Read more

UNINITIALIZED CONSTANT PAPERCLIP::STORAGE::S3::AWS

-
Set up Ruby on Rails with Paperclip and S3 using AWS SDK Paperclip requires the following gems added to your Gemfile. If your paperclip version is 5.1.0 then we are using 'aws-sdk' version 2.3.gem 'paperclip'
gem 'aws-sdk', '~> 2.3' or our paperclip version is 4.1.0 then we need to use 'aws-sdk' version < 2  (note: add version less than 2.0 otherwise you will get paperclip error)gem 'paperclip'
gem 'aws-sdk', '< 2.0'

Run bundle install and restart the Rails server after modifying the Gemfile.

WE CAN FOLLOW THE LINK BELOW :-

https://github.com/thoughtbot/paperclip/issues/2388
https://coderwall.com/p/vv1iwg/set-up-ruby-on-rails-with-paperclip-5-and-s3-using-aws-sdk-v2
Read more

CHECK PASSENGER RUBY PATH

-
When we want to use the passenger with Nginx or Apache. Then we need to find the passenger ruby path. So, we can easily find the ruby path using :-passenger-config --ruby-command When we run then we will get like :-
Command: /home/rohit/.rvm/gems/ruby-2.1.10/wrappers/ruby
  Version: ruby 2.1.10p492 (2016-04-01 revision 54464) [x86_64-linux]
  To use in Apache: PassengerRuby /home/rohit/.rvm/gems/ruby-2.1.10/wrappers/ruby
  To use in Nginx : passenger_ruby /home/rohit/.rvm/gems/ruby-2.1.10/wrappers/ruby
  To use with Standalone: /home/rohit/.rvm/gems/ruby-2.1.10/wrappers/ruby /usr/bin/passenger start
Add that line in passenger.conf file
Read more