Target Based Deduplication

Target based deduplication was one of the initial ways of performing deduplication. This was more on the VTLs when VTLs were launched as a technology, and over time, it seems that this is not much in use now.

All it did was to carry the data to the target backup device – mainly VTL – and store it there. It would then run a deduplication process to match blocks and delete the duplicate data. At a later schedule, system will run a clear garbage type process to finally remove the deleted content and free up the space on the VTL.

When it was launched, it was the only process and was a great process. With newer and better technologies available, target based deduplication has lost its charm since most applications now prefer source based deduplication and reduce data before it travels on the network.

So target based deduplication has its advantages of using minimum source processing and memory cycles in performing deduplication at the target, the hind side being slightly oversizing the target to ensure enough space to accommodate full data before starting the deletion process.

Store less, backup lesser – The Art of Deduplication

Deduplication is not a new concept though the term is relatively new. It originated more from the backup prospective though it has now got good footholds in many IT components.

I know and have been using for over a decade backup technologies that would backup only unique emails and files, and subsequently backup only delta changes. Being the only one doing that, they used their own language. They did not coin the word deduplication. Now everyone does that and calls it File level deduplication. This approach served good for desktop and laptop backups. Then this technology progressed, the current being block level deduplication. This recognises unique blocks of data and ensures that one block gets backed up only once, thereby reducing the amount of data travelling and getting backed up.

Like any other process, deduplication also needs resources to do what it is meant for. Various applications offer various forms like source based, target based and inline deduplication. Each one of them has their own working mechanisms and pros and cons. In the following series of blogs we will discuss the various methods of deduplication along with their pros and cons.

Processing Big Data

While many think it is difficult to manage data beyond a volume, technologists don’t agree with this. Newer technologies keep coming in to handle and process large volume of data. Consider Google search. While you are typing what you want to search, it starts autocompleting for you and starts showing you results as well.

All this is done by using clusters of servers at the backend. Data that goes in is processed by these servers so you get a good pool of processors and memory to take it in. Further to this, the storage network used behind to read/write this data offers a huge choice.

If it is a file based data, you can go for a sale-out NAS. If you have to handle block level data, scale-out SAN options are available. To help the really heavy databases, pure Flash based storage is now available.

Flash based storages help achieve upto a few million IOPS especially when they perform on inline deduplication done in the memory. Scale out storage there ensures that adding more capacity automatically gives you more memory to handle the new IOs and deduplication help control the requirement of storage since it could otherwise go for a big on budget.

How to Store and Manage BIG Data?

While I mentioned in my previous blog that any size of data is no problem, I often get questioned upon how to store and manage the huge volumes. This is a typical concern of an enterprise faced with increasing data size.

Storage vendors have seen and known this problem as it grew, and have scaled-up or rather scaled-out to help handle this massive growth. Both NAS and SAN vendors have gone beyond the traditional methods of upgrading the storage infrastructure by adding additional shelves and disks. The challenge that the traditional method has is that you end up upgrading capacity with shelves and disks with limited enhancements in processing power. This ends up in performance reduction.

The Scale-out method helps upgrade the storage by adding new nodes which include processing power, memory and capacity, thereby keeping the overall performance consistent with practically no dip in user experience. This is true for both SAN and NAS based storages. These storages can be expanded to PBs on a single storage, or even a single file system, by simply plugging in a new node. It is viable commercially also, as the cost per GB goes down as you keep adding more nodes.

So don’t worry about handling your Big Data as the storage devices are now available to store them more efficiently.

How much Data is good for business?

When you talk of sources of data generation, there is an endless list. Any business stream would have a long list to show how data is getting generated and how much data is being generated. Often businesses get scared with so much of data as they think handling it is a mammoth task. Indeed it is a mammoth task as it needs good investments and infrastructure to handle it. However, if utilized properly, the benefits are much higher. The way businesses are competing, it would soon become inevitable to handle it carefully.

The more data you have, more opportunity you get to see how your products, services and customers behave. There are many examples of business being able to analyse their data patterns and offer more discounts or value added services to give their customer a delightful experience. The new databases handling this Big Data have come up with Multi Parallel Processing technologies and the new applications to handle unstructured data ensure that even if you have PetaBytes of data, you can still do real time analysis and produce results in nano seconds.

Let us enjoy this new revolution on the way technology and businesses are getting shaped up and reap the benefits of these.

Is Virtualization the driving factor for Cloud Computing?

Virtualization has been prevalent for decades in various forms. In the recent times, virtualization of servers and desktops over x86-platform has made a significant change in the way IT was being used. Virtualization gives you the flexibility of using multiple virtual servers created out of dynamic resources from a few physical servers. Resource allocation and re-allocation is very convenient, and most of the functions happen online. Most of these functions can be automated based on usage pattern.

For technologists finding ways of sharing resources, virtualization came up as a blessing. Cloud computing allows different organizations/departments to provision and utilize virtual servers for their own individual use while residing on the same physical hardware. The physical hardware hosted at a service provider’s Data Center allows access to different virtual servers separately.

Virtualization platforms have in-built auto-provisioning, security features and billing systems which charge based on periodic usage. Many service providers have developed web based self-service portals that allow users to create and use their own servers without any external intervention.

So sit back and create your own server with custom configuration and reap the dual benefits of virtualization and cloud computing.

Backup or Archive – What suits your need best?

Many organizations have been equating long term retention of backups as archiving whereas backup and archiving are too separate things and should not be treated as same or interchangeable.

Archiving helps reduce the backup load by moving the older data from the production systems to an archival system. This reduces the amount of data on the production systems enhancing performance and reducing backup window for the production systems.

While moving the data, the archival application leaves a stub on the production system. For mailing applications, individual emails are moved from production system reducing the production mailbox size and load considerably. When the user accesses the data from production system, he actually accesses the stub which in turn accesses the data from archival system to fetch it for him. The archival applications offer flexibility in terms of automatically choosing what to archive based on the size, attachments, date & time of creation or access etc. The other advantage is the long term retention of data without any load on production systems and its availability in its native format when you need it.

Backup on the other hand helps preserve multiple copies of the production data on a different media to help in the event of loss of production data. Backup does not move any data from production systems and does not reduce any load there. Moreover, backups are not stored in the native format; you need to recover them back to the native format from the backup device.