Ace Data Corporate Film premieres at CIO Power List event in Lonavla.

https://www.youtube.com/watch?v=4GdVcTCwNBQ

Learn from your mistakes, analytically

As discussed in the previous blog, every business has a risk and needs to find how to decrease the risks. Each negative outcome that occurs presents an opportunity from which to learn new things. In a small scale business, these decisions could be easy and more mind driven being controlled by a set of managers only. As the businesses grow beyond a size, predicting the future course of action and avoiding the past mistakes becomes important. 

The enterprise integrates the predictive model’s scores in order to act upon what has been learned. At each step, the predictive scores foresee where a “blunder” may incur unnecessary risk, thereby guiding the organization to avoid it. In this way, predictive analytics delivers a complete data-driven system for risk management. 

Predictive modeling capabilities are scientifically proven and have benefited from decades of advancement. With a tenured track record of success, predictive analytics boasts mature software solutions that deliver this technology to – and integrate it with – the modern enterprise.

Predictive Analysis used by Insurance Companies

All businesses are run at a risk. Risk is the way business is managed. Every decision an organization takes impacts the risks an enterprise can withstand like the risk of customer defecting, of not responding to an expensive glossy mailer or offering a huge retention discount to a customer who was not leaving even otherwise and in turn missing out on a critical customer who leaves.

The data driven means to compute risk of any type of negative outcome in general is predictive analysis. Insurance companies have used this very well. Insurance companies are augmenting their practices by integrating predictive analysis in order to improve pricing and selection decisions. 

The actuarial methods that enable an insurance company to conduct its core business perform the very same function as predictive models: Rating customers by chance of positive or negative outcomes. Predictive modeling improves on standard actuarial methods by incorporating additional analytical automation, and by generalizing to a broader set

Predictive Analysis and Big Data

Predictive analytics is an enabler of big data: Businesses collect vast amounts of real-time customer data and predictive analytics uses this historical data, combined with customer insight, to predict future events. Predictive analytics enable organizations to use big data (both stored and real-time) to move from a historical view to a forward-looking perspective of the customer.

For example, historical preferences of consumer can be analyzed from their usage patterns and promotional offers can be planned accordingly. The historical data can help predict which promotional offer would be most useful. 

There are a few solutions available that help achieve this. They combine the capabilities of data mining solutions and predictive analysis to provide a single solution for predictive analysis. Vendors may offer proprietary solutions or solutions based on open source technologies. Predictive analytics software can be deployed on-premises for enterprise users or in the cloud for small businesses or for project or team based initiatives.

Predictive Analysis

Organizations are no more asking for just analysis of their data. Traditional Business Intelligence tools have been doing that for all for a long time now.

What we are exploring is getting more useful insights from our data. Organizations are looking for visualization tools and predictive analysis to explore data in new ways and discover new patterns. Big data analysis is not only restricted to processing large volumes of data on a software, it needs equivalent hardware processing capabilities too.

Predictive Analysis is the practice of extracting information from existing data to determine patterns and predict the future outcome & trends. It helps forecasting what might happen in the future with an acceptable level of reliability and includes what-if scenarios and risk assessment. 

Applied to business, predictive analysis models are used to analyze current data and historical facts in order to better understand customers, products and partners and to identify potential risks and opportunities for a company. It uses a number of techniques, including data mining, statistical modeling and machine learning to help analysts make future business forecasts.

Who is a Data Scientist?

Simple words: A practitioner of data science is a data scientist. They apply their skills to achieve a broad spectrum of end results. They have the ability to find and interpret rich data sources, manage large amounts of data despite hardware, software and bandwidth constraints, merge data sources together, ensure consistency of data-sets, create visualizations to aid in understanding data, build mathematical models using the data, present and communicate the data insights/findings to specialists and scientists in their team and if required to a naive audience.

Data science is emerging to meet the challenges of processing very large data sets i.e. "Big Data" consisting of structured, unstructured or semi-structured data that large enterprises produce. A domain at center stage of data science is the explosion of new data generated from smart devices, web, mobile and social media. Data science requires a versatile skill-set. Many practicing data scientists worldwide specialize in specific domains such as the fields of marketing, medical, security, fraud and finance.

Does Real-Time Analysis Help?

Many people ask the question that is there actually a need for real-time analysis. How would it help them in their business and is it worth the kind of investment it needs to get into real-time analysis?

You can analyse your business to see how you can use it for your business development. We can site a few small examples of different streams that organizations have used to see the benefits. Let us start with healthcare – a small device worn on the waist belt injects desired quantities of Insulin to a  diabetic patient while monitoring the sugar level through the device.

Imagine you enter a hotel and the person at reception knows your name, booking details and your preferences. While the second part is already in control through club memberships, the camera on the door takes your picture, searches for the record and by the time you reach the reception, your details are on the desk.

Someone walking near the store gets a message on additional offers on their favourite product for the next 30 minutes.  If you reject a product from the shelf, you get a message of additional offers on the product. Always get a win-win situation for both the store and the consumer.

From Just Analysis to Real-Time Analysis

Human race has been playing with data for ever. Large volumes of data being generated, and reports created of these, have been the traditional approach the way business have been run and expanded. The future prospects were based on some trends of your own organization, market and competitors and all this was based on the performance.

Analysts have been analysing this data to help organization plan the above.

Times have changed now and businesses cannot only rely on past performances to plan their future. You need to capture the current trends and the needs of the consumer today. Analytics have changed now from analysing past data to perform real-time data. Data generated not only from the structured in-house databases but also from non-structured data generated through social media and consumer behaviour.

Tools are available today to collect data from these sources and put them together through what is called the process of “Data Conditioning” into databases to help analyse them. All this data gets processed real time to produce near instant results helping the businesses serve their consumers better.

What is Data Science?

While most of us believe Data Science is handling large volumes of data effectively and efficiently, Data Science is not restricted to this. Merely using data isn’t really what is meant by “data science.” A data application acquires its value from the data itself, and creates more data as a result. It’s not just an application with data; it’s a data product. Data science enables the creation of data products.

Several organizations have used this well and have given the industry several products that help handle Data Science. Key examples are from Google itself who realized the importance of a search engine and further made it more effective using tracking links. Spell checks with suggestions and speech recognition are other examples of Google’s way of creating products out of data. Facebook and Linkedin have used patterns of friendship relationships to suggest other people you may know or should know, sometimes with great accuracy.

The thread that ties most of these applications together is that data collected from users provides added value. Whether that data is search terms, voice samples, or product reviews, the users are in a feedback loop in which they contribute to the products they use. That’s the beginning of data science.

Target Based Deduplication

Target based deduplication was one of the initial ways of performing deduplication. This was more on the VTLs when VTLs were launched as a technology, and over time, it seems that this is not much in use now.

All it did was to carry the data to the target backup device – mainly VTL – and store it there. It would then run a deduplication process to match blocks and delete the duplicate data. At a later schedule, system will run a clear garbage type process to finally remove the deleted content and free up the space on the VTL.

When it was launched, it was the only process and was a great process. With newer and better technologies available, target based deduplication has lost its charm since most applications now prefer source based deduplication and reduce data before it travels on the network.

So target based deduplication has its advantages of using minimum source processing and memory cycles in performing deduplication at the target, the hind side being slightly oversizing the target to ensure enough space to accommodate full data before starting the deletion process.

Inline Deduplication

Inline deduplication looks as a very impressive term. You are made to believe that the magic would happen on the wire but at the end of the day, there are many caveats.

We have had a very recent experience where someone sized it with assumptions and commitments of reducing the backup & recovery windows tremendously but it did not really go as down as it was expected to be going.

Inline deduplication starts working on the source server itself and is followed by some more processing on the network. The left overs are taken care of by the media device’s memory. So, if you think you have a lot of extra resources on the production servers, go for this. If you are low on resources, you should first upgrade the production servers and then you are expected to have atleast two 10 Gig ports dedicated for the deduplication device and a well sized media server.

Just expecting wonders by replacing the backup device would not help much. It will reduce the backup & recovery windows a bit especially if you move from tape to disk while getting deduplication. However, you should be extremely careful in terms of your upgrade plans and the expectations that you set for yourself.

Source Based Deduplication

Choose the unique content from source itself when you start a backup. It does utilize some processing and memory from the source system so size it well.

Source based deduplication is also very powerful in ensuring that you utilize minimum network bandwidth during the backups. The backup application will create blocks of data on source and then store their hashes there at source and send unique data on the network. This is good for backups only if it is sized well. Catalog created by some applications is large enough to cause trouble for the performance of the source system which could be a production system.

Source based deduplication also gives good results for file system backups. A traditional approach takes long for file system backup that has millions of small files taking days for getting written especially during a full backup cycle. Source based deduplication in this case picks up only the changed content of the changed files reducing the amount of data travelling on the network irrespective of the backup level set.

Global deduplication on the target further reduces the amount of data stored.

Deduplication on Storage

When deduplication was launched for storage, it seemed a difficult technology to handle. Like any other technological aspects, deduplication also needs processing power and memory. So  deduplicating everything while storing on the primary storage would not be very effective. The base premise it started off with was reducing the disk storage investment by reducing the content to be stored. In real sense, it did not help much. While it would reduce the number of disk spindles required to store data, lesser spindles mean lesser IOPS so a compromise on performance.

SSD based storages require huge investment. Deduplication there can help reduce the disk requirement. Being SSD and capable of large no. of IOPS per disk, there is no compromise on IOPS while deduplicating. While working on one such customer requirement recently, I realized that this does not end here. Scale out storage further provides more processing power
& memory every time you upgrade and help you with consistent performance. Deduplication also happens inline i.e. you write only what is unique unlike other technologies that do deduplication at rest i.e. you write everything and then run a deduplication process to mark the duplicate content followed by a cleanup process to remove the duplicate content.

Choose deduplication on Storage with a caution, it may not be as fascinating as it looks like.

Store less, backup lesser – The Art of Deduplication

Deduplication is not a new concept though the term is relatively new. It originated more from the backup prospective though it has now got good footholds in many IT components.

I know and have been using for over a decade backup technologies that would backup only unique emails and files, and subsequently backup only delta changes. Being the only one doing that, they used their own language. They did not coin the word deduplication. Now everyone does that and calls it File level deduplication. This approach served good for desktop and laptop backups. Then this technology progressed, the current being block level deduplication. This recognises unique blocks of data and ensures that one block gets backed up only once, thereby reducing the amount of data travelling and getting backed up.

Like any other process, deduplication also needs resources to do what it is meant for. Various applications offer various forms like source based, target based and inline deduplication. Each one of them has their own working mechanisms and pros and cons. In the following series of blogs we will discuss the various methods of deduplication along with their pros and cons.