Here's an obvious statement: the growth of storage will continue to accelerate. We all know this to be the case within the IT world. We see it in our personal lives (think of all of your pictures and videos in the cloud or at home) and within the IT organizations in which we work. Most organizations have some sort of data retention policy that is mandated by law, or by industry or compliance regulations. And even if there is no formal policy, many organizations have a self-imposed (more like CWA) policy to save everything, just in case. The resulting challenge is that there is no foreseeable limit to the volume of data being saved. As well, that information may, in fact, never need to be accessed – yet it can’t be discarded because you just don’t know.
There are many solutions that can be used to provide long term storage for data you define as “archival” – rarely accessed but not discardable. They range from the simplest, such as storing your older information on tape and hoping you can find it when needed, to full disk-based archiving systems that allow you catalog and find the information you need, nearly instantaneously. The challenge is that, while the first solution is inefficient but relatively inexpensive, the second solution is very expensive, especially for data you may or may not need. A just-in-case solution can be an inefficient solution to a difficult problem.
Vicom has observed that organizations of all sizes with large amounts (200TB +) of archival data are looking for something in the middle: a solution that provide the cost-effectiveness and reliability of tape, but includes the cataloging and nearly-instant access of disk solutions. Many people scoff at the idea of tape and feel that tape is dead, no one uses it anymore, etc.. People have visions of large companies with large tape silos with lots of drives and lots of tapes, as there are still many installations out there like that. But tape is still one of the most reliable and cost effective storage solutions, period, for organizations of all sizes. And for long term storage of archival data, tape just can't be beat in terms of price and performance. If we can create a solution that combines cost-effective tape, fast disk storage for frequently-accessed data, and a specialized file system to tie it all together, then you have a solution that rivals full-disk solutions at a fraction of the cost.
But what about access and manageability? The solution mentioned above will be very policy-driven and will provide the file system and programming to handle all of the back end access, so users need simply click on the file they need and the system will retrieve it. The solution will “know” which data is accessed more frequently, and will place it on disk versus tape for near-instant access, a function called “tiering”. Data that resides on tape will available within a minute or two at most. But 30-60+ seconds of wait time for a file that is rarely accessed, in a solution that is much cheaper than a full disk solution, is something our customers see as worthwhile and good tradeoff. This type of solution can be administered and managed by an existing storage administrator, because once your policies are defined and setup, it needs limited management. This is the sort of solution that has become enticing to many of the organizations we tend to work with today.
There are many ways to approach archival storage needs and the above-mentioned solution is just one of them. But it happens to be one that is very cost effective, which is a big concern of organizations that we work with today for data that may or may not need to be accessed, but yet needs to be available, due either to internal mandates or to legal compliance. Whatever you choose to do for archival storage for your organization, make sure it gives you the access and performance you need at the price point that you want.
Andy