Fujitsu Laboratories Develops Technology to Automatically Resolve Performance Problems in Distributed Storage
|
|
Source: The Hoffman Agency
|
Date: Tuesday, 7 August 2012
|
|
Contact Person: Phyllis Wong
|
Email:
|
|
Web Site:
|
Attach:
|
| |
Fujitsu Laboratories Develops Technology to Automatically Resolve Performance Problems in Distributed Storage
Eases access concentration by 70% for popular data items, stabilizes operations for ICT services
Hong Kong, August 7, 2012 ?Fujitsu Laboratories Limited today announced the development of a technology that automatically resolves problems caused by concentrated accessing of popular data items in a distributed storage system in order to curb access-time slowdowns.
Distributed storage combines multiple servers into a single storage. Increasing the number of servers improves storage capacity and performance, making this approach appropriate for storing data that grows day by day. Furthermore, storing replicas of the same data simultaneously on multiple servers increases data reliability and access performance. If, however, there is a sharp increase in accesses to a particular stored data item, the load on the server storing it will increase, which may greatly cause an upturn in user access times.
Fujitsu Laboratories has developed a technology that can instantly detect spikes in popularity for a data item and automatically increase the number of replicas of it to level-off server loads. This automates what had previously been a manual approach for dealing with popularity spikes, and can limit slowdowns in access times. The new technology applies to distributed object storage (1), and in test cases of access concentrations on the Internet, has been proven to ease those concentrations by approximately 70%, resulting in an improvement in access times of tenfold or greater.
This technology makes it possible to stabilize operations on ICT systems where access patterns are difficult to predict.
Details of this technology are being presented at the Summer United Workshops on Parallel, Distributed and Cooperative Processing (SWoPP 2012 Tottori), opening August 1 (Wednesday) in Tottori, Japan.
Background
With the popularity of smartphones and sensors, the volume of data being stored and analyzed is growing rapidly, bringing about new business value. Distributed storage is often used to store massive volumes of data, in which multiple hard drives, solid-state drives, and other storage mechanisms are combined to be treated as a single storage. Storage capacity and performance can be improved by adding more servers. In addition, simultaneously storing replicas of a single data item on multiple servers increases the reliability of data (Figure 1).
Figure 1: Distributed storage - http://www.fujitsu.com/img/PR/2012/20120730-01a.jpg
Technological Issues
In distributed storage, concentrated accessing of a popular data item can result in a situation in which performance may have no correlation to the number of allotted servers. For example, if there is a news story with widespread public interest, most of the load will be concentrated on the server storing that popular data item, so the total number of servers is irrelevant to performance (Figure 2).
Figure 2: Performance slowdown under concentrated access of popular data item ?
http://www.fujitsu.com/img/PR/2012/20120730-01b.jpg
About the Technology
Fujitsu Laboratories has developed a technology called Adaptive Replication Degree, which automatically detects concentrated access of a popular data item and increases the number of servers with replicas of it to distribute accesses. This technology can very rapidly detect and resolve access spikes and prevent access-time slowdowns, enabling consistently stable access performance. It even stabilizes ICT systems where access patterns are difficult to predict. Adaptive Replication Degree automatically handles the process of detecting concentrated accesses and varying the number of replicas, so the process requires no manual intervention.
Details of the technologies in Adaptive Replication Degree are as follows.
1. Using a small amount of memory to rapidly detect data items which sharply increase in popularity, causing access spikes among massive data volumes
To detect sudden access concentrations, Fujitsu Laboratories developed a popularity-estimation engine that estimates popularity weighted by how recently it was accessed (weighted popularity) using a small amount of memory (Figure 3).
The newly developed method logs the number of data accesses only for a fixed number of data items, requiring little memory. When an unlogged data item is accessed, only a minimal access count is replaced. By carrying over the access count, popularity can be estimated with a high degree of accuracy.
Also, by reducing the access counts for every fixed number of accesses, the most recent access is weighted in the count, so that drastic changes in popularity can be detected.
2. Automatic optimization of the number of replicas
Fujitsu Laboratories developed a technique that analyzes how highly concentrated access occurs, which causes the number of replicas to fluctuate, based on the frequency of access during periods of concentrated access (Figure 4). The number of replicas to add is automatically determined when a popular data item is detected. This technique uses two threshold values—one indicating that a spike is underway, another as a sign of an impending spike—to detect periods of concentrated access. The higher the access frequency during those periods, the more replicas are created, so that replicas are increased in proportion to the magnitude of a traffic spike.
Figure 3: Popularity-estimation engine - http://www.fujitsu.com/img/PR/2012/20120730-01c.jpg
Figure 4: Access gradient analysis mechanism - http://www.fujitsu.com/img/PR/2012/20120730-01d.jpg
Results
This technology was tested on 64 servers in a recreation (2) of a real-world traffic spike caused by a breaking story on the Internet about a well-known pop star (Figures 5, 6). The changes in access frequency to each server per hour were measured, and using existing methods, accesses were concentrated on only those servers that contained related data. The access frequency increased by a factor of approximately 2.3 times. Using the new technology, however, the increase in access frequency increase actually dropped to 70% of previous levels, demonstrating that this technology effectively levels off loads.
Figure 5: Changes in access frequency on each server during access concentration under previous method - http://www.fujitsu.com/img/PR/2012/20120730-01e.jpg
Figure 6: Changes in access frequency on each server during access concentration using this new technology - http://www.fujitsu.com/img/PR/2012/20120730-01f.jpg
Another test was conducted using 16 servers to determine access time as viewed from the user's perspective. Figure 7 shows average access time for all data in periods of concentrated access and average access time for popular data in periods of concentrated access. Comparing previous methods with this new technology, all access times for all data in periods of normal access were set against an average. Looking at average access time for all data, access times increased approximately by a factor of 4 times under concentrated access compared to normal access using previous methods, but only approximately by a factor of 1.2 times using the new technology. Looking at popular data, access times increased approximately by a factor of 15 times using previous methods, but only approximately by 1.4 times using this technology.
Figure 7: Relative access times for each method for different data categories, under different load conditions- http://www.fujitsu.com/img/PR/2012/20120730-01g.jpg
Future Plans
Fujitsu Laboratories is continuing to test and improve the performance of this technology with the goal of applying it to products and services in 2013.
________________________________________
Glossary and Notes
1 Distributed object storage:
Storage where data and its metadata are handled as a single logical unit and managed in a distributed form
2 Recreation:
See page view statistics for Wikimedia projects, Wikimedia (online), available from http://dumps.wikimedia.org/other/pagecounts-raw/
- Ends -
Press Contacts
Fujitsu Hong Kong Limited
Yvonne Yew
Tel: (852) 2827 5780
E-mail: yvonne_yew@hk.fujitsu.com
PR Agency, The Hoffman Agency (www.hoffman.com)
Brian Chan / Jessica Ching
Tel: (852) 2231 8105 / 2231 8112
E-mail: bchan@hoffman.com / jching@hoffman.com
About Fujitsu
Fujitsu is a leading provider of information and communication technology (ICT)-based business solutions for the global marketplace. With approximately 170,000 employees supporting customers in over 100 countries, Fujitsu combines a worldwide corps of systems and services experts with highly reliable computing and communications products and advanced microelectronics to deliver added value to customers. Headquartered in Tokyo, Fujitsu Limited (TSE:6702) reported consolidated revenues of 4.5 trillion yen (US$55 billion) for the fiscal year ended March 31, 2011. For more information, please see: www.fujitsu.com.
About Fujitsu Laboratories
Founded in 1968 as a wholly owned subsidiary of Fujitsu Limited, Fujitsu Laboratories Limited is one of the premier research centers in the world. With a global network of laboratories in Japan, China, the United States and Europe, the organization conducts a wide range of basic and applied research in the areas of Next-generation Services, Computer Servers, Networks, Electronic Devices and Advanced Materials. For more information, please see: http://jp.fujitsu.com/labs/en/.
About Fujitsu Hong Kong
Fujitsu Hong Kong is a leading ICT solutions and services provider that creates value for customers through highly reliable and environmentally friendly ICT offerings. With 50 years of experience and as part of the Fujitsu Group ?a global family of ICT infrastructure experts ?the company has been leveraging innovation and technology in its flexible response to the unique needs of each customer. As the hub and gateway for customers exploring business opportunities in and out of China and the Asia Pacific region, Fujitsu Hong Kong is also committed to develop long-term partnership with customers. Our major customers include the Government of HKSAR, Cathay Pacific Airways, PCCW, KGI Asia Limited, CITIC Securities International, Modern Terminals Limited, Octopus, One Media Group and more. For more information, please visit http://hk.fujitsu.com.
All other company or product names mentioned herein are trademarks or registered trademarks of their respective owners. Information provided in this press release is accurate at time of publication and is subject to change without advance notice.
|
| |
|
|
|
|
2000 iPress