New Paradigms to Overcome the Data Bottleneck in Next-Generation Transactional Applications
September 9, 2010By Sumit Kundu
The effectiveness of meeting the needs of the next-generation transactional applications will depend on the ability to support the memory-based, data virtualization layer and the level of integration of this layer with the backend systems. Read on to learn the challenges with traditional transactional systems.
The Flash Crash on May 6, 2010 may have been caused by a traders fat fingers, or perhaps the real reason may never be known. But it highlights the need for more effective pre-trade compliance checks. With algorithmic trading, decimalization of stock quotes and other factors, trading volume and frequency has increased by many folds today. The underlying data management infrastructure that handles fraud and compliance checks needs to be able to keep up with current levels of trading.
In todays post-crisis financial markets, regulation has emerged as an important driver . In this climate, reducing risk is a key business imperative and data management infrastructure that supports that will be critical. This support may involve providing the risk desk a complete position of the intra-day trades and avoid the situation where risk calculations at the end of a day spill over to the following days trading window. A more ideal situation would be to support a real-time view of all the trades that every desk in the company makes at any instant to understand the enterprise-wide risk at any time and take immediate actions.
Challenges with Traditional Transactional Systems
In general, new classes of applications need data to be served to a larger number of users with lower response times or perform temporary operations, which makes accessing and storing data with traditional transactional systems non-ideal for such applications. Such applications can be found in a variety of markets:
These situations point to the need for high performance and scalable data management infrastructure. High frequency transactions create a bottleneck at the data server layer due a number of factors. One constraint to increased performance of traditional OLTP systems is the reliance on the disk system, and the optimizations that are done to reduce the impact of disk I/O do not go far enough. Another constraint with traditional RDBMS systems is data contention, and horizontal scaling techniques, such as sharding, have to be employed to ensure that performance does not degrade as more users are thrown in.
With the increase in web-based applications and growth in e-commerce, there is a critical need to manage session information, which can also hold contextual data that is accessed frequently. Consider the world of a frequent airline traveler with a mobile device as the preferred interface to the digital world. An airline application could detect when the traveler has missed his or her connecting flight and provide information on alternate connecting flights or local hotels instantly when the traveler lands at a connecting airport. In such situations, much of the computations involve working with temporal data and going to the database and retrieving such information adds overhead that makes such systems unable to scale and meet the performance requirements. Further, as the state information changes with a user accessing newer sites, the performance problem compounds due to the writes involved.
In the airline example above, one can expect a spike in workload when flights are cancelled due to bad weather or a terrorist threat. A traditional RDBMS may not be able to support the level of user scalability required in such cases. Traditional OLTP systems strictly adhere to ACID (atomicity, consistency, isolation, durability) properties. While this makes life easier for application developers, the need to achieve consistency across partitioned databases limits the scaling that can be achieved. In developing web applications with partitioned data, some of the consistency requirements can be sacrificed to achieve scaling . By following the BASE (basically available, soft state, eventually consistent) paradigm, web applications can achieve a higher degree of scaling by knowing that data consistency can be in a state of flux. However, this imposes on the application developer a level of understanding of the operations within a transaction to take advantage of this paradigm.
In-memory to the Rescue
But developers accustomed to developing traditional transactional applications may insist on the ACID properties being there. So what can be done? Fortunately, there are a few alternate paradigms that could be considered.
First, to avoid the slow disk-subsystem, we can consider in-memory database systems, which can provide lower latency and increased throughput. If data is temporal and does not need to be saved, then increased performance can be gained without the overhead of saving data to disk . But even when durability is required, applications can gain performance where workloads are compute intensive or involve mostly read transactions. Pre-trade compliance checks is a good example where in-memory databases can provide a significant performance boost. The airline traveler or the user of the e-commerce application would also experience faster response times from such a solution. A key value proposition here is being able to boost the performance by the flip of a switch without requiring any changes to an existing application.
An alternate way to achieve lower latency and increased throughput while maintaining ACID is to take advantage of distributed caching platforms (DCP). Such systems are memory-based and lie at the mid-tier level, beyond the boundaries of the traditional RDBMS. There are several advantages to this approach. First, by moving the data closer to the application, we can avoid the overhead of multiple hops to retrieve the data and significantly improve the response times. Second, we can enable new application development paradigms, such as object-oriented programs developed using a Java framework, by avoiding the programming and translation overheads between object and relational models. Finally, by avoiding the bottleneck at the RDBMS layer, distributed caching platforms can provide elastic scaling that is a key value proposition for cloud computing.
The lookup of information on hotels nearby a certain airport in the frequent airline traveler example above can be done more efficiently using DCPs. Commonly referenced data can be stored in the cache once and referenced by different keys using different indexes from different processes, saving the need to do the lookup in the database for each process. Further, DCPs can be used to distribute application cached data across multiple processes and make application development easier where such distributed architecture is needed. While the RDBMS can continue to be the system of record, DCPs complement by being the synchronization layer for transaction control.
Data Virtualization Enabling Next-generation Applications
The march towards real-time and need for low-latency response will be felt across various types of applications. To satisfy this need, there will be increasing reliance on use of in-memory systems. Instead of ad-hoc measures, next-generation application development will be enabled by the concept of a data virtualization layer that has the following attributes:
In general, most enterprise applications will rely on a disk-based RDBMS as a system of record at the backend, but the effectiveness of meeting the needs of the next-generation transactional applications will depend on the ability to support the memory-based, data virtualization layer and the level of integration of this layer with the backend systems. In this regard, it will be critical for a data virtualization solution to consider optimizations on loading data, cache invalidation and synchronization techniques, among other strategies.
About the Author
Senior Director of Product Management, Sybase
Sumit Kundu is a senior director of product management responsible for Sybase ASE and other data management products. Sumit's interests lie in working at the intersection of business and technology. He is currently involved in leading a project to drive cross product synergies by bringing together various data management products into a single platform to address the business and technology needs of Sybase customers.
 Melanie Rodier, Wall Street Technology, Data Management a Top Priority for Wall Street Firms, June 9, 2010
 Dan Pritchett, ACM Queue, May/June 2008
 Peter Dobler, Sybase ASE 15.5 The Need for Speed,