X

Alcatel-Lucent needed to build a new application to take advantage of structural changes in the dynamics of the industry. With the explosion of smartphones and the growth in data-centric applications, legacy solutions were struggling to keep up. As a result, innovation was suffering and customer satisfaction was declining. A recent survey conducted by an internationally known consulting company supports industry dynamics that are stymying carrier efforts to remain competitive. The proliferation of mobile devices, changing demand as a result of increased customer sophistication, lack of differentiation due to application commoditization, and emerging competitors offering consumers contemporary services were all cited as challenges. Survey participants cited three primary limitations of their billing systems: questionable accuracy due to a variety of dynamic inputs into the application, rigidity resulting in an inability to quickly change plans as user demands shift, and lack of integration with other systems. Among the CSPs who responded, 66% cited billing accuracy as a key challenge because current billing platforms cannot handle the expanding array of new services and devices, and the resulting explosion in the amount of data consumed. 60% said they are concerned with their billing system’s ability to process the volume of data typically generated with digital services. 70% said their billing systems cannot effectively integrate with other key applications.

To scale without barriers, provide superlative customer experience, and continue to introduce new, engaging mobile applications, this very large financial institution had to rethink its infrastructure. Continuing with a relational database and cache-based solution as the intraday system of record would require scaling from 150 servers to more than 1,000 servers. This was not a practical strategy in terms of time, labor, and operating cost. More important for the business, reliance on nightly batch processing from the intraday system to the master DB2 (book of record) was expensive, and still didn’t solve the data inconsistencies between stored and active data. A new solution had to address: The company’s decision to continue to leverage its legacy mainframe database (DB2), which was the compliant system of record for more than 10 million customer accounts. The requirement to process 250 million transactions and 2 million updates a day, and the ability to update stock prices or show balances on 300 million positions in near real time. The ability to create enough compute capacity to eliminate data inconsistencies. The elimination of frequent system crashes due to overloading the RAM-based cache and the subsequent restarts, which frequently took around an hour. The mandate for a cost-effective solution that would address expectations for 1,000% data growth as it executed on its mobile strategy.

Creating a more accurate fraud detection and prevention solution required factoring in unique attributes of the payment provider’s business model. The solution had to consider a myriad of realities, which included the company’s payments, rewards, and purchase protection programs, all of which have different rules and data-processing requirements. Unique profile attributes for 150 million customers, including payment and invoicing preferences, user ID, IP address, devices, location data etc. Mobility and multiple device dynamics that are often entry points for criminals to target mobile and “cross border” transactions. Visibility into payment histories and behavior patterns that help determine the validity of a transaction. Legacy database and infrastructure strategies that are unable to handle the requirements of a contemporary application, including leveraging near real-time data as a critical component of modern fraud algorithms.

Signal, an identity resolution platform, faced significant challenges with their existing data store, Cassandra. The platform was becoming increasingly expensive, unreliable, and nonperforming, leading to large and unpredictable latency responses and uptime issues. These problems were affecting every element of their business processes, resulting in more frequent and severe incidents. The Cassandra clusters were difficult to maintain and required high-touch operations, diverting resources from higher-value projects. Signal's Customer Identity Platform (CIP) had grown to over 550 servers, leading to server sprawl and unpredictable performance, negatively impacting the company's SLAs. The projected 25% data growth further complicated the situation, prompting the search for a new solution.

Playtika has six gaming studios, each with a different game. It was using a different NoSQL solution, which was not helping the company reach its goal of 150,000 reads and 50,000 writes. Further, the architectural limitations of that solution caused Playtika to implement multiple clusters of nodes per studio - something it did not want to persist. There were different workloads on different clusters (some were highly loaded and others were barely used) and it was difficult to scale out with such a configuration. Adding to the problems was that the incumbent NoSQL solution is not ideally built for SSDs because of its internal rebalancing process. That led to the disks being frequently exhausted, malfunctioning and forcing replacements every six to 12 months.

This world’s largest, independent, real-time ad technology platform, AppNexus, empowers companies to build, manage, and optimize their entire online advertising business. Each day, hundreds of companies around the globe buy and sell billions of online ads using AppNexus’ real-time ad serving technology, advanced yield management controls, optimization algorithms, and patented brand and safety monitoring. Managing and effectively leveraging these ever-growing data sets have a real impact on an organization’s bottom line. AdTech platforms like AppNexus need to use what they know (the data) to power what they do (serve relevant ads) in real time. The combination of speed and scale are required to deliver tailored responses and spur immediate consumer interactions within milliseconds. The challenges are many. In this case, AppNexus needs to: Optimize ad campaigns and increase campaign effectiveness, capture and analyze clickstreams, transactions, video and social media data, spot trends and patterns, understand customer sentiment, unearth new relationships, and adjust campaign tactics in real-time. Scale rapidly as business grows. With its infrastructure reaching capacity, a cost-effective database and storage solution with predictable low latency and high throughput is a critical component of its ability to continue providing high-quality service to its customers. Manage rising infrastructure costs and complexity. With infrastructure growth comes in higher maintenance, power consumption, and operational costs.

Deutsche Bank Technology Centre faced significant challenges in managing and analyzing data within its Investment Bank division. The natural data silos and the need for a robust Business Intelligence (BI) system were evident. The existing data warehouse solutions were either too expensive or too slow, such as Vertica and Hive, respectively. Additionally, the bank required a data-driven access control mechanism that could provide record-level granularity and full access to SQL, while reusing existing bank-wide access rules. The challenge was to find a solution that could handle heterogeneous data from 72 different data sources and systems, manage over 100 ETL jobs, and provide a seamless user experience across various UIs.

Over the last few years, Appsflyer had pushed its internal analytics to the limit. Three years ago, Appsflyer started using Looker. Over time, adoption grew to 1,000 Looker users across sales, support, marketing, finance, and HR, including 60 Looker experts who supported other users by building and maintaining the various dashboards. Appsflyer was running their analytics on Amazon Athena, but as usage had grown, so had the challenges. First, the data had become too big for Athena to handle. Appsflyer has 35 petabytes of terabytes of raw data about customers and their activities, with multiple tables consisting of billions to hundreds of billions of rows. Athena could not handle more than 5 billion rows of data in any query. So they had to aggregate the raw data using Databricks to a smaller data set that Athena could handle. Athena also could not handle more than 20 concurrent queries, which had become a problem by the time Appsflyer reached 1,000 Looker users. Second, the ETL process was slow, expensive, and inflexible to change. A spark job often took weeks to change, and days to run if they had to rebuild all the dimensions and history. This was not only too long, since users often required new reports in days, but also very expensive. Updates also became a problem with GDPR regulation. Each time an entity opted out, Appsflyer had to rerun entire Spark jobs to remove them, which not only took a long time to run but was very expensive as well. Third, even with the dramatically reduced data sets, Athena was still too slow. Some queries took minutes to run. Most interactive analytics, such as the executive dashboards, required query times of a few seconds at most. When the company needed a new report that required the detailed results from a join of two multibillion row tables, Appsflyer decided it was time to replace Athena.

As Swedbank's application development teams were building more solutions, the Tech Stream team faced significant overhead in setting up the supporting backend systems. They used a traditional relational database management system (RDBMS) for data storage, but creating a schema for every use case became cumbersome. This inhibited their ability to move quickly for any new solution the market demanded. Additionally, they discovered that their RDBMS was overkill for their needs, as data storage was being handled entirely in memory. They sought a lighter weight solution that would eliminate the time-consuming schema definition process while providing higher levels of performance. Reliability was also a major issue, as their RDBMS was a single point of failure. Security was another pivotal concern, given the sensitive data they maintained.

A few years ago, the team began looking for an in-memory computing platform that supported secure internode communications in their server cluster. In their growing business environment, performance and resilience were critical top-of-mind issues. They needed real-time visibility into the transactional flow between nodes. They felt that pursuing an in-memory solution would provide them the performance levels they needed, but they also needed extremely high levels of business continuity. They also felt they could leverage an in-memory technology as a caching mechanism on the database data for their microservices. Just as importantly, regulatory compliance was a factor in their initiative. Regulations like the Payment Card Industry Data Security Standard (PCI DSS, or sometimes just PCI) was an inherent part of their business. Having a platform that could simplify their compliance effort and reduce the risk of violation would be ideal. One requirement of PCI is implementing processes and technologies that will protect cardholder data from unauthorized access. This generally includes a combination of authentication/authorization controls plus encryption. The former ensures that only authorized personnel can access the data when going through legitimate access points. The latter ensures that the data is protected from a wide variety of system breaches like “eavesdropping” (i.e., when a hacker captures network traffic of transmitted data between nodes).

A leading retailer faced the challenge of managing and integrating vast amounts of data generated from its global operations. With billions in annual revenue translating to thousands of dollars per second, the need for efficient data tracking, maintenance, and analysis was critical. The platform team aimed to build a centralized cache-as-a-service to streamline data integration across various systems, reducing redundancy and overhead. Without this centralized service, standing up new applications would take months, which was unacceptable for a high-load business processing millions of transactions per month.

British Gas-funded start-up Hive needed a technology solution to store large amounts of data in memory for quick access. The company required integration with its current core backend platform and a product that could scale linearly to fit its growing needs. Ease of querying, real-time query capability, and custom queries were essential. Hive also needed to cache state information for millions of devices with simple deployment and management. After trying MongoDB and finding it inadequate for their traffic demands, Hive sought a better solution.

Running experiments and collecting data on sounding rockets is a fine-tuned process, and part of that process entails making adjustments to experiments along the way. For the FLUMIAS payload, scientists need to make sure that captured images are in focus to provide the most value to all stakeholders. And for any given mission, there are usually distinct teams of scientists running experiments, so the system needs to switch between these different experiments to visualize the images and data relevant for the respective teams, enabling them make the appropriate adjustments. The Airbus Defence and Space software engineering team sought a technology that was easy to understand and could fit within their infrastructural requirements. While there were many potential candidates in the market, it was uncommon to find one that had a suitable programming model with solid documentation. They didn’t have the time to get up to speed on unnecessarily complex systems. And with all the other mission support activities that they were responsible for, the team didn’t necessarily have the time to do extensive research to find an ideal technology.

HUK-COBURG was undergoing a digital transformation initiative to leverage technology to give them a competitive advantage. The initiative entailed improving customer satisfaction as well as reducing the cost of maintaining the data infrastructure. They also needed a better way to improve system reliability, as they expect to run 24x7 systems. The business teams required an upgrade to the data flows and processes, so it was up to the architecture team to propose new solutions to meet the business teams’ needs.\n\nIn one part of their infrastructure, they were reading telematics data to calculate insurance premiums for their customers. But data accesses to the mainframe were costly due to the complicated path in which the data was routed. Access required going through multiple firewalls and validation systems. In addition, there were times that the connection was unavailable, which hampered their ability to serve their customers. Their new premium pricing initiative, which relied on the telematics data, continued to grow rapidly since they were moving many existing customers to that pricing program. They anticipated tripling the customer count to the hundreds of thousands, so the current latency and unavailability would soon become huge problems.\n\nThey knew they would save a significant amount of time and cost if they could cache some of the data retrieved from the mainframe. They also knew this could represent a resiliency pattern where the required data would always be available in the cache even if the connection to the mainframe was unavailable.

The global pizza delivery chain initially used Voldemort containers for custom replication between datacenters. Although the performance was acceptable, replication was unreliable, affecting customer order history and profile data. The early adoption of Voldemort made the upgrade path difficult, and there was no tangible support available. This led the development team to explore more modern storage architectures. They experimented with SQL Server and Memcached, but neither achieved the required performance results. Concurrently, a social media project required de-duplication of data from multiple Twitter feeds, leading them to Hazelcast IMDG as a microservices alternative to Voldemort. The Twitter project team realized they needed a solution to coordinate data between multiple nodes, and Hazelcast IMDG's potential as a replacement became evident when combined with SQL Server, providing the necessary flexibility and tooling to meet performance goals.

As one of the largest providers of internet, voice, and media products and services to business and residential users in the US, this media conglomerate’s employees are responsible for handling over one million customer interactions per day. Its customers can access support services via many different channels - call center, website, or from a mobile device using self-service support; all of these channels roll up into the same customer support organization. The challenge was their ability to handle the large volume of events that were system-generated (from within their own infrastructure) or from users interacting with their hardware or software applications. A major objective for the business was to be able to automate as much of the support process as possible and to reduce support response times for their customers. Having the ability to access up-to-date customer account information such as who the customer is, what services they have, where they live, what’s their history, what’s the current state of the devices in their home, etc. would be key to improving the current event-based support model. By having this data available in near real-time, support can quickly identify and analyze areas where there may be problems with a service or product.

Before Ellie Mae adopted Hazelcast, its Encompass application was suffering from two major problems: performance and scalability. Encompass could not function within the approved service level agreements due to the disk-based access methodology of its database vendor, resulting in high latencies and lower throughput. Attempts at caching frequently used data in the Encompass application’s memory with a homegrown caching solution failed due to data inconsistency across application nodes. This internal solution was too burdensome to maintain and did not guarantee high availability, resulting in failed SLAs. Scalability was another major challenge, as Ellie Mae was expecting its business to grow by 25% to 30% every year, and its current application memory setup was not scalable, potentially resulting in significant losses.

Competent fraud detection systems are crucial for organizations to identify financial crimes such as payment card fraud, anti-money laundering, and anti-terrorist financing. The increasing sophistication of fraudsters' systems and the growing volume of financial transactions put immense pressure on existing detection systems. In the UK, payment card spending exceeds £180 billion from 3.9 billion transactions, with financial fraud losses consistently rising. Detection systems must be flexible, report anomalies in near real-time, and be available 24/7 to cope with these challenges.

Gamesys was in the process of building a highly scalable poker platform and was exploring various technology options to store the game state. The solution needed to be capable of being used both for real money gaming and social platforms. For real money gaming, redundancy was crucial to ensure that a player’s money couldn’t be lost, requiring replication of results and idempotent handling of actions. For social platforms, the system needed to be fast, elastic, and cost-effective, capable of scaling up quickly based on fluctuating demand. The challenge was to build a platform that could manage failures without losing a player’s money, was highly available and elastic, and could handle tens of thousands of concurrent players.

Future Grid faced the challenge of processing extreme volumes of data in real-time for Australian utility companies. Traditional relational databases were inadequate for handling the 3 billion data points collected daily, leading to inefficiencies and high costs. The need for a real-time data aggregation solution was critical to enable complex, real-time decisions and overcome issues related to data volume, speed, reliability, resilience, and license costs.

The logistics company faced significant challenges in managing an evolving data model across a large, heterogeneous set of data sources. With almost 50 different relational databases running Oracle, MySQL, and PostgreSQL, the company needed to address performance issues and enable more interactive data exploration. The introduction of a Hazelcast in-memory data grid as a distributed cache improved data retrieval latency but added a new layer of maintenance complexity. The company needed to keep the data model of the data grid in sync with the data sources as the relational database schemas changed over time.

Following the introduction of a new property tax, Irish Revenue’s IT department was tasked with launching a new service for homeowners who were required to declare their property liability and make a declaration online. It soon became apparent that their existing IT architecture wouldn’t be able to cope with two million property owners accessing the website – potentially at the last minute. In addition, Irish Revenue wanted to change the way it managed back-ups, fixes, and upgrades. Historically, these were conducted during the evenings which could result in some services being unavailable as nighttime copies were made and applications tweaked. Therefore, it required a solution which could perform operational tasks seamlessly in parallel with no external effect on service quality. Crucially, Irish Revenue had two prime solution requirements – high availability and performance. Due to its desire to implement an open-source solution which could handle surges in traffic and store data in-memory, Irish Revenue approached Hazelcast.

E-commerce sales are growing rapidly, but traditional inventory management systems struggle to keep up with the high volume and velocity of data. This often results in inconsistencies between inventory databases and customer-facing systems, leading to lost sales and dissatisfied customers. Traditional relational databases and some NoSQL solutions fail to provide the necessary speed and scalability, causing latency issues and poor customer experiences.

Market Data systems in investment banking require the ability to onboard new data feeds quickly and efficiently. Traditional relational databases and NoSQL solutions often fall short due to their reliance on disk-based storage, leading to poor application query performance and unacceptable online customer experiences. Integrating multiple market data feeds without impacting existing throughput is critical, but schema changes in RDBMS can introduce narrow maintenance windows and system impacts. Additionally, trading desk systems demand low latency access to data, which is often measured in microseconds. The complexity and high cost of maintaining these data feeds can stifle innovation and lead to duplication of efforts across multiple development teams.

When system architects approach a problem domain, they often have to turn to multiple technologies. For FX, there will be some mixture of queues, reactive message passing, and flexible data storage requirements. In the past, this would mean using a relational database as the storage, separate queue infrastructure, and some form of in-house code to connect these infrastructure pieces together. Only once this is in place can the development teams begin to think about addressing the actual problem domain. To complicate matters, the database and message queue software will often be provided by different vendors, and internally the teams that manage this infrastructure will often be different. In sum, the number of moving parts and organizational boundaries means that it can be challenging to quickly and cost-efficiently deliver projects on time and on budget. In the past decade, many have turned to NoSQL datastores to solve some of these problems, but they have found that these technologies still do not provide a whole solution. For instance, relational databases struggle to provide efficient expiry of data, whilst most NoSQL datastores do not provide ways in which outside processes can observe data changes reactively. Neither provide scalable, fault-tolerant queuing capabilities.

The ability to quickly and cost-effectively deliver secure and trusted systems is vital for modern businesses. B2C and B2B systems demand reliable and secure communications, with security being a critical feature for managing access for customers, partners, operators, and administrators. These systems must handle security checks for millions of transactions per second without impacting SLAs. The challenge is further compounded by the need for coherent security across multiple channels, including mobile, tablet, desktop, and customer service systems. Failure to modernize access control can lead to costly integration issues and potential security breaches, damaging a company's reputation and leading to serious trouble with data protection agencies. Multi-factor authentication using trusted devices, such as codes sent to customers' mobile phones, is now a standard requirement. This involves fault-tolerant messaging and reliable data expiration, which can be expensive and complex to implement across various systems. The days of simple username and password authentication are over, and IT departments must deliver coherent security across multiple computer systems.

USAA was looking to modernize their web infrastructure to help grow their online business. The previous generation Websphere-based infrastructure at USAA did not provide a unified, fault-tolerant, scalable foundation for their customer-facing applications. Rather, each application required its own authentication and data management, and this was seen as inefficient due to redundant information management. The new infrastructure based on Hazelcast utilizes an In-Memory Data Grid (IMDG) architecture for shared access to numerous services and functions. With easy-to-scale Java infrastructure, application developers can have fast access to cached data and employ a single authentication strategy for all customer-facing applications, from banking to insurance. Another important capability of the new infrastructure at USAA is synchronization among data centers. The previous generation utilized redundant production and QA clusters at two data centers that are relatively closely located. For better data compliance and assurance, USAA is introducing a third data center that is very distant from the existing two and requires that the synchronization and data integrity be well managed. Hazelcast Enterprise includes critical support for WAN Replication for this exact scenario. Fault tolerance is critical in any large-scale solution, and USAA was using a single-copy caching architecture. A key feature of an IMDG is the automatic and seamless backup and restore capability. Hazelcast is a superior technology for data sharding, partitioning, and backup in a distributed environment, providing the IT organization the assurance that data is not lost should a single node or even a data center fail.

Delivering e-commerce systems that provide competitive advantage is one of the toughest challenges facing retailers today. Flexibility and speed are the keys to building a world-beating e-commerce system. The ability to deliver a personalized shopping experience with relevant content at predictable sub-second latency is the difference between business success and failure. Customers will typically wait only 1 to 2 seconds for a page to load before moving on. Speedy response and high availability are absolute musts, and making sense of the huge volumes of data generated by customers is critical. Some of the complex business requirements driving today’s e-commerce systems include generating personalized shopping experiences in microseconds using multiple data streams, omni-channel systems that deliver a consistent shopping experience to the customer across mobile and web, and in-store, removal of data and system silos to present a single view of your data (warehousing, supply chain, social, customer), flexible datastore that allows full-scale updates to an entire product catalog in minutes rather than hours, the ability to meet peak demand workloads and then shrink systems at quieter periods to save compute costs, and dynamic pricing, comparing against competitors’ catalogs, and then adjusting pricing in real-time.

Growing revenue from existing customers is challenging when data is locked in separate silos across various systems. Traditional relational databases and legacy mainframe systems are inadequate for the modern digital world that requires agility and innovation. Real-time analytics on large volumes of data suffer from latency issues, and database-to-grid architectures create more points of failure and increase processing time.

The financial services industry faces demanding technology requirements, necessitating the adoption of advanced solutions like IMDG technology to process data at high speeds. Financial institutions need to run advanced queries and perform complex transactions on large datasets quickly and scalably. The rise in data volume, driven by increased technology use and social media, requires financial organizations to utilize all available data resources for competitive advantage. Additionally, complex regulations across multiple jurisdictions demand extensive data retention and reporting to ensure compliance.