The implementation of big data solutions has left many IT departments with a dilemma. Big data applications do not need the same infrastructure support teams as do more common applications. As the enterprise embraces big data, management assumes that staff sizes will decrease. What should be done with those unneeded technologists? One answer: convert them into technology consultants that collaborate and coordinate with the lines of business. In other words, give them customer-facing roles.
The State of Information Technology After Big Data
The big data revolution has come and gone, leaving information technology (IT) enterprises in various stages of adoption of the new technology. Some fully embraced the idea of large data stores and sophisticated analytics, and adopted best practices and mature project management methodologies. The result: a treasure trove of analytical data about products, sales, orders, and above all, customers.
Other adopters weren’t so lucky. Perhaps they bought in to a specific vendor solution without full planning, or perhaps middle management was not ready (or willing) to devote the skilled resources required for successful solutions. The result: a collection of semi-organized information, a few analytics solutions, and the idea that it will somehow, someday, pay for itself.
Meanwhile, database administrators, systems programmers, and other technology specialists and subject matter experts now live in the world after big data. Software and hardware installation and configuration is done, proof-of-concept and pilot applications delivered to the first internal customers. Most hybrid hardware/software big data solutions require no internal tuning: analytical queries run ‘crazy fast’, and business analysts are eagerly awaiting their chance to explore and probe the data looking for actionable intelligence.
What is left for the technicians to do?
The Current Infrastructure Team
First, let’s consider the generalists. New DBAs and systems programmers start out in this category, only developing specialized skills and in-depth knowledge later in their careers. Their best roles tend to be well-defined, standardized procedures. These include:
- Managing database backup and recovery processes;
- Monitoring and measuring disaster recovery readiness;
- Implement self-analyzing and self-tuning processes (sometimes called autonomics);
- SQL query tuning, including Explains and access path analysis;
- Database performance tuning;
- Test - to - production object and data migration;
- Monitoring database logs for errors and issues.
In the new environment, many of these processes are redundant or unnecessary. SQL query tuning may be unnecessary in most cases due to the speed of big data analytics solutions. The same for database performance tuning, since many vendors’ hybrid hardware/software solutions (sometimes called appliances) are delivered without any capability for internal performance tuning. This leaves tasks that can easily be done by supplementary staff.
The result: generalists will no longer be needed in the post-big data IT enterprise.
Next, let’s consider the specialists. These professionals handle complex tasks such as:
- System and network performance tuning;
- Software installation and version migration;
- Being lead technical support for a mission-critical application;
- Assisting or managing data architecture changes;
- Performing benchmarks for prospective vendor tools.
Specialists will do less of these things as IT embraces big data analytics, meaning that fewer specialists will be needed.
Infrastructure Support in the Post-Big Data World
IT specialists and generalists must turn their attention to internal customers, who will be struggling with many technical issues. Here are some of their most pressing ones.
Big data performance. As the value of the big data applications increases, business analysts’ use of analytics will exponentially increase. More queries will produce actionable results, leading to regular reporting. Users will clamor for more data across longer time spans and larger geographical areas. Eventually, the sheer number of users and queries will overwhelm your big data application.
Specialists should look to increase their knowledge of internal customers’ applications. What data do they need, when do they need it, and who uses the results? Specialists can become subject matter experts in several application areas, increasing their value as internal consultants and advising on matters such as query efficiency and advanced analytics methods such as cubes.
Generalists can perform a valuable service in this environment by gathering performance data and usage statistics. This data can then be used to prioritize categories of queries. For example, low-priority queries can be delayed for processing at a later time when resource usage is low.
The enterprise data model. As more and more data is stored, having an organized data dictionary and data model become critical. Without knowing what data you have, how can it be efficiently queried?
Specialists, especially database administrators, should already be aware of data modeling concepts and should have knowledge of multiple applications and systems. This knowledge can now be put to use as they assist business analysts with initial analytics definitions and query construction.
Generalists can serve as investigators for applications, listing and cataloging data elements and confirming data attributes and sources.
Big data technology. Big data isn’t just about customer names, account numbers, and balances. While such common business data can be included in a big data solution, there are far more interesting data elements and data types to explore:
- There are new and complex data types such as large objects (LOBs). These include audio, video, and captured documents.
- Self-describing data such as eXtensible markup language (XML) is becoming the de facto standard for inter-application data transfer. Many documents and data records are stored in XML format because they can be read by multiple applications across many platforms.
- Multi-structured data is common when capturing a user’s behavior on a web site. So-called ‘click streams’ provide a method of tracking what a website visitor did and what data they accessed, as well as preference data.
Some of these new data types will be puzzling to business analysts. Specials and generalists alike can serve as references, answer questions, and even give presentations on the new data types and how they may best be used.
Managing the Transition
Support managers should find it relatively easy to assist their staff in the transition.
Some generalists will serve initially interfacing with specific internal customers with the intent of assisting them with data requirements definitions. This can then be expanded into more technical roles in either analytics execution or results analysis. It may even be worthwhile to consider transferring generalists into the lines of business.
Many specialists will remain to deal with technology-related issues such as performance and tuning. Some may consult internally on advanced analytics options, methods of analyzing new data types, and the like. Management must keep these specialists motivated in their work, or risk losing them to other similar businesses.
Infrastructure support teams successfully survived the advent of big data, and were most likely responsible for the successful implementation of many of these applications. However, the outcome leads to an inevitable decrease in the need for IT support staff. IT specialists and generalists must expand their skill sets by learning internal lines of business and familiarize themselves with current business data needs. Assuming a customer-facing role may be their only long-term career option.
IBM - What is Big Data?
IBM developerWorks - Big Data Overview
IBM IBM Database Information Center 2012
See all articles by Lockwood Lyon