DB2 management, tutorials, scripts, coding, programming and tips for database administrators
Big data software, hardware, application suites, business analytics solutions ... suddenly, it seems, IT enterprises are deluged with vendor offerings that solve problems it didn't know it had. As you dive into what will most likely be your largest IT project of the year, ensure that you have planned and budgeted for the following items that are unique to big data implementations.
It’s difficult to simply 'drop' big data applications into an existing IT infrastructure and expect to run smoothly. In addition to energy and cooling requirements for new hardware to support the new big data application, other IT areas need to prepare. The major factors that determine whether enhancements will be needed to existing applications include large data storage needs, larger data transmission capacity, and the demands these will place on existing hardware and software.
Big Data implementations are more than just lots of data. Of equal importance is the analytics software used to query the data. Analyzing business data using advanced analytics is common, especially in companies that already have an enterprise data warehouse. It is therefore only natural that your big data application must be integrated with the existing warehouse.
Big data applications do not need the same infrastructure support teams as do more common applications. As the enterprise embraces big data, management assumes that staff sizes will decrease. What should be done with those unneeded technologists? One answer: convert them into technology consultants that collaborate and coordinate with the lines of business. In other words, give them customer-facing roles.
Technical support teams usually support familiar hardware and software configurations. Specialization in particular combinations of operating systems and database management software is common, and this allows some team members to gain in-depth experience that is extremely valuable in an enterprise IT setting. How has big data changed this paradigm?
Big data applications are here to stay. The promise of this technology is the ability to quickly and easily analyze large amounts of data and derive from that analysis changes to customer-facing systems. Management believes that the analysis and subsequent changes will drive up customer satisfaction, market share and profits, hopefully at a reasonable cost.
Many big data application implementations seem to begin with an existing data warehouse, one or more new high-volume data streams, and some specialized hardware and software. The data storage issue is often accommodated by installing a proprietary hardware appliance that can store huge amounts of data while providing extremely fast data access. In these cases, do we really need to worry about database design?
Big data applications and their associated proprietary, high-performance data stores arrived on the scene a few years ago. With promises of incredibly fast queries, many IT shops implemented one or more of these combination hardware and software suites. However, few IT enterprises have implemented metrics that clearly measure the benefits of these systems. The expected monetary gains from big data applications have not yet materialized for many companies, due to inflated expectations. The solution: Measure resource usage, and use these measurements to develop quality metrics.
Load tests give the database administrator (DBA) quite a lot of valuable information and may make the difference between poor and acceptable application performance. But what about a big data environment? Are there any gotchas or traps associated with big data?
With all of the news, articles, white papers and vendor products related to Big Data, it’s easy to forget the data that drives our companies, manages sales, interacts with customers, and supports our mission-critical systems--the "other" data ... the "little" data. If we want to incorporate big data into our enterprise the crucial step is integrating in with our existing data.
Ask database administrators how they implement disaster recovery in their big data environments and you'll get two typical responses: DR plans are not necessary and backups will take up a lot of space and recovery will take too long. Despite this reasoning, a disaster recovery plan for your big data implementation may be essential for your company's future.
Big Data is a new method, a new process, a new IT paradigm for storage and retrieval of data. Newness means change, and some staff may resist change. Mitigate these problems by planning ahead for staff training, user training, and collaboration across teams.