Anthony Ricigliano Business Blog

Modern data management is rapidly changing to accommodate the economic downturn and the growth of new technology. To reduce expenses, many IT shops are reusing legacy storage devices in addition to taking advantage of pay-as-you-go, cloud-based services. However, these distributed systems must be managed effectively to provide viable, affordable solutions to data management.

The Exciting Challenges of the New Infrastructure
This new strategy isn’t without challenges and opportunities. Today’s system designers must determine how to fully leverage the strengths of on-demand hardware to build the best data management platforms for their IT shop. At a minimum, these solutions must:
• Provide a high degree of scalability and a low level of latency by taking full advantage of parallel processing and memory capabilities.
• Provide fast and easy methods to expand and contract resources as demand changes.
• Provide exceptional up-time with minimal outages. The system should be designed to expect errors and recover accordingly without impacts to the end-user.
• Create a global experience spanning both time zones and geographical boundaries to unite business systems and partners.
• Support a variety of workload types including transactional, analytic, pull, and push.
• Increase effectiveness, efficiency, and affordability while promoting growth.

The CAP Theorem
A popular theory called the CAP Theorem states that it is not possible for a distributed storage system to be “consistent, available, and partition tolerant” at the same time. At any given point, only two of these goals are achievable. Because of this, tradeoffs must be made when distributed systems are designed and implemented.

The Eventually-Consistent Design Strategy
Some web developers are trading consistency for uptime when designing their applications. In anticipation of the need to partition the network as the system grows, they have relaxed consistency requirements in order to guarantee a higher degree of availability during and after the partitioning. This means that individual network outages could result in stale data or other minor problems instead of a nonfunctional website. These “eventually consistent platforms” were inspired by online icons like Google, Microsoft, and Amazon; many cloud-based services and open-source projects offer products that use this design structure.

A Different Approach: Enterprise Data Fabric
Although the eventually-consistent design is acceptable for many applications, it’s not a viable solution for any process where consistency is a key concern. For example, inconsistent processing in a financial system could spell disaster with multiple downstream impacts to data accuracy and consistency. There will always be some form of CAP tradeoff in a distributed system, but a new approach called EDF, or enterprise data fabric, promises to provide a better solution for core business functions.

EDF solutions use a shared-nothing approach to scalability. Partitioning uses nodes that are connected to create a seamless and expandable “fabric” that can span application, geographic, and machine boundaries. To scale the available storage space horizontally, EDF simply connects additional machine nodes. Within these data partitions, entries are composed of key/value pairs with an exceptional level of thread-based consistency.

By isolating data, related partitions can be organized and grouped into service entities. This larger unit is deployed on a single storage device where it can be accessed transactionally with complete independence from other service entities. This approach allows the EDF to create fault tolerance using a partial failure mode with fault isolation.

EDF-based systems exploit the variable nature of data by building flexible configurations that allow for consistency, partition-tolerance, and availability tradeoffs based on when and where the application workflow processes the information. When implemented correctly, EDF strategies allow businesses to reach all three CAP goals, but not at the same time or in the same place.

With the right approach, data management across a distributed system can be an effective and affordable solution for modern IT departments. Before choosing a strategy, consider the benefits and potential issues that each one brings to the table.

As high-tech electronic devices become more and more pervasive throughout everyone’s personal life, IT workers are practically demanding that their employers provide the same technology in the workplace as they use in their personal life. Instead of being tied to their desks for a traditional workday, employees prefer to be able to take their work with them as they live their lives. While this can provide a benefit to the IT shop that requires 24X7 coverage, the unattached, alternative-platform worker also presents certain challenges and risks to a company that allows the consumerization of their IT department through the use of WiFi, Smart Phones, and other consumer-based technologies.

WiFi and Data Security
One of the biggest security threats with enabling a mobile workforce is insecure WiFi access. Although an IT shop can secure their on-site WiFi network, employees that use external networks are a whole different story. If an employee decides to work on their presentation over coffee before meeting with their next customer, anything could happen. All it takes is one breach on a machine that contains sensitive data to compromise the entire company. Although the ever-increasing availability of free WiFi offers an attractive option for companies that are trying to trim expenses in every way possible, it can be an expensive savings. While security measures can be put in place on mobile devices, an aircard might be a better way to go. As long as the device is within range of a mobile phone signal, the aircard is just as secure as any other dedicated broadband connection. As additional protection, companies should install utilities on mobile devices that perform scans for viruses and malware each time a connection is made to the corporate network.

Employee-Owned Devices
When employees would prefer to use their own devices on the corporate network, it can present a multitude of challenges. In fact, many companies specifically prohibit anyone from connecting to their network with personal devices due to the security risks. The following list details some of the risks and challenges involved with allowing employees to use their own equipment for work:
• Controlling Access – An employee’s spouse, children, friends, or even a virtual stranger may be given access to their personally-owned devices. Even the most innocent use could result in exposure of sensitive data.
• Virus Infection – As employees use the internet or install programs for their own use, they may infect their electronic devices with viruses or malware that can affect the corporate system.
• Supporting Multiple Platforms – If an employee insists on using an Apple-based laptop and an iPhone when the rest of the company uses PCs and Blackberrys, support can become an issue. The company may incur additional costs if they must support alternative versions of software or provide tech support for hardware problems.

Supporting Diverse Programs
In most cases, employees will install a variety of applications for both personal and business use. From instant messaging (IM) programs to applications that remember personal information, the employee may believe that the new programs help them to do their job better and faster. All may be good until there is a problem. An IT support desk has enough problems when they are in complete control of the type of devices and programs that operate under their corporate umbrella. Consumerization of IT opens the floodgates for an anything-goes attitude. While productivity can be impacted if someone’s personal device is rendered unusable, the entire IT shop can experience an outage when an unsafe application compromises a critical system or database on the host. Rules, procedures, and systematic safety nets should be put into place to protect the corporate network, but what about the individual employee? There’s no simple solution to this problem. While it’s expensive to support a wide-open system, it’s not feasible to implement a use-at-your-own-risk rule for employee-installed programs.

Employee Productivity
In most cases, if a manager can see their employee, they can be assured that they are performing their job responsibilities to some degree. While most workers will appreciate the opportunity to work from home or catch up on a project while they’re waiting for their child’s game to start, some will take advantage of this new job benefit. By monitoring mobile usage through new programs, a company can get an idea of whether or not their employee is working or hanging out on social networking sites. As the programs are refined, it might be possible to tell if someone is actually attending a special training class or if they’re hanging out in the bar around the corner.

The consumerization of IT is an inevitable development. To maintain efficiency and security, corporate IT shops will be forced to change the way that they look at employee productivity, system support, and network security.

While the hardware required to store massive amounts of data becomes cheaper with each passing year, the resulting explosion of stored data content means that companies are forced to devise innovative new ways to meet the challenges of processing this ever-growing wealth of information. Simply storing everything forever because of the low cost of storage media sounds like a good idea to the uninformed, but massive amounts of information stored in databases and flat files can make retrieval, purging, and archiving a difficult process. Recent electronic data laws that require specific periods of retention to allow for auditing in the event of fraud or other wrongdoing only serve to complicate matters even further.

Best Practices for Data Management
Like any other data processing area, the experts in data management have compiled a list of best practices. While each item will not apply to every organization, the individual IT shop should choose the practices that work well for their particular data storage model. With the growth in data warehouses, a data management strategy is critical to the overall success of virtually every business area. Rules and code should be created to make sure that each piece of data is always accurate, that it means the same thing to everyone and every system, and that everyone has access to the most current information.

Data Stewardship
A data steward maintains the metadata registry and ensures each data element’s integrity. This would include making sure that each data element has a clear and precise definition, that the data element is not duplicated unnecessarily, and that each data element has clear and up-to-date documentation that specifies valid values, data sources, and data destinations. When the data element is no longer required, it should be immediately removed from the file structure. Data stewardship ensures consistent use of a defined field between multiple computer systems, allows for easier mapping of data, and reduces migration costs.

Model Driven Integration
By using Unified Modeling Language (UML), some IT shops are using the model-driven strategy to provide application integration solutions. This is an attempt to reduce the costs of meeting the ever-changing demands of the current business world by quickly adapting the existing software infrastructure. It attempts to separate business logic from the underlying system so that individual components can be reused without the need to change them. With this theory, data storage should be kept independent from application design and organized according to the business needs.

Active Data Model
Relational databases are the storage method of choice for most organizations that require retention of massive amounts of data with fast retrieval times. The Active Data Model “actively” refreshes the data that is seen at the client level. The client retrieves data in its current state. Next, it tracks the data created, deleted, or modified by the user, and then passes the information back to additional services for validation prior to permanent storage. Because data at the client level is always up-to-date, code designed to set up or manipulate the data can often be reduced or eliminated.

Organizational Challenges
As the amount of stored data grows, so do the organizational challenges. While no one wants to keep out-dated information, it has become increasingly necessary to do so in case of audits or legal challenges. Old data must be archived once it is no longer needed for instant retrieval, but it must still be kept somewhere that it can be accessed fairly easy when necessary. Due to inconsistent classification of data between systems or between organizations, substantial effort and cost is wasted in trying to reconcile data. In many cases, both systems will be correct, but they may be following different data management rules. When data elements are stored in multiple file systems, data errors can become a major problem. One system may be updated before another one, or certain systems may not be updated at all. When two or more computer systems are merged together due to the growth in mergers and acquisitions, it only compounds the problems if an aggressive data management strategy is not used.

Managing a Technology Project involves managing both the new system components and the programmers and analysts that create them. In many ways, managing the people involved can be a more daunting task that tracking each new piece of code or hardware item. If each person on the team is not kept up-to-date and on the same page, the process can quickly break down and mistakes will be made.

The Right Approach Can Increase the Chances of Success
While the exact approach taken may depend on the organization and the project details, there are a few methods that should always be used. Many project managers like to detail their project within software packages like Microsoft Project, or Sharepoint, but it may not be very effective without communication that goes beyond recording tasks and deadlines. The project manager should realize that while some people work well with a list, most people will need more direction. In addition, the team will probably be made up of an assortment of people with different learning styles. The material should be presented verbally and visually for the best results. At a minimum, the project manager should create a project plan, schedule a launch meeting to explain the project in detail, and then plan on weekly meetings for progress reports and problem resolution.

Improved Human Interaction Can Prevent Project Failures
If a project manager only informs, and doesn’t communicate, there is a high chance that the project will fail. They should be open to all questions, feedback, and suggestions to ensure that everyone understands both their role in the project and the potential cost of a failure. Excellent suggestions about better methods for implementing new technology can sometimes come from surprising sources. If an open-door approach is not maintained, a team member with a great idea could decide to keep it to themselves rather than risk ridicule or rejection. While it is important to go over the minute details of system changes that must be implemented, it is just as important that everyone understands the big picture. If the entire team understands that their next raise is dependent on the revenue increase that a successful project outcome will bring and that a failure could mean layoffs, they will be more likely to put in their best effort. The project manager should also make sure that they are aware of each team member’s vacation plans and personal issues that could result in an absence during a critical phase of the project. While unforeseen events will always happen during a project, asking a few questions can minimize the surprises.

Is Over-Communication Possible?
While anything is possible, it’s very hard to over-communicate during a project. Always ask for elaboration on any answer to make sure that each party understands both the question and the answer. Yes and no questions rarely give the full picture. Frequently, team members will think they have the same technical definition of a business term, but actually bring a slightly different viewpoint to the table. Neither is wrong, just from different perspectives. For example, one person may think that a payment timetable begins when they place an order, while someone in a different area may think that the clock doesn’t start ticking until the product actually arrives.

Communicate at all Levels within the Organization
Effective communication is required within and between all levels of the organization. While executives have very different perspectives than middle management and the technical staff, they will need frequent updates about each project. The executive level should expect weekly updates that let them know whether or not the project is on target to meet the deadline or if the project manager requires additional resources to achieve the ultimate goal. Middle management will also require a weekly update, but will want more details about each task and the testing results. The team will require the most information so that they know if their part is causing a delay in any other area or if they will have to wait on another component before they can complete their part. Communication should go both ways. Projects that involve inter-company partnerships require even more back and forth communication. As the project approaches its target launch dates, meetings may be escalated from weekly to daily when necessary.

Effective Communication Leads to Improved Support
When everyone feels like they are a valuable part of the project, they are more likely to provide the support required for a successful project. Each person involved from management to staff with minimal roles should be included in all communications and feel that they are providing useful input so that they engaged and buy into the importance of success. An executive who believes in the value that the project will bring to the organization will be more likely to pull a few strings when needed to add resources to a project when they are desperately needed. Along the same line, a technician who feels that their input is heard will be more likely to fit your needs into their busy schedule than if they think their ideas are only given a token amount of consideration.