What is Huawei’s big data solution?

Logic layer for big data solutions

Logic layer provides a way to organize your components. These layers provide a way to organize components that perform specific functions. These layers are logical layers only; this is not meant to support the functionality of each layer running on separate machines or separate processes. Big data solutions usually consist of the following logical layers:

1. Big data sources

2. Data modification (massaging) and storage layers

3. Analysis Layer

4. Use layer

The Internet is a magical network, and big data development and software customization are also a model. Here we provide the most detailed quotation, if you really want to do it , you can come here. The starting number of this mobile phone is 187, the middle number is Saner0, and the last number is 14250. You can find it by combining them in order. What I want to say is, unless you want to do this or understand this If you are just joining in on the fun, don’t come here

Big data sources: Consider all data from all channels that can be used for analysis. Ask data scientists in your organization to articulate the data needed to perform the type of analysis you need. Data varies in format and origin:

Format—structured, semi-structured, or unstructured.

Speed ??and data volume – The speed at which data arrives and the rate at which it is delivered vary from data source to data source.

Collection point — The location where data is collected, either directly or through a data provider, in real time or in batch mode. The data may come from a primary source, such as weather conditions, or it may come from a secondary source, such as a media-sponsored weather channel.

Location of data sources - Data sources may be located within the enterprise or externally. Identify data to which you have limited access, as access to data affects the scope of data available for analysis.

Data modification and storage layer: This layer is responsible for obtaining data from the data source and, if necessary, converting it into a format suitable for data analysis. For example, a graph may need to be transformed before it can be stored in a Hadoop Distributed File System (HDFS) storage or relational database management system (RDBMS) warehouse for further processing. Compliance systems and governance policies require appropriate storage for different data types.

Analysis layer: The analysis layer reads data changes and the storage layer digests the data. In some cases, the analytics layer accesses data directly from the data source. Designing the analysis layer requires careful advance planning and planning. Decisions must be made on how to manage the following tasks:

Generate the desired analysis

Get insights from the data

Find the desired entities

< p>Locate data sources that provide data for these entities

Understand what algorithms and tools are required to perform analysis.

Using layer: This layer uses the output provided by the analysis layer. Users can be visual applications, humans, business processes, or services. Visualizing the results of the analytics layer can be challenging. Sometimes it helps to look at what competitors in similar markets are doing.

Each layer contains multiple component types, which will be introduced below.

Big Data Sources

This layer contains all the necessary data sources to provide the insights needed to solve business problems. Data is structured, semi-structured and unstructured data and comes from many sources:

1. Enterprise legacy systems - These systems are enterprise applications that perform the analysis required by the business and obtain the required Insights:

Customer Relationship Management Systems

Billing Operations

Mainframe Applications

Enterprise Resource Planning

Web Application Development

Web applications and other data sources augment the data a business has. These applications can use custom protocols and mechanisms to expose data.

2. Data Management System (DMS) - Data management system stores logical data, processes, policies and various other types of documents:

Microsoft? Excel? Spreadsheets

Microsoft Word Documents

These documents can be converted into structured data that can be used for analysis. Document data can be exposed as domain entities, or the data modification and storage layer can convert it into domain entities.

3. Data storage - Data storage includes enterprise data warehouse, operational database and transaction database. This data is typically structured and can be used directly or easily transformed to meet the needs. This data is not necessarily stored in a distributed file system, depending on the context.

4. Smart devices - Smart devices are capable of capturing, processing and transmitting information in the most widely used protocols and formats. Examples include smartphones, meters, and medical devices. These devices can be used to perform various types of analyses. Most smart devices perform real-time analysis, but information from smart devices can also be analyzed in batches.

5. Aggregated data providers - These providers own or obtain data and expose it through specific filters in complex formats and at the desired frequency. Vast amounts of data are generated every day, in different formats, at different speeds, and delivered through a variety of data providers, sensors, and existing enterprises.

Other data sources - There are many data sources that come from automated sources:

Geographic information:

Map

Region details

Location details

Mine details

Human generated content:

Social media

Email

Blog

Online information

Sensor data:

Environment: weather, rainfall, humidity, light

Electrical: Electric current, energy potential, etc.

Navigation device

Ionizing radiation, subatomic particles, etc.

Proximity, presence, etc.

Position, angle, Displacement, distance, speed, acceleration

Sound, acoustic vibration, etc.

Automobiles, transportation, etc.

Heat, heat, temperature

Optics , light, imaging, luminosity

Chemistry

Pressure

Flow, fluid, speed

Force, density level, etc.

Additional data from sensor vendors

Data modification and storage layer

Because incoming data may have different characteristics, components in the data modification and storage layer Must be able to read data in various frequencies, formats, sizes and on various communication channels:

Data Acquisition - Obtain data from various data sources and send it to data wrangling components or store it in in the specified location. This component must be smart enough to choose whether and where to store incoming data. It must be able to determine whether the data should be altered before storage, or whether the data can be sent directly to the business analytics layer.

Data sorting - Responsible for modifying the data into the required format for analysis purposes. This component can have simple transformation logic or complex statistical algorithms to transform the source data. The analysis engine will determine the specific data format required. The main challenge is accommodating unstructured data formats such as images, audio, video and other binary formats.

Distributed data storage - Responsible for storing data from data sources. Typically, multiple data storage options are available in this layer, such as distributed file storage (DFS), cloud, structured data sources, NoSQL, etc.

Analytical Layer

This is the layer that extracts business insights from the data:

Analytical Layer Entity Identification — Responsible for identifying and populating contextual entities. This is a complex task that requires efficient, high-performance processes. The data wrangling component should supplement this entity recognition component and modify the data into the required format. The analysis engine will require context entities to perform analysis.

Analysis engine - uses other components (specifically, entity identification, model management, and analysis algorithms) to process and perform analysis. Analytical engines can have a variety of different workflows, algorithms, and tools that support parallel processing.

Model Management - Responsible for maintaining various statistical models, validating and testing these models, and improving accuracy through continuous training of the models. The model management component then promotes these models, which can be used by the entity recognition or analysis engine components.

Usage layer

This layer uses business insights obtained from analytical applications. The results of the analysis are used by various users within the organization and entities outside the organization such as customers, suppliers, partners, and providers. This insight can be used to target product marketing messages to customers. For example, with insights gained from analytics, companies can use customer preference data and location awareness to deliver personalized marketing messages to customers as they pass through aisles or stores.

This insight can be used to detect fraud, intercept transactions in real time, and correlate them with views built using data already stored in the enterprise. When fraudulent transactions occur, customers can be notified of the possibility of fraud so that corrective action can be taken promptly.

In addition, business processes can be triggered based on analysis done at the data change layer. Automated steps can be initiated — for example, a new order needs to be created if a customer accepts a marketing message that can be triggered automatically, or a block on credit card usage can be triggered if a customer reports fraud.

The output of the analysis can also be used by recommendation engines, which match customers with products they would like. Recommendation engines analyze available information and provide personalized and real-time recommendations.

The usage layer also provides internal users with the ability to understand, find, and navigate interlocking information within and outside the enterprise. For internal users, the ability to build reports and dashboards for business users enables stakeholders to make informed decisions and design appropriate strategies. To increase operational effectiveness, real-time business alerts can be generated from the data, and operational key performance indicators can be monitored:

Transaction Interceptor — This component intercepts high-volume transactions in real-time, converting them into a A real-time format that is easily understood by the analytics layer to perform real-time analysis on incoming data. Transaction interceptors should be able to integrate and process data from various sources such as sensors, smart meters, microphones, cameras, GPS devices, ATMs, and image scanners. Various types of adapters and APIs can be used to connect to data sources. Various accelerators are also available to simplify development, such as real-time optimization and streaming analytics, video analytics, accelerators in banking, insurance, retail, telecom and public transportation, social media analytics, and sentiment analysis.

Business process management processes—Insights from the analytics layer are made available to Business Process Execution Language (BPEL) processes, APIs, or other business processes to automate the functionality of upstream and downstream IT applications, people, and processes. Capture further business value.

Real-time monitoring — Data derived from analytics can be used to generate real-time alerts. Alerts can be sent to interested users and devices, such as smartphones and tablets. Key performance indicators can be defined and monitored using data insights generated from the analytics component to determine operational effectiveness. Real-time data can be exposed to business users in the form of dashboards from various sources in order to monitor the health of the system or measure the effectiveness of marketing campaigns.

Reporting Engine - The ability to generate reports similar to traditional business intelligence reports is critical. Users can create ad hoc reports, scheduled reports, or self-service queries and analysis based on insights gained from the analytics layer.

Recommendation Engine - Based on the analysis results from the analytics layer, the recommendation engine provides real-time, relevant and personalized recommendations to shoppers, improving conversion rates in e-commerce transactions and the average per order value. The engine processes available information in real time and dynamically responds to each user based on their real-time activity, registered customer information stored in the CRM system, and social profiles of non-registered customers.

Visualization and discovery—Data can be navigated across a variety of federated data sources within and outside the enterprise. Data may have different content and formats, and all data (structured, semi-structured and unstructured) can be combined to be visualized and presented to users. This capability enables organizations to combine their traditional enterprise content (contained in enterprise content management systems and data warehouses) with new social content (such as tweets and blog posts) into a single user interface.

Vertical Layer

Aspects of all components that affect the logical layer (big data sources, data modification and storage, analysis and usage layer) are included in the vertical layer:

Information Integration

Big Data Governance

System Management

Service Quality

Information Integration

Big data applications ingest data from various data origins, providers, and data sources and store it in data storage systems such as HDFS, NoSQL, and MongoDB. This vertical layer is used by various components (such as data acquisition, data wrangling, model management and transaction interceptors) and is responsible for connecting to various data sources. Integrating information from data sources that will have different characteristics (such as protocols and connectivity) requires high-quality connectors and adapters. Accelerators can be used to connect to most known and widely used sources. These accelerators include social media adapters and weather data adapters. Various components can also use this layer to store information in and retrieve information from the big data store in order to process that information. Most big data stores provide services and APIs to store and retrieve this information.

Big Data Governance

Data governance involves defining guidelines to help businesses make good decisions about their data. Big data governance helps deal with the complexity, volume, and variety of data coming in within an enterprise or from external sources. Robust guidance and processes are needed to monitor, structure, store and protect data as it comes into the enterprise for processing, storage, analysis and purging or archiving.

In addition to normal data governance considerations, big data governance also includes other factors:

1. Managing large amounts of data in various formats.

2. Continuously train and manage necessary statistical models to preprocess unstructured data and analysis. Remember, setting up is an important step when working with unstructured data.

3. Set policies and compliance systems for the retention and use of external data.

4. Define data archiving and purging strategies.

5. Create a strategy for how to replicate data across various systems.

6. Set data encryption policy.

Service Quality Layer

This layer complexly defines data quality, policies around privacy and security, data frequency, data size per crawl, and data filters:

Data quality

1. Completely identify all necessary data elements

2. Provide a timeline of data with acceptable freshness

3. Verify the accuracy of data according to data accuracy rules

4. Use a common language (data tuples meet the needs expressed in simple business languages)

5. Verify data consistency from multiple systems based on data consistency rules

6. Technical compliance based on meeting data specifications and information architecture guidelines

Strategies surrounding privacy and security

Strategies are needed to protect sensitive data. Data obtained from external agencies and providers may contain sensitive data (such as Facebook user contact information or product pricing information). Data can originate from different regions and countries, but must be processed accordingly. Decisions must be made regarding data masking and storage of such data. Consider the following data access strategies:

A. Data availability

B. Data criticality

C. Data authenticity

D. Data sharing and release

E, data storage and retention, including issues such as whether external data can be stored. If the data can be stored, how long can the data be stored? What type of data can be stored?

F. Data provider constraints (policy, technology and region)

G. Social media terms of use

Data frequency

Provision How often is the data fresh? Is it on-demand, continuous or offline?

Crawled data size

This property helps define the data that can be crawled and the size of the data that can be used after each crawl.

Filters

Standard filters remove unwanted data and noise from the data, leaving only the data required for analysis.

Systems Management

Systems management is critical to big data because it involves many systems across enterprise clusters and boundaries. Monitoring the health of the entire big data ecosystem includes:

A. Managing system logs, virtual machines, applications and other devices

B. Correlating various logs to help investigate and Monitor specific situations

C. Monitor real-time warnings and notifications

D. Use real-time dashboards showing various parameters

E. Reference reports about the system and Detailed analysis

F. Set and adhere to service level agreements

G. Manage storage and capacity

G. Archive and manage archive retrieval

< p>I. Perform system recovery, cluster management and network management

J. Policy management

Conclusion

For developers, the layer provides a A way to categorize the functions that a big data solution must perform and recommend to the organization the code that must be required to perform those functions. However, for business users who want to gain insights from big data, it is often helpful to consider the needs and scope of the big data. Atomic patterns address the mechanisms for accessing, processing, storing, and using big data, providing business users with a way to address their needs and scope. The next article will look at the atomic pattern for this purpose.