Having a robust data architecture is the backbone of any successful organisation. As a high-level framework for managing your organisation’s data, it provides a blueprint to govern how data is collected, stored, integrated, and used across the whole organisation. With a strong data architecture, organisations can align their data management processes with their strategic objectives. This supports data-driven decisions, streamlines and optimises operations, and provides insights that ultimately drive business growth.
Data architecture is comprised of a number of essential components, that must work together to help mitigate risks and enhance scalability and flexibility. This enables organisations to adapt to the needs of a fast, and continually, evolving market. Let’s explore these data architecture components together.
Key data architecture components
Managing data effectively requires a structured, modular approach. and therefore, data architecture is made up of many different components. These components hold specific roles in the data lifecycle. They ensure business data is collected, stored, processed, secured, and used efficiently and responsibly.
Image 1.0. A diagram showing a generic architecture model comprised of key data architecture components.
1. Data sources
Data sources describe the origin points of data, i.e., where it resides in its native form. This can be in internal systems, like CRMs, Portfolio Management Systems, risk engines, or external sources such as market data that feed client interactions. When a firm is considering its different data sources, they need to be aware of:
- Data variety: Organisational data sources will be rich in variety. Therefore, they will need categorising e.g., structured vs unstructured, semi-structured, unstructured, and their formats will also differ.
- Data velocity: Consider how frequently data is generated i.e., whether it’s through real-time streaming, or periodic batch uploads.
- Data ownership: Ownership over data needs to be clarified and who will be responsible for data accuracy and maintenance.
When examining your data sources, you will also want to consider the quality of the data—its reliability and cleanliness, as well as acknowledging how easily each of your data sources can connect to your overarching data architecture.
2. Data ingestion
Data ingestion refers to the process of collecting and importing your data from various sources into a central location. Efficiency and scalability is key for this process. When it comes to data ingestion, firms should consider:
- Data validation: How valid is the data? Catching incorrect or outdated data at the point of ingestion will help prevent issues further down the line that can disrupt your whole architecture.
- Data security: Moving data around can create vulnerabilities in your sensitive information, so ensuring your data is secured using encryption or access controls is crucial.
- Latency requirements: Consider how quickly the data that is being transferred needs to be available for processing or analysis as some will experience a period of delay before the data begins to be transferred.
- Error handling: Ingestion failures can happen, so having mechanisms in place to detect, log, and recover from these are necessary to prevent further delays.
When analysing the data ingestion process, you need to ensure that the ingestion pipeline can handle increasing data volumes as your business grows, without sacrificing performance.
3. Data storage
Firms deploy numerous methods and technologies to store data, depending on its structure and how it is being used and how frequently. For example, it may be stored via relational databases or data lakes/warehouses and cloud storage. The storage solutions you choose ultimately impact the performance, scalability and accessibility of your data. When deciding on them, organisations should consider aspects like:
- Data storage format: How your data is stored depends on the way you intend to use it. Examples include columnar, row-based, or object storage layouts.
- Data lifecycle management: Policies need to be in place
- Data back-up & redundancy: Losing data can be detrimental so having replication and back-up strategies will ensure data durability and disaster recovery.
When it comes to data storage options, firms will also want to optimise their storage to be compatible with high-frequency access or large-scale analytics. Not only that, but they will need to ensure their data is stored appropriately according to regulatory requirements like GDPR—specifically regarding data location and access.
4. Data models
Data models provide a blueprint that define how data is structured, stored and related within a system. They provide a systemic way to organise, interpret and manage data. The models are instrumental in aligning business and technical requirements and are typically grounded into three levels of abstraction:
- Conceptual data model: This provides a high-level view of data focussing on the core entities and relationships between them without going into any technical details.
- Logical data model: This establishes the structure of data elements and the relationships among them.
- Physical data model: This describes how data is physically stored.
When developing data models, organisations should consider:
- Business requirements: If the data model doesn’t reflect how the business actually works, everything built on top of it—e.g., analytics, applications/systems or reporting—will be flawed.
- Data quality & governance: Inconsistent or poorly governed data structures can cause errors, increase risk and lower confidence in analytics or reporting.
- Performance & scalability: A model that works on paper but fails in performance or flexibility under real-world conditions will limit the system’s usefulness.
5. Data processing
The processing of data is integral for many functions and involves transforming your raw data into insightful information to help you make more accurate decisions. This process involves the sorting, filtering, and aggregation of data which is then used for further analysis. To benefit from data processing, firms have a couple of things to think about:
- Data processing model: Firms must choose between batch, real-time or hybrid processing models, dependent on their latency and business needs.
- Data lineage: To ensure transparency and traceability, data should be tracked to see how it’s transformed from source to output.
- Reusability: Designing modular, reusable processing components will help reduce duplication, save time, and improve maintainability.
Firms also need to ensure they are continuously monitoring their processing jobs for performance, failures, and anomalies.
6. Data governance
Data governance establishes the policies and procedures for managing data assets throughout their entire lifecycle. The aim is to ensure your data quality, integrity, compliance with regulatory requirements and help with managing any data risks. When it comes to data governance, firms must consider aspects like:
- Policy framework: Clear policies for data access, usage, classification and retention are required to keep data secure and in the best possible shape.
- Data stewardship: Having clear roles and responsibilities assigned for maintaining data quality and compliance is crucial.
- Data auditing: Ensuring all data actions are logged and traceable for accountability and regulatory compliance.
When assessing data governance policies, firms will also want to ensure they are sufficiently training any users on governance policies, tools, and best practices to ensure a clear understanding and consistent use of data.
7. Data integration
The data integration component in data architecture is all about bringing together data from multiple sources, to create a more unified and consistent view of information across an organisation.
- Data consistency: Your integrated data needs to be maintained with consistency at the forefront across systems and timeframes. Training and accountability roles can help with this.
- Latency of data: Your business requirements can help determine whether you’ll need to perform real-time integration or batch updates.
- Transformation rules: It should be defined prior to integration how data will be cleaned, standardised, and transformed.
Firms must also take care in selecting the right integration tools that align with their existing tech stack and future scalability needs, or else face the consequences of having to replace legacy systems with newer ones—ultimately creating disruption and the need for downtime.
8. Data security and privacy
Data security is crucial for compliance—and to ensure unauthorised persons don’t gain access to your business data. Firms therefore need to implement robust measures and processes to safeguard their data from falling into the wrong hands. This typically involves thinking about:
- Access controls: Role-based or attribute-based access controls can keep data safe and restricted to only those who need it.
- Encryption: Minimise the risk of breaches by protecting your data with encryption both while at rest, and while in transit.
- Threat detection and incident response: Timely response to security threats or breaches is critical and can determine the resilience and continuity of your business. Monitor for unauthorised access or potential threats with automated tools and develop and test a response plan to security incidents.
- Regulation compliance: Processes to determine how to collect, processes, and store personal information to ensure compliance with regulations such as GDPR.
9. Metadata management
Your metadata holds important information, such as the content, definitions, structure, lineage and usage of data. Together, these help you to parse the context of your data to deduce more accurate insights, as well as ensuring you are effectively using data across your organisation. There are many considerations firms will need to make in relation to their metadata management, including:
- Clear ownership structures: Designating specific individuals to take responsibility will create accountability.
- Data governance: Ensuring your metadata supports your broader governance goals like compliance and data quality.
- Metadata types: Understanding and managing technical metadata (schema, data types, source systems), business metadata (definitions, business rules) and operational metadata (data usage, performance metrics) ensures a complete metadata ecosystem.
Firms will also want to ensure their metadata is high-quality and consistent. Standardisation, validation, and version control processes can help with this. Similarly, to comply with regulations, metadata will need to be secure and support compliance with GDRP or other data regulations.
With your data architecture being made up of so many different pieces, it’s crucial you are maintaining and enhancing each of these to ensure a truly robust data architecture that feeds into your bigger enterprise architecture.
Here at Davies, our Asset and Wealth Management practice provides data architecture design, analytics and management. Want to find out more about what we can do for you? Get in touch today!
