Data quality / reliability, Analytics / BI
Writing Code (SQL, Python, etc), Documentation / data discovery
Data modeling / transformation, Analytics / BI
As a consultant - entirely depends on the client
Writing Code (SQL, Python, etc), Architecture design
Semantics / ontologies / knowledge graphs
Data modeling / transformation, Data quality / reliability
Cloud data warehouse (Snowflake, BigQuery, Redshift, etc.)
Lack of clear ownership, Pressure to “move fast”
Data Ownership is the foundation of everything
Ingestion / pipelines, Infrastructure / platform work
Lakehouse (Databricks, Iceberg/Hudi/Delta + compute engine)
Data modeling / transformation, Data quality / reliability, Infrastructure / platform work, ML / AI
Lack of clear ownership, Tools don’t support good modeling
Hard to maintain over time, Pressure to “move fast”
Data modeling / transformation, Data quality / reliability, Analytics / BI
Writing Code (SQL, Python, etc), Governance / quality checks
Lack of clear ownership, Hard to maintain over time
It is very diverse and not generic at all
Writing Code (SQL, Python, etc), Documentation / data discovery, Data modeling
Data modeling / transformation, Fighting fires
Documentation / data discovery, Architecture design
Hard to maintain over time, Tools don’t support good modeling, Pressure to “move fast”, AI tools produce inconsistent schemas
Event-driven architecture feeding into a centralized warehouse
Data modeling is not the same thing as a pipeline
Data quality / reliability, Infrastructure / platform work
Ingestion / pipelines, Data modeling / transformation
Documentation / data discovery, Data modeling
Tools don’t support good modeling, Pressure to “move fast”
Cloud-native (GCP Cloud Composer, AWS MWAA, etc.)
Writing Code (SQL, Python, etc), Pipeline debugging
Data is like water, it flows. But if not controlled it will pop up/show unexpectedly (because business is also like water, it always finds its way) multiplying operational costs and risks.
Data quality / reliability, ML / AI, Fighting fires
Lack of clear ownership, Hard to maintain over time, Tools don’t support good modeling, Pressure to “move fast”, AI tools produce inconsistent schemas
1. DE for AI requires a different way of thinking (and then building). 2. DEs can no longer work in isolation. They need to work with their upstream and downstream partners. Great DEs always knew this, but many still operate as if they are some engineer who doesn’t need to care about what happens either side of their work.
that thing you said about data modeling as a program, not a project
Pressure to “move fast”, AI tools produce inconsistent schemas
Data modeling / transformation, Infrastructure / platform work
Lack of clear ownership, Hard to maintain over time, Pressure to “move fast”
Writing Code (SQL, Python, etc), Data modeling
Data engineering will likely be the mechanism to ensure organizations can produce useful results with AI. AI will change data engineering but won't eliminate it.
Main org is small and research-heavy. We use simple object storage for public datasets.
It's a much broader discipline than most people think.
Data quality / reliability, Analytics / BI, Fighting fires
I don't know if I can speak to the wider industry, but, within my organization, they don't understand that it is a separate important part of an overarching architecture.
Writing Code (SQL, Python, etc), I don't find AI helpful
Writing Code (SQL, Python, etc), Pipeline debugging, Documentation / data discovery
More about supporting decisions and making direct impact, less discussion about tools
Lack of clear ownership, Hard to maintain over time, Tools don’t support good modeling, Pressure to “move fast”
That good engineers need to understand the business in addition to the tech they use.
Data quality / reliability, Infrastructure / platform work, Fighting fires
How to define with leadership and compile a centralized source of truth for data (system of record)
Writing Code (SQL, Python, etc), Documentation / data discovery, Architecture design
Ingestion / pipelines, Data modeling / transformation, ML / AI
Writing Code (SQL, Python, etc), Pipeline debugging, Documentation / data discovery, Data modeling, Architecture design
Ingestion / pipelines, Data quality / reliability
We are the backbone for good data used by ML and AI.
Hard to maintain over time, Tools don’t support good modeling, AI tools produce inconsistent schemas
Seems like every tool in LLM/AI space requires a K8s cluster with half the Apache foundation to do something a basic CMS could do. Most of the problems are not big enough for that, most companies are not going to drop > $300/m on infra just to manage their prompts.
Data modeling / transformation, Data quality / reliability, ML / AI
Writing Code (SQL, Python, etc), Pipeline debugging, Architecture design
Data quality / reliability, Fighting fires
Lack of clear ownership, Tools don’t support good modeling, Pressure to “move fast”
Writing Code (SQL, Python, etc), Pipeline debugging, Documentation / data discovery, Data modeling, Governance / quality checks
Lack of clear ownership, Hard to maintain over time, Tools don’t support good modeling
The many d different roles engineers play
Successful teams vs unsuccessful teams are often separated by operational discipline.
Data modeling / transformation, Data quality / reliability, Fighting fires
Ingestion / pipelines, Data modeling / transformation, Data quality / reliability
Writing Code (SQL, Python, etc), Pipeline debugging, Documentation / data discovery, Governance / quality checks
Ingestion / pipelines, Data quality / reliability, Analytics / BI, Fighting fires
Ingestion / pipelines, Infrastructure / platform work, Fighting fires
Writing Code (SQL, Python, etc), Pipeline debugging, Documentation / data discovery, Architecture design
SaaS vendors won't solve your problems - it matters very little whether you use Snowflake, DataBricks, AWS or GCP.
Pipeline debugging, Documentation / data discovery
Infrastructure / platform work, Fighting fires
Ingestion / pipelines, Data modeling / transformation, Infrastructure / platform work, Analytics / BI
Writing Code (SQL, Python, etc), Pipeline debugging, Documentation / data discovery, Data modeling
Data modeling / transformation, Analytics / BI, Fighting fires
Ingestion / pipelines, Data modeling / transformation, Data quality / reliability, Infrastructure / platform work
Hard to maintain over time, None / modeling is going well
Data models, and engineering are the foundation of ML and AI. Recent RIFs have both heavily targeted DE teams, while asking for "better" ML and "more" AI models
Writing Code (SQL, Python, etc), Documentation / data discovery, Governance / quality checks
Probably just a better understanding of the basics to enable them to fathom "what is possible" and the requirements to achieving that.
Writing Code (SQL, Python, etc), Documentation / data discovery, Data modeling, Governance / quality checks
Lack of clear ownership, AI tools produce inconsistent schemas, None / modeling is going well
How multi-faceted and advanced it is. It is hard to do good engineering.
Ingestion / pipelines, Data modeling / transformation, Data quality / reliability, Infrastructure / platform work, Analytics / BI, Fighting fires
Lack of clear ownership, Tools don’t support good modeling, Pressure to “move fast”, AI tools produce inconsistent schemas
Hard to maintain over time, Tools don’t support good modeling
It’s more than just moving data from A to B in hopes it becomes useful as a result
Data modeling / transformation, Infrastructure / platform work, Analytics / BI
Documentation / data discovery, Data modeling, Architecture design
what we are actually doing because it is not visible
Homemade; custom solution to solve dbt's shortcomings
Infrastructure / platform work, Analytics / BI
Ingestion / pipelines, Infrastructure / platform work, Analytics / BI
Writing Code (SQL, Python, etc), Pipeline debugging, Documentation / data discovery, Data modeling, Architecture design, Governance / quality checks
Hard to maintain over time, Tools don’t support good modeling, Pressure to “move fast”, None / modeling is going well
Semantics, Ontologies, and Knowledge Graphs, and the design exercises they require - "always be architecting"
Better understanding of how DE/AE positively impact the business and revenue/operations
Ingestion / pipelines, Data modeling / transformation, Data quality / reliability, Infrastructure / platform work, Analytics / BI
Good data engineering is a prerequisite for adopting AI/ML.
Ingestion / pipelines, Data modeling / transformation, Data quality / reliability, Analytics / BI, Fighting fires
Writing Code (SQL, Python, etc), Pipeline debugging, Documentation / data discovery, Architecture design, Governance / quality checks
It is not an afterthought endeavour. It is the backbone of any AI initiative.
In data engineering, a good design is one that adapts well to change
An executive said to me: It doesn't matter how good you become at the bitch work, you're still the bitch. Eye opening. (Was said with politeness and discussing managing different value systems)
Pipeline debugging, I don't find AI helpful
Lack of clear ownership, AI tools produce inconsistent schemas
Architecture design, Governance / quality checks
That demands a constant effort to keep the data flow working because data changes all the time.
Product database, analytics and integrations
Growing convert into the power of data contracts and to some extent data products
Need to work with Business folks in an integrated fashion and make them more curious about technical aspects that can lead to more Systemic thinking and not be reactive and merely demand reports on an adhoc basis(Ex. Slowly changing dimension; Data retention life cycle)
Native tooling (i.e. dbt Cloud, Fivetran)
Ingestion / pipelines, Data modeling / transformation, Analytics / BI
Ingestion / pipelines, Data quality / reliability, Analytics / BI
Data modeling and how create a meaningful depiction of business processes in a data model
How versatile the role is. That it's really a specialized Software Engineering role, more than a Data Analyst role.
That tools aren't the problem; people and process are.
Ingestion / pipelines, Data modeling / transformation, Data quality / reliability, Analytics / BI
Writing Code (SQL, Python, etc), Documentation / data discovery, Data modeling, Architecture design
Data quality / reliability, Infrastructure / platform work, Analytics / BI, Fighting fires
Writing Code (SQL, Python, etc), Architecture design, Governance / quality checks
Ingestion / pipelines, Data modeling / transformation, Data quality / reliability, Infrastructure / platform work, Fighting fires
Without data modeling we are ser to fail
The difference between that and data science
It takes time to get right and doesn’t happen overnight. And if stakeholders pressure for immediate results, it’s Garbage In Garbage Out
Ingestion / pipelines, Data modeling / transformation, Infrastructure / platform work
Data modeling / transformation, Data quality / reliability, Infrastructure / platform work, Analytics / BI
Requirements/expectations must be clearly defined and backed-up with the appropriate resources.
You can still do/be a data engineer without big tech tools and big tech buzzwords. If you can build a system/process that serves some end user a data product that is useful, insightful, or makes life easier for the customer or stakeholder then congrats! you are doing the thing.
Data modeling, Governance / quality checks
That good quality data/analytics engineering takes time! Often I'm pressured to rush stuff and their is a consequent drop off in quality/maintainability.
The thing that works for current situation
Good data for testing will improve application quality
Data Engineers are not just tech resources they need to be very tightly integrated to business processes
Conceptual and logical data modeling are still exceptionally productive
Data modeling / transformation, Infrastructure / platform work, ML / AI
First, there is no substitute for deep process knowledge. Second, not everyone has "Big Data" problems. In fact, the deeper your process knowledge becomes, the less data you need to make high impact business decisions.
Documentation / data discovery, Governance / quality checks
It’s as vast as the software engineering world except everything has a data twist, making it that much harder.
how difficult their requests can be. I know this is hopeless.
Writing Code (SQL, Python, etc), Documentation / data discovery, Architecture design, Governance / quality checks
Data Engineering is not Software Engineering
Operational system designer, which doesnt seem to be a real job.
That data engineering is a very young discipline and its time to throw away the 25 year old prototypes and to start thinking like an engineer. With data, the sequel.
Hard to maintain over time, Tools don’t support good modeling, Pressure to “move fast”
That different use cases of data mean different models and tools, and that data can change over time over it's lifecycle, and that is fine. Not every data set will go through the same tools (ie a source of truth is not a system, it's a determined state in a lifecycle, which might be implemented in a different tool depending on the data) ie sometimes the correct truth source is Kafka, sometimes it's the warehouse.
It’s essential for data insights from AI to BI
You need an "any other comments" question: suggesting the "Which Industry" should be multi-choice?
Layered design rather than marketing driven design
It’s an investment that will pay off if you do it well from inception and continue to do it well, without rushing it.
Combo of home-grown scheduler / Azure Data Factory
Data Engineering isn't about the tools you are using, but seems like most jobs require practice in tools.
Software engineers created all the problems data engineers have to deal with.
Ingestion / pipelines, Data modeling / transformation, Infrastructure / platform work, Analytics / BI, Fighting fires
Orgs suck at data in general and people dont seem to know xxxx about data in general (exceptions do exist).
Ingestion / pipelines, Data modeling / transformation, Data quality / reliability, Analytics / BI, ML / AI
Poorly designed source applications and databases, combined with a lack of clear data ownership
Ingestion / pipelines, Data modeling / transformation, Infrastructure / platform work, Fighting fires
I think your focus on data modeling is exactly right as I see a lack of skills and focus on data modeling to be one of the TWO big issues (quality being the other .. not data quality but quality in the Robert Pirsig sense)
It’s not just software engineering 10 years ago but it has its own challenges that software engineers don’t need to worry about.
That it takes time to build data models that are fit for business, and these don’t just happen overnight or within a “sprint”.
Raw layer - event driven/canonical, datamart level kimball/add-hoc
modern dimensional warehouse / analytics engineering
Business impact and can-do attitude matter a lot more than many junior people realize. Data is a nexus in most organizations and DEs / Analysts / DSs / Etc. often have highly strategic "crow's nest" vantage points yet fail to capitalize on this opportunity.
Prioritising value of Data first rather than Data modernization
It's not a project. It's a program that needs to be treated as a capital investment.
Lack of leadership and talent, tech debt
That is is where the magic (data) is created
Iceberg on S3 w/ Glue & Lake Formation, hand-rolled data platform.
DE is a SWE discipline and not enforcing good discipline gets you in trouble.
There is no one size all fit approach, and a lot technical debt to get to innovation
Ingestion / pipelines, Data quality / reliability, Infrastructure / platform work, Analytics / BI, Fighting fires
Large enterprises are bound by their legacy applications which constrain data engineering initiatives. This is further compounded by the ways of working (off shoring), making it very difficult to adopt industry best practices, largely because decisions are often framed with how to make hybrid architectures work ( old and new platforms). Both states often need to coexist for longer timeframes than we would like to admit
Ingestion / pipelines, Data modeling / transformation, Data quality / reliability, Fighting fires
Data is simply the backend for the backend
The way we spend our working days is the way we spend our working lives
Mack of data modeling expertise will turn your powerful data infra into a giant mess impossible to maintain, with sky-high change costs.
It is complex, touches multiple disciplines
Data Vault (Data Warehouse) and Kimball (Data Marts)
That data engineers (or IT people) don't 'own' data and that data (management) is not IT.
How much time and effort it takes. The complexity behind it.
The skills involved, software engineering requirements for tooling and platforms.
domain knowleedge is key and it must be mandatory part of data projects
Ingestion / pipelines, Analytics / BI, Fighting fires
That we shouldn't do the job of 3 roles/titles while we should maintain the pipelines and the data quality
Data quality / reliability, Infrastructure / platform work, ML / AI
That in practice it is never about how many tools one knows and how shiny the architecture is, but knowing the business and being able to communicate with the stakeholders on their own language (sometimes this literaly means using their vocabulary). This was the main pillar in dimensional modeling and except for the cloud, I don't really see the industry has moved much since it was introduced in 1996.
Data quality can't be compronised even during urgency
Writing SQL is easy. Writing the correct SQL is hard.
I wish that data engineers and business stakeholders understand that both need to Walk on other shoes and give each other as much information as possible.
AI tools produce inconsistent schemas, None / modeling is going well
Data governance isn’t only data engineers responsibility
The "shift left" mindset that drives TDD in systems development applies just as much if not more in data; no amount of data engineering magic will fix poor data quality / definition / constraints at the sources.
Lack of clear ownership, Pressure to “move fast”, AI tools produce inconsistent schemas
Data modeling / transformation, Data quality / reliability, Infrastructure / platform work
Writing Code (SQL, Python, etc), Documentation / data discovery, Data modeling, Architecture design, Governance / quality checks
how to elevate data literacy of other employees
how foundational and important it is for the rest. give proper visibility to DE
Planning to do more than we end up having time for. So IDK, maybe talent?
It’s better when it doesn’t break, even if it’s less visible than fixing failures
i wish it gets same exposure as software engineering, bcoz if you look at it carefully the data we provide is the one that runs or breaks the company.
It's much more important than model development.
We're not order takers, but rather developers who should be incentivized to build great products people love.
Primary goal is to help business end-user, not technology adoption.
Ingestion / pipelines, Data modeling / transformation, Infrastructure / platform work, Analytics / BI, ML / AI
the importance of good data because of the pain it causes.. understanding the dataa is the proff that you did job correctly because its the trail..
Data has to be driven from a shared business roadmap, have an isolated view creates isolated data
Ingestion / pipelines, Data modeling / transformation, Data quality / reliability, ML / AI
That AI / LLM / Agents do not solve magically the need for data / software engineering best practices
Writing Code (SQL, Python, etc), Data modeling, Architecture design
How critical it is for anything to do with data.
Adopt a more functional / declarative style approach in the tools being built
Better coaching and knowledge transfer from the seniors to the newbies is very important.
Build a data team does not magically tell the industry what best to do/ which data to produce or collect.
Clear understanding of business and technical requirements.
Ingestion / pipelines, Data modeling / transformation, Analytics / BI, ML / AI
S3 (parquet, csv, whatever) + Amazon Athena
How valuable DE is and how - in the age of information - it can basically make or break a company. Arguably most important/impactful org in a company
Ingestion / pipelines, Data quality / reliability, Fighting fires
vendors currently build "own" semantics layer into their plattforms. They at least should provide an interface to ingest and maintain "own Knowledge" (own KG, Ontologies) into their proprieatary semantic tier.
If the underlying data/systems are poor then no engineering effort can 100% fix that. Invest in the foundations.
Without engineering, all of the data people clamor over would be utterly unusable and unreliable.
Good + Fast → Not Cheap (it'll cost a lot) Fast + Cheap → Not Good (quality suffers) Good + Cheap → Not Fast (it'll take a long time)
Ingestion / pipelines, Infrastructure / platform work, ML / AI, Fighting fires
Writing Code (SQL, Python, etc), Pipeline debugging, Governance / quality checks
A holistic approach works best, you need methodology, standards, tooling, grit, cash...
The importance of data for AI to function well.
Ingestion / pipelines, Data modeling / transformation, Data quality / reliability, Infrastructure / platform work, Analytics / BI, ML / AI, Fighting fires
I can do data analytics, but my job is to be the one who builds and maintains the platform for those who do data analytics.
Data quality is critical building block.
Data modeling / transformation, Data quality / reliability, Analytics / BI, Fighting fires
Ingestion / pipelines, Data modeling / transformation, Fighting fires
Proper Data Modeling is very important and can not be skipped
The importance of having a source of truth
People don't understand the data ops side of the job. Most people expect that we deploy prototypes to prod by hand basically. Although we use prototyping strategically, the job of "productionalizing" (wishing we had a better word) is under-estimated.
There's a design/engineering aspect to it, not just rushed implementation.
Data is not static, data models evolve hence it must be a core principle not a feature
Ingestion / pipelines, Data quality / reliability, Infrastructure / platform work, Fighting fires
Lack of clear ownership, Hard to maintain over time, Pressure to “move fast”, AI tools produce inconsistent schemas
Every software engineer can be a data engineer, but not every data engineer is a software engineer, and that's perfectly fine.
The relationship w architecture and the opportunity to think about ROI in terms of realizing business value w the most efficient design—> execution.
If the glue is cheap, the product will fall apart eventually
You won’t get any answers to your business questions if it’s not a first class citizen in your planning (and software engineers should ideally be more educated on this topic when building schemas and eventing / and whenever they change them)
Writing Code (SQL, Python, etc), Pipeline debugging, Data modeling, Governance / quality checks
It's as much a business function as a technical one. And it's such a competitive advantage.
Lack of clear ownership, None / modeling is going well
Architecture design, I don't find AI helpful
Nonprofit Director trying to learn data engineering
Im not qualified enough yet to answer. But my current thoughts is not to overload the Data Engineer with tasks that are more for Data Science and Analytics or other Data roles.
Hard to maintain over time, Pressure to “move fast”, AI tools produce inconsistent schemas
AI needs a good Data Engineering pipeline to work
Data can't be an afterthought, these things take time to do! We have a cauldron, not a magic wand!
Ingestion / pipelines, Data modeling / transformation, Analytics / BI, ML / AI, Fighting fires
That it is hard to build good data platforms and pipelines and data engineers can't fix data that doesn't exist / isn't captured
Data modeling. To Software Engineers, ingestion and pipelines make sense. Data modeling and proper data integration to them is a black box. Even product people and some in the business do not know nor care to know how time consuming it can be to model data correctly for humans and machines.
Mix of Inmon, Kimball, OBT, semantic models
There is much more to Semantic layer than BI like dimensions, measures, metrics etc
It's the thing that matters more than data science or analytics
Ingestion / pipelines, Data modeling / transformation, Data quality / reliability, Infrastructure / platform work, Analytics / BI, ML / AI
The importance of data engineering teams in an organization and how to manage them.
Huge numbers of externals building stuff with little oversight and zero quality control
Ingestion / pipelines, Data modeling / transformation, Analytics / BI, Fighting fires
The best tech in the world will always be let down by poor business engagement & poor requirements!
Good practices learned from software engineering matter for industrialisation. DE are not just data wizards but developers.
Data modeling / transformation, Infrastructure / platform work, Analytics / BI, ML / AI, Fighting fires
It's complicated to build data systems since you in principle have to take the full history of the company's systems into consideration for everything you build, while the rest of organization's developers are mainly concerned about the present state.
Data engineering is as much a technical role as it is an understanding the business role. A good data engineer will find that their business acumen will take them as far or farther than their technical acumen can. The industry should understand and embrace this.
We focus mostly on agents and hence use alternative orchestration approaches and tools
AI/Agent enabled data engineering is where I see the greatest pain point and the least amount of tooling support. I really wish the industry had the following: (a) "Cursor for Data engineering" which is not just pipeline building which they do already, but linking engineering work to business knowledge, semantic context etc), (b) Understanding the business - better knowledge management, not the mechanics (well understood) but capturing tacit knowledge or knowledge embedded in non-database domains... they key is not just AI-enablement, but agent-enablement where I have an agent "sidekick" that brings business knoweldge immediately or captures it at the time of need.
How important to properly plan data governance and quality before using it downstream
built in house: https://docs.napkin.run/getting-started/
Acknowledging it exists would be a start
Databricks Jobs via Databricks Asset Bundles
Pipeline debugging, Documentation / data discovery, Architecture design
Often it's assumed that it's "enough" if lots of data is available - there is too little talk or thought about the usefulness of the data and if it's available in formats that ML or analysts can work with without friction. The work to actually usable and used data is underestimated heavily.
Ingestion / pipelines, Data modeling / transformation, Infrastructure / platform work, ML / AI, Fighting fires
Clean data is a hard earned battle. It works because someone cares that it works.
It's potential business value outside of reporting/analytics
That data engineering, done correctly, is every bit as rigorous as software engineering.
Lack of clear ownership, Tools don’t support good modeling, AI tools produce inconsistent schemas
Documentation / data discovery, I don't find AI helpful
That the whole point of data engineering is to serve the business
A few years ago in an airport I saw an ad for a consulting firm that said: "Data is the new oil". But if someone dumped 10,000 barrels of oil on my doorstep what exactly could I do with it? Not of much value unless I can find someone with an oil refinery. If data is the new oil then then the value isn't the amount of raw data or where it came from. Data teams need to find their refinery. The bad news is refineries are complex and expensive. The good news is that those who have a refinery create/control immense value.
That it takes time to build a good foundation, and that is a necessary exercise to go through
Without DE, there is no driving value from data.
Testing data is much harder than software
we are a consultancy, depends on the clients
Data modelling and integration is a political and business problem
Data silos, Relying on older tools, Lack of a data quality framework
Data Engineering is not just about moving data from point A to B, it's about creating a reliable and scalable foundation that won't happen without incorporating software engineering best practices.
One Big Table + Kimball-style dimensional modeling
Stop using lingo like 'poor data quality' instead detail on what is wrong
How much it differs from classical Software Engineering and Data Science/BI
How brittle data ingestion can be and ways to make it more resilient.
Data modeling / transformation, Infrastructure / platform work, Analytics / BI, Fighting fires
Metadata modeling is required for the entity resolution workflows
Documentation - not having perm links in Zendesk
There are no simple solutions, especially in regulated environments.
I wish people understand that data engineering is not all about tools. It about understanding the core concepts and fundamentals.
Tools don’t support good modeling, Pressure to “move fast”, AI tools produce inconsistent schemas
Turnkey no/low code solutions are a pipe dream.
Data quality can't be fixed by one person, team or department no matter how hard you shout.
How challenging enterprise level prioritisation of data engineering is. How to combat the assumption that data work is ‘one and done’ rather than an ongoing capability
It takes long term investment in developing talent and mentorship to get right
Ingestion / pipelines, Data quality / reliability, Infrastructure / platform work, Analytics / BI
The code writing is the easiest part of the process by far
That Data Modelling isnt just something that the data engineers do, it has to be collaborative and owned in part by the business
How enteties are connected for his/her business
Pipeline debugging, Governance / quality checks
Building "fast" very often introduces more problems than it solves. Staging and testing build trust which drive adoption ...
The invisible progress of data engineering is critical to do any of the visible magic of analytics and AI. committing to the discipline will seem like overkill in the short term, but will be money well spent in the medium term.
What the platform e.g. Databricks or MS Fabric offers out of the box
Org very "structured data" oriented, does not know a lot about vector databases, knowledge mining in docs, enrichment (e.g. transcribing images in context of doc) etc.
How a reasonable Data Mesh approach works - beyond theory and in the age of AI. Now, AI is a consumer of datasets and may query tables, datasets, ontologies, ... How to design those data products in Data Mesh, how do we prevent excessive queries etc.
It isn't given particularly understood well or executed effectively
Writing Code (SQL, Python, etc), Data modeling, Governance / quality checks
That it's just the first step in value creation; the ante into the poker game if you will.
Data quality / reliability, Analytics / BI, ML / AI, Fighting fires
It's not just the data but the context of the data
Good data engineering is not only about tool knowledge, but also about modeling and understanding the business pain points.
It’s not a commodity skill. Pressure to outsource
The pain points that dictated why the specialization exists.
You can't as cowboy with data as with software
It takes significantly longer when understanding the business is our problem and not our stakeholder’s problem
Data modeling / transformation, Data quality / reliability, Infrastructure / platform work, Analytics / BI, ML / AI
Lakehouse DWH + S3 lake + other DB stores (grapgh/docment)
Primary (Glue/Spark) w EMR (spark) for large distributed worklaods) + others
Data is a team sport, it requires sponsorship and alignment across both business and technical users to be fully successful.
The range of what a data engineer does. It's a lot more than just pipeline building with python. There's all the hats one wears from building dashboards on BI tools, implementing AI where possible, maintenance of data platform, cost awareness, etc.
That it is a real specialization that can go toe to toe with any traditional software engineering. It’s the same mindset but it gets a bad wrap cause people don’t think “it’s hard to do”.
Data Engineering is not as simple as says
Shifting left is a game changer. Not everything has to be loaded into a data warehouse. Async pub/sub is a great way to persist, share and reuse data.
Retroactively remediating data models is difficult and cannot be solved by engineering teams alone
There are no shortcuts to success - AI is not a magic bullet.
Centralized warehouse migrating to Lakehouse after being acquired
Have a combination of legacy/tech debt and difficulties with leadership direction
Data engineering is a strategic partner in achieving product outcomes, not just the exhaust of the traditional product triad (design, engineering, and PM)
that we aren't going away - but we need to expand our capabilities and partnerships with the business, and adapt to the realities of AI-driven development.
Going from Kimball to mixed with Lakehouse implementation
Personally very interested in the eval data lifecycle underpinning AI-powered apps. None of the observability apps seem to get this right yet.
I lead AI/ML and I wish the ratio of data engineers to data scientist and ai engineers was higher. They’re the lifeblood of our team. Especially those search engineer hybrids.
clickhouse and postgres selfhosted in AWS
dynamic linking between architecture and code
Consultant working on migrations and data quality
Determining the value of the data and its usage so ROI of Data Engineering can be estimated
Technical aspects are later. Requirements and Business analysis is critical. Current State of the data flow and data consumption is critical before solutioning and technical aspects.
We basically have cron based jobs but transformed models are orchestrated with a data transformation framework running prod jobs with a cloud run job on a schedule
kimball at the core, OBT for reporting when needed
Pursue simplicity than complexity in your system, modeling, or pipeline
How much we need business context to be effective or we just become too risk averse which slows us down and leads to an unhealthy spiral!
Software engineering and data engineering are starting to merge. Software engineers need to understand data engineering tools and skills and vice versa. Leadership and hiring managers need to understand this.
Data ownership is politics. Data modeling in theory is too ideal. Data engineers are solving problems the upstream created, not really technical problems, political problems in a technical way.
Intentional data modeling patterns is the flywheel for data engineering, BI, and AI.
Industry should understand that data engineering isn't a one-time job, it a end-to-end practice to become data-driven then ai-driven.
That source system quality directly effects everything downstream
It's more about maturity and ability to change than fancies tools.
Get data analysts to contribute to data tables
Technology is not an end. Bring all together stakeholders around the table to grow the business is the priority.
Don’t ditch quality as it gives an impression of slowing things down - if it does then you need to find competent people (lack of quality is very often due to incompetence)
With data engineering, competitive advantage compounds through faster, safer adaptation and reliable decisions; without it, any advantage is temporary, fragile, and eventually self-sabotaged by bad data.
Data engineering suffers due to bad business modelling and low business integration
Breaking Siloes is the core issue and the core solution, for architecture and data quality.
Ingestion / pipelines, Data quality / reliability, Infrastructure / platform work
You need to have solid data foundations - the basics that have been established for decades now: data quality, governance, appropriate modelling, consistent pipeline development approaches, solid operational structures and engineering practices in places (security, CI/CD, monitoring, alerting, finops, etc) - BEFORE deciding that you want to throw the AI bus into anything and everything, and expecting the LLM genie to get all your sheeeet gleaming up from nowhere just with the flick of a magic AI wand.
Various for different purposes. Domo internal, dbt orchestration, n8n, make
It’s funner than what it looks like or sounds like.
I would hope we would have a consistent architecture by now, but it pain me to say, there isn't. In my circle of influence I try to adopt Data Mesh/ federated ownership to tackle this beast en show a good practice.
How to educate the next generation of bachelors to become Responsible data professionals, anything that helps with that, because foundational knowledge for this group is hard to get a hold of.
It is impossible to be compliant to EU regulation without a data engineering lifecycle. Any vendor that professes it can be solved with AI is a dreamer. Data processing = communication and understanding, otherwise it is just moving data. I could write an essay about it, and maybe in the future I will, but I think your manifesto on data modeling says enough for now.
I wish people would slow down with AI it stresses me. I just got a handle of how to code and build pipelines now it seems everything is put to work by AI
Data pipelines are brittle, but they can be hardened with the right processes, tools, and people. The last part being extremely important. We need people who know the patterns of what works and what fails and why. I think AI struggles with this and it's amazing when you have a data engineer who has worked with many different data pipelines and can assess a situation and provide solutions that work and that are documented and they make their work approachable. They are not trying to be a hero, they are making their work reliable and approachable.
More data does not necessarily mean better results/quality/etc.
We wear multiple hats, please be kind to us and pay us what we actually deserve.
Governance / quality checks, I don't find AI helpful
The importance of a well defined use case.
Team dysfunction shapes pipelines more than technology
To get high quality results from data/automation, somebody has to manage the quality of your data. If an enterprise program for data quality doesn't exist you'll end up with the wild wild West some teams will do it well others won't.
Writing Code (SQL, Python, etc), Pipeline debugging, Data modeling
It should be part of strategic conversations
I think they understand, but a key improvement to products would be standards and better integration. It's often painful to put the tools together that you need for your case when they don't talk to each other natively.
Data Engineer doesn't mean Data Engineer+Data Architect+Infrastructure Engineer+Devops+Security& Governance Specialist
Data is everyone’s responsibility and there’s a direct dependency from strategy and operations on service owners and engineering. Data isn’t just an afterthought and analytics isn’t a series of one of report requests
heterogenous tech debt cannot be replaced by a single tool/software solution
Not sure, Databricks version of pipeline automation- we may not yet have dependencies
It’s not magic and that good things come to those who wait
Software and data are not the same, similar but different challenges.
No amount of technology or process improvements will make up for a lack of aligned definitions of key entities and metrics.
I wish the industry/its leaders, truly understood how important data and data engineering are, not just repeating buzzwords(big data, data driven...), but taking real action and making real investments.
Data modeling / transformation, Infrastructure / platform work, Fighting fires
It should not ditch decades of software engineering best practices
Clear understanding of roles and boundaries
Systems behind pipelines, warehouses and dashboards. Understand why a particular system works the way it does and tradeoffs
That it requires just as much time and rigor as software engineering
building robust data pipelines needs time and planning. Therefore the business context needs to be explained. It is not just „open the csv file and transfer it to database.“ This is not how data helps the collegues.
Talent, Lack of leadership direction, technical debt
Ingestion / pipelines, Analytics / BI, ML / AI
Most data-heavy companies are fundamentally knowledge companies. Outsourcing (internationally or to AI) production and interconnection of knowledge is counterproductive.
That to build a good data model, I need a lot of time from business users to ask them how their world works.
Data modeling / transformation, ML / AI, Fighting fires
Depends on the client, we are a management consulting firm
The majority of data issues start at the source, until there is accountability in the teams generating the data there will always be a need for highly skilled data engineers to create customized transformation pipelines.
Global vs local initiatives, sheer size of teams, aligning best practIces etc etc. List is long…
ALL of the above? I handle multiple teams, plus global initiatives, so hard to frame some of my answers here.
Data in movement vs data at rest, that we have to handle that duality
I wish the wider industry understood the importance of data quality and the challenges we face in ensuring accurate and reliable data for analysis. By the way, we've been using surveyswap.io to get respondents. You fill in surveys and get responses back. Perhaps it's also helpful to you. Anyway, good luck!
Ingestion / pipelines, Data modeling / transformation, Data quality / reliability, Infrastructure / platform work, ML / AI
I wish the wider industry understand how good data engineering enhances AI/ML.
Pressure to “move fast”, None / modeling is going well
Fancy tools and AI won't solve your data fundamentals; they're enablers, not magic sticks.
A quick fix will always come back to haunt you
Data security (regulated industry specific, AI-specific)
The balance between enforcing best practices and delivering a fast ROI is incredibly difficult.
Data teams need non-data stakeholders to be part of the data process. We can enable the business to use data effectively. We can't do it for them.
Big Enterprise Company ... Disjoint Orgs: Very little overlap between complementary but independent teams/tools - ETL/Orchestration related and BI/Analytics.
Data Consumption usage/patterns incl. Data Modeling and Analytics
Involve DEs in the end to end data lifecycle incl BI/Analytics. Avoid unnecessary pre-materialization if, in turn, it helps make Analytics more responsive in small data situations => Some minimal requirement changes/adjacent-to-existing analytical tasks can be enabled.
changing mindset from legacy excel reports
Data engineering is impossible in isolation, it takes regular collaboration with teams across the business to ensure that deliveries provide value
Ingestion / pipelines, Infrastructure / platform work, ML / AI
It is more about process and people than technology.
Data modeling / transformation, Analytics / BI, ML / AI
I wish my company and its consultancy clients understood that to build a solid product you need foundations and that takes time and compromise from both sides. There is still lack of knowledge regarding DE and Data Modelling best practices in many cases and that ends up with lousy deliverables.
Task completion throughput doesn't imply low team performance
Reducing complexity increases reliability. All these data modelling steps increase the amount of points of failure. We have a data vault model + kimball model, but it is very error prone, costly and slow. I do not think it was worth the effort.
Humans actually need to think about and understand the organisations data - AI isn't going to do that for us
It's extremely multifacted, and no one person or team can do EVERYTHING, so be comfortable with imperfection.
Data engineering must provide business value or it's a wasted effort
You can't buy an organizational data transformation- you have to own it which means swallowing humility and putting organic people in charge who actually know what good DataOps looks like
It's complex, has many interdependences and takes time
Be rigorous in software engineering best practices. Leaders do not recognize the importance of it.
Lack of leadership direction + talent/hiring
more clear understanding of what DE is supposed to do & more requirements (and knowledge transfers) about business context, not just "here's our SQL database, you can start working now"
Understand the impact of domain ownership, as an example it is not given that repositories can be combined in one central lake and it is not a case where IAM, RBAC, RLS or whatever is a solution.
Lakehouse w/federated ownership; matrixed in BU citizen DE and DS members
How much money is wasted in companies by not having a strategy in place to follow good data modeling practices that helps create reusable curated data structures. Large organizations will spend 10’s of millions in labor and system costs each year (BU’s each doing their own thing) while also having so much data sprawl that senior leadership teams struggle being data driven at the top. There is also a component of corporate alignment missing as well where data modeling and system design practices tend to follow org structure and culture - this is directly tied to data sprawl. However organizations can fix data sprawl by having a data-modeling-first strategy with senior leadership support (C suite and senior VP’s). This will help break down silos even when org structure naturally supports them (such as orgs built based on acquisitions).
We have domain expertise in data pipelines, AI/ML is an issue where more training is needed.
It’s not about metrics and KPIs but about being able to answer questions and the ability to ask “just one more thing” without any additional engineering effort.
Data Engineering/Modeling is a distinct discipline within, and even slightly outside of, software engineering. Just because the application you built uses Postgres, etc. on the backend does not mean you necessarily have a firm grasp of how those objects should be shaped and used for non-application use cases. This gap is evident whenever I've had to partner with backend engineers on projects.
Not all businesses are handling “big data”
Web data from ai first scraping platforms is inconsistent and cannot match the quality (yet) of traditional scraping platforms. Also, the standards and lawsuits around web data are threatening the open web.
That it is the backbone of the AI initiatives
Medallion “architecture” will not save you
Americans are paid way too much. Third worlders don't care about quality for cultural reasons. Hire in Europe for the single most highly skilled people at less than half the cost
Environment management for BI/data science teams (DevOps, but for teams that aren’t necessarily “dev”)
When you work in a massive organization, even “basic” data engineering such as moving data from a source system to a raw data lake is cumbersome. The fact that industry standards and solutions exist is not an indication that they are fully leveraged inside a massive org.
Data modeling / transformation, Data quality / reliability, Analytics / BI, ML / AI, Fighting fires
It's still the same old song, garbage in, garbage out.
Cloud scheduler, but will use dagster soon
I am in martech team. Bottleneck is waiting for data team to ingest data for us
That we need developers and other people to care about data quality for data if we are going to have data as a competetive advantage.
No one has fucking big data, your stupid app won't grow to Google levels of scale and if it will you will have enough money to rebuild the whole thing from scratch. Stop throwing spark on everything. Also, lakehouses are just dwh with extra steps, more potential mistakes and no actual benefit.
That the core to having good models and Gen Ai use case is still having robust , scalable and quality base data.
That good data takes time and strategy to do it well
Data modeling is more important than ever
That a tactical solution that uses a Kimball approach and a strategic solution that uses an Inmon approach will require a lot of time to reconcile. Lots of organisations fail to reconcile what should be global and local which ends with a messy glocal outcome which helps nobody.
Writing Code (SQL, Python, etc), Pipeline debugging, Data modeling, Architecture design
Getting hired is next to impossible because HR are using rudimentary data engineering to sort most people out
Custom metadata driven, running as Azure Functions
That self-service data engineering may misleadingly produce results faster, but the long term cost is prohibitive
Data modeling / transformation, Data quality / reliability, Infrastructure / platform work, Fighting fires
Data Engineering is not about tool switching based upon wants but tool hopping based upon requirements and architecture designs is ok.
Azure based, Databricks workflows, Data Factory
Importance of data modeling, and the fact that SQL is not dead.
In-house Orchestrator tool ( something similar to Airflow )
Two things: That data engineering isn't about the tools, it's about understanding the business problems you're solving. Said differently, because I always seem to need more words than the average bear. For effen sakes, stop chasing every new framework in the modern data stack. Tools come and go, but the core principles don't change. ....And just because you never learned them or forgot they exist, doesn't mean we don't know what they are. Look it up or ask someone to point you to those who do and just listen and learn.
That just replicating the data model from the source system is a recipe for disaster
That learning tool and hire based on tool experience is not a great way of hiring
Data Engineering is not an abstract version of the Software Engineering. Using the best practices from Software Engineering is critical while writing ETL | ELTs
It takes times, and doesn't work in agile mode
Data Quality needs combined effort from the upstream data producers to the consumers . Everything is not a data team problem to solve in the dark.
Data modeling / transformation, Analytics / BI, ML / AI, Fighting fires
Real blocker are never technical problems
Data engineering is not report sharing teams often come up to data team and ask for reports just for their presentation or validate their approach. A small team often get stuck in a loop just writing queries rather than focusing on pipelines
Backend analytics and operations need to be considered when making front end changes to systems
Bring business people on board and foster communication to build awareness of data
Data modelling is necessary but seems to have been forgotten
Ingested data does not mean that it can be served
I wish the industry understood that Data Engineers need to be domain experts, not just ticket-takers. Often, requests come in as simple data movement tasks, but technical skills alone aren't enough to guarantee reliability. To build trust, engineers need domain training just as much as they need technical specs. Without understanding the business context, we can only check for generic errors (like null values or record count checks). When we are trained in the domain, we can write business-specific Data Quality tests and robust reconciliation logic, ensuring that data doesn't just 'load' successfully, but actually matches the reality across source and target systems. Investing in this domain knowledge transforms us from utility providers into strategic partners who ensure data integrity.
That data engineering is far from data analytics, and that it entails much more than visualizations and reporting
Tech is table stakes. Modeling and good practices is the job. For new people, don't try to learn everything at once. Be strong at a couple of things then add skills incrementally, diving deep into some, in order to differentiate.
Data quality / reliability, Infrastructure / platform work, Analytics / BI
Tools are cattle, not pets. We shouldn't love them. If they've outgrown their usefulness, time to move on
For better data quality, we need business team’s help, otherwise even AI or more engineers won’t be of that much help
Most of the work can be software engineering. Software engineering tools like domain-driven design (to model more precisely the whole data governance processed and expectation of data products both for producers and consumers) and design patterns (hexagonal architecture, separation of concerns -business domain, application environment-, testability thanks to dependence inversion principles...) to reflect the process modelisation into the code in a clean and robust way. All starts with a good organization, a model of what is expected from business and other "parties prenantes" then a code that reflects all of that.
That data architects are not data engineers.
Documentation / data discovery, Architecture design, Governance / quality checks
we are consulting company, so all of the above can be found a each company.
Data Modeling, Streaming, Architecture, Carrer growth
I’d like to see more people talking about foundations; more articles and talks about timeless tools. Also, see projects where those foundations were applied and complemented using AI.
Difference between analytics engineering and data engineering - specifically the combination of skillsets and outcomes. As an example, I see data (software) engineers hired to build dbt models and analytics (BI) engineers being hired to build bespoke ingestion patterns.
Data engineering directly support business goals, hence, it's crucial to build solid long term data strategy.
That it is the most important role in the business...And they need more help and time to do things right, because no one else does anything right.
There’s no magic wand, despite what vendors tell you!
Ownership and lineage have huge ripple effects downstream that are often taken for granted. To use an analogy, whether a river is beautiful and clean or stinks and is full of garbage is whether people upstream can be convinced to stop dumping their trash in the river.
Understanding the purpose of systems you build is oftentimes more important than using state-of-the-art technology and algorithms.
Although Data Engineering has a lot in common and lot to learn from Software Engineering - not all practices are applicable in that domain, neither are these disciplines the same.
Tools and licenses do not solve problems. Data is king. Data Quality first. Domain literacy is equally important for engineers and to focus on why, before the what and the how.
The architecture and tooling is ahead of the use cases. The data is not as useful as people think and the modern data stack is way more advanced than the nature of the business problem. People spend too much on infrastructure without understanding the value of said infrastructure.
How role can impact in the organitation and connect with salary
It’s incredibly more complicated than traditional web and mobile app engineering.
Data engineers do not know what data modeling is
That there is no or very little engineering in "data engineering". I would never step on a bridge built like a pipeline...
Either Airflow, Dagster or Prefect (currently evaluating)
The database is not just an implementation detail. What I wish data engineers understood about data engineering: That tight coupling to any technology is limiting the platforms options (so please use appropriate abstractions when interacting with anything out of your control)
Value data engineering provides to business, there is no direct way of showing impact/metrics in numbers or ROI.
The difference between data architecture for Data analysis and for AI
Files from public data repository in various file formats etc
You need people who are skilled but also willing to up skill with times. Some organisations operates in dinosaur age and it’s a risk to ppl working with those systems, if they are out of job as you suddenly fit nowhere. Catch 22 situation. Burnt myself for being part of this setup. Also if your boss is a dinosaur, your up skill is with useless or a threat to him !
Data modeling / transformation, Data quality / reliability, Analytics / BI, ML / AI
There is not analytics/advanced analytics and/or AI/ML work without a proper data foundation, for which data modeling is absolutely critical and not done over night.
Importance of data modelling at early stages
That data is not technical but rather business artifacts. While tech increases stability in pipelines and reduces workload, communication, ownership and governance is, in my experience, where most projects fails.
Focusing on Business Semantics/Ontologies more than tools
That it is foundational to other investments
Data modeling / transformation, Infrastructure / platform work, Analytics / BI, ML / AI
Star Schemas are not the answer to everything. Data Modeling is not always synonymous with one single modeling approach.
It's more important than data science and data scientists rarely make good data engineers
Investing in data modeling, data quality, semantics, data governance could bring faster ROÍ from data projects and data teams in the long term
Data Lake + Power BI and custom reporting tools
Nearly all of the above (except for compute costs and tool complexity)
Building a solid data infrastructure does not happen overnight, and requires collaboration from the business side. If no one is available to help us understand the crucial human processes at the company, we are not going far. Also, data governance isn’t optional.
At scale there’s a high need for creativity especially when building on top of years of historical decisions that you didn’t make.
You need good data to do AI and therefore data engineers are more important than ever.
Our responsibilities dont fall into a clear box. Especially in government/ smaller organizations
Documentation / data discovery, Architecture design, I don't find AI helpful
data credibility (veracity to build trust) / create stable software (no more bugs please)
Not everything is to be builded over Spark. Some apps do not require new buzzwords oruse the cutting edge technolgy. Also, current architects do not understand the basics of kappa and lambda architectures. Stop thinking about AI is the solution, stop the hype and create stable sowtware that really works (stop thinking "build that now, latter we will smash those bugs" ignoring the technical debt in the mature stages)
Translating infrastructure value to non-technical stakeholders (aka: How not to be a data apologist)
Data engineering is infrastructure: when it works, it’s invisible; when it’s judged, it’s often through failing applications. Good pipelines, schemas, and governance aren’t perceptible as features to most business users, but there should be no doubt that they are non-negotiable prerequisites for any applications, analytics, and/or AI capabilities to succeed at all.
It takes time to build good and reliable data systems
Data Engineering can't fix business process and data stewardship issues.
Data modeling / transformation, Data quality / reliability, ML / AI, Fighting fires
Pain of adopting industry accepted data model will pay off long term vs. build stuff in house from scratch.
Data Engineering is adjacent but not equivalent to Information Technology (IT)
It's not just moving the data from one place to another, after "touching" the data you feel responsible for the quality and reliability of the output, and most of the leadership ignores this.
It is the foundation of good analytics and decision making
We are developing data platform, with hopes to apply federated ownership
Data engineering has very low visibility, but high impact
We spend a lot of time on things that do not shine, digging in data and see why we got a not from source in a column, and not even the source does not know about it
Data analytics team should be included right after the ideation process of a product whose analytics will be required. Don't expect us to magically present out of the box analytics for products or businesses you just told us about while excluding us from all previous workshops/brainstorming sessions. Sorry for ranting.
Data architecture / data engineering foundation is a non negotiable and cannot be replaced by fancy AI
Talent + extremely poor requirements causing refactoring, delays, etc
For my role, cloud understanding is the most important. Which includes lakehouse, semantic modelling, when to use a specific tool and why. Eg streaming data with checkpointing and idempotency checks vs. Batch to send data to a storage account. Cloud is the hardest exp to gain as typically the user has to front the cost to learn these tools
Keep it simple, master fundamentals of data eng'
Data Engineering tasks are not set and forget.
The value of good foundation data model and what is it look like.
How much of data engineering work is actually building relationships and convincing people to trust you.
Continuous improvement means continuos work, there is no "one-off and done" in data
The importance of data quality over speed. Rushing to deliver often creates technical debt.
Databricks + on-prem DW + data marts (i.e., mixed)
Having a semantic understanding of the data is immensely helpful
Data Ownership / Contracts / Data Quality
Data Architecture and Engineering consultant
Onboarding new domain teams effectively into our data mesh network - ensuring appropriate skill level and consistent is maintained
Tools don’t support good modeling, None / modeling is going well
There is difference between data engineering, data analytics and data science.
That consistent data engineering practices is how we operationalize data governance
Importance of Collaboration required to tackle Data Quality
Loads of manual labor are used in place of design and discipline.
Lack of clear ownership, Hard to maintain over time, Tools don’t support good modeling, Pressure to “move fast”, AI tools produce inconsistent schemas, None / modeling is going well
They need to drop adhoc mindset for a strategic approach.
Moving from data engineering focusing on analytics to data engineering serving different purposes (shared architecture with integration, modular architecture to serve different needs and refresh frequency)
Lack of clear ownership, Hard to maintain over time, Tools don’t support good modeling, Pressure to “move fast”, None / modeling is going well
The most difficult part about data engineering is getting the business to care about the input as much as the output.
Dv2.0 (silver) + Dimensional modelling (gold)
We are at the core of AI, stop building new things in top of shitty data if you don't want to obtain shitty results. Prioritize better, please.
The value add/benefits of quality data modelling
Data engineering is software engineering and demands the same engineering rigor (SDLC). Furthermore, data platforms must be treated as internal products, requiring a shift toward genuine product management mindsets.
Data management required components and prerequisites to use your organizations data.
Data Engineering looks different per each Data Engineer.
No good analytics without solid data engineering
That it is an engineering discipline not application support
That it's hard, thorny work that requires continuous investment to tame the entropy. "Another flaw in the human character is that everybody wants to build and nobody wants to do maintenance.” - Kurt Vonnegut
AI projects as the data engineer is the base line because AI need clean and meaning full data to train.
Data is not software, data acculturation is mandatory in an enterprise to have a big adoption and a better leadership in strategic, tactical and operational for the wealth care of the company
It's important part of data strategy. going forward data engineers will be in high demand
Depends on the client company and their current data maturity, business use cases and issues they are currently facing
That it is necessary and dealing with and processing data is not the same as software engineering practices and concepts. They have overlap but in data engineering we have a lot more to consider when it comes to dealing with large-scale data.
AI does not work well without proper data. We are not just plumbers, we can't magically do our work without requirements.
Data/ML/AI Consultant [Presales + Implementation Responsibilities]
Note: I work mostly in AI/ML so the models need to work. So in that light, Data Lineage and importance of your data being able to express the business accurately. There's so much stateless tribal institutional information not captured in the data. Good Data Models, then simple things like understanding how columns of data, even simply it's cardinality for instance, can really make or break the "expression" of that data. We need more thoughtfulness in how we express and capture data in the data models.
Vector Databases - AstroDB + WatsonX Orchestrate Knowledge Base
Better you know your data, faster you catch the issues